Towards more participative methods in the construction of social indicators: survey techniques aimed at determining importance weights

(1)

September 11-13, 2009

Lausanne, Switzerland

Swiss Foundation for Research in Social Sciences

&

the University of Lausanne

World

Association

for Public Opinion

Research

Public Opinion and

Survey

Research

in a

Changing

World

(2)

Filomena Maggino

filomena.maggino@unifi.it

Towards more participative

methods in the construction of

social indicators

:

_:

survey

techniques

aimed

at

determining

importance

(3)

1.Introduction

3.Obtaining weights:

objective ad subjective approaches

2.Obtaining weights:

general principles

4.Conclusions

(4)

1.Introduction

3.Obtaining weights:

objective ad subjective approaches

2.Obtaining weights:

general principles

4.Conclusions

(5)

How to obtain this ?

By involving

By involving individuals

individuals

in the process of

social indicators construction

GOAL

(6)

Methodology

Methodology aimed

aimed

aimed at

at

at constructing

at

constructing

indicators

⇓

“

technology

technology”

technology

⇓

need

need to

to

to have

have

have specialistic

specialistic training in

specialistic

training in

training in order

order

order to

to

apply

apply the procedure in a

the procedure in a

scientific

and

objective

(7)

Actually

the procedure,

even

even though

though

though scientifically

scientifically

defined

defined, , , ,

is

far

_far

from

_from

being

_being

objective

(8)

Indicators

Indicators construction

construction

is

is developed

developed

developed through

through

through different

different

stages

⇓

Each

Each stage

stage

stage requires

requires

a

decision

decision / / / / choice

decision

choice

((((methodological

methodological or

methodological

or

or not

not

not) ) ) )

to

to be

be

be taken

taken

⇓

Some decisions are quite technical

others may involve value judgments

(9)

It is quite difficult to make these decisions

objective

Generally

they are taken through a process accepted and

shared by the

scientific community

But

can we share methodological decisions /choices

by

(10)

1.

1. 1. choosing

choosing

choosing analytical

analytical

analytical approach

analytical

approach

2.

2. 2. choosing

choosing

choosing and

and

and obtaining

and

obtaining

obtaining weights

obtaining

weights

3.

3. 3. choosing

choosing

and

and identifying the

identifying the

aggregating

aggregating technique

technique

4.

4. 4. choosing

choosing

choosing models

models

models and

and

and conceptual

conceptual

approaches

approaches in

in

in order

order

order to

to

to assess

assess

(i)

(i) robustness

robustness

((((iiiiiiii) ) ) ) discriminant

discriminant

discriminant capacity

capacity

In

(11)

Weights

in indicators construction aim at

assigning differential

importance

to the indicators

to be aggregated.

⇓

this process does not represent simply a

technical problem

⇓

weights judgment values

(12)

Weights

⇓

differential

subjective

weights

How

(13)

Determining

and

_and

applying

_applying

differential

_differential

subjective

weights

⇓

solid

conceptual

framework

helping in clarifying how

_{to obtain}

_obtain

_weights

_{to assign}

_assign

_weights

(14)

1.Introduction

3.Obtaining weights:

objective ad subjective approaches

2.Obtaining weights:

general principles

4.Conclusions

(15)

General basic conditions concerning weights

The identified weights

_are

_{non negative}

_numbers

_{add up to unity}

_are

_related

_in

_some

_way

_to

_the

corresponding

score

_{may require to be rescaled in order to have}

an

identical range

(0; 1)

(16)

Reproducing each sub-score’s contribution to CI

a criterion has to be adopted

weighting system

improvement and refinement of the adopted

model of measurement

model of measurement.

(17)

Identification of a generic weighting system needs to take into accounttake into accounttake into accounttake into account :

_{rationale and theoretical framework}

_{on which the}

measurement of the complex characteristics is founded and that will consequently regard the synthetic score

_{meaning and contribution of each sub}

_{meaning and contribution of each sub----score}

_{meaning and contribution of each sub}

_score

_{to the}

synthesis

_{quality of data and statistical adequacy of}

indicators

(18)

proportional size of weightsproportional size of weightsproportional size of weightsproportional size of weights

a.

equal

equal or

equal

differential

differential weighting

differential

aggregation technique to be adopted aggregation technique to be adopted aggregation technique to be adopted aggregation technique to be adopted

b.

compensatory

compensatory or

compensatory

non

----

compensatory

Identification of a generic weighting system needs decisions to be taken

(19)

N.B.

AAAA

whole set of weights

able to express in

a

perfect

perfect way

_perfect

way

the contribution of each indicator

General statements

does not exist

(20)

Equal vs. differential weighting

(21)

First decision

Equal Weighting

_{Different Weighting}

The choice will strongly influence

the final results

(22)

• different components different components different components different components havehavehavehave toto betoto bebebe aggregatedaggregatedaggregatedaggregated by by by by different numbers of indicators

different numbers of indicators different numbers of indicators

different numbers of indicators ( synthetic score = unbalanced structure);

• indicators exist measuring the same componentindicators exist measuring the same componentindicators exist measuring the same componentindicators exist measuring the same component (

double weighted

o

double counting

).

Doubtful

procedure mainly

when

Equal weighting

(23)

Differential weighting

•

• theoretical reflectionstheoretical reflectionstheoretical reflectionstheoretical reflectionstheoretical reflectionstheoretical reflectionstheoretical reflectionstheoretical reflections •

• methodological concernsmethodological concernsmethodological concernsmethodological concernsmethodological concernsmethodological concernsmethodological concernsmethodological concerns

Doubtful

procedure mainly

when

not supported bynot supported bynot supported bynot supported by

(24)

Compensatory and

Non

Non----compensatory

compensatory

aggregation techniques

(25)

In order to avoid incoherencies between

theoretical meaning actual application

of weights of weights

a consistent aggregating technique is needed

by considering

compensability

among

the elementary indicators

(26)

Compensatory

aggregating approach

- additive approachadditive approachadditive approachadditive approach (simple addition)

- geometrical approachgeometrical approachgeometrical approachgeometrical approach (multiplicative technique)

low values compensated by high values

synthetic score does not allow us to return to the original individual profiles

problems of interpretation

problems of interpretation 



(27)

In obtaining

ssss

ubjective

weights

the decisions / choices

do not rely on experts.

Statements in obtaining

subjective weights

(28)

_{theoretical issue}_{theoretical issue}_{theoretical issue}_{theoretical issue}

“importance” is a distinct construct?

_{psychometric properties of importance ratings}_{psychometric properties of importance ratings}_{psychometric properties of importance ratings}_{psychometric properties of importance ratings}

internal consistency and test-retest reliability Identification of a subjective weighting system

needs to take into accounttake into accounttake into accounttake into account :

Statements in obtaining

subjective weights

(29)

the criterion the level

the techniques the approach

Identification of a subjective weighting system: a modelmodelmodelmodel should be chosen by considering

Statements in obtaining

subjective weights

(30)

1.Introduction

3.Obtaining weights:

objective ad subjective approaches

2.Obtaining weights:

general principles

4.Conclusions

(31)

Approaches

for

obtaining

weights

can produce:

a.

statistical statistical statistical statistical methodsmethodsmethodsmethods

b.

multimultimultimulti----attribute attribute attribute approachesattribute approachesapproachesapproaches

c.

scaling approachesscaling approachesscaling approachesscaling approaches

Approaches for obtaining

weights

““““objectiveobjectiveobjectiveobjective weightsweightsweights””””weights

⇒

(32)

a.

a. Statistical methods

Statistical methods

Approaches for obtaining

weights

(33)

SSSS

SSSStatisticaltatisticaltatisticaltatisticaltatisticaltatisticaltatisticaltatistical methodsmethodsmethodsmethodsmethodsmethodsmethodsmethods

preferred when the choice of weights relies preferably on “objective” principle:

1. 1.1.

1. CorrelationCorrelationCorrelationCorrelation AnalysisAnalysisAnalysisAnalysis (CA)(CA)(CA)(CA) 2.

2.2.

2. Principal Component AnalysisPrincipal Component AnalysisPrincipal Component AnalysisPrincipal Component Analysis (PCA)(PCA)(PCA)(PCA) 3.

3.3.

3. Data Envelopment Analysis (DEA)Data Envelopment Analysis (DEA)Data Envelopment Analysis (DEA)Data Envelopment Analysis (DEA)

(34)

Statistical approaches are useful usefuluseful useful useful useful useful useful

in order to identify differential relative weights

to be assigned to each sub-score that will be considered in the synthetic score

Applicability of statistical models to obtain subjective weights Applicability of statistical models to obtain subjective weights Applicability of statistical models to obtain subjective weights Applicability of statistical models to obtain subjective weights

(35)

Statistical approaches has to be carefullycarefullycarefullycarefullycarefullycarefullycarefullycarefully considered

since by removing any control on the weighting procedure from the analysts gives a

false appearance of mathematical objectivity

false appearance of mathematical objectivityfalse appearance of mathematical objectivity

false appearance of mathematical objectivity

false appearance of mathematical objectivity false appearance of mathematical objectivity false appearance of mathematical objectivity false appearance of mathematical objectivity

that is actually difficult to achieve in social measurement

Applicability of statistical models to obtain subjective weights Applicability of statistical models to obtain subjective weights Applicability of statistical models to obtain subjective weights Applicability of statistical models to obtain subjective weights

Moreover

Statistical approaches do not completely meet the requirements of adopting subjective weights

(36)

b.

b. Multi

Multi

Multi----attributes approaches

Multi

attributes approaches

Approaches for obtaining

weights

(37)

Multi

Multi----Attribute Models

Attribute Models

allow

subjective importance weights to be identified at

subjective level

through an indirect approach

by

Multi-attributes

approaches

• managing a certain number of combined comparisons

• applying methods aimed at making decision among different available alternatives

(38)

1. 1. 1.

1. MultiMultiMulti----Attribute Decision MakingMulti Attribute Decision MakingAttribute Decision MakingAttribute Decision Making:

_{Analytic Hierarchy Process}_{Analytic Hierarchy Process}_{Analytic Hierarchy Process}_{Analytic Hierarchy Process (AHP) (pairwise}

comparison)

Among these models we can distinguish:

2. 2. 2.

2. MultiMultiMulti----Attribute Compositional ModelMulti Attribute Compositional ModelAttribute Compositional ModelAttribute Compositional Modelssss:

_{Conjoint Analysis}_{Conjoint Analysis (CA)}_{Conjoint Analysis}_{Conjoint Analysis}

Multi-attributes

approaches

(39)

c.

c. Scaling approaches

Scaling approaches

Approaches for obtaining

weights

(40)

DimensionalityDimensionalityDimensionalityDimensionality Nature of dataNature of dataNature of dataNature of data

Scaling techniqueScaling techniqueScaling techniqueScaling technique

Criterion for testing the modelCriterion for testing the modelCriterion for testing the modelCriterion for testing the model Standard of measurementStandard of measurementStandard of measurementStandard of measurement

Contribution to the measurement of each multiple Contribution to the measurement of each multiple Contribution to the measurement of each multiple Contribution to the measurement of each multiple measures measures measures measures

They

They can

can

can be

be

be classifi

classified through

classifi

ed through

ed through their

ed through

their

their features

features

features::::

(41)

wh

whoooose nature is comparative or preferential

se nature is comparative or preferential

((((in in in in yellow in the following table)yellow in the following table)yellow in the following table)yellow in the following table)

produced by a comparative scaling technique

(in pink in the following table) (in pink in the following table) (in pink in the following table) (in pink in the following table)

Among all these approaches we

need

to

select those approaches that utilize data:

(42)

(43)

-- group weightinggroup weightinggroup weightinggroup weighting: group weightinggroup weightinggroup weightinggroup weighting

-

Thurstone model

(

differential scale

) -

Unfolding model

(

perceptual mapping

)

-- individual weightingindividual weightingindividual weightingindividual weighting: individual weightingindividual weightingindividual weightingindividual weighting