Towards more participative methods in the construction of composite indicators. The case of obtaining weights: from “objective” to “subjective” approaches

(1)

1

International Seminar on

Involving Citizens/Communities in Measuring & Fostering Well-being & Progress:

towards new concepts and tools

Palais de l’Europe, Strasbourg (France)

(2)

Involving Citizens/Communities

in Measuring & Fostering

Well-being & Progress:

towards new concepts and tools

International Seminar on

Palais de l’Europe, Strasbourg (France) November 27 – 28, 2008

(3)

3 Workshop 3

Links between traditional systems of

indicators and those developed with

citizens/communities:

antagonisms,

alternatives or

complementarities?

(4)

filomena.maggino @unifi.it

Towards more participative methods

in the construction of composite

indicators

The case of

weighting

systems

Filomena Maggino

(5)

5 1.Introduction

3.Obtaining weights:

objective ad subjective approaches

2.Obtaining weights:

general principles

4.Conclusions

(6)

1.Introduction

3.Obtaining weights:

objective ad subjective approaches

2.Obtaining weights:

general principles

4.Conclusions

(7)

7 GOAL

GOAL

legitimacy

of Social

of Social Indicators

of Social

Indicators

HOW ?

involving

individuals

in the

in the process

process

GOAL

_{Obtaining a larger}_legitimacy_legitimacy_{of social indicators}

How to obtain this ?

(8)

Methodology

⇓

“

Technology

Technology”

⇓

Specialistic

Specialistic Training

Training

⇓

Scientific

and

Objective

Way

The methodology aimed at constructing indicators refers to and deals with the term “technology”, pointing out the need to have specialistic training in order to apply the procedure in a scientific

(9)

9 the procedure

the procedure

is

far

_far

from

_from

being

_being

objective

and

aseptic

Actuallythe procedure,the procedure,

even though scientifically defined,

is

(10)

Indicators

Indicators construction

construction

Stages

⇓

Decision

Decision /

/

/ Choice

Choice

⇓

Technical / Value judgments

Indicators construction

is developed through differentstages.

⇓ ⇓⇓ ⇓ ⇓ ⇓ ⇓ ⇓

Each stage requires

a decision / choice(methodological or not) to be taken. ⇓ ⇓⇓ ⇓ ⇓ ⇓ ⇓ ⇓

Some decisions are quite technical others may involve value judgments

(11)

11 Objective decisions :

Objective decisions :

quite difficult

generally

Scientific community:

accepted and shared

but

Larger community:

can we share this ?

It is quite difficult to make these decisions objective

Generally

they are taken through a process accepted and shared by the scientific community

But

(12)

Choosing

Choosing…

…

1.

1. 1. …

…

… analytical

…

analytical

analytical approach

approach

2.

2. 2. …

…

… and

…

and

and obtaining

obtaining

obtaining weights

obtaining

weights

3.

3. 3. …

…

… and

…

and

and identifying the

identifying the

identifying the aggregating

identifying the

aggregating

technique

4.

4. 4. …

…

… models

…

models

models and

and conceptual

and

conceptual

conceptual approaches

conceptual

approaches

approaches in

in

order

order to

to

to assess

to

assess

(i)

(i) robustness

robustness

((((iiiiiiii) ) ) ) discriminant

discriminant

discriminant capacity

capacity

In

particular

…

1. choosing analytical approach in order to verify the underlying dimensionality of selected elementary indicators (dimensional analysis) 2. choosing and obtaining weights in order to define the importance of

each elementary indicator to be aggregated (weighting criteria)

3. choosing and identifying the aggregating technique in order to synthetize the elementary indicators values into composite indicators (aggregating-over-indicators techniques)

4. choosing models and conceptual approaches in order to assess

1. the robustness of the synthetic indicator in terms of capacity to produce correct and stable measures (uncertainty analysis, sensitivity analysis)

2. the discriminant capacity of the synthetic indicator (ascertainment of selectivity and identification of cut-point or

(13)

13 Weights

Weights

differential importance

⇓

not simply a technical problem

⇓

Weights

Judgment values

Weightsin indicators construction aim at assigning differential importanceto the indicators to be aggregated. ⇓ ⇓⇓ ⇓ ⇓ ⇓ ⇓ ⇓

this process does not represent simply a technical problem

⇓ ⇓⇓ ⇓ ⇓ ⇓ ⇓ ⇓

(14)

Weights

⇓

differential

subjective

weights

How

(15)

15 Determining

Determining

and

applying

differential

subjective

weights

⇓

conceptual

framework

how to

obtain

weights

assign

weights

Determining

Determining and and applyingapplyingdifferentialdifferentialsubjectivesubjectiveweightsweights ⇓ ⇓ ⇓ ⇓ ⇓ ⇓⇓ ⇓

solid conceptualconceptualframeworkframeworkhelping in clarifying how

_to _obtain_obtain_obtain_obtain _{importance weights at individual-subjective level through}

subjective judgments

(16)

1.Introduction

3.Obtaining weights:

objective ad subjective approaches

2.Obtaining weights:

general principles

4.Conclusions

(17)

17 General statements

∑

=

K

j

ij

i

x

w

CI

1

CI CI CI

CI_iiii composite indicator for case i

KKKK number of indicators to be aggregated

xxxx_ijijijij indicator j to be aggregated for case i w

ww

(18)

Basic conditions

The identified weights

_{non negative}

_{add up to unity}

_{related to the corresponding}

_score

_{rescaled to}

_{identical range}

_{(0; 1)}

General statements

General basic conditions

The identified weights are non negativenumbers

add up to unity

are related in some way to the corresponding score

(19)

19 R

R

Reproduc

eproduc

eproducing

ing

ing each

ing

each CI sub

each

CI sub

CI sub----score

score

score’’’’ssss contribution

contribution

criterion

⇓

weighting system

⇓

adopted

adopted measurement model

measurement model

improvement & refinement

General statements

In order to reproduce as accurately as possible the contribution of each sub-score to the construction of CI

a criterion has to be adopted to define a weighting system a criterion has to be adopted to define a weighting systema criterion has to be adopted to define a weighting system a criterion has to be adopted to define a weighting system

improvement and refinement of the adopted model of measurement improvement and refinement of the adopted model of measurementimprovement and refinement of the adopted model of measurement improvement and refinement of the adopted model of measurement.

(20)

Generic

Generic weighting

weighting

weighting system

system

system identification

identification

needs to

take into account :

rationale and theoretical

framework

meaning and contribution of each

sub

sub----score

score

quality of data

and

statistical adequacy

of

indicators

General statements

Generic weighting system identification needs to take into account :

rationale and theoretical frameworkon which the measurement of the complex characteristics is founded and that will consequently regard the synthetic score

meaning and contribution of each sub-scoreto the synthesis

(21)

21 proportional size

proportional size

a.

equal

or

differential

weighting

aggregation technique

b.

compensatory

or

non

----

compensatory

Generic

Generic weighting

weighting

weighting system

system

system identification

identification

needs

decisions

General statements

Generic weighting system identification needs decisions to be taken

proportional size of weights

a.

a.equalequal or differential weightingdifferential

aggregation technique to be adopted

b.

(22)

N.B.

AAAA

whole set of weights

does not exist

General statements

N.B.

Awhole set of weightswhole set of weights

able to express in a perfect perfect way

the contribution of each indicator

does not exist

(23)

23 Equal vs. differential weighting

Equal vs. differential weighting

(24)

First decision

Weighting

Equal

Different

The choice will strongly influence

the final results

(25)

25 _different

_different

_{different components}

_components

different

different numbers

numbers

synthetic score = unbalanced structure

_indicators

_{indicators exist measuring the same}

_{exist measuring the same}

component

double weighted or double counting

Doubtful

procedure mainly

when

Equal Weighting

General statements

different components have to be aggregated by different numbers of indicators

synthetic score = unbalanced structure

indicators exist measuring the same component

(26)

_{theoretical reflections}

_{methodological concerns}

Doubtful

procedure mainly

when

not supported bynot supported bynot supported bynot supported by

General statements

Differential Weighting

•

•theoretical reflectionstheoretical reflections on the meaning and impact of each indicator on the theoretical reflectionstheoretical reflectionstheoretical reflectionstheoretical reflectionstheoretical reflectionstheoretical reflections synthesis,

•

•methodological concernsmethodological concerns aimed at identifying proper and consistent methodological concernsmethodological concernsmethodological concernsmethodological concernsmethodological concernsmethodological concerns techniques.

(27)

27 Compensatory and

Compensatory and

Non

Non----compensatory

compensatory

aggregation techniques

(28)

In order to avoid incoherencies between In order to avoid incoherencies between In order to avoid incoherencies between In order to avoid incoherencies between

theoretical meaning actual application theoretical meaning actual application theoretical meaning actual application theoretical meaning actual application

of weights of weights

⇓

a consistent aggregating technique is needed a consistent aggregating technique is neededa consistent aggregating technique is needed a consistent aggregating technique is needed

by byby

by cccconsideronsideronsideringonsideringing compensability among ing compensability among compensability among compensability among the elementary indicators

the elementary indicators the elementary indicators the elementary indicators

(29)

29 Compensatory

Compensatory

aggregating approach

additive additive additive additive _{(simple addition)}

geometrical geometrical geometrical geometrical _{(multiplicative technique)}

⇓

low values compensated by high values low values compensated by high values low values compensated by high values low values compensated by high values

⇓

synthetic score does not allow us to return to the original individual profiles

problems of interpretation

problems of interpretation 

problems of interpretation



(30)

In obtaining

ssss

ubjective

weights

the decisions / choices

do not rely on experts

Obtaining subjective

(31)

31

_{theoretical issue}_{theoretical issue}_{theoretical issue}_{theoretical issue}

“importance” is a distinct construct ?

_{psychometric properties of importance ratings}_{psychometric properties of importance ratings}_{psychometric properties of importance ratings}_{psychometric properties of importance ratings}

internal consistency and test-retest reliability Subjective weighting system identification

needs to take into accounttake into accounttake into accounttake into account :

Obtaining subjective

weights statements

(32)

_criterion_criterion_criterion_criterion

_level_level_level_level

_techniques_techniques_techniques_techniques

_approach_approach_approach_approach

Subjective weighting system identification : a modelmodelmodelmodel should be chosen by considering

Obtaining subjective

weights statements

Identification of a subjective weighting system: a modelshould be chosen by considering

the criterion of importance or preference to be adopted

the level at which weights are determined and applied (individualindividual or individualindividualindividualindividualindividualindividual groupgroupgroupgroupgroupgroupgroupgroup weights)

the techniques allowing subjective evaluations and judgments to be collected (explicitly or implicitly)

the approach allowing a subjective importance/preference continuum to be constructed

(33)

33 1.Introduction

3.Obtaining weights:

objective ad subjective approaches

2.Obtaining weights:

general principles

4.Conclusions

(34)

Can

produce weights:

a.

statistical statistical statistical statistical methodsmethodsmethodsmethods

b.

multimultimultimulti----attribute attribute attribute approachesattribute approachesapproachesapproaches

c.

scaling approachesscaling approachesscaling approachesscaling approaches

Obtaining weights

approaches

““““objective

objective

objective””””

⇒

““““subjective

subjective

subjective””””

⇒

scaling approaches scaling approaches scaling approaches scaling approaches can

- handle subjective evaluations and judgments, explicitly or implicitly expressed

- obtain subjective weights at group level and at individual level.

(35)

35 a.

a.

a. Statistical methods

Statistical methods

Obtaining weights

approaches

(36)

Weights Weights Weights

Weights are are are are determineddetermineddetermined throughdetermined throughthroughthrough

1. 1.1.

1. CorrelationCorrelationCorrelation AnalysisCorrelation AnalysisAnalysisAnalysis 2.

2.2.

2. Principal Component AnalysisPrincipal Component AnalysisPrincipal Component AnalysisPrincipal Component Analysis 3.

3.3.

3. Data Envelopment AnalysisData Envelopment AnalysisData Envelopment AnalysisData Envelopment Analysis

the same data on which weights will be applied the concept of efficient performance

Statistical methods

(37)

37 b.

b.

b. Multi

Multi

Multi----attributes approaches

attributes approaches

Obtaining weights

approaches

(38)

Multi

Multi----Attribute Models

Attribute Models

allow

subjective importance weights to be identified at subjective level

through an indirect approach by

Multi-attributes

approaches

• managing a certain number of combined comparisons

• applying methods aimed at making decision among different available alternatives

Weights obtained through these methods are considered more stable than those produced by direct evaluations.

(39)

39

1. 1. 1.

1. MultiMultiMulti----Attribute Decision MakingMulti Attribute Decision MakingAttribute Decision MakingAttribute Decision Making:

_{Analytic Hierarchy Process}_{Analytic Hierarchy Process}_{Analytic Hierarchy Process}_{Analytic Hierarchy Process}_(AHP) pairwisepairwise comparisonpairwisepairwisecomparisoncomparisoncomparison 2.

2. 2.

2. MultiMultiMulti----Attribute Compositional ModelMulti Attribute Compositional ModelAttribute Compositional ModelAttribute Compositional Modelssss:

_{Conjoint Analysis}_{Conjoint Analysis}_{Conjoint Analysis}_{Conjoint Analysis}_(CA)

Among these models we can distinguish:

Multi-attributes

approaches

(40)

c.

c. Scaling approaches

Scaling approaches

Obtaining weights

approaches

(41)

41

_{Dimensionality}_{Dimensionality}_{Dimensionality}_{Dimensionality}

_{Nature of data}_{Nature of data}_{Nature of data}_{Nature of data}

_{Scaling technique}_{Scaling technique}_{Scaling technique}_{Scaling technique}

_{Criterion for testing the model}_{Criterion for testing the model}_{Criterion for testing the model}_{Criterion for testing the model}

_{Standard of measurement}_{Standard of measurement}_{Standard of measurement}_{Standard of measurement}

_{Contribution to the measurement of each}_{Contribution to the measurement of each}_{Contribution to the measurement of each}_{Contribution to the measurement of each}

multiple multiplemultiple

multiple measuresmeasuresmeasuresmeasures

They

They can

can

can be

be

be classifi

be

classified through

classifi

ed through

ed through their

their

their features

their

features

features::::

(42)

(43)

43

_wh_wh_wh_{whoooose nature is comparative or preferential}_{se nature is comparative or preferential}_{se nature is comparative or preferential}_{se nature is comparative or preferential}

_((((in_in_in_{in yellow in the previous table)}_{yellow in the previous table)}_{yellow in the previous table)}_{yellow in the previous table)}

_{produced by a comparative scaling technique}_{produced by a comparative scaling technique}_{produced by a comparative scaling technique}_{produced by a comparative scaling technique}

_{((((in pink in the previous table)}_{in pink in the previous table)}_{in pink in the previous table)}_{in pink in the previous table)}

Among all these approaches we

need

to

select those approaches that utilize data:

(44)

_{group weighting}_{group weighting:}_{group weighting}_{group weighting}_{group weighting}_{group weighting}_{group weighting}_{group weighting}

Thurstone model ₍differential scale₎

Unfolding model ₍perceptual mapping₎

_{individual weighting}_{individual weighting:}_{individual weighting}_{individual weighting}_{individual weighting}_{individual weighting}_{individual weighting}_{individual weighting}

Conjoint model

Scaling models allowing subjective

weights to be obtained

weights to be obtained are

are

Scaling approaches

In our perspective, these models can be distinguished with reference to the possibility to define subjective weights at individual level or at group level (last column of the previous table), in particular:

--group weightinggroup weightinggroup weightinggroup weightinggroup weightinggroup weightinggroup weightinggroup weighting: Thurstone model(differential scale), unfolding model

(45)

45 1.Introduction

3.Obtaining weights:

objective ad subjective approaches

2.Obtaining weights:

general principles

4.Conclusions

(46)

This work aims at This work aims at This work aims at This work aims at

systematically framing the issue and systematically framing the issue and systematically framing the issue and systematically framing the issue and showing the possible approaches in order to showing the possible approaches in order to showing the possible approaches in order to showing the possible approaches in order to obtaining weights in a subjective perspective obtaining weights in a subjective perspective obtaining weights in a subjective perspective obtaining weights in a subjective perspective

⇓

anticipating a anticipating a anticipating a

anticipating a research proposalresearch proposalresearch proposalresearch proposalresearch proposalresearch proposalresearch proposalresearch proposal we are going to we are going to we are going to we are going to define define define define

⇓

clarifying many technical issues clarifying many technical issuesclarifying many technical issues clarifying many technical issues

(47)

47 Difficulties

Difficulties

Obtaining subjective weights requires and relies Obtaining subjective weights requires and relies Obtaining subjective weights requires and relies Obtaining subjective weights requires and relies on the accomplishment of large survey projects on the accomplishment of large survey projects on the accomplishment of large survey projects on the accomplishment of large survey projects

aimed at collecting aimed at collecting aimed at collecting

aimed at collecting ““““importanceimportanceimportance”””” dataimportance datadatadata

_Time_Time_Time_Time

_Resources_Resources_Resources_Resources

_Sampling_Sampling_Sampling_Sampling

_Field_Field_Field_{Field work}_work_work_work

_…_…_…_…

(48)

«Constructing composite indicators should take into account the agreement among citizens concerning the importance to be

assigned to each indicator» (Hagerty and Land, 2007)

Seen in this perspective, this topic can be placed in the ambit of an improvement of

democratic participation to decisions (“res publica”)

(49)

49

Presentation designer: [email protected]