1 Survey Research Methods - PowerPoint Presentation

Slide1

1

Survey Research Methods

2. Reliability, Validity and Scale Construction

Steve Fisher, Robert AndersenandAnthony Heath Department of Sociology UNIVERSITY of OXFORDSlide2

2

The Logic of Sampling & MeasurementSlide3

3

General Problem of Measurement

Reliability & Validity

Reliability Refers to the replicability of the measurement procedure to yield consistent resultsValidity

R

efers to the extent to which the measurement

procedure actually measures the concept that it is

intended to measureSlide4

4

Different aspects of reliability

Test-retest, Inter-observer, Inter-item

Test-retest reliability Consistency of repeated measurements on the same subjects Used for concepts that are believed to be stable N.B. subjects may change; conditioningInter-observer reliabilityRepeated measures by different observers on the same subject

Especially important in coding open-ended questions

Inter-item reliability

Do the items in a composite measure correlate highly

Cronbach’s alphaSlide5

5

Types of Validity

1. Face and Content

Face validityBasically a subjective measure of validity Does it seem that we are measuring what we claim?Content validityDoes the content of the measuring instrument cover the full domain of the concept?

e.g

., measures of left/right ideology requires items tapping different but related things like redistribution, privatisation, government intervention

etc

. Slide6

6

Types of Validity

2. Criterion-related Validity

Criterion-related validityCorrelation with other measures known to have validity. (e.g., questionnaire measures of turnout validated against registers)Must know the criterion itself has been measured wellAppropriate criteria do not always exist

Predictive validity

Does our measure predict expected outcomes

(

e.g.,

Attitudes to taxes can be validated by their ability to predict electoral support for tax-cutting party. But other factors influence voting behaviour, so this is not clear-cut and more theory-dependent)Slide7

7

Types of Validity

3. Construct ValidityBased on a theoretical prediction about the relationship between the concept and other items.

Does the measured concept relate empirically to other measured variables in ways that are theoretically expected (i.e., does the measure yield the expected correlations?)Theory-laden and rather weak but at least it can always be attempted NB the lack of an expected correlation may reflect a bad theory or another measure involved was badly measured If your measure has been taken to test a theory you cannot use the same theory to test the construct validity of the measureSlide8

8

Reliability and validity

Some conclusionsReliability is relatively straightforward to demonstrate, validity is much more difficult and often theory-laden

You don’t necessarily have to demonstrate reliability and validity every time a measurement procedure is usedSometimes there will be existing measures which previous researchers have shown to be valid and reliable and you can simply borrow theseRemarkably few such measures in sociology and political science. See Heath, A. and J. Martin (1997) “Why Are There so Few Formal Measuring Instruments in Social and Political Research?” in Lyberg et. al (eds.) Survey Measurement and Process Quality. New York: Wiley.Slide9

9

Advantage of

Composite MeasuresBased on idea triangulation—several

reliable measures improves validityError scores tend to cancel out when we sum over items Gives greater variability in respondents’ scores

X = T + e

Item

X (observed score)

T (true score)

e (error)

1

2

3

5

5

8

6

7

7

+1

+2

-1

Total

18

20

+2Slide10

10

Measuring abstract concepts

Indices and scalesComposite measures of an abstract concept

Adds together scores assigned to several different measures of a constructAssumes an underlying continuum (i.e., there is structure in the data)—scores represent specific points along the continuumIndices: Individual measures can be distinct, but together they represent a larger abstract concept (e.g., Consumer Price Index, United Nations Index of the best Countries to live)

Scales

:

All indicators measure a

single dimensional

concept

(

e.g

.,

Likert scale

s measuring attitudes)Slide11

11

Measuring attitudinal intensity

Likert-scalesEvenly balanced response choices

Mix direction of questions to avoid response set biasStandard response formats typically have five categories for each itemStrongly agree, agree, disagree, strongly disagree, undecidedDefinitely like—definitely dislikeVery important

—very

unimportant

Definitely true

—definitely

false

Assign scores to the response categories

(1-5)

Sum together items (average if desired)Slide12

12

Using scales to measure political attitudes

For each statement, please circle the category that best reflects your opinion

I believe that I can help change the minds of public officialsSometimes politics and government seem so complicated that a person like me can’t really understand what is going onPeople should vote in elections because each individual vote can make a differenceGenerally, those elected to parliament soon lose touch with peopleThe government cares about what people like me think

I doubt that individual people like me could influence the platforms of political parties

Each of these items are Likert-items with response categories ranging from strongly agree to strongly disagree

Adapted from: Gray, G. and N. Guppy (1999).

Successful Surveys. Research Methods and Practice

. Toronto: Harcourt Brace, p.72. Slide13

13

Some other types of scales to consider

Feeling thermometers

Scale of 0 (very cold)-100 (very warm)Often used to determine how people feel about certain groups (used in AES forBogardus Social Distance ScaleMeasures the distance separating ethnic or other groups from each other Semantic Differential ScalesMeasures subjective feelings using polar opposite adjectives (e.g., light/dark, deep/shallow, modern/traditional, bad/good) Slide14

14

An example from the BSA

Measuring income satisfaction It is sometimes useful to give more detailed response categories

e.g., Which of the phrases on this card would you say comes closest to your feelings about your household’s income these days?Living comfortably on present incomeCoping on present incomeFinding it very difficult on present income

Other answer (WRITE IN)

(Don’t know)

Feel free to be creativeSlide15

15

Constructing Scales

Choose itemsCheck for dimensionality using factor analysisCompute a scale (i.e. sum the scores on the items, recoding if necessary)Check reliability of scale using Chronbach’s alpha and bivariate correlationsMake adjustments and test reliability again if necessarySlide16

16

How big an alpha?

Early stages of research, >0.7Individual level comparisons, >0.9Even with 0.9 the standard error of measurement is almost a third of the standard deviation of test scores. (Nunnally and Bernstein, 1994).Slide17

17

Group work

Spend the next session working in groups in Seminar room A or BAim is to design a set of questions that can be used to build a valid and reliable scale for your conceptBring a draft questionnaire with you next week