Measurement Jessica Goldberg - PowerPoint Presentation

Measurement Jessica GoldbergAssistant Professor of Economics at University of MarylandAffiliate of J-PAL

Course Overview What is Evaluation? Theory of change Measurement Why Randomize?How to Randomize?Sampling and Sample SizeThreats and AnalysisGeneralizabilityStart to Finish j-pal | Measurement & Indicators 2

Measurement Outcomes, Indicators, and Data

Demand for acute healthcare for children Setting: Poor, peri-urban areas in Mali with limited sanitation and infrastructure; high rates of child morbidity and malnutrition Operating area of Mali Health, a health NGO in Bamako Program: Free clinic visits and medications for children under 5 at public clinicsHealth worker visitsMeasurement: Daily(!) data on symptoms, drugs, and doctor visitsExample: Healthcare in Mali j-pal | Measurement & Indicators 3

Lecture Overview What to measurePurpose of measurementTheory of changeHow to measure Sources of measurement Measurement concepts Response processChallenges to measurementPrimary data Administrative dataEthics and IRBj-pal | Measurement & Indicators5

Lecture Overview What to measurePurpose of measurementTheory of change How to measure Sources of measurement Measurement conceptsResponse processChallenges to measurementPrimary data Administrative dataEthics and IRBj-pal | Measurement & Indicators6

Lecture Overview What to measurePurpose of measurement Theory of change How to measure Sources of measurementMeasurement conceptsResponse processChallenges to measurementPrimary data Administrative dataEthics and IRBj-pal | Measurement & Indicators7

Purpose of Measurement (I) j-pal | Measurement & Indicators 8

Lecture Overview What to measurePurpose of measurement Theory of change How to measure Sources of measurementMeasurement conceptsResponse processChallenges to measurementPrimary data Administrative dataEthics and IRBj-pal | Measurement & Indicators9

Theory of Change j-pal | Measurement & Indicators 10

Lecture Overview What to measureTheory of change Purpose of measurement How to measure Sources of measurementMeasurement conceptsResponse processChallenges to measurementPrimary data Administrative dataEthics and IRBj-pal | Measurement & Indicators11

Lecture Overview What to measureTheory of change Purpose of measurement How to measure Sources of measurementMeasurement conceptsResponse processChallenges to measurementPrimary data Administrative dataEthics and IRBj-pal | Measurement & Indicators12

Where can we get data? From existing sources (Secondary data) Publicly available Administrative dataOther secondary dataCollected by researchers(Primary data)Surveys Non-survey methodsj-pal | Measurement & Indicators13 https://commons.wikimedia.org/wiki/File:Cuyahoga_County_US_Census_Form-Herbert_Birch_Kingston_1920.jpghttps://commons.wikimedia.org/wiki/File:US_Navy_090123-N-9760Z-004_Hospital_Corpsman_2nd_Class_Jennifer_Ross_files_medical_records_aboard_the_aircraft_carrier_USS_Nimitz_(CVN_68).jpg

Types and Sources of Data Information about a person/ household NOT about a person/ household Information reported by a person Automatically generated J-PAL | Measurement & Indicators 14 Cognition, anthropometrics Demographics Behavior, beliefs Patience, risk aversion, psychometrics Knowledge Income, expenditure Farming inputs and outputs Quality of medical care Business income taxes Bank transactions Phone data Sales records School/university records, criminal record Prices Weather, air quality Stock markets VAT records

Primary Data Collection Surveys Exams, tests, etc. Games VignettesDirect observationDiaries/logsFocus groupsInterviewsj-pal | Measurement & Indicators15

Primary Data: Modes Interviewer administeredPaper-basedComputer-assisted/ digitalTelephone-based Self-administered Paper Computer/digitalTask platforms like Amazon Mechanical Turkj-pal | Measurement & Indicators16

Primary Data: Who, Where, When? Survey respondent Who has the most information for question of interest? Who is most likely available? Who is least likely to be biased?Example: head of household vs. person at home, mother vs. healthcare provider Survey settingIs the respondent distracted?Will peers influence responses?Example: interview at clinic vs. at home, interview at school vs. at homeSurvey timingCould the survey affect behavior?Example: conduct health test just before or after health worker visit 17 j-pal | Measurement & Indicators

Suppose you want to learn how often each child was visited by a health worker. Who would you ask? The health worker The health worker’s supervisor The child The motherNo one -- use the health worker’s phone recordsj-pal | Measurement & Indicators18

Administrative data Information collected, used, and stored for administrative (i.e., operational) purposes, rather than research purposes.Medical records Grade books Arrest records Bank account dataPersonnel recordsLog booksj-pal | Measurement & Indicators19

Administrative Data: Sources HealthVital statistics officeHealth facilities (e.g., hospitals, clinics) Finance Banks, credit unions Credit rating agenciesEducationSchools and universitiesDepartment of educationJ-PAL’s Catalog of US data sets J-PAL’s collaboration with governments, e.g. in Tamil Nadu (India)Administrative records of program being evaluatedj-pal | Measurement & Indicators20

Why are administrative data useful? The outcomes and metrics required for a study may already be tracked by a government or organization Available retrospectively Enable long-term follow-up May include near census of relevant populationReduce logistical burden and burden on subjectsOften less expensive than surveysMay reduce bias and errorj-pal | Measurement & Indicators21

What source of data do you primarily expect to use in your work? Primary survey dataOther primary (non-survey) dataAdministrative dataPublic data Other secondary data j-pal | Measurement & Indicators 22

Lecture Overview What to measureTheory of change Purpose of measurement How to measure Sources of measurementMeasurement conceptsResponse processChallenges to measurementPrimary data Administrative dataEthics and IRBj-pal | Measurement & Indicators23

Concept of Measurement j-pal | Measurement & Indicators 24 Bullseye represents the construct

Concept of Measurement j-pal | Measurement & Indicators 25 Bullseye represents the construct represents the indicator

Concept of Measurement j-pal | Measurement & Indicators 26 Bullseye represents the construct represents the indicator Many blue dots represent the data

Concept of Measurement j-pal | Measurement & Indicators 27 Construct (Intelligence/ cognitive development) Indicator (IQ Test result) Data (Recorded test scores) https://commons.wikimedia.org/wiki/File:Red_Silhouette_-_Brain.svg

Concept of Measurement j-pal | Measurement & Indicators 28 Construct (Women’s empowerment: Agency) Indicator (Female political participation) Data (Number of women speaking at general village council meetings) Rachel Glennerster *, Claire Walsh, Lucia Diaz-Martin (2018) A practical guide to measuring women’s and girls’ empowerment in impact evaluations

Concept of measurement What type of information represents the construct best? How do we collect this information? How precisely will the recorded information represent the true information? j-pal | Measurement & Indicators29 Construct Indicators Data Collection Data

The Goals of Measurement j-pal | Measurement & Indicators 30 Validity: Accuracy/ Unbiasedness

Validity In theory: how well does the indicator map to the construct we are trying to measure? j-pal | Measurement & Indicators 31 Construct Indicators

The Goals of Measurement j-pal | Measurement & Indicators 32 Validity: Accuracy/ Unbiasedness Precision Reliability

Reliability Is the indicator measured in a way that is consistent and precise? Would we get the same data if we measured several times? j-pal | Measurement & Indicators33 Construct Indicators Data Collection Data

Outcome: weekly consumption Indicator: expenditure in last weekWhich is likely to be the main problem? Validity Reliability BothNeitherJ-PAL | Measurement & Indicators34

The Goals of Measurement j-pal | Measurement & Indicators 35 Reliability Validity

Lecture Overview What to measureTheory of change Purpose of measurement How to measure Sources of measurementMeasurement conceptsResponse processChallenges to measurementPrimary data Administrative dataEthics and IRBj-pal | Measurement & Indicators36

The Response Process j-pal | Measurement & Indicators 37 Error/ Bias Error /Measurement bias Construct Indicators Data Collection Data

Four-Step Response Process j-pal | Measurement & Indicators 38

J-PAL | measurement 39 How many times did you consume rice this this month? Step 1: Comprehension ?

J-PAL | measurement 40 Step 2: Retrieval How many times did I consume rice this month? Step 2: Retrieval How many times did I consume rice this month?

J-PAL | measurement 41 Step 3: Estimation/Judgement 2x +~2x +~2x +~2x =~8x

J-PAL | measurement 42 How many times did you buy rice this this month? Step 4: Response 0, 1, 2-3, 4-6, 7-12, 13+ How many times did you buy rice this this month? Step 4: Response 0, 1, 2-3, 4-6, 7-12, 13+ 0, 1, 2-3, 4-6, 7-12, 13+

Which stage in the response process might produce measurement error? ComprehensionRetrievalJudgement/estimationReportingAll of the above J-PAL | Measurement & Indicators 43

Lecture Overview What to measureTheory of change Purpose of measurement How to measure Sources of measurementMeasurement conceptsResponse processChallenges to measurementPrimary data Administrative dataEthics and IRBj-pal | Measurement & Indicators44

Lecture Overview What to measureTheory of change Purpose of measurement How to measure Sources of measurementMeasurement conceptsResponse processChallenges to measurementPrimary data Administrative dataEthics and IRBj-pal | Measurement & Indicators45

Measurement Error: Social Desirability Bias Example: Q. Were you arrested in the past month? Yes NoRespondents might hesitate to admit arrestsTendency of respondents to answer questions in a manner that emphasize strengths, hide flaws, or avoid stigmaAsk indirectly, ensure privacyj-pal | Measurement & Indicators 46

Measurement Error: Context effects Anchoring Bias Example: Q. In South Africa, some voters reported having to wait more than 5 hours to vote. How long did you have to wait last time you voted? No time (there was no line, or I voted by mail) Less than 10 minutesBetween 10 minutes and 30More than 30 minutes but less than an hourAn hour or moreRespondents will be more likely to give a number on the higher end of the spectrum. Especially if this requires estimation and judgment.Avoid adding anchors to your questions j-pal | Measurement & Indicators 47

Measurement Error: Framing Example:Q1. How many years of education do you have? Q2. Did you go to public or private school? Q3. Did your school provide everyone a quality education? Q4. For the upcoming election, what are the top policy priorities you are looking for from the candidates?Suddenly, education becomes everyone’s top priorityPhrasing of questions or location in the questionnaire changes respondents focus or way of thinkingBe careful of where questions are placed and if they are phrased to express implicit judgement.j-pal | Measurement & Indicators 48

In Ghana young women spend 25 hours per week on unpaid care and services (includes child care) on average. How long do women in your household spend per week caring for their children? 0 – 10 hours 10 – 25 hours More than 25 hours Don’t knowPrefer not to answerJ-PAL | Measurement & Indicators49Counting Women’s work Policy brief 1 (2018) Available at: http://www.cww-dpru.uct.ac.za/sites/default/files/image_tool/images/74/CWW_PB201801.pdf

Measurement Error: Telescoping Bias Example: Many health surveys in developing countries ask about birth history: Q. Did you give birth to a child who passed away within the last year? Respondents may report very salient events that happened more than 12 months agoPeople perceive recent events as being more remote than they are (backward telescoping) and distant events as being more recent than they are (forward telescoping)Visit once at the beginning of the reference period, ask the question. Then ask, “since the last time I visited you, have you…?”j-pal | Measurement & Indicators 50

Other things to consider Question wordingSpecific and easy to understandAvoid double negativesBack-translate and pre-test in other languages Multiple choice questions: answer options that Cover all possible answers Do not overlapIf possible, do not require “check all that apply”Survey length and respondent fatigueSurveyor training/qualityPilotingj-pal | Measurement & Indicators51

Lecture Overview What to measureTheory of change Purpose of measurement How to measure Sources of measurementMeasurement conceptsResponse processChallenges to measurementPrimary data Administrative dataEthics and IRBj-pal | Measurement & Indicators52

Access to Administrative Data Regulations that limit access to identified data South Africa POPIA : Protection of Personal Information ActNigeriaThe Freedom of Information Act of 2011 in section 14 Obligations depend on level of identification and your contract with the data providerData security requirementsFines for data leaksIndividual authorizations or waivers (similar to informed consent) j-pal | Measurement & Indicators 53

Access to Administrative Data: Generating an Electronic File Records are in an unusable formatHand-written records PDF file j-pal | Measurement & Indicators 54To address records in an unusable formatDigitize Clinic ledgers in Mali Tax records in Bangladesh Books and academic papers worldwide

Challenges for measurement Data accuracyReporting biasSelection biasDifferential coverage j-pal | Measurement & Indicators 55

Using Administrative Data: Matching Goal: match administrative with program or survey data Personally Identifiable Information (PII) Name, Identification numbers, Address Photos or biometric characteristicsUnderstand what identifiers the data agency collectsCollect those same identifiers from your study sample at baselineUse numeric identifiers instead of string variablesIdentification number and DOB instead of names and addressesParticipants may not be willing to provide sensitive identifiers E.g. bank account or social security numbers -- emphasize privacy & confidentiality during study enrollment j-pal | Measurement & Indicators 56

Using Administrative Data: Separate Identifiers from Outcomes j-pal | Measurement & Indicators 57

Matching process Exact/DeterministicMinor discrepancies are not well accounted for E.g., typos in name, reversed day and month in DOB Some records are not identified as matches even though they may be (false negatives) j-pal | Measurement & Indicators58 FINDER FILE ADMINISTRATIVE DATA RECORD NAME DOB RECORD NAME DOB 1 Jane Doe 5/1/1950 A Jane Doe 1/5/1950 2 Jonathan Smith 7/1/1975 B John Smith 7/1/1975 3 Sarah Franklin 8/23/1987 C Sarah Franklin 8/23/1987 ü û û

Matching process Fuzzy/ProbabilisticAccounts for the likelihood that identifiers may not align exactly to those in a data system E.g., SSN and last name match, DOB is off by a month  count as a match j-pal | Measurement & Indicators59 FINDER FILE ADMINISTRATIVE DATA RECORD NAME DOB RECORD NAME DOB 1 Jane Doe 5/1/1950 A Jane Doe 1/5/1950 2 Jonathan Smith 7/1/1975 B John Smith 7/1/1975 3 Sarah Franklin 8/23/1987 C Sarah Franklin 8/23/1987 ü ü ü

When choosing identifiers for matching study data to administrative data, which of the following identifiers would be preferable to using an individual’s street address? An email address A government-issued, unique identification number Date of birth All of the aboveB and C60j-pal | Working with Administrative data

How can we ensure that the data are accurate? Unlike with survey data, the researcher does not have a say in the data collection and processing phaseUsing Administrative Data: Determining Data Accuracy 61 j-pal | Measurement & IndicatorsTarget by Sergey Demushkin from Noun Project

To address possibly inaccurate data Cross-reference with other sources to ensure accuracyIdentify the data agency’s quality control protocolChoose indicators that are unlikely to be incorrectly reportedSelect variables that are straightforward and less susceptible to human error Request raw variables Communication between program or implementing partner responsible for collecting data and research team Ask how and why data are collectedUsing Administrative Data: Determining Data Accuracy62j-pal | Measurement & Indicators

Unlike survey data, administrative data are not susceptible to bias. True False 63 j-pal | Working with Administrative data

Using Administrative Data: Reporting Bias From an individual E.g., under-reporting income to qualify for a social welfare programFrom an administrative organization E.g., schools over-report attendance to meet requirements j-pal | Measurement & Indicators 64Teacher by Danil Polshin from the Noun Project – Similar to survey data, but with potentially stronger incentives + Much administrative data is not self-reported

Using Administrative Data: Reporting Bias To address reporting biasIdentify the context in which the data were collected Were there incentives to misreport information? Choose variables that are not susceptible to bias E.g., hospital visit v. value of insurance claim65j-pal | Measurement & Indicators

Using Administrative Data: Selection Bias Administrative records only from individuals or organizations in contact with the administration in questionProgram recipientsApplicants Partner schools and hospitals Etc. Ask: what is the reason for the organization to collect these data?j-pal | Measurement & Indicators66– Unlike with survey data, but no control over selection process+ Some administrative data has near-universal coverage

Using Administrative Data: Differential Coverage Between treatment and control: Differential ability to link individuals to administrative recordsDifferential probability to appear in administrative recordsE.g., victimization as measured by calls to report crime j-pal | Measurement & Indicators 67 Actual impact: 10 unit r eduction in victimization Measured impact: 10 unit increase in victimization

Using Administrative Data: Differential Coverage To address selection and differential coverage bias Identify the data universe Which individuals are included in the data and which are excluded, and why? Identify how the intervention may affect the reporting of outcomes Determine direction in which differential selection might occur and how this might bias effect estimatesCollect baseline survey with identifiers for linkingTo ensure that you are equally likely to link treatment and control individuals to their records and identify differential coverage68j-pal | Measurement & Indicators

Lecture Overview What to measureTheory of change Purpose of measurement How to measure Sources of measurementMeasurement conceptsResponse processChallenges to measurementPrimary data Administrative dataEthics and IRBj-pal | Measurement & Indicators69

Ethics and IRB Two separate questions:Is the program I am evaluating ethical?Is it ethical for researchers to create a control group, who do not get the programme?Are the research procedures ethical? What’s the difference? Research on human subjects is subject to legal oversight (IRB review)Potentially harmful programs implemented by others should be evaluated (even if perhaps not implemented)Data collection and potentially random assignment must be reviewedj-pal | Measurement & Indicators70

Ethics and IRB Institutional Review Boards (IRBs) for research on human subjectsRequired for all research that receives support, directly or indirectly, from the United States federal government Belmont Principles Respect for persons Informed consentProtection for vulnerable populationsBeneficenceMaximize benefits and minimize possible harmsDo not harmProtect subjects’ privacyJustice Benefits and burdens of research should not go to different groupsj-pal | Measurement & Indicators71

IRB Procedures Informed ConsentSubjects must consent to the use of their primary or administrative dataThey should agree to study procedures with full knowledge of risks and benefits IRBs and/or data providers may require that individuals consent to each specific data set that may be usedWaiver of informed consentDifferent levels of scrutiny depending on potential risks and populations involvedAll J-PAL studies must be reviewed by an IRB.j-pal | Measurement & Indicators72

Thank you!