Using CARI and Behavior Coding to Evaluate QuestionnairesJoanne Pascal - PDF document

Using CARI and Behavior Coding to Evalua
Using CARI and Behavior Coding to Evaluate QuestionnairesJoanne PascaleUS Census BureauFederal CASIC WorkshopMarch 2223, 2011What is CARIComputer Audio Recorded InterviewRecords both CATI and CAPIinterviews

directly on to interviewers’laptops
directly on to interviewers’laptops What is Behavior Coding?Systematic method to describe & quantify interviewerrespondent interactionsEach turn of speech (“utterance”) by interviewer or respo

ndent is assigned standardized code(s)Op
ndent is assigned standardized code(s)Opentext notes complement codesAnalytic focus: frequency of behaviors; notes help interpret quantitative data Anatomy of an Interview“Utterance” = everything

one person says before the other person
one person says before the other person says something“Exchange” = pair of utterances: one from interviewer and one from respondentlevel” exchange = first pair of utteranceslevel” exchange

= second pair, etc.“Final outcome&
= second pair, etc.“Final outcome” = respondent’s final answer to the questionTypical Behaviors of InterestInterviewersAsking questions as worded or changing wordingVerifying or skipping ques

tionsProviding clarificationProbing for
tionsProviding clarificationProbing for more detailRespondents Providing straight answersProviding indirect or qualified answersAsking for clarificationOther response that does not answer questionEvaluating

QuestionnairesHow often are Qs asked as
QuestionnairesHow often are Qs asked as worded? What wording changes are most common?What are the consequences of wording How often do Rs give straight answers?How do these factors vary by:QuestionInterview

erTreatment (test control)Mode (CATI CAP
erTreatment (test control)Mode (CATI CAPI)Etc.Conventional Behavior CodingUsually used for CAPI since CATI has centralized monitoringInterviewers carried analog tape recorders and tapesRequested consent; tur

ned on recorder; recorded full interview
ned on recorder; recorded full interviewUnsystematic selection of cases recorded; usually interviewer judgment of respondent’s likelihood to cooperateRulethumb goal: record ~100 casesCARI Behavior Codin

gInterviews are recorded directly on to
gInterviews are recorded directly on to computers; no need for additional equipmentFeasible to record all interviews not just subsetCARI produces image of Q as displayed for interviewer and final data entry o

f R’s answerResearchers can sample
f R’s answerResearchers can sample at varying rates from entire universe: HHs/people based on known characteristicsSelect topic areas; particular questionsAmerican Community Survey (ACS)Replaces Census

“long form”Yearround data coll
“long form”Yearround data collection Annual data releasePersonPerson questionsMultiMail out/mail backCATICAPIACS Content TestTwo main goals for content test:explore improvements to existing Qs (pro

gram participation; veterans; wages)exam
gram participation; veterans; wages)examine alternative Qs for new content areas (computers/internet; place of birth)Data Collection: AugustSeptember: mail out/mail back OctoberNovember: CATI and CAPI intervi

ews conducted and recordedHH and R Char
ews conducted and recordedHH and R Characteristics HH and personlevel characteristics of interest varied by topic areaComputers: Urban vs ruralMore or less than HS educationParental Place of Birth:generation

Hispanic vs NonhispanicCase Allotment b
Hispanic vs NonhispanicCase Allotment by Topic AreaGoal: 1,600 cases (hh interviews) 800 CATI800 CAPI Within each mode:400 English cases100 program participation/computers100 parental place of birth100 wages

100 veterans400 Spanish casesACS Casel
100 veterans400 Spanish casesACS Caseload DispositionCATICAPISample released23,67315,202Completed + partial cases4,523(pending)Recording consent rate88.5%64.8%Cases recorded 4,0054,137Cases selected for be

havior codingACS ScopeFirst two exchang
havior codingACS ScopeFirst two exchanges & final outcome:Interviewer 1level exchangeRespondent 1level exchangeInterviewer 2level exchange (if any)Respondent 2level exchange (if any)Final outcomeCodes applie

d at utterancelevelMultiple aspects of e
d at utterancelevelMultiple aspects of each utterance coded (eg: behavior, language, interruptions)Language and ACSEnglishSpanish translationCrosscultural differences Multiple cultures within Spanishspeaking

populationInterviewers’ language p
populationInterviewers’ language proficiencyInterviewers’ language trainingInterviewer 1Level Initial QAskingAs wordedMajor changeVerificationSkipLanguage (English, Spanish, mix)Interrupted (if

applicable)Language Problems (if applica
applicable)Language Problems (if applicable)Interviewer 1Level DetailIf Major Change (check all that apply):Modified or dropped introductionModified or dropped reference periodOtherIf Verification (check one

):CorrectIncorrectUnclearRespondent Beh
):CorrectIncorrectUnclearRespondent Behavior and 2Levels)Answering BehaviorCodeable answerIndirect answerQualified answerUncodeable responseRequest for clarificationDK or RefusedLanguage (English, Spanish, m

ix)Interviewer 2LevelFollowup BehaviorP
ix)Interviewer 2LevelFollowup BehaviorProbe/clarify/verify correctlyProbe/clarify/verify incorrectlyProbe/clarify/verify (unclear)topic commentOther2. LanguageFinal Outcome GroupRespondent Answer (same as

in Respondent Behavior)Interviewer Data
in Respondent Behavior)Interviewer Data EntryMatches respondent answerDoes not match respondent answerRespondent Misunderstanding (if app)Overall Language Problem (if app)Interviewer 1Level Screennotesrequi

redRespondent Behavior ScreenFinal Out
redRespondent Behavior ScreenFinal Outcome ScreenRESULTS (hypothetical)Interviewer QAskingInterviewer 1LevelInitial QAskingExactMajor changeVerification (correct)Verification (incorrect)Verification (unc

lear)SkippedOtherInaudibleType of Major
lear)SkippedOtherInaudibleType of Major ChangeDropped/modified introductionDropped/modifiedreference periodOther major changeCategory A (fromopen text notes)Category B (fromopen text notes)Category C (fromo

pen text notes)etc.Respondent QAnswerin
pen text notes)etc.Respondent QAnsweringRespondent 1Level QAnsweringCodeableanswerIndirect answerQualified answerUncodeableanswerRequestfor clarificationDon’t knowRefusedOtherInaudibleSequence of Beha

viorsLevel ExchangeLevel ExchangeFinal
viorsLevel ExchangeLevel ExchangeFinal OutcomeIntAskingR QAnswerIntFollowupR AnswerR AnswerData EntryExactCodeableNoneNoneCodeableMatchExactUncodableCorrect probeCodeableCodeableMatchMajor changeCodeableNone

NoneCodeableMatchMajor changeRequest for
NoneCodeableMatchMajor changeRequest forClarifyCorrect probeCodeableCodeableMatchMajor changeRequest forClarifyIncorrect probecodeablecodeableNonMatchEtc.Sequence of LanguageLevel ExchangeLevel ExchangeIntA

skingR QAnswerIntFollowupR AnswerEnglish
skingR QAnswerIntFollowupR AnswerEnglishEnglishNoneNoneSpanishSpanishNoneNoneEnglishEnglishEnglishEnglishSpanishSpanishSpanishSpanishEnglishSpanishNoneNoneEnglishSpanishSpanishSpanishSpanishEnglishSpanishEngl

ishEtc.Subgroup AnalysisQuestion (test
ishEtc.Subgroup AnalysisQuestion (test vs. control)CATI vs. CAPIPerson number in householdCoderInterviewerTopic Area (and subgroups within)HH and R Characteristics by Topic Area Computers: Urban vs ruralMo

re or less than HS educationParental Pla
re or less than HS educationParental Place of Birth:generationHispanic vs NonhispanicCARI SummaryUnobtrusive easy recording mechanismLevels playing field between CATI & CAPIValuable sampling capabilitiesFlex

ibility in defining what behaviors to co
ibility in defining what behaviors to codeImage display offers:Exact display of wording (including fills)Interviewer’s actual data entry valueReal time monitoring of coding qualityData extractable at any