Recommendations for Eurostat and NSIs WP4 ROME April 11 th 12 th 2019 MIMOD MixedMode Designs for Social Surveys FINAL WORKSHOP Dag F Gravem Statistics Norway MIMOD project MixedMode Designs in Social Surveys ID: 809148
Download The PPT/PDF document "Mixed-mode questionnaire designs:" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIsWP4
ROMEApril 11th | 12th 2019MIMODMixed-Mode Designs for Social SurveysFINAL WORKSHOP
Dag F. GravemStatistics Norway
Slide2MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
AgendaOverview of WP4Presentation of deliverablesRecommendations for discussionDiscussant Anette Björnram, Statistics SwedenOpen discussion
Slide3Overview of WP4Main contributorsStatistics Norway (WP leader)CBSIstatSupporting contributorDestatisMixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide4Overview of WP4Recommendations on approaches for developing web questionnaires for mixed-mode surveysQuestionnaire levelQuestion levelQuestionnaire elements“Unimode vs. mode-specific questionnaires”Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide5Overview of WP4Feasibility of mixed-mode questionnaires and usefulness of MIMOD recommendations dependent on several factorsSurvey practices and infrastructureSurvey communication optionsSurvey content and requirements (from Eurostat)Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide6WP4 DeliverablesA paper on mixed mode combinations and experiences at participating NSIsA paper on survey communication in mixed-mode ESS surveysA paper with recommendations for key questionnaire elements, questions and question types in mixed mode settingsMethodological report Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide7WP4 Deliverable 1Mixed mode experiences at European NSIsData from MIMOD surveyMixed mode combinations in useQuestionnaire differencesOpinions on surveys not fit for CAWIGood and bad practicesNational case studiesStats Norway: Mixed mode in a historical perspectiveIstat: Recent experiences from census and aspects of life surveyCBS: The “Omnimode” questionnaire design systemMixed-mode questionnaire designs: Recommendations
for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide8Deliverable 1: Mixed mode combinations in use
Slide9Deliverable 1: Concurrent, sequential & combined mixed mode data collection
Slide10Deliverable 1: Questionnaire differencesError and consistency checks: many, but smallDon’t know and Item nonresponse: many, and some large Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide11Deliverable 1: Fitness of key ESS surveys for CAWILFS: Lack of interviewer supportEU-SILC: Length of surveyMixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide12Deliverable 1: Good practicesGeneral adviceWhen introducing mixed-mode at your NSI, start with a simple survey Gradually introduce web sample to go mixed-mode Pretest, pilot, measure effects Login and completion assistanceMake login as easy as possibleEstablish and man a helpdesk at the times respondents are activeMixed-mode questionnaire designs: Recommendations for Eurostat and
NSIs
Slide13Deliverable 1: Good practicesQuestionnaire developmentAdapt questionnaires stronglyEnsure smartphone complianceUse a respondent-centric web design approachModularize questionnairesAdd open questions to surveys asking for suggestions for improvementMixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide14Deliverable 1: Good practicesCommunication strategiesFind the right communication strategy for each surveyUse ONE web address for all surveysUse login info for routing respondent to the correct oneAdd a brochure with web completion instructions to your cover letterUse recurrent remindersShorten e-mail notificationsUse lotteries for boosting CAWI responseMixed-mode questionnaire designs: Recommendations for Eurostat and
NSIs
Slide15Deliverable 1: Good practicesData collection organizationFind the right data collection strategy for each surveyUse an online first approachDo CAWI reclaim directly in CATICustomize breakoff strategiesIn CAPI/CATI designs, use the same interviewer for the same respondentKeep CAWI option open after introducing other modesTechnical aspectsMonitor servers constantlyEnable restart at breakoff pointsMixed-mode questionnaire designs: Recommendations
for Eurostat and NSIs
Slide16Deliverable 1: Bad practices (don’ts)Don’t send too many remindersDon’t send reminders indiscriminatelyDon’t introduce differences that can lead to bias and time series breaksDon’t underestimate case management and IT infrastructure costsDon’t make questionnaires too longMixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide17National case studies: Statistics NorwayOngoing transition from interviewer-administered to CAWI single mode or mixed mode with CAWIStarting with smaller, non-ESS surveysSurvey specific data collection strategiesAdult Education Survey the first ESS survey with mixed mode data collection
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide18National case studies: IstatEliminating paper questionnaires for the next census (CAWI/CATI/CAPI design)Development of household CAWI questionnaire for the Aspects of daily life surveyPrivacyMixed-mode questionnaire designs:
Recommendations for Eurostat and NSIs
Slide19National case studies: CBSFactors leading to questionnaire differences in conversions to mixed mode:Not being able to start afreshFear of time series breaksSpecific client demands Technical restrictions and limitations.Consequences for the rest of the organization.Time and money restrictions. Mixed-mode questionnaire designs: Recommendations
for Eurostat and NSIs
Slide20National case studies: CBSCBS guidelines for omnimode designKeep the stimulus – question and answer categories – the same across modes Redesign using unimode principles if possible. Don’t use one mode as the primary one. Use a screen-by-screen design to keep the respondent focussed on one question at a time … Unless they are filter or follow-up questions Repeat question texts on next page/screen to give the complete stimulus again Always include all answer options in questions to be read out loud, including No opinion, Refused or Don’t know. Avoid grids if possible, and ask series of questions in all modes instead Minimize instructions and explanations, and present them similarly across modes, preferably in the form of questions Mixed-mode questionnaire designs: Recommendations
for Eurostat and NSIs
Slide21WP4 deliverable 2A paper on survey communication in mixed-mode ESS surveysThe concept of contact modes in survey literatureNSI experiencesCommunication strategiesUse of digital communicationPush-to-web strategiesNational case studiesISTAT’s communication strategyStatistics Norway’s use of digital concact communicationCBS: Push-to-web communication and use of incentives
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide22Deliverable 2: Contact modes Contact phase mode changeFollow-up phase mode changePush-to-web and communication mode variationMixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide23Deliverable 2: NSI experiencesUniform communication strategiesMixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide24Deliverable 2: NSI experiencesTailored communication strategiesMixed-mode questionnaire designs: Recommendations for Eurostat and
NSIs
Slide25Deliverable 2: NSI experiencesReasons why digital communication is little usedLack of e-mail addressesLach of mobile phone numbersQuestionnaire not adapted to smartphoneData protectionHousehold survey designLack of web data collection experienceNegative experiences with use of SMS notifications before CATI callbacks
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide26Deliverable 2: NSI experiencesPush-to-webMany different combinations and protocols, butPractically all sequential mixed-mode survey with CAWI are CAWI first, Followed by interviewer-administered modesReduction of field costs the main motivator
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide27Deliverable 2: More on push-to-webNicolaas and Smith (2017): motivational aspects paper-> web. Motivation to open the mailing Personalisation, type of mailing – enveloped, postcard etc., Appearance before opening.Motivation to read the mailing Personalisation, Easy to read – length, font, vocabulary, appearance – e.g. important, professional Motivation to take part in the survey Clarity about purpose of mailing and survey request, use of persuasive reasons for taking part – including incentives
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide28Deliverable 2: More on push-to-webMotivation to go online Clear instructions for logging into the questionnaire Minimal effort needed to enter login details Multiple access methods; e.g. any internet-enabled device Motivation to complete questionnaire Authentic looking landing page with clear instructions Designed for mobile: short, reduction of clutter and text Avoid question types prone to breakoff: complex question formats and cognitively difficult questions Mixed-mode questionnaire designs:
Recommendations for Eurostat and NSIs
Slide29Deliverable 2: National case studiesIstatRespondent-oriented lettersEmphasizes the importance of the respondent, not the surveyBalances legal requirements, practical information and Focus on readabilityRespondents’ pages on IstatStandardized survey descriptionsSurvey materials: advance letter, questionnaire, instructions, templates
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide30Deliverable 2: National case studiesStats NorwayUse of digital contact communication – no paper letters anymoreRegister with verified e-mail and mobile phone numbersCAWI questionnaires: direct links and two-factor loginData on contact mode use needs to be better integrated in survey management system (see also WP3)
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide31Deliverable 2: National case studiesCBSPush-to-web communication with persons and households: inventory of experimentsAdvance letters, reminders, flyers, envelopes, QR codes, and incentives. Lower linguistic complexity always goodOtherwise, different approaches work better for different subgroups: tailoring!Incentive experiments in LFS with positive resultsIncreased responseBetter representativenessBetter data
qualityCost savingsMixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide32Deliverable 3: recommendations for key questionnaire elements, questions and question types in mixed mode settingsTheoretical frameworks for mixed mode analysisAssessment of questionnairesEurostat model questionnaires and documentationNational implementationsResults from prestesting performed for MIMODICT, EHIS, EU-SILC, AES, LFSAppendix: Test report from Statistics Norway
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide33Deliverable 3: Theoretical frameworksUnimode, mode-specific, generalized mode, mode enhanced designDe Leeuw & Hox: Different design conventions – question format effects Often mode specific designs by survey practicioners Experimental designs often try to minimize question differencesMixed-mode questionnaire designs: Recommendations for Eurostat and
NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide34Deliverable 3: Theoretical frameworksMIMOD predecessor DCSS (Körner et al. 2013)Key factors for differential measurement Type of social interaction – interviewer involvement, respondent controlType of communication – verbal, non-verbal, para-verbal, computer mediatedQuestionnaire design options – visual vs. aural stimulusComputer assistanceHigh risk question typesSensitive, difficult, long, many response options,
complex skip instructionsMixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide35Deliverable 3: Theoretical frameworksCampanelli et al. (2013): Question Characteristics Relevant to Measurement ErrorMode recommendations for 29 different characteristics of individual questions grouped byQuestion content – topic, sensitivity, inherent difficultyQuestion format – open, closed, ratio, ordinal, nominalOne question may have several and conflicting characteristics
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide36Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide37Table of moderecommendations
Slide38Deliverable 3: Campanelli continuedQuestion types not recommended for all modes1) Sensitive questions 4) Subjective, non-sensitive scalar questions 6) Unconstrained textual/verbal open questions 10) Open questions with interviewer coding 14) Mark all that apply response format 16) Ranking questions 18) Visual analogue questions 20) Questions with a high number of response categories 24) Questions with edit checks 29) Questions with show cards
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide39Deliverable 3: Mixed mode pretesting by Statistics NorwayTest-retest designTest subjects were first interviewed in CATI modeThen responded to the same questions in CAWI mode about a week laterRetrospective probingEyetrackingTest of unimode design: Entire ICT questionnaireTest of mode specific design for problematic questions according to Campanelli criteria: Selected questions from ICT, EHIS, EU-SILC, AES and LFS surveys
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide40Assessment of questionnaires: ICT questionnaireICT survey regulation contains no recommendations or limitations regarding which mode to useBut: Model questionnaire presupposes a visual formatBut CATI most used (16 NSIs)Followed by CAWI (15 NSIs)
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide41ICT model questionnaire exampleMixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide42Assessment of questionnaires:ICT questionnaireApplication of Campanelli typology on model questionnaire indicates possible issues A. Mark all that apply format (14) B. Inherent difficulty due to concepts (6) C. Use of instructions and clarification (23) D. Use of Don’t know (25) E. Question length (Dillman)
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide43ICT: National implementationsNetherlandsCATI/CAPI/CAWI designOmnimode approach: everything visible on a screen will be read out loudNorwayCATI onlyEmbedded in 35-minute omnibus«Check all that apply» grids converted to yes/no sequences
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide44ICT questionnaire PretestingStats Norway: Both unimode (omnimode) and mode specific approach testedUnimode: entire questionnaireMode specific: selections with «mark all that apply» and inherently difficult questionsMobile unimode based on existing CATI questionnairePC mode specific version based on model questionnaire
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide45Slide46ICT pretesting: main findingsUnimode: General problems more important than mode-related problemsOverall verbosity – redundant phrasesInsufficient operationalization – too theoretical languageTransformation to yes/no increases questionnaire length and makes it difficult for respondents to have an overviewMode specific PC:Indicate that «check all that apply» on PC can be combined with
yes/no sequence in CATI for the questions that were tested Should be tested quantitatively Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide47Assessment of questionnaires: EHISEHIS: face-to-face modes described as «preferred modes» in methodological manualShow cards suggestionsSeparate document on CATI implementation of questions on alcohol consumptionCAPI used by 15 NSIs, CAWI by 11Note: literature does not recommend face-to-face for personal interviews
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide48Assessment of questionnaires: EHIS Application of Campanelli typology on model questionnaire indicates possible issues A. Sensitive questions (1) B. Long “yes/no for each” question batteries (15) C. Show cards (29) D. Inherent difficulty due to difficult terms and calculations (5)
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide49MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019EHIS question pretestingQuestion on alcohol comsumptionMode-specific test-restestCATI: sequence
of questionsCAWI: Single questions with nine optionsIndication of equivalenceBut 9 options too much for mobile ..?Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Slide50Assessment of questionnaires: EU-SILCMethodological guidelines state that five modes of data collection are possible, PAPI, CAPI, PASI and CAWI«Priority to be given to personal interviews»CATI not covered in legal basis, but allowed on «gentleman’s aggreement» in case of person rather than household samplesRecommendations presuppose interviewer-administered modes: No discussion
on how to implement for self-administeredCATI and CAPI most widely usedMixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide51EU-SILC pretestingMode-specific test-retest of questions on mortgagesInterviewer for motivating and assisting in CATIVisual presentation and automatic calculations in CAWIInspired by business survey questionnaire design
MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide52EU-SILC questionnaire prestestingVisual presentation: increased perceived response burdenSatisficingCalculations with subtractions not intuitiveMeasurement errorLayout not suited for mobile screensDevice effects in addition to possible mode effects?
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide53Assessment of questionnaires: Adult Education Survey (AES)Reference questionnaire and survey guidelines are availableVariable INTMETHOD has nine values Pros and cons of different data collection methods are discussedCAPI and other interviewer-administered modes recommended in guidelinesBut reference questionnaire often presupposes a visual modeCAPI most used mode (17), followed by CAWI (11) and CATI (10)
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide54Assessment of questionnaires:AES questionnaireApplication of Campanelli typology on model questionnaire indicates possible issues A. Mark all that apply format (14) B. Inherent difficulty due to long and complex questions (5) C. Inherent difficulty due to long recall period (5)
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide55AES pretestingMode-specific CAWI test of questions on hours of trainingVisual presentation and automatic calculationsMixed-mode questionnaire designs:
Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide56AES questionnaire pretestingNon-mode specific issues persistWhat should be considered one learning activity?Should breaks be included or excluded?Difficulties calculating average length of daysCalculator does not solve these problemsCan even enhance them with an extra task of evaluation
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide57Assessment of questionnaires: LFSNo model questionnaire, but minimum reqirements in terms of variables found in explanatory notesNo mention of CAWIShow cards mentioned as the «essential difference» between CAPI and CATI …CAPI most common data collection method in LFS W1 (22) CATI most common in W2+ (15)CAWI considered unsuitable for LFS W1 by 20 NSIsRecruitment and identificationNeed for
interviewer clarificationMixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide58Application of Campanelli typology on model questionnaire indicates possible issues A. Inherently difficult questions – including recall (5) B. Open questions with interviewer coding (10) C. Use of instructions (23) D. Show cards (29)
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Assessment of questionnaires: LFSMIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide59LFS pretestingMode-specific test-retest of question on actual working hoursInterviewer for motivating and assisting in CATIVisual presentation and automatic calculations in CAWI
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide60LFS prestestingSimilar setup tested previouslyStatistics Finland in the DCSS project – positive pilot resultsEquivalent results with CATIONS – negative results from cognitive and usability testsAnnoys respondentsSatisficingStats Norway’s results very similar to UK experienceE.g. rounding off to full hours every day (see figure)
Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide61Recommendations and conclusions for discussionDifficult to offer recommendations that contradict existing Eurostat requirements and recommendationsRebuild all model and national questionnaires and documentation with mixed mode in mind – modes will be mixed!Start from scratch instead of continuing on an existing patchworkShorten, modularize, simplifyTake unimode as a
starting pointBut continue exploring mode specific solutions using cognitive and usability testing, plus pilotingMixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019
Slide62Recommendations and conclusions for discussionThe Campanelli typology: useful, but needs updatingLittle in terms of CAWI specific recommendationsOther tools for identifying and analyzing problematic questions?Better facilitation of rapid exchange of experiences and recommendationsIncluding mixed mode pretesting protocolsRisk
of being outdated before publicationDocumentation of surveys and questionnaire fragmentedA better tool than Cros-portal and Circabc?Wiki-based web page?Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs MIMOD project - Mixed-Mode Designs in Social SurveysRome, 11-12 April 2019