/
Review of WP 4 -  Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs Review of WP 4 -  Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs

Review of WP 4 - Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs - PowerPoint Presentation

greemeet
greemeet . @greemeet
Follow
348 views
Uploaded On 2020-08-06

Review of WP 4 - Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs - PPT Presentation

Discussant Anette Björnram Statistics Sweden Brief presentation Working at the cognitive lab at Statistics Sweden since12 years questionnaire design and ID: 799800

cawi question hours questions question cawi questions hours mode cati day unimode worked approach test check recommendations specific format

Share:

Link:

Embed:

Download Presentation from below link

Download The PPT/PDF document "Review of WP 4 - Mixed-mode questionnai..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.


Presentation Transcript

Slide1

Review of WP 4 - Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs

Discussant

:

Anette Björnram,

Statistics

Sweden

Slide2

Brief presentation

Working

at

the

cognitive

lab

at

Statistics

Sweden since12

years

(

questionnaire

design and

testing

)

Before

that

I

worked

at the data

collection

department

for 10

years

.

Leading a

project

preparing

for mixed-mode in the LFS (

introducing

web from the 2:nd

wave

for permanently employed)

The ambition is to

create

a

unimode

questionnaire

that

can

be

used

in CATI as

well

.

Slide3

The need of re-designing

Eurostat´s model questionnaires, technical specifications and

guidelines often recommend or require a specific

mode,

The MIMOD survey shows

that

mixed-mode is “a fact” in ESS surveys (a great variation of mode combinations),

The ongoing work introducing web

into the

mix – including mobile CAWI - also needs to be considered,

WP

have done

a great

job giving recommendations

(based on user tests) how to best redesign key

questions and types of questions

for mixed-mode (

a big help

for NSIs),

Have also pointed out the need of

redesigning Eurostat´s

guidelines

and model questionnaires for mixed-mode data

collection.

Slide4

Unimode or mode-specific approach –

what

to

recommend

?

Statistics

Norway

have

done

user

tests of

both

unimode

and mode -

specific

approaches

.

The test results,

as well as the experiences from a

Statistics Netherlands, indicate

that

a

unimode

approach – which

includes mobile

CAWI as well – should be the general recommendation.

Mode

specific

solutions

require thorough pretesting before implementation

.

Slide5

The review focuses

on:

Is the

unimode

approach the best?

In

such

cases

;

why

is it the best?

Focusing

on

recommendations

for

three

types

of

questions

(or

key

questions

)

tested

in SSBs mode-

specific

test approach

ICT: ”Check-all-

that

-

apply

” in CAWI and ”

yes

/no”

questions

in CATI

EHIS:

Alcohol

consumption

b

ranching

question

in CATI and

single

question

in CAWI

LFS:

Actual

worked

hours

day

-by-

day

calculator

in CAWI and

single

question

in CATI

Slide6

ICT: ”Check-all-that-apply

” in CAWI and

yes

/no

questions

in CATI

The ”check-all-

that

-

apply

” format

works in self-administered modes (CAWI PC and paper-and-pencil) and face-to-face interviews (with showcards) but not in telephone interviews or mobile CAWI.

Slide7

Different ways to handle the ”check-all-

that

-

apply

” format in CATI

An

open

question

with

interviewer codingCould lead to underreporting, Risk of coding errors Risk of

interviewer

effects

Yes

/no

questions

Often

give

higher

reporting

than

the ”check-all-

that

-

apply

format

Longer

completion

time

,

higher

response

burden

,

higher

risk of ”break-off” (

especially

in

self-administered

modes

)

Slide8

Recommendations

from the WP

A

unimode

approach

where

:

”the ICT

model

questionnaire

should be reviewed and presented in a mode-neutral way. The “Check all that apply” questions should be considered replaced with yes/no sequences, as well as shortened or made more relevant.”

“For

some short ones, like the question on Internet connections, a check all that apply format could be acceptable in CAWI, while using a sequence of yes/no questions in CATI

.”

Slide9

Review of the recommendations

I agree with the recommendation of a

unimode

approach, with yes

/no sequences, in order

to avoid

measurement

differences

due to mode specific questions.In Sweden we have problems with the estimates for the ICT Survey. Since

we

use

”check-all-

that

-

apply

in CAWI and

yes

/ no

questions

in CATI,

we

get different

estimates

.

A

yes/no

unimode

-format would probably reduce the problems

.

More

comparable reporting between countries

(

differences

between

countries

due

to ”

question

format

effects

rather

than “actual” differences)

The same

questions

can

be

used

in all

modes,

you

save

time

in

programming

Slide10

Review of the recommendations

One must also have in mind the very important last part of the recommendation as well:

The

“Check all that apply” questions should be considered replaced with yes/no sequences,

as well as shortened or made more relevant

.”

Transforming the check-all-that-apply questions into series of yes/no questions makes the questions longer and more “burdensome”. (That´s why Stats Sweden are not using the yes/no format in CAWI.)

One of the main problems in the SSB test was related to difficult

, unfamiliar and technical

terms.

The

questions needs to be reworked and tested

!

Slide11

EHIS: Alcohol

consumption

branching

in CATI and

single

question

in CAWI

EHIS

model

questionnaire is designed for face-to-face modes with showcards.Also works in some self-administered modes (PC CAWI, Paper and pencil) but not so well in mobil CAWI

A document

with further recommendations for CATI mode on

this and other certain questions (from a Eurostat grant)

I

mprovement

of the European Health Interview Survey (EHIS) modules on alcohol consumption, physical activity and mental health. Final Report

(Berlin 2011

).

Slide12

EHIS: Alcohol

consumption

single

question

(

showcard

)

A long list of

categories

, don’t work well in CATI or mobile CAWI

Slide13

EHIS: Alcohol consumption

branching

Slide14

Recommendations from the WP

“Based

on the user tests,

a mode specific design

could viable for the question on alcohol consumption, with show cards in CAPI, one question in CAWI, and a branching question in CATI

.

/…/

The branching approach could be applied for mobile CAWI while keeping the one question format in PC CAWI.

However

, this may be considered unnecessarily complex in terms of programming and administration.

A

unimode branching approach could also be considered.”

Slide15

Review of the recommendations

I think one

should

consider a

unimode

approach, not only to make it easier to program and administer, but for measurement reasons:

T

here

could

be possible differences in the answers with the branching question instead of the single question, especially since it is a sensitive question.

The response categories can give a “hint” of what is “normal” or “social

desirable” (at least the respondent thinks so). Nine categories in the single question and three categories in the branching question - could give different “ideas” what`s considered as “normal consumption”.The first category may be too ”narrow

”, at

least

for CATI, as p

eople

are more likely to give ”social desirable answers” in

CATI.

Slide16

LFS: Actual worked

hours

day

-by-

day

calculator

in CAWI and

single question in CATI Actual worked hours in the reference week is a key question in the LFS.A challenge

to get the

respondent´s

to

provide

actual

worked

hours

instead

of

contracual

or

usual

working

hours

,

especielly

in

s

elf-administered

modes (CAWI).

The lack of an

interviewer

,

who

can

give

support and

help

the respondent to

remember

, is a problem.

The idea with

the day-by-day calculator is

to help the respondent`s to remember how much they worked day-by-day in the reference week, in order to get a more accurate sum of hours worked, and to prevent satisficing.

Slide17

HWACTUAL: Day-by-day calculator

Slide18

Test results and recommendations

The

MIMOD user test

results; Using

the

day-by-day calculator in CAWI required

more of the respondent, and presupposes a familiarity with calculation setups.

It´s

in line

with

the results at ONS, and also in line with tests for the EU-SILC using a calculator for mortgage paymentConclusion; The mode-specific

solution

cannot

presently

be

recommended

.

Further

testing

,

including

testing

of

other

approaches

,

is

advisable

.

SO…

What´s

is

next

to test?

Should

we

test (

other

)

visual

mode

specific

approach in CAWI?

Or

should

we

try

to

design

a

unimode

question

(or approach),

that

works

for

both

CATI and CAWI?

Slide19

Testing other

approaches

for

this

question

experiences

and

thougts

Statistics Sweden´s

is preparing to get the LFS into the web (from 2:nd wave).At a first round of cognitive interviews of the LFS web questionnaire, we used the same single question as is used in the telephone interview. How many hours did you work in <<week 14>> , i.e. from Monday <<1 of April>> to Sunday <<7 of April>>?Clearly did not work! (3

of 15 test persons

actually

tried

to

count

the

hours

. 2

worked

standard

hours

and

didn´t

have to

count

.)

The rest

made

an

estimate

or

answered

”I

think

it

was

a normal

week

” and

gave

the same

numbers

of

hours

as

contractual

hours

.

Did

not

realize

the

importance

of the

question

. ”A

couple

of

hours

more

or less

can

hardly

make

any

difference

”.

We

had

to do

something

to make the

differences

between

actual

worked

and

contraltual

working

hours

clearer

!

Slide20

Swedish new version of HWACTUAL(to be

tested

soon

)

Add

an

introduction

to the

question

:

This question is one of the most important questions in the survey that concerns how many hours you actually worked << in your main job >> during week <<14>>. Try to answer as accurately as you can.Refers to the period Monday

1 April to

Sunday

7 April

Present

previous

questions

about

abscense

and extra

hours

:

How

many hours did you

work that week?

Please take into account

<<if extra hours: the extra hours you worked>> as

well as

<<if

abscense

: your absence from work because

of <<

abs reasons>>.

We

will

test

this

in the

next

round of

cognitive

intervews

in May.

Slide21

The last recommendation from WP4

The establishment of a Wiki-type forum

for exchange of examples, test results and

discussions.

It´s

a brilliant

idea

!

There is a need of a forum for “easy sharing” of

experiences

and test results (good and bad), since we often struggle with the same

issues

and problems.

Slide22

Thank´s

for

listening

!

a

nette.bjornram@scb.se