Discussant Anette Björnram Statistics Sweden Brief presentation Working at the cognitive lab at Statistics Sweden since12 years questionnaire design and ID: 799800
Download The PPT/PDF document "Review of WP 4 - Mixed-mode questionnai..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Review of WP 4 - Mixed-mode questionnaire designs: Recommendations for Eurostat and NSIs
Discussant
:
Anette Björnram,
Statistics
Sweden
Slide2Brief presentation
Working
at
the
cognitive
lab
at
Statistics
Sweden since12
years
(
questionnaire
design and
testing
)
Before
that
I
worked
at the data
collection
department
for 10
years
.
Leading a
project
preparing
for mixed-mode in the LFS (
introducing
web from the 2:nd
wave
for permanently employed)
The ambition is to
create
a
unimode
questionnaire
that
can
be
used
in CATI as
well
.
Slide3The need of re-designing
Eurostat´s model questionnaires, technical specifications and
guidelines often recommend or require a specific
mode,
The MIMOD survey shows
that
mixed-mode is “a fact” in ESS surveys (a great variation of mode combinations),
The ongoing work introducing web
into the
mix – including mobile CAWI - also needs to be considered,
WP
have done
a great
job giving recommendations
(based on user tests) how to best redesign key
questions and types of questions
for mixed-mode (
a big help
for NSIs),
Have also pointed out the need of
redesigning Eurostat´s
guidelines
and model questionnaires for mixed-mode data
collection.
Slide4Unimode or mode-specific approach –
what
to
recommend
?
Statistics
Norway
have
done
user
tests of
both
unimode
and mode -
specific
approaches
.
The test results,
as well as the experiences from a
Statistics Netherlands, indicate
that
a
unimode
approach – which
includes mobile
CAWI as well – should be the general recommendation.
Mode
specific
solutions
require thorough pretesting before implementation
.
Slide5The review focuses
on:
Is the
unimode
approach the best?
In
such
cases
;
why
is it the best?
Focusing
on
recommendations
for
three
types
of
questions
(or
key
questions
)
tested
in SSBs mode-
specific
test approach
ICT: ”Check-all-
that
-
apply
” in CAWI and ”
yes
/no”
questions
in CATI
EHIS:
Alcohol
consumption
–
b
ranching
question
in CATI and
single
question
in CAWI
LFS:
Actual
worked
hours
–
day
-by-
day
calculator
in CAWI and
single
question
in CATI
Slide6ICT: ”Check-all-that-apply
” in CAWI and
yes
/no
questions
in CATI
The ”check-all-
that
-
apply
” format
works in self-administered modes (CAWI PC and paper-and-pencil) and face-to-face interviews (with showcards) but not in telephone interviews or mobile CAWI.
Slide7Different ways to handle the ”check-all-
that
-
apply
” format in CATI
An
open
question
with
interviewer codingCould lead to underreporting, Risk of coding errors Risk of
interviewer
effects
Yes
/no
questions
Often
give
higher
reporting
than
the ”check-all-
that
-
apply
”
format
Longer
completion
time
,
higher
response
burden
,
higher
risk of ”break-off” (
especially
in
self-administered
modes
)
Slide8Recommendations
from the WP
A
unimode
approach
where
:
”the ICT
model
questionnaire
should be reviewed and presented in a mode-neutral way. The “Check all that apply” questions should be considered replaced with yes/no sequences, as well as shortened or made more relevant.”
“For
some short ones, like the question on Internet connections, a check all that apply format could be acceptable in CAWI, while using a sequence of yes/no questions in CATI
.”
Slide9Review of the recommendations
I agree with the recommendation of a
unimode
approach, with yes
/no sequences, in order
to avoid
measurement
differences
due to mode specific questions.In Sweden we have problems with the estimates for the ICT Survey. Since
we
use
”check-all-
that
-
apply
”
in CAWI and
yes
/ no
questions
in CATI,
we
get different
estimates
.
A
yes/no
unimode
-format would probably reduce the problems
.
More
comparable reporting between countries
(
differences
between
countries
due
to ”
question
format
effects
”
rather
than “actual” differences)
The same
questions
can
be
used
in all
modes,
you
save
time
in
programming
Slide10Review of the recommendations
One must also have in mind the very important last part of the recommendation as well:
The
“Check all that apply” questions should be considered replaced with yes/no sequences,
as well as shortened or made more relevant
.”
Transforming the check-all-that-apply questions into series of yes/no questions makes the questions longer and more “burdensome”. (That´s why Stats Sweden are not using the yes/no format in CAWI.)
One of the main problems in the SSB test was related to difficult
, unfamiliar and technical
terms.
The
questions needs to be reworked and tested
!
Slide11EHIS: Alcohol
consumption
–
branching
in CATI and
single
question
in CAWI
EHIS
model
questionnaire is designed for face-to-face modes with showcards.Also works in some self-administered modes (PC CAWI, Paper and pencil) but not so well in mobil CAWI
A document
with further recommendations for CATI mode on
this and other certain questions (from a Eurostat grant)
I
mprovement
of the European Health Interview Survey (EHIS) modules on alcohol consumption, physical activity and mental health. Final Report
(Berlin 2011
).
Slide12EHIS: Alcohol
consumption
–
single
question
(
showcard
)
A long list of
categories
, don’t work well in CATI or mobile CAWI
Slide13EHIS: Alcohol consumption
–
branching
Slide14Recommendations from the WP
“Based
on the user tests,
a mode specific design
could viable for the question on alcohol consumption, with show cards in CAPI, one question in CAWI, and a branching question in CATI
.
/…/
The branching approach could be applied for mobile CAWI while keeping the one question format in PC CAWI.
However
, this may be considered unnecessarily complex in terms of programming and administration.
A
unimode branching approach could also be considered.”
Slide15Review of the recommendations
I think one
should
consider a
unimode
approach, not only to make it easier to program and administer, but for measurement reasons:
T
here
could
be possible differences in the answers with the branching question instead of the single question, especially since it is a sensitive question.
The response categories can give a “hint” of what is “normal” or “social
desirable” (at least the respondent thinks so). Nine categories in the single question and three categories in the branching question - could give different “ideas” what`s considered as “normal consumption”.The first category may be too ”narrow
”, at
least
for CATI, as p
eople
are more likely to give ”social desirable answers” in
CATI.
Slide16LFS: Actual worked
hours
–
day
-by-
day
calculator
in CAWI and
single question in CATI Actual worked hours in the reference week is a key question in the LFS.A challenge
to get the
respondent´s
to
provide
actual
worked
hours
instead
of
contracual
or
usual
working
hours
,
especielly
in
s
elf-administered
modes (CAWI).
The lack of an
interviewer
,
who
can
give
support and
help
the respondent to
remember
, is a problem.
The idea with
the day-by-day calculator is
to help the respondent`s to remember how much they worked day-by-day in the reference week, in order to get a more accurate sum of hours worked, and to prevent satisficing.
Slide17HWACTUAL: Day-by-day calculator
Slide18Test results and recommendations
The
MIMOD user test
results; Using
the
day-by-day calculator in CAWI required
more of the respondent, and presupposes a familiarity with calculation setups.
It´s
in line
with
the results at ONS, and also in line with tests for the EU-SILC using a calculator for mortgage paymentConclusion; The mode-specific
solution
cannot
presently
be
recommended
.
Further
testing
,
including
testing
of
other
approaches
,
is
advisable
.
SO…
What´s
is
next
to test?
Should
we
test (
other
)
visual
mode
specific
approach in CAWI?
Or
should
we
try
to
design
a
unimode
question
(or approach),
that
works
for
both
CATI and CAWI?
Slide19Testing other
approaches
for
this
question
–
experiences
and
thougts
Statistics Sweden´s
is preparing to get the LFS into the web (from 2:nd wave).At a first round of cognitive interviews of the LFS web questionnaire, we used the same single question as is used in the telephone interview. How many hours did you work in <<week 14>> , i.e. from Monday <<1 of April>> to Sunday <<7 of April>>?Clearly did not work! (3
of 15 test persons
actually
tried
to
count
the
hours
. 2
worked
standard
hours
and
didn´t
have to
count
.)
The rest
made
an
estimate
or
answered
”I
think
it
was
a normal
week
” and
gave
the same
numbers
of
hours
as
contractual
hours
.
Did
not
realize
the
importance
of the
question
. ”A
couple
of
hours
more
or less
can
hardly
make
any
difference
”.
We
had
to do
something
to make the
differences
between
actual
worked
and
contraltual
working
hours
clearer
!
Slide20Swedish new version of HWACTUAL(to be
tested
soon
)
Add
an
introduction
to the
question
:
This question is one of the most important questions in the survey that concerns how many hours you actually worked << in your main job >> during week <<14>>. Try to answer as accurately as you can.Refers to the period Monday
1 April to
Sunday
7 April
Present
previous
questions
about
abscense
and extra
hours
:
How
many hours did you
work that week?
Please take into account
<<if extra hours: the extra hours you worked>> as
well as
<<if
abscense
: your absence from work because
of <<
abs reasons>>.
We
will
test
this
in the
next
round of
cognitive
intervews
in May.
Slide21The last recommendation from WP4
The establishment of a Wiki-type forum
for exchange of examples, test results and
discussions.
It´s
a brilliant
idea
!
There is a need of a forum for “easy sharing” of
experiences
and test results (good and bad), since we often struggle with the same
issues
and problems.
Slide22Thank´s
for
listening
!
a
nette.bjornram@scb.se