Jonas Bertling Director NAEP Survey Questionnaires ETS RWJF amp ETS Workshop Understanding the opportunities for collaboration between the health and education sectors in a culture of health ID: 675551
Download Presentation The PPT/PDF document "New Survey Questionnaire Indicators in P..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
New Survey Questionnaire Indicators in PISA and NAEP
Jonas BertlingDirector, NAEP Survey Questionnaires, ETS
RWJF & ETS Workshop
Understanding the opportunities for collaboration between the health and education sectors in a culture of health
Princeton, May 5, 2016Slide2
Main talking points today
Survey questionnaires in educational large-scale assessments started collecting broader types of contextual variables reflecting the increased policy interest in “noncognitive” skills and additional outcomes.PISA broadened the range of constructs captured with the student questionnaires and introduced health and well-being facets in 2015. PISA 2018 will include an optional well-being questionnaire.NAEP started innovating their survey questionnaires and will introduce a series of new contextual modules (including noncognitive indices) in 2017.2
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
Copyright © 2016 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING
are
registered trademarks of Educational Testing Service (ETS).
36319Slide3
Main talking points today (continued)
NAEP 2017 contextual modules can also be mapped to PISA 2018 draft well-being framework. PISA conceptualizes noncognitive variables explicitly as additional outcomes; NAEP currently models them on the predictorside only. Challenges fall especially into three categories: Measurement, Perceived Intrusiveness and Respondent Burden.3
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide4
Changing role of large-scale assessments
4
How do subgroups perform?
How do students approach learning situations and utilize learning opportunities?
How do learning opportunities differ across the Nation?
How are students in the Nation doing in terms of other outcomes than achievement?
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide5
Shift in policy interest
Measure contextual variablesDescribe test-taker populationsImprove estimates of achievement (conditioning model)Measure constructs of own interest
Report additional outcomes
Measure trend
Compare subgroups
Evaluate policies
Accountability
5
Figure from Bertling,
Borgonovi
, & Almonte (2016)
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide6
Examples
6
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide7
Survey
questionnaires data put achievement results into context …
… and
can add value to
large-scale
assessments by capturing important additional variables.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide8
PISA
Programme for International Student Assessment Organisation for Economic Co-operation and Development (OECD)8Slide9
PISA broadened the range of constructs captured with survey questionnaires since the first cycle in 2000
Since its early cycles, PISA has included questions capturing students’ attitudes and sense of belonging to schoolPISA 2012 introduced new domain-general student attitudes and behaviors (e.g., perseverance, openness for problem solving)PISA 2015 added additional noncognitive outcomes also covering health and well-beingThis trend continues with PISA 2018
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
Figure from Bertling,
Borgonovi
, & Almonte (2016)Slide10
PISA 2015 Questionnaire Framework
Figure from OECD (2016)
J
. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide11
Well-being-related constructs in PISA 2015
General life satisfactionParticipation in physical education classes at schoolEngagement in moderate and vigorous physical exerciseSchool-related well-being (sense of belonging, bullying, test anxiety)A larger number of additional health- and well-being-related questions (e.g., BMI, dietary habits, enjoyment of physical activity, etc.) were piloted but not taken to operational largely due to timing constraints.11
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide12
Well-being questionnaire in PISA 2018
OECD is planning an optional questionnaire devoted to student well-being for PISA 2018 in addition to other noncognitive factors captured with the regular student questionnaires.Developed draft framework for PISA 2018 Module12
*Note: The PISA 2018 well-being framework i
s not a public document and currently remains a DRAFT version
, i
s subject to all the normal OECD copyright provisions
,
and
should
clearly be acknowledged with the OECD as owner and author.
DRAFT*
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide13
NAEP
National Assessment of Educational Progress National Center for Education Statistics (NCES)13Slide14
Innovations underway for the NAEP Survey Questionnaires
Expand constructs to include more contextual factors critical for student achievement
Improve
reporting
of
contextual factors for more robustness & policy relevance
Improve pre-testing and questionnaire
design
for
digital delivery to maximize validity and keep student burden low
I.
II.
III.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide15
NAEP Contextual Variables
Opportunity-to-Learn (OTL) Factors:
(COGNITIVE ASSESSMENT)
(SURVEY QUESTIONNAIRES)
“Noncognitive” Student Factors
:
Achievement
Domain-general
factors (“Core”)
Subject-specific
factors
*not measured with the NAEP survey questionnaires
Students’ exposure to opportunities to acquire relevant knowledge and skills, both at school and outside of school.
Skills, strategies,
attitudes
and behaviors that are distinct from content knowledge and academic skills (e.g., motivational and self-regulatory variables)
OTL
at
School
BUT
:
The term
“noncognitive” is not intended to be interpreted literally. All aspects of human thought and behavior are related to cognition!
Cognitive Ability*
“Noncognitive”
Student Factors
OTL Outside of School
Basic Demographics
Balanced reporting approach based on indices and stand-alone
questions
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide16
16
Implementation for NAEP 2017 Core, Reading, Writing and Mathematics:
Item Development
Pre-testing in Cognitive Labs
Large-scale Pilot Testing
Operational Administration
Noncogs
Domain-general (Core)
OTL Outside of School
OTL At School
“Noncognitive”
Student Factors
Basic Demographics
Subject-specific
1
2
1
2
3
3
4
5
6
7
5
4
Grit/Perseverance (including self-control)
Desire for
Learning
Self-Efficacy/Confidence
Achievement Goals
Interest & Enjoyment
Components of SES
Technology Use
School Climate
OTL
at
School
Out-of-
S
chool
Activities
Resources for Learning and Instruction
Organization of Instruction
Teacher Preparation
5 Core Modules
Additional Subject-specific
Modules
OTL
Outside of School
2014
2014-15
2016
2017
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide17
Mapping NAEP to PISA 2018 WB Framework
17DRAFT*
Grit/Perseverance,
Desire for Learning,
Self-Efficacy
School Climate
SES
*Note: The PISA 2018 well-being framework i
s not a public document and currently remains a DRAFT version
, i
s subject to all the normal OECD copyright provisions
,
should clearly be acknowledged with the OECD as owner and author.
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide18
Difference between PISA and NAEP approaches
18Additional Outcomes
Achievement Predictors
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide19
Challenges
MeasurementPerceived IntrusivenessRespondent Burden19Figure from Bertling,
Borgonovi
, & Almonte (2016)
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide20
Summary
Survey questionnaires in educational large-scale assessments started collecting broader types of contextual variables reflecting the increased policy interest in “noncognitive” skills and additional outcomes.PISA broadened the range of constructs captured with the student questionnaires and introduced health and well-being facets in 2015. PISA 2018 will include an optionalwell-being questionnaire.NAEP started innovating their survey questionnaires and will introduce a series of new contextual modules (including noncognitive indices) in 2017.20
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide21
Summary (continued)
NAEP 2017 contextual modules can also be mapped to PISA 2018 draft well-being framework. PISA conceptualizes noncognitive variables explicitly as additional outcomes; NAEP currently models them on the predictor side only. Challenges fall especially into three categories: Measurement, Perceived Intrusiveness and Respondent Burden.21
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016 Slide22
Questions? Comments?
jbertling@ets.orgTwitter: @JonasBertling
Thank you!
J. Bertling | New Survey Questionnaire Indicators in PISA and NAEP | RWJF-ETS Workshop | 5/5/2016
Copyright © 2016 by Educational Testing Service. All rights reserved. ETS, the ETS logo and MEASURING THE POWER OF LEARNING
are
registered trademarks of Educational Testing Service (ETS).
36319
22