Cognitive complexity Sample items Math Item Design 4 building blocks Learning Progression Item Design OutcomesScoring Assessment Quality BEAR Assessment System Step 2 A match between what is taught and what is ID: 681644
Download Presentation The PPT/PDF document "Design framework Item development" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Design frameworkItem developmentCognitive complexitySample items
Math
Item DesignSlide2
4 building blocks
Learning Progression
Item Design
Outcomes/Scoring
Assessment QualitySlide3
BEAR Assessment System Step 2A match between what is taught and what is assessedConstructive alignment
aligning teaching and assessment to the learning outcomes/standards (Biggs, 2003)
Proposed items are located along the LP map
Item Design Framework
Learning Outcomes/Standards
Assessment Task
Teaching & Instruction
One Framework
(Wilson & Sloane, 2000)Slide4
Item design framework used by Smarter-Balanced Assessment under the evidence-centered design approach (Mislevy, Steinberg, & Almond, 2003)
Defined as the degree to which expectations and assessments work together to improve and measure students learning
Alignment frameworkSlide5
Categorical concurrenceCommonality between the content categories of the standards and those of the assessment items
Range of knowledge correspondence
Number of objectives within each standard covered by item(s)
Balance of representation
Relative coverage of content categories by items in a test
Depth of Knowledge consistencyMatch between the cognitive demand of items and the level of cognitive demand communicated by the wording of the objectives
4 criteria to determine the degree of alignmentSlide6
Universal designDesign item that accurately assess the targeted competency for all students
Ensure item fairness – make sure that items are equally difficult for groups of equal ability (e.g. males and females; urban and rural)
Vocabulary & language
use content-specific language appropriate to the assessed grade
For non-content-specific material, use vocabulary/language from previous grade levels
Item developmentSlide7
Grade appropriatenessDesign items that assess a primary content domain/standard of the appropriate grade“For non-reading items, the reading level is approximately one grade level below the grade level of the test, except for specifically assessed content terms or concepts” (SBAC, 2012)
Using items to link tests
For pre-post test designs, include some items that appeared on previous test(s) to measure student progress
If the time between tests is relatively long (i.e. 2-3 months), the same test can be used
If the time is short (i.e. 2-3 weeks), pick a few items to reuse and include new onesItem developmentSlide8
Use of Modified Bloom’s TaxonomyDefinitionAn example of Cognitive Rigor Matrix (Hess, et al., 2009)Demonstration on how to align standards and proposed item(s) on the LP map
Cognitive complexitySlide9
Modified by Anderson & Krathwohl (2001)
Modified Bloom’s Taxonomy
Old Bloom’s Taxonomy (Bloom, 1956)
Knowledge
Comprehension
Application
Analysis
Synthesis
EvaluationSlide10
Cognitive rigor matrixSlide11
Sample items focus on Application process
Process
Level 1: Recall
& Reproduction
Level
2: Skills & Concepts
Level 3: Strategic Thinking/
ReasoningLevel 4: Extended
ThinkingApply:Carry out or use a procedure in a given situations; carry out (apply to a familiar task) or use (apply) to an unfamiliar task.
Follow simple/routine procedures.Apply an algorithm or formula.Solve routine problems applying multiple concepts or decision points.
Use or apply concepts to solve non-routine problems.
Select or devise an approach among many alternatives to solve a novel problem.Slide12
Sample item*
Intended level: Grade 8
Domain: Expressions and Equations
Content standard
: Analyze and solve linear equations and pairs of simultaneous linear equations
.
CCSS: Analyze
and solve pairs of simultaneous linear equations (8.EE.8
).
Intended claims: 1,
2
, 3 and 4
Max is
organizing
a trip
to the
airport for a party of 75 people.
He
can use two types of taxi.
A
small taxi costs $40 for the trip and holds up to 4
people.
A large taxi costs $63 for the trip and holds up to 7 people.
* Adapted from SBAC
(2013, p.
134)Slide13
Sample item Grade 8
LP level
Specific Standard
Cognitive Rigor
Analyze and solve linear equations and pairs of simultaneous linear equations.
Analyze and solve pairs of simultaneous linear equations
(8.EE.8).
Apply,
Level 3 (Strategic Thinking/reasoning)
How many taxis of each type should Max order, to keep the total cost as low as possible? Explain.Slide14
Sample item Grade 7
LP level
Specific Standard
Cognitive
Rigor
Solve real-life and mathematical problems using numerical and algebraic expressions and equations.
Use variables to represent quantities in a real-world or mathematical problem, and construct simple equations and inequalities to solve problems by reasoning about the quantities. (7.EE.4).
Apply,
Level 2 (Skills & Concepts
Let L be the number of large taxis needed and S be the number of small taxis needed.
a. Write an expression to show the number of taxis needed.
b. If Max orders 6 large taxis, how many small taxis will he need?
b. How much will the cost be? Slide15
Sample item Grade 6
LP level
Specific Standard
Cognitive
Rigor
Apply and extend previous understandings of arithmetic to algebraic expressions.
Write, read, and evaluate expressions in which letters stand for numbers. (6.EE.2).
Apply, Level 1 (Recall and Reproduction)
a. Let x be the number of large taxis
needed. If Max wants to order large taxis only, evaluate x. b. If each person must equally share the cost of taking large taxi defined previously, write an expression for the cost of each person. Let y be the amount of money each person must pay. Calculate y.Slide16
Bloom, B. S. (1956). Taxonomy of educational objectives. Handbook I: The Cognitive Domain
. New York, NY: David McKay Co.
Anderson, L. W., &
Krathwohl, D. (2001).
A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives. New York, NY: Longman.Hess, K., Carloc, D., Jones, B., & Walkup, J., (2009). What exactly do “fewer, clearer, and higher standards” really look like in the classroom? Using a cognitive rigor matrix to analyze curriculum, plan lessons, and implement assessments
. Paper presented at CCSSO, Detroit, Michigan.
Nitko, A. J., & Brookhart, S. (2007).
Educational assessment of students. Upper Saddle River, NJ: Pearson Education, Inc.McMillan, J. H. (2007).
Classroom assessment. Principles and practice for effective standard-based instruction (4th ed.). Boston: Pearson - Allyn & Bacon. Oregon Department of Education. (2014, June).
Assessment guidance. Webb
, N. (2007).
Aligning assessments and standards
. Retrieved from http://
www.wcer.wisc.edu/news/coverStories/aligning_assessments_and_standards.php
Wilson
, M. (2005).
Constructing measures: An item response modeling approach
. New
York, NY: Psychology
Press, Taylor & Francis
Group.
Wilson, M., & Sloane, K. (2000). From principles to practice: An embedded assessment system.
Applied Measurement in Education, 13
(2), pp. 181-208.
Smarter Balanced Assessment Consortium. (2012, April).
General item specifications
.
Smarter Balanced Assessment Consortium. (2013, June).
Content specifications for the summative assessment of the Common Core State Standards for Mathematics
. Revised
draft.
BibliographySlide17
Item Design (Math)
PPT by
the
Oregon Department of Education and
Berkeley Evaluation and Assessment Research Center is licensed under a CC BY 4.0.
You are free to:Share — copy and redistribute the material in any medium or format Adapt
— remix, transform, and build upon the material Under the following terms:Attribution
— You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
NonCommercial — You may not use the material for commercial purposes. ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the
same license as the original. Oregon
Department of Education welcomes editing of these resources and would greatly appreciate being able to learn from the changes made. To share an edited version of this resource, please contact Cristen McLean,
cristen.mclean@state.or.us
.
Creative Commons License