The following materials were prepared for use by
Description: The following materials were prepared for use by CARS and SASS consultants. Item Writing Chris Orem Jerusha Gerstner Christine DeMars Agenda Presentation and Discussion Item writing guidelines Examples Develop, Evaluate, and Revise Items A
Related Topics
Download Presentation
"The following materials were prepared for use by" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. The following materials were prepared for use by CARS and SASS consultants.<br>
slide2. Item Writing Chris Orem
Jerusha Gerstner
Christine DeMars<br>
slide3. Agenda Presentation and Discussion
Item writing guidelines
Examples
Develop, Evaluate, and Revise Items<br>
slide4. A test blueprint is a table of specifications that weights each objective according to how important it is or how much time is spent covering that objective, links objectives to test items and summarizes information
Essential for Competency Testing BLUEPRINT TIPS
List objectives in a table
Identify length of test
Designate number of items per objective
-Evaluate the importance of each objective and assign items accordingly
Often assign one point per item<br>
slide5. Example of BluePrint<br>
slide6. Some items are more appropriate when testing different kinds of knowledge; or when tapping into different kinds of cognitive processes.
We’ll focus on multiple choice items because recruiting faculty to score open-ended items would be difficult. Writing Items<br>
slide7. Multiple-Choice Item ANATOMY http://testing.byu.edu/info/handbooks/betteritems.pdf<br>
slide8. Guidelines for Writing Items<br>
slide9. Content Focus on a single problem when writing an item
Use new situations to assess application
Avoids memorization exercises
Allows for synthesis and evaluation
Keep content of items independent
students shouldn’t be able to use one item to answer another, although a set of items may tap into a shared scenario
Avoid opinion-based items
Address a mix of higher-order and lower-order thinking skills.<br>
slide10. Higher Order Thinking Bloom Haladyna Analysis Synthesis Application Comprehension Knowledge Evaluation Predicting Evaluating Defining Recalling Problem Solving Higher Order Skills Lower Order Skills<br>
slide11. Writing Items to Assess Higher Order Thinking (based on Haladyna) Try item stems such as "If . . ., then what happens?", "What is the consequence of . . .?", or "What would happen if . . .?" (predicting)
Ask students to make a decision based on predetermined criteria, or to choose criteria to use in making a decision, or both (evaluating).
Require the student to use combinations of recalling, summarizing, predicting, and evaluating to solve problems.<br>
slide12. Avoid excess words – be succinct
Use specific, appropriate vocabulary
Avoid bias (age, ethnicity, gender, disabilities)
Write stems and options in third person
Underline or bold negative or other important words
Have others review your items Style/format<br>
slide13. The stem should clearly state the problem
Place the main idea of the question in the stem, not the item options
Keep the stem as short as possible
Don’t provide clues to correct answer in stem (e.g., grammatical clues)
If the stem is a complete sentence, end with a period and begin all response options with upper-case letters.
If the stem is an incomplete sentence, begin all response options with lower case letters.
Use negative stems rarely Writing the stem<br>
slide14. Make sure there is only one correct answer for each item
Develop as many effective plausible options as possible, but three are sufficient (Rodriguez, 2005)
It is better to have fewer options than to write BAD options to meet some quota!
Vary the location of the correct answer when feasible (Flip a coin), or put options in logical order (e.g. chronological, numerical)
Avoid excessive use of negatives or double negatives
Keep options independent
Keep options similar (in format)
Length and wording Writing the distracters<br>
slide15. DO use as distracters:
common student misconceptions (perhaps from open-ended responses from previous work)
words that “ring a bell” or “sound official”
responses that fool the student who has not mastered the objective
DO NOT use as distracters:
responses that are just as correct as the right answer
implausible or silly distracters
Use “all of the above” and “none of the above” sparingly
Don’t use “always” or “never”
Don’t give clues to the right answer Writing the distracters – Cont.<br>
slide16. All the option choices should use parallel grammar to avoid giving clues to the right answer. The best way to increase the reliability of a test is to:
A. increase the test length
B. removing poor quality items
C. Tests should be readable for all test takers.<br>
slide17. What’s wrong with this item? California:
A). Contains the tallest mountain in the United States
B). Has an eagle on its state flag.
C). Is the second largest state in terms of area.
*D). Was the location of the Gold Rush of 1849. What is the main reason so many people moved to California in 1849?
A). California land was fertile, plentiful, and inexpensive.
*B). Gold was discovered in central California
C). The east was preparing for a civil war.
D). They wanted to establish religious settlements. Stem should state the problem<br>
slide18. “Focus on a single problem”; items with multiple clauses may have multiple correct answers depending on which aspect the student focuses. Bleeding of the gums is associated with gingivitis, which can be cured by the sufferer himself by brushing his teeth daily.
A. true
B. false<br>
slide19. The United States should adopt a foreign policy based on:
A). A strong army and control of the North American continent.
B). Achieving the best interest of all nations.
C). Isolation from international affairs.
*D). Naval supremacy and undisputed control of the world’s sea lanes. According to Alfred T. Mahan, the United States should adopt a foreign policy based on:
A). A strong army and control of the North American continent.
B). Achieving the best interest of all nations.
C). Isolation from international affairs.
*D). Naval supremacy and undisputed control of the world’s sea lanes. What’s wrong with this item?- cont. More than One Possible Answer<br>
slide20. Piloting and revising Following these rules does not guarantee that items will perform well empirically. Testing companies, using paid item-writers and detailed writing guidelines, ultimately only use 1/3 (or fewer) of the items operationally.
It is normal and expected to have to revise items after pilot-testing .<br>
slide21. Piloting and revising 1 ITEM ANALYSIS STEPS: Item difficulty
- proportion of people who answered the item correctly
- an item should not be too easy or too difficult
- problems arise from poor wording, trick questions, or speediness
Item Discrimination
correlation of item and total test, should be higher than 0.2 - item as an indicator of the overall test score
the higher the better
can be (but shouldn’t be) negative
Distractor Analysis
frequency of response selections
item is problematic if people with a high overall score are selecting incorrect responses frequently<br>
slide22. The Big Idea Ultimately, you want to know whether students are achieving your objectives
Tests are used to indirectly measure these knowledge, skills, attitudes, etc.
Items you write are a sample of all possible items to measure that objective
the more you write (and the better your items are!) the more reliably you can measure the objective
You want to be sure that the items you create are measuring achievement of the objective, and NOT test-wiseness, reading ability, or other factors<br>
slide23. References Haladyna, T. M. (1999). Developing and validating multiple-choice test items. Mahwah, NJ: Lawrence Erlbaum Associates.
Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24, 3-13.
Downing & Haladyna (2006). Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Associates.
Burton S.J. et al (1991). How to prepare Multiple-Choice Test Items: Guidelines for University Faculty. Brigham Young University Testing Services.<br>
slide2. Item Writing Chris Orem
Jerusha Gerstner
Christine DeMars<br>
slide3. Agenda Presentation and Discussion
Item writing guidelines
Examples
Develop, Evaluate, and Revise Items<br>
slide4. A test blueprint is a table of specifications that weights each objective according to how important it is or how much time is spent covering that objective, links objectives to test items and summarizes information
Essential for Competency Testing BLUEPRINT TIPS
List objectives in a table
Identify length of test
Designate number of items per objective
-Evaluate the importance of each objective and assign items accordingly
Often assign one point per item<br>
slide5. Example of BluePrint<br>
slide6. Some items are more appropriate when testing different kinds of knowledge; or when tapping into different kinds of cognitive processes.
We’ll focus on multiple choice items because recruiting faculty to score open-ended items would be difficult. Writing Items<br>
slide7. Multiple-Choice Item ANATOMY http://testing.byu.edu/info/handbooks/betteritems.pdf<br>
slide8. Guidelines for Writing Items<br>
slide9. Content Focus on a single problem when writing an item
Use new situations to assess application
Avoids memorization exercises
Allows for synthesis and evaluation
Keep content of items independent
students shouldn’t be able to use one item to answer another, although a set of items may tap into a shared scenario
Avoid opinion-based items
Address a mix of higher-order and lower-order thinking skills.<br>
slide10. Higher Order Thinking Bloom Haladyna Analysis Synthesis Application Comprehension Knowledge Evaluation Predicting Evaluating Defining Recalling Problem Solving Higher Order Skills Lower Order Skills<br>
slide11. Writing Items to Assess Higher Order Thinking (based on Haladyna) Try item stems such as "If . . ., then what happens?", "What is the consequence of . . .?", or "What would happen if . . .?" (predicting)
Ask students to make a decision based on predetermined criteria, or to choose criteria to use in making a decision, or both (evaluating).
Require the student to use combinations of recalling, summarizing, predicting, and evaluating to solve problems.<br>
slide12. Avoid excess words – be succinct
Use specific, appropriate vocabulary
Avoid bias (age, ethnicity, gender, disabilities)
Write stems and options in third person
Underline or bold negative or other important words
Have others review your items Style/format<br>
slide13. The stem should clearly state the problem
Place the main idea of the question in the stem, not the item options
Keep the stem as short as possible
Don’t provide clues to correct answer in stem (e.g., grammatical clues)
If the stem is a complete sentence, end with a period and begin all response options with upper-case letters.
If the stem is an incomplete sentence, begin all response options with lower case letters.
Use negative stems rarely Writing the stem<br>
slide14. Make sure there is only one correct answer for each item
Develop as many effective plausible options as possible, but three are sufficient (Rodriguez, 2005)
It is better to have fewer options than to write BAD options to meet some quota!
Vary the location of the correct answer when feasible (Flip a coin), or put options in logical order (e.g. chronological, numerical)
Avoid excessive use of negatives or double negatives
Keep options independent
Keep options similar (in format)
Length and wording Writing the distracters<br>
slide15. DO use as distracters:
common student misconceptions (perhaps from open-ended responses from previous work)
words that “ring a bell” or “sound official”
responses that fool the student who has not mastered the objective
DO NOT use as distracters:
responses that are just as correct as the right answer
implausible or silly distracters
Use “all of the above” and “none of the above” sparingly
Don’t use “always” or “never”
Don’t give clues to the right answer Writing the distracters – Cont.<br>
slide16. All the option choices should use parallel grammar to avoid giving clues to the right answer. The best way to increase the reliability of a test is to:
A. increase the test length
B. removing poor quality items
C. Tests should be readable for all test takers.<br>
slide17. What’s wrong with this item? California:
A). Contains the tallest mountain in the United States
B). Has an eagle on its state flag.
C). Is the second largest state in terms of area.
*D). Was the location of the Gold Rush of 1849. What is the main reason so many people moved to California in 1849?
A). California land was fertile, plentiful, and inexpensive.
*B). Gold was discovered in central California
C). The east was preparing for a civil war.
D). They wanted to establish religious settlements. Stem should state the problem<br>
slide18. “Focus on a single problem”; items with multiple clauses may have multiple correct answers depending on which aspect the student focuses. Bleeding of the gums is associated with gingivitis, which can be cured by the sufferer himself by brushing his teeth daily.
A. true
B. false<br>
slide19. The United States should adopt a foreign policy based on:
A). A strong army and control of the North American continent.
B). Achieving the best interest of all nations.
C). Isolation from international affairs.
*D). Naval supremacy and undisputed control of the world’s sea lanes. According to Alfred T. Mahan, the United States should adopt a foreign policy based on:
A). A strong army and control of the North American continent.
B). Achieving the best interest of all nations.
C). Isolation from international affairs.
*D). Naval supremacy and undisputed control of the world’s sea lanes. What’s wrong with this item?- cont. More than One Possible Answer<br>
slide20. Piloting and revising Following these rules does not guarantee that items will perform well empirically. Testing companies, using paid item-writers and detailed writing guidelines, ultimately only use 1/3 (or fewer) of the items operationally.
It is normal and expected to have to revise items after pilot-testing .<br>
slide21. Piloting and revising 1 ITEM ANALYSIS STEPS: Item difficulty
- proportion of people who answered the item correctly
- an item should not be too easy or too difficult
- problems arise from poor wording, trick questions, or speediness
Item Discrimination
correlation of item and total test, should be higher than 0.2 - item as an indicator of the overall test score
the higher the better
can be (but shouldn’t be) negative
Distractor Analysis
frequency of response selections
item is problematic if people with a high overall score are selecting incorrect responses frequently<br>
slide22. The Big Idea Ultimately, you want to know whether students are achieving your objectives
Tests are used to indirectly measure these knowledge, skills, attitudes, etc.
Items you write are a sample of all possible items to measure that objective
the more you write (and the better your items are!) the more reliably you can measure the objective
You want to be sure that the items you create are measuring achievement of the objective, and NOT test-wiseness, reading ability, or other factors<br>
slide23. References Haladyna, T. M. (1999). Developing and validating multiple-choice test items. Mahwah, NJ: Lawrence Erlbaum Associates.
Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24, 3-13.
Downing & Haladyna (2006). Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Associates.
Burton S.J. et al (1991). How to prepare Multiple-Choice Test Items: Guidelines for University Faculty. Brigham Young University Testing Services.<br>