Interpreting Your Exam Results Item Analysis Presentation Overview Why is understanding your exams item analysis important Psychometrics used to understand your exam results ExamSoft Where to find your exam performance results ID: 1025913
Download Presentation The PPT/PDF document "Item Analysis Presentation" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Item Analysis PresentationInterpreting Your Exam Results
2. Item Analysis Presentation: OverviewWhy is understanding your exam’s item analysis important.Psychometrics used to understand your exam resultsExamSoft:Where to find your exam performance results.How to interpret your exam results.Practice Reading Exam Results
3. Importance of Understanding your Exam ResultsValuable in improving items which will be used again in later tests.Useful to eliminate ambiguous or misleading items in a single test.Valuable for increasing instructors’ skills in test construction.Helps identifying specific areas of course content which need greater emphasis or clarity.
4. Psychometrics to Understand Your Exam ResultsThere are several psychometrics used in ExamSoft to interpret your exam results.Difficulty IndexCorrect %Discrimination IndexPoint BiserialResponse FrequencyKuder-Richardson 20 (KR-20)We will define these and their ranges through out this presentation.
5. ExamSoftSummary Report and Item Analysis
6. ExamSoft: Navigation
7. ExamSoft: Navigation (cont.)From the left-hand navigation, go to your course’s title (e.g. DEN XXXX Course Discipline).Next select the plus sign.A drop down menu will appear.Click on the current year.
8. ExamSoft: Navigation (cont.)From the right-hand window, your exams from that year will populate.Click on the name of the exam in blue.NOTE: The exam being viewed is non-graded and is designed to orient students with ExamSoft.
9. ExamSoft: Navigation (cont.)From the assessment’s tabs, mouse over “Report/Scoring”.From it’s drop down menu click on Summary Report or Item Analysis.We will review both.Let’s start with the Summary Report.On the next screen click the “Use Top 25” button.
10. ExamSoft: Summary ReportKuder-Richardson 20 (KR-20) is theinternal consistency or “reliability”of an assessment score due to good(or poor) assessment construction. The degree to which an assessment is free from randomerror, yielding the same resultacross multiple administrationsthe same cohort.
11. ExamSoft: Summary Report – KR-20The KR-20 is influenced by:Number of test items on the exam.Student performance on each item.Variance for the set of student test scores.Range: 0.00 – 1.00Values near or equal to 0.00 – the test is unreliable and contains random error unassociated with content-specific difficulty and exam-taker comprehension.Values near or equal to 1.00 – the test is highly reliable an would indicate the same rank order of exam-taker performance each time when administered to the same cohort.DMD exams should have a KR-20 of ≥ .50Note: a common cause for a low KR-20 is a low number of test items.
12. ExamSoft: Summary Report (cont.)Question PerformanceThe Correct % (Difficulty Index) column displays the percentage of students who answered the question correctly.The Point Biserial column displays the exam-taker’s answer on a single item and the exam-taker’s overall performance on the assessment.The Discrimination Index displays the % of high performers who answer an item correctlyminus the % of lower performers who answered the item correctly.The Response Frequencies display the distractor analysis of students’ exam answers.
13. ExamSoft: Summary Report – Correct % (Difficulty Index)The Correct % column displays the number of students who answered a question correctly. This is part of the Difficult Index.Difficulty Index – the number of exam-takers who answered the item correctly divided by the total number of exam takers. (An indication of how “easy” a question is.)Questions should perform above 50% - half of the class answered it correctly (aka – understood the question and the teaching materials).If they perform below 50% - less than half of the class answered it correctly,then the Discrimination Index, Point Biserial and Response Frequency columns should be reviewed.Note: The most common error is a question that confuses students in how it was asked rather than the material it covered.
14. ExamSoft: Summary Report – Point Biserial (Student Performance)The Point Biserial column displays the exam-taker’s answer on a single item and the exam-taker’s overall performance on the assessment. (Measures test item discrimination)Range: -1.00 to 1.00A negative result indicates low-scoring exam-takers who performed well on this question than the high-scoring exam-takers.A positive result indicates high-scoring exam-takers performed well on this question than low-scoring exam-takers.Questions that discriminate well have scores between .3 and .7.As a general rule, a point biserial of ≥ .20 is desirable.
15. ExamSoft: Summary Report – Discrimination Index (Question Performance)The Discrimination Index displays the % of high performers who answer an item correctly minus the % of lower performers who answered the item correctly.Range: -1:00 – 1:00-1.00 – Substantially more low performers answered the item correctly than high performers.1.00 – Substantially more high performers answered the item correctly than low performers.0-0.2 – not discriminating0.2-0.4 – starting to become discriminating0.4 and above – strongly discriminatingNote: Most commonly, a negative to 0 discrimination can indicate if a question was misleading or the answer key was incorrectly entered.
16. ExamSoft: Summary Report – Response FrequencyThe Response Frequencies display the distractor analysis of students’ exam answers.Addresses the performance of incorrect response options.Incorrect options should be plausible but incorrect.If no one chooses a particular option, the option is not contributing to the performance of the test.Note: The presence of one or more implausible distractors can make the item artificially easier than it should be.
17. ExamSoft: Item AnalysisFrom the tabs, mouse over Report/Scoring and select Item Analysis
18. ExamSoft: Item Analysis (cont.)From the Item Analysis page check:Question ID/RevQuestion TextInclude the question titleAnswer choice textMultiple Choice Response, Upper/Lower & Disc IndexPerformance HistoryThen click the View Report button
19. ExamSoft: Item Analysis (cont.)The Item Analysis generally shows the same data as the Summary Report with a few exceptions.It shows the full question stem and answer choices.It shows answer choices up through “J”.How many students did not answer the question.Average time students took to answer the question.
20. Let’s Practice!
21. Reading Your Results - ExamSoftLet’s Practice interpreting exam results with the example used in this presentation.ExamSoft – below is question 1 both in the Summary Report and Item Analysis view.First look at the Correct %column.This is the Difficulty Index. Check to see if it is above or below 50%.If it is below, 50%, then proceed to the next step.Our example is 47%, .47.Future Exams: If the Correct % is above 50% still take the time to look at the Point Biserial,Discrimination Index, andResponse Frequencies.These can show other questiondeficiencies, such as being too easy.
22. Reading Your Results (cont.) - ExamSoftNext check the Discrimination Index column.Range: -1:00 – 1:000-0.2 – not discriminating0.2-0.4 – starting to become discriminating0.4 and above – strongly discriminatingThis question’s Discrimination Index is 0.54, which means it is strongly discriminating. Indicating, strong question performance, which is desirable.Note: Negative numbers indicate higher performers answered the question incorrectly. This can indicate the question is potentially confusing or a student is reading too much into the question.
23. Reading Your Results (cont.) - ExamSoftNext check your Point Biserial column.Range: -1.00 to 1.00Questions that discriminate well have scores between .3 and .7.As a general rule, a point biserial of ≥ .20 is desirable.This question’s Point Biserial is 0.46.Indicating, student performance is a health discrimination, which is desirable. Note: Questions that are very easy or very difficult will have low Point Biserial scores. Negative scores suggest:The question was incorrectly keyed.The question was poorly constructed or confusing.The content was inadequately taught.
24. Reading Your Results (cont.) - ExamSoftNext, let’s look at the Response Frequency.The Response Frequencies display the distractor analysis of students’ exam answers.In this question, two of the distractors appeared to be plausible to students.Consider how the question / answers are worded as it may have been confusing to students.Note: Two answers were not selected by students. This means they are not contributing to the exam’s performance. Consider revising them in future versions of the exam.
25. Reading Your Results (cont.) - ExamSoftOverall Results for this question:Below 50% difficulty index (Correct %) – check the Discrimination Index, Point Biserial and Response Frequency.Strong Discrimination Index – 0.54Strong Point Biserial – 0.46Response Frequency – Review distractors that received the most answers and consider revising distractors that received 0 responses for future versions of this exam.Conclusion: This is a strong question but the distractors may be an issue.Consider looking further into the most commonly selected distractors to see if they were confusing. If yes, adjust the answer key to accept both answers, and revise for the future.Consider revising distractors that received 0 responses for future versions of the exam.
26. Further HelpIf you would like to schedule a one-on-one session discussing your exam results, contact:Christina Ryan, Office of Academic Affairs352-273-5949cryan@dental.ufl.eduThank you!
27. ReferencesVallejo-Elias, Johana, “Interpretation of Discimination Data from Multiple-Choice Text Items.” file:///C:/Users/chaskins/Downloads/ExamQuestionStatistics.pdfMaxinity, “Measuring Item Reliability Part 1 – Item Discrimination Index” https://maxinity.co.uk/blog/item-discrimination-index/