Easier to understand Easier to measure More formulas like stats Base for validity Need for validity Does test measure what it claims Can test be used to make decisions Validity Reliability is a necessary but not a sufficient condition for validity ID: 1028097
Download Presentation The PPT/PDF document "Reliability or Validity Reliability gets..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Reliability or ValidityReliability gets more attention:Easier to understandEasier to measureMore formulas (like stats!)Base for validity
2. Need for validityDoes test measure what it claims?Can test be used to make decisions?
3. ValidityReliability is a necessary, but not a sufficient condition for validity.
4. Validity: a definition“A test is valid to the extent that inferences made from it are appropriate, meaningful, and useful”Standards for Educational and Psychological Testing, 1999
5. “Face Validity”“looks good to me”!!!!!!!
6. Trinitarian view of ValidityContent (meaning)Construct (meaning)Criterion (use)
7. 1) Content Validity“How adequately a test samples behaviors representative of the universe of behaviors the test was designed to measure.”
8. Determining Content Validity describe the domain specify areas to be measured compare test to domain
9. Content Validity Ratio (CVR)Agreement among raters if item is:EssentialUseful but not essentialNot necessary
10. 2) Construct validity “A theoretical intangible” “An informed, scientific idea”-- how well the test measures that construct
11. Determining Construct validitybehaviors related to constructsrelated/unrelated constructsidentify relationshipsmulti trait/multi method
12. Multitrait-Multimethod MatrixCorrelate scores from 2 (or more tests)Correlate scores obtained from 2 (or more) methods
13. Evidence of Construct ValidityUpholds theoretical predictionsChanges (?) over time, gender, trainingHomogeneity of questions(internal consistency, factor or item analysis)Convergent/discriminantMultitrait-multimethod matrix
14. Decision Making How well the test can be used to help in decision making about a particular criterion.
15. Decision TheoryBase rateHit rateMiss rateFalse positiveFalse negative
16. 3) Criterion Validity“The relationship between performance on the test and on some other criterion.”
17. Validity coefficientCorrelation between test score and score on criterion measure.
18. Two ways to establish Criterion Validity A) Concurrent validity B) Predictive validity
19. Determining Concurrent validityAssess individuals on constructAdminister test to lo/hi on constructCorrelate test scores to prior identificationUse test later to make decisions
20. Determining Predictive validityGive test to group of peopleFollow up groupAssess laterReview test scoresIf correlate with behavior later can use later to make decisions
21. Incremental validityValue of including more than one predictorBased on multiple regressionWhat is added to prediction not present with previous measures?
22. Expectancy dataTaylor-Russell TableNaylor-Shine TablesToo vague, outdated, biased
23. Unified Validity - Messick“Validity is not a property of the test, but rather the meaning of the scores.”Value implicationsRelevance and utility
24. Unitarian considerationsContentConstructCriterionConsequences
25. Threats to validityConstruct underrepresentation (too narrow)Construct-irrelevant variance (too broad) construct-irrelevant difficulty construct-irrelevant easiness
26. Example 1Dr. Heidi considers using the Scranton Depression Inventory to help identify severity of depression and especially to distinguish depression from anxiety. What evidence should Dr. Heidi use to determine if the test does what she hopes it will do?
27. Example 2The newly published Diagnostic Wonder Test promises to identify children with a mathematics learning disability. How will we know whether the test does so or is simply a slickly packaged general ability test?
28. Example 3Ivy College uses the Western Admissions Test (WTA) to select applicants who should be successful in their studies. What type of evidence should we seek to determine if the WAT satisfies its purpose?
29. Example 4Mike is reviewing a narrative report of his scores on the Nifty Personality Questionnaire (NPQ). The report says he is exceptionally introverted and unusually curious about the world around him. Can Mike have any confidence in these statements or should they be dismissed as equivalent to palm readings at the county fair?
30. Example 5A school system wants to use an achievement battery that will measure the extent to which students are learning the curriculum specified by the school. How should the school system proceed in reviewing the available achievement tests?
31. Project homework question What content or construct is your measure assessing? (explain your answer)What do you think congruent and discriminate constructs would be to the one in your measure?How would you determine the content or construct validity of your measure?How would you determine the criterion validity of your measure?Why would you use those approaches?
32. Project homework question Select a standardized instrument from MMY to use as a comparison for your measure?Copy the relevant data.Why did you select that instrument?How would you use it to help standardize your measure?
33.