Rosenkrantz MS EdS Guiding Steps for the Evaluation or Creation of a Scale A Starter Kit March 20 2018 Applied Psychometric Strategies Lab Applied Quantitative and Psychometric Series ID: 744399
Download Presentation The PPT/PDF document "Abbey Love, MS, & Dani" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Abbey Love, MS, & Dani Rosenkrantz, MS, EdSGuiding Steps for the Evaluation or Creation of a Scale: A Starter KitMarch 20, 2018
Applied Psychometric Strategies Lab
Applied Quantitative and Psychometric SeriesSlide2
What is a “bad” scale?I saw in someone’s dissertation a newly published scale…
It depends!Slide3
But, will bad measurement really hurt my study?Slide4
Yes! That’s why we are here. Big Picture: Poor measurement is an ethical concern because
If the measurement is problematic, the reliability of our findings is compromised
…in other words…Our degree of trust in our results is in question, which means our statistical conclusion validity is in question! Slide5
Sample specific measurement challenges that may occurLow reliability or large measurement error around scoresUncertainty about using a total score for each person or subscale scores for each person
Not all response options are being used or low item response variabilityPoor factor structure solutions due to cross-loading issues on multiple factors, low loadings on factors, and/or influences such as item phrasingSlide6
Examples From Applied WorkCognitive Flexibility Inventory for DissertationOnly original paper explored factor structureCross loading issues on originalPoor recovery of factor structure in my sampleHad to reduce items for better fit, a controversial decision
Internal structure assessment of Objectified Body Consciousness ScalePoor fit with trans women, indicating inappropriate to use without further studySlide7
Why does good measurement matter?Slide8
If you use a well established measure you will likely find the followingHigh reliability which will result in low measurement erroraccurate effect size estimates (d, R2
)better captures effects of interest (Beta, path coefficients)improves inferential techniques (more accurate SEs and ultimately statistical decisions)Confidence in how to score the scale (total and/or subscales)
All response options are being used Strong recovery of factor structure solution with near zero cross-loadings across factors, high loadings on intended factors, and/or minimal influence due to method factorsSlide9
Looking for a Scale? Guiding Steps for the Evaluation or Creation of a Scale
1.) Evaluation of psychological scales - Should I use a scale I found?
2.) Scale development – I can’t find a scale. What steps do I need to develop a scale to measure a psychological construct?Slide10
How do I know if my scale is “good?”Slide11
Good scales… have ongoing and multiple sources of evidence that can be used to evaluate the validity of the interpretation of the scale for a particular use.Slide12
Sources of Validity in Instrument Development
Evidence based on...
Test contentAm I measuring what I planned to measure? Slide13
Sources of Validity in Instrument Development
Evidence based on...
Test contentAm I measuring what I planned to measure? Reponses processesAre my participants understanding the items on my scale in an expected way? Slide14
Sources of Validity in Instrument Development
Evidence based on...
Test contentAm I measuring what I planned to measure? Reponses processesAre my participants understanding the items on my scale in an expected way? Relations to other variables
Do
the items I have chosen to represent my construct relate to other variables in an expected way? This can include convergent and discriminant evidence. Slide15
Sources of Validity in Instrument Development
Evidence based on...
Test contentAm I measuring what I planned to measure? Reponses processesAre my participants understanding the items on my scale in an expected way? Relations to other variables
Do
the items I have chosen to represent my construct relate to other variables in an expected way? This can include convergent and discriminant evidence.
Internal structure
What
is the degree to which the items on my scale are conforming to the construct and how I intend to interpret the scale? Slide16
Sources of Validity in Instrument Development
Evidence based on...
Test contentAm I measuring what I planned to measure? Reponses processesAre my participants understanding the items on my scale in an expected way? Relations to other variables
Do
the items I have chosen to represent my construct relate to other variables in an expected way? This can include convergent and discriminant evidence.
Internal structure
What
is the degree to which the items on my scale are conforming to the construct and how I intend to interpret the scale? Slide17
Sources of Validity in Instrument Development
Evidence based on...
Test contentLiterature review, content specification, expert judgesReponses processesCognitive interviewsRelations to other variablesAnalysis
of the relationship of the scale scores to variables external to the scale (correlational evidence)
Internal structure
Factor analysis,
measurement invariance
**See
Standards for Educational and Psychological Testing
Slide18
Determining If You Should Use An InstrumentLength and ContentDoes the scale represent the breadth of the construct?
Reliability Is the score reliability from your scale reasonable, using similar samples?*
*Depends on the seriousness/specificity of the reliability issue…Slide19
Determining If You Should Use An InstrumentPrevious Samples
Has the scale been used with samples similar to your sample of interest? Intended PerformanceHas the scale previously performed as intended based on review of past psychometric analyses?
EFA, CFA, correlational, SEMSlide20
Determining If You Should Use An InstrumentScoring How has the scale been scored in the past?
Was sufficient testing done to evaluate appropriateness of using a total score, if needed? Slide21
Is one EFA okay?When is it enough?Consider whether there is psychometric evidence for your specific sampleSlide22
Where can I find good scales?Slide23
Where To Find Instruments: Literature Review the literature on your construct and scales that measure your construct, paying close attention to:Definitions about the constructReliability
Factor StructureSubscales vs. total scoresExploratory Factor AnalysisConfirmatory Factor Analysis
Validation sampleMeasurement invarianceSlide24
Where To Find Instruments: Reviews Mental Measurements Yearbook (MMY)A tool to locate information about commercial tests and measuresIssues from 1938-2017
Provides factual information on published testsCritical test reviews written by:Professionals and psychometricians in education, psychology, speech/language/hearing, law, health care, and other related fields Slide25
Developing an Instrument If NeededSlide26
Developing an Instrument If NeededSlide27
Developing an Instrument If NeededSlide28
Developing an Instrument If NeededSlide29
Developing an Instrument If NeededSlide30Slide31
Best Practices In Instrument Development Recognize Instrument Development as an ongoing process, not a one time eventSlide32
Helpful References for Scale ConstructionAmerican Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.
DeVellis, R. F. (2012). Scale development: Theory and applications. Los Angeles, CA: Sage.
Kline, P. (1986). Making tests reliable II: Personality inventories. In P. Kline (Ed), A Handbook of Test Construction: Introduction to psychometric design (pp. 59-76). London, United Kingdom: Methuen. Thorndike, R. M., & Thorndike-Christ, T. (2010). Measurement and evaluation in psychology and education. Boston, MA: Prentice Hall.Willis, G. B. & Artino, A. R. (2013). What do our respondents think we’re asking? Using cognitive interviewing to improve medical education surveys. Journal of Graduate Medical Education, 5, 353-356. doi:10.4300/JGME-D-13-00154.1 Slide33
What things did we want to get into, but did not have time to do so?Best practices in CFA and EFABifactor AnalysesSEMUsing IRTMeasurement Invariance or DIFCognitive diagnostic modelsMultilevel SEM and IRTSlide34
What did I learn?Think twice before using a scale to measure a construct of interestGood measurement mattersInstrument development is an ongoing processConsider gathering psychometric evidence to support the intended use of the scale within your studySlide35Slide36Slide37
Glossary of Terms
Psychological ConstructVariable that is not directly observable
PsychometricsThe science of measuring mental capabilities and processesMeasurement ErrorDifference between the participants’ answers and the true scoreTotal ScoringScores from multiple items are summed to create a composite score Factor AnalysisA method to explore the underlying structure of a set of items Negatively phrased itemsExample: I feel unhappy. Positively phrased itemsExample: I feel happy.Factor pattern loadingThe contribution of an item to the latent factor (think standardized regression weights)Slide38
Attitudes Towards Computers [Strongly disagree] [Disagree] [Agree] [Strongly Agree] 1. A computer is a useful tool for my everyday life. 2. Using a computer makes my life easier.3. Computers help me gain access to my community.
4. Computers allow me to learn new things.6. Having a computer at my house enables me to stay connected.7. Computers are a negative influence on my life.Slide39
Attitudes Towards Computers [Strongly disagree] [Disagree] [Agree] [Strongly Agree] 1. A computer is a useful tool for everyday life. 2. Using a computer makes my life easier.3. Computers help me gain access to my community.
4. Computers allow me to learn new things.5. Computers make my life better.6. Having a computer at my house enables me to stay connected.
7. Computers are a negative influence on society.PSYCHOLOGICAL CONSTRUCTSlide40
Attitudes Towards Computers [Strongly disagree] [Disagree] [Agree] [Strongly Agree] 1. A computer is a useful tool for everyday life. 2. Using a computer makes my life easier.3. Computers help me gain access to my community.
4. Computers allow me to learn new things.5. Computers make my life better.6. Having a computer at my house enables me to stay connected.
7. Computers are a negative influence on society.RESPONSE FORMATTOTAL SCORESlide41
Attitudes Towards Computers [Strongly disagree] [Disagree] [Agree] [Strongly Agree] 1. A computer is a useful tool for everyday life. +2. Using a computer makes my life easier.
+3. Computers help me gain access to my community.+4. Computers allow me to learn new things.
+5. Computers make my life better. +6. Having a computer at my house enables me to stay connected. +7. Computers are a negative influence on society. -+ POSITIVELY PHRASED ITEMS- NEGATIVELY PHRASED ITEMSSlide42
Measurement Issues That Arise From Instruments With Weak Psychometric EvidenceProblematic scales may not perform as designed in your sample Inconsistent results across studies
Results we cannot trustSlide43
Handling Gaps in the LiteratureLimited dimensionality or internal structure assessment studies*There is some debate on this; some say do CFA if history of EFAs present in literature VS some say do CFA based on theorySlide44
xxxx