A Preliminary Psychometric Investigation SmartEvals and Psychology Departmental Instrument Dr Chin and Dr Nettelhorst Winter 2018 Why Psychometrics Validity is simply the property of an assessment tool that indicates the tool does what it says it does ID: 761404
Download Presentation The PPT/PDF document "A Preliminary Psychometric Investigation..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
A Preliminary Psychometric Investigation: SmartEvals and Psychology Departmental Instrument Dr. Chin and Dr. Nettelhorst Winter 2018
Why Psychometrics?Validity is simply the property of an assessment tool that indicates the tool does what it says it doesAnd if it does that, then test scores actually have meaning!Wrapped up in the concept of validity, is reliabilityWhen we administer a test, we would like to know how much of their scores reflects “truth” and how much reflects error
Methods for Estimating ValidityContent ValidityCriterion Related ValidityFactorial ValidityExploratory Factor Analysis (EFA)Confirmatory Factor Analysis (CFA)Construct Validity
Methods for Estimating ReliabilityTest-Retest ReliabilityAlternate Forms of ReliabilityInternal Consistency ReliabilitySplit Half ReliabilityCronbach’s Coefficient AlphaInter-Rater Reliability
Overall Analytical StrategyDepartmental Instrument N = 76Exploratory Factor Analysis (EFA)Internal Consistency Reliability (Cronbach’s Alpha)SmartEvals Instrument N = 57Exploratory Factor Analysis (EFA) Internal Consistency Reliability (Cronbach’s Alpha)
Departmental InstrumentSeventeen-item self-report measureExample: ”The instructor was enthusiastic about the subject matter”Likert-type scaleStrongly AgreeAgreeNeither Agree nor DisagreeDisagreeStrongly Disagree
Departmental-EFA Model SpecificationsConducted three standard EFAs with oblique geomin rotation.1-factor, 2-factor, 3-factors, and 4-factorsConducted two bifactor EFAs with bi-geomin rotationOne higher order factor and two subfactors One higher order factor and three subfactors
Departmental-EFA Model SpecificationsUsed weighted least squares estimation (WLSMV) to model the data. Inspection of EigenvaluesExamined Global Fit IndicesHigh CFI, TLI values (>.95), low RMSEA (<.06), and low SRMR (>.08) values were interpreted as indicative of strong model fit.Examined Factor Loadings> .30
Departmental-Bifactor model with two subfactors Refer to Handout for Pattern of Factor Loadings
Perceived Competency Student Assistance V1 E1 E2 E3 E4 E5 E6 E7 E8 E9 E10 E11 E12 E13 E14 E15 E16 E17 V2 V3 V4 V5 V6 V7 V8 V9 V10 V11 V12 V13 V14 V15 V16 V17 Teaching Effectiveness
Departmental-Bifactor model with two subfactorsReliability
Departmental-Bifactor model with two subfactors Reliability Total Score .92 Excellent) Factor 1 (Perceptions of Competency) .76 (Acceptable) Factor 2 (Assistance to Students) .83 (Good)
SmartEvals InstrumentTen-item self-report measure (excluding items 1 and 2)Example: ”The expectations of class were clearly communicated at the beginning of the course”Likert-type scaleStrongly AgreeAgreeNeutralDisagree Strongly Disagree
SmartEvals-EFA Model SpecificationsConducted two standard EFAs with oblique geomin rotation.1-factor and 2-factor modelsUsed weighted least squares estimation (WLSMV) to model the data. Inspection of EigenvaluesExamined Global Fit Indices High CFI, TLI values (>.95), low RMSEA (<.06), and low SRMR (>.08) values were interpreted as indicative of strong model fit. Examined Factor Loadings > .30
SmartEvals-One factor model Refer to Handout for Pattern of Factor Loadings
SmartEvals-Two factor model Refer to Handout for Pattern of Factor Loadings
V1 E1 E2 E3 E4 E5 E6 E7 E8 E9 E10 V2 V3 V4 V5 V6 V7 V8 V9 V10 Teaching Effectiveness Cronbach’s Alpha = .865 (Good)
Implications & Future ResearchPreliminary evidence supports the use of total scores for SmartEvals and Departmental InstrumentLarger sample size needed to determine generalizability of resultsConcerns over overfitting the data due to small sample sizeOther types of validity:Convergent Validity: Does the SmartEval function equally when administered in class versus online? Divergent Validity: Does the SmartEval total score distinguish between overall course quality and students’ grades in the course? Construct Validity: Can the SmartEvals detect changes in teaching effectiveness when instructors implement a novel teaching technique?
Thank you!