Why So Much Attention on Rubric Quality? CAEP
Description: Why So Much Attention on Rubric Quality? CAEP Standard 5, Component 5.2: The providers quality assurance system relies on relevant, verifiable, representative, cumulative and actionable measures, and produces empirical evidence that
Related Topics
Download Presentation
"Why So Much Attention on Rubric Quality? CAEP" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Why So Much Attention on Rubric Quality? CAEP Standard 5, Component 5.2:
The provider’s quality assurance system relies on relevant, verifiable, representative, cumulative and actionable measures, and produces empirical evidence that interpretations of data are valid and consistent [emphasis added].
And…<br>
slide2. Optional CAEP Review of Assessment Instruments CAEP has announced that its accreditation process will now allow the early submission of all key assessment instruments (rubrics, surveys, etc.) used by an Educator Preparation Provider (EPP) to generate data provided as evidence in support of CAEP accreditation. Once CAEP accreditation timelines are fully implemented, this will occur three years prior to the on-site visit to allow EPPs time to react to formative feedback from CAEP.<br>
slide3. A Reality Check Regarding Current Rubrics: Commonly Encountered Weaknesses Using overly broad criteria
Using double- or multiple-barreled criteria
Using overlapping performance descriptors
Failing to include all possible performance outcomes
Using double-barreled descriptors that derail actionability
Using subjective terms, performance level labels (or surrogates), or inconsequential terms to differentiate performance levels
Failing to maintain the integrity of target learning outcomes: a common result of having multiple levels of “mastery”<br>
slide4. Overly Broad Criterion<br>
slide5. Double-barreled Criterion<br>
slide6. Overlapping Performance Levels<br>
slide7. Possible Gap in Performance Levels<br>
slide8. Double-barreled Descriptor<br>
slide9. Use of Subjective Terms<br>
slide10. Use of Performance Level Labels<br>
slide11. Use of Surrogates for Performance Levels<br>
slide12. Use of Inconsequential Terms<br>
slide13. Failure to Maintain Integrity of Target Learning Outcomes<br>
slide14. Value-Added of High Quality Rubrics For Candidates, Well-designed Rubrics Can: Serve as an effective learning scaffold by clarifying formative and summative learning objectives (i.e., clearly describe expected performance at key formative transition points and at program completion)
Identify the critical indicators aligned to applicable standards/ competencies (=construct & content validity) for each target learning outcome
Facilitate self- and peer-evaluations<br>
slide15. Value-Added of High Quality Rubrics For Faculty, Well-designed Rubrics Can: Improve assessment of candidates’ performance by:
Providing a consistent framework for key assessments
Ensuring the consistent use of a set of critical indicators (i.e., the rubric criteria) for each competency
Establishing clear/concrete performance descriptors for each assessed criterion at each performance level
Help ensure strong articulation of formative and summative assessments
Improve validity and inter- and intra-rater reliability of assessment data
Produce actionable candidate- and program-level data<br>
slide16. Attributes of an Effective Rubric Rubric and the assessed activity or artifact are well-articulated.<br>
slide17. Attributes of an Effective Rubric (cont.) Rubric has construct validity (i.e., it measures the right competency/ies) and content validity (rubric criteria represent all critical indicators for the competency/ies to be assessed).
How do you ensure this?<br>
slide18. Attributes of an Effective Rubric (cont.) Each criterion assesses an individual construct
No overly broad criteria
No double- or multiple-barreled criteria<br>
slide19. Attributes of an Effective Rubric (cont.) To enhance reliability, performance descriptors should:
Provide clear/concrete distinctions between performance levels (there is no overlap between performance levels)
Collectively address all possible performance levels (there is no gap between performance levels)
Eliminate (or carefully construct the inclusion of) double/multiple-barrel narratives<br>
slide20. Attributes of an Effective Rubric (cont.) Contains no unnecessary performance levels. Common problems encountered when multiple levels of mastery are present include:
Use of subjective terms to differentiate performance levels
Use of performance level labels or surrogates
Use of inconsequential terms to differentiate performance levels
Worst case scenario: failure to maintain the integrity of target learning outcomes<br>
slide21. Attributes of an Effective Rubric Resulting data are actionable
To remediate individual candidates
To help identify opportunities for program quality improvement
Based on the first four attributes, the following meta-rubric has been developed for use in evaluating the efficacy of other rubrics…<br>
slide22. Some Additional Helpful Hints In designing rubrics for key formative and summative assessments, think about both effectiveness and efficiency
Identify critical indicators for target learning outcomes and incorporate those into your rubric
Limit the number of performance levels to the minimum number needed to meet your assessment requirements. HIGHER RESOLUTION IS BEST ACHIEVED WITH MULTIPLE FORMATIVE LEVELS, NOT MULTIPLE LEVELS OF MASTERY!
Populate the target learning outcome column first (Proficient, Mastery, etc.)
Make clear (objective/concrete) distinctions between performance levels; avoid the use of subjective terms in performance descriptors
Be sure to include all possible outcomes
Don’t leave validity and reliability to chance
Most knowledgeable faculty should lead program-level assessment work; engage stakeholders; align key assessments to applicable standards/competencies; focus on critical indicators
Train faculty on the use of rubrics
Conduct and document inter-rater reliability and fairness studies<br>
The provider’s quality assurance system relies on relevant, verifiable, representative, cumulative and actionable measures, and produces empirical evidence that interpretations of data are valid and consistent [emphasis added].
And…<br>
slide2. Optional CAEP Review of Assessment Instruments CAEP has announced that its accreditation process will now allow the early submission of all key assessment instruments (rubrics, surveys, etc.) used by an Educator Preparation Provider (EPP) to generate data provided as evidence in support of CAEP accreditation. Once CAEP accreditation timelines are fully implemented, this will occur three years prior to the on-site visit to allow EPPs time to react to formative feedback from CAEP.<br>
slide3. A Reality Check Regarding Current Rubrics: Commonly Encountered Weaknesses Using overly broad criteria
Using double- or multiple-barreled criteria
Using overlapping performance descriptors
Failing to include all possible performance outcomes
Using double-barreled descriptors that derail actionability
Using subjective terms, performance level labels (or surrogates), or inconsequential terms to differentiate performance levels
Failing to maintain the integrity of target learning outcomes: a common result of having multiple levels of “mastery”<br>
slide4. Overly Broad Criterion<br>
slide5. Double-barreled Criterion<br>
slide6. Overlapping Performance Levels<br>
slide7. Possible Gap in Performance Levels<br>
slide8. Double-barreled Descriptor<br>
slide9. Use of Subjective Terms<br>
slide10. Use of Performance Level Labels<br>
slide11. Use of Surrogates for Performance Levels<br>
slide12. Use of Inconsequential Terms<br>
slide13. Failure to Maintain Integrity of Target Learning Outcomes<br>
slide14. Value-Added of High Quality Rubrics For Candidates, Well-designed Rubrics Can: Serve as an effective learning scaffold by clarifying formative and summative learning objectives (i.e., clearly describe expected performance at key formative transition points and at program completion)
Identify the critical indicators aligned to applicable standards/ competencies (=construct & content validity) for each target learning outcome
Facilitate self- and peer-evaluations<br>
slide15. Value-Added of High Quality Rubrics For Faculty, Well-designed Rubrics Can: Improve assessment of candidates’ performance by:
Providing a consistent framework for key assessments
Ensuring the consistent use of a set of critical indicators (i.e., the rubric criteria) for each competency
Establishing clear/concrete performance descriptors for each assessed criterion at each performance level
Help ensure strong articulation of formative and summative assessments
Improve validity and inter- and intra-rater reliability of assessment data
Produce actionable candidate- and program-level data<br>
slide16. Attributes of an Effective Rubric Rubric and the assessed activity or artifact are well-articulated.<br>
slide17. Attributes of an Effective Rubric (cont.) Rubric has construct validity (i.e., it measures the right competency/ies) and content validity (rubric criteria represent all critical indicators for the competency/ies to be assessed).
How do you ensure this?<br>
slide18. Attributes of an Effective Rubric (cont.) Each criterion assesses an individual construct
No overly broad criteria
No double- or multiple-barreled criteria<br>
slide19. Attributes of an Effective Rubric (cont.) To enhance reliability, performance descriptors should:
Provide clear/concrete distinctions between performance levels (there is no overlap between performance levels)
Collectively address all possible performance levels (there is no gap between performance levels)
Eliminate (or carefully construct the inclusion of) double/multiple-barrel narratives<br>
slide20. Attributes of an Effective Rubric (cont.) Contains no unnecessary performance levels. Common problems encountered when multiple levels of mastery are present include:
Use of subjective terms to differentiate performance levels
Use of performance level labels or surrogates
Use of inconsequential terms to differentiate performance levels
Worst case scenario: failure to maintain the integrity of target learning outcomes<br>
slide21. Attributes of an Effective Rubric Resulting data are actionable
To remediate individual candidates
To help identify opportunities for program quality improvement
Based on the first four attributes, the following meta-rubric has been developed for use in evaluating the efficacy of other rubrics…<br>
slide22. Some Additional Helpful Hints In designing rubrics for key formative and summative assessments, think about both effectiveness and efficiency
Identify critical indicators for target learning outcomes and incorporate those into your rubric
Limit the number of performance levels to the minimum number needed to meet your assessment requirements. HIGHER RESOLUTION IS BEST ACHIEVED WITH MULTIPLE FORMATIVE LEVELS, NOT MULTIPLE LEVELS OF MASTERY!
Populate the target learning outcome column first (Proficient, Mastery, etc.)
Make clear (objective/concrete) distinctions between performance levels; avoid the use of subjective terms in performance descriptors
Be sure to include all possible outcomes
Don’t leave validity and reliability to chance
Most knowledgeable faculty should lead program-level assessment work; engage stakeholders; align key assessments to applicable standards/competencies; focus on critical indicators
Train faculty on the use of rubrics
Conduct and document inter-rater reliability and fairness studies<br>