1 PHOTO BY NSHIMIYIMANA ALEXIS SOMA UMENYE Using the modified Angoff method to set fluency benchmarks in Rwanda Norma Evans Evans and Associates Sharon Haba Soma Umenye Rwanda 6262019 ID: 805094
Download The PPT/PDF document "6/26/2019 SOMA UMENYE ACTIVITY" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
6/26/2019
SOMA UMENYE ACTIVITY
1
PHOTO BY NSHIMIYIMANA ALEXIS / SOMA UMENYE
Using the modified
Angoff
method to set fluency benchmarks in Rwanda
Norma Evans
Evans and Associates
Sharon
Haba
Soma
Umenye
Rwanda
Slide26/26/2019
2
Why set new standards and benchmarks?
Previous P1 – P3 Kinyarwanda Benchmarks Processes used, challenges encounteredP1-P3 ORF benchmarks established using modified
AngoffProcesses used, challenges encounteredAchievements and next steps
Overview
Slide36/26/2019
3
2013 P3 ORF benchmarks - median method
Non reader
Beginning
Emerging
Competent
Proficient
Benchmark
‘meeting expectations’
P3
0 CWPM
1 to 19 CWPM
20 to 32 CWPM
33 to 47 CWPM
47+ CWPM
33 CWPM
Slide46/26/2019
4
2014 ORF and RC Benchmarks, P1, P2 and P3
Non reader
Beginning
Emerging
Competent
Proficient
Benchmark
‘meeting expectations’
P1
0 CWPM
1 to 19 CWPM
20 to 32 CWPM
33 to 47 CWPM
47+ CWPM
1 CWPM
P2
0 CWPM
1 to 19 CWPM
20 to 32 CWPM
33 to 47 CWPM
47+ CWPM
20 CWPMP30 CWPM1 to 19 CWPM20 to 32 CWPM33 to 47 CWPM47+ CWPM33 CWPM
Sores in red = minimally meeting grade level expectations at each grade level
Slide56/26/2019
5
Difficulty replicating median method
Median method only applicable to EGRA
Documented challenges with median method (Jukes et al., 2018)
Insufficient % of pupils reading with comprehension to produce consistent results
Other challenges
USAID SOMA UMENYE
Slide6Establishing cut scores
“
There are no exact or correct cut scores for a test, but only more or less defensible ones. Defensibility is based largely on the method used to set the limit scores
.’
Ferrara, Perie & Johnson, 2008
6/26/2019
6
Slide76/26/2019
USAID SOMA UMENYE
7
Most defensible method for establishing cut scores?
Have
panels of experts (curriculum specialists, master teachers)
look at the questions asked or the tasks assigned and come to consensus as to whether pupils in each category (not meeting expectations, partially meeting, meeting, exceeding) would be able to answer them correctly.
Zieky
&
Perie
, 2006; Livingston &
Zieky
, 1984;
Zieky
&
Perie
, 2008
Angoff
method
Bookman method
Edel
method
Nedelsky
method
International best practices
Most widely used
Sireci
&
Biskin
, 1992 &
researched
Cisek
, 2012
Only method that can be used for both EGRA and LARS-type assessments
Slide8The modified
Angoff
method
Yes – No method
Slide96/26/2019
USAID SOMA UMENYE
9
For every evaluation instrument
% pupils in each category (short, medium, long term)
Alignment with curriculum
Evaluation specialists – 1 day
Curriculum specialists – 2 days
Evaluation & curriculum specialists (master teachers) 3 days
Decision makers
Slide106/26/2019
USAID SOMA UMENYE
10
Part A: Development of generic assessment framework (policy-level descriptors)
Does not meet expectations
Partially meets expectations
Meets grade-level expectations
Exceeds grade-level expectations
Pupils are performing well below expected levels. They have very limited mastery of the knowledge, skills and competencies outlined in the curriculum. As a result, they cannot complete most basic grade-level tasks.
Pupils are performing below expected levels. They have partial mastery of the knowledge, skills and competencies outlined in the curriculum. As a result, they are able to complete some (or partially complete some) basic, grade-level tasks.
Pupils are performing at expected levels. They have developed sufficient mastery of the knowledge, skills outlined in the curriculum to successfully complete most basic tasks.
Pupils are performing beyond the expected level. They have developed superior mastery of the knowledge, skills and competencies outlined in the curriculum. As a result, they can successfully complete all basic tasks as well as some complex tasks.
Slide116/26/2019
USAID SOMA UMENYE
11
Part B: Adaptation of generic framework to reading in early primary
Does not meet expectations
Partially meets expectations
Meets grade-level expectations
Exceeds grade-level expectations
Pupils are unable to identify most sounds in words or to consistently connect sounds to letters. As a result, they make
many
errors reading the simplest grade-level texts, so many that they not understand most of what they are reading and can only rarely figure out the meaning of new words. They do not like to read.
Pupils read grade-level texts very slowly - usually syllable by syllable – hesitantly, and with limited confidence, particularly when faced with longer texts. They make
some errors
when reading familiar words or simple texts, often skipping over words entirely. They rarely go back to self-correct. As a result, they are able to answer
some
basic literal comprehension questions or figure out the meaning of some new words.
Pupils read
most
grade-level texts accurately. They usually self-correct when they make a mistake and are able to figure out the meaning of most new words by using simple clues in the text or illustrations. They are able to answer
most
literal comprehension questions and demonstrate confidence in their reading abilities.
Pupils read grade-level texts easily and fluently, respecting tones and syllable duration. They are able to figure out the meaning of all new grade-level words and . answer all literal and simple inferential comprehension questions, including making accurate and logical predictions and making personal connections between the text their own lives. They like to read and have confidence in their reading abilities.
Slide126/26/2019
USAID SOMA UMENYE
12
Development of grade-specific performance descriptors
Slide136/26/2019
USAID SOMA UMENYE
13
Alignment of “meets expectations” for P2, P3 with SDG 4.1.1a
Draft SDG 4.1.1 a Performance descriptor
(November 2018)
“Meets grade level expectations” in assessment framework
P2
Students read and comprehend most of the written words in an instrument given to them, particularly familiar ones, and extract explicit information from sentences.
Students read P2-level texts accurately but not necessarily quickly. They are able to answer basic literal comprehension questions. locate information in these texts and understand meaning of words in texts.
P3
Students read aloud written words accurately and fluently. They understand the overall meaning of sentences and short texts. They identify the texts’ topic.
Students read P3 texts with enough accuracy and fluency.
They an
swer literal and some basic inferential questions about these texts, make reasonable predictions of what might happen and summarize accurately main events or ideas.
6/26/2019
USAID SOMA UMENYE
14
Using Modified
Angoff
to set ORF and RC cut scores for different performance categories
30 P1 to P3 master teachers from 5 regions (6 per region)
2 – 3 Kinyarwanda specialists per grade level
P3
P2
P1
Discussion leader
Slide156/26/2019
USAID SOMA UMENYE
15
Teacher selection process
Profile of teachers
Perceived by local or district education authorities as Master early primary teachers
Minimum 5 years of experience teaching Kinyarwanda in Early primary
Currently teaching Kinyarwanda in P1, P2 or P3Active participation in early grade reading trainingsFormer or current early grade reading trainer or facilitatorPublic school teachers
Slide166/26/2019
USAID SOMA UMENYE
16
Region
Experience
Grade 1
Grade 2
Grade 3
North Province
5 to 10 years
1 urban
1 rural
NA
1 urban
1 rural
10+ years
NA
1 urban
1 rural
NA
South province
5 to 10 years
NA
1 urban1 ruralNA10+ years1 urban1 ruralNA1 urban1 ruralTwo Grade 1, 2 and 3 teachers per province
Example, 2 of 5 provinces
Slide176/26/2019
USAID SOMA UMENYE
17
Process – Preparation for Modified
Angoff
Step 1
: Validating performance descriptors for grade level
Step 2: Learning how to collect oral reading fluency dataStep 3:
Collecting ORF and RC data in neighboring school (1/2 day)
Slide186/26/2019
USAID SOMA UMENYE
18
Detour
Slide196/26/2019
USAID SOMA UMENYE
19
Does not meet expectations
Partially meets
Meets expectations
Exceeds expectations
Teachers, curriculum specialists read through the performance description for fluency for “meets expectations ”.
They identified a pupil who would be a “borderline pupil” for this category
Step 4:
Understanding how the
Angoff
method works
Borderline pupil –
Pupil at the lowest point of a performance category
Slide206/26/2019
USAID SOMA UMENYE
20
Step 5:
For the category “meets expectation”
, read down the sentences in the text and circle the last word their borderline pupil would attempt to read in 1 minute
Slide21The
Angoff method – 2 rounds of scoring
Round 1
Round 2
Slide226/26/2019
USAID SOMA UMENYE
22
Number of
cwpm
Step 6:
Sharing and discussing individual assessments (Round 1)
Step 7
:
Re-scoring and sharing Round 2 assessments.
P3
Standard deviation 5.9
Standard
deviation 2.2
Step 8 :
Calculating mean score for Round 1 -
41
Slide236/26/2019
USAID SOMA UMENYE
23
Slide246/26/2019
24
Slide256/26/2019
USAID SOMA UMENYE
25
Draft ORF Standards (CWPM) , Modified
Angoff
0Below categorization
Does not meet
Partially meets
Meets expectations
Exceeds
Angoff
Benchmarks
P1
0
1 to 6
7
to 11
12
to 21
22
+
12
P2
0
1 to 7
8
to 27
28
to 31
32+
28
P3
0
1 to 18
19
to 40
41
to 53
54+
41
Previous Benchmarks
1
20
33
Scores in
red
are the mean, round 2 scores for each performance category
Slide266/26/2019
USAID SOMA UMENYE
26
Regional Comparison, ORF benchmarks, other Bantu languages
Slide276/26/2019
USAID SOMA UMENYE
27
Step 9 – Proposed adjusted scores (+/- 2 SE Mean)
Below categorization
Does not meet
Partially meets
Meets
Exceeds
Benchmarks
P1
0 CWPM
1 to 7
8 to 14
15 to 25
26+
15 CWPM
P2
0 CWPM
1 to 14
15 to 24
25 to 35
36+
25 CWPM
P3
0 CWPM
1 to 24
25 to 39
40 to 50
51+
40 CWPM
Slide286/26/2019
USAID SOMA UMENYE
28
Slide296/26/2019
USAID SOMA UMENYE
29
What we achieved…
Rwanda’s expectations now aligned with regional expectations – and with international expectations (SDG 4.1.1a)
Because the decision-making process was rooted in local expert knowledge, level of confidence that benchmarks are accurate
Ministry ownership of new benchmarks and standards
Change agents (master teachers) in each district who own these benchmarks, can explain and defend them, as well as teach others how to monitor progress towards them.
Learning assessment specialists from all subject areas able to replicate the modified
Angoff
method to set benchmarks and standards for other subject areas (systemic capacity)
Slide306/26/2019
USAID SOMA UMENYE
30
Next steps
Repeat the process, to generate grade or term-specific cut scores and benchmarks for other foundational skills
Implement an accountability framework (3 pilot districts) where teachers measure, each term, pupils’ progress with respect to foundational skills and implement remediation activities for pupils in two lowest categories
Develop an electronic dashboard so that school-based performance with respect to benchmarks is entered, aggregated at local, district and national level and used to direct resources to low-performing schools/areas.
Slide316/26/2019
SOMA UMENYE ACTIVITY
Sharon Haba
Director, MEL
Soma
Umenye
/Chemonicsshaba@soma-umenye.org
Norma Evans
Techncial
Director
Evans and Associates
n.evans.associates@gmail.com
Slide326/26/2019
USAID SOMA UMENYE
32
Recommendations for those of you working with countries thinking about setting stands
Choose your teachers carefully.
Involve local district staff, trainers (those who know teachers in the area) in the selection
Trust your master teachers
. They know the range of children’s reading performance at given grade levels and can describe it.
Teach them to collect fluency data
, and have them collect fluency data on a wide range of pupils before setting standards, so that they are able to attach quantitative descriptors to their qualitative judgments.
Program this for the end of the school year
.
Have your assessments specialists from all disciplines involved in the process
,
so they can lead their own benchmarking sessions for their respective disciplines.
Slide336/26/2019
USAID SOMA UMENYE
33
Comparison, P2 Benchmarks “Meets expectations’ - Median (2014, 2016) and
Angoff
method (2019)
25
30
35
40
45
50
25
cwpm
38
cwpm
35
cwpm
45
cwpm
25
cwpm
Median method, score 25
th
and 75
th
percentile
38
cwpm
2014
2016
2019 ANGOFF method
35
cwpmv
Slide346/26/2019
USAID SOMA UMENYE
34
Meets expectations
Round 1
Round 2
P1
Median
14
12
Min
12
12
Max
20
17
P2
Median
23
28
Min
4
22
Max
30
30
P3
P3
Median
34
41
Min
25
37
Max
42
44
Slide356/26/2019
USAID SOMA UMENYE
35
Modified
Angoff
method – 3 part process
2 days
3 days