6/26/2019 SOMA UMENYE ACTIVITY - PowerPoint Presentation

Slide1

6/26/2019

SOMA UMENYE ACTIVITY

1

PHOTO BY NSHIMIYIMANA ALEXIS / SOMA UMENYE

Using the modified

Angoff

method to set fluency benchmarks in Rwanda

Norma Evans

Evans and Associates

Sharon

Haba

Soma

Umenye

Rwanda

Slide2

6/26/2019

2

Why set new standards and benchmarks?

Previous P1 – P3 Kinyarwanda Benchmarks Processes used, challenges encounteredP1-P3 ORF benchmarks established using modified

AngoffProcesses used, challenges encounteredAchievements and next steps

Overview

Slide3

6/26/2019

3

2013 P3 ORF benchmarks - median method

Non reader

Beginning

Emerging

Competent

Proficient

Benchmark

‘meeting expectations’

P3

0 CWPM

1 to 19 CWPM

20 to 32 CWPM

33 to 47 CWPM

47+ CWPM

33 CWPM

Slide4

6/26/2019

4

2014 ORF and RC Benchmarks, P1, P2 and P3

Non reader

Beginning

Emerging

Competent

Proficient

Benchmark

‘meeting expectations’

P1

0 CWPM

1 to 19 CWPM

20 to 32 CWPM

33 to 47 CWPM

47+ CWPM

1 CWPM

P2

0 CWPM

1 to 19 CWPM

20 to 32 CWPM

33 to 47 CWPM

47+ CWPM

20 CWPMP30 CWPM1 to 19 CWPM20 to 32 CWPM33 to 47 CWPM47+ CWPM33 CWPM

Sores in red = minimally meeting grade level expectations at each grade level

Slide5

6/26/2019

5

Difficulty replicating median method

Median method only applicable to EGRA

Documented challenges with median method (Jukes et al., 2018)

Insufficient % of pupils reading with comprehension to produce consistent results

Other challenges

USAID SOMA UMENYE

Slide6

Establishing cut scores

“

There are no exact or correct cut scores for a test, but only more or less defensible ones. Defensibility is based largely on the method used to set the limit scores

.’

Ferrara, Perie & Johnson, 2008

6/26/2019

6

Slide7

6/26/2019

USAID SOMA UMENYE

7

Most defensible method for establishing cut scores?

Have

panels of experts (curriculum specialists, master teachers)

look at the questions asked or the tasks assigned and come to consensus as to whether pupils in each category (not meeting expectations, partially meeting, meeting, exceeding) would be able to answer them correctly.

Zieky

&

Perie

, 2006; Livingston &

Zieky

, 1984;

Zieky

&

Perie

, 2008

Angoff

method

Bookman method

Edel

method

Nedelsky

method

International best practices

Most widely used

Sireci

&

Biskin

, 1992 &

researched

Cisek

, 2012

Only method that can be used for both EGRA and LARS-type assessments

Slide8

The modified

Angoff

method

Yes – No method

Slide9

6/26/2019

USAID SOMA UMENYE

9

For every evaluation instrument

% pupils in each category (short, medium, long term)

Alignment with curriculum

Evaluation specialists – 1 day

Curriculum specialists – 2 days

Evaluation & curriculum specialists (master teachers) 3 days

Decision makers

Slide10

6/26/2019

USAID SOMA UMENYE

10

Part A: Development of generic assessment framework (policy-level descriptors)

Does not meet expectations

Partially meets expectations

Meets grade-level expectations

Exceeds grade-level expectations

Pupils are performing well below expected levels. They have very limited mastery of the knowledge, skills and competencies outlined in the curriculum. As a result, they cannot complete most basic grade-level tasks.

Pupils are performing below expected levels. They have partial mastery of the knowledge, skills and competencies outlined in the curriculum. As a result, they are able to complete some (or partially complete some) basic, grade-level tasks.

Pupils are performing at expected levels. They have developed sufficient mastery of the knowledge, skills outlined in the curriculum to successfully complete most basic tasks.

 

Pupils are performing beyond the expected level. They have developed superior mastery of the knowledge, skills and competencies outlined in the curriculum. As a result, they can successfully complete all basic tasks as well as some complex tasks.

Slide11

6/26/2019

USAID SOMA UMENYE

11

Part B: Adaptation of generic framework to reading in early primary

Does not meet expectations

Partially meets expectations

Meets grade-level expectations

Exceeds grade-level expectations

Pupils are unable to identify most sounds in words or to consistently connect sounds to letters. As a result, they make

many

errors reading the simplest grade-level texts, so many that they not understand most of what they are reading and can only rarely figure out the meaning of new words. They do not like to read.

 

Pupils read grade-level texts very slowly - usually syllable by syllable – hesitantly, and with limited confidence, particularly when faced with longer texts. They make

some errors

when reading familiar words or simple texts, often skipping over words entirely. They rarely go back to self-correct. As a result, they are able to answer

some

basic literal comprehension questions or figure out the meaning of some new words.

Pupils read

most

grade-level texts accurately. They usually self-correct when they make a mistake and are able to figure out the meaning of most new words by using simple clues in the text or illustrations. They are able to answer

most

literal comprehension questions and demonstrate confidence in their reading abilities.

Pupils read grade-level texts easily and fluently, respecting tones and syllable duration. They are able to figure out the meaning of all new grade-level words and . answer all literal and simple inferential comprehension questions, including making accurate and logical predictions and making personal connections between the text their own lives. They like to read and have confidence in their reading abilities.

Slide12

6/26/2019

USAID SOMA UMENYE

12

Development of grade-specific performance descriptors

Slide13

6/26/2019

USAID SOMA UMENYE

13

Alignment of “meets expectations” for P2, P3 with SDG 4.1.1a

Draft SDG 4.1.1 a Performance descriptor

(November 2018)

“Meets grade level expectations” in assessment framework 

P2

Students read and comprehend most of the written words in an instrument given to them, particularly familiar ones, and extract explicit information from sentences.

Students read P2-level texts accurately but not necessarily quickly. They are able to answer basic literal comprehension questions. locate information in these texts and understand meaning of words in texts.

 

P3

Students read aloud written words accurately and fluently. They understand the overall meaning of sentences and short texts. They identify the texts’ topic.

Students read P3 texts with enough accuracy and fluency.

They an

swer literal and some basic inferential questions about these texts, make reasonable predictions of what might happen and summarize accurately main events or ideas.

Slide14

6/26/2019

USAID SOMA UMENYE

14

Using Modified

Angoff

to set ORF and RC cut scores for different performance categories

30 P1 to P3 master teachers from 5 regions (6 per region)

2 – 3 Kinyarwanda specialists per grade level

P3

P2

P1

Discussion leader

Slide15

6/26/2019

USAID SOMA UMENYE

15

Teacher selection process

Profile of teachers

Perceived by local or district education authorities as Master early primary teachers

Minimum 5 years of experience teaching Kinyarwanda in Early primary

Currently teaching Kinyarwanda in P1, P2 or P3Active participation in early grade reading trainingsFormer or current early grade reading trainer or facilitatorPublic school teachers

Slide16

6/26/2019

USAID SOMA UMENYE

16

Region

Experience

Grade 1

Grade 2

Grade 3

North Province

5 to 10 years

1 urban

1 rural

NA

1 urban

1 rural

10+ years

NA

1 urban

1 rural

NA

South province

5 to 10 years

NA

1 urban1 ruralNA10+ years1 urban1 ruralNA1 urban1 ruralTwo Grade 1, 2 and 3 teachers per province

Example, 2 of 5 provinces

Slide17

6/26/2019

USAID SOMA UMENYE

17

Process – Preparation for Modified

Angoff

Step 1

: Validating performance descriptors for grade level

Step 2: Learning how to collect oral reading fluency dataStep 3:

Collecting ORF and RC data in neighboring school (1/2 day)

Slide18

6/26/2019

USAID SOMA UMENYE

18

Detour

Slide19

6/26/2019

USAID SOMA UMENYE

19

Does not meet expectations

Partially meets

Meets expectations

Exceeds expectations

Teachers, curriculum specialists read through the performance description for fluency for “meets expectations ”.

They identified a pupil who would be a “borderline pupil” for this category

Step 4:

Understanding how the

Angoff

method works

Borderline pupil –

Pupil at the lowest point of a performance category

Slide20

6/26/2019

USAID SOMA UMENYE

20

Step 5:

For the category “meets expectation”

, read down the sentences in the text and circle the last word their borderline pupil would attempt to read in 1 minute

Slide21

The

Angoff method – 2 rounds of scoring

Round 1

Round 2

Slide22

6/26/2019

USAID SOMA UMENYE

22

Number of

cwpm

Step 6:

Sharing and discussing individual assessments (Round 1)

Step 7

:

Re-scoring and sharing Round 2 assessments.

P3

Standard deviation 5.9

Standard

deviation 2.2

Step 8 :

Calculating mean score for Round 1 -

41

Slide23

6/26/2019

USAID SOMA UMENYE

23

Slide24

6/26/2019

24

Slide25

6/26/2019

USAID SOMA UMENYE

25

Draft ORF Standards (CWPM) , Modified

Angoff

0Below categorization

Does not meet

Partially meets

Meets expectations

Exceeds

Angoff

Benchmarks

P1

0

1 to 6

7

to 11

12

to 21

22

+

12

P2

0

1 to 7

8

to 27

28

to 31

32+

28

P3

0

1 to 18

19

to 40

41

to 53

54+

41

Previous Benchmarks

1

20

33

Scores in

red

are the mean, round 2 scores for each performance category

Slide26

6/26/2019

USAID SOMA UMENYE

26

Regional Comparison, ORF benchmarks, other Bantu languages

Slide27

6/26/2019

USAID SOMA UMENYE

27

Step 9 – Proposed adjusted scores (+/- 2 SE Mean)

Below categorization

Does not meet

Partially meets

Meets

Exceeds

Benchmarks

P1

0 CWPM

1 to 7

8 to 14

15 to 25

26+

15 CWPM

P2

0 CWPM

1 to 14

15 to 24

25 to 35

36+

25 CWPM

P3

0 CWPM

1 to 24

25 to 39

40 to 50

51+

40 CWPM

Slide28

6/26/2019

USAID SOMA UMENYE

28

Slide29

6/26/2019

USAID SOMA UMENYE

29

What we achieved…

Rwanda’s expectations now aligned with regional expectations – and with international expectations (SDG 4.1.1a)

Because the decision-making process was rooted in local expert knowledge, level of confidence that benchmarks are accurate

Ministry ownership of new benchmarks and standards

Change agents (master teachers) in each district who own these benchmarks, can explain and defend them, as well as teach others how to monitor progress towards them.

Learning assessment specialists from all subject areas able to replicate the modified

Angoff

method to set benchmarks and standards for other subject areas (systemic capacity)

Slide30

6/26/2019

USAID SOMA UMENYE

30

Next steps

Repeat the process, to generate grade or term-specific cut scores and benchmarks for other foundational skills

Implement an accountability framework (3 pilot districts) where teachers measure, each term, pupils’ progress with respect to foundational skills and implement remediation activities for pupils in two lowest categories

Develop an electronic dashboard so that school-based performance with respect to benchmarks is entered, aggregated at local, district and national level and used to direct resources to low-performing schools/areas.

Slide31

6/26/2019

SOMA UMENYE ACTIVITY

Sharon Haba

Director, MEL

Soma

Umenye

/Chemonicsshaba@soma-umenye.org

Norma Evans

Techncial

Director

Evans and Associates

n.evans.associates@gmail.com

Slide32

6/26/2019

USAID SOMA UMENYE

32

Recommendations for those of you working with countries thinking about setting stands

Choose your teachers carefully.

Involve local district staff, trainers (those who know teachers in the area) in the selection

Trust your master teachers

. They know the range of children’s reading performance at given grade levels and can describe it.

Teach them to collect fluency data

, and have them collect fluency data on a wide range of pupils before setting standards, so that they are able to attach quantitative descriptors to their qualitative judgments.

Program this for the end of the school year

.

Have your assessments specialists from all disciplines involved in the process

,

so they can lead their own benchmarking sessions for their respective disciplines.

Slide33

6/26/2019

USAID SOMA UMENYE

33

Comparison, P2 Benchmarks “Meets expectations’ - Median (2014, 2016) and

Angoff

method (2019)

25

30

35

40

45

50

25

cwpm

38

cwpm

35

cwpm

45

cwpm

25

cwpm

Median method, score 25

th

and 75

th

percentile

38

cwpm

2014

2016

2019 ANGOFF method

35

cwpmv

Slide34

6/26/2019

USAID SOMA UMENYE

34

 

Meets expectations

Round 1

Round 2

P1

Median

14

12

Min

12

12

Max

20

17

 

P2

Median

23

28

Min

4

22

Max

30

30

 

P3

P3

Median

34

41

Min

25

37

Max

42

44

Slide35

6/26/2019

USAID SOMA UMENYE

35

Modified

Angoff

method – 3 part process

2 days

3 days