Uses of a Statewide Longitudinal Data System to Evaluate and Inform Programs, Policies, - PowerPoint Presentation

Uses of a Statewide Longitudinal Data System to Evaluate and Inform Programs, Policies, and Resource AllocationsPresented February 13, 2013 26th Annual Management Information Systems [MIS] Conference 1 Deborah T. Carran, Jacqueline Nunn, Sara Hooks Johns Hopkins University Stacey N. Dammann York College

Background: Linking Data SetsPart C Early Intervention Services data (Birth – 3)Part B Special Education Services data (4, 5 – 21) General Education Services dataK Children assessed for K Readiness WSS-K, scaled then scores at 3 levels Readiness (Developing, Approaching, Fully Ready)Maryland State Assessment (MSA), scaled then scored at 3 levels (Advance, Proficient, Basic) Math and Reading assessments administered annually in grades 3 through 8 2013 MIS Conference 2

2013 MIS Conference3

Benefits of Using the Maryland IDEA Scorecard Data at the State, District, and School levels that drills down to the student level Data that allows users to identify students in need of targeted interventions in the alert categories (attendance, academics, suspension, and mobility) Data to create an action plan to monitor student progress within targeted interventions Reporting functions to support monitoring of progress towards targets of interventions 4 2013 MIS Conference

How to use these data at the state level?DescriptiveWho do we serve?LongitudinalWhat is the educational placement of children served in EIS by K and Grade 3? Is Fall K WSS is a successful predictor of later standardized test performance? Comparative Is there a difference in standardized achievement performance within educational services (Gen Ed and Sp Ed) for children served in EIS? 2013 MIS Conference 5

Tracking Plan2013 MIS Conference6

MethodPARTICIPANTS42% tracking matchMissing dataOutcomesWSS-K (2006-07)RSAA and MSAA (Spring 2011) ProcedureThree studies presented 2013 MIS Conference 7

Missing DataN = 5,328 participants received EIS 42.1% (n = 2,245) matched 57.9% ( n = 3,083) unmatched (missing data)Is there a significant difference between matched and unmatched participants on characteristics: gender, Part C eligibility category, Part C MA, Race, age of entry, age of exit, and/or months in EIS? 2013 MIS Conference 8

Gender 2013 MIS Conference 9 Results of the Chi-Square statistic assuming equal probabilities indicated no significant difference between Unmatched and Matched participants for gender [ X 2 (1, N = 5,328) = 0.93, p > .05]. Inspection of cell counts and percentages indicated that the distribution for gender was similar for both groups.

Part C Eligibility   25% Delay Atypical Develp / Behav High Probability N (%) N (%) N (%) Unmatched 1,902 (61.9%) 463 (15.1%) 707 (23.0%) Matched 1523 (67.8%) 376 (16.7%) 346 (15.4%) Total 3,425 (64.4%) 839 (15.8%) 1053 (19.8%) 2013 MIS Conference 10 Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Part C Eligibility [ X 2 (2, N = 5,317) = 47.23, p < .001]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were eligible due to a Condition with a High Probability than Matched participants. Conversely, a greater proportion of Matched participants were eligible based on 25% Delay than Unmatched participants.

Part C Eligibility cont. 2013 MIS Conference 11

Part C Medical Assistance 2013 MIS Conference 12 Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Part C Medical Assistance [ X 2 (1, N = 5,278) = 17.49, p = .001]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were eligible for Medical Assistance while receiving EIS.

Race   Am Ind/Alaska Native Asian or Pacific Islander Black/AfAm Hispanic Multiple White Unmatched 2 (0.1%) 121 (3.9%) 971 (31.5%) 237 (7.7%) 112 (3.6) 1640 (53.2%) Matched 5 (.2%) 91 (4.1%) 606 (27%) 201 (9%) 75 (3.3%) 1267 (56.4%) Total 7 (.1%) 212 (4%) 1577 (29.6%) 438 (8.2%) 187 (3.5%) 2907 (54.6%) 2013 MIS Conference 13 Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Race [ X 2 (5, N = 5,328) = 16.76, p < . 05]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were Black or African American while a smaller proportion of participants in the Unmatched group were Hispanic than in the Matched group.

Entry/Exit Age in Months 2013 MIS Conference 14

Summary – Missing DataGender – the two groups are similarEligibility – significantly greater proportion of Unmatched participants in High Probability CategoryMedical Assistance – significantly greater proportion of Unmatched participants received MARace – significantly greater proportion of Black or African American participants in Unmatched group with significantly greater proportion of Hispanic participants in Matched group Age – Matched participants were significantly younger at age of entry than Unmatched participants. Matched participants received EIS longer than Unmatched participants. Age of exit was similar for both groups. 2013 MIS Conference 15

Outcome InstrumentsWork Sampling System-Kindergarten (WSS-K)Assesses 7 Domains, scaled then scored at 3 levels (Proficient, In Process, Needs Development)Personal and Social DevelopmentLanguage and Literacy Mathematical ThinkingScientific ThinkingSocial StudiesThe Arts Physical DevelopmentReading State Accountability Assessment (RSAA ) scaled then scored at 3 levels (Basic, Proficient, Advanced)Math State Accountability Assessment (MSAA) scaled then scored at 3 levels (Basic, Proficient, Advanced) 2013 MIS Conference 16

ProcedureStudent data tracking system links records of students in General Ed, Special Ed (Part B) and EIS (Part C)Identify children in 3 rd Grade 2010-11 school year with birth dates between Sept. 1, 2001 and Aug. 31, 2002At the state level, student identifiers entered in student data tracking system to obtain student service level (Gen Ed or Sp Ed), student outcomes, and demographic info Deidentified data provided to researchers 2013 MIS Conference 17

Studies and RQ for this PresentationImpact of Early Intervention on K ReadinessWho was served by EIS?Does EIS impact later K Readiness Scores? Tracking Children Receiving Early Intervention Services (Part C Services Birth to 3) into Elementary School to Grade 3What is the educational placement at Grade 3 for children who received EIS?What is the SAA performance at Grade 3 for children who received EIS ? 2013 MIS Conference 18

Different Data Set3. K Readiness and Grade 3 SAA PerformanceIs there a relationship between K Readiness and Grade 3 RSAA and MSAA scoring?What subscales of the WSS predict RSAA and MSAA scoring? 2013 MIS Conference 19

Study 1: Impact of Early Intervention Services on K ReadinessDoes level of service provided to children (Birth – 3) enrolled in early intervention services (EIS) programs enhance their later performance on the Kindergarten Work Sampling System (WSS-K)? 2013 MIS Conference 20

K Readiness: Method2,245 children Who were eligible and received EIS services in MD linked with MD MMSR scoresBorn between Sept 1, 2001 and Aug 31, 2002EIS services WSS-KSummary scaled composite score Hierarchical Linear Regression 2013 MIS Conference 21

Participants2013 MIS Conference22

EIS Services & WSS-K Average Scores Service Description Children WSS-K WSS-K N Mean SD Audiology 474 73.2 13.1 Family Counseling/Training 375 71.7 13.2 Occupational Therapy 467 71.0 14.4 Physical Therapy 565 73.0 13.9 Special Instruction 1145 71.8 13.5 Speech/Language Therapy 1553 74.4 12.9 Other Services Than Above 335 73.1 13.6 At least 1 Service 2245 74.5 12.8 2013 MIS Conference 23 Max WSS-K score of 90

VariablesOutcome: WSS-KPredictors:Demographics: FaRMs, Gender, Minority Earliest Age of Child that Part C Services Begin(-) receive services earlier better prepares child to enter KTime in Program (+) indicates longer time in program better prepares child to enter K Total Minutes Services (+) indicates longer time in program better prepares child to enter K 2013 MIS Conference 24

Hierarchical Regression Results VariablesModel 1 Model 2 β β FaRMs -.170** -.171** Gender -.048* -.044* Minority -.120** -.103** Age Svc Began (months) -.069* Time in Program (days) -.066* Total Minutes Services -.138** R 2 .061 .086 R 2 Change .025** 2013 MIS Conference 25

EIS – WSS ConclusionsDemographic Controls: WSS-K was higher for students not economically disadvantaged, higher for girls, and for White studentsAge Svc Began (-): For every month earlier a child starts receiving services, he/she is expected to score .017 SD increase on the WSS-KSupports previous findingsTime in Program (-): inconclusive; possible that children who are in the program for longer times have more severe disabilities Total Minutes Services (-): inconclusive, possible that children with more severe disabilities will have more/longer servicesCorrelation Time in Program & Total Min Svc r = .37, p < .001 2013 MIS Conference 26

Study 2Tracking Children Receiving Early Intervention Services (Birth to 3) into Elementary School by Service Level: How do they Compare with their Peers? 2013 MIS Conference 27

BackgroundPart C Services (IFSP)Part b/B Services (IEP)Little research examining outcomes of children receiving Part C services, mostly Part B services (Cole, Dale, Mills & Jenkins, 1993; Daley & Carlson, 2009; Peterson et al., 2004; Walker et al., 1988) Focused on developmental progress over short term; not longitudinal or growth trajectoriesEnrollment in special education changes for children as they move through elementary schoolLimitations (disability, small samples, covariates) 2013 MIS Conference 28

First Steps: Descriptive AnalysesChildren in 3rd grade (within birth cohort)What is their Education Service (Gen Ed, Sp Ed)?Who were the children receiving EIS? 2013 MIS Conference 29

Participants2013 MIS Conference30

Total Sample, Gen Ed and HI Sp Ed Characteristic N % Total Sample 52,584 100.0 General Education 47,928 91.1 Special Education (Part B) 4,656 8.9 Disability Codes 4, 6, 8, and 9 3,994 86.7 04-Speech or Language Impairments 1,898 40.7 06-Emotional Disturbance 224 4.8 08-Other Health Impairments 721 15.5 09-Specific Learning Disabilities 1,151 24.7 All Other Disability Codes* 665 14.3 2013 MIS Conference 31 *Autism, Deaf, Deaf-Blindness, Developmental Delay, Hearing Impaired, Intellectual Disability, Multiple Disabilities, Orthopedic Impairment, Traumatic Brain Injury, Visual Impairment

Gender N % Males 26,449 50.3 Females 26,135 49.7 New Race Descriptions American Indian/Alaskan 140 0.3 Asian 3,108 5.9 Black/African American 17,044 32.4 Hispanic 6,611 12.6 Multiple Races 2,258 4.3 Native Hawaiian/Pac Islander 33 0.1 White 23,390 44.5 Eligible for Free and Reduced Meals No 29,892 56.8 Yes 22,692 43.2 Limited English Proficiency Identified No 47,081 89.5 Yes 4,056 7.7 Exited 1,447 2.8 2013 MIS Conference 32

Students in Grade 3 Historically Tracked to EIS Characteristic N % Third Grade Students Received EIS Part C 2,482 100.0 General Education Grade Three 1,628 65.6 Special Education Grade Three 854 34.4 Disability Codes 4, 6, 8, and 9 (SL, ED, OHI, SLD) 617 72.2 All Other Disability Codes 237 27.8 Gender Males 1,646 66.3 Females 836 33.7 Limited English Proficiency Identified No 2,353 94.8 Yes 97 3.9 Exited by Grade Three 32 1.3 Free and Reduced Meals in Grade Three     No 1,627 65.6 Yes 855 34.4 Minority Status     Yes, Minority 1,086 43.8 No, Minority 1,396 56.2 2013 MIS Conference 33

Students in Grade 3 Tracked to EIS, EIS Characteristics Part C Eligibility of Grade 3 Students Born Sept. 1, 2001 – Aug. 31, 2002 25% Delay Atypical Development High Probability Condition Student Placement Grade Three 2011 (n = 1,674) (n = 400) (n = 408) N % N % N % General Education Grade Three 1,093 65.3 284 71.0 251 61.5 Special Education Grade Three 581 34.7 116 29.0 157 38.5 2013 MIS Conference 34 65.5%, n = 1,628 enrolled in general education at grade three

Average State Assessment Scores by Educational Service at K and Grade 3 2008 WSS-K 2011 RSAA 2011 MSAA   N M SD M SD M SD General Ed Gr 3 47928 77.9 10.9 430.8 38.2 429.9 41.1 No EIS 46300 77.9 10.9 430.9 38.2 429.9 41.1 Yes EIS 1628 77.2 11.1 427.8 39.1 428.6 41.7 Special Ed Gr 3 3994 67.8 13.5 368.0 120.6 364.6 114.4 No EIS 3377 68.0 13.4 371.5 117.3 367.1 111.2 Yes EIS 617 67.2 13.9 349.2 135.9 350.9 129.8 2013 MIS Conference 35

Hierarchical Regression AnalysesOutcome: RSAA Grade 3, MSAA Grade 3Three Models:Model 1, 2, 3 Demographics: FaRMs, Gender, MinorityModel 2, 3 Part C EISModel 3 EIS x FaRMs EIS x Gender EIS x Minority 2013 MIS Conference 36

Hierarchical Regression Results: WSS-K WSS-K General Ed Special Ed Model 1 Model 2 Model 3 Effect Size Model 1 Model 2 Model 3 Effect Size Variable B (SE) B (SE) B (SE) Partial ŋ 2 B (SE) B (SE) B (SE) Partial ŋ 2 Constant 81.57**(.10) 81.60**(.10) 81.60**(.10) .592 72.95** (.50) 73.20** (.52) 73.65** (.55) .248 FARMS -3.57**(.12) -3.58**(.12) -3.56**(.12) .003 -3.73** (.50) -3.83** (.50) -3.79** (.56) .002 Gender -2.46**(.11) -2.45**(.11) -2.49**(.11) .000 -1.89** (.51) -1.86** (.51) -2.54** (.55) .004 Minority -1.98**(.12) -1.99**(.12) -1.98**(.12) .001 -3.62** (.50) -3.61** (.50) -3.63** (.56) .009 EIS   -.80** (.29) -1.16* (.56) .000   -1.28* (.62) -4.37** (1.37) .003 EIS * FARMS     -.47 (.68) .000     -.35 (1.32) .000 EIS * Gender     1.09 (.60) .000     4.22** (1.37) .001 EIS * Minority     -.40 (.66) .000     .53 (1.31) .000 R 2 . 057 .057 .057   .056 .057 .060   R 2 Change   .0001** .000     .001* .003*   2013 MIS Conference 37 * p ≤ .05, ** p ≤ .01

WSS-K ResultsGeneral EdWSS-K higher for students not FaRMs, for girls, and for White studentsEIS students scored lower (.7 M diff)No interaction effects Special Ed WSS-K higher for students not FaRMs, for girls, and for White studentsEIS students scored lower (.8 M diff)Sig interaction EIS x Gender F > M NO EIS (2.9 M diff) M > F EIS (1.7 M diff) 2013 MIS Conference 38

Hierarchical Regression Results: RSAA RSAA Grade 3 General Ed Special Ed Model 1 Model 2 Model 3 Effect Size Model 1 Model 2 Model 3 Effect Size Variable B (SE) B (SE) B (SE) Partial ŋ 2 B (SE) B (SE) B (SE) Partial ŋ 2 Constant 451.50**(.29) 451.70**(.30) 451.76**(.30) .592 399.95**(4.10) 404.91**(4.19) 406.34**(4.44) .248 FARMS -22.31**(.35) -22.34**(.35) -22.31**(.36) .003 -19.31**(3.99) -21.19**(3.99) -20.24**(4.36) .002 Gender -10.15**(.32) -10.04**(.32) -10.14**(.32) .000 5.16 (4.06) 5.79 (4.05) 4.08 (4.39) .004 Minority -11.26**(.35) -11.35**(.35) -11.39**(.36) .001 -46.56** (4.00) -46.75** (3.99) -48.16** (4.37) .009 EIS   -5.49**(.89) -7.64**(1.71) .000   -27.86**(5.16) -39.27**(11.48) .003 EIS * FARMS     -1.00 (2.08)     -5.64 (10.86) EIS * Gender     3.08 (1.84)     12.72 (11.48) EIS * Minority     1.30 (1.98)     9.38 (10.75) R 2 .153 .154 .154   .054 .061 .060   R 2 Change   .001** .000     .007** .000   2013 MIS Conference 39 * p ≤ .05, ** p ≤ .01

RSAA ResultsGeneral EdRSAA higher for students not FaRMs, for girls, and for White studentsEIS students scored lower (3.1 M diff)No interaction effects Special Ed RSAA higher for students not FaRMs and for White studentsEIS students scored lower (22.3 M diff)No interaction effects 2013 MIS Conference 40

Hierarchical Regression Results: MSAA MSAA Grade 3 General Ed Special Ed Model 1 Model 2 Model 3 Effect Size Model 1 Model 2 Model 3 Effect Size Variable B (SE) B (SE) B (SE) Partial ŋ 2 B (SE) B (SE) B (SE) Partial ŋ 2 Constant 448.58**(.32) 448.77**(.32) 448.81**(.33) .550 390.64**(3.84) 394.63**(3.92) 396.59**(4.16) .248 FARMS -22.64**(.38) -22.67**(.38) -22.57**(.39) .003 -18.99**(3.74) -20.50**(3.74) -20.10**(4.08) .002 Gender -2.24**(.35) -2.13**(.35) -2.28**(.35) .000 18.68**(3.81) 19.19**(3.79) 15.52**(4.11) .004 Minority -14.36**(.38) -14.45**(.38) -14.47**(.39) .001 -53.53**(3.75) -53.68**(3.74) -53.01**(4.09) .009 EIS   -5.35**(.96) -7.68**(1.85) .000   -22.53**(4.84) -37.81**(10.76) .003 EIS * FARMS     -3.46 (2.24)     -3.77 (10.18) EIS * Gender     4.85* (1.99) .000     25.01* (10.76) .001 EIS * Minority     1.02 (2.15)     -2.25 (10.08) R 2 .142 .142 .143   .078 .083 .084   R 2 Change   .001** .000*     .005** .000   2013 MIS Conference 41 * p ≤ .05, ** p ≤ .01

MSAA ResultsGeneral EdWSS-K higher for students not FaRMs, for girls, and for White studentsEIS students scored lower (1.3 M diff)Sig interaction EIS x GenderF > M No EIS (1.9 M diff)M > F EIS (4.2 M diff) Special Ed WSS-K higher for students not FaRMs, for girls, and for White studentsEIS students scored lower (16.2 M diff) Sig interaction EIS x Gender M > F No EIS (10.4 M diff) M > F EIS (42.9 M diff) 2013 MIS Conference42

General Ed EIS SAA ConclusionsDemographic Controls: WSS-K, RSAA, & MSAA was higher for students not economically disadvantaged, higher for girls, and for White students EIS: WSS-K, RSAA, & MSAA when FaRMs, Gender, and Minority are controlled, students who received EIS scored lower than their Gen Ed peers. EIS Interactions: WSS-K & RSAA no significant interactions; MSAA EIS x Gender (F EIS) 2013 MIS Conference 43

Special Ed EIS SAA ConclusionsDemographic Controls: WSS-K, RSAA, & MSAA was higher for students not economically disadvantaged and for White students; females scored higher on WSS-K & MSAAEIS: WSS-K, RSAA, & MSAA when FaRMs, Gender, and Minority are controlled, students who received EIS scored lower than their Sp Ed peers EIS Interactions: WSS-K & MSAA significant EIS x Gender (F EIS lower in K and more at Grade 3) 2013 MIS Conference 44

Youth in EIS have lower average scores than their peers in Gen Ed and Sp Ed2013 MIS Conference 45

Female EIS in Sp Ed (HI) Services have lower average scores than their peers2013 MIS Conference 46

Study 3: K Readiness and Grade 3 MSA PerformanceIs performance on the WSS-K predictive of Grade 3 high stakes testing (Reading MSA and Math MSA)? 2013 MIS Conference 47

BackgroundMaryland Model for School Readiness (MMSR) Pearson Work Sampling System AssessmentAssesses 7 Domains, scaled then scored at 3 levels (Proficient, In Process, Needs Development)Personal and Social Development Language and LiteracyMathematical ThinkingScientific ThinkingSocial Studies The ArtsPhysical Development 2013 MIS Conference 48

MethodParticipants/ProcedureGeneral Ed Fall K Work Sampling System student scores (2003, 2004, 2005) N = 152,105 Matched to Grade 3 Spring MSA Math, MSA Read student scores (2006, 2007, 2008) N = 100,958 Match Rate 66% for Reading & MathInstrumentsWSS is a 30 item instrument with scaled items coded 1 - 3 MSA is a high stakes test coded 1 - 3 2013 MIS Conference 49

Kindergarten MMSR Composite Score Distribution by Grade 3 MSA Performance Level MSA N % WSS-K M WSS-K SD Math Basic 7,876 17.71% 63.98 5.71 Proficient 55,777 55.25% 72.21 5.52 Advanced 27,304 27.05% 78.38 5.25 Total 100,957 100.00% 72.42 3.51 Reading         Basic 17,695 17.52% 64.78 5.67 Proficient 64,907 64.28% 72.63 5.62 Advanced 18,376 18.20% 79.02 4.91 Total 100,978 100.00% 72.42 3.85 2013 MIS Conference 50

Matching Students K to Grade 3 Kindergarten School Year N WSS-K Scores Grade 3 MSAA Grade 3 MSAA Grade 3 RSAA Grade 3 RSAA N Students Match Rate (%) N Students Match Rate (%) 2002-2003 54,452 36,032 66.17% 36,026 66.16% 2003-2004 50,024 33,986 67.94% 34,008 67.98% 2004-2005 47,629 30,940 64.96% 30,944 64.97% Total 152,105 100,958 66.37% 100,978 66.39% 2013 MIS Conference 51

MethodCausal Comparative Descriptive DesignCART analysisBinary decision trees representing a series of rules that lead to membership in a class or valueVariables in the dataset are analyzed and split on a value to best predict outcome By following the splits in the decision tree to the targeted outcome, observers can discern the patterns and combinations of variables that best predict either the presence or absence of the desired outcome 2013 MIS Conference 52

Grade 3 Math Decision TreeMIS DC 2013 Data Conference53 All Participants

Grade 3 Math Decision TreeMIS DC 2013 Data Conference54 IIA3: Begin Phonemic Aware

Grade 3 Math Decision TreeMIS DC 2013 Data Conference55 IIA3: Begin Phonemic Aware IIA3: Begin Phonemic Aware

Grade 3 Math Decision TreeMIS DC 2013 Data Conference56 IIA3: Begin Phonemic Aware IIIB1: Number & Quantity IIA3: Begin Phonemic Aware

Grade 3 Math Decision TreeMIS DC 2013 Data Conference57 IIIC2:Recog/Dup/Extend Patterns IIA3:Begin Phonemic Aware IIA3:Begin Phonemic Aware IIIB1: Number & Quantity

Grade 3 Math Decision TreeMIS DC 2013 Data Conference58 IIA3: Begin Phonemic Aware IIIB1: Number & Quantity IIA3: Begin Phonemic Aware IIIC2: Recog /Dup/Extend Pattern IIIA1: Strategy Use for Math

Grade 3 Math Decision TreeMIS DC 2013 Data Conference 59 IIA3: Phonemic Aware IIA3: Phonemic Aware IIIB1: Number & Quantity IIIC2: Patterns IIIA1: Strategy Use Math IIA1: Listening

Grade 3 Math Decision TreeMIS DC 2013 Data Conference60 IIA3: Phonemic Aware IIIC2: Patterns

Grade 3 Math Decision TreeMIS DC 2013 Data Conference61 IIIC2: Patterns IBI: Rules & Routines IIA3: Phonemic Aware

Grade 3 Math Decision TreeMIS DC 2013 Data Conference62 IIIA1: Strategy Use for Math IBI: Rules & Routines IIIC2: Patterns IIA3: Phonemic Aware

Grade 3 Math Decision TreeMIS DC 2013 Data Conference 63 IIIA1: Strategy for Math IBI: Rules & Routines IIIC2: Patterns IIA3: Phonemic Aware IIA3: Phonemic Aware IIA3: Phonemic Aware IIA1: Listening IIIA1: Strategy for Math IIIC2: Patterns

Math Decision Tree Nodes Descriptions Terminal Node Segment: Predicted Outcome # Students Average Math Score Demonstrates beginning phonemic awareness (IIA3) Shows understanding of number and quantity (IIIB1) Recognizes, duplicates, and extends patterns (IIIC2) Begins to use and explain strategies to solve mathematical problems (IIIA1) Gains meaning by listening (IIA1) Follows classroom rules and routines (IB1) 1 3 2,838 1.495 3 3 {1,2,3} {1,2,3} {1,2,3} {1,2,3} 2 3 4,833 1.709 1 { 2,3} {1,2,3} {1,2,3} {1,2,3} {1,2,3} 3 3 2,173 1.723 { 2,1} {1,2,3} {3,2 } 1 {1,2,3} {1,2,3} 4 2 14,620 1.927 { 2,1} {1,2,3} {3,2 } {2,3} {1,2,3} {1,2,3} 5 2 4,647 2.027 { 2,1} {1,2,3} 1 {1,2,3} {3,2 } {1,2,3} 6 2 4,857 2.188 { 2,1} {1,2,3} 1 {1,2,3} 1 {1,2,3} 7 2 4,091 2.143 1 {1,2,3} { 1,3} {1,2,3} {1,2,3} {1,2,3} 8 1 3,908 2.246 1 {1,2,3} 1 {1,2,3} {1,2,3} {1,2} 9 1 4,164 2.324 1 {1,2,3} 1 {1,2,3} {1,2 } 1 10 1 10,019 2.453 1 {1,2,3} 1 {1,2,3} 1 1 2013 MIS Conference 64

Ranked (Highest to Lowest) Significant WSS-K Indicators used in the Predictive Math Model SAA Math Demonstrates beginning phonemic awareness (IIA3) Shows understanding of number and quantity (IIIB1) Recognizes, duplicates, and extends patterns (IIIC2) Begins to use and explain strategies to solve mathematical problems (IIIA1) Gains meaning by listening (IIA1) Follows classroom rules and routines (IB1) 2013 MIS Conference 65

Grade 3 Math Score Distribution: Comparing Training & In Time Validation Predicted Outcome Training Average MSA Score In Time Average MSA Score Actual Training Outcome: % Students Per Segment Actual In Time Validation Outcome: % Students Per Segment 1 (Basic) 2 (Proficient) 3 (Advanced) 1 ( Basic) 2 (Proficient) 3 (Advanced) Advanced 435 434 6.16% 49.80% 44.04% 6.99% 48.34% 44.67% Proficient 411 411 19.08% 59.88% 21.04% 19.14% 59.48% 21.38% Basic 385 384 42.05% 50.82% 7.13% 43.63% 49.50% 6.88% Total 414 414 18.95% 55.04% 26.01% 19.46% 54.17% 26.37% 2013 MIS Conference 66

ConclusionFall WSS-K is a moderately successful predictor of later standardized Math test performance2013 MIS Conference 67

Grade 3 Reading Decision TreeMIS DC 2013 Data Conference68

Grade 3 Reading Decision TreeMIS DC 2013 Data Conference69 IIA3: Phonemic Aware

Grade 3 Reading Decision TreeMIS DC 2013 Data Conference70 IIA3: Phonemic Aware IIA3: Phonemic Aware

Grade 3 Reading Decision TreeMIS DC 2013 Data Conference71 IIA3: Phonemic Aware IIA3: Phonemic Aware IIIA1: Strategy f/Math

Grade 3 Reading Decision TreeMIS DC 2013 Data Conference72 IIA3: Phonemic Aware IIA3: Phonemic Aware IIIA1: Strategy f/Math IIA1: Listening

Grade 3 Reading Decision TreeMIS DC 2013 Data Conference73 IIA3: Phonemic Aware IIA3: Phonemic Aware IIIA1: Strategy f/Math IIA1: Listening IIIA1: Strategy f/Math

Grade 3 Reading Decision TreeMIS DC 2013 Data Conference74 IIA3: Phonemic Aware IIA3: Phonemic Aware IIIA1: Strategy f/Math IIA1: Listening IIIA1: Strategy f/Math IIIB1: Number & Quantity

Grade 3 Reading Decision TreeMIS DC 2013 Data Conference75 IIA3: Phonemic Aware IB1: Rules & Routines

Grade 3 Reading Decision TreeMIS DC 2013 Data Conference76 IIA3: Phonemic Aware IB1: Rules & Routines IIC2: Print Concepts

Grade 3 Reading Decision TreeMIS DC 2013 Data Conference77 IIA3: Phonemic Aware IB1: Rules & Routines IIIC2: Patterns IIB1: Effective Comm

Grade 3 Reading Decision TreeMIS DC 2013 Data Conference78 IIA3: Phonemic Aware IB1: Rules & Routines IIIC2: Patterns IIB1: Effective Comm IIA3: Phonemic Aware IIIA1: Strategy f/Math IIA1: Listening IIIA1: Strategy f/ Math IIIB1: Number & Quantity

Read Decision Tree Nodes Descriptions Terminal Node Segment: Predicted Outcome # Students Average Reading Score Demonstrates beginning phonemic awareness (IIA3) Begins to use and explain strategies to solve mathematic problems (IIIA1) Gains meaning by listening (IIA1) Shows understanding of number and quantity (IIIB1) Follows classroom rules and routines (IB1) Shows some understanding of concepts about print (IIC2) Speaks clearly and conveys ideas effectively ( IIB1) 1 3 4,164 1.566 3 3 {1,2,3} {1,2,3} {1,2,3} {1,2,3} {1,2,3} 2 3 3,466 1.708 3 { 2,1} {1,2,3} {1,2,3} {1,2,3} {1,2,3} {1,2,3} 3 3 2,176 1.732 2 3 {1,2} {1,2,3} {1,2,3} {1,2,3} {1,2,3} 4 2 15,345 1.875 2 {2,3} {1,2} {1,2,3} {1,2,3} {1,2,3} {1,2,3} 5 2 4,419 1.949 2 {1,2,3} 1 { 1,3} {1,2,3} {1,2,3} {1,2,3} 6 2 4,274 2.084 2 {1,2,3} 1 1 {1,2,3} {1,2,3} {1,2,3} 7 2 5,384 2.083 1 {1,2,3} {1,2,3} {1,2,3} {1,2} {1,2,3} {1,2,3} 8 2 2,609 2.123 1 {1,2,3} {1,2,3} {1,2,3} 1 {1,2} {1,2,3} 9 2 1,601 2.156 1 {1,2,3} {1,2,3} {1,2,3} 1 1 {3,2 } 10 1 12,545 2.307 1 {1,2,3} {1,2,3} {1,2,3} 1 1 1 2013 MIS Conference 79

Ranked (Highest to Lowest) Significant WSS-K Indicators used in the Predictive Reading Model SAA Reading Demonstrates beginning phonemic awareness (IIA3 )* Gains meaning by listening (IIA1 )* Begins to use and explain strategies to solve mathematical problems (IIIA1 )* Shows understanding of number and quantity (IIIB1 )* Follows classroom rules and routines ( IB1)* Recognizes, duplicates, and extends patterns (IIIC2)* Speaks clearly and conveys ideas effectively ( IIB1 ) 2013 MIS Conference 80 *also significant predictors in MSA math model

Grade 3 Reading Score Distribution: Comparing Training & In Time Validation Predicted Outcome Training Average MSA Score In Time Average MSA Score Actual Training Outcome: % Students Per Segment Actual In Time Validation Outcome: % Students Per Segment 1 (Basic) 2 (Proficient) 3 (Advanced) 1 ( Basic) 2 (Proficient) 3 (Advanced) Advanced 444 444 5.35% 58.62% 36.03% 5.50% 58.44% 36.06% Proficient 421 421 17.94% 66.41% 15.65% 17.87% 66.32% 15.81% Basic 398 398 39.03% 56.66% 4.31% 38.27% 57.89% 3.84% Total 422 422 18.81% 62.96% 18.23% 18.76% 63.08% 18.16% 2013 MIS Conference 81

Additional Analyses/ConclusionsSexFor math and reading, similar performanceEthnicityFor math and reading, Whites and Asian/Pac Islanders realized predicted performance at higher levels; African-American, Hispanic, & Native American realized predicted performance at lower levelsFaRMs For math and reading, student receiving FaRMs realized predicted performance at lower levels 2013 MIS Conference 82

Conclusions Fall K WSS is a modestly successful predictor of later standardized test performanceMath SAA performance was better predicted than Reading SAA Overidentification of middle group (Proficient)Findings support the importance of early literacy skillsFindings support the persistent ‘gap’ in student achievement, evident from K 2013 MIS Conference 83

Summary QuestionsAre early intervention services effective in preparing children to enter kindergarten? Value of EIS?Continue funding of Birth to five model?Is level of K preparedness predictive of grade 3 high stakes testing performance? Value of high quality preK programs (QRIS)? Fund additional preK programs? Are early intervention services effective in preparing children to be successful in school?Value of EIS? Further tracking studies looking at child outcomes. 2013 MIS Conference 84