Evaluation Results 2016 Katie Tosh Director of Measurement amp Outreach Boston After School amp Beyond Contents Enrollment amp Attendance Findings ACT Framework Update Measurement Findings ID: 626069
Download Presentation The PPT/PDF document "Boston Summer Learning Community" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Boston Summer Learning Community
Evaluation Results, 2016
Katie Tosh, Director of Measurement & Outreach
Boston After School & BeyondSlide2
Contents
Enrollment & Attendance Findings
ACT Framework
Update
Measurement Findings
Program Practices
Youth Social-emotional Skill Growth
Trends over Time
Summary & Recommendations
Slide3
Focus on Best Practices and Quality ImprovementSlide4
Enrollment and AttendanceSlide5Slide6
The RAND study on summer learning found significant advantages in math, ELA, and social-emotional skills for high attenders (≥ 20 days). Two factors are at play here:
duration of program
and
retention of students in the program
. The above graph is for all BSLC students, regardless of program duration. However, among students who were scheduled to attend at least 20 days, 70% of those students attended 20 or more days of summer programming.Slide7
Avg
: 84.8%
Range
: 58% to 100%Slide8
BSLC By Date Average Rate of Attendance, 2015 and 2016
2015 dates
2016 dates
2015 by date average rate of attendance
2016 by date average rate of attendance
July 3, 2015
July 4, 2016Slide9
ACT Framework UpdateSlide10
Potential Revision to the ACT Framework (draft)
ACT Skills
Framework
ACHIEVING
Critical
Thinking
Creativity
Perseverance
CONNECTING
Social Awareness & Relationships
Communication
Teamwork
THRIVING
Growth Mindset
Efficacy
Self-Regulation
Update to reflect recent research and youth development work nationally
Align terminology of ACT Framework and measurement tools
Established shared definitions of skillsSlide11
Outcomes Associated with ACT Skills:
Achievement (academics, life)
and
Behavioral (personal, social)
ACT Skills
Framework
Associated Outcomes from Research Review
ACHIEVING
Critical
Thinking
College success and workforce readiness; Employee selection by management
Creativity
Improved
academic achievement
Perseverance
High education attainment; Higher undergraduate GPAs; Fewer
career changes; Work habits and task persistence; Math achievement
CONNECTING
Social Awareness & Relationships
Improved academic test scores; Better job performance; Improved grades;
Healthier relationships;
Improved social skills; Improved self-efficacy, problem-solving
Communication
Healthier social relationships;
Improved self-efficacy
Teamwork
Improved
academic achievement;
Improved self-efficacy
THRIVING
Growth Mindset
Improved performance
and
academic
achievement;
Lower stress levels
Efficacy
Improved academic achievement;
Academic persistence; College GPA, persistence; Improved work performance;
Improved perseverance
Self-Regulation
Positive academic outcomes (grade promotion, test scores,
course grades); College GPA and retention;
Positive social outcomes (better impulse control, more stable relationships, increased empathy and perspective taking, more constructive response to anger)Slide12
Measurement ResultsSlide13
What Questions Are We Answering?
Program Practices
To what extent did BSLC Programs deliver practices that build ACT skills “most of the time”?
Youth Social-Emotional Skill Growth
What percent of summer program participants demonstrated skills “usually/often” or “always” at the start and end of the summer program?
What percent of summer program participants reported improvements in their skills over the summer?
=
Adult rating (observer if program practice data; teacher/staff if youth skill data)
=
Youth rating (of either program practice or self-report on skills)
Presentation LegendSlide14
Achieve:
Program Practices
Almost Always
Most of the Time
Sometimes
Never
=
Observer Rating, BSLC 2016 (n=119 programs)
=
Youth Rating, BSLC 2016 (n=94 programs; *pilot scale, n=15 programs)
On average, observers and youth rate
P
rograms
as
Meeting
the
Benchmark
in Achieve
Practices
% of programs
meeting the benchmark in practices that build
:
75%
Perseverance
62%
Critical Thinking
(observer perspective)Slide15
Achieve:
Youth Skill Growth
=
Pre Staff/Teacher Rating
of Youth Skills (n=1478 students, 41 programs)
=
Post
Staff/Teacher Rating of Youth Skills (n=1478 students, 41 programs)
% of
youth
reporting improvements at the end of the program in:
82%
Perseverance
88%
Critical Thinking
(n= 1279 students,
25 programs)Slide16
Connect:
Program Practices
Almost Always
Most of the Time
Sometimes
Never
=
Observer Rating, BSLC 2016 (n=119 programs)
= Youth Rating, BSLC 2016 (n=94 programs; *pilot scale, n=15 programs)
On average, observers and youth rate
P
rograms
as
Meeting
the
Benchmark
in
Connect Practices
% of programs
meeting the benchmark in practices that build:
91%
Relationships
with Adults
87%
Relationships
with Peers
66%
Communication
77%
Teamwork
(observer
perspective)Slide17
Connect:
Youth Skill Growth
=
Pre Staff/Teacher Rating
of Youth Skills
(n=1478 students, 41 programs)
=
Post
Staff/Teacher Rating of Youth Skills (n=1478 students, 41 programs)
% of
youth
reporting improvements at the end of the program in:
83%
R
elationships with Peers
78%
Relationships with
Adults
94%
Empathy
(n= 1279 students,
25 programs
)Slide18
Thrive:
Program Practices
Almost Always
Most of the Time
Sometimes
Never
=
Observer Rating, BSLC 2016 (n=119 programs)
= Youth Rating, BSLC 2016 (n=94 programs; *pilot scale, n=15 programs)
On average,
youth
rate
P
rograms
lower than observers in Thrive Practices
% of programs
meeting the benchmark in practices that build:
91%
Self-Regulation
(observer
perspective)Slide19
Thrive:
Youth Skill Growth
=
Pre Staff/Teacher Rating
of Youth Skills
(n=1478 students, 41 programs)
=
Post
Staff/Teacher Rating of Youth Skills (n=1478 students, 41 programs)
% of
youth
reporting improvements at the end of the program in:
54%
Self-Regulation
(n= 1279 students,
25 programs
)Slide20
Cross-cutting Practices help build all ACT Skills
Almost Always
Most of the Time
Sometimes
Never
=
Observer Rating, BSLC 2016
(
n=119 programs)
=
Youth Rating, BSLC 2016
(n=94 programs; *pilot scale, n=15
programs)Slide21
Program Organization
and Structure
Supportive Environment
Engagement in
Activities and Learning
Program Practice, Observer Perspective: BSLC 2013-2016
Almost Always
Most of the Time
Sometimes
Never
= 2013, 2014, 2015
= 2016
On average, the BSLC met or exceeded the benchmark in all areas of program quality, with scores very similar to the previous year.Slide22
Supportive Environment
Engagement in
Activities and Learning
Program Practice, Youth Perspective: BSLC 2014-2016
= 2014, 2015
= 2016
Youth rated programs very similarly on aspects of program quality in Summer 2016 as compared to the previous two summers. Program strengths were maintained from one year to the next and room for improvement still exists in identifying and incorporating best practices in areas of challenge. Slide23
How did returning programs fare?
There was no difference for Returning sites
in
youth or observer ratings for
2016 versus 2015 across 18 out of 23 areas
of program
quality
Observers rated Returning sites lower* in 2016 than 2015 in:
Organization
of Activity
Space AdequacySocial-Emotional Environment
Staff Build Relationships and Support Individual YouthYouth Relationships with PeersHowever, Returning sites
exceeded the benchmark in all 23 domains
of program quality
(*paired samples t-test,
p<0.05
)Slide24
Summary & RecommendationsSlide25
Summary: Reach and Dosage
Reach
: 127 programs serving 10,084 youth
Average program duration
: 6.5 weeks (range 1-10
wks
)
Average Rate of Attendance
: 84.8% (range 58%-100%)Slide26
Summary: Program Practice
On average, observers and youth rated programs
at or above the benchmark in 9 out of 10 areas of practice
ACT practices and cross-cutting practices
66% - 91% of programs hit the benchmark
of
quality practice,
depending
on
which skill the practices are meant to build
Room for improvement
remains in:
Achieve
and
Thrive
practices
(
critical thinking, perseverance,
self-regulation
)
Opportunities for Leadership & Choice
Over time, the growing BSLC has
maintained strong positive trends
in above-benchmark performanceSlide27
Summary: Youth Skill Growth
On average, youth participants achieved
statistically significant growth in all ACT skills
, as rated by program staff and youth self-report
Areas of
highest skill performance or growth
:
Student self-report:
empathy
Program staff rating:
peer relationships, adult relationships
Areas of the
lowest skill performance or growth
:
Students self-report:
self-regulation
Program staff rating:
perseverance, critical thinking,
communication
, and
self-regulationSlide28
Recommendations & Next Steps
Review
PRISM Report
with staff
What are your program’s goals for youth?
How does
your program practice support those goals
? Staff hiring and training?
What does the data reveal about your strengths? Challenges?
How does your staff’s experience align to or differ from your data?
What are 1-2 areas to focus on maintaining? Improving?Attend upcoming
professional development sessionsRead up on
best practices
Summer Insight Center
Measurement Tools
Potential areas of focus
for BSLC partners:
Strategies to maintain
high program attendance
or improve low attendance
Maintain and/or strengthen
program practices
that relate to your program’s content and goalsIncorporate best practices for academic instructional quality if your program has an academic focus
Create opportunities to give youth leadership, autonomy, and choiceSlide29
Workshop 1
- National Study RCT Findings: Implications for Program Leaders (first floor)
2016 SUMMER LEARNING SEMINAR
From Research to Practice: Maximizing the Potential of Summer
Learning
Workshop 2
- Unpacking PRISM Data: Quality of Program Practice (downstairs)
Workshop 3
- Supporting Social-Emotional Development (downstairs)