Program Quality Reviews (PQR) - PowerPoint Presentation

Slide1

Program Quality Reviews (PQR)FY2019 – FY2022Community Adult Learning Centers and Correctional Institutions

Adult and Community Learning Services (ACLS)

Slide2

What Adult Education Program Directors Can Expect

Before, During and After a Program Quality Review

Slide3

CONTENTS

1

Overview of Program Quality Reviews

2

How to Prepare for a Program Quality Review

3

What to Expect During a Program Quality Review

What to Expect

A

fter a Program Quality Review

4

Slide4

01

Overview of Program Quality

Reviews

Slide5

Purpose

5

Slide6

PQR Cohorts

6

All programs fully reviewed during a four-year funding cycle.

Cohorts organized by performance and local areas:

Year 1

(

FY19)-Cohort 1

Year 2 (FY20)-Cohort 2

Year 3 (FY21)-Cohort 3

PQR Master Schedule released in the fall of each year.

Slide7

Program Quality Review ScheduleExamples

7

Slide8

PQR Team

8

ACLS Curriculum Specialist

Program Quality Review Consultant

Adult Education Director (from another local workforce area)

ACLS Program Specialist (if not a member of the Program Quality Team)

ACLS Program Quality Team Member

ACLS Review Team Lead (a Program Quality Team member)

Local Workforce Development Board Representative

Slide9

PQR Team-cont.

Depending on the size of the program, a Review Team may be comprised of anywhere from two to seven members.Each Review Team will have two members of the Program Quality (PQ)TeamThe Local Workforce Development Board representative is always invited, but depending on schedules may or may not be part of the team.

9

Slide10

Indicators of Program Quality

The following Indicators of Program Quality (IPQ) will be the focus:IPQ 3-Career Pathways CollaborationIPQ 4-Curriculum and InstructionIPQ 7-Organizational CapacityIPQ 8-Program LeadershipIndicator 5 (Student Performance) data is reviewed by the team before the visit and may be included in interviews depending on performance-related concerns. Indicators will be rated using IPQ-specific rubrics

10

Slide11

Ratings

Ratings will:Be assigned at the standard and indicator level;Be based on solid evidence;Be used as corroborative evidence in future funding decisions;Capture the program’s performance at the time of the visit;Inform the ACLS policy infraction protocol (PIP);Inform the site visit TA process;Inform the program’s continuous improvement process.

11

Slide12

Ratings-cont.

Limited evidence: The program demonstrates little to no evidence related to the Indicator and/or significant concerns are noted.Developing: The program demonstrates inconsistent evidence related to the Indicator and/or moderate concerns are noted.Proficient: The program demonstrates consistent evidence related to most standards of the Indicator and/or minor concerns are noted.Exemplary: The program demonstrates consistent evidence related to all standards of the Indicator and is a potential exemplar in this area.

12

Slide13

13

Data Collection

Classroom Observations

Classroom observations do not exceed 30 minutes each and are conducted in such a way as to not disrupt classroom activities. The Team will visit several classes and use a tool and rubric to gather evidence on activities and practices related to four essential elements of high-quality instruction: scaffolding, differentiated instruction, checking for understanding, and student engagement.

Document Review

Interview Questions

During the visit, the Team will interview program leadership, a sampling of teachers, advisors, and students. No written response to the interview questions is expected.

For some of the Indicators, ACLS requires programs to have specific documents prepared for the day of the visit, prior to the team’s arrival.

Slide14

Interviews

Each interview must be held in a secure, quiet location.More than one interview may be scheduled at the same time during the visit.Each participant must be present for the entirety of the interview and is given complete freedom to speak.Attendees should not attend multiple group interviews.In general, supervisors and supervisees should not be in the same group interview.

14

Slide15

15

Program Leadership

Interviews-cont.

Teachers

Students

Advisors

1. 5 hourMust include staff who oversee the AE program directly and are responsible for its daily management (program director, assistant director, lead teacher). Must include program’s director’s supervisor.

1 hourMust represent advisors who work with students at all levels.

.5 hourShould include maximum 10 studentsMust represent all levels and content areas serviced by the programProgram staff must not be in attendance.

1 hour

Must represent all levels and content areas services by the program

Depending on program size, the group may vary in size. Final number to be decided during preparation phone call

.

Slide16

Interview Questions

Interview questions will relate to the 4 Indicators reviewed during the visit.The team may ask questions related to Indicator 5 (Student Performance) depending on concerns related to MSG performance.The team will ask questions related to NRS follow-up.

16

Slide17

Document Review

Indicator of Program QualityDocuments RequestedPage LimitWhen IPQ 4-Curriculum and InstructionLesson plansCorresponding scope and sequences and curriculum units for the observed classes.Lesson plan-2 pagesCorresponding scope and sequences and curriculum units-3 pages eachLesson plan-the morning of the visitScope and Sequences and curriculum units-2 weeks before the visitIPQ 7-Organizational Support Resumes and job descriptions for the program’s leadershipResume and job description program leaders and supervisor- 2 pages each2 weeks before the visitIPQ 8-Educational LeadershipThree copies of actual feedback director provided to teachers including but not limited to supervision, classroom observations, and annual evaluation reviews. N/A2 weeks before the visit

17

Slide18

Structure of the Day – Sample Schedule #1

18

ActivityActivity LengthScheduleStaff InvolvedIndicators of Program Quality FocusTeam arrival and set-up15 min8:30-8:45PQR TeamN/AFacility Tour15 min8:45-9:00PQR Team Members & Program Director7Interview with program leadership90 min9:00-10:30PQR Team Member (s) & Program Leadership3, 4, 7, 8Classroom Observations60 min10:30-11:30PQR Team Members & Teaching Staff4Student Focus Group Interview30 min11:30-12:00PQR Team Members & Students3, 4, 7 ,8Teacher Focus Group Interview60 min12:00-1:00PQR Team Members & Teaching Staff3, 4, 7 ,8Lunch & Team Check-in30 min1:00-1:30PQR Team3, 4, 7 ,8Advisor Focus Group Interview60 min1:30-2:30PQR Team Members & Advisors3, 7 , 8Team Work Time (aggregate classroom observations data, sort evidence, build preliminary finding, prepare report-out)105 min 2:30-4:15PQR Team3, 4, 7 ,8Report Out15 min4:15-4:30PQR Team and Program Leadership Staff3, 4, 7 ,8

Sample Schedule for a smaller program, 2-3 member team. (*

Note: Start and

end times of activities are subject to change.)

Slide19

Structure of the Day – Sample Schedule #2

19

ScheduleTeam members 1, 2, 3Team members 4 , 5, 6, 78:30-8:45Team arrival and set-up8:45-9:00Facility Tour9:00-10:30Interview with program leadership10:30-11:30Classroom ObservationsAdvisor Focus Group Interview11:30-12:00Student Focus Group InterviewClassroom observations12:00-12:30Lunch and Team check-in12:30-1:30Teacher Focus Group Interview1:30-4:15Team Work Time (aggregate classroom observations data, sort evidence, build preliminary finding, prepare report-out)4:15-4:30Report-out

Sample Schedule for a larger, multi-site program, 6-7 member team

(*

Note: Start and end times of activities are subject to change.)

Slide20

Logistics

The program director should inform the team of:Parking logisticsFood allergy restrictions in the buildingMeeting place in the morningDebrief roomInterview roomTeam members cannot accept or solicit any meals.Program director needs to be available throughout the day for any clarifying questions.

20

Slide21

End of Day Report-Out

Program leadership and their supervisor are required to attend.The team will only share a set of preliminary findings, most likely on one of the four Indicators only.The team will not provide ratings or extensive feedback at this time.

21

Slide22

02

How to Prepare for a Program Quality

Review

Slide23

Before the Program Quality Review

23

Slide24

Before the Program Quality Review

24

Slide25

Useful Documents

Program Leadership and staff participating in the PQR should review the following documents in preparation for the visit:Program Quality Review ProtocolIndicators of Program QualityACLS Performance Standards

25

Slide26

03

What to Expect During a Program Quality

Review

Slide27

During the Program Quality Review

27

Slide28

05

What to Expect After a Program Quality

Review

Slide29

Written Report

29

Slide30

THANK YOU

781.338.3850

http://www.doe.mass.edu/acls

/

75 Pleasant Street,

Malden, MA

02148

Adult and Community Learning Services (ACLS)