Qualitative Program Evaluation Research: Employing the Comp - PowerPoint Presentation

Slide1

Qualitative Program Evaluation Research: Employing the Complementary Analysis Research Matrix Application (CARMA)

LeAnn Grogan Putney, PhDEducational Psychology & Higher Education Professor and Department ChairUniversity of Nevada, Las VegasJoan Wink, PhDProfessor EmeritaCalifornia State University, Stanislaus

joan@joanwink.com

EQRC – Las Vegas, NV

February 11, 2014

putneyl@unlv.nevada.eduSlide2

YOU DO KNOW THESE HEAPS OF DATA NEED MORE ANALYSIS, RIGHT?

…I TOLD YOU TO USE CARMA TO GET OUT OF THE HEAPS AND OVER TO CONCEPTS!Slide3

Vygotsky’s Conceptual Development – AKA, THE HEAPS3

Heaps

Complexes

Concepts

Generalized understandings

Group similar items

Connections among itemsSlide4

Critical Action/Complementary Analysis Research Matrix Application (CARMA)4

CARMA helps you move from Heaps of Data to Complexes of Relationships toConcepts related to a theoretical perspective or producing a grounded theory Putney, L. G., Wink, J & Perkins, P. (2006). Teachers as Researchers: Using the Critical Action Research Matrix Application for Reflexive Classroom Inquiry. Florida Journal of Teacher Education, Vol IX, p. 23 - 35.Slide5

5

NOTE-TAKING

EXPECTATIONS

Who are the intended participants?

Who is supposed to provide what service?

How are participants to be served?

What will be produced in the activity?

NOTE-TAKING

EVIDENT IMPLEMENTATION

Who are the actual participants?

Who is actually providing what service?

How are participants actually served?

What is actually being produced?

NOTE-MAKING / ALIGNMENT OR DEPARTURE?



Compare/contrast expected with evident using Action Researcher Interpretations



For each question do you see alignment or departure?

o

What

might be the implications?

NOTE-REMAKING / RECOMMENDATIONS/



Modify or maintain program activity?



In what ways?



For which participants?



With what intended outcomes?Slide6

Complementary Analysis Research Matrix Application [CARMA]6Slide7

Complementary Analysis Research Matrix Application (CARMA)7

Content of fieldnotesNoteTakingDescriptions of setting/people/activitiesMapping of physical spaceDirect quotes or substance of what was saidNote Making Observer comments in margins – interpretations & questionsNote RemakingWhat did you learn – maintain or modify?Reflective Cycle – transformative stageSlide8

Potential Uses for CARMA8

Program evaluation Applied classroom research Self study (action research) orTeam approach with classroom teacherEducational researchSchool wide behavioral practiceSingle-sex based classesComparison of several schools – English Language Learners in high ranking schools for “best practices”Policy analysisCritical incident studySlide9

Scale of use9

MicroMacroLessonClassroomSchoolDistrictCommunity

StateNational

Continental

GlobalUnit/Project/Lesson/Guest speaker (Teacher’s, Speaker’s, and Students’ expectations)

Classroom

project/

Tolerance

School-wide project/Action

Research – 111 teachers

District-wide project/

Empowerment

Community-wide/Hospice – EPY 716

2008/Critical Incident dissertationSlide10

Scale of use10

State-wide project/Proposition 227 in California- dissertationNational-level project/No child Left Behind policy analysis – dissertationContinental level/Quality Assurance in European Higher Education Area - policy analysis – dissertationGlobal level/recommendations for future research studiesMicroMacroLesson

ClassroomSchool

DistrictCommunity

StateNationalContinental

GlobalSlide11

Participant discussionFor your own research, what WAS YOUR intended vs enacted?DID YOU ENCOUNTER WHAT YOU EXPECTED?DO YOU NEED TO COLLECT MORE DATA OR ARE YOU SEEING REPEATING PATTERNS?

11CARMA APPLIED TO YOUSlide12

Complementary Analysis Research Matrix Application (CARMA)12

WHAT - Tool for critical evaluation, to help the researcher/evaluator to describe and interpret what is being accomplished in the setting, and in what ways the cultural space is being utilized by the participants, bymeasuring program expectations, against evident implementation of practices and outcomes, from the perspectives of all levels of participants (Putney, Wink & Perkins, 2006). Slide13

Complementary Analysis Research Matrix Application (CARMA)13

WHY - Roots of CARMA:Vygotskian view of life as a social construction, steeped in the context of our cultural experiencesto evaluate whether the context of a program is meeting the needs of the constituents, we would have to determine what the constituents and program administrators are socially constructing through the program implementation . Slide14

Complementary Analysis Research Matrix Application (CARMA)14

WHY - Roots of CARMA:Ethnographic perspective of learning from people about their cultural patterns of interaction (Agar, SBCDG)people, who work in a common setting for individual yet related purposes, begin to co-construct norms and expectations for being together in their local time and space . Slide15

Complementary Analysis Research Matrix Application (CARMA)15

HOW – Observations and Interviews of constituentsFieldnotes – written account of observationsContinuous notes, jottings, head notes,“cooked up”Slide16

teacher’s reflections on her own practice16

Example from action researchPutney, L. G., Wink, J & Perkins, P. (2006). Teachers as Researchers: Using the Critical Action Research Matrix Application for Reflexive Classroom Inquiry. Florida Journal of Teacher Education, Vol IX, p. 23 - 35.Slide17

Step 1 – NoteTaking Notations on classroom expectations17Slide18

Step 2 – Notetaking from observations what is evident 18Slide19

Step 3 – results and conclusions after contrastive analysis 19Slide20

Step 4 - Recommendations - decisions20