How to Conduct Effective Assessment - PowerPoint Presentation

How to Conduct Effective Assessment When Nobody Wants to They Can’t Agree What to But We Have to The history of a large community college English Department’s struggle to implement a district wide assessment program. Presented by: Beth Baker-Brodersen, Bret Ross, and Chelli Gentry DMACC

DMACC Founded March 1966 6 campuses AnkenyBooneCarrollNewtonUrbanWest Student Demographics 55% Female 45% Male Average age is 23 40% Full-time 60% Part-time Student Enrollment 25,000 Offer 168 career programs, certifications, and technical programs as well as liberal arts transfer degrees

Purpose AccountabilityHLCAQIPImprove Student learningTeaching

Assessment at DMACC Shared process of purposeful, systematic measurement used to document, reflect upon and improve student learning.Engaging in assessment activities involves four processes that result in improved student learning.Collaboration with peers to identify clear, valid and appropriate course and program competencies.Systematic collection of evidence that the identified competencies are addressed.Thoughtful dialog to arrive at a collective interpretation of the data.Agreement to use data to improve both teaching and learning.

Assessment Process

Course Level Assessment Represents the largest part of DMACC’s comprehensive assessment programFaculty work collaboratively to develop an assessment tool to assess the course competencies for a target courseT he assessment enables faculty to identify trends in student learning to analyze, monitor and enhance the course curriculum

Continued Course assessment works well for liberal arts courses for transfer degreesAppropriate methods of course level assessment include: testing, portfolios, projects evaluated by rubrics, or any other systematic method of collecting data Use same method of assessment across the district

DMACC ENG Assessment: A Brief History, 2002-2009Beth Baker-Brodersen English professor District Chair of Communications DMACC West Campus

A Delicate Endeavor “Comp. I Assessment Analysis: Fall 2006 through Spring 2009” Many of our composition faculty feel alienated or are just plain disinterested in the subject of writing assessment. They especially feel disconnected by an assessment that happens outside of the classroom for purposes of program evaluation. This distrust is understandable considering the teaching load most composition faculty are faced with every semester and the relative isolation created by a multi-campus institution.

Guiding Principles of Assessment Conference on College Composition and Communication, “Writing Assessment: A Position Statement” Designed/evaluated by well-informed faculty for clearly understood purposesElicit a variety of pieces, preferably over timeEncourage and reinforce good teaching practicesGrounded in current and relevant research on language learning and best practices

2002-2009 2002/2003: developed and implemented a timed-writing based on reading prompt, with opportunity for revision later in the semester2004: Ed White, recognized expert in writing assessment, visits DMACC and suggests portfolio approach 2005-2009: Portfolio assessment developed, refined, and implemented; Ed White visits again

DMACC’s English 105 Assessment Project: The Nuts and Bolts (Mostly Nuts) Presenter: Bret RossChair, ENG 105 Assessment Committee

My Qualifications DMACC English faculty memberMember of the first project’s assessment committeeBaseball scorer

Section 2 of ENG 105 (Comp 1) Competencies 2. Practice reading as an active part of the writing process. 2.1 Use effective reading techniques such a rereading, annotating, and summarizing. 2.2 Differentiate between main and supporting ideas. 2.3 Distinguish between objective and subjective material. 2.4 Understand connotation and denotation. 2.5 Demonstrate sensitivity to discriminatory language. 2.6 Analyze the content, expression, and context of the writing. 2.7 Adapt material for specific writing purposes.

The Former ENG 105 Assessment Borrowed ideas from the old ENG 105 Portfolio AssessmentLow stakesRubric designUniversalityHolistically scored writing Data Collection

The Resurrection—Combining Old and New New ideasAssess a single competency2.6 Analyze the content, expression, and context of verbal or visual text. (revised) Key Idea: No single essay promptEach ENG 105 Instructor should choose a text for student analysisAssessment as Experiment—Hypothesis:‘Tradeoff sample stability for increased instructor involvement’

Results Two pilot scoring sessions conducted by Boone DMACC ENG facultyBoone project given go ahead in February, 2013ENG 105 Assessment Committee formed in Spring 2013Developed student essay constraintsDeveloped phased-in faculty involvement schedule Revised scoring rubric

Results: Scoring Rubric 0 1 2 34 5Inappropriate thesis; paper’s focus is not on critical reading; paper reads as a character description, narrative, short story, etc. No clear analytical thesis statement  May have an analytical thesis statement, but material does not support it; the paper’s focus is unclear   Has an analytical thesis statement Has an analytical thesis statement; may also forecast techniques to be explored in paper Has an insightful analytical thesis statement No use of analytical terminology to describe or interpret the text May include some analytical terms, but terms are misused to describe text May understand analytical terms, but does not cite textual evidence of their use Understands analytical terms, but description of text is haphazard or weak Understands analytical terms and describes text generally well; adequate use of textual evidence Has command of analytical language to describe text; uses interesting textual evidence No interpretation and does not include any reference to a text at all Little or no interpretation: Material is dominated by summary of the text or personal taste asides Unclear interpretation: may try to analyze, but ends up with broad summary of points or personal tastes Interpretation of text is apparent but not developed adequately Interpretation of text is generally thorough, but may have isolated logical gaps in development Interpretation of text is thorough and consistent; may exceed assignment expectations Exhibits no written consideration of text’s audience, purpose, or context Has little consideration of the text’s audience, purpose, or context Has little consideration of the text’s audience, purpose, or context     Exhibits some consideration of the text’s audience, purpose, or context Adequately considers the text’s audience, purpose, or context   Weaves audience and contextual considerations into analysis seamlessly Material lacks sufficient development: paper appears to be short answer exercises, or may be only a paragraph or two in length Material lacks analytical focus, organization, and clear expression Material may have initial analytical focus, but deviates; lacks clear organization and expression Material is focused on analysis, but occasional weak expression and organizational problems occur Analysis is well focused, organized, and expressed, but isolated problems exist Superficial errors may exist, but analysis shows strong composition skills overall

Results: Essay Scoring Excerpt (from Fall 2014) Folder 12 *(out of 15) Reader I Reader IIADJ/AVScore A 10 1 B 1 3 1 C 0 0 2 D 2 3 2.5 E 0 1 DQ F 2 5 5 G 1 4 3 H 2 3 2.5 I 3 5 3 J 1 3 1 K 3 4 3.5 L 3 4 3.5 M 2 3 2.5N031O111P121.5Q011R142S121.5T131 Key:Regular=AveragedBold=AdjustedGreen=AdjudicateDQ=DisqualifiedB.E.G.I.N.= Coded Essays (Concurrent Student Essays) 20 Essays/15 Folders= 300 Student Essays

Results: Fall 2013 (FT only) & Spring 2014 (FT +ADJ)

Results: Fall 2013 & Spring 2014 * 22 DQ Essays * 68 DQ Essays Avg. Score: 2.24 Passing Scores (>=3) 77 (27.6%) Avg. Score: 2.27 Passing Scores (>=3) 64 (31.8%)

*Results—Fall 2014 (All ENG 105 Faculty) *Essays screened for DQs before scoring AVG. Score: 2.45 Passing Essays 110 (37%)

Deliverables Process BenefitsSteady improvement in faculty buy-inIncreased communication between faculty segmentsSpurred development of other assessment projectsCreated “Assessment Community” projects Product BenefitsFulfillment of AQUIP requirementsResults of Assessment Communities (pending)Improvement of student reading skills (?)

Conclusions and Outlook At this point process benefits trump product benefitsIn order to improve the product we need to expand the processWe need a forum to present and discuss ENG 105 assessment resultsFrom this discussion, we need to focus on best practices toward developing students’ reading skills We need a specific and recurring “ENG Faculty Development Day”

Wrap Up Faculty consensus is not easyDo somethingGet as much buy-in as possibleFind championsCooperation collaboration are a necessary part of the processProduce resultsRemain patientCreate change

Questions Thank you

Appendix Works Consulted for the ENG 105 Assessment Project 2011-PresentEmail copy sent to all DMACC ENG faculty, Fall 2014.Example of a ‘Default Essay Assignment’ for the ENG 105 Assessment