University of South Florida Office of Institutional Effectiveness and Review Welcome and thank you The Office of Institutional Effectiveness Academic Planning and Review Melanie Wicinski PhD ID: 759705
Download Presentation The PPT/PDF document "Academic and Administrative Assessment I..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Academic and Administrative Assessment Information Session
University of South FloridaOffice of Institutional Effectiveness and Review
Slide2Welcome and thank you!
The Office of Institutional Effectiveness, Academic Planning and ReviewMelanie Wicinski, PhDAssistant Director Institutional Effectiveness & Assessment
Slide3Agenda
Academic assessment reports/SAM academic report Components and requirements of an assessment plan Administrative assessment reports System for Assessment Management (SAM)
Slide4Academic Assessment Plans/Reports
Annual report of each degree and certificate program. Program learning goals and learning outcomesPlans for future program developmentFormative assessment of the degree or certificate program, not summative assessment of students. Non-comprehensiveNon-linearProgram-focusedLooking for opportunities for growth and developmentContinuous
Slide5Components of an Assessment Plan in SAM
1. Program/Department Mission Statement2. Program Learning Goals (LGs) 3. Learning Outcomes (1 or more per LG) i. Statement of Learning Objective ii. Method of Assessment iii. Performance Targets iv. Results v. Use of Assessment Results
Slide6Components of an Assessment Plan in
SAMBreakdown
Slide7Learning Goals v. Learning Outcomes
Learning Goals
Not always measureable, broad statementsSkills, Content, knowledge, or tasks students will haveExample: “Development of Critical Thinking Skills.”Undergraduate RequirementsContent Specific KnowledgeCritical Thinking SkillsCommunication Skills *Graduate degrees and Certificates: no minimum reporting requirement
Learning Outcomes
A demonstrable skill, measureable change in behaviorSpecific Statements What students will know and demonstrate after the course(s)Example: “Majors in this program will be able to conduct original research in this discipline using appropriate methods.” Includes 5 detailed parts
Slide8Method Of Assessment Section
Should be the most detailed section under the Learning Outcome Less flexible, has several requirements of what should be includedExamples of Assessment MethodsWritten Student WorkStudent Presentation/PerformancePortfolio of Student WorkPart of a Lab ReportEmbedded test questionsPart of a thesis or dissertationStandardized tests*Internship/Practicum evaluation*
Slide9Method of Assessment: Requirements
1. A clear description of the assessment method2. A statement on how the assessment specifically measures the task, information or competency stated in the Learning Outcome3. Context of the Assessment 4. How the assessment will be scored5. Which students in the program will be assessed
Slide10Method of Assessment: Requirements Cont.
6. If a sample of student work will be analyzed in lieu of all students, include information on the sample 7. If employing a rubric, information is needed on the rubric used and how it was developed and validatedE.g. How Inter-rater reliability will be addressed8. Information on how the rubric will be calibrated9. Information on who will be reviewing and rating the assessment.
Slide11Method of Assessment: Inappropriate Assessments
Grades CANNOT be used as outcomes (summative assessment of student work, not formative assessment of program work)Summative Student Assessments (the passing or failing of a culminating exam, course or program) CANNOT be used (summative vs. formative)Student/faculty/Alumni Surveys CANNOT be used (indirect measures)Publication in an academic journal CANNOT be used as a student learning outcome (external factors) Job placement CANNOT be used (external factors)
Slide12Performance Targets: Target for measured program performance
SHORT statement on what benchmark(s) the program wants to meetThis is not how we think our students will perform, but what we want our program to doShould be worded in terms of performance on the rubric or rating instrument“Performance target will be considered met if 75% of students achieve an overall score of 4/5 or higher”“Performance target will be considered met if 80% of students assessed will receive a final score of “Commendable” or higher.
Slide13Assessment Results
SHORT statement on the final results of the assessmentStated in terms of the rubric or grading instrumentDoes not need to include commentary about the results and why or why not the performance targets were metDoes not need to be, and should not be, perfectDon’t be afraid to report that the target was not met
Slide14Use of Assessment Results: Do
Should include specified, actionable “next steps” the program will take, using the assessment results, to develop or improve on an programmatic level: Curriculum Mapping/redevelopment Revisit or revise the assessment method/rubricRevisions to plan of study/program structureDevelopment of new modules/courses Faculty development
Slide15Use of Assessment Results: Don’t
Remember: The responsibility is on the program, not its students. Use of results section should indicate programmatic improvementsThis is not an assessment report of student achievementDo not include: tutoring effort, sending students to the writing center, counseling students, providing students feedback, or improved recruitment
Slide16Certificates
The following three possibilities apply to certificates:If ALL of the students in the certificate program are non-degree seeking, non-financial aid students, then you can state that in SAM and do not have to complete a program assessment.If even one student who is degree-seeking and receives financial aid is enrolled in the certificate program then you have to provide an assessment (I know some certificates can be taken as electives in a degree-seeking program and this is where this may come into play).If your certificate is a sub-set of a full degree program (the classes in the certificate are really just a portion of a full degree program), then your certificate assessment can be identical or a subset of your assessment reported for the degree program.
Slide17Rubric for Academic Program Assessment
Cycle
Element
Quality Level
Missing Data
Initial
Emerging
Developed
Highly Developed
Planning
Mission
May be the departmental mission
Mission statement is missing
A statement is present, but is not a mission, but a description of the program
Mission statement is present and appropriate
Program Goals
Should match the Mission Statement and each should be one complete statement
Undergraduate
Three must be covered (Content Specific Knowledge, Communication, Critical Thinking)
Any one portion of the three required Goals are not completed (Content Specific Knowledge, Communication, Critical Thinking).
A statement is present, but not a Goal
Goals are succinctly and clearly stated
Graduate and Certificate
No required amount/content program goals
No Goal present
Not a learning Goal
Learning Goals are listed, but not aligned with mission statement
Learning Goals are listed and clearly aligned with the mission statement
Learning Outcome Statement
Should be actionable and realistic
No Learning Outcome present or not revised from previous year
The statement is present, but not stated as a learning outcome
The Learning Outcome is present, but may not be an appropriate student learning outcome
The Learning Outcome may need minor revisions
The Learning Outcome is present and well-developed and clearly stated
Method of Assessment
Should include type of assessments, how the program is being measured, and should provide information specific to the stated measure
No stated method of assessment or not revised from previous year
Method is inappropriate; not connected or specific to the learning outcome; use of IDM, GO, external factors
Method is appropriate, but may need further description, including more information about rubric, raters, or context
Method may need minor clarification
Method is clear, concise and well-defined. Meets all reporting requirements
Performance Targets
Should be thought of as PROGRAM benchmarks. Indicate what numbers would indicate that the program has met their programmatic goal
No performance target present or targets do not correspond with methods
Performance targets present, but stated in terms of assignment, course, or degree completion
Perfomance targets are present, but may need minor clarification
Performance targets are appropriate and well-stated
Reporting
Assessment Results
Include specific findings, including number of students included in the assessment, and any other appropriate statistical information
No Assessment Results present
Assessment results present, but unclear how they relate to methods
Assessment results present, but not specific
Assessment results present and clear, but missing minor data (such as: number of students)
Assessment results present and clear. All data reported appropriately
Use of Assessment Results
Should include actionable items for PROGRAM improvement
No Use of Results present
Use of Results statement indicates no changes are necessary
Use of Results statement does not indicate what changes will be made based off data
Use of Results statement is present and clear; may need minor clarification
Use of Results statement is present and clear and specifically indicates the changes that are being made
Slide18The Phases of the Academic Reporting Cycle(Note: For all colleges, except Education and Business)
Important Notes:
Changes to your plan can be made at any time.
Reviews by OIE are intended to be helpful – this is a conversation.
Adapted
from
Rodriguez
& Frederick, Miami
Dade
College
2013
Slide19Academic Reporting Cycle DatesNote: For all colleges except Education and Business
Academic Assessment ReportingPlan YearPlan Due DatesPlan Approval DateFinal Report Date2016March 31, 2016April 15, 2016December 15, 20162017January 15, 2017January 30, 2017December 15, 20172018January 15, 2018January 30, 2018December 15, 2018
Slide20The Phases of the Academic Reporting Cycle(Note: For Education and Business ONLY)
Adapted
from Rodriguez & Frederick, Miami Dade College 2013
Important Notes:
Changes to your plan can be made at any time.
Reviews by OIE are intended to be helpful – this is a conversation.
Slide21Academic Reporting Cycle DatesNote: For Education and Business ONLY
Academic Assessment Reporting
Plan Year
Plan Due Dates
Plan Approval Date
Final Report Date
2015-2016
N/A
N/A
October 1,
2016
2016-2017
October 15, 2016
October 31, 2016
October
1,2017
2017-2018
October 15, 2017
October 31, 2017
October
1,2018
Slide22Administrative Assessment Plans
Very similar to Academic plansThey are different in the following ways:Learning Outcomes vs. ObjectivesSame type of cycle, but different due datesNot nearly as strenuous and detailed – RIGHT NOW!May include graduation rates, alumni surveys, employer surveys, etc.Ask: What are the goals for your support area? What can you improve? What goals might you have been given from leadership?
Slide23Example of Administrative Assessment Objective
Objective: Encourage funded and unfunded research involving community-based partnerships.Method: Track grant applications captured by Office [name] by using [software].Performance Target: Increase community-based grant applications by 5%.Results: There were ### community-based grant proposals submitted this past calendar year, representing a #.#% increase in proposals. There were ### community-based grant awards in that year, representing an #.#% in awards, to a total of $#. Already in 2016, there is a total of $# in awards.Use of Results: These data are also self-identified on the … Form of the Office [name]. A task force is working on easier methods of gathering and recording community-based grants and funding, as well as community-based research that is done without external funding.
Slide24Slide25The Phases of the Administrative Reporting Cycle
Adapted
from Rodriguez & Frederick, Miami Dade College 2013
Important Notes:
Changes to your plan can be made at any time.
Reviews by OIE are intended to be helpful – this is a conversation.
Slide26Administrative Reporting Cycle Dates
Slide27How does USF organize academic and administrative program assessment plans?
“SAM” The System for Assessment Management Database maintained by the Office of Institutional Effectiveness Available online with NetID/USF password
Slide28Accessing the System for Assessment Management
The pink (or orange)
paper provided in your training materials has detailed instructions for logging in.Recommendation: Once you get to the login page, bookmark it!
Slide29Slide30How do you know what your plan was rated?
Slide31Overall Plan Rating Scale
If a plan receives a yellow or red overall rate, we will select that is “not complete”
Programs will be notified and asked to make changes to sections that still need clarification or modification
When secondary adjustments have been made, the “
P
lan complete?” button should be changed to “Yes”
Slide32Take Away Points
Formative of program, not summative of studentsNot comprehensive assessment of all program efforts, make it manageableUse samples when possibleNot linear, if one area has met the performance target and is going well, move on to another areaDevelop a tentative five year assessment progressionBrainstorm possible area of improvement to tackle over the next 5 yearsUniversity-wide 5 year strategic plans will be coming to the college and unit level.
Slide33Questions?
PLEASE CONTACT OUR OFFICE ALONG THE WAY! We are here to help!