An institutional case study of using standardised testing to measure and develop students generic skills Professor Stuart Brand and Jamie Morris European First Year Experience Conference ID: 910960
Download Presentation The PPT/PDF document "The Collegiate Learning Assessment (CLA+" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
The Collegiate Learning Assessment (CLA+): An institutional case study of using standardised testing to measure and develop students’ generic skills.
Professor Stuart Brand and Jamie Morris
European First Year Experience Conference
Utrecht 2018
Slide2Institutional Context – Birmingham City University
Post 1992 University
~24,000 Students8 Campuses moving to 2/3
Vocational approach to learningMulticultural student populationCommuter students (69%)
Slide3National Context - the Teaching Excellence Framework (TEF)
The TEF aims to recognise and reward excellence in teaching and learning, and help inform prospective students' choices for higher education.
The TEF was developed by the Department for Education in England.It was voluntary to participate in.
Universities can qualify for awards by using the following metrics...
Slide4TEF cont…The Teaching Excellence and Student Outcomes Framework (TEF) aims to recognise and reward excellence in teaching, learning and student outcomes.
Slide5TEF cont…The TEF is being developed via a phased approach:
Slide6International Context - The RAND Report (2015)
http://www.hefce.ac.uk/pubs/rereports/year/2015/learninggain/
Commissioned by the Department for Business Innovation and Skills (BIS), the Higher Education Funding Council for England (HEFCE) and the Higher Education Academy (HEA)
Rand Europe
Slide7What is ‘Learning Gain’?
The Office for Students characterise ‘Learning Gain’ as the following:An attempt to measure the improvement in knowledge, skills, work-readiness and personal development
made by students during their time spent in higher education.In our case - what level of skills do students enter and leave with when they undertake their course of study?
How do curriculum and extra-curricular activities contribute to the development of these skills?
Slide8Group Activity – Discussion On your tables, discuss the following questions:
What does ‘Learning Gain’ mean to you and your own subject/discipline?
Do you do anything already in regards to tracking the ‘value added’ to your programmes/courses?
If so, what data do you use and how do you collect it?
Slide9HEFCE and ‘Learning Gain’
The Higher Education Funding Council for England (HEFCE) released a funding call in 2015 for new learning gain projects in selected Universities throughout England
Decisions released formally at end of July 2015
13 collaborative projects fundedBirmingham City bid with three partners: Coventry, Liverpool John Moores and Staffordshire
Slide10Project Overview - Revised Proposal
Original Proposal – longitudinal study, 1,000 students with 4 testing points.
Revised to:
2 longitudinal studies from 2015-18 and 2016-19Follows an additional ~1000 students2 Cross sectional study in Semester 2 of 2016-17 and 2017-18 academic year Test a mixture of L4 (1st year), L5 (2nd year) and L6 (3rd year) 4 testing points reduced to 3:Autumn of year 1Spring of year 2Spring of year 3
Slide11MethodologiesStandardised tests
- These measure the skills students acquire, which may be generic or specialised (for example, a skill specific to a discipline).
Self-reporting surveys - Such surveys ask students to report the extent to which they think they have gained knowledge and developed skills.
Other qualitative methods - These encourage students to reflect on their learning, acquired skills and skills gaps. Mixed methods - This draws on a range of tools and indicators to track improvement in performance, (for example through a combination of grades, student learning data and student surveys).
Slide12Emergent approachesCognitive gain
– what students think and knowSoft skills development – affective measures of attitudes and how students feel
Employability and careers readiness – mainly behavioural measures of activities students have undertaken in preparation for the world of work
Slide13The Council for Aid to Education (CAE)
In 2002, CAE, a not for profit based in NYC, launched a national effort to assess the quality of undergraduate education.
CLA+ (Collegiate Learning Assessment) is a web administered, value added assessment programme.
The aim of CLA+ is to provide an objective assessment about the critical-thinking skills a student possesses as they enter and exit college.
Slide14The Collegiate Learning Assessment (CLA+)
The Collegiate Learning Assessment (CLA+) is a performance based test that measures critical thinking, problem solving, analytic reasoning, and writing skills.
The CLA+ uses a Performance Task (60”)
and 25 Selected-response Questions (30”).Can take up to 2 hours to complete - Tutorials and Demographic Survey.PT responses are scored (1-6) by staff leads in each institution - requires scorer training.
Slide15Why CLA+? (CLA post 2011)Generates an individual student report based on a Mastery Level (1-5)
Mastery Levels are formed via a combination of scores for the Performance Task and Selected-Response Questions.
Provides a rich source of data from the U.S and other participating countries e.g. Italy, Germany, Chile and Peru using the CLA+.
Key point - Emphasis on highlighting areas of development for the student.
Slide16How have we embedded the CLA+? – Optional vs Compulsory1st and 2nd year of testing - financial incentives
Low attendance - first two years
Successful engagement with the Birmingham City Business School
Embedded into Professional Studies Module as formative assessment. Importance of engaging programme and modular level staff
Slide17Findings from Optional Testing – Focus Groups
Structure – Issues with lack of spell check and viewing content.
Most students believed that the test was not subject independent.
The time the test was taken in the academic year was an issueStudents would take the CLA+ test if employers and other HEIs valued the scores from it.Students who had taken the test twice noted enhancement of other skills e.g. time management
Slide18Focus Group Findings (optional testing) – cont…
Cross-sectional study Spring 2017 (n=78)
Students also reported specific disciplines may be advantaged in the test
Slide19Autumn (Fall) 2017 Testing – Compulsory
201 L4 students tested (81% response rate) in a week
Previous issues with recruitment no longer encountered
Focus on one participating School - Birmingham City Business School Testing embedded into a core moduleTied directly to summative assessment – E-Portfolio
Slide20MKT4020 - Professional Development
Developed via the University’s Transforming the Curriculum (TtC) process
CLA+ embedded into Week 2 of the Module, focusing on Critical Thinking:
LO1 - Undertake an audit of skills and capabilities, reflecting on the outcomes and engage with a programme of professional development.LO3 - Evidence skills in the context of professional development and evaluate impact. Other sessions include:Building competency with contemporary IT software e.g. Microsoft OfficeEnhancing presentation skills Forming a professional portfolio and profile via e-portfolio and Social Media
Slide21MKT4020 – Professional Development
Reports distributed in Week 7 employability week in bespoke activity
Tied into extra curricular awards framework – Graduate+
Majority of the reports collected! CLA+ individual student report submitted as evidence for summative assessment Electronic copies also available to send to students who could not attend.
Slide22Results – Autumn (Fall) 2017
Slide23Cont..
Slide24Slide25Fall – Mastery Levels by Ethnicity
Slide26Native vs 2nd Language
Slide27Summary of Testing Numbers 2015/2018 - Birmingham City University
Institution total:2015/16 academic year - 220
2016/17 academic year - 299 2017/18 academic year – 259
Overall Total (Across 4 partners):2015-2018 longitudinal study – 845 students tested 2016-2019 longitudinal study – 693 students tested2017 (Spring) cross-sectional study – 201 students tested2017 (Autumn) cross-sectional study – 248 students tested2018 (Spring) cross-sectional study – 167 students testedTotal number of tests administered – 2,154
Slide28CLA+ as a diagnostic tool – Future Developments Interest from numerous staff at BCU on utilising the test as part of a foundation degree.
Could also provide comparative data if L4s are tested.CLA+ to be continue being used in Birmingham City Business School programmesCAE to provide anonymised, aggregated analysis of student performance across 4 partners
Slide29Group Activity – Discussion Point On your tables, discuss the following questions:
Do you use diagnostic testing in your practice/institution? How so and what is its purpose? Do you use any other diagnostic methods in your practice?
In your experience, what works and what doesn’t in relation to diagnostics?
Slide30Thank you for listening!
Any questions?