JoAnne Baird Therese Hopfenbeck David Andrich amp Gordon Stobart Knowledge Economy economic importance The Audit Society important societal control function assessment defines what counts as valuable learning through these mechanisms ID: 808650
Download The PPT/PDF document "March 3, 2015 Assessment & Learnin..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
March 3, 2015
Assessment & Learning: fields apart?
Jo-Anne Baird, Therese
Hopfenbeck
David
Andrich
& Gordon
Stobart
Slide2Knowledge Economy
economic importance
The Audit Society
important societal control functionassessment defines what counts as valuable learning through these mechanismsMultiple, high-stakesassessment’s domination over learning Assessment is agenda-setting21st Century has seen interesting developments already Build cumulatively on what has already been done
The need for a review on Assessment and Learning
Slide3What do we mean by theory - functions
AbstractionAbductive reasoningWallander
has a theory that
Schwarzman killed IngaDistinguished from practiceIt’s just a theoryNormativeHow things ought to be ExplanatoryDescriptiveMay be causalMay be predictiveMay be formalised in logic or mathematical equationsMarch 3, 2015Page 3
Slide4What do we mean by theory - focus
Scientific theoryRelates to empirical phenomenaHas an internal logic
Should be empirically testable
Substantive theoryLearningDevelopmental psychologyTest theoryMetrologyPsychometricsMarch 3, 2015Page 4
Slide5Overview
Relationships between substantive learning theory & assessment
Theoretical and philosophical dilemmas
Case Studies - applicationsInternational testsAssessment for LearningConclusionsMarch 3, 2015Page 5
Slide6March 3, 2015
Assessment and Learning
Jo-Anne Baird & David
Andrich
Slide7Behaviourist theory of learning
Learning is demonstrated in behaviour
Mental processes are not important
Study of animals tells us about human learning (eg rats & pigeons)Learning as a reaction to stimuli in the environment, such as teaching
Slide8Behaviourist
approach to assessmentControl conditions
Measure memory for facts
Compare performance with criteria or normsGlobal score for performance on ability in subject areaNorm- or criterion-referenced
Slide9Cognitive-constructivist theory of learning
Learning occurs in the brainCognition, especially meta-cognition important
Memorisation
of facts not so impressiveBuilding of mental models of the worldIntegrate and build upon previous knowledge and learningNovice-expert differences
Slide10Cognitive-constructivist assessments
Higher order skillssynthesis
Evaluation
Problem solvingExtended tasksAssessed in terms of novice-expert continuum
Slide11Socio-cultural theory of learning
Learning is a social eventLearning is situated and context-dependent
Learning is value-laden
Learning does not happen within, but between people
Slide12Socio-constructivist approach to assessment
Holistic, qualitative feedback emphasised
Authentic tasks important
Groups as well as individuals assessedSelf- and peer-assessment importantEngagement with criteria
Slide13Theories of learning and assessment practices
Have learning theory and assessment practice informed each other?
Or have they been growing apart?
Slide14Chronology and relationships
Learning theories have been contemporaneous Cognitive psychology superseded behaviourism
Vygotsky’s
work overlapped considerably with behaviourist thinkingCognitive constructivism and social constructivismLinks between forms of assessment and learning theory not cleare.g.Multiple choice format can be used to assess cognitionImplications of theories for assessment practice not straightforwarde.g. Skinner (1989) Good instruction demands two things: students must be told immediately whether what they do is right or wrong and, when right, they must be directed to the step to be taken next. March 3, 2015Page 14
Slide15Assessment and psychometrics: they are different
Lawn (2008) Crossing the Atlantic – history of the development of different approaches to assessment in European countries and the USEarly 80s ‘Rasch
wars’ in the UK
Nuttall, Gipps, Broadfoot, Black, Harlen … argued for a more educationally sound approach to assessment that was learner-centredBaird & Black (2013) outlined how psychometrics does not fit well with a range of educational assessment purposes: curricula change (construct), criteria public, correlation between questions, qualification-focus (not item), multiple dimensions in tests, pre-testing not always feasible …Psychometrics literature – how has it related to learning theoryMarch 3, 2015Page 15
Slide16Psychometric models – representational measurement
The Ferguson Committee – 1940 British Association for the Advancement of Science
No evidence that psychological assessments were quantitative
Based upon Campbells 1920s arguments against psychophysicsRebuttal by Stevens (1946), giving us the different forms of ‘measurement’ – nominal, ordinal, interval, ratio [learning can be measured in each of these ways]Measurement as a product of the instrumentLuce and Tukey’s additive conjoint measurement – mathematical proof that ratio scales could be produced from transformations of ordinal variables. BUT the assumptions of additive conjoint measurement are not met by assessment data. For example, transitivity (If A>B and B>C then A>C): item parameters change across time and sub-populationsMichell (1997) says this isn’t measurement in the scientific sense, but he says that about all of psychological assessment and by implication educational assessmentMarch 3, 2015Page 16
Slide17Psychometric models – classical test theory
Based upon centuries of statistical work
Lord and
Novick (1968) made the great leap forwardBUT suffers from the same problems as representational theoryTrue score = learning part of the equation, but what is it? TheoreticalMarch 3, 2015Page 17
Slide18Psychometric models – latent trait theory
Come from factor analytic models produced by Spearman (1904)
One parameter (difficulty), two parameter (& discrimination), three parameter (& guessing) typically used.
Multi-dimensional forms also availableOnly the Rasch form (one parameter) can deal with the transitivity problemsUnlike in the representational theory approach, Rasch is probabilistic, so deviations from the model are handledMore than one parameter causes problems for transitivityBUT still does not deal with Michell’s criticismsAre psychological constructs quantifiable? Not just a problem for psychometrics – for the assessment field broadlyMarch 3, 2015Page 18
Slide19What does it mean to assess attainment?
Cronbach & Meehl (1955)A construct is some postulated attribute of people, assumed to be reflected in test performance. In test validation the attribute about which we make statements in interpreting a test is a construct.
Educational attainment constructs – grading & scoring scheme
Assessment by associationAll based upon correlationDimensionalityHow can we combine different things?InvarianceDo the scores mean the same thing across tests and groups of students?March 3, 2015Presentation title, edit in header and footer (view menu)Page 19
Slide20Physics envy
Michell (2008) – requirements for real numbers should be satisified
Realist - numbers
exist independently from humansKane (2008)Educational assessments do not meet these stricturesPhysics is held up as an idealistic example of measurement, but Physical measurements took a long time to develop – theories & apparatusMeasurement in physics shows inconsistency across too (Hedges, 1987)Need externality to our measures & multiple ways of measuring to substantiate that there is a real phenomenonMarch 3, 2015Presentation title, edit in header and footer (view menu)Page 20
Slide21Creation of constructs
Theory-basedEmprically
-driven
Subject-matter expert devisedPolicy-drivenAgenda-setting activity“Different methods and theories have implications for the ways in which concepts such as learning or educational reform or fairness are formulated, studied and promoted as a practical activity. Perhaps more profoundly and subtly, these methods and theories affect the ways human beings are represented and, ultimately the ways they come to understand themselves and others … Moss (2005)“…it may not always be clear to what extent an attribute is conceptually independent of the methods of measurement, especially in human science applications.” Maul (2013)March 3, 2015Page 21
Slide22Philosophical position
Field is essentially modernist, Borsboom claims realist too
Attributes have an independent existence
They are discoverable using scientific methodsNeopragmatic, postmodern test theoryAgnostic as to the nature of the attributes and their independent existenceAccept that the attributes might be defined in part, or entirely, by the assessment apparatusUse triangulation to advance knowledge of the attribute and measurement systemMarch 3, 2015Presentation title, edit in header and footer (view menu)Page 22
Slide23Fields apart
Goldstein, Laming and others have pointed out that there is no theory in test theory – only mathematical modelsMcGrath (2005) – psychometrics has caused the problem
Borsboom
blames psychologists for lack of theory underlying psychological testsAndrich has argued that it should be a collaboration between substantive theorists and psychometriciansSijtsma (2006) – models of learning that underpin test design are often either not referred to or remain in puberty, infancy or even at the foetal stageHas psychometrics (or assessment) helped us to understand learning?March 3, 2015Page 23
Slide24An answer to somebody else’s problem – Baird & Black (2013)
Measurement systems have their own, internal logicMeasurement doesn’t tell us about the phenomenon of interest
Nothing about a set of numbers tells you what they measure (
Maraun, 1988)Educational attainment – setting out the construct has an intentional elementMarch 3, 2015Presentation title, edit in header and footer (view menu)Page 24
Slide25How could assessment inform learning theory and vice versa?
Empirical data can test theory and help to move it forwardWe do not seem to have used it systematically
Craft knowledge of examiners and other educators – raises questions about what kind/level of theory we expect from assessment data
Mark Wilson making serious attempts to do thisDependency of the data upon curriculum exposure and other population characteristics leads to problems with invariance: frame of reference needs to be taken more seriouslySociocultural learning theory do not fit well with standardisation-as-fairness principles of assessmentAssessment is an expression of educational valuesMarch 3, 2015Page 25
Slide26March 3, 2015
International tests & learning
Therese
Hopfenbeck & Jo-Anne Baird
Slide27International tests
Slide28International tests influence learning based upon three processes
1) what counts as valuable learning 2) how national assessment systems are developed around the world
3)
how students approach learning since there is evidence that students adopt their learning approaches according to the tests givenExample: the case of Norway
Slide29THE CURRICULA APPROACH – TIMSS, Science, grade 8
Most underground caves are formed by the action of water on
Granite
LimestoneSandstoneShaleCognitive domain: Factual Knowledge, main topic Earths structure and physical features.
Slide30Slide31Slide32How international tests have influencednational assessment systems
Norway : Introduced national tests in 2004. The reading tests are based upon the PISA reading framework (Frones
et al 2012).Denmark: Introduced national tests after low performing in PISA (Egelund, 2008).Japan: changed item format on their national tests to more open-responses like those in PISA (Schleicher, 2009).Korea: PISA like tasks on their University Entrance Exam (Schleicher 2009).Germany: introduction of national educational standards and more focus upon external assessment (Ertl, 2006)
Slide33Slide34Literature review in three steps
Step 1Broad search AHELO, PIAAC, PIRLS, PISA, TIMSSMore than 1000 articles detected.
Step 2
Narrowing to peer-reviewed (805) No grey literature, but categorized reports, book reviews and conference papers separately.Step 3Quality checks of relevant articles based upon reading abstracts.Developing categories based upon model-article and coding in End-note. Meetings with discussions on categories and coding.Still in process of quality checking.
Slide35Published peer-reviewed articles from 1993 - 2013
Slide36March 3, 2015
Presentation title, edit in header and footer (view menu)
Page
36Interest in ScienceSecondary analysis of student questionnairedata
Slide37March 3, 2015
Presentation title, edit in header and footer (view menu)
Page
37
Slide38March 3, 2015
Formative assessment & learning
Gordon
Stobart & Therese N. Hopfenbeck
Slide39Assessment for learning
Slide40Slide41March 3, 2015
Presentation title, edit in header and footer (view menu)
Page
41
Slide42Elevers forutsetninger for å lære kan styrkes dersom de
Forstår
hva de skal lære og hva som er forventet av dem.Får tilbakemeldinger som forteller dem om kvaliteten på arbeidet eller prestasjonen.Får råd om hvordan de kan forbedre seg.Er involvert i eget læringsarbeid ved blant annet å vurdere eget arbeid og
utvikling. Fire prinsipper om vurdering, Utdanningsdirektoratet.
Slide43The challenges
Lack of theoretical consensus on AfL and formative assessmentFew researchers use the original articles
Development of stories – which is flawed if you look at the original work
March 3, 2015Page 43
Slide44The
vast majority of studies on AfL are small-scale action research designs and are published in a wide range of journals.
A
concern for the review is that current definitions of formative assessment/AfL cover a wide range of teaching and learning practices while research designs often lack an action theory (what is causing change), often accompanied by a lack of systematic data collection (for example baseline data before a research initiative).March 3, 2015Page 44
Slide45Overselling?
Whilst claims for large effect sizes are regularly made in the literature, the evidence for these has increasingly been critiqued, for example by Bennett (2011) and Kingston and Nash (2011)
.
The effects of formative assessment upon learning have been over-sold by some authors. This is unfortunate because the limited empirical research suggests a modest, but educationally significant, impact on teaching and learning.March 3, 2015Page 45
Slide46“Good
instruction demands two things: students must be told immediately whether what they do is right or wrong and, when right, they must be directed to the step to be taken next.
We presume that Skinner meant that students be directed to the next step even if they were wrong, although he did not write
that”. B.F. Skinner, in a letter to Science in 1989March 3, 2015Page 46
Slide47Assessment models need to have a closer relationship with learning
Educational assessment is a social construct – it is often an intentional, agenda-setting activity
Test theory provides statistical models which will vary in utility with context
Cognitive learning theory is the current model in assessmentInternational tests influence learning through policyAssessment for learning influences learning through practiceAssessment outcomes have powerful effects – the numbers have a life of their own. Unless we change our practices, the effects will be more detrimental upon learning
Conclusions
Slide48March 3, 2015
Page 48