T ime for Multi-State - PowerPoint Presentation

Slide1

Time for Multi-State Models of Vocabulary Acquisition?

Rob Waring

w

aring.rob@gmail.comSlide2
Slide3

Assessing vocabulary acquisition

Which components do we assess?

receptive

?

productive? use

?

form? meaning?

Which test type?

m/c? translation? vocab knowledge scales? other?

Which words?

general?

t

echnical?

l

ow-hi frequency?

Stakes

high - for formal assessment, grades etc.

low

-

for research?Slide4

Problems with our current test battery

Translation

L1 –> L2 or L2 –> L1

Low volume–

only a few dozen words at best

before we kill the subjects :)

Notoriously difficult to score

Inter-rater

reliability issues

Criteria for successful answer – award half points?

Easy to make arbitrary choices about what is correct

Not sensitive to partial knowledge

Scores can be affected by test strategy use

Lack of knowledge of L1 word for an L2 equivalent

Not all words can be easily translated

Etc

.Slide5

Problems with our current test battery

Multiple choice

Low volume–

only a few dozen words at best before we kill the subjects :)

Notoriously difficult to make

Which distractors?

Sensitive or not?

Contextualized or not?

Equating vocabulary frequency of the distractors and the target

Use of definitions, synonyms,

or ?

???

25% is a

giveaway (unless there’s ‘I don’t know)

Often need correction for guessing

Scores can be affected by test strategy use

Etc.Slide6

Problems with our current test battery

Knowledge scales

e.g.

Wesche

and

Paribakht

(1996)Slide7

Problems with our current test battery

Knowledge scales

e.g.

Wesche

and

Paribakht

(1993)

A

*?%$#>& mess

!

-u

sing

ordinal data nominally

-multiple aspects of knowledge at the same level – internal inconsistency -productive and receptive all mixed up -totally arbitrary scoring -unclear what gains mean (e.g. t1 mean 2.5, t2 mean 2.7 t3 mean 2.8) -compare S1 25@I + 25@V vs. S2 25@ II + 25@IIISlide8

Assumptions underlying scales

We move from the receptive to productive

receptive productive

But this assumes receptive knowledge is complete before we can produce

Huh?Slide9

Assumptions underlying scales

We

could have

separate scales

receptive

productive

But any gains on the receptive are not seen on the receptive and vice versa

Huh?Slide10

Assumptions underlying scales

We

start with a threshold receptive knowledge

productive

receptive

But any gains on the productive still aren’t seen in the receptive and vice versa

Huh?Slide11

A solutionSee each of the stages

as

states

of

knowledge

not a scale

Recognize the data are ordinal, not nominal

Develop linear scales of a single aspect of vocabulary knowledgeSlide12

Simple

state

model

0 I

do not

understand

(the meaning of) this

word

1 I understand

(the meaning of) this

word

a little

2 I understand

(the meaning of) this

word quite well 3 I understand (the meaning of) this word very wellTest

design

Understand

Can u

se in a sentence

Apple 0 1 2 3 0 1 2 3

Book 0 1 2 3 0 1 2 3

Curtain 0 1 2 3 0 1 2 3Slide13

Build a matrix

3

x

xxx

2

x

xx

1

x

0

xxx

0

1

2

3

Understand

UseSlide14

Track data over time

3

h

2

e

fg

1

c

d

0

ab

0

1

2

3

Understand

Use

3

g

h

e

2

c

f

1

b

d

0

a

0

1

2

3

Understand

Use

t1

t2Slide15

3d representationsSlide16

Advantages of State Models

Any words, phrases,

collocations,

etc. can be

tested

Fast

data collection – hundreds per

hour (esp. if digitally collected)

Direct access to

knowledge (subject reports what they know)

Knowledge is not mediated through assumptions for what a test is assessing

E.g. sensitive

vs

insensitive targets, with or without context

Can track a single word or multiple words over timeE.g. verbs vs nouns vs adjectivesCan see how does derivative knowledge developCan see at what stage can learners use systemic knowledge e.g. inflectionalAllows us to see changes or development over timeAllows us to see patterns in developmentAllows us to look at whole lexicons, not just wordsAny variable (subject to declarative knowledge) can be used on the axes (meaning, use, pronunciation, etc.)Slide17

Issues with State Models

Not suitable for high-stakes

testing

Assumes

subjects have access to declarative knowledge

Unclear what math to use for

analysis (to me at least!)

Adding levels to get finer detail leads to

massive increases in data needed for reliability

a

need for clear labels for each state

a three state model is crude (I don’t know, I think I know, I know)

a 7 state model is too fine (I don’t know, ?. ?, ?, ?, ?, I know perfectly)

Polygraphs need careful attention (the various meanings of

bank might need contextualizing)Labels for states determine what you are testingI can use it vs. I can use it in a sentence vs. I can use it in speechSlide18

Issues with State Models

Hard to do for listening

We

may need to adjust the data for accuracy of

reporting

Hard to validate self-reports

-Can

use non-words to validate reports (

splonk

,

merd

,

thyde

)

-Will need to validate any test instrument with a pilot population before mass-usee.g. Give oral check (e.g. m/c, translations) test with pilot populations to validate their rating of say state 2, is actually state 2Slide19

Issues with State Models

Need to validate knowledge reports are not random

Give subjects several tests including a subset of test A words in test B a

few days

apart

P

ilot the test instrument with some subjects first. We should find most data are

orange

Same knowledge

3

2

1

0

0

1

2

3

t1

t2Slide20

Questions for you…What other ways could a state model of vocabulary be used?

Is there an application in your own

area?

What math would be appropriate to use on these data?Slide21

Thanks for your time!

Rob Waring

w

aring.rob@gmail.com