Engagement 2.0 Bryant Patten - PowerPoint Presentation

Slide1

Engagement 2.0

Bryant Patten

Director of Learning and Technology

MedU

Slide2

MedU

We build Virtual Patient (VP) cases

501(c)(3) non profit

Hanover, NH

14 people

29,000 new students each year

1,000,000+ cases viewed / year

Slide3

Disclosure

Employee of MedU

Slide4

Slide5

Slide6

By engagement we mean:

" the extent to which students are willing and

able to take on the learning task at hand.”

or

"the learner actively engages in cognitive processes for learning.”

Our focus is cognitive engagement

Slide7

History pre 2013

Tracked time on case and performance

never showed performance

Time on card / all cards not meaningful

Slide8

Why

Committed to continuous improvement of our metrics

Research

Slide9

Engagement Score Algorithm V1.0 2013

Components

answerScore

timeScore

toolbarScore

summaryScore

Equally weighted

Slide10

Summary Statement / Machine Learning

Lightside System

Current version is 0 or 1

Code directly embedded in CASUS display system

Slide11

BACKGROUND

Isolated online environments require learner autonomy and may not inherently foster learner

cognitive engagement

.

Many clerkship directors are using time on case as an indicator of engagement, but empirical evidence suggests this approach is not optimal.

Little is known about the factors in a virtual patient (VP) case that will promote

cognitive engagement

, which we define as “

the degree to which students are willing and able to take on the learning task at hand

.”

OBJECTIVES

To develop and validate a

computer-generated dynamic engagement score

based on student interactions with MedU VP cases.

METHODS

Engagement Score Development

We developed an engagement score that includes

four equally weighted components

based on student interactions with the case, each of which is tracked by the VP software.

A

scoring algorithm

and preliminary

cut-points

for determining low, moderate or good engagement were developed after reviewing log data from 20 randomly selected students.

Engagement Score ValidationContent: Six medical educators were surveyed to establish content validity of the score components. Response process: Four faculty members reviewed log data for 10 cases and scored student engagement as either low, moderate or good. We then assessed rater agreement with the empirically derived scoring cut-points using Pearson correlation, and we assessed inter-rater reliability using intra-class correlation for these ratings.Consequence: We displayed the engagement score to students as a routine aspect of MedU case use.

Development and Validation of an Engagement Metric for Virtual Patient Cases

Norm Berman, MD, Anthony Artino, PhD

Geisel School of Medicine at Dartmouth and Uniformed Services University of the Health Sciences

RESULTSEngagement Score ComponentsTime on page: > 20 secondsMCQ answer accuracy: cumulative percentUse of clinical reasoning toolbar: scaled score (0–12)Summary statement automated analysis and case match: binary score (0, 1)Total Score Values: Red < 0.3; Yellow 0.3–0.5; Green > 0.5Validity EvidenceContent: All educators agreed that the components of the score reflect engagement.Response process  Mean Pearson correlation = 0.98Mean inter-rater reliability = 0.98Consequence: Display of engagement score to students impacts their behavior. Good engagement increased from 72% in week 1 to 86% in week 5.

CONCLUSIONS

A machine-generated engagement metric, based on student actions in a VP case, is feasible. Validity evidence suggests these scores may reflect important aspects of students’ cognitive engagement with the VP cases. IMPLICATIONSThe engagement score appears to be a good indicator of student interaction with MedU cases, and may be better than time on case.The engagement score, as an indicator of cognitive engagement, can serve as an important outcome measure in efforts to improve the design of VPs.The next step in collecting validity evidence for our engagement scores will include correlating these scores with students’ self-reported cognitive engagement using a survey instrument that is currently being validated.

REFERENCES

Artino AR. Think, feel, act: motivational and emotional influences on military students online academic success.

J Comput High Educ (2009) 21:146–166 Berman NB, Fall LH, Smith S, Levine DA, Maloney CG, Potts M, Siegel B, Foster-Johnson L: Integration strategies for using virtual patients in clinical clerkships. Acad Med 2009, 84(7):942–949.

BACKGROUND

METHODS

RESULTS

OBJECTIVES

Engagement

CONCLUSIONS

REFERENCES

IMPLICATIONS

Slide12

Response from educators

Strong favorable response

Some summative use

Request for more student information

Slide13

Response from students

Anecdotal negative reaction when used summatively

Intense interest in algorithm when used summatively

We do NOT tell students score components

Odd combination of Digital Native and Immigrant

Slide14

Performance of students

Close to 90% getting green engagement

80% are getting credit on summary statement

65% correct answers

Slide15

Engagement Score Algorithm V1.5

CORE Components

answerScore

timeScore

clicks (hyperlinks & images)

Equally weighted

Not Yet Displayed

Slide16

Concerns

Students gaming system

Allowing faculty full access via publication

Tension between faculty & students

requiring engagement vs spying on me

Slide17

Engagement Score Algorithm V2.0

Currently Under Development

Components

multi-part answerScore

timeScore

toolbarScore (with semantic analysis)

click tracking

multi-part summaryScore

targeted engagement tools

Slide18

Summary Statement / Machine Scoring V2.0

iParadigms System (turnitin)

Five values of 0, 1 or 2

Accuracy

Narrows DD

Semantic qualifiers

Transformative language

Global summary

Web services model

Slide19

Possible Future Directions

Next Generation Machine Learning

Parameterized Algorithm

Feed our LA work

Standards and Collaboration

Slide20

Engagement Standard

Does one exist?

Is it time to create one?

Slide21

tin can

Slide22

IMS Caliper Analytics™ Interoperability Standards Reach Candidate Final Release Status

- May 6, 2015

Slide23

Slide24

bryant.patten@med-u.org

http://www.med-u.org