Cognitive Task Analysis - PowerPoint Presentation

Slide1

Cognitive Task Analysis

Dick ClarkCenter for Cognitive TechnologyRossier School of EducationKeck School of Medicine University of Southern Californiaclark@usc.edu - www.cogtech.usc.eduPSLC October 15, 2013Slide2

Why the interest in Cognitive Task Analysis (CTA)?

What evidence supports CTA’s use in education?

How is it implemented? Examples? Exceptions?Next steps in research.

Topics

2Slide3

Methods for identifying the cognitive strategies used by

experts and novices to perform complex tasks.Supports decisions on WHAT to teach - not how.Important because of evidence that +/- 70% of expert decisions and many actions are implicit – automated and nonconscious -- in order to circumvent limits on WM.When CTA used to design instruction, 1σ increase in learning and

.5σ decrease in time to learn.

Preliminary evidence of increases in task self-efficacy and persistence (decreased dropout) in higher education courses.

Why

Cognitive Task Analysis?

3Slide4

Brief History of CTA

Recent developments in long history of Task AnalysisGilbreth’s 1890 – 1930 QUERTY keyboard, 3X bricklayingCrandall & Gretchell-Leiter (1993) identified 30% more indicators of distress in premature babies with Klein’s CTA (Crandall, Klein & Hoffman, 2006).Chao & Salvendy (1994) examined four different methods of capturing the strategies experts use for three debugging tasks. Average of 40% procedural steps and 30% explanations

Increased to average of 80% of steps after interviewing 6 expertsCost-benefit diminishes beyond 4 to 6 experts

Why do experts recall different IF – THEN

steps?

4Slide5

Chao & Salvende, (1994)

5Slide6

Decision step recall

increase with more experts

Chao & Salvende, (1997) Figure 4

6Slide7

PhD Students (intermediates) vs. Psychology Faculty

Feldon (2010)

7Slide8

70% Decisions Missing and 4 to 6 Experts to Remedy

Other studies, including partial replications of Chao & SalvendeTrauma Surgeons (Campbell, 2010; Crispen 2010; Sullivan et al, 2011; Velmahos et al, 2006)Psych faculty teaching experimental design (Feldon, 2010)Expert instructors consistently describe 30% of decisions but but about 60% of actions when teaching.With CTA the decisions identified reached 90 to 100% with four to six experts.

Most of our studies focused on surgical procedures because of disputes about “expertise” and surgeon’s legally required to report mistakes. 8Slide9

Variation in SME Action and Decision Steps

(Crispen, 2010 – Cricothyrotomy procedure)

9Slide10

Percent of decisions identified with each new SME

Crispen, 2010; Figure 6

10Slide11

Expert Knowledge Provided During Teaching

Sullivan, Yates, Clark, Green, Tang, Cestero, Plurad, Lam & Inaba (In Press)

11Slide12

Unexpected Result: Controversial CVC Procedure

Figure 4: Yates, Sullivan & Clark (2011)

12Slide13

Exception: Two CTA studies of catheter procedure

Clark, 2014)

13Slide14

CTA in Instructional Design

14Slide15

CTA in Instructional Design

Gucev (2012) randomized double blind experiment on CTA in Ultrasound Guided Regional Anesthesia

15Slide16

Gucev CTA Study Design and Results

Both experimental and control groups:Same tasks and conceptual knowledge required by the American and European Societies of Regional Anesthesia.Same instructional methods (conceptual knowledge first then demonstration and practice).Participants were second and third year medical students.Experimental group received CTA content for Societies tasks and the control group received the approved Societies content and tasks.Results – benefits of CTA on learning and performance over controls:Declarative knowledge effect size d = 1.43 (42%)Procedural knowledge effect size d = 1.65 (45%)Effect size for the time for task performance was d = -1.12 (-37%)

16Slide17

Benefit of Cognitive Task Analysis?

Hoffman (1998) 38% better with CTA – changed textbooks on prenatal infections.Velmahos et al (2002) 35% better surgical decisions, improved transfer, 25% quicker, no important errors.Tofel-Grehl & Feldon (2013) meta analysis (57 comparisons). Hedges g = .88 (31%) overall but g =1.56 (44%) for PARI-type CTA methods and g = .39 (16%) for Klein’s CDM method

.Biology lab course significantly better performance and lower dropout (Feldon et al, 2010; Feldon & Stowe, 2009).

Instruction based on CTA is consistently more effective than Behavioral Task Analysis or

“

self

report

”

.

17Slide18

CTA vs. Traditional Instruction - Biology Lab Reports

Universal Lab Report Rubric CriteriaTreatment Mean(SD)

Control Mean

(SD)

F

p-value

Discussion: Conclusions based on data

Conclusion is clearly and logically drawn from data provided. A logical chain of reasoning from hypothesis to data to conclusions is clearly and persuasively explained.

0.90 (.50)

0.77 (0.48)

4.378

.037*

Discussion: Alternative explanations

Alternative explanations are considered and clearly eliminated by data in a persuasive discussion.

0.43

(0.52)

0.28 (0.44)

6.171

.

014*

Discussion: Limitations of design

Limitations of the data and/or experimental design and corresponding implications discussed.

0.70

(0.63)

0.54 (0.57)

4.703

.

031*

Discussion: Implications of research

Paper gives a clear indication of the implications and direction of the research in the future.

0.31

(0.46)

0.21 (0.40)

3.463

.

064

Discussion: Total Score

2.34 (1.49)

1.78

(1.37)

9.501

.

002**

Feldon et al. (2010); Feldon & Stowe (2009)

18Slide19

Biology 101 Attrition (Withdraw Rates)

CTA Condition

Control Condition

Fisher’s Exact

(2-sided)

2-week Enrollment

142

172

-

Final Enrollment

140

158

-

Overall Dropouts

2

14

p=.005**

Biology Majors

1

3

p=.334

Non-Majors

1

11

p=.010**

Women

1

8

p=.041*

Men

1

6

p=.072

Feldon et al. (2010); Feldon & Stowe (2009)

19Slide20

CTA with Online Faculty at Kaplan University

20

CTA with four of the most effective online faculty teaching intro courses.

P

lan:

Identify the strategies reported by most of the experts interviewed.

Translate

them into a

Likert

-

type

values survey that would be offered to a large random sample of 280 online instructors in different

fields.

“How likely are you to advise a new instructor to use ……?”

Correlate the rankings of the items by individual faculty with their student’s

learning and

retention

data.

Use the items that predicted the greatest success to help hire new

faculty, train

existing faculty and evaluate the

results.Slide21

Results of Kaplan U Survey Based on CTA

21

DROPOUT

: With every .5 increase in survey ranking of items, student dropout decreased 1.6% (a low score of 1 predicts a dropout rate of 41% whereas a score of 5 predicts a significantly lower rate of 29.4%)

GPA:

With every .5 increase in survey ranking of items, GPA increased about .15 points. A score of 3.0 on the survey would predict a GPA of 2.1 whereas a score of 5.0 on the survey would predict a GPA of 2.5.

RETENTION:

Faculty who valued making themselves available by phone, calling students who were not actively participating and who tried to help students recover from problems had an 81% chance of higher retention rates in academic programsSlide22

22

Content based on a CTA of career service advisors with highest placements“Kaplan Way” design and deliveryRandomized controlled study (treatment n: 63; control n: 67)15% improvement in performance (key metric: job placements)

Example: Kaplan Career Services AdvisorsSlide23

Cost of CTA?

Taken from Clark, 2014

23Slide24

What is Cognitive Task Analysis?100 + strategies for capturing the implicit and explicit strategies experts use to perform complex tasks based on Newell & Simon’s “Human Problem Solving” (1972).Goal is to enhance human or machine learning and performance.Four types of CTA processes (Marsha Lovett’s 2x2):

24Slide25

What is Cognitive Task Analysis?Yates (2007) sorts prescriptive CTA methods by outcome: Those that capture declarative (what) and/or procedural (how) and/or Strategic (when) expert knowledge.Our emphasis is on a blending of the three varieties of CTA methods that capture all three types of knowledge identified by Tofel-Grehl & Feldon (2013) meta analysis as the most productive:CDM (Critical Decision Method; Klein et al, 1989).

PARI (Precursor, Action, Result, Interpretation; Hall et al, 1995).

CPP (Concept, Process, Principle, Procedure; Clark, 2014

).

25Slide26

What is Cognitive Task Analysis?Three to six experts selected because they are consistently and recently successful (not simply “experienced”) and NOT instructors. Evidence that each expert has different implicit knowledge about same tasks and that instructors invent “superstitious” steps.Results of interviews corrected by experts and edited into one “gold standard” approach for novices based on maximum efficiency and accuracy. Range of problem examples and performance scenarios are also collected from experts for use in instruction.

Goal is to develop a succinct and accurate procedure (when and how) to perform as basis for demonstrations and practice exercises.

Emphasis on IF – THEN decisions.

26Slide27

What is Cognitive Task Analysis? Six TasksTask 1. Outline sequence of tasks “as performed on the job”If no necessary sequence, teach easier tasks before more difficult tasks.Place prerequisite knowledge first.If safety is an issue – “

Safety first”.

“

In about 30 seconds, describe the actions and decisions you implement to achieve the goal of this task.

Interview experts with recent, consistently successful experience who are NOT full time instructors.

27Slide28

Surgery Task Sequence

Task 4

Introduce intravenous dilator and catheter

Select catheter & choose insertion site

Task 1

Immobilize patient, prepare site

and insert catheter needle

Task 2

Introduce guide wire and

incise skin around wire insertion

Task 3

Prepare lumens and secure line with

non-absorbable sutures

Task 5

28Slide29

Performing substantive

examinations

Issuing communications

or votes (including pre-

examination results)

Re-examining

applications

Examining amendments

Discussing with applicant

Writing further communication(s)

or refusal

Example Course Outline: Examining patent applications

Preparing search

reports

Analyzing

applications

Determining

mean features

of invention

Classifying

applications

Performing

searches

Determining

search

strategies

Using

search

tools

Evaluating

search

results

Writing pre

-

examination results

Determining

claimed subject

matter

Determining

novelty &

inventive steps

Identifying

relevant EPC

requirements

Comparing

documents with

invention

Selecting relevant

documents

Determining described invention

Determining claimed invention

Finding lack of unity

29Slide30

What is Cognitive Task Analysis? Six Tasks

Task 2) For each task, describe clearly enough so that trainees can read and applyContext (Where, When)Condition or Cue (What Starts the task)Sequence of Actions and Decisions (How)Finish this step before going on to step 3 –Tasks or task sequence may change when you see performance stepsCan estimate time required to train at this point

Interview 2-3 experts with recent, successful experience

30Slide31

Task

2: Actions and DecisionsExplain each action in the sequence you perform them Things people do (start with action verbs)Explain each decisionDescribe as “IF” and “THEN” sentences

MOST IMPORTANT: Write steps clearly enough so that a trainee could read and then do what you are describing

.

31Slide32

Start by

deciding

among three sites for catheter placement.IF

the neck is accessible and can be moved, and the head and neck are free of excessive equipment,

THEN

select jugular placement.IF

neck is inaccessible or cannot be moved,

THEN

select subclavian.

IF

the

subclavian veins are

thrombosed and

there is no injury to the IVC,

THEN

select femoral vein placement.

Catheter Placement Steps -Decision Procedure

32Slide33

Catheter Placement Steps

Dilator and catheter insertion for Triple Lumen catheters: Step 13A: Thread the guide wire into the tip of the dilator. Direct the dilator down the wire slowly and through subcutaneous tissue (3 – 4 cm).

33

33Slide34

Patent Examination Procedure Example

34Slide35

What is Cognitive Task Analysis? Six Tasks

Task 3) Collect task-related information about:Supplies and equipment (and location)Performance standards (speed, quality)Common novice performance errorsReasons (Personal Benefits and Personal Risks)35Slide36

Task

4) Identify conceptual knowledge related to procedure:Facts (required statements about anything)Concepts (define new terms – get examples)Processes (how things work)Principles (what causes things to happen) Conceptual knowledge is important IF people must remember something to tell someone else about it – or IF they must apply it to adjust a procedure to solve an unexpected or novel problem

What is Cognitive Task Analysis? Six Tasks36Slide37

Knowledge Types

Presentation During Instruction

Practice and Assessment During Instruction

Type of Information

Example

Objective is to Remember

Proxy for Remember

Objective is to Use or Apply**

Proxy for Use if application is impossible **

Procedure

When to use;

List of action and decision steps

Demonstration of

when and how to perform

Recall when to use; Recall action and decision steps

Reorder steps;

Recall next or missing steps

Decide when to use;

Perform the steps (actions and decisions)

Critique performance or output of actions and decisions

Supportive

Conceptual Knowledge

Fact

Statement of fact

Statement of fact

Recall fact

Recognize fact when presented with distractors

Recall fact in task context

Concepts

(Terms with definitions and example)

List of

defining attributes

Examples and Non—examples of concept

List defining attributes verbally or in writing

Recognize defining attributes when presented with distractors

Identify or generate examples and non-examples

Critique someone else’s identification or generation of examples

Process

(How something works)

List of phases, events and causes at each phase

Examples; simulations of phases, events, and causes

Recall phases, events, and causes

Recognize phases, events, and causes;

Recall missing phases, events, and causes

Identify causes of faults in a process;

Predict events in a process

Critique someone else’s description of causes or prediction of events in a process

Principle

(Cause and effect relationship)

Statement of cause and effect relationship

Examples, demonstration, simulation of cause and effect relationship

Recall the principle

Recognize the principle;

Recall missing elements of the principle

Decide if principle applies;

Predict an effect;

Apply the principle to solve a problem, explain a phenomenon or make a decision

Critique someone else’s application of the principle to solve a problem, explain a phenomenon or make a decision

Knowledge

Integration

Explain the interconnections among conceptual knowledge components, or the conceptual foundation of procedures, or the procedural implementation of conceptual knowledge components

Opportunities (including instructions,

templates, rubrics)

to

self-explain, discuss, present,

describe or select their reasoning about interconnections among knowledge components, for example the principle(s) that justify the application of a procedure.

Knowledge Transfer

Multiple and varied contexts for examples

Multiple and varied contexts for practice

and assessment.

Opportunities for students to explain how they would use the knowledge in other contexts

© 2011 Atlantic Training Inc.

37Slide38

What is Cognitive Task Analysis? Six Tasks

Task 5: Collect five authentic problems trainees will learn to solve One for demonstration during trainingOne for practice and feedbackOne for progress checkTwo for competency tests

38Slide39

What is Cognitive Task Analysis? Six Tasks

Task 6) Give CTA document from SME A to SME B, C, D, E, etc.) to “correct”.Flynn (2013) found reviews of one CTA interview by 3 SMEs more efficient and effective than 4 complete interviews.Develop a

“gold standard” CTA for training and/or job aid development – use language novices will understand.Pull CTA into training design that includes:

Performance objectives and reasons

References to prior knowledge (analogies, examples)

Conceptual knowledge underlying procedureDemonstration of procedure (worked example)

Part and whole task

practice with feedback

39Slide40

CTA Problems and Exceptions

Cannot use Expert-based CTA IF:No experts available and/orNew (novel) tasks, technology, science, processes, orIf “experts” not consistently succeeding at

taskProblems using CTA:Analyst training requires many hours of practice.

“Clients” resist added front end expense of structured interviews and/or have used an ineffective CTA method in the past.

Experts sometimes hold back their “secret sauce” and/or reject the gold standard believing it demeans their skills.

40Slide41

Next Steps in CTA Research

Need to focus research on most effective of the 100+ CTA methods.Clear operational definition of CTA methods.Data mining to extend and/or replace structured interviews.Why are different experts aware of different tasks and steps?Better understanding of how declarative and procedural knowledge interact during task performance (as task elements change).Cost-effectiveness of different types of CTA for instruction.Analysis of why

CTA:Decreases time to learn, Increases self-efficacy, Increases persistence and

Increases transfer.

41Slide42

42Slide43

References

Evidence for most claims and references in this presentation and a review of the research on CTA can be found at:www.cogtech.usc.eduAccess the “Publications” tab

43