AutoLearn’s authoring tool € A piece of cake for teachers - PowerPoint Presentation

AutoLearn’s authoring tool€ A piece of cake for teachers Martí QuixalFundació Barcelona Media – Universitat Pompeu FabraCo-authors: Susanne Preuß, Beto Boullosa and David García-NarbonaJoint work with Toni Badia, Mariona Estrada, Raquel Navarro, John Emslie, Alice Foucart, Mike Sharwood Smith, Paul Schmidt, Isin Bengi-Öner and Nilgün Firat€ Research funded by the Lifelong Learning Programme 2007-2013 (2007-3625/001-001) 1

OutlineMotivation and goal A tool for authoring ICALL materialsUsing and evaluating of AutoTutorConcluding remarks2

OutlineMotivation and goal A tool for authoring ICALL materialsUsing and evaluating of AutoTutorConcluding remarks3

Motivation: ICALL’s ironyICALL figures 119 projects from 1982 to 2004 (Heift & Schulze, 2007)Half a dozen ICALL systems are continuously used in real-life instruction settings (Amaral & Meurers, submitted; Heift & Schulze 2007)Crucial aspectsAppropriate integration in the learning context (Levy 1997, 200-203)Successfully restrict learner production in terms of NLP complexity (Amaral and Meurers, submitted)4

Motivation: FLTL requirementsICALL meeting FLTL Feedback has to be coherent with syllabus, teaching approach, and activity goals (focus on form/content)Ideally has to respond to real-life needs (Amaral 2007)5

Motivation: involve FLTL practitionersFLTL experience Integration of out-of-class work thoughtfully and coherently designed – with the needs of the learner in mind (Levy and Stockwell, 2006: p. 11 – 12)CALL traditionAuthoring tools have a long tradition in CALL, but practically inexistent for ICALL (Levy 1997, chap. 2, Toole and Heift 2002)6

Goal: one NLP response to shape language technology to the needs of the teachers (and learners) byallowing for feedback generation focusing both on form and on contentproviding a tool and a methodology for teachers to author/adapt ICALL materials autonomously7

OutlineMotivation and goal A tool for authoring ICALL materialsGeneral functionalities (and GUI)Answer specificationsNLP-resource generationUsing and evaluating of AutoTutorConcluding remarks8

Context: AutoLearn projectGoalsIntegrate existing ICALL technology in Moodle (ALLES project, Schmidt et al. 2004, Quixal et al. 2006)Evaluate such an FLTL paradigm for different learning scenarios ParticipantsFundació Barcelona Media Universitat Pompeu Fabra, Barcelona (coord., NLP-based applications, HCI)Institut für Angewandte Informationsforschung, Saarbrücken (NLP)Heriot-Watt University, Edinburgh (FLTL, SLA)Bogazici University, Istanbul (FLTL practice)9

AutoTutor: ICALL for MoodleDefinitionA web-based software solution to assist non-NLP experts in the creation of language learning materials using NLP-intensive processing techniquesTeacher perspectiveA tool to create, manage, and track ICALL activities including (NLP-based) feedback on form/contentLearner perspective To do the exercises and get feedbackTo track own activity10

AutoTutor: architecture and process 11

AutoTutor: functionalities (I) Teacher perspective ERROR MODEL ANSWER MODEL 12

AutoTutor: functionalities (II) Teacher perspective13

AutoTutor: functionalities (III)14Teacher perspective

AutoTutor: doing activities Learner perspective15

AutoTutor: immediate feedbackLearner perspective16

Step two: specific language checkingStep one: general language checkingShort () on the NLP side17

ATACK: answer specification (I)A questionDefine the ecological footprint in your own words.Possible answersThe ecological footprint relates to the impact of human activities on our environment. It is an indicator that measures the surface needed to produce our resources and absorb the waste we generate.18

ATACK: answer specification (II)Divide answers into blocks (or chunks)The ecological footprint relates to the impact of human activities on our environment. It refers to an indicator that estimates the surface needed to produce our resources and absorb the waste we generate.19Divide answers into blocks (or chunks)

ATACK: answer specification (III) BB1B220

ATACK: answer specification (IV) AAB1B2E D E 21 C

ANSWER MODEL ATACK: NLP resource generationERROR MODEL22Teacher GUI

ATACK’s underlying NLP strategyTechnical aspectsProcessing is a pipeline (no stand-off annotation)Shallow template-based (info) chunking using both relaxation techniques and buggy rulesKURD: constraint-based formalism – Finite State Automaton enhanced with unification (Carl and Schmidt- Wigger 1998)23

ATACK: underlying NLP formalism (I)24 Description partAction partMarkersVariablesQuantifiers Operators

ATACK: underlying NLP formalism(II)two-level analysis 25B2a1a2... a1 ... a2 ... a1 a2 ... a1 ... a2 ... b1 b2 ... b1 ... b2 ... b1 b2 ... b1 ... b2 ... A B1 C D1 D2 E

“an indicator” LemmaStuffMissing = ?$-1a{tag~:c1_}, ^Aa{ lu = a;the ,disc =_}, Ba { lu = indicator ,disc =_} ::: Ar {disc= C,tag =b10}g{ flaga = small,flage = an }, Br{disc= C,tag =b10}g{ flaga = small;det_a , flagb = noun_sg,flage = indicator }, j(rule=@end10). ATACK : “ info ” chunker Word- level Word level with extra- stuff Lemma-level Lemma-level with extra stuff Lemma-level with stuff missing Some “ key ” words ( concept words ) For each “info”-chunk a corresponding analysis rule is created 26

ATACK: “global” well- formednessBlock order combinations are checked for (deviant structures inc.)ori_chunked_gap_no_need= ?$-1a{chunked=a_G_A_C2_D2_F_E_;a_G_A_C1_D1_D1_E_; a_G_A_H_B1_C2_D2_F_E_;a_G_A_H_B1_C1_D1_D1_E_; a_C2_D2_F_E_;a_C1_D1_D1_E_}, +Aa{flagc~=_,lu~=@;&at,snr~=1000}e{flagc=g,style=no_need} ::: Ar{style=no_need,bstyle=no_need,estyle=no_need}, $-1g{style=no_need}.27Correct block orderCorrectness in within block“ Blended ” structures Missing blocks

OutlineMotivation and goal A tool for authoring ICALL materialsUsing and evaluating of AutoTutorConcluding remarks28

AutoLearn: testing in real-life (I) Pre-PhaseRecruitingTwo workshops with over 60 participantsSystem usageMaterial development (training plus development)Material selectionTesting action preparation (book PC-labs, etc.)Analysis of testing actionQuestionnaires for learnersQuestionnaires and interviews with teachers29 78% would only use ICALL materials if ready-made.

AutoLearn: testing in real-life (II) Participants in testing3 universities: 5 different classes (EN, DE)7 secondary school teachers: 10 classes (EN)5 language school teachers: 5 classes (EN)30

AutoLearn: training ICALL developers4-hour course (2 sessions), plus 4 control meetings (and individual work)How to plan, pedagogically speaking, a learning sequence including ICALL materials?What can NLP do for you?How do you use ATACK’s GUI?31

Learnt from cooperation with teachersDesigning FLT materials knowing in advance that they will be part of an ICALL system is more difficult than selecting activities from books The notion of timeThe notion of spaceThe lack of expertise in using ICALL/NLP results into overdemanding or not challenging NLP tasks32

Learning to restrict NLP complexity (I) 33Which is your attitude concerning responsible consumption? How do you deal with recycling? Do you think yours is an ecological home? Are you doing your best to reduce your ecological footprint? Make a list of 10 things you could do to reduce, reuse or recycle your waste at home.

Learning to restrict NLP complexity (II) Which is both the challenge and the opportunity of managing our waste?If we do not recycle the stock of aluminium and steel in our society, where would they come from?What consequence has the 1994 packaging directive on people’s behaviour?For which two types of products have hazaradous substances been prohibited in their production?What should we require from Europe to become a recycling society?34

Analysing AutoTutor’s performance Activity type: reading comprehensionQ1: Explain in your words what the ecological footprint is.Q2: What should be the role of retailers according to Timo Mäkelä?35QuestionInv. Tot 1st 2 73 2nd 21 100

Building a “gold standard” Out of 173 manually reviewed attemptsQuestionCorr.Part. Incorr. Inv. Tot 1st 36 23 12 2 73 2nd 14 29 36 21 100 36

Quantitative analysis (accuracy) MESSAGESREAL ERRORSPERCENTAGE Form Cont Form Cont Form Cont CORRECT ANSWERS 31 139 15 71 48,4 51,1 PARTIALLY CORRECT 8 84 7 42 87,5 50 INCORRECT ANSWERS 41 30 39 18 95,1 60 MESSAGES REAL ERRORS PERCENTAGE Form Cont Form Cont Form Cont CORRECT ANSWERS 6 45 8 20 100 44,4 PARTIALLY CORRECT 29 110 18 57 62,1 51,8 INCORRECT ANSWERS 20 93 21 77 100 82,8 37

Main causes of misbehaviour MISBEHAVIOURPHASE 1PHASE 2Connection failed1 0 Bad use of the system 1 1 System misleading learner 4 2 False positive (L1-driven, OOV) 22 33 Inappropriate focus on form 35 21 Artificial separation of messages 0 61 Poor specifications 1 62 TOTAL 64 180 38

Misbehaviour in formal aspects39 Inappropriate focus on form“Rare” entries

Artificial separation 40

Poor specifications41 Semantic extension of answerSyntactic flexibility

OutlineMotivation and goal A tool for authoring ICALL materialsUsing and evaluating of AutoTutorConcluding remarks42

Conclusions: improve coverage To reduce the effects of poor specificationsSupport material designers with semantic driven techniques for expansion of their possible answersAdd functionalities to teacher interface to easily extend exercise models or specific feedback messages using learner answers inappropriately handled by the system43

Conclusions: improve accuracyTo reduce false positivesAdapt general (non-customizable) NLP resources to better handle L2 learner profilesTo reduce the effects of artificial separationBetter exploit the information provided by teachers in the block definition processUse a parser that allows for the grouping of syntactic or/and informational units 44

Conclusions: general messageIt was Feasible to overcome ICALL’s ironyPossible to meet some FLTL requirementsIncredibly useful to involve real-life teachers and learners in testingNLP developers have to work closely together with FLT trainers45

Martí Quixal (marti.quixal@barcelonamedia.org) Fundació Barcelona Media – Universitat Pompeu FabraDiagonal 177, planta 10E-08018 BarecelonaAcknowledgements: thanks to Holger Wunsch, Ramon Ziai and Detmar Meurers for their very useful comments on a rehearsal of this presentation Thanks for your attention! Questions or remarks ? http://autolearn.barcelonamedia.org/ http://parles.upf.edu/autolearn/ http://parles.upf.edu/autolearnTutorKit 46

ReferencesAmaral 2007 Amaral & Meurers, submitted J. Burstein, S. Wolff, and C. Lu, Using lexical semantic techniques to classify free-responses, 227–244, in Breadth and depth of semantic lexicons, ed. by Evelyne Viegas, Kluwer, Dordrecht, 1997.(Carl and Schmidt-Wigger 1998)Kathleen Graves. Designing Language Courses: A Guide for Teachers. Boston, MA: Heinle & Heinle, 2000.(Heift 2001) do they read itHeift & Schulze 2007Heift 2003(Levy 1997, 200-203)Levy and Stockwell, 2006(Pujolà 2001, 2002) Quixal 2006 Toole and Heift 2002Schmidt 2004Ziai 200947

ReferencesFrom Graves 2000 (p 149)When teachers are required to strictly adhere to a textbook and timetable there is little room for them to make decisions.On the other hand, the majority of teachers are not paid or do not have the time in their schedules to develop all the materials for every course they teach.48

AutoLearn’s authoring toolA piece of cake for teachers Martí Quixal (marti.quixal@barcelonamedia.org)Some more slides (bonus-track)49

Conclusions: user behaviourLearners do not always go through the two correction stepsEven if they do not read it, what else can we do for them thanks to NLP techniques? (Heift 2001)Layered feedback (Pujolà 2001, 2002)Adaptive feedback (Heift 2003, Ziai 2009)Intelligent visual feedback (?)We want the teacher to monitor and enhance system behaviour, but also to overview the learning process!50

Motivation: real life experiencesReal-life TeachersThe teacher role can be either very passive or very active. None of them is desirable.“The majority of teachers are not paid or do not have the time in their schedules to develop all the materials for every course they teach.” (Graves 2000, p. 149)51

AutoTutor: functionalities (#) Teacher perspective52

AutoTutor: working process ATACKATAP53

ATACK: underlying NLP formalism (II)In ALLES KURD was extended:(Schmidt et al. 2004, Quixal et al. 2006)Text level structure, where nodes are sentences, instead of words, and rules can be applied to feature bundles associated with sentences (info percolation procedure)jump operator: so that rule order application could be controlled externally54

ATACK: “info” chunkerFor each “info”-chunk a corresponding analysis rule is createdthat estimatesorigaprule_B1_8 = Aa {ori=that;That}, ^Za{ori~=estimate}, *Za{ori~=estimate}e{c=adv|adj}|{c=v,typ=prt}, Aa{ori=estimate} ::: Ar{chunked=B1}d{flag=***},Zg{flagc=g},j(rule=@end2). 55

ATACK: exercise-specific lexicon For each unknown word a rule is created to avoid a false alarmUnknown = Aa{lu=Ecolabel}e{problem=unknown} : Ad{problem=unknown}. 56

ATACK: error modelling For each trigger/message pair a corresponding rule is created“a measure that measures”  “This is correct…”match_lit = Aa{ori=a}e{gram=gramprop}, Aa{ori=measure}e{gram=gramprop}, Aa{ori=that}e{gram= gramprop }, Aa { ori = measures }e{gram= gramprop } ::: Au{style= gramprop,gram = gramprop }r{ gramprop = This is correct but check whether it is stylistically appropriate. }, $-1g{style= gramprop }. 57

Results from teacher work (I) 58

Context: AutoLearn main activitiesYear 2008Migration and adaptation to new platformAdaptation of contents to new scenariosTesting with ca. 250 learners Year 2009Analysis of first testing phaseDevelopment of an ICALL authoring toolTesting with ca. 400 learners and 6 teachers creating materialsWork dissemination in FLTL practitioners agorae59

Reviewed teacher work (I) 60