UsingKnowledgeComponentModelingtoIncreaseDomainUnderstandinginaDigital - PDF document

1 UsingKnowledgeComponentModelingtoIncreas
UsingKnowledgeComponentModelingtoIncreaseDomainUnderstandinginaDigitalLearningGameHuyNguyenCarnegieMellonUniversityhn1@andrew.cmu.eduYeyuWangUniversityofPennsylvaniayeyuw@upenn.eduJohnStamperCarnegieMellonUniversityjstamper@cs.cmu.eduBruceM.McLarenCarnegieMellonUniversitybmclaren@cs.cmu.eduABSTRACTKnowledgecomponents(KCs)denetheunderlyingskillmodelofintelligenteducationalsoftware,andtheyarecrit-icaltounderstandingandimprovingtheecacyoflearningtechnology.Inthisresearch,weshowhowlearningcurveanalysisisusedtotaKCmodel-onethatwascreatedafteruseofthelearningtechnology-whichcanthenbeimprovedbyhuman-centereddatasciencemethods.Wean-alyzeddatafrom417middle-schoolstudentswhousedadigitallearninggametolearndecimalnumbersanddecimaloperations.Ourinitialresultsshowedthatproblemtypes(e.g.,orderingdecimals,addingdecimals)capturestudents'performancebetterthanunderlyingdecimalmisconceptions(e.g.,longerdecimalsarelarger).ThroughaprocessofKCmodelrenementanddomainknowledgeinterpretation,wewereabletoidentifythedicultiesthatstudentsfacedinlearningdecimals.Basedonthisresult,wepresentanin-structionalredesignproposalforourdigitallearninggameandoutlineaframeworkforpost-hocKCmodelinginatu-toringsystem.Moregenerally,themethodweusedinthisworkcanhelpguidechangestothetype,contentandorderofproblemsineducationalsoftware.KeywordsKCModel,DecimalNumber,DigitalLearningGame1.INTRODUCTIONIntheviewofKCmodeling,student'sknowledgecanbetreatedasasetofinter-relatedKCs,whereeachKCis\anacquiredunitofcognitivefunctionorstructurethatcanbeinferredfromperformanceonasetofrelatedtasks"[22].AKC-basedstudentmodel(whichwerefertoasKCmodel)hasbeenemployedinawiderangeoflearningtasks,suchassupportingindividualizedproblemselection[11],choos-ingexamplesforanalogicalcomparison[35]andtransition-ingfromworkedexamplestoproblemsolving[43].AgoodKCmodelisvitaltointelligenteducationalsoftware,par-ticularlyinthedesignofadaptivefeedback,assessmentofstudentknowledgeandpredictionoflearningoutcomes[24].AnewareaineducationaltechnologythatcouldpotentiallybenetfromKCmodelsisdigitallearninggame.Whiletherehasbeenmuchenthusiasmaboutthepotentialofdig-italgamestoengagestudentsandenhancelearning,fewrigorousstudieshavedemonstratedtheirbenetsovermoretraditionalinstructionalapproaches[32,34].Onepossiblereasonisthatmostdigitallearninggameshavebeende-signedinaone-size-ts-allapproachratherthanwithper-sonalizedinstructioninmind[9].AdoptingKCmodelingtechniquescouldthereforebeanimportantrststepinmeet-ingindividualstudents'learningneedsandmakingdigitallearninggamesamoreeectiveformofinstruction.Acriti-calquestioninthisdirectioniswhetheraKCmodelcanbecreatedaftertheuseofthelearningtechnology,inordertobetterunderstandthetargetedlearningdomainandtohelpinimprovingthetechnology.Inourstudy,weexplorethisquestioninthecontextofagamethatteachesdecimalnumbersanddecimalopera-tionstomiddle-schoolstudents.WestartedwithaninitialKCmodelbasedonproblemtype(e.g.,addingdecimals,completingsequencesofdecimals),thenusedthehuman-machinediscoverymethod[51]toderivenewKCsandfor-mulatethebestttingmodel.Fromthisimprovedmodel,werstdiscussndingsaboutstudents'learningofdecimalnumbersandproposepotentialchangestotheinstructionalmaterialsthataddressawiderrangeoflearningdiculties-aprocessknownas\closingtheloop"[24].Then,weoutlineageneralframeworkforaddingKCmodelstoeducationalsoftwareinapost-hocmanneranddiscussitsbroaderim-plicationsindigitallearninggames.2.BACKGROUNDInthissection,werstpresentbackgroundinformationabouttwoaspectsofstudentmodelingthatarerelevanttoourwork:(1)KCmodeling,atechniquethatrepresentsstu-dents'knowledgeaslatentvariables,and(2)thecurrentstateofstudentmodelingindigitallearninggames.Then, wedescribethegameenvironmentusedfordataanalysis.2.1KCModelingTraditionally,KCmodelshavebeendevelopedbydomainexperts,usingCognitiveTaskAnalysismethodssuchasstructuredinterviews,thinkaloudprotocolsandrationalanalysis[45].Thesemethodsresultinbetterinstructionaldesignbutarealsohighlysubjectiveandrequiresubstantialhumaneort.Toaddressthisshortcoming,awiderangeofpriorresearchhasfocusedoncreatingKCmodelsthroughdata-driventechniques.Someoftheearliestworkoniden-tifyingandimprovingKCmodelswasdonebyCorbettandAnderson[11]withtheearlyLISPtutors.Inthiswork,plottingoflearningcurvesshowed\blips"or\peaks"inthecurveswhichindicatednewKCsthatwerenotaccountedforintheinitialmodel.Byusingacomputationalmodeltotthedatainlearningcurves,[5]showedhowLearningFactorAnalysis(LFA)couldautomatetheprocessofidenti-fyingadditionalKCsineduca

2 tionalsoftware.LFAtakesasinputaspaceofhy
tionalsoftware.LFAtakesasinputaspaceofhypothesizedKCs,whichcanbediscoveredthroughvisualizationandanalysistools[51].Oncethereareseveralhuman-generatedKCmodels,theycanbecom-binedbymergingandsplittingskillsusingmachinelearningtechniquesthataimtoimprovetheoverallt[23].Itisimportanttodeneagoodmodel,butitisnotal-waysclearhowtodoso.Goodnessoftisbestmeasuredbycrossvalidation,butthistechniqueistimeconsumingandcomputationallyexpensiveforlargedatasets.Further-more,thereisnoconsensusonhowcrossvalidationshouldbeperformedoneducationaldata[50].Tworelatedandeasy-to-computemetricsaretheAkaikeinformationcrite-rion(AIC)andBayesianinformationcriterion(BIC),whichaddressovertbymeasuringpredictionaccuracywhilepe-nalizingcomplexity.Ingeneral,alowerAIC/BIC/crossval-idationscoreindicatesabettermodel.Incasetheydonotagree,[50]showedthatAICcorrelateswithcrossvalidationbetterthanBIC,throughananalysisof1,943KCmodelsinDataShop.However,thesescoresalonedonotportraythefullpicture;aspointedoutby[3],manystudentmodelingtechniquesthataimtopredictstudentlearningachieveneg-ligibleaccuracygains,\withdierencesinthethousandthsplace,"suggestingthattheyarealreadyclosetoceilingper-formance.Inresponse,[28]broughtattentiontoanotherimportantcriterion-whetherthemodelisinterpretableandactionable.Astheauthorsargued,evenslightimprovementcanbemeaningfulifitrevealsinsightsonstudentlearningthatgeneralizetoanewcontextandleadtobetter,empiri-callyvalidatedinstructionaldesigns.Forinstance,somere-searchhasbeensuccessfulinredesigningtutorunitstohelpstudentsreachmasterymoreeciently,basedonanalysisofpreviousKCmodels[24,27].Ouranalysisfollowstheestablishedprocessoutlinedabove,inwhichwestartedwithabasichuman-generatedKCmodel,thenidentiedpotentialimprovementsusinglearningcurveanalysis,andevaluatedthenewmodelbyAIC,BICandcrossvalidation.Wealsoderivedinstructionalinsightsfromthismodelastherststepinclosingtheloop.2.2StudentModelinginGamesAspointedoutby[2],knowledgeindigitallearninggamesishardertorepresentthanknowledgeintutoringsystemsbecausethestudents'thinkingprocess,aswellaslearningobjectives,maynotbeasexplicit.Thepopularstudentmodelingtechniquesforlearninggamesarethosethatcanrepresentuncertainty,suchasBayesianNetworks(BN)[31]andDynamicBayesianNetworks(DBN)[8].Forinstance,inUseYourBrainz,byapplyingBNtoeachlevelofthegametoestimatetheproblem-solvingskillsoflearners,re-searcherswereabletovalidatetheirmeasuresofstealthas-sessment[46].[10]appliedDBNinPrimeClimb,amathgameforlearningfactorization,tobuildanintelligentpeda-gogicalagentthatresultsinmorelearninggainsforstudents.Follow-upworkby[30]renedandevaluatedtheexistingDBN,yieldingsubstantialimprovementinthemodel'stestperformancepredictionaccuracy,whichinturnhelpsbet-terestimatestudents'learningstatesinfuturestudies.Asanotherexample,[42]employedDBNtopredictresponsesonpost-testquestionsinCrystalIsland,animmersivenarrative-basedenvironmentforlearningmicrobiology.Recentresearchhasproposedentirelydata-drivenmeth-odsfordiscoveringKCmodelsinatutoringsystem[17,26].However,mostKCmodelsemployedindigitallearninggameshavebeengeneratedmanuallybydomainexperts.Forinstance,inZombieDivision,theKCswereidentiedbymathteachersascommonprimefactorssuchas\dividebytwo"and\dividebythree"[2].Similarly,thedesignersofCrystalIslandlabeledthegeneralcategoriesofknowl-edgeinvolvedinproblem-solvingasnarrative,strategic,sce-nariosolutionandcontentknowledge[42].Therstat-tempttoreneahuman-generatedbaselineKCmodelusingdata-driventechniquesindigitallearninggameswasdonebyHarpsteadandAleven[18].Theirapproach,whichwasappliedtoBeanstalk,agamethatteachestheconceptofphysicalbalance,isbasedon[51]'shuman-machinediscov-erymethod,whichisverysimilartoours;however,therearenotabledierencesinthelearningenvironments.Inparticu-lar,thedomainofdecimalnumbersinvolvesmanymorerulesandoperationsthanBeanstalk'sdomainofbeambalancing;inturn,ourdigitallearninggamealsoincorporatesmoreac-tivities(e.g.,placingnumbersonanumberline,completingsequences,assigningnumberstoless-thanandgreater-thanbuckets).Therefore,ourKCmodelingprocesstakesintoac-countnotjusttheinstructionalmaterialsbutalsoelementsoftheinterfaceandproblemtypes,whichcouldbemoregeneralizabletootherlearningenvironments.2.3ADigitalLearningGameforDecimalsDecimalPointisasingle-playergamethathelpsmiddle-schoolstudentslearnaboutdecimalnumbersandtheirop-erations(e.g.,adding,ordering,comparing).Thegameisbasedonanamusementparkmetaphor(Figure1),wherestudentstraveltovariousareasofthepark,eachwithadif-ferenttheme(e.g.,HauntedHouse,SportsWo

3 rld),andplayavarietyofmini-gameswithinea
rld),andplayavarietyofmini-gameswithineachthemearea,eachtarget-ingacommondecimalmisconception:Megz(longerdecimalsarelarger),Segz(shorterdecimalsarelarger),Pegz(thetwosidesofadecimalnumberareseparateandindependent)andNegz(decimalssmallerthan1aretreatedasnegativenumbers)[21,47].Eachmini-gamealsoinvolvesoneofthefollowingproblemtypes:1.NumberLine-locatethepositionofagivendecimalnumberonthenumberline. 2.Addition-addtwodecimalnumbersbyenteringthecarrydigitsandtheresultdigits.3.Sequence-llinthenexttwonumbersofagivense-quenceofdecimalnumbers.4.Bucket-comparegivendecimalnumberstoathresh-oldnumberandplaceeachdecimalina\lessthan"or\greaterthan"bucket.5.Sorting-sortagivenlistofdecimalnumbersinas-cendingordescendingorder. Figure1:Ascreenshotofthemainmapscreen.Ineachthemearea,andacrossthedierentthemeareas,theproblemtypesareinterleavedtoimprovemathematicslearning[41]andintroducevarietyandinterestingameplay.Figure2showsthescreenshotsoftwomini-games-AncientTemple(aSequencegame)andPegLegShop(anAdditiongame).Eachmini-gamerequiresstudentstosolveuptothreeproblemsofthesametype(e.g.,placethreenumbersonanumberline,orcompletethreenumbersequences).Stu-dentsmustanswercorrectlytomovetothenextmini-game;theyalsoreceiveimmediatefeedbackabouttheiranswers.Tofurthersupportlearning,afteraproblemhasbeensolved,studentsarepromptedtoself-explaintheiranswerbyselect-ingfromamultiple-choicelistofpossibleexplanations[7].Apriorstudyby[34]showedthatDecimalPointpromotedmorelearningandenjoymentthanaconventionalinstruc-tionalsystemwithidenticaldecimalcontent.Follow-upstudiesby[37]and[19]thentestedtheeectofstudentagency,wherestudentscanchoosetheorderandnumberofmini-gamestheyplay.Thesestudiesrevealednodier-encesinlearningorenjoymentbetweenlow-andhigh-agencyconditions,but[19]foundthatstudentsinahigh-agencyconditionhadthesamelearninggainswhileplayingfewermini-gamesthanthoseinlow-agency,suggestingthatthehigh-agencyversionledtomorelearningeciency.Post-hocanalysesby[52]examinedthedierentmini-gamesequencesplayedbyhigh-agencystudentsandfoundthat,consistentwiththereportsin[19],thosewhostoppedearlylearnedasmuchasthosewhoplayedallmini-games.Thisresultleadstoimportantquestionsabouttherightamountofinstructionalcontenttomaximizelearningeciency.Toanswerthesequestions,wewouldneedamorene-grainedmeasureofstudentlearningusingin-gamedataratherthanexternaltestscores.TheKCmodelingworkpresentedhererepresentstherststepinthisdirection. (a)AncientTemple (b)PegLegShopFigure2:Screenshotsoftwomini-games.2.3.1ParticipantsandDesignWeobtaineddatafromtwopriorstudiesofDecimalPointinvolving484studentsin5thand6thgrade,inallstudyconditions[19,37],andremovedthosestudentswhodidnotnishalloftherequiredmaterials,reducingthesampleto417students(200males,216females,1declinedtorespond).Thestudentsplayedeithersomeorallofthe24mini-gamesinFigure1,dependingontheirassignedagencycondition,asdescribedpreviously.Whenselectingamini-game,stu-dentswouldplaytwoinstancesofthatgame,withthesameinterfaceandgamemechanicsbutdierentquestions.Stu-dentsinthehigh-agencyconditionalsohadthechoicetoplayathirdinstanceofeachmini-gameonce.Insubsequentanalyses,weuseanindexof1,2and3todenotethein-stancenumber,e.g.,AncientTemple1,AncientTemple2andAncientTemple3.Foradetaileddescriptionoftheexperimentaldesignofpriorstudies,referto[19,37].2.4DatasetWeanalyzedstudents'in-gameperformancedata,whichwasarchivedintheDataShoprepository[49]indatasetnumber2906.Thedatasetcoversatotalof613,055individualtrans-actions,whichrepresentactionstakeninthemini-gamesby417studentsinsolvingdecimalproblems.3.METHODS&RESULTSWestartedwiththebaselineKCmodelsderivedfromtwosetsoffeaturesthatDecimalPointwasbuiltupon.TheseinitialmodelsweretusingtheAdditiveFactorsModel(AFM)method[6],andthelearningcurveswerevisualizedinDataShop.AFMisaspecicinstanceoflogisticregres-sion,withstudent-correctness(0or1)asthedependentvari-ableandwithindependentvariabletermsforeachstudent,eachKC,andtheKCbyopportunityinteraction.Itisageneralizationofthelog-lineartestmodel[54]producedbyaddingtheKCbyopportunityterms.Wethenchosethemodelwithbettertandanalyzeditslearningcurves.Eachmodelwasrunon42,637observationstaggedwithKCs.3.1BaselineModelsOurrstbaselinemodel,calledDecimalMisc,consistsoffourKCsthatarethemisconceptionstargetedbythemini-games:Megz,Segz,Negz,Pegz[21].Becauseeachmini-gamewasdesignedbasedonasinglemisconception(KC), wecreatedamodelthatmapseachmini-gamequestiontoitscorrespondingKC.Thesecondmodel,ProblemType,insteadmapseachmini-gamequestiontoitsproblemtype(oneofNum

4 berLine,Addition,Bucket,SortingandSequen
berLine,Addition,Bucket,SortingandSequence).Table1showsthetstatisticsresultsofthesetwomodels.Table1:Fitstatisticsresultsofthetwobaselinemodels.RMSEindicates10-foldcross-validationrootmeansquarederror,stratiedbyitem.Val-uesthatindicatebesttareinbold. Model(#ofKCs) AIC BIC RMSE DecimalMisc(4) 30,699.27 34,379.97 0.3292 ProblemType(5) 29,504.09 33,202.12 0.3231 Ascanbeseen,ProblemTypeoutperformsDecimalMiscinallthreemetrics-AIC,BICandRMSE.Inotherwords,theactualproblemtypescapturestudents'learningbetterthantheunderlyingmisconceptions.Insubsequentanalyses,wethereforefocusedonimprovingtheProblemTypemodel.TherststepisidentifyingpotentialimprovementsinthelearningcurveofeachKC.Ingeneral,agoodlearningcurveissmoothanddecreasing[51].Smoothnessindicatesthatnostepismuchharderoreasierthanexpected,andadecreasingcurveshowsthatstudentswerelearningwellandthereforemadefewererrorsatlateropportunities[36].FromFigure3,weobservedthatthelearningcurvesofNum-berLineandBucketarereasonablygood.ThelearningcurveofAdditionstaysatroughlythesamelowerrorratethroughout(10%),buttherearesuddenpeaks,suggest-ingthatsomeproblemswereharderthanothersandthusshouldberepresentedbyaseparateKC.ThelearningcurveofSequencedecreasesbutnotsmoothly;thezigzagpatternindicatesthatstudentswerealternatingbetweeneasyandhardproblems.Again,havingseparateKCsforthelowsandhighsofthecurvewouldlikelyyieldabettert.ThelearningcurveofSortingisneitherdecreasingnorsmooth;therefore,thisKCneedstobefurtherdecomposed.3.2ImprovedKCModels3.2.1KCdecompositionTondpossibledecompositions,wefollowedthehuman-machinediscoverymethodoutlinedin[51]andconsultedpriorliteratureonstudents'learningofdecimalnumbers.Belowwepresentouranalysisofeachproblemtype.NumberLine.Asitslearningcurveisalreadygood,weturnedtorelatedworkonthegameBattleshipNumberline[29],wherestudentshavetoplacegivenfractionnumbersonanumberline.Theauthorsfoundthat,onanumberlinethatrunsfrom0to1,studentshavebetterunderstandingwhenadjustingfrom0or1(e.g.,1/10or9/10)thanfrom1/2(e.g.,3/7).Sincedecimalnumberscanbetranslatedtofractionsandviceversa,we(tentatively)experimentedwithapplyingthendingsof[29]toourmodel.Inparticular,wedecomposedtheNumberLineKCintoNumberLineMid(thenumbertolocateliesbetween0.25-0.75)andNumberLineEnd(thenumbertolocateliesbetween0-0.25or0.75-1). Figure3:LearningcurvesoftheKCsinProblemType.Thex-axisdenotesopportunitynumberforeachKCandy-axisdenoteserrorrate(%).Theredlineplotsalloftheactualstudents'errorrateateachoppor-tunity,whilethebluelineisthecurvetbyAFM.Addition.ThereareeightitemsinanAdditiongame:fourtextboxesforcarrydigits-carryTens,carryOnes,carry-Tenths,carryHundredths-andfourtextboxesfortheresult-ansTens,ansOnes,ansTenths,ansHundredths(seeFigure2bforanexample).Previously,alloftheseitemshadthesameKClabelofAddition,butweexpectedthatsomedig-itswouldbehardertocomputethanothers.Forinstance,thecarryHundredthsdigitisalways0,becauseourprob-lemsonlyinvolvenumberswithtwodecimalplaces.Ontheotherhand,becausethefocusofAdditionproblemsistotestthatstudentscancarryfromthedecimalportiontothewholenumberportion(i.e.,probingforthePegzmiscon-ception),thecarryOnesdigitisalwaysexpectedtobe1.ItwasindeedthecasethatcarryOnes,alongwithansOnes,ac-countsforalargeportionofthepeaksinAddition'slearningcurve(Figure3).Themostcommonerrorinthesepeaks,however,comesfromcarryTensandansTensinthemini-gameThirstyVampire1.Forthemajorityofstudentsinoursample(87.5%),ThirstyVampire1wastherstAddi-tionproblemtheyencountered,anditsquestion(7.50+3.90)wasalsotheonlyonewithacarryinthetensplace;inotherwords,itwasboththerstandhardestquestion.Forthisreason,wedecidedtodecomposetheAdditionKCinto:Addition_Tens_NonZeroappliestothecarryTensandansTensiteminThirstyVampire1.Addition_OnesappliestocarryOnesandansOnesinallAdditionmini-games.Otheritems(e.g.,carryTenths,carryHundredsansTenths)retaintheKClabelAddition.Sequence.InaSequencemini-game,studentshavetoenterthelasttwonumbersinanincreasingarithmeticse- quence,basedonthepatternoftherstthreegivennumbers(e.g.,Figure2a).Inthewaythequestionsweredesigned,therstnumbertollinalwaysrequiresanadditionwithcarry,whereastheseconddoesnotinvolveacarry.Wethereforehypothesizedthattherstnumberismoredi-cultthanthesecond,whichwasconrmedbyinspectionofthelearningcurve:thealternateupanddownpatternsdepictstudents'errorratesastheylledintherstandsecondnumberineachsequence.Wefurtherdistinguishedbetweennumberswithtwodecimaldigitsandthosewithone,astheformershouldbemorediculttoworkwith.Insummary,wedec

5 omposedtheSequenceKCintofourKCs:Sequence
omposedtheSequenceKCintofourKCs:Sequence_First_OneDigit(rstnumber,withonedecimaldigit),Sequence_First_TwoDigits(rstnumber,withtwodecimaldigits),Sequence_Second_OneDigit(secondnum-ber,withonedecimaldigit),Sequence_Second_TwoDigits(secondnumber,withtwodecimaldigits).Bucket.AsthelearningcurveofBucketisalreadygood,wedidnotfurtherdecomposethisKC.Sorting.ThelearningcurveofSortingremains atatarounda25%errorrate.Sincetherearenooutstandingblipsorpeaksinthiscurve,weinsteadusedDataShop'sPerformanceProlertooltoplotthepredictedandactualerrorratesofeachmini-gameproblem(Figure4).Weidenti-edvemini-gameproblemsinwhichtheactualerrorratewaslargerthanpredictedbyatleast5%;inotherwords,theseproblemswereharderthanexpected.Therefore,welabeledveofthem-RocketScience1,RocketScience2,JungleZipline2,BalloonPop2andWhacAGopher1-byaseparateKCcalledSortingHard,whileotherproblemsre-mainedinSorting.WewillcharacterizethemathematicalfeaturesoftheseSortingHardproblemsinSection4.2. Figure4:VisualizationoftheSortingKC'sgoodnessoftwithrespecttotenSortingmini-gameswiththehighesterrorrates.Thebars(shadedfromleft)showtheactualerrorratesandthebluelineshowspredictederrorrates.3.2.2Newmodelresult&comparisonTable2showsthetscoresoftheoriginalProblemTypemodel,themodelsresultingfromindividualKCdecompo-sitions,andthenalmodelcombiningalldecompositions,calledCombined.ApartfromProblemTypeandCombined,thenameofeachothermodelindicateswhichoriginalprob-lemtypeKCisdecomposed.Forinstance,theSortingmodelhassixKCs-SortingHard,Sorting,NumberLine,Bucket,Addition,Sequence-wherethelastfourareiden-ticaltothoseinProblemType.WecanthereforeseethatdecomposingtheoriginalSortingKCaloneresultsinade-creaseofAICby231.91andBICby214.59.Table2:Fitstatisticsresultsoftheoriginalandnewmodels,sortedbyAICindescendingorder.Valuesthatindicatebesttareinbold. Model(#ofKCs) AIC BIC RMSE ProblemType(5) 29,504.09 33,202.12 0.3231 NumberLine(6) 29,492.48 33,207.83 0.3233 Sorting(6) 29,272.18 32,987.53 0.3215 Sequence(8) 29,159.27 32,909.25 0.3234 Addition(7) 29,025.77 32,758.43 0.3235 Combined(12) 28,436.07 32,255.34 0.3196 Figure5showstheresultinglearningcurvesoftheabovedecompositions.WeobservedthreeKCswithissues:(1)Sequence_First_TwoDigitsisa atcurvewhichindicatesnolearning,(2)SortingHardremainsathigherrorrates,and(3)Addition_Tens_NonZerohastoolittledata(becauseitonlyappliestoThirstyVampire1).ThreeotherKCs-Addition,Addition_Ones,Sequence_Second_Digits-havelowand atcurves,suggestingthatstudentsalreadymas-teredthemearlyonanddidnotneedasmuchpractice(i.e.,theywereover-practicingwiththeseKCs).TheremainingKCshavesmoothanddecreasingcurves.Mostnotably,wewereabletoxthezigzagpatternintheoriginalSequencecurve,reducethepeaksintheAdditioncurve,andcapturetheSortingproblemsthatdore ectstudents'learning.OtherthanNumberLine,allofthenewmodelsresultedinbetterAICandBICscores.TheCombinedmodel,whichincorporatesalldecompositions,isthebestt;whencom-paredtoProblemType,itsAICscoreislowerby1068.02anditsBICislowerby946.78.UsingDataShop'sPerformanceProlertool,wewerealsoabletovisualizethedierencesbe-tweenthesemodelsinFigure6.HereweseethatforeachofthenewKCs,theCombinedmodel'sprediction,representedbytheblueline(squarepoints),isclosertotheactualerrorratethantheProblemTypemodel'sprediction,representedbythegreenline(roundpoints).Hence,thecombinationofourKCdecompositionsresultedinabettertvisually.4.DISCUSSION4.1ComparisonofBaselineModelsWefoundthattheProblemTypemodel,whichmapsmini-gamequestionstoproblemtypes,isabettertforstudentlearningthantheDecimalMiscmodel,whichmapsmini-gamequestionstounderlyingmisconceptions.Hereweout-linetwopossibleinterpretations.First,whileeachquestionwasdesignedtotestonemiscon-ception,studentsmaydemonstrateothermisconceptionsintheiranswers.Forexample,themini-gameJungleZipline1,labeledasSegz(shorterdecimalsarelarger),asksstudentstosortthedecimals1.333,1.33,1.3003,1.3fromsmallesttolargest.Ananswerof1.3003,1.333,1.33,1.3wouldmatch KCswithissues Lowand atKCs GoodKCs Figure5:LearningcurvesoftheKCsinCombined.Thex-axisdenotesopportunitynumberandy-axiserrorrate(%).Theredlineplotstheactualstudents'errorrateateachopportunity,whilethebluelineisthecurvetbyAFM. Figure6:VisualizationoftheCombinedandProblem-Typemodels'goodnessoftwithrespecttothenewKCs.Thebars(shadedfromleft)showtheactualerrorrates.TheblueandgreenlineshowpredictederrorratesofCombinedandProblemTyperespectively.theSegzmisconception,butweobservedthat25%ofthein-correctanswerswere1.3,1.33,1.333,1.3003,whichinsteadcorrespondst

6 oMegz(longerdecimalsarelarger).Asanother
oMegz(longerdecimalsarelarger).Asanotherexample,themini-gameCaptureGhost1,labeledasMegz,asksstudentstodecideifeachofthefollowingnumbers-0.5,0.341,0.213,0.7,0.123-issmallerorlargerthan0.51.14%oftheincorrectanswersstatedthat0:5�0:51andalso0:341�0:51,whichdemonstratesbothSegzandMegz,re-spectively.Ingeneral,inaproblemsolvingenvironmentlikeDecimalPoint,measuringstudents'misconceptionsshouldbebasedontheiractualanswers,notthequestionsalone.Therefore,aKCmodelthatmapseachquestiontoitshy-pothesizedmisconceptionmaynotcapturethestudents'fullrangeoflearningdiculties.Twoalternativeapproachesusedbyotherresearchfortrackingdecimalmisconceptionsare:(1)measuringthematalargergrainsize,suchaswholenumber,roleofzeroandfraction[14],and(2)usingerro-neousexamplesinsteadofproblemsolvingquestions[21].InthecontextofKCmodeling,wecouldapplyourprocesstoanexistingdatasetofstudentlearningofdecimalnumbersfromerroneousexamples,suchasthedatasetfrom[33].Fromacognitiveperspective,[44]pointedoutthat\dierentkindsofknowledgeandcompetenciesonlyshowupinter-twinedinbehavior,makingithardtomeasurethemvalidlyandindependentlyofeachother."Theauthorsconducted aseriesofstudiestoteststudents'conceptualknowledgeofdecimalnumbersandproceduralknowledgeoflocatingthemonanumberline.Eachstudyemployedfourcommonhy-potheticalmeasuresofeachkindofknowledge,butrevealedsubstantialproblemswiththemeasures'validity,suggestingthatitisdiculttoreliablyseparatetestsofconceptualknowledgeandproceduralknowledge.Inourcontext,thedecimalmisconceptionsre ectconceptualknowledgewhiletheproblemtypesrequireacombinationofbothconceptualandproceduralknowledge.Therefore,dierentiatingprob-lemsbytheirtypescreatesclearerKCdistinctionsthanbytheirassociatedmisconceptions,becausetheformermatchesmorecloselywithstudents'actualperformance.4.2InterpretationoftheNewKCsHerewediscusstheinsightsfromourearlierKCdecompo-sitionresults,usingacombinationoflearningcurveanaly-sesanddomain-specicinterpretations.WhiletheexamplequestionswecitearespecictothoseinDecimalPoint,thendingsaboutstudentlearningareapplicabletoanyothereducationaltechnologysystemindecimalnumbers.NumberLine.Unlike[29],wedidnotobservethatstudentshavemoredicultywithnumberscloseto0.5thanwithnumberscloseto0or1.DecomposingNumberLineintoNum-berLineEndandNumberLineMidresultsinincreasesinBICandRMSE,whichareindicativeofovert.Furthermore,theoriginallearningcurveofNumberLineisalreadysmoothanddecreasing(Figure3),soitisunlikelythatanydecomposi-tionwouldyieldsignicantimprovements.Moregenerally,thisresultsuggeststhatstudentscouldlearntoestimatethemagnitudeofagivendecimalnumberbetween0and1rea-sonablywell,eventhoughtheymayhavedicultywiththeequivalentfractionformintheway[29]reported.Toexplainthisdierence,weshouldnotethatstudentstendnottoper-ceivedecimalsandfractionsasbeingequivalent[47],hencedicultieswithfractionsmaynottranslatetodicultieswithdecimalnumbers.As[12]pointedout,afractiona/brepresentsboththerelationbetweenaandbandthemag-nitudeofthedivisionofabyb,whereasadecimalnumber,withouttherelationalstructure,moredirectlyexpressesaone-dimensionalmagnitude.Therefore,studentsoftenhavehigheraccuracyinestimatingdecimalnumbersthanfrac-tionsonanumberline[53].Thendingsfromouranalysisand[29]furthersupportthisdistinction.AdditionandSequence.Theseproblemtypesbothin-volvecomputingthesumoftwodecimalnumbers,andasourdecompositionsshowed,thedicultyfactorliesincar-ryingdigitstothenexthighestplacevalue.InthecaseofAddition,therstquestion,whichalsohappenstobethemostchallenging,istoadd7.50and3.90,whichrequirestwocarries,onetotheonesplaceandonetothetensplace.Theerrorrateisthereforehighestforthisquestion(therstpeakinFigure3),butdecreasesatlater(easier)opportuni-ties.TheoriginallearningcurveofSequenceproblemshasazigzagpatternduetothestudentsalternatingbetweenadditionswithandwithoutcarry.Distinguishingbetweenthesetwotypesofoperations,andalsoonthenumberofdecimaldigits,didresultinabettermodelt.WealsonotethattheerrorratesinSequenceproblemsaregenerallyhigherthaninAdditionproblems.Apossibleinterpretationisthat,whiletheunderlyingadditionoperationsaresimilar,theSequenceinterfacedoesnotlayoutthecarryandresultdigitsindetailastheAdditioninterfacedoes(Figure2).Aspointedoutby[25],foraddingandsubtractingdecimalsofdierentlengths,incorrectalignmentofdecimaloperandsisthemostfrequentsourceoferror.SinceAdditionproblemsalreadysupportedthisalignmentviatheinterface,studentswerelesslikelytomakemistakesinthem.BucketandSorting.Theseproblemtypesbothinvolve

7 performingcomparisonsinalistofvedec
performingcomparisonsinalistofvedecimalnumbers,butindierentmanners.Bucketproblemsrequirecompar-ingeachnumbertoagiventhresholdvalue,whileSort-ingproblemsrequirecomparingthenumbersamongthem-selves.Accordingto[40],orderingmorethantwodecimals(Sorting)couldreveallatenterroneousthinkingwhichmerecomparisonofpairs(Bucket)cannot.Consistentwiththisnding,ourresultsalsoshowedthatstudentswereabletolearnBucketproblemswellbutstruggledwithSorting.OurhypothesisisthataSortingproblemrequirestwoseparateskills:(1)comparingindividualpairsofnumber(inalistofvenumbers,studentsmayperformuptotencompar-isons),and(2)orderingthenumbersonceallthecompar-isonshavebeenestablished.Thecurrentinterfaceonlyasksforthenalsortedlist,soitwouldneedtoberedesignedtoallowfortrackingstudentmasteryofeachofthesetwoskills.Furthermore,byexaminingtheveproblemscatego-rizedasSortingHard,weidentieduniquechallengesthatwerenotpresentelsewhereinDecimalPoint.Firstistheissueofnegativenumber-themini-gameBalloonPop2,withanerrorratecloseto60%(Figure4),asksstudentstosortthesequence8.5071,-8.56,8.5,-8.517indescendingorder.Giventhatstudentsmayholdmisconceptionsaboutboththelengthandsignofdecimalnumbers[21],andthatnootherSortingproblemsinvolvenegativenumbers,itisclearwhystudentsfacedsignicantdicultiesinthiscase.Thesecondissueisanothercommonmisconception-thata0immediatelytotherightofthedecimalpointdoesnotmatter(e.g.,0.03=0.3)-which[39]referredtoasroleofzero.Itcouldbeinvokedinthemini-gameRocketScience1,whichasksstudentstosort0.14,0.4,0.0234,0.323inascend-ingorder;inparticular,19%oftheincorrectanswersput0.0234between0.14and0.323,implyingtheincorrectbe-liefthat0.0234=0.234.Previousstudieshavealsoreportedthat9thgradersandevenpre-serviceteachersdemonstratedthismisconceptioninsimilarsortingtasks[20,38].Further-more,studentsmaystillhavethismisconceptionevenafterabandoningothers[13].Accordingto[24],therearefourstepstoredesignatutorbasedonanimprovedcognitivemodel:(1)resequencing,(2)knowledgetracing,(3)creatingnewtasks,and(4)changinginstructionalmessages,hintandfeedback.Basedonthisframeworkandouranalyses,wederivedthefollowinglessonsfordesigninginstructionalmaterialsinourdigitallearninggameandothertutoringsystemsindecimalnumbers:1.ArrangetheeasyAdditionproblems(withoutorwithonecarry)atthebeginning.Thenumberoftheseeasyproblemscanalsobereduced,asoverpracticeisal-readyoccurringbasedonthenumberofproblemsstu-dentsareattemptingwithlowerrorrates.2.DesignmoreAdditionproblemswithvaryingdicul-ties(thosewithmorecarriesaremoredicult)and positiontheminincreasingorderofdiculty.3.LeavetheoperandeldsblankinAdditionproblemssothatstudentscanpracticealigningdecimaldigits.GettingfeedbackonthisalignmenttaskcouldinturnhelpthemsolveSequenceproblemsbetter.4.ProvidemorescaoldinginSortingproblems,byrstaskingstudentstoperformpairwisecomparisonsofthegivennumbers,thenhavingthemplacethenumbersinorder.Thersttaskcanbeusedtotrackmiscon-ceptionsandthesecondtotracktheskillofordering.5.DesignquestionsinotherproblemtypesbesidesSort-ing(e.g.,NumberLine,Bucket)thataddresstheroleofzeromisconception,asitmaybestrongerandpersistlongerthanothermisconceptions.4.3AdvantagesofPost-hocKCModelingWhile,ingeneral,KCmodelingmethodscanbeappliedtoanydomain,domainknowledgeisstillcriticalfortheinter-pretationoftheimprovedmodelsandanunderstandingofthenewlydiscoveredKCs.Wehaveshownthatwecanapplymethodsinapost-hocmannertoadatasetinaneducationaldomaintobothachieveabetterunderstandingandcreateabetterttingKCmodel.OurndingsalsodemonstratethatthetypeofKCmodelingweusedcanhelpguidechangestothetypes,contentsandorderofproblemsthatareusedinadecimallearninggame(andeducationaltechnologymoregenerally).Fromatheoreticalperspective,thesearchspaceforaKCmodelinagivendomainwillbesomewherebe-tweenaSingleKCmodel,whereeverysteprepresentsthesameKC,toaUniqueStepmodel,whereeverystephasitsownKC.IfweincludetheoptionoftaggingasinglestepwithmultipleKCs,thespacecouldgetinnitelylarger,butinapracticalsensemulti-codedstepscouldbecombinedtoasingleKCbyconcatenatingtheKCsonagivenstep.Sev-eralautomatedprocesseshavebeenappliedtocreateKCmodelsbysearchingthepossiblespace,suchasQ-Matrixsearch[48],buttheyhavethelimitationofcreatingmodelswithunlabeledskills.Themethodsthatweuseddonotfacethisproblembecausewestartedwithafullylabeledmodelandworkedfromthere.Usingvisualandcomputationalanalysesonthelearningcurves,wewereabletomakeim-provementsbycombiningtheoutputofttingmodelswithdomainknowledge.TheoriginalAdditionKCisanexcel-lentexampleofthisapproa

8 chinaction.Whiletheoverallcurvedidshowad
chinaction.Whiletheoverallcurvedidshowadecliningerrorrate,everyfouropportu-nitieslookedasifthestepsweregettingharder(seeFigure3).Methodologically,thiswasaclearopportunityforim-provementandlikelyafeaturewhereeachsuccessivestepinaproblembecameharder.Sureenough,thiswasthecaseaseachoffourproblemstepsrequiredacarry,andthehardestproblemrequiredtwocarries.Thisisoneexamplewhichdemonstratesthatwewereabletonotonlygetabettert-tingmodel,butalsoattainadeeperdomainunderstanding.4.4FutureWorkInournextstudy,wewillusethebestKCmodelfromthisworkasatestofhowwellitperformswithanewpopula-tionofstudents.Thereisalsopotentialinconnectingourworkwithearlierstudiesofstudentagencyindigitallearn-inggames.Inparticular,[37]and[19]reportedthateventhoughstudentsinthehigh-agencyconditioncouldchoosetoplayanymini-gameinanyorder,theydidnotlearnmorethanthoseinthelow-agencycondition,whoplayedaxednumberofmini-gamesinadefaultorder.[19]speculatedthattheformermightbefocusedonselectingmini-gamesbasedontheirvisualthemes(e.g.,HauntedHouse,WildWest-seeFigure1)ratherthanlearningcontent.Toaddressthisissue,wecouldemployanopenlearnermodel[4]thatdis-playstheestimatedmasterylevelofeachdecimalskilltothestudents,wheretheskillsaretheKCsinourbestmodel.Inthisscenario,weexpectthatstudentswhoexerciseagencywouldbeabletomakeinformedselectionsofmini-gamesbasedonanawarenessoftheirlearningprogress.Atthesametime,digitallearninggamesareintendedtoengagestudentsandpromotelearning.Therefore,wewanttoexploretheinteractionsbetweenenjoymentandlearning,particularlyinhowbesttobalancethem.Justaslearningcanbemodeledbyknowledgecomponents,canenjoymentalsobemodeledby\funcomponents,"andhowwouldtheybeidentied?Webelieveourdigitallearninggameisanexcellentplatformforthisexploration,becauseeachmini-gamehasaseparatelearningfactor(thedecimalquestion)andenjoymentfactor(thevisualthemeandgamemechan-ics).Itisalsopossibletotrackstudents'enjoymenteitherthroughin-gamesurveysorautomatedaectdetectors[1].Asournextstep,wewilldesigntwostudyconditions,onethatemploysatraditionalopenlearnermodelandonethatcapturesandre ectsstudents'enjoyment,usingtheveproblemtypes(wordedinamoreplayfulway,e.g.,ShootinginsteadofSorting,becauseallSortingmini-gamesinvolveshootingobjectssuchasspaceship)astheinitialfuncompo-nents.Findingsfromthisfollow-upstudywouldthenallowustoreneourenjoymentmodelandprovideinsightsintowhetheralearning-drivenorenjoyment-drivengamedesignyieldsbetteroutcomes.InthedirectionofKCmodeling,asmentionedin[19]and[52],itispossiblethatthethegamecontainsmorelearningmaterialsthanrequiredformastery,orthatsomestudentsmayhaveexhibitedgreaterlearningeciencythanothers.WiththeKCmodelidentiedinthiswork,wecanthenapplyBayesianKnowledgeTracing[11]toassessstudents'masteryofeachKCandverifythepresenceoflearningef-ciencyorover-practice.Anotherareaweplantostudyiswhetherindividualdierencesamongthestudentsintheirgameplayandlearningcouldleadtofurtherimprovementinpredictingskillmasterybasedonthebest-tKCmodel,similartopreviousresearchdoneinanintelligenttutorforgeneticslearning[15].Theseindividualdierencescouldbeaccountedforbyotherfeaturesinthegameoutsideoftheidentiedcognitive-denedKCs[16].5.CONCLUSIONPreviousworkhasbeendoneonreningKCmodelsfored-ucationalsystemsinthemannerwehaveshownhere[51],althoughourresearchfocusedontheapplicationofthere-nementtechniquestoadigitallearninggame.WefoundthatmodelingKCsbyproblemtypesyieldsabettertthanmodelingbytheunderlyingmisconceptionsthatwerebeingtested.Furthermore,therenedKCmodelalsoshowedushowtoimprovetheoriginallearningmaterials,inparticularbyfocusingonthemorechallengingandpersistentmiscon-ceptions,suchasthoseinvolvingmultiplecarries,roleofzeroandnegativenumbers.Moregenerally,wedemonstratedhowlearningcurveanalysiscanbeemployedtoperform post-hocKCmodelinginatutoringsystemwithvarioustypesoftask.Inturn,ourworkopensupfurtheroppor-tunitiestoexploretheinteractionofstudentmodelswithlearning,enjoymentandagency,whichwouldultimatelycontributetothedesignofalearninggamethatcanadap-tivelybalancetheseaspects.6.ACKNOWLEDGEMENTSThisworkwassupportedbyNSFAward#DRL-1238619.TheopinionsexpressedarethoseoftheauthorsanddonotrepresenttheviewsofNSF.SpecialthankstoJ.Eliza-bethRicheyandErikHarpsteadforoeringvaluablefeed-back.ThankstoScottHerbst,CraigGanoe,DarlanSantanaFarias,RickHenkel,PatrickB.McLaren,GraceKihumba,KimLister,KevinDhou,JohnChoi,andJimitBhalani,forcontributionstothedevelopmentoftheDecimalPointgame.7.REFERENCES[1]R.Baker,S.Gowda,M.Wixon,J.Kalka,A.Wagner,A.Salvi,V.Aleven

9 ,G.Kusbit,J.Ocumpaugh,andL.Rossi.Sensor-
,G.Kusbit,J.Ocumpaugh,andL.Rossi.Sensor-freeautomateddetectionofaectinacognitivetutorforalgebra.InEducationalDataMining2012,2012.[2]R.Baker,M.J.Habgood,S.E.Ainsworth,andA.T.Corbett.Modelingtheacquisitionof uentskillineducationalactiongames.InInternationalConferenceonUserModeling,pages17{26.Springer,2007.[3]J.BeckandX.Xiong.Limitstoaccuracy:Howwellcanwedoatstudentmodeling?InEducationalDataMining2013.Citeseer,2013.[4]S.BullandT.Nghiem.Helpinglearnerstounderstandthemselveswithalearnermodelopentostudents,peersandinstructors.InProceedingsofworkshoponindividualandgroupmodellingmethodsthathelplearnersunderstandthemselves,InternationalConferenceonIntelligentTutoringSystems,volume2002,pages5{13.Citeseer,2002.[5]H.Cen,K.Koedinger,andB.Junker.Learningfactorsanalysis{ageneralmethodforcognitivemodelevaluationandimprovement.InInternationalConferenceonIntelligentTutoringSystems,pages164{175.Springer,2006.[6]H.Cen,K.R.Koedinger,andB.Junker.Isoverpracticenecessary?-improvinglearningeciencywiththecognitivetutorthrougheducationaldatamining.Frontiersinarticialintelligenceandapplications,158:511,2007.[7]M.T.Chi,N.DeLeeuw,M.-H.Chiu,andC.LaVancher.Elicitingself-explanationsimprovesunderstanding.Cognitivescience,18(3):439{477,1994.[8]C.Conati,A.Gertner,andK.Vanlehn.Usingbayesiannetworkstomanageuncertaintyinstudentmodeling.Usermodelinganduser-adaptedinteraction,12(4):371{417,2002.[9]C.ConatiandM.Manske.Evaluatingadaptivefeedbackinaneducationalcomputergame.InInternationalworkshoponintelligentvirtualagents,pages146{158.Springer,2009.[10]C.ConatiandX.Zhao.Buildingandevaluatinganintelligentpedagogicalagenttoimprovetheeectivenessofaneducationalgame.InProceedingsofthe9thinternationalconferenceonIntelligentuserinterfaces,pages6{13.ACM,2004.[11]A.T.CorbettandJ.R.Anderson.Knowledgetracing:Modelingtheacquisitionofproceduralknowledge.Usermodelinganduser-adaptedinteraction,4(4):253{278,1994.[12]M.DeWolf,M.Bassok,andK.J.Holyoak.Fromrationalnumberstoalgebra:Separablecontributionsofdecimalmagnitudeandrelationalunderstandingoffractions.Journalofexperimentalchildpsychology,133:72{84,2015.[13]K.DurkinandB.Rittle-Johnson.Theeectivenessofusingincorrectexamplestosupportlearningaboutdecimalmagnitude.LearningandInstruction,22(3):206{214,2012.[14]K.DurkinandB.Rittle-Johnson.Diagnosingmisconceptions:Revealingchangingdecimalfractionknowledge.LearningandInstruction,37:21{29,2015.[15]M.Eagle,A.Corbett,J.Stamper,B.M.McLaren,R.Baker,A.Wagner,B.MacLaren,andA.Mitchell.Exploringlearnermodeldierencesbetweenstudents.InInternationalConferenceonArticialIntelligenceinEducation,pages494{497.Springer,2017.[16]M.Eagle,A.Corbett,J.Stamper,B.M.McLaren,A.Wagner,B.MacLaren,andA.Mitchell.Estimatingindividualdierencesforstudentmodelinginintelligenttutorsfromreadingandpretestdata.InInternationalConferenceonIntelligentTutoringSystems,pages133{143.Springer,2016.[17]J.P.Gonzalez-BrenesandJ.Mostow.Dynamiccognitivetracing:Towardsunieddiscoveryofstudentandcognitivemodels.InternationalEducationalDataMiningSociety,2012.[18]E.HarpsteadandV.Aleven.Usingempiricallearningcurveanalysistoinformdesigninaneducationalgame.InProceedingsofthe2015AnnualSymposiumonComputer-HumanInteractioninPlay,pages197{207.ACM,2015.[19]E.Harpstead,J.E.Richey,H.Nguyen,andB.M.McLaren.Exploringthesubtletiesofagencyandindirectcontrolindigitallearninggames.InInternationalLearningAnalytics&KnowledgeConference.Springer,2019.[20]J.HiebertandD.Wearne.Proceduresoverconcepts:Theacquisitionofdecimalnumberknowledge.Conceptualandproceduralknowledge:Thecaseofmathematics,pages199{223,1986.[21]S.Isotani,D.Adams,R.E.Mayer,K.Durkin,B.Rittle-Johnson,andB.M.McLaren.Canerroneousexampleshelpmiddle-schoolstudentslearndecimals?InEuropeanConferenceonTechnologyEnhancedLearning,pages181{195.Springer,2011.[22]K.R.Koedinger,A.T.Corbett,andC.Perfetti.Theknowledge-learning-instructionframework:Bridgingthescience-practicechasmtoenhancerobuststudentlearning.Cognitivescience,36(5):757{798,2012.[23]K.R.Koedinger,E.A.McLaughlin,andJ.Stamper.Automatedstudentmodelimprovement.InternationalEducationalDataMiningSociety,2012.[24]K.R.Koedinger,J.C.Stamper,E.A.McLaughlin,andT.Nixon.Usingdata-drivendiscoveryofbetterstudentmodelstoimprovestudentlearning.InInternationalConferenceonArticialIntelligencein Education,pages421{430.Springer,2013.[25]M.Y.LaiandS.Murray.Whatdoerrorpatternstellusabouthongkongchineseandaustralianstudents:understandingofdecimalnumbers?2014.[26]R.V.Lindsey,M.Khajah,andM.C.Mozer.Automaticdiscoveryofcognitiveskillstoimprovethepredictionofst

10 udentlearning.InAdvancesinneuralinformat
udentlearning.InAdvancesinneuralinformationprocessingsystems,pages1386{1394,2014.[27]R.LiuandK.R.Koedinger.Closingtheloop:Automateddata-drivencognitivemodeldiscoveriesleadtoimprovedinstructionandlearninggains.JournalofEducationalDataMining,9(1):25{41,2017.[28]R.Liu,E.A.McLaughlin,andK.R.Koedinger.Interpretingmodeldiscoveryandtestinggeneralizationtoanewdataset.InEducationalDataMining2014.Citeseer,2014.[29]D.Lomas,D.Ching,E.Stampfer,M.Sandoval,andK.Koedinger."battleshipnumberline":Adigitalgameforimprovingestimationaccuracyonfractionnumberlines.SocietyforResearchonEducationalEectiveness,2011.[30]M.ManskeandC.Conati.Modellinglearninginaneducationalgame.InAIED,pages411{418,2005.[31]J.MartinandK.VanLehn.Studentassessmentusingbayesiannets.InternationalJournalofHuman-ComputerStudies,42(6):575{591,1995.[32]R.E.Mayer.Computergamesforlearning:Anevidence-basedapproach.MITPress,2014.[33]B.M.McLaren,D.M.Adams,andR.E.Mayer.Delayedlearningeectswitherroneousexamples:astudyoflearningdecimalswithaweb-basedtutor.InternationalJournalofArticialIntelligenceinEducation,25(4):520{542,2015.[34]B.M.McLaren,D.M.Adams,R.E.Mayer,andJ.Forlizzi.Acomputer-basedgamethatpromotesmathematicslearningmorethanaconventionalapproach.InternationalJournalofGame-BasedLearning(IJGBL),7(1):36{56,2017.[35]K.MuldnerandC.Conati.Evaluatingadecision-theoreticapproachtotailoredexampleselection.InIJCAI,pages483{488,2007.[36]A.NewellandP.S.Rosenbloom.Mechanismsofskillacquisitionandthelawofpractice.Cognitiveskillsandtheiracquisition,1(1981):1{55,1981.[37]H.Nguyen,E.Harpstead,Y.Wang,andB.M.McLaren.Studentagencyandgame-basedlearning:Astudycomparinglowandhighagency.InInternationalConferenceonArticialIntelligenceinEducation,pages338{351.Springer,2018.[38]I.J.Putt.Preserviceteachersorderingofdecimalnumbers:Whenmoreissmallerandlessislarger!.Focusonlearningproblemsinmathematics,17(3):1{15,1995.[39]L.B.Resnick,P.Nesher,F.Leonard,M.Magone,S.Omanson,andI.Peled.Conceptualbasesofarithmeticerrors:Thecaseofdecimalfractions.Journalforresearchinmathematicseducation,pages8{27,1989.[40]A.RocheandD.M.Clarke.Whendoessuccessfulcomparisonofdecimalsre ectconceptualunderstanding.Mathematicseducationforthethirdmillennium:Towards2010,pages486{493,2004.[41]D.Rohrer,R.F.Dedrick,andK.Burgess.Thebenetofinterleavedmathematicspracticeisnotlimitedtosuperciallysimilarkindsofproblems.Psychonomicbulletin&review,21(5):1323{1330,2014.[42]J.P.RoweandJ.C.Lester.Modelinguserknowledgewithdynamicbayesiannetworksininteractivenarrativeenvironments.InSixthAIandInteractiveDigitalEntertainmentConference,2010.[43]R.J.Salden,V.A.Aleven,A.Renkl,andR.Schwonke.Workedexamplesandtutoredproblemsolving:redundantorsynergisticformsofsupport?TopicsinCognitiveScience,1(1):203{213,2009.[44]M.SchneiderandE.Stern.Thedevelopmentalrelationsbetweenconceptualandproceduralknowledge:Amultimethodapproach.Developmentalpsychology,46(1):178,2010.[45]J.M.Schraagen,S.F.Chipman,andV.L.Shalin.Cognitivetaskanalysis.PsychologyPress,2000.[46]V.J.Shute,L.Wang,S.Grei,W.Zhao,andG.Moore.Measuringproblemsolvingskillsviastealthassessmentinanengagingvideogame.ComputersinHumanBehavior,63:106{117,2016.[47]K.Stacey,S.Helme,andV.Steinle.Confusionsbetweendecimals,fractionsandnegativenumbers:Aconsequenceofthemirrorasaconceptualmetaphorinthreedierentways.InPMECONFERENCE,volume4,pages4{217,2001.[48]J.Stamper,T.Barnes,andM.Croy.Extractingstudentmodelsforintelligenttutoringsystems.InProceedingsoftheNationalConferenceonArticialIntelligence,volume22,page1900.MenloPark,CA;Cambridge,MA;London;AAAIPress;MITPress;1999,2007.[49]J.Stamper,K.Koedinger,R.Baker,A.Skogsholm,B.Leber,J.Rankin,andS.Demi.Pslcdatashop:Adataanalysisserviceforthelearningsciencecommunity.InInternationalConferenceonIntelligentTutoringSystems,pages455{455.Springer,2010.[50]J.Stamper,K.Koedinger,andE.Mclaughlin.Acomparisonofmodelselectionmetricsindatashop.InEducationalDataMining2013,2013.[51]J.StamperandK.R.Koedinger.Human-machinestudentmodeldiscoveryandimprovementusingdatashop.InInternationalConferenceonAIinEducation,pages353{360.Springer,2011.[52]Y.Wang,H.Nguyen,E.Harpstead,J.Stamper,andB.M.McLaren.Howdoesorderofgameplayimpactlearningandenjoymentinadigitallearninggame?InInternationalConferenceonArticialIntelligenceinEducation.Springer,2019.[53]Y.WangandR.S.Siegler.Representationsofandtranslationbetweencommonfractionsanddecimalfractions.ChineseScienceBulletin,58(36):4630{4640,2013.[54]M.WilsonandP.DeBoeck.Descriptiveandexplanatoryitemresponsemodels.InExplanatoryitemresponsemodels,pages43{74.Springer,