Leveraging the talent of hand animators to create three dimensional animation - PDF document

Eurographics/ACMSIGGRAPHSymposiumonComputerAnimation(2009)E.GrinspunandJ.Hodgins(Editors)LeveragingtheTalentofHandAnimatorstoCreateThree-DimensionalAnimationEaktaJain1,YaserSheikh1,JessicaHodgins1;21CarnegieMellonUniversity2DisneyResearch,Pittsburgh AbstractTheskillsrequiredtocreatecompellingthree-dimensionalanimationusingcomputersoftwarearequitedifferentfromthoserequiredtocreatecompellinghandanimationwithpencilandpaper.Thethree-dimensionalmediumhasseveraladvantagesoverthetraditionalmedium—itiseasytorelightthescene,renderitfromdifferentview-points,andaddphysicalsimulations.Inthiswork,weproposeamethodtoleveragethetalentoftraditionallytrainedhandanimatorstocreatethree-dimensionalanimationofhumanmotion,whileallowingthemtoworkinthemediumthatisfamiliartothem.Theinputtoouralgorithmisasetofhand-animatedframes.Ourkeyinsightistousemotioncapturedataasasourceofdomainknowledgeand`lift'thetwo-dimensionalanimationtothreedimensions,whilemaintainingtheuniquestyleoftheinputanimation.Amotioncaptureclipisprojectedtotwodimensions.First,timealignmentisdonetomatchthetimingofthehand-drawnframesandthen,thelimbsarealignedtobettermatchtheposeinthehand-drawnframes.Finallythemotionisreconstructedinthreedimen-sions.Wedemonstrateouralgorithmonavarietyofhandanimatedmotionsequencesondifferentcharacters,includingballet,astylizedsneakywalk,andasequenceofjumpingjacks.CategoriesandSubjectDescriptors(accordingtoACMCCS):I.3.3[AnimationfromMotion/VideoData,AnimationofArticulatedFigures,BelievableMotion,ComputerVisionforAnimation]: 1.IntroductionCharacteranimationistheartofmakinganinanimatechar-actermoveasifitpossessesauniquepersonality[ JT95 ].Thisarthasitsoriginsinthetraditionalhand-drawnstyle[ McC14 ],thoughthereisrichvarietyinthetoolsandtech-niquesusedtoday—sketchinggures,manipulatingpup-pets,andposingcomputer-generatedcharacters.Eachstyleofanimationrequiresthattheartistdevelopadifferentsetofskillstocreateavisuallycompellingendproduct[ Las94 ].AnimatorswhocreatetraditionallyanimatedmoviessuchasSnowWhiteandtheSevenDwarfsandPrincessMononokearetrainedinthemediumofpencilandpaper.Themagicoftheirartisintimatelyconnectedtothetwo-dimensional(2D)medium;thereislittleautomatictransferenceoftheirskilltocomputergraphicssoftware,withitsslidersandin-versekinematics.Althoughnotintuitivetotraditionallytrainedartists,thethree-dimensional(3D)computer-generatedmediumhas Figure1:Aframefromahand-drawnanimationandthecorrespondingthree-dimensionalframe.severaladvantages—itiseasiertomovethecameraaround,varylighting,createshadows,andaddphysicalsimulations.Manyoftheanimatedmoviesreleasedinthepastdecadehavebeenfullycomputer-generatedfeatures[ Wik09 ].Inthispaper,weproposeamethodtoleveragetheuniquetalentofc\rTheEurographicsAssociation2009. EaktaJain,YaserSheikh,JessicaHodgins/LeveragingtheTalentofHandAnimatorstoCreateThree-DimensionalAnimation Figure2:Input:Handanimation.Output:3Danimation.Ourkeyinsightistousemotioncapturedataasdomainin-formation.Wegenerate3Danimationthatmatchesthestyleelements(poseandtiming)ofthehandanimation.traditionalanimatorstocreate3Danimationwhileallowingthemtocontinuetoworkwithpencilandpaper.The2Danimationworkowstartswithananimatorwhosketchesthecharacter'smovementat6-12framespersec-ond.Acleanup/inbetweeningartistcleansuptheanima-tor'slinesanddrawsintheremainingframestogenerate24framesforeverysecondofanimation.Ourgoalistoliftthe2Danimationtothreedimensionsonacomputergeneratedcharacter(Figure 1 )atthisstageinthepipeline.Thiswork-owisquitedifferentfromthetypical3Dcomputergener-atedanimationpipeline,whichstartswithmodelingachar-acter,riggingitforintuitivecontrol,andnallyanimatingit—allwithina3Dcomputergraphicssoftwarepackage.Theproblemoflifting2Danimationto3Dcanbeframedasa3Dreconstructionproblem[ Kan81 , LC85 ]becausein-formationaboutthethirddimensionexistsasamentalmodelintheartist'shead,butisnotdirectlyrepresentedinthedrawing.Reconstructionismademorechallengingbecausetheparametersunderlyingtheartist'sdrawingscanchangefromframetoframe.Thesizeandshapeofthecharacter'slimbsmaychangeintentionallytoconveyinformationaboutmovement(e.g.tension),orunintentionally,asaresultofimprecisionornoiseinthedrawings.Thecameraexistsintheartist'shead,andparameterssuchaseldofviewmayvarythroughoutasequence.Inaddition,handanimationandcomputer-generatedanimationmaybeperceiveddifferentlybyviewers—forexample,JohnLasseterhypothesizesthatholdingaposeforseveralframesisacceptableforahandanimatedsequence,butmakesthemotionlooklifelessin3D[ Las94 ].Mostimportantly,inourexperience,creatingcompellingthree-dimensionalanimationismorethanminimizinganim-agedistancemetric[ BSB07 ],[ SB03 ].Wewanttocapturethestyleoftheartist'shandanimation,andbystylewemeanthenuancesofposeandtimingthatgiveacharacteritsper-sonality.Therearemanycomponentsthatgointoanartist'suniquestyle:squashandstretch,anticipation,followthroughandexaggeration[ JT95 ].Weobservethatsomeofthesestyleelementsarecontainedintheposesoftheanimatedcharacter.Inthispaper,wefocusoncreating3Danimationwhichcaptures(1)theposeofthecharacterineachframe,and(2)thetimingoftheseriesofframes.Wedonotcapturesignicantmodelchangessuchassquashandstretch.Theinputtoouralgorithmisasequenceofhandanimatedframes(Figure 2 ).Ourkeyinsightistoleveragemotioncap-turedataasasourceofdomainknowledgetoaidinresolvingambiguitiesandnoiseinthe2Ddrawings.Weshowthat3Danimationcanbecreatedbymodifyingmotioncaptureani-mationwithstyleelementsfromthehand-drawnframes.Wemodifythetimingofthemotioncaptureposestomatchthehand-drawnposesviadynamictimewarping.Weintroduceatranslationandscaleinvariantposedescriptortocaptureposeinformation.Wepresentresultsonavarietyofhandanimationsincludingaballetsequence,astylizedsneakywalkandajumpingjackssequence.Wealsoevaluateouralgorithmquantitativelyusingamotioncapturesequenceasinput.2.RelatedWorkThegraphicscommunityhasrecognizedthatthemediumofpencilandpaper(orstylusandtablet)isintuitiveinmanyrespects,forexample,torapidlysketchcharactermo-tion[ TBvdP04 ],orkeyposes[ DAC03 ].InSection 2.1 ,wediscusstheworkthatismostcloselyrelatedtooursinthatthe3Dposeisbeingmodiedorreconstructedbasedonanartist's2Dinput.Thecomputervisioncommunityisalsoin-terestedinreconstructingarticulated3Dposefromimagesforsuchapplicationsasmarker-lessmotioncapture,activityrecognitionandgenerationofnovelviewpoints.InSection 2.2 ,wediscussrepresentativeworksthatassumeaknownskeleton,andusemotioncapturedataasdomainknowledgeinreconstruction.2.1.3DAnimationfromaHandAnimator'sInputAnimationresearchershaveattemptedtouseinputfromtra-ditionallytrainedartiststomakemotioncaptureanimationmoreexpressive[ LGXS03 ].Liandcolleaguesstartwithamotioncaptureanimation,andaskanartisttoredrawcer-tainkeyframestomakethemmoreexpressive.Auseralterstheposeandlimblengthsoftheskeletoninthecorrespond-ingmotioncaptureframetomatchthedrawingascloselyaspossible.Thealgorithmthenwarpsthemeshtomatchtheartist'sdrawingandprovidesaseamlesstransitiontoprevi-ousandsubsequentframes.Ourworkattemptstoexchangethehuman-in-the-loopforamoreautomaticmethodthatsetstheskeletalposesforanentirehand-animatedsequence.Davisandcolleaguesgenerateallpossible3Dposesun-derlyingagivenhand-drawnstickgure[ DAC03 ].Theydesignauserinterfacetopresenttheseposesinorderofde-sirability.Theonusofpickingthe`right'3Dposerestsontheuseroftheinterface.Ourworkcomplementsthisworkaswegenerateonlyone3Dposeforeachinputdrawing—theonusisonourmethodtoselectthe`right'3Dposesothatthenalanimationcapturesthestyleoftheoriginalhand-animation.c\rTheEurographicsAssociation2009. EaktaJain,YaserSheikh,JessicaHodgins/LeveragingtheTalentofHandAnimatorstoCreateThree-DimensionalAnimationIn[ BLCD02 ],Breglerandcolleaguestracka2Danimatedcharacterandretargettheanimationontonewcharactersin2Dand3D.Theirmethodrequiresthattheanimatorprovidekeyposes,whichtheiralgorithminterpolatesandwarps.Inthecaseofretargettingtoanew3Dcharacter,theanimatorwillhavetogeneratekeyposesin3D,mostlikelyusingacommercialanimationpackage.Webuilduponthispreviousworktobeabletogeneratealltheskeletalposeswithoutrequiringthattheartistinteractwith3Dsoftware.2.2.3DHumanPoseReconstructionfromImagesWenotethatmuchresearchhasbeendevotedtostudyingtheproblemofreconstructing3Dposefrom2Dimagedata.Anextensivesurveyoftechniquescanbefoundin[ MG01 ]and[ Gav99 ].Alargebodyofworkassumesaknown3Dskeletonandtriestondtheskeletalparametersthatminimizeanimagedistancemetric[ LC85 , SB01 , Tay00 , DCR01 , ST03 , Smi07 ].Notably,Taylor[ Tay00 ],DiFranco[ DCR01 ]andSminchis-escu[ Smi07 ]track3Dhumanposeusingmonocularinputdata.DiFrancoandcolleagues[ DCR01 ]assumethatcam-eraparametersand2Dcorrespondencesareknownforeveryframetobetracked,anddeterminethe3Dposesthatmin-imizetheimageprojectionerrorunderkinematicanddy-namicconstraints.Taylor[ Tay00 ]doesnotassumethattheinputimageswereacquiredusingacalibratedcamera,butasksausertochoosethedesiredposefromthesetofallpos-siblesolutions,foreveryinputimage.Sminchisescu[ ST03 ]estimatesarticulatedhumanmotionusingimagefeatureslikeopticalow,motionboundariesandedgeenergies.Ingeneral,imagesprovidearichsetoffeaturesthatcanbeusedfortrackingrobustlyandhandlingnoise[ SB03 ]—ourinputsarelinedrawings,whichlackthisrichness.Eventhoughreconstructionofhumanposeshasbeendonewithouthumanmotionmodels[ BM98 , DCR01 , CTMS03 ],recentreconstructiontechniqueshavebenetedfromlearningprobabilisticmodelsofhumanmotion—eitherfrommotioncapturedata[ HLF99 , Bra99 , SBF00 , SBS02 , AT04 , AT06 , UFF06 ],or,from3Drangescandata[ BSB07 ].Inparticular,Sidenbladh,BlackandSigal[ SBS02 ]observethatforhighdimensionaldata(suchashumanmotion)itiseasiertomatchtosegmentsinahugedatabase,thanitistomodelthedatabase.Theygeneratea3Dposebypickingasamplefromthedatabaseinsuchawaythatthepreviouslygeneratedposesareagoodmatchtothesample'sprevioushistory.This3Dposecanbeusedasapriorforimage-basedtrackingasin[ SB01 ].Themostimportantdifferencebetweenourworkandtheexistingbodyofresearchisthatthevisioncommunityhasinputthathasatrue3Dinterpretation.Incontrast,theskillofanartistliesinhowcleverlytheycanconveythestyleofthemotion,andtheirtechniquesmayincludemodifyingperspective,alteringlengthsoflimbsandcurvingbones—asaresult,thereislikelynocorrect3Dinterpretationof Figure3:3Dmotioncaptureposesareprojectedto2Dusingauser-speciedcamera.thehand-drawnframesthattspreciselytoahierarchicalskeleton.Visionresearchersareinterestedintrackinghumanguresinareliableandautomatedmanner,whereasweareinterestedincapturingtheessenceofa2Dcharacter'smove-ment.Inaddition,inthepresenceofnoise,thereisatrade-offbetweentrackingpreciselyandmaintainingnaturalnessofmovement.Wechoosenaturalnessandsmoothnessofmo-tionoverprecisetrackingwhenfacedwiththistrade-off.3.ApproachTheinputtoouralgorithmisasequenceofhand-drawnim-ages(Figure 3 ).Auseridentiesvirtualmarkersontheani-matedgure'sjointsineveryinputframe,denoted˜xh.TheyalsospecifyanorthographiccameraRthatapproximatestheimaginarycamerausedbytheartisttocreatetheirdraw-ings.Boththeseinputscanbereasonablyintegratedintothe2Danimationworkowbecausethecleanupartistalreadytouchesupeveryframeoftheanimationwhenitreachestheirstation.Inpractice,ittakeslessthanaminuteperframetomarkoutthevirtualmarkers,andacoupleofminutespersequencetospecifythecamerausingasimpleGUI.Wealsoassumethata3Dmodelhasbeenconstructedandriggedwithasimpleskeletonthatrepresentsagoodapproximationtothehand-drawngure.Wetakeadvantageofmotioncaptureasasourceofdata(see,forexample,mocap.cs.cmu.edu)andassumethatamo-tioncapturesegment,X3Dmocap,isgiventhatisareasonablematchtothehandanimatedsequence(Figure 3 ).Thetimingofthismotioncapturesegmentdependsonthestyleoftheactorwhowasmotioncaptured,andisboundbythelawsofthephysicalworld(e.g.gravity).Incontrast,theartistusestimingtocommunicateemotionortempo,andmaychoosenottorespectphysicallaws[ Bab09 ].Therststepinourmethodistowarpthemotioncapturesegmenttoalignwiththetimingofthehandanimation(Figure 4 ).Thesecondstepistoaligntotheposeintendedbytheartistineachframeoftheirdrawing(Figure 5 ).Wenowdescribeeachofthesestepsinmoredetail.c\rTheEurographicsAssociation2009. EaktaJain,YaserSheikh,JessicaHodgins/LeveragingtheTalentofHandAnimatorstoCreateThree-DimensionalAnimation3.1.DescriptionofSkeletonThecompleteskeletonfora3DcharacterisparametrizedbythethreerotationvaluesforMjointsandthe3Dworldpositionoftheroot.TheposeatatimeisdenotedbyXja,whichisa(3M+3)1vector.Forthepurposeofliftingthe2Dposetothreedimensions,weuseasimpliedparametrization—the3DworldpositionsofNvirtualmarkers.Then,the3DposeatagiventimeisX3D=hX1T;X2T;:::;XNTiT:(1)whereeachXi2R3isthe3Dpositionofavirtualmarker,andX3Disa3N1vector.Thesemarkerpositionscon-taininformationabouttwoofthethreejointanglesforeachjoint.Wecomputethethirdjointangle,yawrotationabouttheboneofthelimbsegment,separatelytoobtainthecom-pleteposeXja,whichcanbesenttoacommercialsoftwarepackageforrendering(Section 3.6 ).Inourexamples,Niseither19or24,andMiseither20or21.Thischaractermodelishierarchicalinthateveryjointisdenedrelativetoitsparentinthehierarchicalchain.Anexamplelimbhierarchywouldbepelvis-upperback-lowerback-rightclavicle-righthumerus-rightradius-righthand.Wedenethehierarchicalchainsintermsofbothjointanglesandcorrespondingvirtualmarkers.3.2.PoseDescriptorWeintroduceaposedescriptortoextracttranslationandscaleinvariantinformationabouta2Dpose—theintuitionbehindthisdescriptoristhatacharactercanbeinthesameposeatdifferentlocations,andtwocharacterscanhavethesameposeeveniftheirrelativelimblengthsaredifferent.Weareinspiredbyshapecontexts[ BMP02 ],whichareintendedtobedescriptorsformatchingtwoinputshapes.Ourgoalistheinverse—togeneratea3Dposetomatchthedescriptorextractedfromtheinputhanddrawing.Foragiven2Dpose,thedescriptorstartsattheroot(whichisthepelvis)andtravelseveryhierarchicallimbchain.Foreverylinkinthechain,wedeterminetheposi-tionvectorofthechildmarkerinacoordinateframexedtoitsparent.AsillustratedinFigure 5 ,thepositionvectorforthewristwouldbecalculatedwithrespecttoacoordinateframexedtotheelbow.Thereferenceorientationforthiscoordinateframecanbeabsolute(i.e.orientedalongthex-axisoftheworldcoordinateframe),orrelative(i.e.orientedalongthecorrespondinglimb,inthiscase,rightradius).TheposedescriptorPforagivenposewouldbethevectorofpo-laranglesforthepositionvectorsofKvirtualmarkersoftheskeletalmodel,P=[1;2;:::;K]T(2)whereKisthenumberoflimbsthatareneededtocharac-terizethepose.Forallourexamples,weuseK=8(leftandrighthumerus,radius,femurandtibia). Figure4:Theprojectedmotioncapturesegmentisretimedtomatchthetimingofthehandanimatedframes. Figure5:Theposedescriptorconsistsofin-the-image-planeanglesforeverylimbsegment.Thelimbsegmentsoftheprojectedmotioncaptureposearemodiedtomatchtheposedescriptorforthehanddrawnposeviaplanarrotation.3.3.TemporalAlignmentGiventhesequenceofhand-drawnposes˜xh,weusethedynamictimewarpalgorithmtore-timethemotioncap-turesegmentX3Dmocap.Thisalgorithmalignstwovariablelengthtimeseries,subjecttomonotonicityinthetimedi-mension.Theoptimalalignmentminimizesthetotaldis-tancebetweenwarpedelementsinthetwoseries[ SC90 ],[ Ell03 ].Inordertodeneadistancemetrictocomparehand-drawn2Dposeswithmotioncaptureposes,weprojecteach3DposeX3DmocapusingtheorthographiccameraRtoget2Dposes˜x2D.WecomputetheposedescriptorsPhandP2Dforeverypose˜xhand˜x2D.Thedistancebetweenahand-drawnposeandamotioncapturepose(either3D,oritspro-jection)isdenedasthesumoftheinternalangledifferencebetweentheanglescomprisingthecorrespondingposede-scriptors,d(X3Dmocap;˜xh)=d(˜x2D;˜xh)=d(P2D;Ph)(3)d(P2D;Ph)=Kåi=1(2Di;hi)(4)wherereturnstheinternalangledifferencebetween2Diandhi.c\rTheEurographicsAssociation2009. EaktaJain,YaserSheikh,JessicaHodgins/LeveragingtheTalentofHandAnimatorstoCreateThree-DimensionalAnimation Figure6:Thereconstructed3Dposeminimizestheerrorinprojectedmarkerpositions.Wedeterminethejointanglescorrespondingtothe3Dmarkerpositions,andestimateyawfrommotioncapturedata.3.4.2DPoseAlignmentGiventheretimedmotioncapturesegment,ourgoalistogenerate3Dposestomatchtheposedescriptorsforthecor-respondingartistdrawnposes.Becausetheinputprovidesinformationaboutonlytwodimensionsofa3Dpose,weprojectthe3DmotioncapturedposesX3Dtotwodimensionswiththeuser-denedcameraR.Theprojectedmotioncap-tureposesaredenoted˜x2D.Weachieveourgoalintwosteps:rst,modifytheprojectedmotioncaptureposes˜x2D(Figure 5 ),andthen,reconstructthe3Dmarkerpositionsfromthemodiedprojections(Figure 6 ).Wedividethecharactermodelinto`upperbody',whichconsistsofthehierarchicalchainscontainingthetwoarmsandthehead,and`lowerbody',whichconsistsofthejointchainsinvolvingthelegs.Wemakethisdistinctionbecauselimbsthatareincontactwiththegroundneedtobehandleddifferentlyfromlimbsthatarenotincontact.Westartatthepelvisandworkourwaythrougheachhi-erarchicalchainoftheupperbody.Eachlimbsegmentoftheprojectedmotioncaptureposeisrotatedintheimageplanesothatthein-planepolarangleisthesameasthedesiredposedescriptor,thatis,thecorrespondingpolarangleinthehanddrawnpose(Figure 5 ).Thelowerbodyisnotmodiedinthisstep.Themodiedposeisdenotedx2D.Throughthismodica-tionstep,weattempttoextracttheposeofthehand-drawncharacterwhilelteringoutmuchofthenoiseandimpreci-sion.3.5.3DMarkerReconstructionGiventhe2Dposex2D,weinferthemissingthirddimen-sion.Wend3DvirtualmarkerpositionsX3DsuchthatX3D=argminX3D(ep+1el+2er)(5)whereepistheprojectionerror,elistheerrorinlimblengths,eristhemotioncaptureregularizationterm,and1and2aretheassociatedweights.Increasing1permitsthelimblengthstochangemore(accountingforsmallamountsofsquashandstretchintheartistdrawnpose),andincreas-ing2biasesthe3Dreconstructiontowardsthemotioncap-turepose.ThersttermepintheerrorfunctionminimizesthesumofthesquareddistancebetweentheprojectionofX3Dandthe2Dmarkerpositionsthatdescribetheposex2D,ep=RX3Dx2D:(6)Equation 6 isunderconstrainedbecausethereare3Nun-knownsand2Nconstraints.Becauseweassumethatthein-dividuallimbsofthe3Dcharactermodelarerigidbodies,weenforceconstantlimblengthsthroughtheseconderrortermel.Thepositionsofthetwovirtualmarkersontheendsofagivenlimbcanbedenotedby(xi;yi;zi)and(xj;yj;zj).Then,thelengthofthelimbis(xixj)2+(yiyj)2+(zizj)2=L2(7)whereListhelengthofthegivenlimb.WecanlinearizeEquation 7 usingaTaylorseriesexpansionaroundthebestmatchmotioncaptureposeandstackingthelengthequationsforeachlimbtoyieldel=AlimbX3Dblimb:(8)whereAlimbandblimbarefunctionsoftheretimedmotioncaptureposeandL.InSection 3.4 ,wediscussedhowtomodifytheprojec-tionofthebestmatchmotioncaptureposeX3D.Weusethis3Dposeasapriorforthereconstructionfromthemodiedprojectionx2D.Theregularizationtermiser=X3DX3Dmocap:(9)Thenalobjectivefunction(Equation 5 )canberewrittenintermsofthesethreeerrorterms.X3D=argminX3DRX3Dx2D+1AlimbX3Dblimb+2X3DX3DmocapWecanformulatealinearleastsquaressystemtoestimatetheoptimalX3Dthatminimizesthisobjectivefunction,inclosedform,W24RAlimbI35X3D=24x2DblimbX3Dmocap35(10)WAfullX3D=bfull:(11)whereWcontainstheweightsofthevariouserrorterms.ThesetofsolutionsX3Dliesinalowdimensionalsub-spaceofthesetofallpossibleposes[ SHP04 ].Eachposec\rTheEurographicsAssociation2009. EaktaJain,YaserSheikh,JessicaHodgins/LeveragingtheTalentofHandAnimatorstoCreateThree-DimensionalAnimationcanbemodeledasalinearcombinationofbasisvectors,X3D=µ+Påi=1bici(12)=µ+CXb(13)whereciisabasisvector,biisthecorrespondingweight,Cisa3NPmatrixofbasisvectors,XbisavectorofweightsforeachofthePbasisvectors,andµisthemean.PrincipalComponentAnalysisndsthesetofbasisvectorsthatmaximizevarianceinalowerdimensionalrepresenta-tion.WeperformPCAonanactivityspecicdatabasetondthesetofbasisvectorsforeachexamplemotionseparately.TheweightedleastsquaressysteminEquation 10 canthenbewrittenasWAfull(CXb+µ)=bfull;(14)WAfullCXb=bfullAfullµ;(15)WAXb=b(16)WecanndtheleastsquaressolutiontoEquation 16 andreprojectXbtogetthe3DmarkerpositionsX3D.Asthisisalinearsystem,thesolutionistheglobalminimum,isnumericallystable,andcanbefoundinclosedform.3.6.ComputationofJointAngles:Roll,Pitch,YawEverylimbisdescribedbythethreejointangles(roll,pitchandyaw)relativetoitsparentlimb.Becausetherootjointforourcharactermodelisthepelvis,westartbyrecoveringtherotationofthepelviswithrespecttotheworldcoordi-nateframeviaprocrustesanalysis(detailsareprovidedassupplementarymaterial)[ JCG04 ]andworkourwaydowneachhierarchicalchaintogeneratethefullposeXja.ThemarkerpositionsX3Dgiveustwooutofthethreejointanglesforalimbsegment,rollandpitch.Wecomputeyawrotationfrommotioncapturedata.Thejointanglesforourskeletalmodel(Section 3.1 )describetherotationofthelimbsegmentinthexyzordering—whenweconvertthisdescriptiontothezyxordering,andarefunctionsof3Dmarkerpositions(rollandpitch),andzisthe`yaw'angle,whichcannotbecomputedfrommarkerpositions.Welookuptherotationofthecorrespondinglimbsegmentinthemotioncapturepose,andsimplyusethatztocompletethegenerated3Dpose.4.ResultsWedemonstrateourmethodonfourhandanimationsbytwoartists.Theexamplesincludeballet,astylizedsneakywalk,ajumpingjackssequence,anda`happyower'sequence.Theballetdancerisdrawntobeafullyeshedcharacter,whiletherestofthecharactersaredrawnasstickgures.Allourresultsarebestseenasvideo.Figures 7 and 9 areframestakenfromthevideosforthe Figure7:Notethattheheadbobsfromsidetosideinthe3Danimationgeneratedbyourmethod,andthearmsmatchthehandanimationmuchmorecloselythanthemotioncapturedposes.balletsequenceandthejumpingjacksequence.Thetoprowcontainsframesfromthehandanimatedsequence,themid-dlerowisthe3Danimationgeneratedbyourmethod,andthebottomrowisthetimealignedmotioncapturesegment.Theskirtandponytailontheballetdancer,andthehaironthejumpingcartoonwereseparatelysimulated.InFigure 7 ,wecanseetheheadmovefromsidetosideinthe3Danima-tion(asinthehandanimation).Wefoundthatitwasquitedifcultforahumansubjecttoperformavigorousactionlikejumpingjackswhilebobbingtheirhead.Asaresult,ourmotioncapturedatabasedoesnotcontainanyjumpingjackssequencewithsignicantside-to-sideheadmovement.Acomparisonofthecharacter'sarmsintheframesinFigure 7 alsoillustrateshowouralgorithmmodiesthemotioncap-turesegmenttobettermatchthehand-drawnposes.InFig-ure 9 ,weseehowthe3Danimationgeneratedbyourmethodmatchestheballetanimation.Figure 8 showssampleframesfromthe`happyower'and`sneakywalk'sequences.Thefeetoftheowerareclampedtothegroundastheywereinthehand-drawnanimation.Thefreeparametersinourmethodaretheweightsoflimblengthconstraints1andregularization2,thenum-berofprincipalcomponentsPandthelengthofthesmooth-ingwindowwusedtosmooththeinputmarkers˜xh.Inprac-tice,1=0:001and2=0:1workwellforalmostallthec\rTheEurographicsAssociation2009. EaktaJain,YaserSheikh,JessicaHodgins/LeveragingtheTalentofHandAnimatorstoCreateThree-DimensionalAnimation Figure8:Resultsonahappyowerandastylizedwalk.Table1:Reconstructionerrorwhencameraparametersarevaried. DqDfmeanrmserrormaxrmserror(degrees)(degrees)(meters)(meters) 000.00450.0173500.00460.0173-500.00450.01741000.00480.0179-1000.00470.01750100.00500.01690-100.00510.0180 Baselineistheerrorbetweengroundtruthmarkerpositionsandtimealignedmotioncapturemarkerpositions:meanrmserror=0.0062,maxrmserror=0.0257resultswegenerated.Thepirouettesequenceintheballetexamplerequiredhigherregularization2=1because2Dinformationwithoutanytemporalcoherenceisnotsufcienttodisambiguatewhetherthearmsareinfrontofthetorsoorbehindit.WechoseP=20foralltheexamples.Wealsoevaluateourmethodquantitativelybyprojectingamotioncapturewalksequenceto2Dusingaknowncam-era.Theprojectedposesarethetargetposes(˜xh)fortheeval-uation.TheresultsaregraphedinFigure 10 .Theticksonthex-axisrepresentthe3Dpositionsofthevirtualmarkersi.e.theelementsofthevectorX3D.Ticks1through42repre-sentmarkersonthe`upperbody',thatis,thelimbsegments Figure9:Thearmsandtorsobettermatchthehandanima-tioncomparedtothemotioncapturedata. Figure10:Errorin3Dmarkerposition.thatweremodiedtomatchtheposedescriptorofthehand-drawnpose.TheredcurveistheRMSerrorbetweenthe3Dmarkerpositionsofthebestmatchmotioncaptureseg-mentandthegroundtruthmarkerpositions,fora100framesegment.ThebluecurveistheRMSerrorbetweenthe3Dmarkerpositionsgeneratedbyourmethod(called3Drecon-struction),andthegroundtruthmarkerpositions.WealsoinvestigatehowtheerrorinspecifyingRaffectsreconstructionerror.Becauseweassumeasimpleortho-graphiccameramodel,thecameracanbeparametrizedbytheazimuthandelevationangles.ThegreencurveinFig-ure 10 plotstheRMSerrorinmarkerposition(overallNmarkers)whentheazimuthisoffby5.InTable 1 ,wereportthemeanandmaximumRMSerrorin3Dmarkerpositionsc\rTheEurographicsAssociation2009. EaktaJain,YaserSheikh,JessicaHodgins/LeveragingtheTalentofHandAnimatorstoCreateThree-DimensionalAnimationwhentheazimuthandelevationareoffby5and10.ThebaselineisthemeanandmaximumRMSerrorbetweenthemarkerpositionsofthebestmatchmotioncaptureseg-mentandgroundtruth(theredcurveinFigure 10 ).Forer-rorsupto10,theerrorinthemarkerpositionsgeneratedbyourmethodislessthanthebaseline.Thisdemonstratesthattheposedescriptorisrobustenoughtotolerateasmallmismatchbetweentheuser-speciedcameraandtheartist'simaginarycamera.Ourmethodtakesapproximately2minutestogeneratejointanglesfora200frameinputsequence—ittakesaround7secondstocomputetheposedescriptorsforalltheframes,andaround1secondtocomputethe3Dreconstructionusingtheleastsquaresminimization.Mostofthe2minutesarespentintimealignmentandincomputingjointanglesfrommarkerpositions.5.ConclusionandDiscussionWehavedemonstratedamethodthatgenerates3Danima-tionofhumanmotionfromtraditionalhand-drawnframes.Ourkeyinsightisthatdomaininformationcanhelpresolveambiguitiesandhandlethenoisethatisinherentin2Dhanddrawings.Ourapproachextractsthestyleelementsoftheartist'sdrawingsandtransfersthemto3D.Weusemotioncapturedataasastartingpoint,anddescribeaposedescrip-tortoquantifythedifferencebetweenamotioncapturedposeandahand-drawnpose.Bymodifyingtheprojectedmotioncaptureposein2D,andthenliftingthemodiedposeto3D,wecreatenew3Danimation.Anassumptionmadeearlyoninourmethodisthatmodelchangessuchasthelengtheningoflimbs,orthecurvingofbones,are`noise',notinputdata.Thetechniqueof“squashandstretch”isusedtogreateffectbyhandanimators;ourmethodgeneratesthe3Dposesforarigidskeletonanddoesnotaddressthesignicantchangesinshapeseenintradi-tionalsquashandstretch.Anaturalnextstepwouldbetoin-corporatethiselementintothe3Danimationswegenerate.Therecanbemanywaystocomparehumanposes,forexam-ple,imagemeasurementssuchasmoments[ Bra99 ],variouslterresponses[ SB01 ],chamferdistances[ BSB07 ],orthebasicbit-xor[ CTMS03 ],allofwhichareglobaldescriptors.Histogramsofshapecontexts[ AT04 , AT06 ]arecompara-tively`quasi-local'inthattheyencodesilhouetteshapeoverangularandradialbinscenteredatregularlyspacedpointsonthesilhouette.Thesedescriptorscouldbeusefulifwewerecomparingchangesinthesilhouetteshapesofindivid-ualbodypartsandnotonlytheorientationsofthebones.Onelimitationofthisapproachisthatwearedependentinafewessentialwaysonthemotioncapturesegmentthatwestartwith.Becausefootplantsarepickedfromthemo-tioncapturedata,wecannotchangethegivenmotioncap-turewalktoawiderstancewalkinourresults.Also,theyawrotationforeverylimbiscomputedfrommotioncap-turedata.So,ifthedatabasecontainsahandwavewiththepalmfacingforward,wecannotchangethattohavethepalmfacebackward.Despitethis,motioncapturedataprovidesagoodstartingpointbecauseitisasequenceofsmooth,nat-uralandcoherent3Dposes.Thissequenceisnotthe3Dan-imationwedesirebecauseitisinthestyleoftheactorwhowascaptured,andthemotionisfundamentallylimitedbythelawsofphysics.Animation,ontheotherhand,involvescon-veyingtheuniquestyleofthecharacterandthe`plausiblyimpossible'inthecontextofphysics.Ourmethodusesmo-tioncapturedataasastartingpointtocreatethedesired3Dsequence—onewhichcontainsthesamesetofmoves,butinthestyleandpersonalityoftheanimatedcharacter.Itwouldbeinterestingtothinkabouthowtoextendthisapproachformotionsthatcannotbeeasilycapturedinamotioncapturestudio.Anotherlimitationisthattheposedescriptorcanonlyhandlein-the-image-planemismatchinlimbangles—wehavenotincludedforeshorteninginourdenition.Thislim-itationisperhapsnotsocrucialbecausetheanimatorpickscameraanglesthatshowthemotionoffwellaspartoftheanimationprocess.Wechoosethisposedescriptoroverthesumofsquareddistancesbetweenvirtualmarkerposi-tions[ HLF99 , DCR01 , Tay00 ]asitisalocaldescriptorandcanbeusedtocapturesubtledifferencesintheangularorien-tationofindividualbodyparts.Also,thepropertiesoftrans-lationandscaleinvariancearewellsuitedforcomparingmo-tioncapturedposesandhand-drawnposesbecausetheactorandthecharactercouldoccupydifferentpartsoftheimage,andmayhavedifferentbodyproportions.Thethirdlimitationofourmethodisthatweuseasimpleorthographiccameramodelinsteadofthegeneralperspec-tiveform.Thischoiceallowsustoformulatethe3Drecon-structionproblemasalinearsystem.Wecanthensolvetheminimizationinasinglestepbyndingtheleastsquaresso-lution.Thisformulationinuencedmanyofourdesigndeci-sions,forinstance,representingposeasthesetof3Dmarkerpositionsinsteadofhierarchicaljointangles.Ourworkisanimportantstepinleveragingtheuniquetal-entoftraditionalhandanimatorstocreateanimationsinthe3Dmedium.Manyinterestinginteractionsarepossibleinthisscenario.Forexample,onecanphysicallysimulatetheballetdancer'sskirtoncewehavethedancer'smotionin3D,andcompositeitontothehandanimationtoreducethebur-denonthehandanimator.Or,allowahand-drawncharactertointeractwithphysicalsimulationsinthe3Denvironment,battingaball,forexample.AcknowledgmentsThankstoGlenKeaneandTomLaBaffforprovidingtheinputanimations,andtoJustinMaceyandMosheMahlerforhelpwithmotioncaptureandvideo.ThanksalsotoAu-todeskfortheirdonationofthe3DanimationandrenderingpackageMaya.c\rTheEurographicsAssociation2009. EaktaJain,YaserSheikh,JessicaHodgins/LeveragingtheTalentofHandAnimatorstoCreateThree-DimensionalAnimationReferences[AT04]AGARWALA.,TRIGGSB.:Learningtotrack3dhumanmotionfromsilhouettes.InICML'04:21stInter-nationalConferenceonMachinelearning(2004),ACM,p.2.[AT06]AGARWALA.,TRIGGSB.:Recovering3dhu-manposefrommonocularimages.PatternAnalysisandMachineIntelligence,IEEETransactionson28,1(2006),44–58.[Bab09]BABBITTA.:"...animationfollowsthelawsofphysics-unlessitisfunnierotherwise.".Wikipedia,May2009.[BLCD02]BREGLERC.,LOEBL.,CHUANGE.,DESH-PANDEH.:Turningtothemasters:Motioncapturingcar-toons.ACMTransactionsonGraphics21,3(July2002),399–407.[BM98]BREGLERC.,MALIKJ.:Trackingpeoplewithtwistsandexponentialmaps.ComputerVisionandPat-ternRecognition,IEEEComputerSocietyConferenceon0(1998),8.[BMP02]BELONGIES.,MALIKJ.,PUZICHAJ.:Shapematchingandobjectrecognitionusingshapecontexts.PatternAnalysisandMachineIntelligence,IEEETrans-actionson24,4(2002),509–522.[Bra99]BRANDM.:Shadowpuppetry.InICCV'99:In-ternationalConferenceonComputerVision(1999),IEEEComputerSociety,p.1237.[BSB07]BALANA.,SIGALL.,BLACKM.,DAVISJ.,HAUSSECKERH.:Detailedhumanshapeandposefromimages.In2007ConferenceonComputerVisionandPat-ternRecognition(CVPR2007)(June2007),pp.1–8.[CTMS03]CARRANZAJ.,THEOBALTC.,MAGNORM.A.,SEIDELH.-P.:Free-viewpointvideoofhumanactors.ACMTransactionsonGraphics22,3(July2003).[DAC03]DAVISJ.,AGRAWALAM.,CHUANGE.,POPOVI´CZ.,SALESIND.H.:Asketchinginterfaceforarticulatedgureanimation.InSCA'03:ACMSIG-GRAPH/EurographicsSymposiumonComputerAnima-tion(2003),pp.320–328.[DCR01]DIFRANCOD.E.,CHAMT.-J.,REHGJ.M.:Reconstructionof3-dguremotionfrom2-dcorrespon-dences.In2001ConferenceonComputerVisionandPat-ternRecognition(CVPR2001)(Dec.2001),pp.307–314.[Ell03]ELLISD.:Dynamictimewarp(dtw)inmatlab.Webresource,available:http://www.ee.columbia.edu/dpwe/resources/matlab/dtw/,2003.[Gav99]GAVRILAD.M.:Thevisualanalysisofhumanmovement:Asurvey.ComputerVisionandImageUnder-standing:CVIU73,1(1999),82–98.[HLF99]HOWEN.R.,LEVENTONM.E.,FREEMANW.T.:Bayesianreconstructionof3dhumanmotionfromsingle-cameravideo.InNeuralInformationProcessingSystems:NIPS(1999),pp.820–826.[JCG04]JOHNC.GOWERG.B.D.:ProcrustesProb-lems.OxfordUniversityPress,USA,2004.[JT95]JOHNSTONO.,THOMASF.:TheIllusionofLife:DisneyAnimation.DisneyEditions;RevSubedition,1995.[Kan81]KANADET.:Recoveryofthethree-dimensionalshapeofanobjectfromasingleview.Artif.Intell.17,1-3(1981),409–460.[Las94]LASSETERJ.:Trickstoanimatingcharacterswithacomputer.InACMSIGGRAPHCourseNotes(1994).[LC85]LEEH.-J.,CHENZ.:Determinationof3dhu-manbodyposturesfromasingleview.ComputerVision,Graphics,andImageProcessing30,2(1985),148–168.[LGXS03]LIY.,GLEICHERM.,XUY.-Q.,SHUMH.-Y.:Stylizingmotionwithdrawings.InSCA'03:ACMSIGGRAPH/EurographicssymposiumonComputerani-mation(2003),pp.309–319.[McC14]MCCAYW.:Gertiethedinosaur,1914.[MG01]MOESLUNDT.B.,GRANUME.:Asurveyofcomputervision-basedhumanmotioncapture.ComputerVisionandImageUnderstanding:CVIU81,3(2001),231–268.[SB01]SIDENBLADHH.,BLACKM.J.:Learningimagestatisticsforbayesiantracking.InICCV'01:InternationalConferenceonComputerVision(2001),pp.709–716.[SB03]SIDENBLADHH.,BLACKM.J.:Learningthestatisticsofpeopleinimagesandvideo.InternationalJournalofComputerVision54,1-3(2003),181–207.[SBF00]SIDENBLADHH.,BLACKM.J.,FLEETD.J.:Stochastictrackingof3dhumanguresusing2dimagemotion.InECCV'00:6thEuropeanConferenceonCom-puterVision-PartII(2000),Springer-Verlag,pp.702–718.[SBS02]SIDENBLADHH.,BLACKM.J.,SIGALL.:Im-plicitprobabilisticmodelsofhumanmotionforsynthesisandtracking.InECCV'02:7thEuropeanConferenceonComputerVision-PartI(2002),Springer-Verlag,pp.784–800.[SC90]SAKOEH.,CHIBAS.:Dynamicprogrammingal-gorithmoptimizationforspokenwordrecognition.Read-ingsinspeechrecognition(1990),159–165.[SHP04]SAFONOVAA.,HODGINSJ.K.,POLLARDN.S.:Synthesizingphysicallyrealistichumanmotioninlow-dimensional,behavior-specicspaces.ACMTrans-actionsonGraphics23,3(Aug.2004).[Smi07]SMINCHISESCUC.:LearningandInferenceAl-gorithmsforMonocularPerception.ApplicationstoVi-sualObjectDetection,LocalizationandTimeSeriesMod-c\rTheEurographicsAssociation2009. EaktaJain,YaserSheikh,JessicaHodgins/LeveragingtheTalentofHandAnimatorstoCreateThree-DimensionalAnimationelsfor3DHumanMotionUnderstanding.PhDthesis,UniversityofBonn,FacultyofMathematicsandNaturalSciences,2007.[ST03]SMINCHISESCUC.,TRIGGSB.:EstimatingAr-ticulatedHumanMotionwithCovarianceScaledSam-pling.InternationalJournalofRoboticsResearch22,6(2003),371–393.[Tay00]TAYLORC.J.:Reconstructionofarticulatedob-jectsfrompointcorrespondencesinasingleimage.In2000ConferenceonComputerVisionandPatternRecog-nition(CVPR2000)(June2000),pp.677–685.[TBvdP04]THORNEM.,BURKED.,VANDEPANNEM.:Motiondoodles:aninterfaceforsketchingcharactermo-tion.ACMTransactionsonGraphics23,3(Aug.2004),424–431.[UFF06]URTASUNR.,FLEETD.J.,FUAP.:Temporalmotionmodelsformonocularandmultiview3dhumanbodytracking.ComputerVisionandImageUnderstand-ing104,2(2006),157–177.[Wik09]WIKIPEDIA:Listofanimatedfeature-lengthlms,2009.c\rTheEurographicsAssociation2009.