Robotics and Autonomous Systems Learning from demonstration and adaptation of biped - PDF document

RoboticsandAutonomousSystems47(2004)79–91 LearningfromdemonstrationandadaptationofbipedlocomotionJunNakanishi,JunMorimoto,GenEndo Inthispaper,weintroduceaframeworkforlearningbipedlocomotionusingdynamicalmovementprimitivesbasedon 1.IntroductionTherehasbeenagrowinginterestinbipedloco-motionwiththerecentdevelopmentofadvancedhu-manoidrobots.Manyofexistingsuccessfulwalkingalgorithmsusethezeromomentpoint(ZMP)crite-crite-foroff-linemotionplanningplanningandon-line Correspondingauthor.E-mailaddress:jun@atr.jp(J.Nakanishi). J.Nakanishietal./RoboticsandAutonomousSystems47(2004)79–91inroboticapplicationstoachievedesignatedtasksin-volvingrhythmicmotionwhichrequiresinteractionsbetweenthesystemandtheenvironment.Examplesincludebipedlocomotionlocomotion,quadrupedlocomo-locomo-,jugglingjuggling,drummingdrumming,andplayingwithaslinkytoyy.Neuraloscillatorshavedesir-ablepropertiessuchasadaptationtotheenvironmentthroughentrainment.However,itisdifculttodesigninterconnectionandfeedbackpathwaysofneuralos-cillators,andtomanuallytuneallopenparametersinordertoachievethedesiredbehavior.Otherapproachesincludeincludeinanefforttode-signasimplecontrollerbasedonphysicalintuition.ThecontrolstrategyproposedinRef.Ref.isquitesimpleandeasytoimplement.However,itrequiresmanualtuningofthecontrolparametersandaccuratetorque-controlledactuators.Inthispaper,wesuggestanapproachtolearningbipedlocomotionfromdemonstrationanditsadapta-tionthroughcouplingbetweenthepatterngeneratorandthemechanicalsystem.Motivatedbyhuman’scapabilityoflearningandimitatingdemonstratedmovementsofateacher,imitationlearninghasbeenexploredasanefcientmethodformotorlearninginrobotstoaccomplishdesiredmovementsements.Inourpreviouswork,weproposeddynamicalmovementprimitivestoencodecomplexdiscreteandrhythmicmulti-jointmovementsthroughimitationlearninglearning.Dynamicalmovementprimitivesareformulatedasasetofautonomousnon-lineardifferentialequa-tionswithwell-denedattractordynamics.Demon-stratedtrajectoriesarelearnedusinglocallyweightedregression,andtheoutputofdynamicalmovementprimitivesservesaskinematicmovementplans,e.g.,desiredtrajectories,forarobot.ThispaperpresentstheideaofusingtherhythmicmovementprimitivesbasedonphaseoscillatorsoscillatorsasaCPGtolearnbipedlocomotionfromdemon-stration.Comparedwithneuraloscillators,oneoftheappealingpropertiesofphaseoscillatorsisthatthedesiredphaserelationshipamongoscillatorscanbespeciedinastraightforwardmanner.InRef.Ref.,acomprehensiveformulationofphasecoordinationofcoupledphaseoscillatorsisproposed.Applicationsofcoupledphaseoscillatorshavebeenexploredinthegaitcontrolofmulti-leggedrobotsrobotsandthecontrolofabipedrobotrobot.Inadditiontousingphaseoscillators,ourmovementprimitivehasvariousdesirablepropertieswhicharebenecialforbipedlocomotion.Forexample,itcanlearnademonstratedtrajectoryrapidly,anditiseasytore-scalethelearnedrhythmicmovementintermsofamplitude,frequencyandoffsetofthepatternspatterns.Intheapplicationofrhythmicmovementprimitivestobipedlocomotion,weintroducecouplingtermstothemovementprimi-tivestoachievethedesiredphaserelationshipamonglimbsfollowingtheformulationproposedinRef.Ref..Wealsoproposeanadaptationalgorithmforthefrequencyofwalkingbasedonphaseresettingresettingandentrainmentbetweenthephaseoscillatorandmechanicalsystemusingfeedbackfromtheenviron-ment.FrequencyadaptationofaCPGisbenecialwhenthedesiredfrequencyofthecoupledsystemisnotexactlyknowninadvance.InRef.Ref.,asimilarideaofusingcoupledphaseoscillatorsasapatterngeneratorforbipedlocomotionwasproposed.Intheirmethod,desiredjointtrajecto-riesofthelegsaregeneratedfromanominaltrajec-toryatthetipofeachlegdenedbyacombinationofsimpleprescribedfunctionsofphasethroughinverseerse.IncomparisontoRef.Ref.,webe-lievethatourmethodhastheadvantageofexibilityinencodingcomplexmovementsbyimitationlearn-ingandthepotentialcapabilityofimprovinglearnedmovementsthroughreinforcementlearninglearning.Wedemonstratetheeffectivenessoftheproposedcontrolstrategybynumericalsimulationsandexperimental2.BipedrobotWeuseaplanar5-linkbipedrobotdevelopedinRef.Ref..Theheightoftherobotis0.4mandtheweightisabout2kg.Thelengthofeachlinkofthelegis0.2m.Themassofthebodyis1.0kg,thethighis0.43kgandtheshankis0.05kg.Themotionoftherobotiscon-strainedwithinthesagittalplanebyatetherboom.Thehipjointsaredirectlyactuatedbydirectdrivemotors,andthekneejointsaredrivenbydirectdrivemotorsthroughawiretransmissionmechanismwiththere-ductionratioof2.0.Thesetransmissionmechanismswithlowreductionratioprovidehighbackdrivabil-ityatthejoints.Footcontactwiththegroundisde-tectedbyfootswitches.Therobotisanunderactuatedsystemhavingroundedsoleswithnoankles.Thus,it J.Nakanishietal./RoboticsandAutonomousSystems47(2004)79–91ischallengingtodesignacontrollertoachievebipedlocomotionwiththisrobotsincenoactuationcanbeappliedbetweenthestancelegandthegroundcom-paredtomanyoftheexistingbipedrobotswhichhaveatfeetwithanklejointactuation.3.DynamicalmovementprimitivesInthissection,weoutlinetherhythmicdynamicalmovementprimitivesoriginallyproposedinRef.Ref.,whichwewilluseasaCPGforbipedlocomotioninthispaper.3.1.RhythmicdynamicalmovementprimitivesRhythmicdynamicalmovementprimitivesencodeperiodicbehavioralpatternsasanoutputofasetofnon-lineardynamicalsystemscomposedofadynamicalsystemwithaphaseoscillatorandatransformationdynamicalsystemwithanon-linearfunctionapproximator.Considerthefollowinglimitcycleoscillatorchar-acterizedintermsofanamplitudeandaphaseacanonicaldynamicalsystemwhichgeneratesbasicrhythmicpatterns:isatemporalscalingfactor,thedesired(relative)amplitude,andisapositiveconstant.Notethatthephasedynamicscanbewrittenasisthenaturalfrequency.Whentherearemultipleoscillators,wewillintroducecouplingtermsamongtheoscillators(seeSection4.1).Thisrhythmiccanonicalsystemisdesignedtoprovideanamplitudesignalsignalcos
andphasevariablemod
tothefollowingsecond-ordertransformationdynamicalsystem(
),wheretheout-isusedasthedesiredtrajectoryfortherobot:

aretimeconstants,isanoffsetoftheoutputtrajectory.isanon-linearfunctionapproximatorusinglocallinearmodelsmodelsofthe
 istheparametervectorofthethlocalmodelwhichwillbedeterminedbylocallyweightedlearn-learn-fromademonstratedtrajectorySection3.2).EachlocalmodelisweightedbyaGaus-siankernelfunctionexp
isthecenterofthethlinearmodel,andcharacterizesitswidth.Analpredictioniscalculatedbytheweightedaverageofthepredictionsoftheindi-vidualmodels.AsdemonstratedinRef.Ref.,theam-plitude,frequencyandoffsetofthelearnedrhythmicpatternscanbeeasilymodiedbyscalingtheparam-individually.3.2.ImitationlearningwithdynamicalmovementAnimportantissueishowtolearntheparametersinthenon-linearfunctiontocharacterizetheoutputofadynamicalmovementprimitiveforagivendemonstratedtrajectory.Givenasampleddatatargettargetthelearningproblemisformulatedtondtheparam-Eq.(6)usingincrementallocallyweightedregressiontechniquetechniqueinwhichisupdatedby PPvv P  target J.Nakanishietal./RoboticsandAutonomousSystems47(2004)79–9179–91
1]isaforgettingfactor.Wechosethislocallyweightedregressionframeworkasitcanau-tomaticallyndthecorrectnumberofnecessaryba-sisfunction,andcantunetheparametersofeachGaussiankernelfunctiontoachievehigherfunc-tionapproximationaccuracy.Moreover,itlearnstheofeverylocalmodeltotallyindepen-dentofallotherlocalmodels,whichminimizesinter-ferencebetweenlocalmodelsandprovidesameanstorobustlyclassifyarhythmicpatternwiththehelpoftheparametersparameters.4.RhythmicdynamicalmovementprimitivesasaWeusetherhythmicdynamicalmovementprimi-tivesintroducedinSection3.1asaCPGforbipedFig.1illustratestheproposedcontrolar-chitectureinthispaper.Eachjointisequippedwithamovementprimitivewhichgeneratesthedesiredjointtrajectory.Wedenetheindexandthecor-respondingnameofthejointasLefthip( HIP),andLeftknee(2,L KNEE),Righthip3,R HIP),andRightknee(4,R Anadditionaloscillator()isallocatedtoprovideareferencephasesignaltothelimboscillators,whichisadjustedbythegroundcontactinformationattheinstanceofheelstrike.Section4.1introducescou-plingtotheoscillatorsofthemovementprimitivestoachievethedesiredphaserelationshipbetweenthe Fig.1.Proposedcontrolarchitectureforbipedlocomotionwithdynamicalmovementprimitives.Section4.2proposesafrequencyadaptational-gorithmofthelearnedperiodicmovementsthroughtheinteractionamongthecoupledoscillators,robotandenvironment.4.1.Inter-andintra-limbphasecoordinationWeintroducecouplingamongtheoscillatorstoreg-ulatethedesiredphaserelationshipbetweenthelimbsoftherobot.Thiskindofcouplingismotivatedfromabiologicalpointofviewwhereithasbeenhypothe-sizedthatcouplingamongneuraloscillatorsplaysanimportantroleincoordinatingthedesiredphasere-lationshipoflimbmovementsinlocomotionandgaitgait.Considerthefollowingcouplingtermsfortheos-isapositiveconstantgain,andisanel-ementofthewhichcharacterizesthecouplingwithotheroscillators.Thisformofcouplingappearsinvariousstudiesofcoupledoscillatorsandtheirapplication,e.g.,e.g.,.Inthispaper,weemploytheformulationinRef.Ref.tospecifythede-siredphaserelationship.InRef.Ref.,Cisdenedtobeasymmetricmatrixwherethediagonalelements0forall,andoff-diagonalelementschosenasfollows:1:oscillatorsaredesignedtobeinphasesuchthat0(mod21:oscillatorsaredesignedtobeoutofphasesuchthat(mod2AsnotedinRef.Ref.,anarbitraryphasedifferenceotherthan0orcanbespeciedbyintroducingachangeofcoordinates,orequivalenttohavinganoffsetinthecouplingterms.Inthispaper,wedesignthedesiredphasedifferenceamongthecanonicaloscillatorssuchthatthelinksofeachlegmoveinphase(withzerophasedifference),andtheleftandrightlegsmoveoutofphase(withphasedifference)bydeningthephaseoftheos-cillatoras0attheinstanceofheelstrikeofthecorrespondingleg.Morespecically,werequire,and J.Nakanishietal./RoboticsandAutonomousSystems47(2004)79–91.Thus,theconnectionmatrixischosentobe1011104.2.FrequencyadaptationoflocomotionSection4.1introducedinternalcouplingoftheos-cillatorstocoordinatethephasedifferenceamongthelimbsoftherobot.Thissectionconsidersinterac-tionbetweentheenvironmentandtheCPGtoachieveself-tuningofthenaturalfrequencyoftheoscillatorsandsynchronizationoftheCPGwiththeperiodicbe-havioroftherobot.4.2.1.SynchronizationofcoupledoscillatorswithfrequencyadaptationBeforeintroducingtheproposedfrequencyadapta-tionlawforthebipedrobotCPG,letusconsiderthebehaviorofthefollowingdynamicsoftwocoupled0arenaturalfrequenciesoftheos-cillators,andarepositivecouplingconstants.Then,theoscillatorsrunwiththephasedifferenceatthecoupledfrequencywhentheyareentrained.When,thephasedifferenceremainsnon-zero.However,ifthenthephasedifferenceoftheseoscillatorswillbezero.Thus,weintroduceanupdatelawofthenatu-ralfrequencytoachievesynchronizationoftheseoscillatorswithzerophasedifference:isapositiveconstant.Itisstraightforwardtoseethat.Thus,thephasedifferencewillbezerosuchthat4.2.2.FrequencyupdatelawandphaseresettingofInthissection,weintroduceanadaptationalgo-rithmoftheCPGinordertoadjustthefrequencyofthelearnedperiodicmotionsbytherobotthroughtheinteractionamongtheCPG,robotandenvironment.AsdepictedinFig.1,theproposedcontrolsystemcanberegardedasacouplingoftheCPGandtheme-chanicaloscillator(robot)whichisanalogoustothecoupledoscillatorsystemdiscussedinSection4.2.1Forthispurpose,werstintroduceareferenceoscillator()whichwillbesynchronizedwiththelocomotionoftherobotthroughtheadaptationmech-anismdescribedbelow.Thisreferenceoscillatorcanbeconsideredasaphaseestimatoroflocomotionbythediscreteheelstrikeinformationdetectedbyfootswitches.Then,additionalcouplingisintroducedtothelimboscillatorswithtoachievethedesiredrelativephaseMotivatedbythesynchronizationmechanismofthecoupledoscillatorsinSection4.2.1,weproposethefollowingphaseresettingandfrequencyupdatelaw.Theycanbeinterpretedasadiscretizedversionofphasecouplingandfrequencyupdateattheinstanceofheelstrike:$heelstrikeheelstrikeistheDirac’sdeltafunction,isthenumberofsteps,andheelstrikeisthephaseofthemechanicaloscillator(robot)atheelstrikedenedasheelstrike0attheheelstrikeoftheleftleg,andheelstrikeattheheelstrikeoftherightleg.isthemeasuredfrequencyoflocomotiondenedby isthesteppingperiodoflocomotion(halfperiodwithrespecttotheoscillator).Atthesametime,naturalfrequenciesofallthelimboscillatorsareupdatedattheinstanceofheelcontactsuchthat.Notethatitispossibletodirectlyintro-ducephaseresettingtothelimboscillatorsasseeninin.However,introductionofareferenceoscil-latorallowsphaseestimationdependingonmultipleeventsandmulti-modalinformation.Moreover,con-tinuousphasecouplingofthelimboscillatorswiththe J.Nakanishietal./RoboticsandAutonomousSystems47(2004)79–91 15.5 16 16.5 17 17.5 0.6 0.4 0.2 0 0.2 0.4 Time (sec)L_HIP (rad) L_HIP desR heel strike 15.5 16 16.5 17 17.5 0.2 0 0.2 0.4 0.6 0.8 1 Time (sec)L_KNEE (rad) L_KNEE desL heel strkeR heel strike Fig.2.Jointtrajectoriesfortheleftlegandheelstriketimingforfourperiods(eightsteps)ofwalking(simulation).referenceoscillatorhavingphaseresettingalleviatestheproblemofdiscontinuitytothedesiredjointtra-Thephaseresettingalgorithmismotivatedfromabiologicalperspectiveaswellasamathe-maticalpointofview.Phenomenaofphaseresettingorphaseshiftareobservedinmanybiologicalos-cillatorsresultingfromexternalperturbations,e.g.,circadianpacemakers,biochemicaloscillators,andhumanngertappingneuralnetworksorks.Phaseresettingisrelatedtothestabilitypropertiesofneu-ralrhythms,whichcanbeanalyzedbyexaminingthephase-dependentresponsesagainstperturbations.Arecentstudystudyinvestigatedtheroleofphaseresettinginbipedlocomotion.Numericalstudiesininsuggestpossiblecontributionofphasereset-tingduringwalkingtogaitstabilityagainstexternal5.NumericalsimulationsAsademonstratedtrajectory,weusethemo-tioncapturedataofhumanwalkinginRef.Ref.(29-year-oldmale,173cm,83.5kg,righthipandknee).WeidentiedtheperiodandfrequencyofthispatternbythepowerspectrumestimationwithFFTandautocorrelationas17sand855Hz,respectively.ThedynamicsoftherobotarederivedusingSD/FASTandintegratedusingtheRunge–Kuttaalgorithmat1msstepsize. http://www.sdfast.comThegroundcontactforceiscalculatedusingalinearspring-dampermodel.Alow-gainPDcontrollerisusedateachjointtotrackthedesiredtrajectorywhichistheoutputofthemovementprimitive.Awalkingpatternfromthedemonstratedtrajectoryislearnedwiththedynamicalprimitives.Wemanu-allydesignedthedesiredtrajectoryfortheinitialstepoflocomotionfromastandingpositionatrest,andtheproposedCPGcontrollerisactivatedatheelcon-tactoftherststep.Theamplitudeparameterofthedynamicalprimitivesissetto7,andtheoff-375isintroducedtothekneejoints.Forthescalingofthenaturalfrequencyoftheoscillator,theadaptationlawproposedinSection4.2.2isusedwiththeinitialfrequencyof83rad/s(periodofoscillationis1.3s).Theseparametersaredeterminedempiricallyfromtrialanderror.Fig.2illustratesthedesiredandactualjointtrajec-toriesfortheleftleg,andthetimingofheelstrikeafterastablepatternwaslearnedbythephasereset-tingalgorithm.Fig.3showsthetorquecommandfortheleftleg,whichindicatesthatthekneejointswingspassivelysinceitrequiresalmostnotorque(see1–15.3s).Fig.4depictsonestepofwalking.Fig.5(left)showstheadaptationoftheperiodoflocomotionFig.5(right)showsthelearningcurveofthefre-quencyoftheCPGwithdifferentcouplingconstants2,0.5and0.8inEq.(18).Thesteppingperiodapproached0.387s,andtheresultantCPGfrequencywas12rad/s,whichroughlycorrespondstothenaturalfrequencyoftheswinglegmodeledasasim-pliedlinearpendulum,usingtheproposedadaptationlaw. J.Nakanishietal./RoboticsandAutonomousSystems47(2004)79–91 15.5 16 16.5 17 17.5 0.5 0 Time (sec)L_HIP torque (Nm) L heel strkeR heel strike 15 15.5 16 16.5 17 17.5 0.5 0 Time (sec)L_KNEE torque (Nm) L heel strke Fig.3.Torquecommandtothelefthipandkneejointsforfourperiods(eightsteps)ofwalking(simulation). Fig.4.Snapshotsofwalkingsimulationforonestepat15frames/s(1frame66ms).Robustnessagainstexternalperturbationsisevalu-atedbypushingtherobotforwardandbackwardwithexternalforces.Forcesareappliedforadurationof0.1satdifferenttimingduringasinglestep(atanin-tervalof0.1radfrom0to2ofthephaseoftheref-erenceoscillator).Whenaforwardperturbingforceisapplied,therobotcouldcopewithupto9.1N(max)at1radand2.2N(min)at7radoftheper-turbingforces.Whenabackwardperturbingforceisapplied,therobotcouldcopewithupto2.4N(max) 100 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of stepsHalf period (sec) K=0.2K=0.5K=0.8 1 2 3 4 5 6 7 8 9 Number of stepsomega (rad/s) K=0.8 Fig.5.Frequencyadaptationofwalkingviaentrainment(simulation).Left:adaptationofperiod.Right:learningcurveofthefrequencyoftheCPG.9radand1.0N(min)at4and5radoftheperturbingforces.Incontrast,withoutphaseresetting,therobotonlycouldcopewithmuchsmallerdisturbances,forexample,therobotonlytol-eratedupto3.9Noftheforwardperturbingforceap-pliedatThesimulationresultsdemonstrateself-adaptationofthefrequencyoflocomotionandrobustnessofwalkingagainstdisturbancebytheproposedalgo- J.Nakanishietal./RoboticsandAutonomousSystems47(2004)79–91 16.5 17 17.5 18 18.5 0.5 0 L_HIP (rad) L_HIPL_HIP desL heelstrke 16.5 17 17.5 18 18.5 0 Time (sec)L_KNEE (rad) L heelstrkeR heelstrike Fig.6.Jointtrajectoriesfortheleftlegandheelstriketimingforfourperiods(eightsteps)ofwalking(experiment).6.ExperimentalimplementationWeimplementedtheproposedcontrolframeworkonourbipedrobot.Intheexperimentalimplementa-tion,ourinitialattempttoachievebipedlocomotionusingthehumandemonstratedtrajectorywasnotsuccessfullargelyduetomechanicallimitationoftheexperimentalsystemanddiscrepancyinthegroundcontactconditionbetweensimulationsandexperi-ments.Thus,weusedanothertargettrajectorywhichwasexperimentallyobtainedfromanactualtrajectoryofsuccessfulrobotlocomotionusingastatemachinecontroller.Thestatemachinecontrollerisdesignedtocoordinatethelegmovementswiththephysicalstateoftheleggedsystembasedontheideapresentedin 16.5 17 17.5 18 18.5 1 0 1 2 Time (sec)L_HIP torque (Nm) L_HIP torqueL heelstrke 16 16.5 17 17.5 18 18.5 1 0 1 2 Time (sec)L_KNEE torque (Nm) L heelstrke Fig.7.Torquecommandtothelefthipandkneejointsforfourperiods(eightsteps)ofwalking(experiment).Toinitiatelocomotionintheexperiments,werstsuspendedtherobotwiththelegsswingingintheair,andthenplacedtherobotonthegroundman-ually.Thus,theinitialconditionofeachrunwasnotconsistent,andoccasionallytherobotcouldnotstartwalkingorfelloverafteracoupleofstepswhenthetimingwasnotappropriate.Fig.6tratesthedesiredandactualjointtrajectoriesfortheleftleg,andthetimingofheelstrike.Fig.7showsthetorquecommandfortheleftleg.Someoscilla-tioninthetorquecommandforthehipjointcanbeseen.Thisisduetonoisyjointvelocitysignalsobtainedfromnumericallydifferentiatedjointan-glesmeasuredbyopticalencoders.Wearecurrentlyplanningtousegyrostoobtainsmootherjointve-locitiestoimprovetheperformanceofthetrackingcontroller.Notethatalimitonthetorquecommand J.Nakanishietal./RoboticsandAutonomousSystems47(2004)79–91 100 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Number of stepsHalf period (sec) 0 0 1 2 3 4 5 6 7 8 9 Number of stepsomega (rad/s) Fig.8.Frequencyadaptationofwalkingviaentrainment(experiment).Left:adaptationofperiod.Right:learningcurveofthefrequencyoftheCPG.isimposedat1.5Nm.Fig.8(left)showsthepe-riodadaptationandFig.8(right)showsthelearningcurveofthefrequencyoftheCPG.Steppingpe-riodforatypicalwalkingexperimentwasaround0.37s.Inthisexperiment,theinitialfrequencyoftheoscillatorwassetto71rad/s(periodofoscil-lationis1.1s),andtheadaptationgaininEq.(18)wasdecreasedaccordingtoanannealingprocedure,where05andisthenum-berofsteps,asisneededinmostgradientdescentprocedure.Weintroducedanoffsetforphase$heelstrikeheelstriketoadjustthetimingoffootcontact,whereischosentobe8rad.Theseparametersaredeterminedempirically.Notethatphaseresettingwithanoffseteffectivelychangestheperiodofoscillation. Fig.9.Walkingoversurfaceswithdifferentfrictionpropertiesandaseesaw-likemetalsheetwithaslightchangeintheslope.Robustnessoftheproposedalgorithmisevaluatedbytestingwalkingoversurfaceswithdifferentfric-tionpropertiessuchascarpet,corksheet(3mmthick)andaseesaw-likemetalplate(2mmthick).Themetalplatewasplacedsothattheinclinationoftheslopeslightlychangeslikeaseesawwhentherobotwalksoverit(theheightofthecenteris7mm).TherobotcoulddealwiththechangeintheenvironmentasdepictedinFig.9Notethatevenifweusethelearnedtrajectoryfromtheactualrobotwalkingpattern,therobotcouldnotwalkbyjustreplayingitasadesiredtrajectory.Phaseresettingusingfootcontactinformationwasnecessary.Thisimpliesthatappropriateon-lineadjustmentofthephaseoftheCPGbysensoryfeedbackfromtheenvi-ronmentisessentialtoachievesuccessfullocomotion.Inaddition,empiricallywefoundthattheproposedcontrollerachievedmuchmorerobustwalkingcom-paredtothestatemachine-basedcontrollerwhichweoriginallydesigned. J.Nakanishietal./RoboticsandAutonomousSystems47(2004)79–917.ConclusionInthispaper,weproposedamethodforlearn-ingbipedlocomotionfromdemonstrationanditsfrequencyadaptationusingdynamicalmovementprimitives.Inthedynamicalmovementprimitives,kinematicmovementplansaredescribedinasetofnon-lineardifferentialequationswithwell-denedattractordynamics,anddemonstratedtrajectoriesarelearnedusinglocallyweightedregression.Speci-cally,weuserhythmicdynamicalmovementprimi-tivesbasedoncoupledphaseoscillatorsasaCPG,andintroducedafrequencyadaptationalgorithmthroughinteractionsamongtheCPG,mechanicalsystemandtheenvironmentmotivatedbythesynchronizationofcoupledoscillators.Numericalsimulationsandex-perimentalresultdemonstratetheeffectivenessoftheproposedcontrolalgorithmtoachievesteady-statewalkingroughlyatthenaturalfrequencyofthecou-pledsystem.Wealsoevaluatedrobustnessagainstdisturbanceinnumericalsimulationsandexperiments.Futureworkwilladdressinitiationandterminationofwalking,andon-linebalancecompensation.Wewillalsoconsidercollectionofhuman’swalkingdataundervariousbehavioralconditions.Inourcurrentstudy,weusedasimplephaseresettingmechanisminwhichthephaseoftheCPGisforcedtoberesettoaspecicvalueattheinstanceofheelstrikeregard-lessofthecurrentphaseoftheCPG.Inthefuture,weareinterestedinthegeneralizationoftheideaofphaseresettingtodeterminephase-dependentreactionagainstexternalperturbationssuchasrecoveryfromstumblingbydesigninganappropriatephaseresettingcurvee.Formalmathematicalanalysiswillbere-quiredtounderstandtheprincipleofperiodicstabilityofalimitcyclesolutiontothedynamicsofacombinedoscillatorandmechanicalsystem.Inthelongrun,wearehopefulthatourapproachmayprovideinsightintoatheoreticallysounddesignprincipleofbipedloco-motioncontroltoachievehuman-likenaturalwalking.AcknowledgementsWewouldliketothankAukeIjspeertatEPFL,SwissFederalInstituteofTechnology,LausanneandSeiichiMiyakoshioftheDigitalHumanResearchCenter,AIST,Japan,andChrisAtkesonatCarnegieMellonUniversity,forvaluablediscussionsandhelp-fulcomments.ThisresearchwassupportedinpartbyNationalSci-enceFoundationgrantsECS-0325383,IIS-0312802,IIS-0082995,ECS-0326095,ANI-0224419,aNASAgrantAC#98-516,anAFOSRgrantonIntelligentControl,theCommunicationsResearchLaboratoryofJapan,theERATOKawatoDynamicBrainProject,fundedbytheJapanScienceandTechnologyAgency,andtheATRComputationalNeuroscienceLaborato-References[1]M.Vukobratovic,B.Borovac,D.Surla,D.Stokic,BipedLocomotion—Dynamics,Stability,ControlandApplication,Springer,1990.[2]S.Kagami,T.Kitagawa,K.Nishiwaki,T.Sugihara,M.Inaba,H.Inoue,Afastdynamicallyequilibratedwalkingtrajectorygenerationmethodofhumanoidrobot,AutonomousRobots12(2002)71–82.[3]A.Takanishi,M.Tochizawa,H.Karaki,I.Kato,Dynamicbipedwalkingstabilizedwithoptimaltrunkandwaistmotion,in:ProceedingsoftheIEEE/RSJInternationalWorkshoponIntelligentRobotsandSystems,1989,pp.561–566.[4]K.Hirai,M.Hirose,Y.Haikawa,T.Takenaka,Thedevelopmentofhondahumanoidrobot,in:ProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,1998,pp.1321–1326.[5]S.Kagami,F.Kanehiro,Y.Tamiya,M.Inaba,H.Inoue,Autobalancer:anonlinedynamicbalancecompensationschemeforhumanoidrobots,in:B.R.Donald,K.Lynch,D.Rus(Eds.),AlgorithmicandComputationalRobotics:New,AKPetersLtd.,2001,pp.329–340.[6]J.Yamaguchi,A.Takanishi,I.Kato,Developmentofabipedwalkingrobotcompensatingforthree-axismomentbytrunkmotion,in:ProceedingsoftheIEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,1993,pp.187–192.[7]K.Yoneda,S.Hirose,Tumblestabilitycriterionofintegratedlocomotionandmanipulation,in:ProceedingsoftheIEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,1996,pp.870–876.[8]S.Mochon,T.A.McMahon,Ballisticwalking,JournalofBiomechanics13(1980)49–57.[9]K.Matsuoka,Sustainedoscillationsgeneratedbymutuallyinhibitingneuronswithadaptation,BiologicalCybernetics52(1985)367–376.[10]G.Taga,Y.Yamaguchi,H.Shimizu,Self-organizedcontrolofbipedallocomotionbyneuraloscillatorsinunpredictableenvironment,BiologicalCybernetics65(1991)147–159.[11]S.Miyakoshi,G.Taga,Y.Kuniyoshi,A.Nagakubo,Threedimensionalbipedalsteppingmotionusingneuraloscillators—towardshumanoidmotionintherealworld,in: J.Nakanishietal./RoboticsandAutonomousSystems47(2004)79–91ProceedingsoftheIEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,1998,pp.84–89.[12]K.Hase,N.Yamazaki,Computationalevolutionofhumanbipedalwalkingbyaneuro-musculo-skeletalmodel,ArticialLifeandRobotics3(1999)133–138.[13]G.Endo,J.Morimoto,J.Nakanishi,G.Cheng,Anempiricalexplorationofaneuraloscillatorforbipedlocomotion,in:ProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,2004,pp.3036–3042.[14]Y.Fukuoka,H.Kimura,A.H.Cohen,Adaptivedynamicwalkingofaquadrupedrobotonirregularterrainbasedonbiologicalconcepts,InternationalJournalofRoboticsResearch22(3–4)(2003)187–202.[15]S.Miyakoshi,M.Yamakita,K.Furuta,Jugglingcontrolusingneuraloscillators,in:ProceedingsoftheIEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,1994,pp.1186–1193.[16]S.Kotosaka,S.Schaal,Synchronizedrobotdrummingbyneuraloscillator,in:ProceedingsoftheInternationalSymposiumonAdaptiveMotionofAnimalsandMachines,[17]M.M.Williamson,Neuralcontrolofrhythmicarmmovements,NeuralNetworks11(1998)1379–1394.[18]J.Pratt,G.Pratt,Intuitivecontrolofaplanarbipedalwalkingrobot,in:ProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,1998,pp.2014–2021.[19]H.Miyamoto,S.Schaal,F.Gandolfo,Y.Koike,R.Osu,E.Nakano,Y.Wada,M.Kawato,Akendamalearningrobotbasedonbi-directionaltheory,NeuralNetworks9(1996)[20]H.Miyamoto,M.Kawato,Atennisserveandupswinglearningrobotbasedonbi-directionaltheory,NeuralNetworks11(1998)1317–1329.[21]S.Schaal,Isimitationlearningtheroutetohumanoidrobots?TrendsinCognitiveSciences3(6)(1999)233–242.[22]A.Ijspeert,J.Nakanishi,S.Schaal,Learningattractorlandscapesforlearningmotorprimitives,in:S.Becker,S.Thrun,K.Obermayer(Eds.),AdvancesinNeuralInformationProcessingSystems,vol.15,MIT-Press,2003,pp.1547–1554.[23]E.Klavins,D.E.Koditschek,Phaseregulationofdecentralizedcyclicroboticsystems,InternationalJournalofRoboticsResearch21(3)(2002)257–275.[24]S.Ito,H.Yuasa,Z.weiLuo,M.Ito,D.Yanagihara,Amathematicalmodelofadaptationinrhythmicmotiontoenvironmentalchanges,in:ProceedingsoftheIEEEInternationalConferenceonSystems,ManandCybernetics,1997,pp.275–280.[25]K.Tsujita,K.Tsuchiya,A.Onat,Adaptivegaitpatterncontrolofaquadrupedlocomotionrobot,in:ProceedingsoftheIEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,2001,pp.2318–2325.[26]K.Tsuchiya,S.Aoi,K.Tsujita,Locomotioncontrolofabipedlocomotionrobotusingnonlinearoscillators,in:ProceedingsoftheIEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,2003,pp.1745–1750.[27]M.Kawato,Transientandsteadystatephaseresponsecurvesoflimitcycleoscillators,JournalofMathematicalBiology12(1981)13–30.[28]J.Peters,S.Vijayakumar,S.Schaal,Reinforcementlearningforhumanoidrobotics,in:ProceedingsoftheThirdIEEE-RASInternationalConferenceonHumanoidRobots,[29]J.Morimoto,G.Zeglin,C.G.Atkeson,Minimaxdifferentialdynamicprogramming:applicationtoabipedwalkingrobot,in:ProceedingsoftheIEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,2003,pp.1927–1932.[30]S.Schaal,C.G.Atkeson,Constructiveincrementallearningfromonlylocalinformation,NeuralComputation10(8)(1998)2047–2084.[31]A.J.Ijspeert,Aconnectionistcentralpatterngeneratorfortheaquaticandterrestrialgaitsofasimulatedsalamander,BiologicalCybernetics84(5)(2001)331–348.[32]Y.Kuramoto,ChemicalOscillations,Waves,andTurbulence,Springer-Verlag,1984.[33]K.Sekiyama,J.Nakanishi,I.Takagawa,T.Higashi,T.Fukuda,Self-organizingcontrolofurbantrafcsignalnetwork,in:ProceedingsoftheIEEEInternationalConferenceonSystems,ManandCybernetics,2001,pp.2481–2486.[34]T.Yamasaki,T.Nomura,S.Sato,Possiblefunctionalrolesofphaseresettingduringwalking,BiologicalCybernetics88(6)(2003)468–496.[35]Y.Ehara,S.Yamamoto,IntroductiontoBody-Dynamics—AnalysisofGaitandGaitInitiation,IshiyakuPublishers,2002(inJapanese).[36]J.K.Hodgins,Bipedgaittransitions,in:ProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,1991,pp.2092–2097. JunNakanishireceivedtheB.E.andM.E.degreesbothinmechanicalengi-neeringfromNagoyaUniversity,Nagoya,Japan,in1995and1997,respectively.HereceivedthePh.D.degreeinengineer-ingfromNagoyaUniversityin2000.HealsostudiedintheDepartmentofElec-tricalEngineeringandComputerScienceattheUniversityofMichigan,AnnAr-bor,USA,from1995to1996.HewasaResearchAssociateattheDepartmentofMicroSystemEngi-neering,NagoyaUniversity,from2000to2001,andwasapresi-dentialpostdoctoralfellowattheComputerScienceDepartment,theUniversityofSouthernCalifornia,LosAngeles,USA,from2001to2002.HejoinedATRHumanInformationScienceLab-oratories,Kyoto,Japan,in2002.HeiscurrentlyaresearcheratATRComputationalNeuroscienceLaboratoriesandwiththeCom-putationalBrainProject,ICORP,JapanScienceandTechnologyAgency.Hisresearchinterestsincludemotorlearningandcontrolinroboticsystems.HereceivedtheIEEEICRA2002BestPaperAward. J.Nakanishietal./RoboticsandAutonomousSystems47(2004)79–91 JunMorimotoreceivedhisB.E.incomputer-controlledmechanicalsystemsfromOsakaUniversity,Osaka,Japan,in1996,M.E.ininformationsciencefromNaraInstituteofScienceandTechnol-ogy,Nara,Japan,in1998,andPh.D.ininformationsciencefromNaraInstituteofScienceandTechnology,Nara,Japan,in2001.HewasaResearchAssistantatKawatoDynamicBrainProject,ERATO,JST,from1999to2001.HewasapostdoctoralfellowattheRoboticsInstitute,CarnegieMellonUniversity,Pittsburgh,PA,from2001to2002.HeiscurrentlyaresearcheratATRCom-putationalNeuroscienceLaboratories,Kyoto,Japan,andwiththeComputationalBrainProject,ICORP,JapanScienceandTechnol-ogyAgency.HeisamemberofJapaneseNeuralNetworkSoci-ety,andRoboticsSocietyofJapan.Hisresearchinterestsincludereinforcementlearningandrobotics. GenEndoreceivedhisB.E.andM.E.degreesbothinMechano-AerospaceEn-gineeringfromTokyoInstituteofTech-nology,Tokyo,Japan,in1996and1998,respectively.HereceivedthePh.D.de-greeinengineeringfromTokyoInstituteofTechnologyin2000.HejoinedSonyCorporation,Tokyo,Japan,in2000.HeiscurrentlyacollaborativeresearcheratATRComputationalNeuroscienceLabo-ratories,Kyoto,Japan.HeisamemberofRoboticsSocietyofJapan.HereceivedtheBestPaperAwardofRoboticsSocietyofJapanin2002.Hisresearchinterestsincludebiologicallyinspiredmobilerobot,mechanicaldesignandcontrol. GordonChengistheheadoftheDepart-mentofHumanoidRoboticsandCom-putationalNeuroscience,ATRComputa-tionalNeuroscienceLaboratories,Kyoto,Japan,andwiththeComputationalBrainProject,ICORP,JapanScienceandTech-nologyAgency.Beforetakingupthisposition,heheldfellowshipsfromtheCenterofExcellence(COE),Science,andTechnologyAgency(STA)ofJapan.BothofthesefellowshipsweretakenattheHumanoidInteractionLabo-ratory,IntelligentSystemsDivisionattheElectroTechnicalLabo-ratory(ETL),Japan.AtETLheplayedamajorroleindevelopingacompletelyintegratedhumanoidroboticssystem.HereceivedaPh.D.insystemsengineeringfromtheDepartmentofSystemsEngineering,TheAustralianNationalUniversity,andBachelorandMasterdegreesinComputerSciencefromtheUniversityofWollongong,Australia.Hisindustrialexperienceincludesconsul-tancytoandasanationalsystemsmanagerforamajortransportcompany.Hewasalsothedirectorofthecompany,G.T.I.Com-puting,specializinginnetworking/transportmanagementsystemsinAustralia.Hisresearchinterestsincludehumanoidrobotics,biomimeticofhumanvision,computationalneuroscienceofvi-sion,actionunderstanding,human-robotinteraction,activevision,mobilerobotnavigationandobject-orientedsoftwareconstruction.HeisasocietymemberoftheIEEERobotics&AutomationandComputerSociety.HeisontheeditorialboardoftheInternationalJournalofHumanoidRobotics.Heisaprogramco-chairforthenextIEEEInternationalConferenceonHumanoidRobots. StefanSchaalisanAssociateProfessorattheDepartmentofComputerScienceandtheNeuroscienceProgramattheUniversityofSouthernCalifornia,andanInvitedResearcherattheATRComputa-tionalNeuroscienceLaboratoryinJapan,whereheheldanappointmentasHeadoftheComputationalLearningGroupduringaninternationalERATOproject,theKawatoDynamicBrainProject(ER-ATO/JST).HeisalsoanAdjunctAssistantProfessorattheDepartmentofKinesiologyofthePennsylvaniaStateUniversity.BeforejoiningUSC,Dr.SchaalwasapostdoctoralfellowattheDepartmentofBrainandCognitiveSciencesandtheArticialIntelligenceLaboratoryatMIT,anInvitedResearcherattheATRHumanInformationProcessingResearchLaboratoriesinJapan,andanAdjunctAssistantProfessorattheGeorgiaInstituteofTechnology.Dr.Schaal’sresearchinterestsincludetopicsofstatisticalandmachinelearning,neuralnetworks,computationalneuroscience,non-lineardynamics,non-linearcontroltheory,andbiomimeticrobotics.Heapplieshisresearchtoproblemsofar-ticialandbiologicalmotorcontrolandmotorlearning,focusingonboththeoreticalinvestigationsandexperimentswithhumansubjectsandanthropomorphicrobotequipment. MitsuoKawatoreceivedtheB.S.de-greeinphysicsfromTokyoUniversityin1976,theM.E.andPh.D.degreesinbio-physicalengineeringfromOsakaUniver-sityin1978and1981,respectively.From1981to1988hewasafacultymemberandlecturerofOsakaUniversity.From1992hebecameadepartmentheadofDepartment3,ATRHumanInformationProcessingResearchLaboratories.From2003,DirectorofATRComputationalNeuroscienceLaboratories.From2004,hehasbeenjointlyappointedastheDirectoroftheComputationalBrainProject,ICORP,JST.From1996to2001hewasjointlyappointedasadirectorofKawatoDynamicBrainProject,ERATO,JST.HehasbeenjointlyappointedasvisitingprofessorofKanazawaInstituteofTechnology,NaraInstituteofScienceandTechnologyandOsakaUniversity.Forthelast15 J.Nakanishietal./RoboticsandAutonomousSystems47(2004)79–91yearshehasbeenworkingincomputationalneuroscience.HewasawardedYonezawafounder’smedalmemorialspecialawardoftheInstituteofElectronics,InformationandCommunicationEngineersin1991,outstandingresearchawardoftheInternationalNeuralNetworkSocietyin1992,OsakaSciencePrizein1993,10thTsukaharaNaka-akiraMemorialAwardin1996andTokizaneToshihikomemorialaward.HeisagoverningboardmemberofJapaneseSocietyofNeuroscienceandJapanNeuralNetworkSocietyandMemberofExecutiveCommitteeofInternationalAssociationfortheStudyofAttentionandPerformance.