M.I.TMediaLaboratoryPerceptualComputingSectionTechnicalReportNo.398Nov - PDF document

M.I.TMediaLaboratoryPerceptualComputingSectionTechnicalReportNo.398November,1996.RevisedJune1998Appearsin:Presence:TeleoperatorsandVirtualEnvironments,Vol.8,No.4,August1999,pp.367-391.TheKidsRoom:APerceptually-BasedInteractiveandImmersiveStoryEnvironmentAaronF.Bobick,StephenS.Intille,JamesW.DavisFreedomBaird,ClaudioS.PinhanezLeeW.Campbell,YuriA.Ivanov,ArjanSchÈutte,AndrewWilsonTheMITMediaLaboratory20AmesStreetCambridge,MA02139(kidsroom@media.mit.edu)AbstractTheKidsRoomisaperceptually-based,interactive,nar-rativeplayspaceforchildren.Images,music,narration,light,andsoundeffectsareusedtotransformanormalchild'sbedroomintoafantasylandwherechildrenareguidedthroughareactiveadventurestory.Thefully-automatedsystemwasdesignedwiththefollowinggoals:(1)tokeepthefocusofuseractionandinteractioninthephysical,notvirtualspace;(2)topermitmultiple,collab-oratingpeopletosimultaneouslyengageinaninteractiveexperiencecombiningbothrealandvirtualobjects;(3)tousecomputer-visionalgorithmstoidentifyactivityinthespacewithoutrequiringtheparticipantstowearanyspe-cialclothingordevices;(4)tousenarrativetoconstraintheperceptualrecognition,andtouseperceptualrecog-nitiontoallowparticipantstodrivethenarrative;(5)tocreateatrulyimmersiveandinteractiveroomenvironment.WebelievetheKidsRoomisthe®rstmulti-person,fully-automated,interactive,narrativeenvironmentevercon-structedusingnon-encumberingsensors.Thispaperde-scribestheKidsRoom,thetechnologythatmakesitwork,andtheissuesthatwereraisedduringthesystem'sdevel-opment.I.MotivationandBackground1IntroductionWeareinvestigatingthetechnologiesrequiredtobuildperceptually-basedinteractiveandimmersivespacesÐspacesthatrespondtopeople'sactionsinareal,physi-calspacebyaugmentingtheenvironmentwithgraphics,video,soundeffects,light,music,andnarration.Usingcomputervision-basedactionrecognitionandothernon-encumberingsensingtechnologiestointerpretwhatpeoplearedoinginaspace,aroomcanautomaticallyprovideentertainingfeedbackinanaturalwaybymanip-ulatingthephysicalenvironment.Forexample,akitchenmightuseaudioandvideotoguideitsoccupantsthrough Ademonstrationoftheproject,includingvideos,im-ages,andsoundsfromeachpartofthestoryisavailableathttp://vismod.www.media.mit.edu/vismod/demos/kidsroomandcomplementsthematerialpresentedhere.thepreparationofarecipe;acafemightobservehowpeo-pleareinteractingandchangelighting,video,andmusictoenliventheatmosphere;and,achild'sbedroommightstimulateachild'simaginationbyusingimagesandsoundtotransformitselfintoafantasyworld.InthispaperwedetailourexperienceconstructingandtestingtheKidsRoom,afully-automated,interactive,nar-rativeplayspaceforchildren.Thespacetheatricallyresem-blesachildren'sbedroom,completewithfurnitureinclud-ingamovablebed.Undercomputercontrolandinresponsetothechildren'sactions,theroomusestwolargeback-projectedvideoscreens,fourspeakers,theatricallighting,threevideocameras,andamicrophonetocarrythechildrenthroughastory.TheKidsRoomexperiencewasdesignedprimarilyforchildrenagessixtotenyearsold,anditlaststentotwelveminutes,dependinguponhowtheparticipantsactintheroom.Throughoutthestory,childreninteractwithobjectsintheroom,withoneanother,andwithvirtualcreaturesprojectedontothewalls.Theactionsandinter-actionsofthechildrendrivethenarrativeactionforward.Mostimportantly,thechildrenareawarethattheroomisresponsive.Thetextisdividedintothreemajorsections,coveringthemotivation,implementation,andanalysisoftheproject.Wehaveincludeddetailsatalllevelsofdesignrangingfrombroadconsiderationssuchasmodelingthefocusofattentionoftheuser,totechnicalissuessuchasvisualsensorintegration,totheimplicationofimplementationdetails.Ourgoalisthatthelessonswelearnedfromthisprojectwillbevaluabletoothersconstructingsuchperceptually-controlled,interactivespaces.2ProjectgoalsTheinitialgoalofourgroupwastheconstructionofanenvironmentthatwoulddemonstratevariouscomputervisiontechnologiesfortheautomaticrecognitionofaction.Aswedevelopedthedesigncriteriaforthisprojectitbe-cameclearthatthiseffortwasgoingtobeanexplorationandexperimentinthedesignofinteractivespaces.Thegoalsthatshapedourchoiceofdomainwerethefollowing:Actioninphysicalspace.Becauseourcomputervi-sionresearchfocusesontherecognitionofactions performedbyhumans,werequirethattheusersbeengagedinactivitytakingplaceintherealphysicalenvironment,notthevirtualscreenenvironment.Ourgoalwastoaugmentarealspace,stimulatingtheimag-inationusingvideo,light,andsound,butnotreplacingthenatural,real-worldactivitywithwhichpeoplearecomfortable.Vision-basedremotesensing.Remotesensingpermitsunencumberedactivityinthephysicalspace,notre-quiringuserstowearsensors,head-mounteddisplays,earphones,microphones,orspecially-coloredcloth-ing.Also,computervisiontrackingandactionrecog-nitiontechniquesallowapersontoeasilyenterandexittheroomatanytimenaturallywithouttrouble-somesensorrequirements.Multiplepeople.ªInteractiveºentertainmentspacesaremoreengagingandsocial(andfun!)whenonecanplayaspartofagroup.However,previousworkincomputervisionandfully-automatedinteractivespaceshasprimarilyconsideredenvironmentscon-tainingonlyoneoratmosttwopeople.Ourgoalistoallowmultiplepeopleintheenvironment,interact-ingnotonlywiththeenvironmentbutalsowitheachother.Ifunencumberedbyhead-mounteddisplays,peoplewillnaturallycommunicatewitheachotherabouttheexperienceasittakesplace,andtheywillwatchandmimiconeanother'sbehavior.Useofcontext.Oneofourresearchtopicsistheuseofcontexttoincreasethereliabilityofvision-basedsens-ing.Ourgoalistobuildasystemthatisnotonlyawareofthecontextofthesituation(e.g.thecurrentpositioninastory-line)butthatalsomanipulatescontextbycontrollingmuchoftheenvironment.Presence,engagement,andimagination.Wewantanenvironmentthatistrulyimmersive,perceptuallyandcognitivelyengaging,andwhereparticipantsdonotneedtoaskforoutsidehelp.Furthermore,wewanttocreateanarrativeexperience,wherethestoryandactionofthepeoplewithrespecttothestoryaretheprimaryfocus.Theexperienceshouldengageuser'simaginationmuchlikeachildren'sstorybook;itisnotnecessarilyrequiredordesirabletoprovideacompletefancifulrenditionofavirtualworld.Theexperienceshouldbecompellinginthesensethattheusersshouldbemoreconcernedwiththeirownactionsandbehaviorsthanwithhowtheinteractivesystemworks.Children.Wewanttheenvironmenttobetailoredtochildren.DavenportandFriedlander[10]andDruinandPerlin[14]bothobservedthatadultsvisitingtheir Someoftheªdirectsensingºtaskscouldbeimplementedusingotherdevicessuchasmicro-switchestodetectpresenceofapersononabed.Becauseourfocusisoncomputervisionwedidnotemploysuchdevices.However,evenifaroomisdenselywiredwithsensingdevices,thedif®cultproblemofunderstandingwhatishappeninginthespacestillneedstobeaddressed,andthatisafundamentalcomponentofourresearchonunderstandingactioninthevisiondomain[5].interactiveinstallationssometimeshaddif®cultyim-mersingthemselvesinthenarrative.Childrenalreadyliketoplaywitheachotherinrealspacesenhancedbyimaginaryconstructs(e.g.couchesascaves)andcanbeeasilymotivatedbysupplementaryimagery,soundandlighting.The®naldesignoftheKidsRoomwasintendedtoachieveeachofthesegoals.Theideaofachildren'splayspaceimmediatelyaddressesmostoftheconcerns:alargeencompassingroomwithmultiplechildrenbeingac-tiveinanengagingactivity.Thegoalofexploitingandcontrollingcontextwasaccommodatedbyembeddingtheexperienceinanarrativeenvironment,wherethereisanaturalstory-linetodrivethesituation.3InteractioninaugmentedenvironmentsBedersonandDruinclassifyworkoncomputerinterac-tiveinterfacesystemsintothosethatfocusonbuildingin-terfaceswhereinformationissuperimposedonthephysicalworldandthosethatembedinformationintothephysicalworlditself[4].Themajorityofresearchinthehumancomputerinterfaceandcomputergraphics®eldshasfo-cusedonsystemsofthe®rstform,whereausermustweargearsuchasglovesensors,specially-coloredclothing,ormicrophones.TheKidsRoomisanexampleofsystemsofthesecondform,wherethecomputerinterfacebecomesunobtrusivelyembeddedinthephysicalworlditself.Suchsystemshavebeenalternatelytermedªaugmentedenvi-ronments,ºªimmersiveenvironments,ºªintelligentroomsº[31],ªsmartroomsº[23],andªinteractivespaces.ºOneearlyexampleofaugmentingaphysicalspacewastheªMediaRoomºprojectofBoltandNegroponte[6].Theirsystemallowedausersittinginachair,ostensiblyinhisorherfuturelivingroom,tointeractwithascreenbypointingandtalking.Theirgoal,asisours,wastoaugmentspacesthatwearecomfortablewith,butthetechnologyavailableatthattimerequiredtheuseofbodygearforsensinggestureandspeech.Eventoday,mostworkinaugmentedenvironmentsrequirescumbersomesensingandhead-mounteddisplaygear[1].Researchonphysical,remotelysensed,interactivespacesbeganwithKrueger'sVideoplacesystem[20].Kruegerdesignedinstallationsthatexploredmanydiffer-entmodesofinteraction,mostofwhichentailedlargebodygesture.Inoneexample,theuserinteractedwithhisorherownsilhouetteonvideoscreens.TheALIVEprojectimprovedonKrueger'ssystembyreplacingspecialblue-screeninghardwarewithcomputervisionalgorithmsthatcantrackthepositionandgesturesofasinglepersonmovinginfrontofanarbitrarilycom-plex,staticbackground[22].Asingleusercaninteractwithvirtualcreaturesbywatchinghisorherownimagesuperim-posedwithbehavior-basedcreaturesonalargevideowall.Theusermustorienttowardsthevideowalltoobservetheinterestingaction.TheªIntelligentRoomºsystemconsistsofseveralcam-erasandtwolargescreensinasmallroom[31].Asingle Wealsonotethatchildrenaremoreforgivingofthesmallglitchesintiming,animations,andrecognitioncertaintobepresentinsuchanexperimentalfacility;ourhopeisthatchil-drenwillfocusmoreonhavingfuninthespacethanon®guringouthowitworksorhowtobreakit. personistrackedusingcomputervision,andsimplepoint-inggesturesarerecovered.Userscanutteralexiconofabout25sentencesintoalapelmicrophone.ThecomputerwillunderstandinstructionssuchasªComputer,whatistheweatherhere?ºandwillusethecitythatthepersonispointingtoononeoftwolargescreenstoretrieveweatherforecastinginformationthatisthendisplayedontheotherscreen.Theroomiscontrolledbyadistributedagent-basedarchitecture[8].Thegoaloftheprojectistoremovethecomputerfromthehuman-computerinterface.Analternativeapproachtothedesignofinteractivespacesistomediatecomputerinteractionthroughthema-nipulationofreal,physicalobjects.Druin,forinstance,constructedastuffed-animaldollcalledNoobie[13].Chil-dreninteractedwithNoobiebysqueezingthedoll'slimbsandwatchingadisplayembeddedinitsbelly.Insteadofbringingchildrentothevirtualspaceonthescreenandforcinginteractionwithspecialdevices,theinterfacewasbroughtintotheworldofthechildrenandembeddedintodeviceswithwhichtheywerealreadycomfortable(alsosee[33,15]).DruinandPerlinsetouttoconstructimmersivephysicalenvironmentsforadultsthatrespondedtomovementandtouchinrealphysicalspaces[14].Theysupplementedarealenvironmentwithasimplenarrativeandin1993debutedaninteractiveinstallationwiththreestories:onehumorousstoryaboutbabysitting,anothermoreseriousnarrativeaboutheavenandhell,anda®nalmurder-mysteryscenario.Thesystemusedcomputercontroloflighting,sound,video,andphysicalpropsandsensorembeddedinobjects.Interestingly,DruinandPerlincommentthatsomeadultswereconfusedwiththewholeideaofanimmersiveexperience.Oneparticipantcommented,ªIdidn'tthinkIshouldtouchanything.Youknow,Momalwayssaid,Donottouch!ºNarrativeininteractivephysicalspaceswasfurtherex-ploredbyDavenportandFriedlander[10].Theywantedpeopletoªfeelasthoughtheywerewalkingthroughacom-putermonitorintoamagiclandscape.ºTheyconstructedafour-world,human-controlledinstallationwherethenar-rativewasªactualizedbythetransformativeactionsofthevisitormovingthroughit.ºTheroomusedlight,sound,video,andcomputerdisplays.Eachpersoninthespacehadahumanguide,anotherªuserºofthesystem,outsidethespacecommunicatingwithhimorherviacomputers.Avarietyofartisticexperimentshavebeenundertakeninvolvingcomputerizedspaces.Areviewofthisworkisbeyondthescopeofthispaper,butreferencesandacriticalanalysisofsomeoftheexperimentsareavailable[21,26].AnotableinstallationisMasakiFujihata'sBeyondPagesfeaturingavirtualbookwhoseillustrationsofobjectssuchasalampanddoorreacttotheuser'sgestures.ArtistsChristaSommerer,LaurentMignonneau,andNaokoTosahaveintegratedcomputervision,computergraphics,andemotionandspeechrecognitiontechniques[30,32].BedersonandDruinbelieve,asdowe,thatthebestimmersivephysicalenvironmentswillhavemulti-modalinputsandoutputs[4].Increasingcomputationalspeedisnowmakingitpossibletoexploredomainslikeimmer-siveof®ceenvironments[35,18],livingspaces,andthe-aterperformances[25].Thetwomajorobstaclestobuild-ingfully-automatedreactivespacesare(1)®ndingpracti-cal,computationally-feasiblesensingmodalitiesthatcanbeusedtounderstandavarietyofdifferenttypesofhumanac-tionandinteractionand(2)developingacomputationally-feasiblecontrolmechanismandinter-communicationar-chitectureforcoordinatingperceptualinput,narrativecon-trol,andperceptualoutputsystems.Boththesegoalsre-quirethatwefurtherourunderstandingofhowwerepresentactions,interaction,time,andstory.4RecognitionofactionMostvirtualrealitysystemsmapagivencon®gurationofthesensoroutputsdirectlytosomesystemresponse.Forinstance,intheALIVEsystem,thepositionofaperson'shandsandheadareestimatedusingcomputervision,andtherelativepositionoftheseobjectsisusedtodetermineifapersonismakingagesture[36].TheKidsRoommovesbeyondjustmeasurementofpo-sitiontowardsrecognitionofactionusingmeasurementandcontext.Althoughmanyofthemechanismsusedaresimple,theKidsRoomcombinesthesensoroutputswithcontextualinformationprovidedbythestorytorecognizeoveradozensimpleindividualandgroupactionsinspeci®ccontexts.Examplesincludemovingthroughaforestinagroup,rowingaboat,anddancingwithamonster.Theserecognizedactionsdrivethestoryandcontrolthenarrative.Throughoutthispaper,wewilllumpalltypesofac-tionrecognitiontogether.However,withinthecomputervisioncommunity,researchersaredevelopingataxonomyofactionbasedonthecomputationalrepresentationsandmethodsrequiredtounderstandeachactiontype(e.g.ataxonomyofmovement,activity,andaction[5]).Manyac-tionsofinterestrequirethatcontextualknowledgebeusedforrecognitioninadditiontosensedmotionandpositionin-formation.Insimplecontexts,directmeasurementofbodypositioncansometimesbeusedtorecognizeactivity.How-ever,asthecomplexityofanenvironmentincreases,manydifferentmeasurementsmaycorrespondtothesameac-tions,or,dependingonthecontext,thesamemeasurementmaycorrespondtodifferentactions.Strongercontextualconstraintsarerequiredtoextractactionlabelsfromper-ceptualmeasurements.TheenvironmentoftheKidsRoomisrichenoughtobegintoexploresomecontext-sensitiverecognitiontasks.II.ImplementationandExperience5ThePlayspaceTheKidsRoomtheatricallyre-createsachild'sbedroom.Thespaceis24by18feetwithawire-gridceiling27feethigh.Twoofthebedroomwallsresemblerealwallsofachild'sroom,completewithrealfurnitureanddecoration.Theothertwowallsarelargevideoprojectionscreens,whereimagesareback-projectedfromoutsideoftheroom.Behindthescreensisacomputerclusterwithsixmachinesthatautomaticallycontroltheroom.Computer-controlledtheatricalcoloredlightsontheceilingilluminatethespFourspeakers,oneoneachwall,projectsoundeffectsandmusicintothespace.Finally,therearefourvideocamerasandonemicrophoneinstalled.Figure1showsaviewofthecompleteKidsRoominstallation.Theroomcontainsseveralpiecesofrealfurniturein-cludingamovablebed,whichisusedthroughoutthestory.Becausetheotherfurnitureisnotexplicitlytrackedbythe Figure1:TheKidsRoomisa24by18footspaceconstructedinourlab.Twowallsresemblethewallsinarealchildren'sroom,completewithpostersandwindows.Theothertwowallsarelargeback-projectionscreens.Computer-controlledlightingsitsonagridsuspendedabovethespace.Thedoortothespace,whereallroomparticipantsenterandexit,ispicturedintheleftmostcorneroftheroom.computervisionsystem,itissealedshutandfastenedtotheground.Fourcoloredrugswithanimaldrawingsandsimu-latedstonemarkersonthe¯oorareusedasreferencepointsduringthenarrative.Coloredcinderblocksonthe¯oorpre-vententhusiasticchildrenfrompushingthebedthroughthescreens.Figure2showsthelayoutoftheroom'sinterior.Figure3showsthefourcameraviews.Cameraoneisthetopview,whichisusedfortrackingpeopleandforsomemotiondetection.Camerastwoandthreeareusedtorecognizebodymovementswhenchildrenarestandingonthegreenandredrugs,respectively.Finally,camerafourprovidesaspectatorviewofmostoftheroomandthetwoprojectionscreens;thisviewisdisplayedtospectatorsoutsidethespaceandprovidesvideodocumentation.Inadditiontothevisualinput,asinglemicrophoneisinthespacethatisusedtodetecttheloudnessofshouts.Theroomhas®vetypesofoutputformotivatingpar-ticipants:video,music,recordedvoicenarration,soundeffects,andlighting.Still-framevideoanimationispro-jectedonthetwowalls.Voicesofthenarratorandmonsters,aswellasothersoundeffects,aredirectionallycontrolledusingthefourspeakersandappeartocomefromparticularregionsoftheroom.Somesoundeffects,suchasmon-stergrowlingandboatcrashes,areparticularlyloudandcanvibratethe¯oor,providingvisceralinput.Aswedis-cusslater,lightingmustremainconstantwhenthevisionalgorithmsareoperating;however,becausethestorycanbeusedtodeterminewhenvisionisandisnotrequired,itispossibletouselightingchangesandcoloredlightingtomarkimportanttransitions.SixcomputerspowertheKidsRoom.OneSGIIndyR5000workstationisusedfortrackingpeopleandthebed,playingsoundeffects,andMIDIcontroloflightoutput.AsecondSGIIndyR5000workstationisusedforactionrecognitionfromcamerasoneandtwo,sendingMIDImu-siccommandstothemusiccomputer,andamplitudeau-diodetection.AthirdSGIIndyworkstationisusedforactionrecognitionfromcamerathree.TwoDECAlpha-Stationsareusedfordisplayingstill-frameanimations,oneperscreen.OneoftheAlphaStationsalsorunstheroom'scontrolprocess.AMacintoshisusedforrunningStudioProMIDIsoftwareconnectedtoaKorg5R/Wsynthesizer.Finally,assortedvideo,lighting,andsoundequipmentarerequiredtocompletetheinstallation.6TheStoryTheKidsRoomguideschildrenthroughaninteractive,imaginativeadventure.Inspiredbyfamouschildren'ssto-riesinwhichchildrenaretransportedfromtheirbedroomstomagicalplaces(e.g.[2,34,27]),thestorybeginsinachild'sbedroomandprogressesthroughthreeotherworlds.Wewilldescribethelastworldindetail,togivethereaderafeelforthestory,it'scharacters,andtheinteractivere-sponsivenessoftheentiresystem. Weusedthehardwarewehadavailable,butcurrentPCsequippedwithreal-timevideodigitizerswouldsuf®ce.Includestwohigh-resolutionvideoprojectorsandwall-sizedscreens,4SonyHandyCamcolorvideocameras,4speakersanda12-channel4-outputmixerandampli®er,onemicro-phone,14lights(11white,3colored),aMIDI-basedlightboardcontroller,andvideodistributionampli®ers. Figure2:TheKidsRoomisfurnishedlikearealchildren'sroom,completewithfurniture,decorations,andamovablebed.Rugsandstone-likemarkersareusedthroughoutthenarrative.Fourspeakersprojectsoundintothespace.Coloredcinderblocksatthebaseofthelargeprojectionscreensprotectthescreens(whichmakeuptherightandbottomwallsinthisimage)fromthemovablebed.Thesquareonthe¯oorinthebottomleftmarkstheªdoor,ºwherepeopleenterandexitthespCamera1Camera3 Camera2Camera4Figure3:ThreecamerasoverlookingtheKidsRoomareusedforcomputervisionanalysisofthescene.Camera1isusedfortrackingthepeopleandthebedinthespaceandalsofordetectingmotionduringparticularpartsofthestory.Cameras2and3areusedtorecognizeactionsperformedbypeoplestandingontheredandgreenrugs.Camera4isusedtoprovideaviewofmostoftheroomandthescreensforspectatorsoutsidethespace.Wenotetherethattheprimarystoryisatraditionallinearnarrative,asopposedtothe(typicallyweakly)non-linearbranchingstory-linesfoundinmanymulti-mediapresentations.Individualresponsesmadebytheroomarereactiveandinthatsensenon-linear.Aswewilldiscussintheanalysissection,weneededastrongnarrativetomotivategroupbehaviorandtoprovidesuf®cientcontext.Wewillarguethatthislinearstructureinnowayreducedtheinteractivitybecausethepacingandindividualreactionsoftheroomarecompletelydeterminedbytheparticipants.Theonlyinstructiongiventothechildrenpriortoen-teringtheroomwasthatthiswasamagicroom,butthattotransformtheroomtheyneededtolearnthemagicpass-word.Tolearnthepasswordtheyshouldtryªaskingthefurniture.º6.1ThebedroomworldChildrenentertheKidsRoomoneatatimethroughtheªdoorºinonecorneroftheroom.Thetrackingalgorithm Figure4:Whenparticipants®rstenterthespace,theprojectionscreensdisplaythewallsofabedroom.attendstothisregion,checkingforpeopleenteringandexitingthespace.Whimsical,curiousmusicplayssoftly,andtheprojectionwallsdisplayscenesfromabedroom,asshowninFigure4.Whenatleastonechildapproachessomepieceoffurniture(e.g.thebluedeskorthegreenfrogrug),eachofwhichhasadistinctpersonality,thefurniturespeaksandascavengerhuntforthemagicwordcommences,asshowninFigure5a.Forexample,aclothestrunksays(withsuitableaccent),ªAye,matey,I'mthepiratechest.Idon'tknowthemagicword,butthefrogontherugmightknow.ºThechildrenthenruntotherugwiththefrogpaintedonit,andtherugfrogseeminglyspeaks,sendingthemtoyetanotherpieceoffurniture.Thesystemrandomlyselectstheorderingsothattheinteractionisslightlydifferenteachtimethesystemisrun.Evenasituationassimpleasthebedroomrequireshan-dlingcontingencies.Ifthechildrengetconfusedanddonotgotothecorrectplace,thesystemwilleventuallyresbyhavingsomefurniturecharactercallthechildrenover,ªHey,overhere,it'sme,theyellowshelf!ºThisgamecontinuesforafewiterations,usuallywithrunning,screaming,laughingchildren.Eventually,onepieceoffurnitureknowstherandomly-chosenmagicword(e.g.ªskullduggeryº)andrevealsittothechildren.Assoonasthepasswordisdisclosed,allofthefurniturestartchantingthewordloudly,ensuringthechildrenhearandrememberit.Amother'svoicesoonbreaksin,silencingthefurniturevoices,andtellingthekidstostopmakingnoiseandtogotobed.Whentheydo,thelightsdropdown,andaspotononewallishighlighted,whichcontainstheimageofastuffedmonsterdoll.Themonsterstartsblinkingandspeaks,askingthechildrentoloudlyyellthemagicwordtogoonabigadventure:ªOnthecountofthreeyellthemagicword:One,two,three,6.2TheforestworldAfterthekidsyellthemagicwordloudly,theroomdarkensandthetransformationoccurs.Imagesonthepro-jectionwallsgraduallyfadetoimagesofacartoonfantasyforestland,andcolored,¯ashinglightscombinedwithmys-teriousmusicplayduringthetransformation,asshowninFigure6.Simultaneously,agrandfatherly-voicednarratorÐthevoiceandpersonalityoftheroomÐsays,ªWelcometotheKidsRoom.It'snotwhatitseems.Whatyoumight Thereisnospeechrecognitioncapability,withonlythevolumeofsoundbeingmeasured.InnorunoftheKidsRoomdidthechildreneveryellanythingbutthemagicword,illustratinghowacompellingnarrativewillconstrainbehavior.seeherearethingsdreamtinyourdreams.ºThenarra-toralwaysspeaksincouplets,enhancingtheexperienceofbeingimmersedinachildren'sstorybook.Asthelightscomeup,thenarratortellsthechildrenthattheyareintheªForestDeep,ºthatmonstersarenear,andthattheymustfollowthepathtotheriver.Astone-pathismarkedonthe¯ooroftheroomandthechildrenquicklyrealizethatisthepaththeyshouldfollow.Theroomprovidesencouragingnarrationiftheydonotdosoandinstructsthemtotostayinagroupandtoremainonthatpath.Iftheydeviatefromthepathªhintsºaregiventoinducethebehavior.Hintsareloudlywhisperedsug-gestionsmadeinasoftfemalevoicetoprovideadditionalinstructionwhenneeded;theirpowerwillbediscussedlaterintheanalysissection.Astheytraversethepath(movingaroundtheroomsev-eraltimes),monstersareheardgrowlingfromafar.ThenarratorwarnsªThemagicbedisnowatree.Hidebehindwheremonsterscannotsee.ºWhenthechildrenmovebe-hindthebed,themonstersstopgrowling,andthechildrencontinueonthepath(seeFigure5b).Iftheydonothide,theloudgrowlingintensi®esandadifferentnarrationen-couragesthekidstogetbehindthebed.Afterashortwalk,thechildrenreachtheriverworld.6.3TheriverworldAsthenarratorannouncesªYou'vecertainlymanagedagloriousact.You'vearrivedattheriverandyou'restillintact,ºimagesofariverdissolveontothetwoscreens.Oneviewshowstheriverprogressingforward(seeFigure7)andtheotherviewshowsthesidewaysmovingriverbank.Intheriverworld,thechildrenaretoldtheªmagicbedºisnowaboat.Theyareencouragedtopushthebedtothecenteroftheroomandªjumpinsideºbyclimbingontop.Whenthechildrenstartmakingrowingmotions,theriverimagesstartmoving.Ifsomeonegetsofftheboat,asplashingsoundisheard.Thenarratorthenshoutsaªpassengeroverboardºcoupletandencouragesthechildtogetbackonthebed.Soon,logandrockobstaclesappearinthepathoftheboat,asshowninFigure7a.Asinstructedbythenarrator,thechildrenmustengageincollaborativerowing(makingrowingmotionsonthecorrectsideofthebed)toavoidtheobstacles.Iftheysuccessfullynavigatetheobstacle,afemalevoicesoftlywhispershintssuchasªNicejob.ºWhentheydonot,theyhearaloudcrashingsound,oftenmotivatingthekidstophysicallyplay-actacrashasinFigure7b,andtheyreceivesomewhisperedhintsa (a)(b)Figure5:(a)Childrenaretoldonlytoªaskthefurnitureforthemagicword.º(b)Duringtheforestworld,childrenmusthidebehindthebedtostoptheloudgrowlingofdistantmonsters. Figure6:Lightingeffectsareusedtomarkspecialtransitionsinthestory.Herecoloredlights¯ash,transitionalmusicplays,andthescreensgraduallyfadefrombedroomtoforestasthenarratorwelcomesthechildrentotheforestworld.Graphicsaresimplestory-bookanimations.However,combinedwithmusic,narration,andlightingeffects,thetransformationcapturestheattentionofpeopleintheroomandgivestheroomasomewhatmagicalquality.howtoavoidtheobstacles.Eventually,theimageofashoreappears,andthechil-drenareinstructedtoªlandtheboatºbypushingthebedtowardstheªshorelineºonthescreen.Astheydo,thesystemproducesloudgrindingnoisesasifthebedisbe-ingpushedontothesand.Suddenly,themother'svoiceisheardinthedistanceagaintellingthechildrentostopmovingfurnitureandtoputthebedbackwheretheyfoundit.Ifnecessary,additionalhintsareprovideduntilthebedisreturnedtoapproximatelyitsoriginalposition.Atthispointthechildrenhavereachedthemonsterworld.6.4ThemonsterworldForestimagesdisplayedonthetwoscreens,tenseforestmusicplayinginthebackground,andjunglesoundsliketwigscrackingandowlshootingannouncethearrivaltothemonsterland.Togiveafeelfortheinteractionwepresentthisworldasanannotateddialog.Thenarratorspeaksinacomforting,deep,somewhatmischievousvoice. Welearnedbyexperiencethatchildrenprefertocrashandtypicallypaddletowardstheobstacles!Narrator:ªWelcometoMonsterLand.It'sagreatspot.Timetohavefun,readyornot.ºMonstersareheardgrowlingsoftly,butcannotyetbeseen.Childrenareoftenhuddledonthebedwaitingexpectantly.Suddenly,thereareloudroarsandthemonstersappearonthetwoscreens.Themonsters,showninFigure8a-b,arelargerthanthechildrenandhaveafriendly,goofy,cartoonlook.Astheycontinuetogrowl,theroomspeaksandsuggeststhatifthekidsyell,themonstersmightbequiet.Thekids,inunison,yell.Iftheshoutisloud,thestorycontinues.Ifnot,theroomrespondswithencouragement.Narrator:ªGetthosemonstersbackinline.Trythatshoutonemoretime!ºIfthekidsstilldonotscream,theroomresponds:Narrator:ªWell,Ican'tsaythatwasaveryloudshout,butperhapsthemonsterswill®gureitout.ºEitherway,themonstersstopgrowlingandshowsurpriseforamoment.Energeticmusicstartstoplay. (a) (b)(c)Figure7:(a)Thespeedoftheboatiscontrolledbyhowvigorouslypeopleonthebedarerowing.Ifeveryonestopsmakingmotions,theboatimagerywillstopmovingforward.Obstaclessuchastheloginthisseriesofimagesapproachtheboat.Theycanbeavoidedbyrowingstronglyontheappropriatesideofthebed(i.e.ifthelogisonthetheleftasshown,thenrowontheleft-handsideofthebed).(b)Audiofeedbacksuchasloudcrashesandnarrationsignalwhenobstacleshavebeenhitoravoided.Crashestendtoevokeexpressiveresponsesfromchildrenandsubsequentlymoreenthusiasticrowing.(c)Achildandmotherrowtheboattogether. (a)(b)Figure8:(a)Achilddancingwithamonster.(b)Childrenspinningduringthemonsterdance.Narrator:ªThemonstersinviteyoutoshimmyanddance.Gostandonarugandyou'llgetyourchance.ºWhenthechildrenareonrugs(sometimespromptedmul-tipletimesindifferentways),theroomcontinues.Thevisionalgorithmusedinthisworldrequirestherebeonlyonechildoneachrugtogetanon-occludedviewofparticipant.Therefore,ifthesystemdetectsmultiplepeo-pleonarug,oneofthemonstersinthestoryrespondsinhisraspyvoice:Monster2:ªHey,onlyonekidperrugplease,so'swecanseewhat'sgoin'on.ºThroughoutthissectionofthestory,thesystemdetectswhenchildrengetoffofarugandthecharactersinthestoryrespondaccordingly.Monster2:ªHey,besuretostayonyourrugthere,thanksalot.ºWheneachrughasasinglechildonit,themonstersbegintoteachthechildrenfourdancemoves:Monster1:ªHey,I'mgoingtodoacrouch.Youwatchme®rst,thenyoutryit.Todoitright,crouchdownandtouchyourtoes.ºThemonster,representedusingstill-frameanimationasillustratedinFigure9doesthemove,andthensays,ªYour turn!ºThesystemwatchesthechildrenontwooftherugs.Whenachildisobservedhavingdonethecrouchmove,themonstersrespondwithpositivecomments:Monster1:ªYo!Kidontheredrug,youdancelikeapro!ºThemonsterscontinue,teachingthechildrenthreeothermoves:throwingthearmstomakea`Y',a¯appingmovewitharmsextended,andaspinningmovealsowitharmsout.Oncethechildrenknowthemoves,themusicchangesandbecomesmoreenergetic.Narrator:ªNowthatyouknowthemonstermoves,seeiftheycancatchyourgrooves!Themonsterswillbeabletocopyifyoudothemoves,butdon'tbesloppy.ºThesystemwillnowrespondtoanyofthefourmovesmadebythechildrenonthefronttworugs.Themonsterswillmimicamovethatachildperforms.Ifthesystemhasahighcon®dencethatitknowswhichmovethechildisperformingthenthemonsteralsooffersspokencon®rmationsuchasªAwesome¯aps.ºIfthechildrenstopdoingrecognizablemoves,themonstercharacterschoosemovesthemselvesandawhisperedhintlike,ªTrya¯ap,spin,`Y',orcrouchºisheard.Thisdancephasecontinuesforafewminutes(seeFig-ure8b).Thenthemusicgrowslouderandfaster:Narrator:ªNowit'stimetodoyourowndance.Let'sseeyoudanceandboogieallaroundtheroom!ºThemonsters,yellingkudossuchasªFeelthegroove!ºandªShakethatfunkything!º,doavarietyofmonstermoves,includingsomenewones.Themusic'scharacter,tempoandvolumecausethechildrentorunaroundtheroomanddonewdancemoves.Suddenly,themother'svoiceisheardagain,andallmusicandsoundabruptlyend:Mom:ªItoldyoukidstogotobed,andImeanbusiness.ºIfthechildrenallgetonthebed,thestorymoveson.How-ever,ifnot,themothercharactercontinuestoencouragethedesiredaction.Mom:ªI'mnotfoolingaround.Getonthatbed.Allofyou!ºAssoonaseveryone'sbackonthebed,themonstersrespondtotheendofthescene:Monster1:ªHeyy'all,let'sgetquiet,andnexttimeyoucomeback,we'llhavearockin'goodtime.º Thechildrenontheblueandyellowrugsnearthebackoftheroomaretrackedbuttheiractionsarenotanalyzedbythesystem,becauseperformingrecognitiononthoserugswouldrequireadditionalcamerasandcomputersnotatourdisposal.Childrenareonlyontheserugswhenmorethantwopeopleareintheroom,andinthosecasesthechildrenareclearlyawareoftheinteractionbetweenthemonstersandtheirplaymates.Thelightsdropdownandcoloredtransitionlightingisused.Transformationalmusicplays,asthemonsterssaygoodbye,ªTakeiteasy!Byebye!ºTheforestfadesbacktotheroom,andasthelightsslowlycomeup:Narrator:ªThanksforourMonsterLandvisitwithyou.We'veenjoyedthiswilddream,andwetrustyoudidtoo.ºExitmusicplaysasthechildrenleavethespace.7PerceptualTechnologyAsdiscussedinsection4,intheKidsRoomwebothmeasurethepositionandmovementofmultiple,interactingpeopleandthenusethatdataandcontextualinformationtorecognizeactionusingvision-basedperceptualalgorithms.Thissectionbrie¯ydescribestheperceptualmethodsusedbytheKidsRoom.Discussionofthedif®cultiesween-counteredwitheachmethodaredeferreduntiltheanalysissection.7.1ObjectTrackingMostimmersiveenvironmentswillneedtokeeptrackofthepositionsofpeopleandobjectsinthespace.IntheKidsRoom,wetrackthepositionsofuptofourpeopleandthemovablebed.Someworlds,likethebedroomworld,usepositionalinformationtoknowwhetherpeoplearenearcertainobjects:thepiecesoffurniturespeakonlywhenapersonisnear.Thepositionalinformationisalsousedtokeeptrackofwhetherpeopleareinagroupandwhethertheyaremovingornot.Mostimportantly,positioninfor-mationisusedtocreateknowncontextsforothervisionprocessesbyensuringthatpeopleareinexpectedregionsoftheroom.TheKidsRoomtrackingalgorithmusestheoverheadcameraviewofthespace,whichminimizesthepossibofoneobjectoccludinganother.Further,lightingisas-sumedtoremainconstantduringthetimethatthetrackerisrunning.Ourroomlightingisdesignedtominimizebright-nessvariationacrossthescene,butinpracticeanobject'sobservedcolorandbrightnesscansigni®cantlychangeasitmovesabout.Backgroundsubtractionisusedtosegmentobjectsfromthebackground,andtheforegroundpixelsareclusteredinto2Dblobregions.Thealgorithmthenmapseachpersonknowntobeintheroomwithablobintheincomingimageframe.Whenblobsmergeduetoproximityoftwoormorechildren,thesystemmapsmorethanonepersontoagivenblob.Thesystemusescolor,velocityestimation,andsizeinformationtodisambiguatethematchwhentheblobslaterseparate.Itisimportantforthealgorithmtokeeptrackofhowmanypeopleareintheroom,whichisachievedbyhavingeveryoneintheroomenterandexitthroughaªdoorºregion.Thecontext-sensitive,non-rigidobjecttrackingmethodisfullydescribedandtestedelsewhere[17].Figure10showstheimageviewusedfortrackingandtheoutputofthetrackingsystem.Eachpersonisrepresented Duringscenetransitions,thelightingvariesandthevisionsys-temisdisabled.Thestoryandtimingofthenarrativearedesignedsuchthatnobodywouldexitduringatransitionandnobodyeverdid;ahumangate-keeperpreventedpeoplefromenteringduringthosetimes.Whenlightingstabilized,there-fore,thesystemknewthenumberofpeopleintheroomonthebedandcouldinitializethetrackeraccordingly. Figure9:Thedancemovesaretaughttothechildrenbythemonstersusingstill-frameanimation.The®rstsequenceshowsonemonsterdoingthespinmove:ªPutyourarmsoutandspinaroundlikeatop.ºThesecondsequenceshowsanothermonsterdoingthe`Y'pose:ªThrowyourarmsupandmakea`Y'.ºThethirdsequenceshowsªFlapyourarmslikeabird,ºandthelastsequenceshowsªCrouchdownandtouchyourtoes.ºbytherectangleboundinghisorherblob.Theboxinthelowerleftcornerrepresentstheªdoorºregionoftheroom,wherepeoplecanenterandexit.Thetrackingalgorithmisnotlimitedtofourpeople,buttheKidsRoomnarrativewasdesignedforamaximumoffourparticipants.7.2MovementdetectionEarlierwemadethedistinctionbetweenmeasuringmovementandrecognizingaction.Astronglyconstrain-ingcontext,however,canallowinferenceofactiondirectlyfrommovement.Forexample,intheriverworldmeasure-mentsofmotionenergyusedinconjunctionwithcontextualknowledgeareemployedtorecognizeparticipant'srowingactions.Theamountofmotiononeachsideofthebedisusedbythecontrolprogramtodecideiftheboatismov-ing(i.e.passengersareªrowingº)andifthepeoplehaveavoidedobstaclesintheriverbyrowingvigorouslyonthecorrectsideoftheboat.TherowingdetectionalgorithmpresumesthateveryoneisªinsidetheboatºÐallonthebed.Thenarrativeen-couragesparticipantstoestablishandmaintainthiscontext(e.g.ªTuckyourhandsandfeetrightin,thehungrysharksareeagertosin.º),andasimplevisionalgorithmbaseduponthesizeofthebedisusedtocon®rmthatthecontextisineffectachieved.Whentheblobsizeisaboutright,everyoneisassumedªintheboatºandthebedorientationiscomputed.Oncethesystemknowseveryoneisonthebedandknowshowthebedisoriented,itcanuseasimpletesttocheckifthereismorerowingontheleftorrightside.Thealgorithmcomputesthepixel-by-pixeldifferencebetweenconsecutivevideoframes.Ifsomeonemovesquickly,alargedifferencebetweenframesisdetected.Thediffer-enceoveraregionistherowingenergy,whichismeasuredoneachsideofthebedandscaledbythenumberofpeopleintheboat.Thecontrolprogramthenusestheseenergymeasurestodetectwhetherornotpeoplearerowingandonwhichsidemostoftherowingisoccurring.Figure11showstheoutputofthesystemwhenapersonisrowingontheleft,right,andbothsidesofthebed.7.3ActionrecognitioninthemonsterworldMoresophisticatedmotionanalysisisusedduringthedancesegmentofthemonsterworldtorecognizethefouractionsofªmakinga`Y',ºcrouching,¯appingandspin-ning.Wechosethesemovesforseveralreasons:theyarefun,naturalgesturesforchildren;theyareeasytodescribeandanimateusingstill-frameanimation;theyareeasytorepeatinaboutthesamewayeachtime;andtheyallowustodemonstrateafewdifferentrecognitiontechniquesusingcomputervision.Eachofthereal-timeapproachesforrecognitionde-scribedbelowareruninparallelasthekidsperformdancemoves.Thevisionsystemreportswhichmovesitthinksisbeingperformed,aswellasitscon®denceinthatas-signment.Allofthevisionprocessesusebackgroundsubtractedimageswhichcontainonlyasilhouetteoftheperson.Theyalsorequireatrainingphasepriortorun-timewhentheactionmodelsareconstructed.7.3.1GeneraldynamicsThe®rstandsimplesttechnique,fordetectingcrouch-ing,usesthesizeofthebackground-differenceblob.Onceinthemonsterworld,theªstandingºblobshapeforapersonisinitializedassoonasthepersonmovesontotherug.Theblobshape,whichismodeledusinganellipsematchedtotheblobdata,iscomparedateachtimewiththeªstandingºmodel.Iftheelongationoftheblobreducessigni®cantly,thealgorithmwillsignalthatacrouchhastakenplace.Fig-ure12bshowsaperson'simageblobandtheellipsemodelforstandingandcrouchingpositions. Forallmoves,thecontrolsystemignoresthemovereportedbythevisionsystemifthetrackinghasindicatedthepersonisnot Figure10:Theleftimageshowsaviewwiththreepeopleintheroomfromtheoverheadcamerausedfortracking.Therightimageshowstheoutputofthetrackingsystem,whichisdescribedandevaluatedelsewhere[17].Allthreepeopleandthebedarebeingtrackedastheymoveabout.Theboxinthelowerleftdenotestheroom'sdoorregion,whereallobjectsmustenterandexit.TheKidsRoomtracksuptofourpeopleandthebed. Figure11:Theseimagesshowthemotionenergythatisdetectedfromtheoverheadcamerawhenapersonisªrowingºastheysitontopofthebed.Theellipserepresentsthepositionandorientationofthebed,extractedbythesystem;thecoloredpixelsindicatewherethesystemdetectedmotion.Theleft,middle,andrightimagesshowrowingontheleft,right,andbothsidesoftheboat,respectively.Theamountofmovementatanytimeiscomparedwiththemaximummovementdetectedsofartocomputehowvigorouslypeoplearerowingandonwhichsideofthebed.7.3.2PoserecognitionThesecondrecognitiontechniqueusestheshapeoftheperson'sbackground-subtractedblobtoidentifywhentheperson'sarmsareraisedupintheairina`Y'.Hereweuseapatternrecognitionapproachtoclassifythebackgroundsubtractedimagesoftheperson.Moment-basedshapefeatures[16]arecomputedfromthetheblobimageslikethoseshowninFigure12dandarestatisticallycomparedtoadatabaseoftrainingexamplesofpeopleªmakinga`Y'.º7.3.3MovementrecognitionThelasttechniqueusedtorecognizemonstermovesusesrecognitionofmotiontemplates[11].Inthismethod,suc-cessivevideoframesofthebackgroundsubtractedimagesofthepeoplearetemporallyintegratedtoyieldaªtemporaltemplateºoftheaction.Templatesforthe¯apandspinmovesareshowninFigure13.Thesetemplatedescrip-tionsrepresentthemovementoversometimeintervalwithasinglevector-valuedimage.Therangeofdurationofin-tegrationisdeterminedbytrainingexamplesoftheactions.Astatisticalmoment-baseddescriptionoftheactiontem-plateisthenusedtomatchtoadatabaseofexamplesofthemoves.7.4EventDetectionInadditiontorecognizinglargebodymotionsofin-dividuals,mostimmersiveenvironmentsneedtobeable ontherug.todetectotherªeventsºiftheyaretoprovideinteresting,reactivefeedback.Forexample,theKidsRoomusestheoutputofthethetrackertoanswerquestionssuchasªIseveryoneinagroup?º,ªIseveryoneonthebed?º,ªIsev-eryoneonthepath?º,ªIseveryonemovingaroundthepathorstandingstill?º,andªIssomeonenearaparticularob-ject?ºTheKidsRoomusesstraightforwardmethodstocomputeanswerstothesequestions.Theªin-a-groupºde-tectorreceivesthepositionofeachpersonfromthevisiontrackerandvalidatesthateverypersoniswithinsomepre-determineddistanceofanotherperson.8StoryControlTechnologiesInadditiontotheperceptualinputtechnology,theKids-Roomhasanarrativecontrolprogram,alightingcontrolprogram,midimusiccontrolprograms,andnetworkingprotocols.8.1NarrativecontrolThenarrativecontrolprogramoftheKidsRoomqueriesthesensorprogramsforinformationaboutwhatishappen-ingintheroomatagiventimeandthendetermineshowtheroomrespondssothatparticipantsareguidedthroughthenarrative.Forexample,whensomeoneenterstheroomthesystemmuststarttrackingthepersonandthecontrolprogrammustimmediatelylearnoftheperson'spresence.Similarly,ifeveryoneleavestheroom,thestorymustfratitscurrentpointinsteadofcontinuingonasiftherewerestillparticipants.Themaincontrolprogramisanevent (a)(c) (b)(d)Figure12:(a)Apersonperformingacrouchmove.(b)Aperson'sbackgrounddifferenceblob.Overlayedontopisanellipsemodeloftheblob.The®rstimageshowsapersoninthestandingposition.Thesecondshowsthesamepersoncrouching.Thedifferenceinelongationoftheellipsemodelisusedtodetectcrouchingmovement.(c)Apersonperforminga`Y'move.(d)Theblobimageusedtodetectthe`Y'move.ThisbinaryimageismatchedtoasetofmodelsofYmovesusingmoment-basedshape-features. Figure13:Twoofthedancemoveactionsarerecognizedusingamotiontemplatematchingmethod[11].Thetopleftimagesshowapersondoinga¯apmove.Thesystemdetectsthe¯apmovebymatchingmotionmodels(whichhavebeencomputingusingadatabaseofexample¯apmoves)tothemotiontemplateshown.Similarly,thebottomimagesshowapersondoingthespinmoveandthecorrespondingmotiontemplate.Thetoppartoftheblobisgeneratedbythemovingarms.Thebottompartisgeneratedbyshadowsfromthearms.IntheKidsRoom,shadowswereincorporatedintothemodelsofthemoves.loop,muchlikethoseinthegameindustryandincom-mercialsoftwarelikeDirectorMAX.Theeventloopcontinuouslymonitorsthestateoftheroom,checkingallinputsasoftenaspossible.Dealingwithreal-time,physicalinteractionrequirescontrolstructuresmorecomplexthanthoserequiredinthetypicalkeyboard-mousesituation,becauseactionstakesomeamountoftimeduringwhichthestateoftheactuatingdevicesmaychange.TheKidsRoomcontrolstructurepar-tiallyhandlesthoseproblemsbyusingthenotionoftimers,andassociatingatimerwitheacheventinterval.Exampleusesoftimersareensuringthatnon-complimentarysoundsdonotplaysimultaneously,thatbackgroundsoundsappearcontinuous,andthatnarrationsarespacedappropriately.Thetimingproblemsweencounteredwillbediscussedinsection10.6.8.2MusicandsoundcontrolTheKidsRoomhasanoriginalscorewrittenforthisinteractiveinstallation.Themusicconsistsof50shortMIDIsegments,manyofwhichcanbeconcatenatedtoformmusicalphrasesthatgraduallyincreaseincomplexity.Theselectionofmusicalsegments,tempo,andvolumeisundercomputercontrolandischangedbasedupontheactionintheroomandtheprogressionofthestory.Computercontrol ofthemusicissuchthatthecontrolprogramcaninterruptmusicabruptlyorattheendofamusicalphrase,dependinguponthesituation.Whenthecontrolprogramcallsforaparticularsoundeffect,thesound®leisstreamedtoaprocessthatadjuststhevolumeofthesoundinthefourspeakerstolocalizethesoundinaregionoftheroomspeci®edbythecontrolprogram.8.3LightingControlThecomputervisiontrackingandrecognitionalgo-rithmsrequirethattheroombewell-litandthatthelightingsettingscanbereliablysetpriortoeachrun.Consequently,speciallightingisusedonlyintransitionsegmentsduringwhichtimethevisionalgorithmsaredisabled.EventhismodestuseoflightingeffectsenhancestheambianceoftheKidsRoom.ThelightingisfullycomputercontrolledusingaMIDIlightboard.Someofourrecenteffortsinusingau-tomatedvisionsystemsintheater[25]useamulti-camerasegmentationmethodthatisinvarianttolightingchangess8.4AnimationControlTocapturetheimaginative¯avorofastory-bookandtopreventthevideoeffectsfromdominatingtheattentionofthechildren,theKidsRoomuseslayered,still-frame,cartoon-likeanimationsequences.Thecontrolprogramrequestsananimation,likeªblue-monster-crouchºforaparticularscreenatacertainframerate(usuallyabout2framespersecond),andseveralframesarestreamedtothedisplay.Abene®tofsuchstory-bookanimationisthatwedonotneedtotightlysynchronizethemotionoftheanimatedcharacterstothatofthechildren,but,likeasto-rybook,thestill-framecartoonscanconveyrichcharacteractivity.8.5ProcessControlTheKidsRoomcontrolarchitectureisbasedonaclient-servermodel.Thecontrolprogramistheclientthatcom-municateswithtenserverstoreceiveinformationaboutthestateoftheroomandtocontroltheoutput.Thesensorserversaretheobjecttrackerserver,themotiondetectorserver,thetwoactionrecognitionservers,andthescreamdetectorserver.Theoutputserversarethedirectionalsoundserver,themusicserver,thelightsserver,andthetwodis-playservers.CommunicationisachievedusingtheRPCprotocol.Theserverarchitecturehasproveneffectiveforallowingdifferentindividualstoworkondifferentcomponentsofthesystemusingthecomputersystemmostappropriatefortheparticulartask.AsnotedbyCoen[8],itiscriticalforanylargedistributedroomcontrolmechanismthatindividualcomponentscanbestoppedandstartedwithoutrequiringarebootoftheentiresystem.III.EvaluationandAnalysisIntheremainderofthispaperweevaluatethetheKids-Roomwithrespecttoourinitialprojectgoalsanddescribeissuesraisedwhichwouldimpacttheconstructionofanysimilarenvironment.9AchievingprojectgoalsWereviewthegoalsofSection2consideringnotonlyhowwellthegoalswereachievedbutalsothein¯uencethosegoalshadonthedevelopmentofvisionalgorithmsandtheoverallsuccessoftheproject.9.1Realaction,realobjectsOneofourprimarygoalswastoconstructanenviron-mentwhereactionandattentionwasfocusedprimarilyintheroom,notonthescreens.Wewantedarichenviron-mentthatwouldwatchwhatthechildrendoandrespondinnaturalways.WebelievetheKidsRoomachievesthisgoal.Childrenaretypicallyactivewhentheyareinthespace,runningfromplacetoplace,dancing,andactingoutrowingandexploringfantasies.Theyinteractwitheachotherasmuchastheydowiththevirtualobjects,andtheirexplorationoftherealspaceandthetransformationofrealobjects(e.g.thebed)addstotheexcitementoftheplay.Wewereonlypartiallysuccessfulintheuseofrealobjectstoenhancetheexperience.TheonlymanipulatedobjectintheKidsRoomisthebed;itisrolledaround,jumpedon,andhidbehind,anditisacriticalpartofthenarrative.However,twomajorobstaclesÐtrackingandnarrativecontrolÐpreventedusfromincorporatingmoreobjectsintotheroom.TheKidsRoomtrackingalgorithmsetsalimitoffourpeopleandonebedinthespacebecausewhenmorepar-ticipantsorlargeobjectsarepresentthespacebecomesvisuallycluttered,debilitatingthetrackingalgorithmandinterferingwithperceptualroutines.Thesecondandper-hapsmoreseriousproblemisthatasmoreobjectsareaddedtoaspace,thebehaviorofthepeopleinthespacewillbe-comelesspredictable,becausethenumberofwaysinwhichobjectsmaybeused(ormisused)increases.Fortheroomtoadequatelymodelandrespondtoallofthesescenariositwillrequirebothespeciallycleverstorydesignandtremen-dousamountsofnarrationandcontrolcode.Themoreperson-objectinteractionsthatthesystemfailstohandleinanaturalway,thelessengagingandsentienttheentiresystemfeels.Tofurtherachievethegoalofthekeepingtheactionontheparticipantssideofthescreens,wedesignedthevisualandaudiofeedbacktoonlyminimallyfocustheat-tentionofthechildren.Typically,virtualrealitysystemsusesemi-realistic,three-dimensionalrenderedscenesandvideoastheprimaryformofsystemfeedback.Wede-cided,however,thatinordertogivetheroomamagical,theatricalfeelandinordertokeeptheemphasisofthespaceintheroomandnotonthescreens,imageswouldhaveatwo-dimensionalstory-booklookandvideowouldconsistofsimple,still-frameanimationsofthoseimages.DuringmuchoftheKidsRoomexperience,thevideoscreensareemployedasmood-settingbackdropsandnotasthecenteroftheparticipants'attention.Audioisthemainformoffeedbackintheroomassounddoesnotrequireparticipantstofocusonanyparticularpartoftheroom.Childrenarefreetolistentomusic,soundeffects,andnarrationastheyplay,runaboutthespandtalktooneanother.Duringthesceneswheresoundistheprimaryoutputmechanism,suchasthebedroomandforestworlds,thechildrenarefocussedontheirownactivityinthespace.Combiningambientsoundeffectswithappropriatemusiccansetatonefortheentirespace.Audiofeedbackcanbefurtherenhancedbyusingspatiallocalization.Evenwithjustfourspeakers,theKidsRoom monstergrowlssoundliketheyarecomingfromtheforestsideoftheroom,andwhenthefurniturespeaksthesoundoriginatesfromapproximatelythecorrectpartoftheroom.9.2RemotevisualsensingIntheKidsRoomtherearenoencumbrancesonorre-quirementsofpeoplewhoenterthespaceexceptthattheymustenteroneatatime.Further,occupantsintheroom(particularlyyoungchildren)aretypicallyunawareofhowtheroomissensingtheirbehavior.Therearenoobvioussensorsintheroomembeddedinanyobjectsorthe¯oor.Thecamerasarepositionedhighabovethespace,welloutoftheline-ofsightandvisibleonlyifsomeoneislookingforthem.Thisenhancesthemagicalnatureoftheroomforallvisitors,especiallyforchildren.Theyarenotpushingbuttonsorsensors,theyarejustbeingthemselves,andtheroomisresponding.9.3Multiple,collaboratingpeopleAnotherdesigngoalwastocreateasystemthatcouldrespondtotheinteractionofmultiplepeople.Sinceself-consciousnessseemstodecreaseasgroupsizeincreases,thekindofrole-playingencouragedbytheKidsRoomismostnaturalandfunwithagroup.Also,whenunen-cumberedbyhead-mounteddisplays,peoplewillnaturallycommunicatewitheachotherabouttheexperienceasittakesplace,andtheywillwatchandmimiconeanother'sbehavior.Forinstance,duringtherowingscene,childrenshouttooneanotheraboutwhattodo,howfasttorow,andwheretorowandplay-acttogetherastheyhitvirtualobstacles.Groupsoffriendsandparent/childpairshaveanespeciallygoodtime.9.4ExploitingandcontrollingcontextGiventhedif®cultyofdesigningrobustperceptualsys-temsforrecognizingactionincomplexenvironments,westrovetousenarrativetoprovidecontextforthevisional-gorithms.Mostofthevisionalgorithmsdependuponthestorytoprovideconstraint.Theboatrowingexamplede-scribedearliertypi®essuchasituation.Itiscurrentlywellbeyondthestateoftheartofcomputervisiontorobustlyrecognizeagroupofcloselysituatedpeoplerowingaboat.IntheKidsRoom,contextmakesitalmosttrivial.Anotherexampleisthemonsterdancescenewherethestorywasconstructedtoensurethateachcamerahasanon-occludedviewofachildperformingthedancemoves.Potentiallyinterferingchildrenarecajoledbythemonsterstostandinlocationsthatdonotinterferewiththesensing.Theadvantageofanactivesystemoverthatofamonitoringsituationistheopportunitytonotonlyknowthecontextbuttocontrolitaswell.9.5Presence,engagement,andimaginationThepowerofacompellingstory-linecannotbeover-statedwhenconstructingaspaceliketheKidsRoomthatintegratestechnologyandnarrative.Theexistenceofastoryseemstomakepeople,particularlychildren,morelikelytocooperatewiththeroomthanresistitandtestitslimits.IntheKidsRoomawell-craftedstorywasusedtomakeparticipantsmorelikelytosuspenddisbeliefandmorecuriousandlessapprehensiveaboutwhatwillhappennext.Thestorytiesthephysicalspace,theparticipant'sactions,andthedifferentoutputmediatogetherintoacoherent,rich,andthereforeimmersiveexperience.Someexistingworkhasstudiedthecriteriathatleadtothefeelingofªimmersionºorªpresenceºinvirtualenviron-ments[28].Herewejustnotethatoursystemmeetseightofthetencriteriacommonlyidenti®edasimportantforcre-atingafeelingofpresenceinavirtualspace[29].OneofthetwocriteriatheKidsRoomdoesnotmeet,ªasimilarityinvisualappearanceofthesubjectsandtheirrepresenta-tioninthevirtualenvironment,ºdoesnotapplytoasystemwherepeopleareinteractingintherealworld.Criteriaforpresencethataremetincludehigh-resolutioninformationbeingpresentedtotheappropriatesenses,freedomfromsensingdevices,easilyperceivedeffectsofactions,anabil-itytochangetheenvironment,andªvirtualºobjectsthatrespondspontaneouslytotheuser.Theremainingunmetcriteria,thatthesystemshouldadaptovertime,isnotmetexplicitly,butasdiscussedlatertheKidsRoomdoesallowtheusertocontinuouslyandnaturallycontrolthepaceoftheexperience.Animmersivespaceismostengagingwhenparticipantsbelievetheiractionsarehavinganeffectupontheenvi-ronmentbyin¯uencingthestory.TheKidsRoomusescomputervisiontoachievethisgoalbymakingtheroomresponsivetotheposition,movements,andactionsofthechildren.Immediatelyupontheir®rstinteractionwiththeroomthechildrenrealizethatwhattheydomakesadiffer-enceinhowtheroomresponds.Thisperceptualsensingenhancesandenergizesthenarrative.AgoalthatwascriticaltoobtainingthepresenceoftheKidsRoomwastonaturallyembedtheperceptualcon-straintsintothestory-line.Forexample,inthemonsterdancescene,thevisionsystemsrequirethatthereisonlyonechildperrugandthatallchildrenareonsomerug.Onewaytoimposethisconstraintwouldhavebeentohavethenarratorsay,ªOnlyonekidperrug.Everyonemustbeonarug.ºInstead,themonsterstellthechildrentostayontherugsandthattherecanbeonlyoneperrugªso'swecanseewhat'sgoin'on.ºThatthemonstershavesomevisualconstraintsseemsperfectlynaturalandmakeschildrenlesslikelytofeelrestrainedortoquestionwhytheyneedtoengageinsomeparticularbehavior.Nomatterhowwellaninteractivestory-lineisdesigned,participants,especiallychildren,willdotheunexpectedÐespeciallywhenthereareuptofouroftheminteractingtogether.Thisunpredictablebehaviorcancausetheper-ceptualsystemtoperformpoorly.Therefore,wedesignedthestorysothatsucherrorswouldnotdestroythesus-pensionofdisbelief.Whenperceptualalgorithmsfail,thebehavioroftheentireroomdegradesgracefully.OneexampleofthisprincipleintheKidsRoomisinthewaythevisionsystemprovidesfeedbackduringthemonsterdance.Ifachildisignoringtheinstructionsofthecharactersandistooclosetoanotherchildonarug,therecognitionofthemovementsofthechildontherugwillbepoor.Therefore,whenactionsarenotrecognizedwithhighcon®dence,themonstersonthescreenwillanimate,doingthelow-con®denceaction,butthemonsterwillnotsayanything.Tothechild,thisjustappearsasthoughthemonsterisdoingitsownthing;itdoesnotappearthattheªmonsterºisanywayconfused.ThischoicewaspreferredoverthepossibilitythatthemonstersaysªGreatcrouchºwhilethechildisactuallyspinning.Similarly,wetriedtominimizethenumberofstorysegmentsthatrequiredaparticularsingleactiononthepart ofallparticipants.Forinstance,togetaparticularpieceoffurnituretospeak,onlyonechildneededtobeinitsproximity;ifallchildrenneededtobeclosetoittheymightneverdiscoverhowtomakethefurnituretalk.Finally,togivetheKidsRoomnarrativeacohesive,im-mersivefeel,therearethematicthreadsthatrunthroughoutthestory.Forexample,thecarefulobserverwillnotethatthestuffedanimalsonthewallsinthechildren'sbedroom,showninFigure4aresimilartothemonstersthatappearlaterinthestory,showninFigure9.Someofthefurniturecharactersinthe®rstworldhavethesamevoicesasthemonstersinthemonsterworld.Theartworkhasthesamestory-bookmotifinallfourscenes,andseveralobjectsintheroomontheshelvesbecomepartoftheforestworldbackdropduringthetransformation.9.6ChildrenassubjectsBuildingaspaceforchildrenwasbothwonderfulandproblematic.Thepositivesincludethetremendousenthu-siasmwithwhichthechildrenparticipate,theirwillingnesstoplaywithpeerstheydonotknow,thedelighttheyex-periencewhenbeingcomplimentedbyvirtualcharacters,andtheircompletedisregardofminortechnicalembarrass-mentsthataroseduringdevelopment.Childrenprovideduniquechallengesaswell.Thebe-haviorofchildren,particularlytheirgroupbehavior,isdif®-culttopredict.Further,childrenhaveshortattentionspansandoftenmoveaboutwithexplosiveenergy,leavingthelongerplayingnarrationsbehind.Youngchildrenaresmallcomparedtoadults,whichcancreateproblemswhende-velopingvisionalgorithms.Inbalance,havingchildrenastheprimaryusergroupnotonlyinspiredustothinkimaginativelybutalso,quitefrankly,madetheprojectallthemorefuntoconstruct.10ObservationsandfailuresThereweresomeissuesthatwefailedtoconsiderinthedesignphaseoftheKidsRoomthatareimportantfordevelopingotherinteractive,immersivespaces,particularlythoseforchildren.Wepresentseveralinanefforttopreventothersfromrepeatingourmistakes.10.1Groupvs.individualactivityTheinteractionintheKidsRoomchangessigni®cantlydependinguponthenumberofpeopleinthespace.First,asmentionedpreviously,allsystemtimingsdifferdependinguponthenumberofpeopleintheroom.Thereisonlyasmallwindowoftimeoutsideofwhicheachunitoftheexperiencebecomestooshortortoolong±andtheidealtimingchangesbasedonthenumberofpeoplearound.Sinceautomaticallysensingwhenpeoplearegettingborediswellbeyondourcurrentperceptioncapability,theKids-Roomusesanadhocproceduretoadjustthedurationofmanyactivitiesdependinguponthenumberofpeopleinthespace.Ingeneral,themorechildrenthereareinthespace,themorefastpacedtheroomappearstobe,becauseassoonasonechild®guresoutthecauseandeffectrelationshipbetweensomeactivityandresponsetheotherchildrenwillfollow.Asinglechildismorehesitantandthereforeneedsmoretimetoexplorebeforetheroominterjects.Also,alonechildoftenrequiresmoreinterventionfromthesystemtoguidehimorherthroughtheexperience.A®nalconsiderationwhendevelopingforgroupactivityistheimportanceofparticipantsbeingabletounderstandcauseandeffectrelationships.Iftoomuchishappeningintheroomandthereisnotareasonableexpectationwithinthechild'smindofstrongcorrelationbetweensomeactionandareasonableresponse,thechildwillnotunderstandthatheorshehascausedtheactiontohappen.10.2Exploratoryvs.scriptednarrativespacesInourinitialdesignoftheKidsRoom,weplannedtocreateaprimarilyexploratoryspace,modeledsomewhatonpopularnon-linearcomputergameslikeMyst[7].Wede-signedandbuiltprototypesforthe®rstandsecondworldsusingthismodel.Inthe®rstworld,therewasnotalk-ingfurniture.Instead,whenchildrenwalkednearobjectstheymadedistinctivesounds:movingneartheshelveswithamirrorwouldmakecrashingsound,steppingontherugswithanimalpictureselicitedthecorrespondinganimalnoises.Ourhopewasthatchildrenwouldenter,®gureoutthattheycouldmakesuchsounds,andthenexploretheroom,graduallycreatingafrenzyofsoundsandactivity.Itdidn'twork.Whenwebroughtinsomechildrenfortestingwefoundthattheydidnotunderstandthattheywerecaus-ingthesounds±therewassimplytoomuchgoingonaseachchildexploredindependently.Thesamewastrueforsomeadults.Evenchildrenaloneintheroomhadtrou-bleidentifyingcauseandeffectrelationships.Wehadalsoplannedtodevelopanexploratorysecondworldusingfor-estsounds,forestimages,andcreepy,exploratorymusic.Again,testingprovedtheconcepttooweak.Themostsigni®cantproblemwasthattheexplorationªplotºdidnotencourageparticularactions,nordiditcausepeopletoactinagroupfashion.Usersdidnotshareanycommongoals.Otherauthorshaveobservedthatex-ploratory,puzzle-solvingspacescansometimesmakeitdif®cultforadultstoimmersethemselvesinaninteractiveworld[14,10].Whenastoryisaddedtothephysicalenvi-ronment,atheatrical-likeexperienceiscreated.Oncethetheatricalnatureofthesystemisapparent,itiseasierforpeopletoimaginetheirrolesand,iftheyarenottooself-conscious,actthemout.Furthermore,recognizingactionissimplerinastory-basedenvironmentbecausethenumberofactionpossibilitiesatanymomentismoreconstrained.primafaciecriticismofalinearstory-lineisthatthesystemlosesitsinteractivenature.Thisisperhapstrueformundaneinterfacessuchasmiceandkeyboards,astherearenodynamicstotheactionsperformed.Foramulti-person,room-sizedenvironment,however,theinteractioncomesfrommakingphysicalexertion,controllingthepacetheadventure,recognizingthattheroomunderstandswhatishappeningandisresponding,andinteractingwithfellowusers.Finally,althoughthescenesandtheªplotºaresimpleandlinear,theactionswithineachscenearenot.Thesystemrespondsappropriatelytouseractionsdependinguponthecontext.Onlythechangesincontextarelinear.10.3Anticipatingchildren'sbehaviorFromtestingwithchildren,welearnedthattherearethreeaspectsofchildren'sbehaviorinthespacethatwehadnotadequatelyconsideredduringnarrativedevelopment. Weintendamorepsychologicallyloadedtermthanspeed. First,thestorymusttakeintoaccountthechildren'sªbe-havioralmomentum.ºTheKidsRoomiscapableofmakingchildrenexceedinglyactive.Bytheendofthe®rstbed-roomworldwhenthefurnitureallstartloudlychantingthemagicword,thechildrenareoftenrunningenergeticallyaroundtheroom.Nextthechildrenenduponthebed,shoutthemagicwordloudly,andthetransformationoc-curs.Thelocationisnowtheforestworldandthechildrenareinstructedandexpectedtoexplore.However,thetrans-formationtypicallycalmsthemdownandtheirtendencyisoftentoremainonthebed.Wefoundthroughtestingthatafairlydirectinstruction(e.g.ªFollowthepath...º),sometimesrepeatedseveraltimes,isrequiredtogetthemtostartmovingagain.Whendesigningforaspacewherephysicalactionisthefocus,behavioralmomentumneedstobeconsidered.Arelatedproblemwastheneedforattention-grabbingcues.Particularlywhenkidsareinastateofhighphysi-calactivity,theyalmostneverhearthe®rstthingthattheroomsaystothem.Sincewedidnotanticipatethisprob-lem,sometimeschildrenmissedimportantinstructions.Wemodi®edthenarrativesothatitrepeatssomecriticalin-structionsmorethanonce.Ideally,weshouldhavebuiltattention-grabbingnarrativeintothestory-lineforeverycriticalnarration.Finally,childrenneedtoclearlyunderstandthecurrenttask.Thelesscertaintheyareofwhattodo,themoreunpredictabletheirbehaviorbecomes.Perhapsbecausetheyhadneverexperiencedaroomsuchasthisbefore,thechildrenseemmoreinclinedtowaitforthingstohappenthantoexploreandtrytomakethingshappen.Thechildrenenjoyedtheexperiencemoreoncethesystemwasmodi®edsothattherewasalwaysacleartaskandwhenthosetaskschangedquickly.10.4AvoidingrepetitivenessandhintsOnewaytobreakthesuspensionofdisbeliefoftheexpe-rienceisforthesystemtoexactlyrepeatasinglenarrationasittriestoencouragesomebehavior.Unfortunately,inaspaceliketheKidsRoombuilttoencouragechildrentophysicallymovearound,instructionsdoneedberepeated.Forexample,inthedancesegmentofthemonsterworld,thecontrolprogramcontinuallychecksifsomeonehassteppedofftheirrugoriftwopeopleareonthesamerug.Ifsome-onedriftsoffarugmorethanonce,narrationisneeded,butrepeatinganarrationjustplayedmomentsbeforeimpartsamechanicalsensetotheresponsesandcausestheentireexperiencetofeellessalive.Onesolutionwedevelopedwastousetwodifferentnarrators.Themainnarratorhasadeep,malevoiceandspeaksinrhymeslikeagrandfatherreadingastory-book.Thesecondnarrator,withasoft,whispered,femalevoice,deliversªhints.ºThe®rsttimesomeonegetsoffarug,themonsterswilltellthepersontogetbackon.Afterthat,however,avoicewhispersahint,ªStayonyourrug.ºThistypeoffeedbackiseasilyunderstoodbyroompartic-ipantsbutdoesnotbreakthe¯owofthestoryandprimarynarration.Thedeliveryofhintsbyadifferentvoiceandtypicallyfromadifferentsounddirectionthanthenarratormadethemperceptuallysalient,increasingtheireffective-ness.Becausethehintswerenotlongrhymingcoupletsitwaseasytohavemultiplephrasestoencourageasinglebehavior,reducingtherepetitionproblem.10.5PerceptuallimitationsSomeperceptual-sensingdif®cultiesandrelatedissuesfollow.10.5.1Perceptually-basedenvironmentalconstraintsAmajorchallengewhendesigningtheKidsRoomwastominimizetheimpactofoursensorsandoutputmodalitiesonthedevelopmentofaninterestingstoryenvironment.Someconstraintsarelistedbelow.Visionalgorithmsgenerallyrequirebrightlighting,butlargeprojectiondisplaysappeardimwhenplacedinbrightspaces.Largevideoscreensdisplayingvideoviolatetheas-sumptionofstaticbackgroundusedbythevisionalgo-rithms.Wewereforced,therefore,tochoosecameraandrugpositionssothatpeopleintheroomwouldneverappearinfrontofascreenintheimageviews±aseriouslimitationforaspaceliketheKidsRoomor(evenworse)theCave[9].Recentworkmotivatedbythisproblemmayalleviatethisconstraint[19,12].Eveninaspaceaslargeas24feetby18feetwithahighceiling,placingcamerassothattheresultingviewswerenon-occludedprovedtricky.Forexample,althoughtheoverheadviewprovidesanon-occludedviewfortracking,wewouldpreferto®ndatrackingsolutionthatdoesnotrequirea20foothighceiling.Further,duetoviewpoint,occlusion,andstorycon-straints,thereisabsolutelyno¯exibilityintheposi-tioningoftheredandgreenrugsandtheircorrespond-ingcameras.Everyspace-relateddecisionrequiredcarefulconsid-erationofimagingrequirements.Forinstance,rugsandcarpethadtobeshort-haired,notshaggy,topre-ventthebackgroundfromchangingaspeoplemovedaround,andobjectswerepaintedina¯atpainttominimizespecularity.ThefourspeakersintheKidsRoomprovidereason-ablelocalizationwhenlistenersarenearthecenteroftheroom.However,whenaparticipantisnearaloud-speakerplayingasound,thatsinglespeakertendstodominatethepositionalperceptandthespatialillusionbreaksdown.Soundlocalizationisimportantifrealobjectsinaspacearetobegivenªpersonalitiesºusingsoundeffects(e.g.asinthebedroomworld),becausetheeffectisdestroyedifthesoundisnotperceivedtocomefromtheobject.TheKidsRoomhasnomechanismsforunderstand-ingspeech,onlyloudness.AddingunencumberingspeechrecognitioncapabilitytoaspaceliketheKids-Roomwithloudmusic,loudsoundeffects,andloudchildrenremainsasigni®cantchallenge.10.5.2Objecttrackingdif®cultiesTheKidsRoomtrackingalgorithmdoesanexcellentjobofkeepingtrackofwherepeoplearebutoccasionallymakeserrorswhenkeepingtrackofpeopleare[17].Inotherwords,thetrackersometimesswapstwopeople,thinking thatoneistheother,butinnormaloperationdoesnotloseapersonaltogether.TheKidsRoom,therefore,wasdesignedwiththeex-pectationthatperfecttrackingofidentitymightbeprob-lematic.Theroomusesinformationaboutwherepeopleare,butwhenreferringtoindividualsitusesenvironmen-talindicators,notabsolutelabels.Forinstance,insteadofsaying,ªGreatjobkidnumberoneºitwillsay,ªGreatjob,kidontheredrug.ºWhiletherearesomeimprovementsthatcouldbemadetothetrackingalgorithm,perfecttrackingofidentityisunlikely.However,unlikeconventionalsurveillancetasks,animmersiveenvironmentthatmustkeeptrackofidentitycanmanipulatepeopleinitsenvironmentsothatwhenitbecomesuncertainofidentityitcanªbootstrapºitselfau-tomatically.Forinstance,asystemcontrollinganenviron-mentwithatelephonemightphysicallycalltheroom,asktospeaktoaparticularpersonand,whenthatpersoncomestothephone,reinitializetracking.Integratingsuchªboot-strappingºdevicesintoanarrativerequirescarefulstorydevelopmentandwillrestrictthedesigner's¯exibility.10.5.3Monsterworldactionrecognitiondif®cultiesAllofthemonsterworldactionrecognitionstrategiesmakeassumptionsabouttheviewingsituation.First,im-agesofthechildcannotbeoccludedbyotherchildren.Carefulcameraandrugplacementminimizedthisproblem,althoughoccasionallywhenlargeadultsenterthespacethesystem'srecognitionperformancewillsufferslightlyduetosmallocclusions.Second,sinceitisdif®culttoremoveshadowsaccurately,shadows(andthereforelightpositions)wereincorporatedintothemotionmodels.Thisturnedouttoyieldamorerobustrepresentationbutrequiresthatthelightingsetupdoesn'tchangebetweenthetrainingphaseandrun-time.Third,themotion-templaterecognitional-gorithmsusedintheKidsRoomarelimitedtorecognizingactionsofindividualpeople,whichpreventstheinteractivenarrativefromexplicitlyrecognizingsomemulti-personaction,suchaspeopleshakinghands.10.5.4Eventdetectionandnon-cooperationOneproblemweencounteredwhendesigningtheKids-Roomwasthatªsimpleºeventsarestronglycontext-sensitiveandmemory-less.Forexample,ourªin-a-groupºdetectorwillsignalfalsecontinuouslyifonemischievouschildrefusestocooperatewiththeremainingchildren.Inthiscase,amorerobustdetectormightignoretherene-gadegiventhatthechildhasn'tbeenfollowingtherulesforawhile.Similarly,ifonechildisscaredandremainsonthebedwhileotherchildrenexploretheforestonthepath,theªin-a-groupºdetectorshouldignorethischildaswell.Weaccommodatedsuchpossibilitiesnotby®xingthedetectors,whichremainsinterestingfutureworkincontext-sensitiveactionrecognition,butbyensuringthatnowheredidthestorystallendlesslyifsomerequestedorexpectedbehaviorwasnotobserved.10.6SensitivitytotimingMultiplepeopleinaspaceincreasesthenumberofpos-siblesituationsandresponsesthatarerequired,therebymakingnarrativetimingcontroldif®cult. GroupactivityintheKidsRoomisalwaysrecognizedusinginputfromthepersontracker,notusingmotiontemplates.Ourlackofanysystematicapproachtocheckingforinconsistenttimingswasmostpainfullyapparentaswetestedthenearly-completedsystem.Sinceadvancingthestoryforwardmanuallyoftencreatestimingproblems,theonlywaytoreallytesttheroomistoputpeopleinitandrunthenarrativerepeatedly.Theproblemisthattheroomprovidesa10-12minuteexperience,andthoroughtestingofeverytimingscenarioisoutofthequestionduetothelargenumberofpossibletimingsituations.Further,oncetheroomistuned,anysmallchangetoanytiming-relatedcoderequireshavingseveralpeoplearoundtointeractinthespace.Ouronlymethodofaddressingthisproblemistocreatemodularstoryfragmentssuchthatthetimersinonefragmentdonotaffectthoseinanother.WenotethatanalternativemechanismtotimersistousetheAIconceptofplanningtomodelthechangeinthestatesoftheworld[3].Recently,Pinhanez,Mase,andBobick[24]haveproposedtheuseofintervalscriptswhereallthesensingandactuatingactivitiesareassociatedwithtemporalintervals,whoseactivationisdeterminedbytheresultofaconstraintsatisfactionalgorithmbasedontheinputfromthesensorsandthepastinteraction.Sucharepresentationmayfacilitateautomaticallycheckingforproblematictemporally-relatednarrativecontrolsituationsof¯ine.10.7CommunicationmodelTheKidsRoomlacksarichmodelofwhatinformationhasbeencommunicatedtotheusersatanygiventimeandhowsequencesofinstructionscanbepresentedtotheusersinanaturalway.Forexample,intheriverworld,therearetimeswhenaroomnarrationispresentinginformationtothechildrenontheboat.Inthemiddleofthatnarration,achildªjumpsoverboardºbygettingoffthebed.Thesystemdetectsthiseventimmediately,buthasnowayofpromptlyindicatingthistothechildren.Thesolutionisnotassimpleascuttingoffthemainnarratormid-phrase.First,abruptlycuttingnarrationsdestroysthesentientfeelofthecharac-ters.Second,evenifanarrationcanbenaturallycutoff(e.g.usingcuephraseslikeªoh!º),thesystemthenneedsamodelofwhatpartialinformationhasbeenconveyedtothechildrenandhowtonaturallypickupthenarrationwhentheoverboardactivityhasended.Signi®canttimemayhaveelapsedduringtheoverboardactivity,forexample,whichrequiresamodi®cationoftheoriginalnarration.Exactlyhowsuchacommunicationmodelshouldberepresentedandusedforreasoningisanopenresearchquestion.10.8WastingsensingknowledgeSometimesthesystemhasthecapabilitytodetectsomesituationbutnowayofinformingtheparticipants.Giventheimpoverishedsensingtechnology,noknowledgeshouldbewasted±allinformationshouldbeusedtoenhancethefeelingofresponsiveness.Forexample,weencounteredthisproblemwhenthechildrenshout.Supposetheroomasksforthekidstoshoutthemagicword;thekidsshoutbutnotloudly.Theroomthenrespondswith,ªTrythatshoutonemoretime.ºInitiallywefeltthatchildrenliketoshout,sowewouldencouragethemdoittwice.Theproblemwasthattheroomrespondedwithambiguousnarration.Aretheydoingitagainbecauseitwasn'tgoodenoughorforsomeotherreason?Worse,ifnobodyshoutedanything,thenarrationmildlysuggestedthattheroomactuallyheardashout.As weimprovedthesystem,weaddedhintsandnewnarrationtotryandensurethatwhenthesystemknowssomething(e.g.howloudascreamis)itletstheparticipantsknowthatitknows.Theeffectisaroomthatfeelsmoreresponsive.Similarly,therearetimeswhenthesystemisdealingwithuncooperativeparticipantsand,despiteseveralat-tempts,hasnotelicitedthedesiredactivity.Another(shout-ing)exampleiswhenthenarratorrequeststhechildrenshoutªBequiet!ºtothemonsters.Ifaftertworequeststhechildrendonotshout,thestorycontinues,ignoringtheirlackofcooperation.However,itisimportanttoexplicitlyacknowledgethatthesystemunderstandsthatsomethingiswrongbutisignoringtheproblem.Inthisexample,thenarratorsaysªWell,Ican'tsaythatwasaveryloudshout,butperhapsthemonsterswill®gureitout.º10.9PerceptualexpectationGiventhetechnologicallimitationsofunencumberedsensing,theinteractionmustbedesignedastonotestab-lishanyperceptualexpectationsonthepartoftheuserthatcannotbesatis®ed.Forinstance,useofsomespeechrecog-nitionintheKidsRoommightproveproblematic.Ifachildoradultseesthattheroomcanrespondtoonesentence,theexpectationmaybeestablishedthatthecharacterscanun-derstandutterance.Anyimmersiveenvironmentwhichencouragesorrequirespeopletotestthelimitsoftheper-ceptionsystemismorelikelytofeelmoremechanicalthannatural.TheKidsRoomisnotentirelyimmunetothisproblem,butwebelievewehaveminimizedtheªresponsivenesstest-ingºthatpeopledobymakingthesystem¯exibletothetypeofinputitreceives(e.g.intheboatscenemostanylargebodymovementwillbeinterpretedasrowing)andbyhavingcharactersinthestoryessentiallyteachthepartici-pantswhattheycanandcannotrecognize(i.e.thealloweddancemovesinthemonsterland).11SummaryandcontributionsTheKidsRoomwentfromwhite-boardsketchestoafullly-operationalinstallationineightweeks.Thispaperhasdescribedthestory,technologyanddesigndecisionsthatwentintobuildingthesystem.WebelievetheKids-Roomisthe®rstperceptually-based,multi-person,fully-automatedinteractive,narrativeplayspaceeverconstructed,andtheexperienceweacquireddesigningandbuildingthespacehasallowedustoidentifysomemajorquestionsandtoproposeafewsolutionsthatshouldsimplifyconstructionofcomplexspacesinthefuture.WebelievetheKidsRoomprovidesseveralfundamen-talcontributions.First,unlikemostpreviousinteractivesystems,theKidsRoomdoesnotrequiretheusertowearspecialclothing,glovesorvests,doesnotrequireembed-dingsensorsinobjects,andhasbeenexplicitlydesignedtoallowformultiplesimultaneoususers.Second,itdemonstratesthatnon-encumberingsensorscanbeusedforthemeasurementandrecognitionofindivid-ualandgroupactioninarich,interactive,story-basedex-perience.Relativelysimplyperceptualroutinesintegratedcarefullyintoastrongstorycontextareadequateforrec-ognizingmanytypesofactionsininteractivespaces.Finally,webelievetheKidsRoomisauniqueandfunchildren'senvironmentthatmergesthemysteryandfantasyofchildren'sstoriesandtheaterwiththespontaneityandcollaborativenatureofreal-worldphysicalplay.CreditsandAcknowledgmentsTheKidsRoomowesmuchofitssuccesstothefactthatitsconceptanddesignwasdeveloped,collaboratively,byalloftheauthors.AaronBobickactedasadvisorandproducer.StephenIntille(interactioncontrol;productionissues)andJimDavis(visionsystems)werethechiefarchi-tectsofthesystem.FreedomBaird,withhelpfromArjanSchÈutte,wrotetheoriginalscript,handledmostnarrativerecording,andprovidedvideodocumentation.ClaudioPinhanezwasinchargeofmusicandlightdesignandcon-trol.LeeCampbellworkedonsoundcontrolandrelatedissues.YuriIvanovandAndrewWilsonwrotetheanima-tioncontrolcode.ComposerJohnKleinwrotetheoriginalKidsRoomscore,andartistAlexWeissmancreatedthecomputerillustrations.Theauthorswouldliketothank:GloriannaDavenportforhersupport;EdMcCluney,JohnUnderKof¯er,Lau-rieWard,andMichelleKemperforcharacternarrations;HugoBarraforworkonMIDIlightcontrol;GregTucker;andLaurieWardforadministrativesupport.WealsoowethankstoOpCodeandMarkoftheUnicornforsoftwaredonations.ThisworkwasfundedbytheMITMediaLab-oratory'sDigitalLifeConsortium.References[1]R.T.Azuma.Asurveyofaugmentedreality.Presence:TeleoperatorsandVirtualEnvironments6(4):355±385,August1997.[2]J.M.Barrie.PeterPan.EPDutton,1988.[3]JosephBates,A.BryanLoyall,andW.ScottReilly.Anarchitectureforaction,emotion,andsocialbe-havior.InProc.oftheFourthEuropeanWorkshoponModelingAutonomousAgentsinaMulti-AgentWorld,S.MartinoalCimino,Italy,July1992.[4]B.B.BedersonandA.Druin.Computeraugmentedenvironments:Newplacestolearn,work,andplay.InAdvancesinHuman-ComputerInteraction,volume5,chapter2.AblexPub.Crop.,Norwood,NJ,1995.[5]AaronF.Bobick.Movement,activity,andaction:theroleofknowledgeintheperceptionofmotion.Phil.Trans.RoyalSocietyLondonB,352:1257±1265,[6]R.A.Bolt.TheHumanInterface.Wadsworth,Inc.,Belmont,California,1984.[7]BroderbundSoftware.Myst.AninteractiveCD-ROM,1994.[8]M.H.Coen.Buildingbrainsforrooms:designingdistributedsoftwareagents.InProc.oftheConf.onInnovativeApplicationsofArti®cialIntelligencepages971±977.AAAIPress,1997.[9]C.Cruz-Neira,D.J.Sandin,andT.A.DeFanti.Surround-screenprojection-basedvirtualreality:ThedesignandimplementationoftheCAVE.InProc.ofSIGGRAPHComputerGraphicsConference,pages135±142.ACMSIGGRAPH,August1993. Figure14:KidsofallageshaveªplayedºintheKidsRoom.Childrenhavebeenwillingtowaitinlineuptoonehourtoexperiencetheroomasecondandthirdtime.[10]G.DavenportandG.Friedlander.Interactivetransfor-mationalenvironments:Wheeloflife.InE.BarrettandM.Redmond,editors,Contextualmedia:multi-mediaandinterpretation,chapter1,pages1±25.MITPress,Cambridge,MAUSA,1995.[11]J.W.DavisandA.F.Bobick.Therepresentationandrecognitionofactionusingtemporaltemplates.Proc.ComputerVisionandPatternRecognitionpages928±934.IEEEComputerSocietyPress,June[12]J.W.DavisandA.F.Bobick.Arobusthuman-silhouetteextractiontechniqueforinteractivevirtualenvironments.InLectureNotesinArti®cialIntelli-gence,volume1537,pages12±25.Springer-Verlag,[13]A.Druin.Noobie:theanimaldesignplaystation.InProc.ofHumanFactorsinComputingSystems(CHI)volume20,pages45±53,1988.[14]A.DruinandK.Perlin.Immersiveenvironments:aphysicalapproachtothecomputerinterface.InProc.ofHumanFactorsinComputingSystems(CHI),pages325±326,April1994.[15]J.GlosandM.Umaschi.Onceuponanobjectcomputationally-augmentedtoysforstorytelling.InProc.oftheInt'lConf.onComputationalIntelligenceandMultimediaApplications(ICCIMA),pages245±249,February1997.[16]M.Hu.Visualpatternrecognitionbymomentin-variants.IRETransactionsonInformationTheoryIT-8(2),1962.[17]S.S.Intille,J.W.Davis,andA.F.Bobick.Real-timeclosed-worldtracking.InProceedingsoftheIEEEComputerSocietyConverenceonComputerVisionandPatternRecognition,pages697±703,LosAlami-tos,CA,June1997.IEEEComputerSociety,IEEEComputerSocietyPress.[18]H.IshiiandB.Ullmer.Tangiblebits:towardsseam-lessinterfacesbetweenpeople,bits,andatoms.InProc.ofHumanFactorsinComputingSystems(CHI)pages234±241,March1997.[19]Y.Ivanov,A.F.Bobick,andJ.Liu.Fastlightinginde-pendentbackgroundsubtraction.InIEEEWorkshoponVisualSurveillance±VS'98,pages49±55,January1998.AlsoappearsasMITMediaLabPerceptualComputingGroupTR#437.[20]M.W.Krueger.Environmentaltechnology:Makingtherealworldvirtual.InCommunicationsoftheACMvolume36,pages36±37,1993.[21]MargotLovejoy.PostmodernCurrents:ArtandArtistsintheAgeofElectronicMedia.AnnArbour,London,England,1989.[22]P.Maes,A.Pentland,B.Blumberg,T.Darrell,J.Brown,andJ.Yoon.ALIVE:Arti®ciallifeinterac-tivevideoenvironment.Intercommunication,7:48±49,1994.[23]A.Pentland.Smartrooms.Scienti®cAmerican274(4):68±76,April1996.[24]ClaudioS.Pinhanez,KenjiMase,andAaronF.Bo-bick.Intervalscripts:adesignparadigmforstory-basedinteractivesystems.InProc.ofHumanFactors inComputingSystems(CHI),pages287±294,March[25]C.S.PinhanezandA.F.Bobick.It/I:Theatrewithanautomaticandreactivecomputergraphicscharacter.Proc.ofSIGGRAPH'98Sketches,July1998.[26]FrankPopper.ArtoftheElectronicAge.ThamesandHudson,London,England,1993.[27]M.Sendak.WheretheWildThingsAre.HarpercollinsJuvenileBooks,1963.[28]T.B.Sheridan.Musingsontelepresenceandvirtualpresence.Presence:TeleoperatorsandVirtualEnvi-ronments,1(1):120±125,1992.[29]M.SlaterandM.Usoh.Presenceinimmersivevir-tualenvironments.InIEEEVirtualRealityAnnualInternationalSymposium,pages90±96,1993.[30]ChristaSommererandLaurentMignonneau.Artasalivingsystem.Leonardo,30(5),October1997.[31]MarkC.Torrance.Advancesinhuman-computerin-teraction:theintelligentroom.InWorkingnotesoftheCHI95researchsymposium,May1995.[32]NaokoTosa,HidekiHashimoto,KaoruSezaki,Ya-suharuKunii,ToyotoshiYamada,KotaroSabe,RyosukeNishino,HiroshiHarashima,andFumioHarashima.Network-basedneuro-babywithrobotichand.InProc.ofIJCAI'95WorkshoponEntertain-mentandAI/Alife,August1995.[33]M.Umaschi.Softtoyswithcomputerhearts:buildingpersonalstorytellingenvironments.InCHIExtendedAbstracts,pages20±21.ACM,1997.[34]WaltDisneyProductions.Bedknobsandbroomsticks.Movie,1971.[35]M.Weiser.Somecomputerscienceissuesinubiq-uitouscomputing.InCommunicationsoftheACMvolume36,pages74±84,1993.[36]ChristopherWren,AliAzarbayejani,TrevorDarrell,andAlexPentland.P®nder:Real-timetrackingofthehumanbody.IEEETrans.PatternAnalysisandMachineIntelligence,19(7):780±785,July1997.