GestureClass ContextGestureTerminal Example Mnemonicick ick letter savesthelePunctuated selfcontained lasso scribble orcrop taporpause deletesinkunderitmnemonic lasso letterorscribble ID: 296907
Download Pdf The PPT/PDF document "gestureswhileinkingbothtomakenovicesawar..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
gestureswhileinkingbothtomakenovicesawareoftheexistenceofgestures,andtoprovidefeedbackduringmulti-strokegesturese-quences.ThebodyofthispaperdetailsFluidInking'scontributionsinthreeareas,including:Pen-basedmenuinteraction:acceleratedbutton-freeinvoca-tion;gesturalshortcuts;andtear-offs.Gesturingwhileinking:mnemonicicks,terminalpunctua-tion,andspeculativefeedback.Gesturesandwidgets:abbreviatedgestures;directmanipula-tion;widgetsfromgestures;pushbuttongestures.2RELATEDWORKTheMathPad2system[13]demonstratedtherudimentarynotionoftap-punctuatedgestures.Thisworkbroadlyextendstheconceptthroughgestureregularization,pausepunctuation,dynamicfeed-backandmenudiscovery.FlowMenus[6],markingmenus[10],andScriboli[7]arerelatedgesturetechniquesthatshareacommonnotionofaradialmenu.EachofthesesystemsiscompatiblewithFluidInkingandcouldbeusedasabackbonefordiscoveringourgestureset.Ineachcase,weperceivethevalueofourgesturesasbeingalternativeandpo-tentiallysimplerandmorenatural“shortcuts”formenuoperations.Inaddition,wenotethattoourknowledgenoneofthesedesignshaveconsideredinteractionusingonlyabutton-freestylus.Moran,etal.[16]presentedasuiteofgesture-basedinterac-tiontechniquestargetingcollaborativeelectronicwhiteboardinter-action,manyofwhicharecloselyrelatedinspirittoours.Yet,becauseoftheiropennessprinciple,whichdoesnottoleraterestric-tionsonthekindsofinkmarksthatcanbedrawn(i.e.,bypre-deningcertaintypesofinkmarksasgestures),theymadeonlyanabortiveattemptatusingapostxindicator(terminalpunctuation)toavoidthemodeswitchbetweeninkingandgesturing[19].Theyencountereddifcultieswhentryingtodistinguishtheirdouble-tappunctuationfromregular“choppy”handwritingandabandonedthetechnique,infavorofpenmodeswhich,astheyreport,comeatthe“costofusershavingtobevigilant.”Theirexperiencenotwith-standing,wedesignedafullgesturesetusingdifferentpunctuationthat,inpractice,isgenerallyunaffectedbyviolationsoftheopen-nessprinciple.Wealsoaddressproblemstheynotedwiththelearn-abilityofalargegesturesetwithourhybridFlowMenudesign.Anumberofsystemshaveexploredthenotionofmodelessges-turaluserinterfacesforinterpretingobjectsinthemiddleoffreeforminput.Theseinterfacesdonotrequirepunctuationbecausetheirdomain-specicdrawingrestrictionsallowgesturestoco-existwithinking.Forexample,Alvarado,etal[1]presentedasystemforcreatingandunderstandingmechanicalsketches,andGross,etal[5],describedcontextualrecognitionofgesturesymbolsfromwithinfree-handdrawings.TheFlyPentopComputer[14],releasedasaproductintheFallof2005,wasdevelopedinparallelwithourwork,anddespiteitslackofanactivedisplayscreen,incorporatesthenotionofaFly-Con—agesturesimilartoourtechniqueforcreatingwidgetsfromgestures.Bothtechniquesuseamnemonicandanencirclinglasso;however,ourdesignsdifferwithregardtopunctuation—theFly-Conrequireseitheradouble-taporapause,whereasoursrequiresanadditionalmimeticormnemonicgesturesymbolforthebuttonaction,followedbyasingletap.Thedifferenceinourdesignscanbeattributedlargelytoourincreasedgesturesetscope—thatis,werequiretheadditionalgesturesymbolbecausewewanttosupportmorethanasingleactiononanencircledmnemonic.Similarly,weavoidtheuseofdoubletapsinageneralgesturecontext,be-causetheirpotentialpresencewouldnecessarilyintroduceadelayinrecognizingsingletaps.Wealsofoundthatsingletapswere,ingeneral,sufcientforrobustrecognition.OtherthanthesimilarityoftheFlyContoourtechniqueformakingwidgetsfromgestures,ourapproachesarequitedifferent.Manyothersystemscombinegesturalinterpretationwithfree-formannotation,suchasLanday'sSILK[12]andMynatt'sFlat-land[18];however,theytypicallyuseexplicitprioractions(e.g.,buttonpress)todisambiguategesturesfromnotes.Animportantexceptionistheclassofsystemsthatusemulti-modalinputtodis-tinguishgesturesfromeachotherandpotentiallyfromfree-formink.Inparticular,speechandgesturehybrids,suchastheseminalworkdoneinQuickSet[4],enableuidtransitionsfrominkingtocommandperformance,butrequirebothsuitablehardwareandasuitableenvironmentforspeechrecognition.SaundandLank[20],however,presentatechniqueforavoidingpriorselectionofmodeduringinking,thatisrelatedbothtoourter-minalpunctuationgesturesandtothegeneralnotionofsuggestiveinterfacesdevelopedbyIgarashi[8].However,theirtechniqueonlyavoidspriormodeselectionforthesub-taskofformingaselection.Also,the“select”buttonmayinterferewithsubsequentdrawingandcanrequirene-grainedtargeting.Mankoff,etal.[15]considergeneraltechniquesforresolvingambiguityinrecognition-basedinterfaces.However,ourdesireformodelessdrawingandgesturingguaranteesthattheoreticallyallgestureswillbeambiguouswithpossiblenotations.Thus,anex-plicitmediation-basedresolutionsystemwouldnotbeappropriatebecauseitwouldnecessarilydisturbinteractionow.Instead,ourworkfocusesonthecomplementaryproblemofhowtodesignges-turesthatwillnot,inpractice,requireambiguityresolution.Buxton[3]notedthevalueofmakingagesturematchthementalmodelofitsactiontounifyacommandandargumentsinasingle“chunk”thatavoidserrorsofsyntax.Althoughthishasbeenoneofmanyimportantcriteriainourdesign,webelievestrictadher-encetohisnotionthatsinglewordorphraseconceptsshouldmapone-to-onetosinglestrokegesturescanresultingesturesthatarephysicallyawkwardorabstract.Thisunistrokephrasingalsodoesnotallowforgeneraluseofmnemonicssincemanysymbols(e.g.,“x”or“t”)naturallyconsistofmultiplestrokes.Finally,ZhaoandBalakrishnan[22]foundrelativeadvantagesformulti-strokeges-tureswhencomparingmultiplelinestoasinglezig-zagformarkingmenus.TheGEditsystem[9]didnotexploretheintegrationoffree-forminkingwithgestures,butdidpresentagesturesetwithseveralpar-allelstoourwork.Itsgesturesetusesamarkingmenu,whichbydenitionprovidesgesturalshortcutsforaccessingmenufunction-ality,althoughnoneofthesystem'snon-markingmenugestureswerepresentedasmenushortcuts.GEditalsousedstyluspauseinectionstodistinguishbetweeninteractiveandnon-interactivecopyanddragoperations,butdidnotuseitinoursenseofpunc-tuationtodistinguishinkfromgesture,nordiditexplorepressure-basedandothertimingoptimizations.AlthoughGEditfocusedonunistrokegestures,itdidhaveasinglemulti-strokeexceptioninthespecicinstanceofdistinguishingagroupselectionfromitsac-tion,andoneexampleofusingalettergesturetoindicateanactionmnemonically.Ourworkexpandsuponbothoftheseconcepts.3OVERVIEWGiventhescopeofFluidInkingitisusefultoconceptualizethesys-temintermsofanalogiestoconventionalWIMPinterfaces.Thuswewilldescribewhatwebelievearethreefamiliarcomponents:amenusystem,shortcuts,andwidgets.First,themenusystemisintendedtobethestartingpointforusersofthesystem.Thatis,anovicewouldusethemenutodiscoverallfunctionality—inourcase,wechosetouseaFlowMenu[6]butbelievethatnoneofourdesigndecisionswouldexcludeaconventionalmenuortool- GestureClass ContextGestureTerminal Example Mnemonicick ick( )letter savesthelePunctuated: self-contained lasso( ),scribble( ),orcrop( )taporpause deletesinkunderitmnemonic lasso( )letterorscribble( )taporpause copiesinkcontainedinthelassomimetic lasso( )strokehook( )taporpause appliesNEmenuoptiontolassocontentsTable1:Overviewofgestureclassicationsequences.Basedonthestructureofgesturesequences,wedistinguishtwoclassicationsofshortcuts(Table1):mnemonicickshaveaprexick,andterminalpunctuationgestureshaveapostxtaporpause.5.1MnemonicFlicksWeadaptedMoyle,etal.'suseoficks[17]toworkinafree-forminkingcontext.Naiveapplicationofickingdidnotworkbecauseoffrequentambiguities,suchastryingtodistinguishaickfromaquicklydrawn`1'.However,ickscanbedisambiguatedfromreg-ularinkingwhenusedaspartofamulti-strokegesture.Mnemonicicksareanextensible,easy-to-learn,gesturemechanismanalo-goustomodier-keys.Amnemonicickconsistsofalinedrawnquicklyfrombottom-lefttoupper-right,followedbyanoverlap-pingmnemoniccharacter(Figure3).Thedirectionalrestrictionisnotstrictlynecessarybut,inourtesting,providedbetterdisam-biguationfromgeneralinkingandwassimplerthanoverloadingickdirectionwithfunctionalmeaning.Representativeexamplesofmnemonicicksareick-Stoshowasavedialog,ick-Utodis-playanUndowidget,andick-etochangethestylusmodetoaninteractiveeraser.Notethatthespatiallocationrequiredbythersttwofunctionsissettothestartoftheick.5.2TerminalPunctuationThestructureofterminallypunctuatedgesturesisstraightforward—withoneortwostrokes,indicateaspatialcontextandages-turefunction,andwithterminalpunctuationinsidetheprecedingstrokes2invokethegesture.Despiteitssimplicity,thisstructureaffordsconsiderabledesignlatitude.Wehaveexploredthreedifferentformsofterminalpunctuation:pausing,tapping,andclickingaspecialhardwarebutton(e.g.,con-trolkey).Wedenepausepunctuationtobeholdingthestylusstill(within6pixels)attheendofagesturestrokeforbetween200msand500msdependingonstyluspressureandgesturecontext.Wedenetappunctuationasafasttappingmotionofthestylusagainstthedisplay.Althoughwebelievethatindividualusersshouldtrainthepauseandtaprecognizersforoptimalresults,wehavefoundthatwriter-independentrecognitionalgorithmsworkreasonablyaslongastheconceptisunderstood(i.e.,tappingisafast“tap”,notadeliberate“press”andreleaseora“dot”).Ourtaprecognizerconsidersanumberoffeaturesincludingamongothersthestroke'sboundingbox,arclength,andtemporalduration.Thesefeaturesenablerecognitionbasedoncorrelationsbetweentaptimingandgeometry;forinstance,asuser'smovethestylusfaster,theyaremorelikelytoinputtapswiththequalitativeappearanceoflinesinsteadofpoints.Last,sincethereinevitablycanbesituationswheretapandpauseinterpretationfails(includingwhendepictingtheFluidInkinggesturesetitself),wetreatclickingthecontrolkeyastappunctuationatthelastknownstyluslocation,andpressing-and-holdingthekeyaspausepunctuation.2Foreaseoflearning,thisruleisstatedgenerally;however,itisonlyen-forcedforsomegestures.Infact,advancedusersmayenableanoptionthatfurtherparameterizessomegesturefunctionalitybasedonthetaplocationwithinoroutsidethepreviousstrokes. Figure4:Left-to-right:Acropstrokeisdrawnindicatingapasteac-tion.Thepasterectangle'slower-leftcornerisalignedwiththecropstroke'scorner.Thepastedselectioniscopiedbydrawinga'C'overit.Inadditiontomerelypunctuatingtheendofagesture,terminalpunctuationcanalsoparameterizegesturefunctionality.Forexam-ple,weusetapsandpausestodistinguishbetweenone-shotandinteractivecommandvariants.Thenotionhereisnotthatpausing,forinstance,isideallysuitedtoinvokinganinteractivecommandvariant,butratherthatitsimpliesthegesturesetdesignandmakesiteasierfortheusertorecallbothbatchandinteractivealternatives.Infact,dependingonapplicationworkow,itmightbemoreap-propriatetomaptapandpauseparameterizationsdifferently,forinstanceifperformingmultipleinteractiveactionsinarowweretheexpectedcase.Nonetheless,webelievethatourmappingfortapandpausepunctuationiseffectivewhenconsideredfromtheperspectiveofgesturerecall.Whenperformanceistheircriteria,weexpectuserswouldavoidthepausealternativebylearningaspecializedwidgetorgesturealternative.Terminallypunctuatedgesturescomeintwoforms:self-containedgesturesindicatebothanoperandandanactioninasin-glestroke,whilecompoundmnemonicandmimeticgesturessepa-rateoperandandactionspecicationintotwoormorestrokes(Ta-ble1).5.2.1Self-containedgestures.Examplesofself-containedgesturesarethepaste,selectanddeletegestures.Thepastegestureisindicatedbydrawingacropmark(e.g., )thatspecieswheretoplacethecorrespondingboundingboxcorneroftheclipboardcontents3andthentappingorpausingwithinthecrop'sconvexhull(seeFigure4).Therequirementfortappinginsidethecropmarksisalwaysenforcedtoavoidconictswithgeneralinking;forinstance,whenwritingthenumber“7.0”.Pausepunctuationengagesavariantofpastebypresentingamenuofpastebuffers.Theselectgestureisindicatedbydrawingalasso4aroundagroupofobjects.Tappunctuationisnotrequiredtobewithinthelasso.Pausepunctuationselectsthestrokesandthenraisesanin-teractiveFlowMenu.3Thecropmarkmaybedrawninanyofthefourcardinalorientations( , , , );conceptually,theuserisdrawingonecorneroftheboundingboxofthethingstobepasted,anditisthiscornerofthatboundingboxwhichisalignedwiththecornerofthecropstroke.4Anycurvetravelingatleast25%ofitsarclengthawayfromitsstartandthenreturningwithin25%ofitsarclengthfromitsstart. Figure8:DrawingthroughaTranScalerwidgetfromoutsidescalestheselectionaboutthecorneroppositethehandle.Drawingfromwithinmovestheselection.astopthanapause.Ifgesturesareforgotten,theself-containedtechniqueforbringingupamenu,press-and-hold,canbeusedwiththeshortened200mstimeout.Wenotethat,ignoringpunctuation,somegesturesmaybesub-setsofothergestures.Inthesecases,gestureabbreviationscauseconictswhich,inourcurrentimplementation,gesturesetdesign-ersmustavoid.Weconsideredusingtimeoutstoallowthelongergesturesequencestobeentered,butwewerenotsatisedwiththedelaythisimposesonrecognitionofshortersequences.6.2DirectManipulation.Tomoveink,weinitiallyprovidedonlyatap-basednon-interactivemimeticgestureanditspause-basedinteractivecounterpart.Nei-therprovedsufcientsincemimeticmovementdidnotaffordpre-cisionandpausingwasdisruptiveforsomeusers.WedesignedtheTranScalerwidgettoaddresstheseshortcomingsandtoaddscal-ing.Whenaself-containedselect(orpaste)gestureismade,aninter-actionwidgetisinstantiatedatthetap(orpause)location.SimilartoApitz,etal.[2],thiswidgetleveragescrossings,however,weusethemtomakeanadditionalscalingmodeavailableonanoth-erwisetraditional(i.e.,pressanddrag)movehandle.Toscale,thestylusmuststartoutsidethewidgetandcrossintoitalonganap-proximatelystraightline.Whenacrossingisdetected,theselectioninteractivelyscalesaboutthecorneroppositethewidget(orapre-viouslychosenpoint)(Figure8).Thewidgetis30pixelssquare,whichislargeenoughtosupportquicktargetingwithapen.How-ever,wenoticedthatuserswantingtoquicklymoveanobjectwouldperformthelassoandtapgesturefollowedimmediatelybyadragstroke.TheywouldthenoftenmisstheTranScalerwidgettargeteventhoughitwaslocatedrightwheretheirstylushadbeenattheendofthetap.Thisproblemarosebecausetheycoarticulatedthebeginningoftheirdragmotionwiththeendoftheirtapstrokeasthestyluswasliftingoffthedisplay.Bythetimethestyluscontactedthedisplayagainforthedrag,ithadalreadymovedsomedistancemakingitdifculttoreexivelytargetthewidget.Weaddressedthis“recognition”errorbydoublingthepickareaofthewidgetfortherst150msaftertheselectiontap.AsecondpotentialconictwiththeTranScaleroccurswhenanintendedgestureaccidentallystartsontheTranScalerorpassesthroughitalonganinitialstraightline,causinginadvertentdragsandscales.TheseconictscanbeavoidedbyinitiallytappingawayfromtheselectionorbyremovingtheTranScalerbyscribblingoverit.6.3WidgetsfromGesturesTocomplementtheabilitytotear-offamenuitem,wealsoprovideamechanismforagesturetobedirectlyconvertedintoawidgetthatresemblesLanday'stechniques[12]forsketchinginterfaces.WeextendLanday'sworkbyfacilitatingend-usersincreating,us-ing,anddeletingmini-interfacesontheywhileinafree-formink-ingcontext.Inadditiontopossiblybeingmoredirectthantearingoffmenuitems,creatingwidgetsfromgesturesnaturallyindicatesthesize,andlocationofthewidget.Widgetscanbecreatedforanygesturebyinserting,beforethegesture'sterminalpunctuation,alassoaroundthegestureandwriting`B'(orchoosing“Button”fromthemenu.)Forexample,adeletebuttonwouldbecreatedbyrstdrawingadeletegesturescribble,thenlassoingit,writing`B',andtapping.6.4PushbuttonGestures.AllbuttonsinFluidInkingareFluidButtonswhich,inadditiontobeingselectableanddeletablelikeinkstrokes,supportadditionalgesturalbehavior.Forexample,insteadofusingapausetobringupaFlowMenu,theFlowMenu'sbehaviorcanbemappedtoaFluid-Buttontoprovideinstantaccess,orapastebuttoncanbedragged-and-dropped.Inaddition,wesupporttwoclassesofslide-offbe-haviors:dragselection,andparameterrenement.Aslide-offbe-havioristriggeredwhenthestyluspressesdownonabuttonandtravelsmorethan10pixelshorizontallyfromthecontactpoint.Ver-ticaldisplacementsareignoredsothatbuttonpressescanbecan-celledbyreleasingaboveorbelowthebutton.6.4.1DragSelectionDragselectionbehaviorsareusedforbuttonsthathaveanessen-tiallyunparameterizedaction,suchascut,delete,etc.Inthesecases,sliding-offtriggersaselectionmechanismthatbacktracksthroughtimeaddingpreviousstrokestotheselectionsetasthesty-luscontinuestoslide.Forexample,sliding-offa“cut”buttonini-tiallyselectsthelaststrokedrawn,thenthestrokebeforeit,etc;whenthestylusliftsallselectedstrokesarecut.6.4.2ParameterRenementParameterrenementbehaviorsareusedforbuttonsthatspecifyscalarparameters.Clickingonthebuttontypicallysetsthedefaultvaluefortheparameter,generallythelastvalueset.Sliding-offthebuttoncreatesavirtualsliderforadjustingthescalarvalue.Inourprototype,weuseaparameterrenementFluidButtonforUndo/Redo;slidingtotheleftperformsanundoactionevery10pixels,andslidingtotherighttriggersredoactions.Forcolorset-tingbuttons,slidingoffthebuttonaddsorsubtractsthecolorfromtheselectedstrokes.7DISCUSSIONThefocusofthispaperhasbeenontechniquesthatcanbeper-formedwiththemostgeneralhardware,abutton-freestylus.Thisfocuswasmotivatedlargelybyearlierreactionstoaseriesofde-signsweprototypedforspecializedhardware.Inparticular,wewatchedpeopleusinggesturalinterfaceswhileeatingorholdingacellphone—bothimpracticalwithanyofourbimanualtech-niques.Wealsoobservedmanyusers,includinggestureexperts,strugglewithstylus-buttontechniquesespeciallyiftheyhadtoad-justtheirnaturalpengrip.Ontheotherhand,wenoticedmanyusersfarewellwithspecializedhardware.Ourinformalobserva-tionsrevealedonesalienttrend—thatthedistributionofindividualpreferenceseemedsomewhatbalanced.Somepeoplelikebuttonsforgoodreasons,andothersdonotforjustasgoodreasons.Con-sequently,weprovideabalancedgesturedesignthatcanbeusedwithoutanyhardwarebuttonsatthecostofpotentialrecognitionerrors,orwithhardwarebuttonsatthecostofgeneralityandawk-wardnessforsomeusers. [12]JamesA.Landay.SILK:sketchinginterfaceslikekrazy.InConfer-enceonHumanFactorsinComputingSystems,pages398–399.ACM,1996.[13]JosephJ.LaViola,Jr.andRobertC.Zeleznik.MathPad2:asystemforthecreationandexplorationofmathematicalsketches.InPro-ceedingsofthe2004SIGGRAPHConference,pages432–440.ACMSIGGRAPH,2004.[14]LeapFrogEnterprises,Inc.TheFlyPentopComputerUserGuide.PartoftheFlyPentopComputer,2005.[15]JenniferMankoff,ScottE.Hudson,andGregoryD.Abowd.Inter-actiontechniquesforambiguityresolutioninrecognition-basedin-terfaces.InSymposiumonUserInterfaceSoftwareandTechnology,pages11–20.ACM,2000.[16]ThomasP.Moran,PatrickChiu,andWilliamvanMelle.Pen-basedinteractiontechniquesfororganizingmaterialonanelectronicwhite-board.InSymposiumonUserInterfaceSoftwareandTechnology,pages45–54.ACM,1997.[17]MichaelMoyleandAndyCockburn.Thedesignandevaluationofaickgesturefor`back'and`forward'inwebbrowsers.InTheFourthAustralianUserInterfaceConference,pages39–46,2003.[18]ElizabethD.Mynatt,TakeoIgarashi,W.KeithEdwards,andAnthonyLaMarca.Flatland:Newdimensionsinofcecomputing.InConfer-enceonHumanFactorsinComputingSystems,pages346–353.ACM,1999.[19]E.Pedersen,K.Mccall,T.Moran,andF.Halasz.Tivoli:Anelectronicwhiteboardforinformalworkgroupmeetings.InConferenceonHu-manFactorsinComputingSystems,pages391–398.ACM,1993.[20]EricSaundandEdwardLank.Stylusinputandeditingwithoutpriorselectionofmode.InSymposiumonUserInterfaceSoftwareandTechnology,pages213–216.ACM,2003.[21]RobertZeleznik,TimothyMiller,LoringHolden,andJosephJ.LaVi-ola,Jr.Fluidinking:Usingpunctuationtoallowmodlesscombinationofmarkingandgesturing.TechnicalReportCS-04-11,BrownUni-versityComputerScienceDepartment,box1910,BrownUniversity,Providence,RI02912,USA,July2004.Availableatftp://ftp.cs.brown.edu/pub/techreports/04/cs04-11.ps.Z.[22]ShengdongZhaoandRavinBalakrishnan.Simplevs.compoundmarkhierarchicalmarkingmenus.InSymposiumonUserInterfaceSoft-wareandTechnology,pages33–42.ACM,2004.