Figure1:Fivesnapshotsinthedesignofacaronahillinmechanicalengineering.N - PDF document

Figure1:Fivesnapshotsinthedesignofacaronahillinmechanicalengineering.Notethatthetopofthegroundisdrawnrst,followedbythecarbase,toensurethattheyareparallel.Thenwheels(whichactastheconnectionbetweenthecarandtheground)wereadded.Thewheelswereattachedtothecaronlyafterthecarwascompleted.Therststartedobject,theground,wascompletedlastsinceitspurposeincompletionwasmoreintermsofgeneralfunctioningwhenattachingittoaCADsystemthanindesign.trainingset.Whilethesystemsaboveallowlimitedamountsofinterspersing,theystillrestricttheuserinawaythatseemsunnecessary,giventhefactthathumansdon'trequiresuchrestrictionsinordertorec-ognizeshapes.Wecannotsimplyignoretheneedforinterspersing.Inmanydomains,userswillinterspersestrokes,startingtodrawoneshape,beforenishinganother.Inourexperienceobservinguserssketchinavarietyofdomains,wehavefounditnotuncom-monforsomeonetodrawpartofashape,stop,continuedrawingotherobjectsindiagram,andthencomebacktonishtheorigi-nalshape.Forexample,InUMLclassdiagrams,usersinterspersedstrokestoimproveoveralllayout.Additionally,theauthorshavewitnessedinterspersinginsoftwaredesign,whereUMLclassdia-gramdesignerssometimesinitiallydrawconnectionsassimplede-pendencyassociations,untilmostoftheclasseshavebeendrawn,atwhichpointtheywillhaveabetterunderstandingoftheemergingdesign,andmakeabetterdecisionaboutthetypeofassociationthatwouldbeappropriatebetweentwoobjects.Thiswillcausethemtoadddifferentkindsofarrowheadstotheassociationsdrawnearlier,producinganinterspersedsketch.Theauthorshavealsowitnessedinterspersinginelectricalengineering;sketchersoftenaddvoltagedirectionsonlylaterinthedesignprocessaftermostofthecircuithasbeendrawn.Innitestatemachines,arrowheadswereoftenaddedasanafterthought.Figure1showsanexampleofinterspers-inginmechanicalengineering.Ourgoalistomakesketchsystemsasnaturalaspossiblebyplacingasfewrequirementsontheuseraspossible.Recognitionerrorsduetointerspersingtendtobemoredisturbingtousersthanthoseduetomessydrawingsince,fromthehuman'sperspective,theshapeisclearlydrawnandobvious.Thispaperpresentsanindexingtechniqueforsketchrecognitionthatexaminesallsubsetswithinthe2powersetspace(whereisthenumberofshapesonthescreen)whenattemptingtorecognizenewshapes,butusesefcientindexingtokeeprecognitionperformanceclosetoreal-timethroughfastpruning.Thispaperalsoreportstimingdatathatsupportstheclaimthattherecognitionofnewshapescanbekeptclosetoreal-timeevenwhenallpossibleshapesubsetsare2RECOGNITIONWehavechosentoperformrecognitionusingageometricmethodthatcomposesshapeshierarchicallyfromtheirsubshapes,testingthegeometricconstraintsbetweenthem.Forthemostpart,therecognitionparadigmissimilartothoseusedin[HD05].However,ourapproachinthatourrecognitionalgorithmdoesnotplaceinter-spersingrequirementsontheuser,butstillperformsrecognitionisreal-timedespiteanexponentialqueryspace.2.1GeometricConstraintsOurindexingtechniqueworksbyindexingthegeometricprop-ertiesofashapeforuseinrecognition.Wehavecreatedavo-cabularyofgeometricconstraints(Table2.1),somewhatsimilartothoseusedby[HD05],withwhichtodescribeshapes,suchas Orientation-Con-straints: HORIZONTALVERTICALNEGS-LOPEPOSLOPEABOVELEFTHORIZLIGNVERTLIGNPOINTSOWNPOINTSEFTPOINTSIGHT,andPOINTS Orientation-Independent ACUTEACUTEIRACUTEEETBISECTSCOINCIDENTCOLLINEARCONCENTRICCONNECTSCONTAINSDRARDEREQUALNGLEEQUALREAEQUALENGTHINTERSECTSLARGERLONGERMEETSNEAROBTUSEOBTUSEIROBTUSEEETONNEIDEOPPOSITEIDEPARALLELPERPENDICULAR,andSAMEIDE CompositeConstraints(subset): SMALLERBELOWRIGHTABOVEEFTABOVEIGHTBELOWEFTBELOWIGHTCENTEREDBOVECENTEREDELOWCENTEREDEFTCENTEREDIGHTCENTEREDLESSHANLESSHANQUAL OtherCon- EQUALGREATERHANGREATERHANQUALORNOT Table1:GeometricConstraintsPERPENDICULAR.Constraintsmaybeeitherorientation-dependentororientation-independent.ThevocabularyalsoincludesEQUALGREATERHAN,andGREATERHANQUAL,allowingcompari-sonofanytwonumericpropertiesofashape(e.g.,statingthattheheightisgreaterthanthewidth).Thevocabularyalsocontainsanumberofconstraintsthatcanbecomposedfromotherconstraints.Weincludethesetosimplifydescriptionsandtomakethemmorereadable.TheconstraintmodiersORNOTarealsopresenttoallowdescriptionofmorecomplicatedconstraints. Figure2:Anarrowwithanopenhead.Figure3showsthegeometriccomponents(threelines)andtheconstraintsthatmakeupanOPENRROW.Newshapesaredenedhierarchicallyintermsofpreviously-denedshapesandthecon-straintsbetweenthem.Forexample,Figure5showsashapede-scriptionforaTRIANGLERROWcreatedfromanOPENRROWandaline.Geometricconstraintsdenetherelationshipsbetween sectioninashapedenitionisavailabletosimplifyhierarchically-deneddescriptions,allowingcomponentstoberenamedforeaseofuselater.Forexample,theOPENRROWdenitionhasaliasesfor defineshapeOpenArrow"Anarrowwithanopenhead"Linehead1LineshaftLinehead2coincidentshaft.p1head1.p1coincidentshaft.p1head2.p1coincidenthead1.p1head2.p1equalLengthhead1head2acuteMeethead1shaftacuteMeetshafthead2Pointheadshaft.p1Pointtailshaft.p2Figure3:Thegeometricdescriptiontorecognizeanarrowwithanopenhead,asshowninFigure2. Figure4:Anarrowwithatrianglehead.thosecomponents.Forexample,theOPENRROWshapedenitionrequiresthattheHEAD1andSHAFTmeetatasinglepointandformanacuteanglefromlineHEAD1tolineSHAFTwhentravelinginacounter-clockwisedirection.2.2SignalNoiseversusConceptualVariationsShapesareautomaticallyrecognizedbasedonthegeometricde-scriptionoftheshape.Therecognitionhandlessignalnoise,butconceptualvariationsmustbeincludedintheshapedescriptions.Bysignalnoisewemeantheunintentionaldeviationsintroducedintoashapebytheimprecisionofhandcontrol.Forinstance,whendrawingasquare,allfoursidesmayturnouttobeofdifferentlengthseventhoughthesketchermeantforthemtobethesamelength.Byconceptualvariations,wemeantheallowablevariationsinasymbolthataredrawnintentionally.Forexample,acapacitorinanelectroniccircuitmaybedrawnastwoparallellines,orasonestraightandonecurvedline(seeFigure6).Inoursystem,signalnoiseishandledbytherecognitionsystem.Forexample,thesystemcansuccessfullyrecognizeaquadrilateralwithunevensidesasasquarebecausetheEQUALENGTHstrainthassomebuilt-intolerance(discussedbelow).Thus,shapesshouldbedescribedtothesystemwithoutaccountingforsignalnoise,i.e.,asifdrawnperfectly(e.g.,asquareshouldbedescribedashavingequallengthsides).Asthesystemdoesnotautomaticallytakeintoaccountthepossibleconceptualvariations,theymustbeprovidedforintheshapedescriptions(e.g.,thetwodifferentwaysacapacitorcanbedrawnmustbespeciedintheshapedescriptions).Othersignalnoiseincludesasketcherintendingtodrawasingleline,butusingseveralstrokestodoso.Inorderforthesystemtodealwiththesephenomena,itrstjoinslinesbymergingoverlap-pingandconnectinglines.Figure7showsthestepsthatgointorecognizingasquare.Figure7ashowstheoriginalstrokes.Fig-ure7bshowstheoriginalstrokesbrokendownintoprimitives.Thesystemhasrecognizedthestrokesaslinesorpolylines;thegureshowsthestraightenedlinesthatwererecognizedbytherecogni- HEADTAILtosimplifytheTRIANGLERROWdefineshapeTriangleArrow"Anarrowwithatriangle-shapedhead"OpenArrowoaLinehead3coincidenthead3.p1head1.p2coincidenthead3.p2head2.p2Lineshaftoa.shaftLinehead1oa.head1Linehead2oa.head2Pointheadoa.headPointtailoa.tailFigure5:Thegeometricdescriptiontorecognizeanarrowwithatriangle-shapedhead,asshowninFigure4. Figure6:Acapacitorcanbedrawnwithtwolinesoralineandacurve.tionsystem.(Thedotsrepresenttheirendpoints.)Figure7cshowstheprimitives(linesegments,inthisexample)joinedtogethertoformlargerprimitives(againlines,inthisexample)usingthemerg-ingtechniquesdescribedabove.Figure7dshowsthehigher-levelrecognitionperformedontherecognizedshapes;themethodforthisisdescribedintthenextsection.Ahigher-levelshapecanthenusethesquareasoneofitscomponents.2.3ConstraintTolerancesInourapproach,signalnoiseishandledbytheshaperecognizerbygivingeachconstraintitsownerrortolerance,chosentobeascloseaspossibletoperceptualtolerance,i.e.,thetolerancethathumansuse.Humanperceptualtoleranceiscontext-dependent,dependingonboththeshapeinquestionandothershapesonthescreen.Ta-ble2.3showsconstraintsandtheerrortolerancesweuse.Notethatsomeconstraintshaveanabsolutetolerance,whileothersarerela-tive.Someconstraintshaveanegativetolerance,whichmeanstheconstrainthastobenotonlygeometricallysatised,butalsoper-ceptuallysatised,meaningthathumanshavetobeabletoperceivethattheconstraintistrue.Togiveanexample,ashapethatisleftofanothershapebyonepixelisgeometricallyleftofanothershape,butisnotperceptuallyleftofanothershape,asitdifcultforahu-mantoperceivesuchasmalldistance.Toensurethataconstraintperceptuallysatised,weaddabufferzonetothetolerance.Perceptualerrortolerancesweredeterminedempiricallyandbasedfromgroupingrulesandsingularitiesfromgestaltprinci-ples[Gol72].Groupingrulesattempttomimichowpeoplepercep-tuallygroupobjects,usingconceptssuchasconnectedness,near-ness,andotherprinciples.Singularitiesdescribewhichgeomet-ricalshapepropertiesaremostnoticeable.Forinstance,humansareparticularlysensitivetohorizontalandverticallinesandcanquicklylabelalineashorizontalornothorizontal,andidentifyifalinedeviatesfromhorizontalorverticalbyaslittleasvede-grees.Humanshaveconsiderablymoredifcultyidentifyinglinesatotherorientations,suchas35degrees,andwouldhaveamuchmoredifculttimedeterminingifaparticularlinewas30,35,or40degrees.Humanstendtogrouptogethertheanglesbetween15and75degreesaspositively-slopedlines[Gol72]. Figure7:Stagesofsquarerecognition;a)originalstrokes,b)primitiveshapes,c)joined/cleanedprimitives,andd)higher-levelrecognition.[HD02]Tocalibratehowsensitivepeopleare,weshowedninepeopleatotalof116linesinarandomorientationbetween0and180de-grees.Userswereaskedtoreporttheorientationofthelineasaccu-ratelyaspossible.Wegroupedthelinesintotwogroups1)angleswithorientationswithin10degreesofhorizontalorvertical(0-10,80-100,and170-180degrees)and2)angleswithorientationsnotwithin10degreesofhorizontalorvertical(10-80,100-170).Whenlabelinglinesfromgroup1(nearhorizontalorvertical),usershadameanerror(absolutevalueofthereportedangleminustheactualangle)of2.8degreesandavarianceof4.95.Whenlabelinglinesfromgroup2(farfromhorizontalorvertical),theusershadameanerrorof7.64andavarianceof25.77.Asshownbythevariance,largeerrorsbetweentheactualorientationandthecorrectorienta-tionwerecommon:24lineshadanerrorgreaterthan10;8lineshadanerrorgreaterthan15;and2lineshadanerrorgreaterthan20.Thetwogroupsweresignicantlydifferentwithapvalueoflessthan.001.Ourusergroupcontainedbothwomenandmen,aswellasavarietyofcomputerscientists,otherengineers,andnon-mathematicallyconcentratedprofessionalsinthestudy.Thegroupswerenotstatisticallydifferentinoursmallsample.Whendrawinganidealpositively-slopedline,userstendtodrawananglecloseto45degreesandwhendrawingaidealhorizontallinetheytendtodrawananglecloseto0degrees.However,sincedifferencesaremuchmoreperceptuallyimportantclosetohori-zontal,wehavemadthetolerancelessfornearsingularities(suchHORIZONTALVERTICALPARALLEL,andPERPENDICULARBasedonthemeanerrorofonly2.8degreesforanglesingularitiesshowninthefootnote,onemayexpectthesingularitytolerancestobeevensmaller.However,becausetheperceptualerrortolerancesareusedtoremovepossibleshapesfromourrecognitionresults,wechooseerrortolerancesthatareslightlysmaller.3INDEXINGLGORITHMOurcurrentimplementationisdoneinthreestages:1)domain-independentprimitivending,2)domain-independentconstraintindexing,and3)domain-dependentshapeformation.3.1Domain-IndependentPrimitiveFindingLow-levelrecognitionisperformedoneachstrokewhenitisdrawn.Duringprocessing,eachstrokeisbrokendownintoacollectionofprimitiveshapes,includingline,arc,circle,ellipse,curve,point,andspiral,usingtechniquesfromSezgin[SSD01].Cornersused horizontal angle 10degrees vertical angle 10degrees posSlope angle 35degrees negSlope angle 35degrees coincident xlocation 10pixels coincident ylocation 10pixels bisects xlocation (length/4)pixels bisects ylocation (length/4)pixels near xlocation 50pixels near ylocation 50pixels concentric xlocation (width/5)+10pixels concentric ylocation width/5+10 sameX xlocation 20pixels sameX width 20pixels sameY ylocation 20pixels sameY height 20pixels equalSize size (size/4)+20pixels parallel angle 15degrees perpendicular angle+90 15degrees acute angle+40 30degrees obtuse angle+130 30degrees contains minX,Y&maxX,Y -5pixels above ylocation -5pixels leftOf xlocation -5pixels larger size -5pixels Table2:Constraintsandtheirerrortolerances.(Note:size=lengthofdiag-onalofboundingbox.Weusethisformulainsteadoftheareatoenableustocomparelinestotwo-dimensionalshapes.Rubineusesthesameformulaforsizewhencalculatingfeaturenumberve.[Rub91])forsegmentationarefoundusingacombinationofspeedandcur-vaturedata(asinSezgin[SSD01]).Bybreakingstrokesdownintotheseprimitives,andperformingrecognitionwithprimitives,wecanrecognizeshapesdrawnusingmultiplestrokes,andhandlesit-uationsinwhichasinglestrokewasusedtodrawmultipleshapes.Ifastrokehasmultipleprimitiveinterpretations,allinterpreta-tionsareaddedtoapoolofinterpretations,butasingleinterpreta-tionischosenfordisplay.Forexample,boththeLINEandARCinterpretationoftheSTROKEinFigure8Awillbeaddedtothepoolforrecognitionusinganyoftheinterpretations.(Appropriatebookkeepingisperformedtoensurethatmultipleinterpretationsareeffectivelymanaged.)Incaseswhenastrokeisdeterminedtobecomposedofseveralprimitives,(e.g.,thePOLYLINEinterpretationinFigure8B),theTROKEissegmented,thesegmentedsubstrokesaddedascompo-nentsoftheoriginalfullSTROKE.Furtherinterpretationscanuseeitherthefullstroke,astheCURVEdoesinFigure8B,oroneormoreofthesegmentedsubstrokes.Thisallowsseveralshapestobedrawnwithasinglestroke.3.2Domain-IndependentConstraintIndexingWewouldprefertoplaceasfewdrawingrequirementsaspossibleonthesketcher,andmustasaconsequencendawaytodealwiththeexponential.Whileoursolutiondoesnoteliminatetheexpo-nential,wecanuseindexingtohelppruneasignicantnumberofshapepossibilitiesquickly.Theindexingprocessoccursonlyonceforeachshape,whenitstrokeisdrawnornewshapeisrecognized.Eachlowlevelshapeisindexedaccordingonlytoitsownpropertiesandnotintermsofthosearoundit,andthus,thetimeittakestoindexalloftheshapesislogarithmicintermsofthenumberofshapesonthescreen.(Itislogarithmicbecauseindexinginvolvestheinsertionofshapesintoasortedhashmap,whichtaketimeoflogn.)Foreachnewshapethatisrecognized,asignicantportionoftherecognitiontimeis Figure8:Multipleinterpretationsandtheiridentiersareaddedtotherecognitionpool.InA,Stroke0hastwointerpretations;Line1andArc2areeachcomposedfromStroke0.InB,Stroke0canbebrokendownintothreeStrokes(1,2,3).Stroke0hastwointerpretations,Curve4composedofStroke0(andthusalsoStrokes1,2,3)andthreelines:Line5composedofStroke1,Line6composedofStroke2,andLine7composedofStroke3.spentperformingtheshapeindexing.Therecognitionprocessisstillexponential,asitmustbeifwearestilltoconsiderallpossiblesubshapes.However,inpractice,itisveryfastbecausemostoftherecognitionworkisbeingperformedintheindexingstage.Astheworkthatisperformedintheindexingstageneednottakeintoaccountothershapesonthescreen,asmoreshapecandidatesarepresentonthescreen,recognitiontimeincreasesmoreslowlythanitwouldotherwise.Whenanewshapeisrecognized,thesystemcomputesitsprop-erties,includingitsorientation,angle,location,andsize,andplacesthatshapeinahashmapwiththekeybeingthevalue.Eachshapeproperty(suchasx-location)hasitsownhashmap,permittingquickretrievalofshapeswiththedesiredpropertyvalue.Forin-stance,theanglehashmapisusedwhensearchingforhorizontallinesorlinesofequalangle.Whenalineisrecognized,itsangleismeasuredandcategorizedasHORIZONTALPOSLOPEVERTICAL,orNEGLOPE.Thecategoryisusedasthekeyforahashmapwhosevaluesarealinkedlistofshapes.Thisallowsconstant-timeinsertionandretrievalofalistofallofthelinesinaparticularorien-tation.Wealsowanttondparallellines,soexactanglesareusedtoaddtheshapetoasortedmaplist.Thisallowsforalogarithmic-timeinsertionandretrievalofthelistoflinesthatareclosetoapar-ticularangle.Similarly,locationandsizearealsostoredinabothacategorizedhashmapaswellasacontinuoussortedhashmaplistsinceitisoftenconvenienttogetarangeofvaluesdependingontheconstraint.Sinceitisfastertoretrieveshapeswithanglesclosetoourpredenedcategories,thesystemdoessoforthoseconstraintswhereitisappropriate.Thecompletelistofpropertiesareindexedforeachshapeandforeachofitscomponents.Thecomponentsareshapespecic,buttogiveanexample,thecomponentsforalinearethetwoendpoints,thecenter,andtheboundingbox.Thelistofpropertiesforeachshapeis:shapetype,shapename,shapeangle,center-xlocation(notethatthecenter-xlocationofanendpointisjustthexlocationoftheendpointitself),center-ylocation,min-xlocation,min-ylo-cation,max-xlocation,max-ylocation,size(actuallythelengthofthediagonaloftheboundingbox),height,width,andstrokelength.3.3HighLevelShapeRecognitionOncepropertiesarecomputedandindexinghasbeendone,thesys-temtriestoseewhetherahigher-levelshapecanbemadefromthisnewshapeandshapesalreadyonthescreen.Weneedtocheckwhetherthisnewshapecanbeapartofanyoftheshapesdenedinthedomaindescription.Foreachshapetemplateinthedomain,thesystemassignsthenewshapetoeachpossibleslotcomponent.Iftherearedomainshapes,andeachshapeiscomposedofcomponents(),thenthejustprocessedshapeisassignedtoeachslotseparatelyindifferentshapetemplates.Figure9showsanexample.Anewlyinterpretedlineisaddedtothesystem.Thesystemcheckstoseewhetherthenewlyinterpretedlinecanbeused Figure9:Atemplateismadeforeachpossibleslot.tocreateanyoftheshapesinthedomain.(Inthisexample,wearecheckingonlythedomainshapeOPENRROW.)Thesystemcre-atesthreetemplates,oneforeachcomponentoftheOPENRROWofthecorrecttype(inthiscasealine),assigningthejustprocessedlinetoadifferentcomponenttoseewhetheranOPENRROWbeformedwiththatcomponent.Thesystemthencomputesthefunctionij,whichreturnsalistijofshapesoftypeandthecomponentsthatmakeuptheseshapes,whichcanbeformedwiththejustprocessedshapeassignedtocomponent.Forexample,ifthedomaindescriptionincludes10shapedescriptions,andOPENRROWisthethirdde-PENRROW(showninFigure3).Anarrowhasthreeslots(oneforeachline).Ifthesystemputstherecentlydrawnshapeintoslot2,thenSHAFT.Thus,returnsalistofallofthepossibleOPENRROW'swiththemostrecentlydrawnshapeactingastheSHAFTofthestroke.Thelengthofijmaybe0(nosuchinterpretationsarepossible),1(oneinterpretationispossible),1(multipleinterpretationsarepossible,seeFigure10).Thenewshapecanbeplacedinanyslotinanytemplateprovideditisoftherighttype.(Anarccan'tbeplacedinalineslot.)isthe Weusethreetemplateshereforexplanationsimplicity,butinactuality,whenasinglelineisdrawn,thesystemcreatessixarrowtemplates.Therecognitioncreatestwocopiesofeveryline,oneineachdirection.(i.e.,thesecondisequaltotherstwiththeendpointsipped.)Eachdirectionallineisassignedoneatatimetoeachcomponentofthreearrowtemplates.Thisisactuallynotaphenomenonappliedspecicallytolines,butanygroupofmultipleinterpretationsusingasinglesubshape. Figure10:MultipleOPENRROWinterpretationsarepossibleusingthecen-terstrokeastheSHAFTunionofallthepossibleshapesformedwiththenewshape.ij))Atthispoint,eachtemplatecurrentlyhasonlyoneslotlled(withthenewshape).Tocomputeij,thesystemas-signsalistofalloftheothershapesonthescreentotheotherslotsonthetemplate,sothateachslotonthetemplateholdsthepossi-bilitiesforthatslot.Thenextstageistoreducethepossibilitiesforeachslot.Thisisdonefromtheindexingdatastructuresthatwerecreatedpreviously.Thefundamentalideabehindtheuseoftheindexedvaluesistoallowquickpruningtopreventthetemplatesfrombranching(whichhelpstolimittheinherentexponential).Thesystemholdsalistofuncheckedconstraints.Onemainfea-tureofouralgorithmisthesmartorderingofconstrainttestbasedonthenumberofcomponentsineachslot.Inordertoremoveashapefromthelistofpossibleshapesinaslot,wehavetobesureashapecannotbeformedusingthatshape.Ifonlyoneofthecom-ponentslotsreferencedbytheconstraintcontainsmorethanonepossibleshape,thenwecandetermineforcertainwhichshapessat-isfythatconstraint,andareallowedinthenalshapecomposition.bethelistofconstraintsfromthatarenotyetsolved.Letbethenumberofslotsforconstraintwithmorethanonecom-ponentinthepossibilitieslist.Foreachconstraintwithonlyoneslot(ornoslots)withmorethanonecomponent(1),wecomputetheconstraintforeachpossiblecombination(whichisalinearnumberofcomputationssinceonlyoneslotoftheconstrainthasmultiplepossibilities).Wethenremovealloftheshapesfromtheappropriateslotthatwouldmakeconstraintfalseandthenre-moveconstraintfromthelistofunsolvedconstraints3.4PruningPossibilitiesbyPreventingBranchingFigure11walksthroughanexample.LineL7hasbeenaddedtothescreenwherepreviousshapesL1,L2,L3,L4,L5,andL6al-readyexist.Inthisexample,thereareseveralshapetemplates,withthesmallestshapetemplatethatoftheOPENRROW,composedofthreelines.Thereadermaybelefttowonderwhathappensbe-foreL7isaddedtothescreen.InitiallyL1isaddedtothescreen,butnoshapetemplatesexistthatcontainonlyasinglesubshape,sonothinghappens.Next,L2isdrawn,andagain,therearenoshapetemplateswithonlytwosubshapes,sonothinghappens.Subse-quently,whenL3isdrawn,theredoesexistashapetemplatewiththreesubshapes(theOPENRROW).Uptothispointwehaveal-readydeterminedthatitisimpossibletomakeashapewithonlyL1andL2(asthereweren'tenoughcomponentstomakeupasingleshape).Atthispointweknowthatashapeexistsonthescreencontainallthreelinessincenoshapetemplatesexistwithfewerthanthreelines.Butmoreimportantlyandmoregeneraliz-ablytoothercases,sinceallattemptshavefailedtomakeshapesoutofthepreviouslyexistingshapesonthescreen,anyshapethatnowexistsonthescreenmustcontainshapeL3.Thus,thealgo-rithmonlyattemptstomakeshapetemplateswithL3.Threedif-ferentshapetemplates(x2,seepreviousfootnote)arecreatedfortheOPENRROW,whereL3issetastheonlypossibilityfortheSHAFTLINE1,andLINE2respectively.WethenaddbothL1andL2tobothoftheotherremainingtwoslots,leavinguswith:Template1:shaftL3head1L1L2head2L1L2Template2:shaftL1L2head1L3head2L1L2Template3:shaftL1L2head1L1L2 Figure11:Anexampleshowinghowtheconstrainttemplatesarepruned.ThesystemthenrunsthealgorithmdescribedinFigure11,andndsthattherearenoshapesonthescreen.Atthispoint,allof thepreviouslyperformedworkisthrownout,andtheprocessisrepeatedforL4,andsoon.NotethatwithL4,sincewewereunabletomakeashapewiththesubshapeL1,L2,andL3,weknowthatanyshapethatmayexistonthescreenmustcontainL4.Eventually,L7isaddedtothescreen,andwearriveatthesitu-ationinFigure11.Thetoprightshowstheinitialtemplatewhenassigningthenewline(withaparticularorientation)totheHEADslotofthearrow.Notethatallothershapesonthescreenareadded.Thesystemthenattemptstosatisfyeachoftheconstraints,remov-ingshapesthatdonotsatisfyaconstraintfromthetemplate.1)ThesystemtriestosatisfytheconstraintEQUALENGTHandre-movesallshapesintheHEAD1slotofthetemplatethatarenotofequallengthtoL7.TheEQUALENGTHconstraintisnowcom-pletedandremovedfromthelistofconstraintsyettobesatised.2)ThesystemattemptstosatisfytheLONGERconstraint,butsincebothargumentshavemorethanoneshapeintheslot,theconstraintispostponed.3)ThesystemtriestosatisfytheCOINCIDENTstraintandremovesL2,L3,andL5fromtheHEAD1slot.BecauseL4isnowtheonlypossibilityfortheHEAD1slot,itisremovedfromtheSHAFTslot,sincethesameshapecannotbeusedforbothslots.4)ThesystemtriestosatisfythesecondCOINCIDENTstraint,butsincenoneoftheshapesintheSHAFTslotcanbecoin-cidentwithL4,theSHAFTslotisempty,andthesystemdeterminesthatanOPENRROWcannotbeformedwiththenewshapellingintheHEAD1slot.Moregenerally,eachtimeanewlineisdrawn,thesystemisasking,"couldthislinebepartofanyknownshapebuiltfromitandallpreviouslines?"Itdoesthisbyputtingthenewlineisallpossibleslotsineachtemplate,puttingalluninterpretedpreviouslydrawnlinesinalltheotherslots,andtreatingthetemplateasacon-straintsatisfactionproblem,withthegeometricconstraintsreningthevalues(thelinesassignedtotheslots).Notethataquestionmaybeaskedastowhyapreviouslydrawnimpossibletemplatearenotalreadyprunedfromthelistsincethesepreviouslycomputedvaluescouldbesavedandcheckedrst.Allpreviouslycomputedtemplatesarenotsavedforseveralreasons:1.Firstoff,allpossiblesub-templatesconsistingonlyofearlierstrokesarenotactuallycomputed,sincemanybranchesarecutoffveryearly.Thismeansthatwewouldrsthavetolookupeachtemplatecombinationtoseeifwehaveseenitbefore.Thislookuptakesaboutthesameamountoftimeascomputingtheconstraintitself(whichisalsojustalookup).2.Second,inasystemwithmanyshapetemplates�(10or�50),savingallpreviouslycomputedtemplateswouldrequireasig-nicantamountofspacethatcouldotherwisebeusedmoreeffectively.Anysystemthatdidthiswouldeitherquicklyrunoutofheapspaceortakeasignicanttimesinktoreadandwritetodiskallofthenecessarytemplateinformation.ThissavingofalltemplatesisinfactwhatJessdoes.LookingattheResultssection(Section4)fortimingcomparisondata,wecanseethattheJessmethodofsavingallpossibletemplatesalongthewaycausesatimeincreaseratherthanaspeedup.3.Third,itisunclearwhatuse,ifany,previouslycomputedpartially-lledtemplateswouldbe.Whileusersdosketchinaninterspersedmanner,manyshapesaredrawninanon-interspersedmanner.Inthiscase,previouslycomputedtem-plateswouldbeoflimitedvalueonceahigh-levelinterpreta-tionwasfoundusingasubshapeinthetemplate.Thiswouldimplythateitherthesystemtakesthespace-timehitforsavingthetemplates,orthesystemmustdoasignicantamountofgarbagecollectionaftereveryhigh-levelinterpretation,anditisunclearwhatthetimesavingswouldactuallybe.Usingourmethod,allincompletetemplatesarediscardedaftereachshapetemplateisexaminedaftereachstroke.Thisisaconscioustradeoffof,sacricingspeedforspace.TheResultssectionshowsthatourspeedtradeoffstillallowsouralgorithmtoperforminreal-time.Futureworkislefttoseeifamoreeffectivemanner(suchasonlyasimplehashcode)ofstoringimpossiblesubtemplatescouldbeusedtoimprovebothspeedandspace.Findingtheshapesthatdonotsatisfyisaquickprocesswhenusingtheindexingtablesformedabove.Foreachconstraint,since1,onlyoneslotisbeingrenedatatime.Thus,thesys-temcomputesthevalue(s)thatsatisfytheconstraintforthatslot.Thesystemusestheindexingdatastructurestoobtainalistofalloftheshapesonthescreenwiththatparticularvalue(s)(e.g.,forHORIZONTAL,itwouldretrievealllineswithananglenear0).Theintersectionofthislistandthelistofshapesintheslotiscomputed,andshapesthatarenotintheintersectionareremovedfromtheslot.Itispossiblethatallshapesareremovedfromtheslot,whichimpliesthatthisshapecannotbeformedwiththesetofshapesintheslot,andallprocessingonthattemplateishalted.Thiscycleisrepeateduntil:1)thetemplateisdeterminedimpossible;2)alloftheconstraintsaresolved,andeachslothasonlyoneshapeinit;or3)alloftheremainingconstraintshavemorethanonecom-ponent(1),andthecycleisstuck(seethenextparagraphforwhathappens).Aftereachcycle,therearesomeslotsthatcontainonlyoneshape;consistencycheckingoccursasthesystemremovestheseshapesfromalloftheotherslotstomakesurethesameshapeisnotusedinmultipleslots.Itispossiblethatalloftheremainingunsolvedconstraintshavemorethanonecomponentremaining(1).Inthiscase,thesystembranchesthetemplate,choosingtheslotwithmorethanoneremainingpossibleassignmentsthathasthefewestsuchpossibleassignments.Itmakesanewcopyofthetemplateforeachofthepossibleassignmentsforthatslot,placingonlyoneineachtem-plate,thencontinuestryingtosolvetheremainingunsatisedcon-straintsoneachofthetemplates.Thisbranchingprocesscanofcoursecauseitsownexponentialslowdown.Thesystem'snearreal-timeperformanceresultsfromthefactthat1)branchingdoesnothappenoftenbecausemostoftheshapesonthescreendonotobeytherequiredconstraints,andthus,manyshapesareremovedfromthepossibilitylistatonce.(ConsidertheCOINCIDENTconstraint.Itisuncommonformanyshapestobedrawnatthesamelocation,somanypossibilitiesareremovedsimultaneouslyfromthepossibilitieslist.)And,2)evenintheworstcasewhereeveryqueryresultsinabranchingfactor,theprocessofcheckingtheconstraintsisasmallproportionoftheoverallrunningtime(seetheResultssectionbelow)Thisisbecausetheexponentialpartofthealgorithmperformsonlylistretrievals(whichhasbeenmadefastwiththeuseofsortedhashmapsandotherdatastructures)andlistintersections.Attheendofthisstage,wehavealistofalloftheshapein-terpretationsandtheircomponents.Allinterpretationsareaddedtotherecognitionsystem,butasingleinterpretationischosenfordisplay.Thesystemchoosestodisplaytheinterpretationthatiscomposedofthelargestnumberofprimitiveshapes(i.e.,thesolu-tionthataccountsformoredata).Creatingshapeswiththelargestnumberofprimitiveshapesalsoresultsinfewermore-complicatedshapesllingthescreen.Forexample,inFigure12,wechoosethesquareinterpretationsratherthanthearrowfordisplaypurposes,asthesquareaccountsforfourprimitive,simplifyingthediagramtoonlytwohigh-levelshapes,whereasthearrowinterpretationac-countsforonlythreelines,simplifyingthediagramtothreehigh-levelshapes.Thespeedupofthisalgorithmisduetothreethings:1.Mostoftheheavycomputationisperformedinthe(linear-time)indexingstage.2.Ratherthanrstcreatingalistofallpossibleinterpretationsofashape,subtreesareprunedastheyaregenerated. Figure12:Theleftshowstheoriginallydrawnstrokes.Themiddleinterpre-tationismadeupofalineandasquare.Therightinterpretationismadeupofanarrowandtwolines.3.Constrainttestingusesasmartorderingofconstraints;i.e.,theconstraintthatcouldcausethebiggestpruningeffectistested4RESULTS4.1StressTestWhenrecognizingnewshapes,itisthecurrentlyunrecognizedstrokesthatcausedifcultiesinrecognition,asrecognizedstrokescanbeprunedawayfromtherecognitionsystem.Thuswetestedoutsystemthroughastresstestwhichmeasuredthetimeittookforanewsymboltoberecognizedwhenthereisalargeamountofunrecognizedstrokesonthescreen.AllofourresultsweretestedandmeasuredonatabletPCwith1GBofRAManda1.73GHzprocessor.Tocompareouralgorithmagainstastandardcachingsystemthattriesallsubsetpossibilities,theauthorsimplementedanalternateversionofthealgorithmthatinsteadusestheRetealgorithminJess[FH01]tohandlethecachingandpruninginrecognition.Theexponentialinherentintestingallsubsetsoverwhelmedthesystem,whichslowedunacceptablyafterabout30individualstrokes(tak-ingonehourtorecognizethenextshape).Usingtheindexingalgorithmdescribedabove,westress-testedrecognitionresultsinthedomainofmechanicalengineering,ado-mainconsistingof15distinctshapes.Thethesystemrecognizedaresistorcontainingsixlinesinlessthanasecond,with189othershapes(besidesthesixlinescreatingtheresistor)onthescreen(seeFigure13(a)),5ofwhichwerehigher-levelshapessuchasresistors,andtheother186wererandomlinesonthescreen.Additionally,wewantedtoseewhatpercentageofrecognitiontimefocusesoneachpartofrecognition.Ourgoalwastomaketheexponentialpartofthealgorithmasmallpartoftherunningtimeofrecognition.Asdenedabove,recognitionconsistsof1)Cornerdetectionorstrokefragmentation,wherestrokesarebrokendowntotheircompositeparts,whichtakesintermsofthenumberofshapesonthescreen.2)Indexing,whichtakeslognintermsofthenumberofshapesonthescreen.3)High-levelrecognition,orcombiningstrokestoformhighlevelobjects,whichisexponentialintermsofthenumberofshapesonthescreen.Thegoalofthispaperwastominimizethisthirdpartofthealgorithm.Theauthorsanalyzedtherunningtimeoftherecognitionsys-temonadiagramofoverahundredshapes,usingJProbe[QS06],anddeterminedthat,withmanyunrecognizedshapesonthescreen,approximately74%ofthetimewasspentoncornernding.Ap-proximately,25%ofthetimewasspentindexingthelinesdrawn.Lessthan1%ofthetimewasusedtoperformthehigh-level(ex-ponential)recognition.Becauseoftheindexingandearlycheck-ingofconstraints,thesystemwasabletoperformeffectivepruningandcaching,andpreventtheexponentialaspectofrecognitionfromcausingoverwhelminglaginrecognition.Asaresult,anexponen-tialrecognitionalgorithmrunsinwhatcanbeconsideredclosetoreal-time,evenwith186shapesonthescreen.4.2MultipleDomainTestInadditionaltoourstresstest,wetestouralgorithmonthreedif-ferentadditionaldomainsinnaturalusage.Domain1wasJapaneseKanji,consistingof27shapesofupto15primitivespershape,withanaverageof10primitivescomposingeachshape.ThetotaltimetorecognizetheshapesinFigure13(b)was3517milliseconds.Domain2wasmilitarycourseofactiondiagrams,consistingof23shapes,ofupto14primitivespershape,withanaverageof�7primitivescomposingeachshape.ThetotaltimetorecognizetheshapesinFigure13(c)was8858milliseconds.Domain3wasbi-ologydiagrams,consistingof6shapes,ofupto10primitivespershapes,withanaverageof�5primitivescomposingeachshape.ThetotaltimetorecognizethedrawnshapesintheFigure13(d)was481milliseconds.Figure13(e)showstheoverallstatisticscombinedforallthreedomains,withthetotaltimeequaling15660milliseconds.Accuracyforallthreedomainswasunfortunately/fortunately100%.5DISCUSSION5.1LimitationsThisalgorithmcanbeusedtodescribeawidevarietyofshapes,butislimitedtothefollowingclassofshapes.Thisalgorithmreliesongraphicaldescriptionsofshapes.Theshapesmustbedrawnusingthesamegraphicalcomponentseachtime.Forinstance,wecannotdescribeabstractshapes,suchaspeopleorcats,thatwouldbedrawninanartisticdraw-Likewise,sincethisalgorithmrequiresgeometricdescrip-tions,ifashapeishighlyirregularandcomplicated,sothatitcannotbebrokendownintosubshapesthatcanbedescribed,itwillbecumbersometodene.Theshapesmustbecomposedsolelyoftheprimitivecon-straintsthatwedeneinthispaper,andtheseconstraintsmustbeenoughtoensurethatallshapesareappropriatelydifferen-tiablefromtheothershapesinthedomain.Pragmatically,itisdifculttodescribecomplexlycurvedshapes.Curvesareinherentlydifculttodescribeindetailbe-causeofthedifcultyinspecifyingacurve'scontrolpoints.Futureworkincludesinvestigatingmoreintuitivewaysofde-scribingcurves.5.2RecognitionLimitationsWearecurrentlylimitedtorecognizeonlythoseshapesthataredescribableusingtheconstraintsabove.Itispossibletocontriveanexamplethatwillcauseourap-proachtobranchveryoften,causingsignicantslowdown.Thiswillariseifsomeonedrawssimilarshapesrepeatedlyontopofthemselves.Thesystemwouldhaveaverylargenum-berofbranches,becauseitwouldbedifculttoeliminatealargenumberofpossibilitiesinaslotforasingleconstraint.Shapesthatarehighlysymmetricwillcausemanyinterpre-tationsthatlookidenticalbutthathavedifferentlineassign-ments.Forexample,considerasquare.Thesquarecanberotatedandeachofthelinesipped,andthusdependingontheformofthedescription,sixteeninterpretationsmayex-ist.Alloftheseinterpretationsmustbekeptbecausehigher-levelshapesformedfromthatsquaremayrequirethelabelingthatoccursinanyoneoftherecognitions.Thus,thenumberofshapesintherecognitionpoolcangrowexponentiallyasshapesarerecognized.Luckily,inpractice,mostinterpreta-tionsareruledoutbasedontheshapedescription.6PREVIOUSORKIndexingdatatoprovidespeed-upisofcoursenotanewidea.Ithasbeenused,forexample,inthecloselyrelatedeldofvisionobjectrecognition[SAB97][SM92][AKJ02][LVB93].Indexing (a)RecognitionStressTest (b)RecognitiontimesforJapaneseKanji (c)RecognitiontimesforMilitaryCourseofActionDiagrams (d)RecognitionTimesforBiology (e)RecognitiontimesforallthreedomainscollatedFigure13:RecognitionResults sketchestosearchforphotoshasbeendoneby[BP97][KKOH92].Indexinghasalsobeenusedinsketchrecognitiontoindexsoftwaredesignmeetings[HGDS02].Thisworkappearstobetherstuseoftheideainsupportofunconstrainedsketchrecognition.Bybreak-ingthestrokesdowntoline,curve,andellipsesegments,wewereabletodeneshapesingeometricterms,thenperformindexingontheseterms,aluxurynotaffordedwhenindexingphotoimagesornon-geometricsketches(suchasanartist'ssketch).7FUTUREORKWewouldliketoremovesomeofthelimitationsdescribedabovebycombiningthetechniquespresentedinthispaperwiththosethathaveprovedusefulinworkinvision.Byprocessingandindexingvisionfeaturesusedforrecognition,andconcurrentlyindexingongeometricpropertiesasdescribedinthispaper,wecanquicklyac-cessshapesthathavetheneededvisualandgeometricfeatures.Asaverysimpleexample,visionrecognitiontechniquescaneasilylo-cateareasofhighdensityink,orshading,whichwearecurrentlynotabletorecognizeusingourgeometricrecognitiontechniques.Wewouldliketocombinevision-andsketch-basedfeaturestoper-formmorerobustrecognitionandperhapsrecognizealargerclassofobjects.8CONCLUSIONThispaperdescribesanindexingalgorithmforuseinsketchrecog-nition.Shapesareindexedastheyarebeingdrawnorandwhentheyarerecognized,usingavocabularyofgeometricpropertiesthatpermitsfastlookupsubsequently.Geometricconstraintindexingallowsustosearchthroughallpossiblesubsetsoftheshapesonthescreenquickly,facilitatingrecognitionofhigher-levelshapes.Becausewecanquicklysearchthroughallpossiblesubsetshapes,wenolongerhavetolimitthesearchspaceandcanallowsketch-erstointerspersethedrawingofincompleteshapeswithlittlespeedpenalty.Byenablingalessconstraineddrawingstyle,weprovideamorenaturalsketchrecognitionuserinterface.Ourresultsshowthatthesystemcontinuestoruninnearreal-time,evenwhenalargenumberofstrokecandidatesexistaspossiblesub-componentsofa9ACKNOWLEDGEMENTSThispaperissupported,inpart,byNSFIISCreativeITGrant#0757557.EditingcommentswereappreciatedfromJacobEisen-stein,PaulBogen,BrandonPaulson,KatieDahmen,MarkEaton,HenryChoi,andJoshuaPeschel.TheauthorswouldliketothankthehelpofPaulTaeleandBrianEoffinthecompletionofguresforthispaper.Additionally,theywouldliketothankthemem-bersoftheSketchRecognitionLabatTAMUandthemembersandalumnioftheDesignRationalGroupatMIT.Additionally,asthisworkwasasignicantpartof[Ham07],theauthorswouldliketothankthosewhohelpedhelpedinthatendeavor.EFERENCES[AD04]ALVARADOC.,DAVISR.:Sketchread:Amulti-domainsketchrecognitionengine.InProceedingsofUIST'04pp.23–32.[AKJ02]ANTANIS.,KASTURIR.,JAINR.:Asurveyontheuseofpatternrecognitionmethodsforabstraction,indexing,andre-trievalofimagesandvideo.PatternRecognition35[Alv00]ALVARADOC.:ANaturalSketchingEnvironment:BringingtheComputerintoEarlyStagesofMechanicalDesign.Mas-ter'sthesis,MIT,2000.[BP97]BIMBOA.D.,PALAP.:Visualimageretrievalbyelasticmatchingofusersketches.IEEETransactionsonPatternAnal-ysisandMachineIntelligence19(2)(1997),121–132.[CGFJ02]CAETANOA.,GOULARTN.,FONSECAM.,JORGEJ.:JavaSketchIt:Issuesinsketchingthelookofuserinterfaces.SketchUnderstanding,Papersfromthe2002AAAISpring[FH01]FRIEDMAN-HILLE.:Jess,thejavaexpertsystemshell.http://herzberg.ca.sandia.gov/jess,2001.[Gol72]GOLDMEIERE.:Similarityinvisuallyperceivedforms.InPsychologicalIssues(1972),vol.8:1.[Ham07]HAMMONDT.:Ladder:Aperceptually-basedlanguagetosimplifysketchrecognitionuserinterfacedevelopment.PhDThesis,MassachusettsInstituteofTechnology,2007.[HD02]HAMMONDT.,DAVISR.:Tahuti:AgeometricalsketchrecognitionsystemforUMLclassdiagrams.AAAISpringSymposiumonSketchUnderstanding(March25-272002),59–[HD05]HAMMONDT.,DAVISR.:LADDER,asketchinglanguageforuserinterfacedevelopers.Elsevier,ComputersandGraph-ics28(2005),518–532.[HGDS02]HAMMONDT.,GAJOSK.,DAVISR.,SHROBEH.:Anagent-basedsystemforcapturingandindexingsoftwaredesignmeet-ings.InProceedingsofInternationalWorkshoponAgentsInDesign,WAID'02[KKOH92]KATOT.,KURITAT.,OTSUN.,HIRATAK.:Asketchre-trievalmethodforfullcolorimagedatabases-querybyvi-sualexample.11thIAPAInternationalConferenceonPatternRecognition(1992),530–533.[Lon01]LONGA.C.:Quill:aGestureDesignToolforPen-basedUserInterfaces.EECSdepartment,computersciencedivision,U.C.Berkeley,Berkeley,California,December2001.[LTC00]LANKE.,THORLEYJ.S.,CHENS.J.-S.:AninteractivesystemforrecognizinghanddrawnUMLdiagrams.InPro-ceedingsforCASCON2000(2000),p.7.[LVB93]LADESM.,VORBRUGGENJ.,BUHMANNJ.,LANGEJ.,VONDERALSBURGC.,WURTZR.,KONENW.:Distor-tioninvariantobjectrecognitioninthedynamiclinkarchitec-IEEETransactionsonComputers42(3)(March1993),[MF02]MAHONEYJ.V.,FROMHERZM.P.J.:Threemainconcernsinsketchrecognitionandanapproachtoaddressingthem.SketchUnderstanding,Papersfromthe2002AAAISpring(Stanford,California,March25-272002),AAAIPress,pp.105–112.[QS06]QUESTOFTWAREI.:JProbe.website,http://www.quest.com/jprobe,2006.[Rub91]RUBINED.:Specifyinggesturesbyexample.InGraphics(1991),vol.25(4),pp.329–337.[SAB97]STEVENSM.R.,ANDERSONC.W.,BEVERIDGEJ.R.:Ef-cientindexingforobjectrecognitionusinglargenetworks.InProceedingsofIEEEInternationalConferenceonNeuralNet-[Sez06]SEZGINT.M.:SketchInterpretationUsingMultiscaleStochasticModelsofTemporalPatterns.PhDthesis,Mas-sachusettsInstituteofTechnology,May2006.[SM92]STEINF.,MEDIONIG.:Structuralindexing:Efcient3-dobjectrecognition.IEEETransactiononPatternAnalysisAndMachineIntelligence(1992),125–125.[SSD01]SEZGINT.M.,STAHOVICHT.,DAVISR.:Sketchbasedinterfaces:Earlyprocessingforsketchunderstanding.InTheProceedingsof2001PerceptiveUserInterfacesWorkshop(Orlando,FL,November2001).[WWL07]WOBBROCKJ.,WILSONA.,LY.:Gestureswithoutli-braries,toolkitsortraining:a$1recognizerforuserinterfaceprototypes.InProceedingsofUIST RecognizingInterspersedSketchesQuicklyTracyA.HammondSketchRecognitionLabDepartmentofComputerScienceTexasA&MUniversityRandallDavisComputerScienceandArticialIntelligenceLabMassachusettsInstituteofTechnologyBSTRACTSketchrecognitionistheautomatedrecognitionofhand-drawndi-agrams.Whenallowinguserstosketchastheywouldnaturally,usersmaydrawshapesinaninterspersedmanner,startingasec-ondshapebeforenishingtherst.Inordertoprovidefreedomtodrawinterspersedshapes,anexponentialcombinationofsubshapesmustbeconsidered.Becauseofthis,mostsketchrecognitionsys- e-mail:hammond@cs.tamu.edue-mail:davis@csail.mit.eduindesign;3)recognizetheshapeasawholetoallowformorepow-erfulediting;4)beautifydiagrams,removingmessandclutter;and5)notifythesketcherthattheshapeshavebeenrecognized.