KeywordsExternalrepresentations!Thinking!Interactivity!Sensemaking!Cos - PDF document

KeywordsExternalrepresentations!Thinking!Interactivity!Sensemaking!Coststructure1IntroductionThisessayisaninquiryintowhythinkingandsensemaking,sooften,isinteractive.ByÔinteractiveÕImeanabackandforthprocess:apersonalterstheoutsideworld,thechangedworldalterstheperson,andthedynamiccontinues.Readingatextsilentlyisnotaninteractiveprocess,formypurposeshere,thoughitisextremelyactive.Readingandunderliningthetext,orreadingandsummarizingit,evenreadingandmovingoneÕslips,are.Thepuzzlethatinteractionraisesaboutsensemakingandthinkingcanbeposedlikethis.Inaclosedworld,consistingofapersonandanexternalrepresentationÑadiagram,illustration,spokeninstruction,orwrittenprob-lemstatementÑwhydopeopledomorethanjustthinkintheirheads?Ifweassumethereisnoonetoask,notooltogeneratenovelresults,noclocktoprovidechronometricinput,noprocesstorunandobserve,thenthereisnothingexternal,nooracleortool,thatapersoncanconsultormanipulate,thatyieldsnewinformation.Theenvironmentcontainsnothingthatcouldnotbeinferredthroughreßection,atleastinprinciple.Sowhybothertomark, deepentheirsemanticandpragmaticprocessingoftheinstructions.Inbothcases,peopleneedtoestablishacoordinationbetweenwhatgoesoninsidetheirheadsandwhatgoesonoutside.Theyconstructacorrespondence,acoordinationrelation,synchronization.Becausethesecoordinationprocessesarenotcost-free,Fig.2overlysimpliÞestherelationbetweeninternalandexternalpro-cesses.Afurtherprocessneedstobeincluded:thecouplingprocessitself,thespecialactionsperformedtoestablishacognitivelink.Figure3illustratesthisaddedcost-ladenprocess:anchoring(seeKirsh2009b,foraninitialdiscus-sionofthisthirdcostspace).SeealsoHutchins(2005)foranaccountofperceptualanchoringandFauconnierandTurner(2002)foranchoringinmentalspaces.AsimportantasthisanchoringorgroundingprocessisIrestrictmyfocus,intheremainderofthiswork,towaysweinteractwithrepresentationstoalterthecognitiveterrainratherthantheinteractionsweperformtoprepareourselvestoengagetheexternalpartofthatterrainthroughanchoring.2MaterialityanditsconsequencesTheargumentothersandIhavelongadvancedisthatpeopleinteractandcreateexternalstructurewhenthinkingbecause:ThroughinteractionitiseasiertoprocessmoreefÞ-cientlyandmoreeffectivelythanbyworkinginsidetheheadalone(Clark2008;Kirsh intermsofthementalcounterpartsoflines,particularlythemorelinesthereare,orthemorecomplexthestructure.Eventhoughsomepeoplecandothingsintheirheadsthatotherscannot,thereisalwaysapointwhereinternalistcognitivepowersareoverwhelmedandphysicalrealizationisadvantageous(seeKirsh2009b).Thus,althoughfroma toseetheeffectofmovementondeformationsofthetrapezoid.Forsythereliedonhislistenersseeingthevisibleannotation,thetrapezoidalstructure,asheexplainedthe positionskeepthevolumeoftheshapeconstant,orwhichmovementsensurethetopplaneremainsparalleltothebottomplane.ChoreographersÞndsuchquestionshelpfulwhenthinkingaboutbodydynamicsandwhentheywanttocommunicateideasofshearingandtorsiontotheirdancers.Buttheyarehardtounderstandifthegroupdoesnotshareavisualorprojectedimageofatransformingshape.Physicallyreifyingashapethroughannotationaddssomethingmorethanjustprovidingasharedreference;itprovidesapersistentelementthatcanbemeasuredandreliablyidentiÞedandre-identiÞed.MeasurementissomethingonedoesafteralineorstructurehasbeenidentiÞed.Thisneednotalwaysrequireanexternalpres-ence.Somepeopleareabletograspthestructureofasuperimposedtrapezoidpurelybymentallyprojectinganinvisiblestructureontothebody.Theylistentothespeaker;watchhisgestures,andproject.Butevenforthese insideignoresthemedium-speciÞcnatureofencoding.Thecurrentviewinpsychologyisthatwhenwevisuallyper-ceiveanexternalstructure,theinformationthatentersisstoredÞrstinvisuo-spatialstore(Baddeley2000;Logie1995),beforebeingprocessedforuseinlatermentalpro-cesses.Sincetheformastructureisencodedinprofoundlyaffectshoweasilyitcanbeusedinaprocess,itisanopenquestionhowmuchinternalprocessingisnecessarytoconvertanexternalstructureintoaninternalstructurethatisusable.Accordingly,itcannotbeassumed,withoutargumentthatthecostsarealwayslowerinperceptuallyretrievinginformationthanÔinternallyÕretrievinginfor-mation,evenifthatinformationiscomplexandvolumi-nousandsomethingwewouldnormallyassumeismoreefÞcientlystoredexternally.Thestrengthofthisconcernisobviousiftheinformationelementtobeperceivedisburiedinvisualclutter.Muchwilldependonvisualcom-plexity,theforminformationisencodedin,howeasyitistoperceivethestructurewhenitiswanted,andsoon.EvenwhenanobjectofthoughtispresentinaclearanddistinctwayÑasForsytheÕsgraphicalannotationsareÑitstillmustbeperceived,thengestalted,andconceptualized.Dowereallyknowtherelativecostofgraspinganexternallyrepresentedcontentversusaninternallyrepresentedone?Theimplicationisthatusingtheworldasexternalstoragemaybelessimportantasapuresourceofcognitivepowerthanusingtheworldforexternalcomputation.Thingsintheworldbehavedifferentlythanthingsinthemind.Forexample,externalrepresentationsareextendedinspace,notjustintime.Theycanbeoperatedonindifferentways;theycanbemanuallyduplicated,andrearranged.Theycanbesharedwithotherpeople.Toolscanbeappliedtothem.ThesedifferencesbetweeninternalandexternalrepresentationsareincrediblysigniÞcant.Theyarewhatmakesinteractivitysointeresting.Iturnnowtoanotherofthesedifferences:thepossibilityofmanuallyreorderingphysicaltokensofstatements.Becauseofrearrangement,itispossibletodiscoveraspectsofmeaningandsigniÞcanceÑimplicationsÑthatarehardtodetectfromanoriginalstatementwhenviewediniso-lation.Byreorderingandrearrangingwhatisclosetowhat,wechangeatokenÕsneighborhood;wechangethespaceofwhatiscognitivelynear.4RearrangementThepowerofphysicalrearrangement,atleastforvehiclesofpropositions,suchassentences,logicalformulae,pic-torialnarratives,isthatitletsusvisuallycomparestate- construct thinking.Itiseasiertohavethesamethoughttomorrow,if Scalemodelsaretangiblerepresentationsofanintendeddesign.Theyserveseveralfunctions:1.Theycanserveasasharedobjectofthoughtbecausetheyarelogicallyandphysicallyindependentfromtheirauthor.Theycanbemanipulated,probed,andobservedindependentlyoftheirauthorÕspriornotionabouthowtointeractwiththemodel.Thisisvitalfortalkingwithclients,displayingbehavior,functionalityanddetectingunanticipatedsideeffects.Itmakesthempublicandintersubjective.2.Modelsenforceconsistency.Theassumptionbehindmodeltheoryinmathematicsisthatifaphysicalstructurecanbefoundorconstructed,theaxiomsthatitinstantiatesmustbeconsistent(NagelandNewman1958).Unlikeadescriptionoftheworld,oramentalrepresentation,anyactualphysicalmodelmustbeself-consistent.Itcannotrefertopropertiesthatarenotsimultaneouslyrealizable,becauseifitisavalidmodelitcountsasanexistenceproofofconsistency.Inamanypartsystem,partAcannotbeinconsistentwithpartBiftheybothcansimultaneouslybepresentinthesamesuperstructure.Similarly,themovementofpartAcannotbeinconsistentwiththemovementofpartBifthetwocanberunsimultaneously.Buildit,runit,andtherebyproveitispossible.Inconsistencyis ambiguity. seeperspectivesandrelations.Whenourinteractioniscontrolledprecisely,orinterpretedasmovementalongatimeline,wecanjuxtaposesnapshotsintimeforcompar- theprocessmanageable.Indeed,werewetodisplaythecomputationalcostproÞles(measuredintermsofspeedaccuracy)forperformingacalculation,suchasaddingnumbersintheheadvs.usingalgorithmsortoolsintheworld,itwouldbeclearwhymostyoungpeoplecannolongerdomucharithmeticintheirheads.Toolsreshapethecoststructureoftaskperformance,andpeopleadaptbybecomingdependentonthosetools.Asecondreasonwecomputeoutsideratherthaninsidehastodowithadifferentsortofcomplexity.Oneofthetechniquesofreformulationinvolvessubstitutionand x?3)2 6x=7.Bysubstituting,weget(x?3)2=16,whichyieldsx=1or- Persistence,reordering,andreformulationlargelyexplainwhyexternalizinginformationandrepresentationmayincreasetheefÞciency,precision,complexity,anddepthofcognition.Andiftheseaspectsofinteractionwithexternalrepresentationsdonotexplaintheextrapowertobehadthensimulationdoes.Still,thereisanotheraspecttocon-sider:howexternalprocessesmayincreasethebreadthofcognition.Toexplorethisaspectconsideragain,whywepreferonemodalitytoanotherforcertaintypesofthinking.Everyrepresentationalsystemormodalityhasitsstrengthsandweaknesses.Aninferenceorattributethatisobviousinonesystemmaybenon-obviousinanother.ConsiderFig.9Ñamusicalnotation.Thereferentofthenotationisapieceofmusic.MusicissoundwithaspeciÞcpitchorharmony,volume,timber,andtemporaldynamics.TheÔhomeÕdomainofmusic,therefore,issound.Visualnotationformusicisparasiticonthestructureofsound.Primafacie,thebestrepresentationtomakesenseofmusicalstructureismusicitself;wegotothesourcetounderstanditsstructure(seefootnote2).Iftherearetimeswhenthesourcemediumisrequiredtorepresentthecontentofathought,thenafurtherreasontoexternalizecontentandmanipulateitoutsideisthatforsomeproblems,thenaturalrepresentationofthecontentonlyexistsoutside.Arguably,nooneÑoratbestonlyafewpeopleÑcanhearmusicintheirheadthewayitsoundsoutside.Mentalimagesofsoundshavedifferentpropertiesthanactualsounds.Evenifitispossiblefortheexperienceofthementalimageofmusictobeasvividanddetailedasperceptionoftherealthing,fewpeopleÑotherthanthemusicallygifted,theprofessionalmusician,orcomposer fordance,gesture,andperhapsthepliabilityofclayaretoo.Musicalinstruments,likewise,permitrapidproductionofsound.Theseoutermediaortoolsforcreatingmediasup-portfastwork.Theyenableustoworkwithplasticmedia.Inthisrespect,theyenableustoworkoutsidemuchlikethewayweworkinside,usingvisualorauditoryimagesforwordsorideas,whichmostofusworkwithatthespeedofthought.Ifexternalmanipulabilitymatchestheinternalrequirementsonspeed,thenanexternalmediumhastheplasticitytobeacandidateforthinkingin.8UsingmultiplerepresentationsDespitethevalueoflisteningtomusic,therearetimeswhennotationdoesrevealmorethanthemusiconehaslistenedtoÑinstanceswhereanon-naturalrepresentationcanbemorerevealingandintuitivethantheoriginalrep-resentation.Becauseanotationalrepresentationusesper-sistent,spaceconsumingrepresentations,earlyandlaterstructurescanbecompared,superimposedandtransformedusingnotationspeciÞcoperators.Aswithlogicandjigsawpuzzles,itisusefultohavetangiblerepresentativesthatcanbemanipulated.Inthesecases,asubjectwhomovesfromonerepresentationtotheothermayextendcognition.Bymovingbetweenlisteningtomusic,andwritingitdowninanotation,orlisteningandthenreadingthenotation,orsometimesviceversa,acomposerorlistenermaybeabletoexplorecertainelementsofmusicalstructurethatareotherwiseinaccessible.Themorecomplicatedthestructureofthemusic,themorethisseemstobetrue.Withoutinteractingwithmultiplerepresentationscertaindiscover-ieswouldsimplybeoutofreach.Visualdesignerswhomovebetweenpenandpaper,3Dmockupsandrapidprototypesarefamiliarwiththesametypeofprocess.9ConstructionandtoolsTheÞnalvirtueofexternalinteractionIwilldiscussis,insomeways,thesummationofpersistence,rearrangement, thereareoperationsthatcanbeperformedonexternalrepresentationsthatcannotbeperformedoninternalrep-resentations,andthat,somehow,theseareessential.Arethereepistemicactivitieswecanperformoutsidethatwecannotduplicateinside,notbecauseoftheircomplexity,butbecausetherearephysicalpropertiesandtechnologiesavailableontheoutsidethatwecannotduplicatemen-tallyÑoperationswecannotmentallysimulatewithsufÞ-cientrealismtodeliverdependableanswers?ConsiderFig.10.Thedotsinthetwoimagesontheleftarerelatedtooneanotherbyarotationof4".Thisisessentiallyinvisibleunlessthetwoimagesaresuperim-posed,asintheimageontheright.Superimpositionisaphysicalrelationthatcanberepeatedanynumberoftimes,asisrotation.Bothrequirecontroloverphysicaltransfor-mations.Inthecaseofsuperposition,thepositionofthelayersmustbecontrolledprecisely,andinthecaseofrotation,theanglemustbecontrolledprecisely.Aretheresuchfunctionsinthebrain?Thebrainprocesswearerequiringisanaloginnature.Forover25years,adisputehasragedoverwhetherbrainssupportanalogprocessesorwhethermentalimageryisdrivenbynon-analogmeans(Pylyshyn).Wecansidestepthisquestion,though,byappealingtoaninprin-cipledistinctionbetweentypesofprocesses.Inanimpor-tantpaper,VonNeumann(1948)mentionedthatsomeprocessesinnaturemightbeirreduciblycomplex.Anydescriptionofoneofthoseprocesseswouldbeascomplexastheprocessitself.Thus,tosimulateormodelthatpro-cessonewouldhavetorecreateallthefactorsinvolved. Andifthereareoutlierhumanswhohavethisoddabilitytheywillnecessarilyfailasthenumberofdotsorthenumberofsuperimpo-sitionsincreases4Hutchins(2001).5EntryfromWikipediaonKimPeektheinspirationforthecharacterinthemovieRainMan:ÔÔHereadsabookinaboutanhour,andremembersalmosteverythinghehasread(É)Hisreadingtechniqueconsistsofreadingtheleftpagewithhislefteyeandtherightpagewithhisrighteyeandinthiswaycanreadtwopagesattimewitharateofabout8Ð10sperpage.Hecanrecallthecontentofsome12,000booksfrommemory.Teamchessisvirtuallyunknown,sotheevidenceforteamsbeingnobetterthanindividualsrests,sofar,ontheirbeinglittleornoteamchessmatches.OnenotableexceptionisAlexanderAlekhinewho Thisholdswhetherthesimulationormodelingisbeingperformedinternallyorexternally.VonNeumannputitlikethis:ÔÔItisnotatallcertainthatinthisdomainarealobjectmightnotconstitutethesimplestdescriptionofitself,thatis,anyattempttodescribeitbytheusualliteraryorformal-logicalmethodmayleadtosomethinglessmanageableandmoreinvolved.ÕÕ(p311)DavidMarrinvokingthesameidea,spokeofType2processeswhereanyabstractionwouldbeunreliablebecausetheprocessbeingdescribedevolvesastheresultofÔÔthesimultaneousactionofaconsiderablenumberofprocesses,whoseinteractionisitsownsimplestdescrip-tionÕÕ(Marr1977).Proteinfoldingandunfoldingareexamplesofsuchprocesses,accordingtoMarr.7Otherexamplesmightbethenbodyproblem,thesolutiontocertainmarketequilibriumproblems,situationswheretheoutcomedependsonthevotingofnparticipants,andcer-tainquantumcomputations.Thehallmarkoftheseproblemsisthatthereexistsphysicalprocessesthatstartandendinaninterpretablestate,butthewaytheygetthereisunpredictable;thefac-torsmediatingthestartandendstatearelargeinnumber,andonanyindividualrunareimpossibletopredict.Todeterminetheoutcome,therefore,itisnecessarytoruntheprocess,anditisbesttoruntheprocessrepeatedly.Notractableequationwillworkaswell.Howaretheseproblemstobesolvedifwehavenoaccesstotheprocessorsystemitself?Thenextbestthingistorunaphysicallysimilarprocess.Forexample,tocom-putethebehaviorofannbodysystem,suchasoursolarsystem,ourbesthopeistoconstructasmallanalogversionofthatsystemÑanorreryÑthenrunthemodel,andread delÕsproof.NewYorkUniversityPress,NewYorkPirolliPL(2007)Informationforagingtheory:adaptiveinteractionwithinformation.OxfordUniversityPress,CambridgePylyshynZ(2001)Istheimagerydebateover?Ifso,whatwasitabout?In:DupouxE(ed)Language,brain,andcognitivedevelopment:essaysinhonorofJacquesMehler.MITPress,Cambridge,MA,pp59Ð83QuineWVO(1960)Wordandobject.MITPress,Cambridge,MARobbinsP,AydedeM(eds)(2009)TheCambridgehandbookofsituatedcognition.CambridgeUniversityPress,Cambridge,pp264Ð306RussellDM,CardS,PirolliP,SteÞkM(1993)Thecoststructureofsensemaking,ProcCHI1993SacksO(2008)Musicophilia:talesofmusicandthebrain.AlfredA.Knopf,NewYorkScaifeM,RogersY(1996)Externalcognition:howdographicalrepresentationswork?IntJHumComputStud45:185Ð213ShepardRN(1978)Externalizationofmentalimagesandtheactofcreation.In:RandawaBS,CoffmanWE(eds)Visuallearning,thinkingandcommunication.AcademicPress,NewYorkStrawsonPF(1959)Individuals:anessayindescriptivemetaphysics.Methuen,LondonVonNeumannJ(1948/1961)Generalandlogicaltheoryofautomata.In:TaubAH(ed)JohnvonNeumann:collectedworks.VolumeV:designofcomputers,theoryofautomataandnumerical