D.H.WhalenKhalilIskarousHaskinsLaboratories,NewHaven,CTMarkK.TiedeHask - PDF document

D.H.WhalenKhalilIskarousHaskinsLaboratories,NewHaven,CTMarkK.TiedeHaskinsLaboratories,NewHaven,CT,andMassachusettsInstituteofTechnology,CambridgeDavidJ.OstryHaskinsLaboratories,NewHaven,CT,andMcGillUniversity,Montreal,Quebec,CanadaHeikeLehnert-LeHouillier etal.,1999),butthelimitedtimeforacquisitionisstillaseverelimitationonitsusefulness.Computedtomog-raphyhasalsobeenusedclinically(Larsson,Mancuso,&Hanafee,1982;Stutley,Cooke,&Parsons,1989).Anotherapproachtotonguemeasurementinspeechwasthedevelopmentofsystemsthattrackasmallnumberofpointsonthesurfaceofthetongue.Thesedependedeitheronagreatlyconstrained,andthereforemuchsafer,X-raymicrobeamsystem(Abbs&Nadler,1987;Kiritani,1986;Kiritani,Itoh,&Fujimura,1975)oralternatingmagneticfieldsgeneratedbycoilsplacedoutsidethehead,thatis,electromagnetometry(Perkelletal.,1992;Scho¨nleetal.,1987).Inbothcases,trackingisperformedonaforeignobject(goldpelletorreceivercoil)thathastobegluedtothetonguesurface.Whilethisinterferesminimallywitharticulationinmostcases(Weismer&Bunton,1999),placementisalengthyanddifficultprocedurethatisnottoleratedbyallpotentialparticipants.Thesesystemshaveallowedforextensivetrackingofpointsonthetonguesurfaceinrealtime.Manyareasinphoneticsandspeechsciencehavebenefitedfromexperimentsusingthesedevices:speechmotorcontrol(Hoole&Nguyen,1997;Lo¨fqvist&Gracco,1994;Perkell,Zandipour,Matthies,&Lane,2002;Westbury,1994a),phoneticvariation(Gick,Iskarous,Whalen,&Goldstein,2003),speecherrors(Pouplier&Goldstein,2002),andspeechdisorders(vanLieshout,Alfonso,Hulstijn,&Peters,1993;Weismer,Yunusova,&Westbury,2003).Imagingofthefullsurfaceofthetongue,alongwiththerestofthevocaltract,hasbeenbestaccomplishedtodatebymagneticresonanceimaging(MRI;Baer,Gore,Boyce,&Nye,1987;Baer,Gore,Gracco,&Nye,1991;Lakshminarayanan,Lee,&McCutcheon,1991;Rokkaku,Hashimoto,Imaizumi,Niimi,&Kiritani,1986).Bymanipulationofmagneticgradients,MRImakespossiblevolumetricimagingofvocaltractstruc-tures,withdistincttissuetypes,includingaspectsoftonguemusculature,distinguishedbytheirdifferingconcentrationsofimageableprotons.However,variousfactorsmakethisformofimaginglessthanideal,in-cludingthesupinepositionofthespeaker,theexpenseoftheexperimentalsessions,thenoisetheequipmentgenerates,andtherelativelyslowsamplingrate.Itsimagingofthevelumisunrivaled,however.Thisisespeciallytrueofrecentenhancementsincluding‘‘tagged’’MRI(Niitsuetal.,1994;Stoneetal.,2001),inwhichanexcitationgridissuperimposedonthetissuesothatdeformationsofcubesoftonguetissue(e.g.,duringspeech)canbemeasured(Stone,Epstein,&Iskarous,2004).Also,afewexperimentalsystemssupportsame-planesamplingratesofupto20Hz,makingimagingofdynamicspeechapossibility(Demolin,Metens,&Soquet,2000;Narayanan,Nayak,Lee,Sethy,&Byrd,2004).Thefinalimagingsystemforthetonguethatwewilldiscussisultrasound.TheapplicationofultrasoundtothevisualizationofspeecharticulationwaspioneeredbyStoneandcolleagues(Morrish,Stone,Sonies,Kurtz,&Shawker,1984;Stone&Davis,1995;Stone,Faber,Raphael,&Shawker,1992;Stone&Lundberg,1996;Stone,Sonies,Shawker,Weiss,&Nadel,1983),withsubstantialinputfromothersaswell(e.g.,Kaburagi&Honda,1994b).Hardwarehasnowimprovedtothepointwherethesignalsreturnedbystandardsettingsforultrasoundmachinescanreadilybeinterpretedastonguesurfacesandareamenabletoawidevarietyofmathematicaldescriptions(seeFigure1).AscanbeseeninFigure1,ultrasoundcapturesanalmostcompleteviewofthetongue,givingusmuchmoreinformationaboutthepharyngealregionthanisavail-ableinpointparameterizationsystems.Imagingspeedusingmodernultrasoundscannerscanreach200Hz,butsystemsthatrequireanalogvideorecordingoftheimagesforcetheframeratetodropto30Hz.Availabilityofdigitalvideocapability,however,hasremovedthisobstacle,andourlaboratoryhasbeguncollectingdataat200Hz.Spatialresolutionisdeterminedbyseveralimagingparametersandisabout1mm.Iftheprobeisallowedtomovewiththejaw,thenthepositionofthetongueiscapturedonlyrelativetothejaw—thepositionofthetonguewithinthevocaltractisnotcaptured.Ifweareonlyinterestedintheshapeofthetongue,thenmeasurementinajaw-centeredsystem Figure1.UltrasoundimageofthetonguesurfaceduringtheEnglishvowel/c¸/.Notetheshadowcastbythehyoidboneattheleftsideoftheimage. JournalofSpeech,Language,andHearingResearchVol.48June2005 issufficient,butifwearealsointerestedinconstric-tions,thelocationofthetonguewithinthevocaltractmustbemeasured.Onewaytoavoidthisproblemistoimmobilizetheheadandtheprobe(Munhall,Ostry,&Parush,1985;Ostry,Keller,&Parush,1983;Parush,Ostry,&Munhall,1983;Stone&Davis,1995;Wrench&Scobbie,2003).Becausethedistancebetweentheheadandproberemainsconstant,theshapeofthetongueiscapturedinahead-centeredcoordinatesystem,insteadofajaw-centeredone.However,whilethejawisstillabletomove(especiallyifanacousticstandoffisused),itisimpededbythestationaryprobe.Ithasbeenes-tablishedinmanyexperimentsonspeechmotorcontrolthatimpedingthejaw,evenbyslightamounts,triggerscompensatorymechanismsinthetongue(Kelso,Tuller,Vatikiotis-Bateson,&Fowler,1984;Lindblom,Lubker,&Gay,1979).Soitisnotknownwhetherthedatacollectedusingastationaryprobesetupreflectthesamepatternsasspeechutteredinalessrestrictivesetting.Immobilizationoftheheaditselfandrestrictingthegeneralpostureofthespeakercouldalsoleadtodeparturefromnaturalspeechpatterns.Further,somespeakers(e.g.,children,theelderly,andpersonswithcertainspeechdisorders)mayfinditdifficulttoundergotherequiredimmobilization.Thepresentarticledescribesasystemthattakesadvantageofultrasoundbutthatdoesnotrequireim-mobilization.Thesystem,theHaskinsOpticallyCor-rectedUltrasoundSystem(HOCUS),incorporatesbothultrasoundimagingofthetongueandopticaltrackingoftheproberelativetothehead,andthustonguesurfacedatacanbereorientedtoberelativetothehead.Thehead,probe,andjawareallowedtomove,buttheirmo-tionistrackedandcanthereforebeusedtocorrectthetonguemeasurementtoahead-basedcoordinateframe.Theprobemayeitherbeheldtoafixedorientationtocol-lectcross-sectionaldataduringrunningspeechormovedtodifferentorientationsduringsustainedphonationtoobtainmultiplecross-sectionsforthree-dimensionalreconstruction.Opticaltrackingdatacanalsobeob-tainedforthelipsandjaw(and,conceivably,othervis-iblestructures)inawaythatprovidesoneofthemostcompletemeasurementsofthevocaltractduringrun-ningspeechthatwehaveseen.Theimplementationissuesthathavetobedealtwithare(a)collectingthemeasurements,(b)aligningthemintime,(c)puttingthemintoacommoncoordinatesystem,and(d)extract-ingthemostrelevantspeechinformation.Theseissuesaredescribedbelow,afterwhichwewillconclude.Themaintwopiecesofequipmentinthesystemareanultrasounddeviceandanopticaltrackingdevice.Fortheultrasound,wecurrentlyuseanAlokaSSD-1000oranSSD-5500,thoughthesystemcouldfunctionwithvirtuallyanyultrasoundscanner.ThedatapresentedherewerecollectedwiththeSSD-1000,recordedonvideotape(withsimultaneousrecordingoftheaudiosignal).Fortheopticaltracking,weuseanOptotrakthree-dimensionalSystem3020fromNDI,whichalsosupportsconcurrentaudiorecording.Inthissystem,acameratracksthemotionofinfraredemittingdiodes(IREDs)placedontheprobeandhead(Ostry,Vatikiotis-Bateson,&Gribble,1997;Vatikiotis-Bateson&Ostry,1995).Aswiththeultrasound,thereareothertrackingsystems,whichuseeitheractiveorpassivemarkers,thatcouldprovidethesamefunctionality.Whatisimportantistheabilitytotrackmultiplepointsinthreedimensionsinrealtime.Theultrasoundtransceiverweuseisa3–5-MHzvariable-frequencyintercostalprobe.Ithasaviewingangleof90andanimagingdepthofabout17cm.Thisprobehasacurvedsurfacethatmakesitidealforobtainingsagittalimagesofthetongue.The90typicallyallowsusasfullaviewofaspeaker’stongueasispossible.Thehyoidboneisalimitingfactorintheposteriordirection,andtheshadowofthejawlimitstheviewintheanteriordirection.Becausetheultrasoundsignalreflectsoffofvari-oustissuesinproportiontothechangesinacousticalimpedance,wecanonlyimagethatpartofthevocaltractthathasacontinuousnonairmediumbetweenthetransceiverandtheobjectofinterest;theimpedanceoftheairistoogreattoallowthesignaltopassbeyondtheairboundary.Forthetongue,thismeansthatthetipwillnotbevisiblewheneveritprojectsoutoveranairpocket.Theanterior-mostpointofthetonguedetect-ableinultrasound,however,isnotfarfromthepositionofthe‘‘tonguetip’’markersinpointparameterizationsystems,whichistypically1cmposteriortotheactualtip(e.g.,Engwall,2003;Westbury,1994b).Anexperi-mentcomparingpointsourceandultrasoundtonguetrackingiscurrentlyunderwayandwilladdressthisissuesystematically.Knowingwheretheultrasoundimageisinrelationtotherestofthevocaltractisoneofthemostdifficultproblemstosolve.Itisnotnecessarytosolvethisproblemforeveryuse,asmanyinterestingfactsaboutthetongueshapecanbederivedfrom‘‘transceiver-centric’’images(Iskarous,Whalen,&Mattingly,2001),butthemostusefulinformationisobtainedbyalign-ingtheultrasoundimagerelativetotheupperskullsothatconstrictionscanbemeasuredorinferred.InHOCUS,wetracktheheadusingatleastthreeIREDsplacedonasetofgoggles.WedonotplacetheIREDsdirectlyontheheadsincethereareveryfewflesh-pointsontheheadthatdonotshiftrelativetotheskull.Therefore,wechoseasystemthatallowsforsomeofWhalenetal.:HaskinsUltrasoundSystem thisslidingtooccurbeneaththegoggles,whilethemainpointofstabilityremainsatthebridgeofthenose.Thegogglesareattachedtoanadjustableelasticbandthatistightenedsothatthegogglescannotmoverelativetothebridgeofthenose.Figure2showstheplacementoftheIREDsonthegogglesandontheWehavetypicallykeptthetransceiverinplacebyhandforHOCUS,thoughsecuringthetransceivertotheheadwithelasticbandshasalsoworked.Thehandheldapproachisnotoptimalsincethehandcanshift,butitdoeshavetheadvantageofallowinggreaterfreedomofthejawbecausetheprobeisheldatafairlyconstantpressure.However,becausethemotionoftheprobeistracked,itispossibletodetermineexactlyinwhichframestheprobehasrotatedorslidoutofplane,andthosedatacanbediscarded.Wedescribebelowaprocedurefordeterminingwhetherthemagnitudeofproberotationandtranslationoutofthemidsagittalplaneisacceptable.Thesystemweareaimingforisonethatinvolvessomerestrainttoholdthealignmentoftheprobetoasingleplaneandtokeepcontactwiththeskin;nomorerestraintshouldbeneeded.Thelessconstrainedthespeaker,themorelikelyitisthatherspeechwillresemblehernaturalspeechpatterns.Becauseweareabletocorrectforthemovementofthetransceiverrelativetothehead,oursystemallowsgreaterfreedomofmovementandlesspotentialim-pairmentofthespeechitselfthanheadrestraintsys-temsallow.Furthermore,becausetheprobe’spositionistracked,baddatacanbeidentifiedanderrorcanbequantified.Theultrasoundimageiscollectedatasamplingratethatisafunctionofthesettingsfordepthofimage,lineresolution,andscanangle.WiththeSSD-1000,weoftenusesettingsthatresultina57-Hzframerate.Torecordcontinuouslyinananalogsetting,therateisloweredbytheVCRto30Hz.Forshortburstsofdatathatfitwithintheultrasoundmachine’sinternalvideobuffer(24framesforourcurrentscanner),itispossibletocollectthedata,thenoutputeachframeofultrasounddatatothevideotapesothateachinternalframeisre-cordedmultipletimesonthevideotape.Afterthevideoframesareenteredintothecomputer,itispossibletoselectasinglevideoframethatrepresentsanultra-soundframe.Thesynchronizationwiththesoundislostinthistechnique,anditisdifficulttoaligntheacquisitionwithaparticularutterance,butitispossibletouseforspecialcircumstancesinwhichthehighersamplingrateiscrucial.Moresophisticatedultrasoundmachinessupportthecollectionofdigitalvideo,whichshouldallowthevideoframeratetobethesameastheultrasoundmachine-internalrateusedforcontinuousrecording.Wehavebeguntocollectdataat200HzusingarecentlyacquiredAloka5500scanner,butresultspresentedherearefromdatacollectedat57Hz,down-sampledto30HzbytheVCR.TheOptotraksystemsupportsmultipleIREDsandconcurrentacousticrecording.IndividualIREDsaretypicallytrackedat200Hz.Ourversionhasanoperatingrangeof2–4mfromthecamera.Root-mean-squareaccuracyat2mfromthecamerais0.1mmforcoordinatesand0.15mmforthe-coordinate.LipmarkersareshowninFigure2.DataCollectionHOCUSsessionsbeginwiththesigningofinformedconsentbytheparticipant.Acoordinatesystemisdefinedwiththeoriginattheupperincisorsandahorizontalplanecorrespondingtotheindividual’socclusalplane(Westbury,1994a).Thecoordinatesys-temisestablishedbysimultaneouslyrecordingtheposi-tionsofmarkersonthegogglesandonatriangle,heldbetweentheindividual’steeth,containingmarkersatknownlocations.Areferencepositionfortheprobeisrecordedseparately,withtheprobeheldbeneaththechinandtheteethclenched.Duringtheexperimentwetrackthemotionoftheprobeasarigidbodyrelativetothegogglesandhencerelativetotheocclusalplane.Next,weobtainatraceofthehardpalate.Thisisaccomplishedbyhavingtheparticipanttakeamouthfulofwaterandforceitupintocontactwiththehardpalate;thisisfollowedbyswallowingthewater.Thisensuresvisualizationofthepalatebecausetheimpedance Figure2.LocationoftheOptotrakinfraredemittingdiodes(IREDs)onthegoggles,theultrasoundtransceiver,andthelips.Theintersectingaxes,labeledA,B,andC,representthethreedirectionsaroundwhichrotationmayoccur:A,rotationaroundthelateraldimension(pitch);B,rotationaroundtheanterior/posteriordimension(roll);andC,rotationaroundtheverticaldimension(yaw). JournalofSpeech,Language,andHearingResearchVol.48June2005 differencebetweenwaterandairresultsinapatternthatisvisuallydistinctfromthedifferencebetweenwaterandbone.Weinstructtheparticipanttoforcethewaterupintocontactwiththehardpalate,sinceotherwisewhatisvisibleistheairlayerabovethewa-terandbelowthepalate(seeFigure3).Thepalatetracecanalsobeverifiedfromlaterportionsoftherunwhenthespeakerswallows(Wrench&Scobbie,2003).Be-causetheheadistrackedduringtheexperiment,theextractedboundaryofthepalatecanbeinsertedintoeveryframeofthespeechtrials.Afterthepalatetrace,weapplywhateveradditionalIREDsweneedfortrackingvisiblearticulators.Thesetup(includingocclusalplanedeterminationandpalatetrace)generallytakesabout20min.TheOptotrakdatacollectionworkswithindividualtrials,eachstartingwithabeep.Speechtrialsareusually1–2minlong.Forcalibration,theultrasoundtransceiverispositionedmanuallywhilethespeakerpronouncessomenonsensesyllables,andthemidsagittalpositionisselectedbasedonvisualinspectionoftheprobeandtheimage.Ex-ternally,theprobeisinlinewiththenose;internally,thetongueimagetendstobeatitsmostextremevalueatthemidline(fornongroovedsounds).Therearewaysofgivingtheholderofthetransceivermorefeedbackaboutthedesiredorientation.Forexample,alaserpointercanbeattachedtotheprobe,andthespeakercankeepthelaserpointwithinatargetonthewall(Gick,2002).Wehavenotadoptedthatstrategy,becausewewantthespeakertofocusfullyonthespeechtask.Wealsohavemeansofdetectingframesinwhichtheprobeorientationwasbeyondacceptablelimits.Althoughwehavesofardescribedthecollectionofmidsagittalimages,itisalsopossibletouseultrasoundinthecoronaldirection.Thiscanbeusefulindetermin-ingthepresenceanddepthofgroovingofthetongueatvariousindividuallocations.Multiplescansofasus-tainedphonationtakenatdifferentparasagittaloffsetsmaybecombinedtoproduceathree-dimensionalstaticrepresentationofthetongue(Honorofetal.,2003).Byusingmultipleutterances,itisalsopossibletobuildupathree-dimensionalreconstructionthatsimulatesavideoclipbytakingasingleframeattimepointsindifferentut-terancesthatareequivalenttotheframesthatwouldhaveoccuredinrealtime(Lundberg&Stone,1999).Consider-ableimageprocessingisrequiredforsuchreconstructions.DataAnalysisThevideorecordingoftheultrasoundsignalhastobedigitizedsothatfurtherprocessingcantakeplaceonthecomputer.(Thisprocess,ofcourse,isavoidedwiththedigitalvideosystem.)Thisportionofthepro-cessisthesameasthatforanyvideoinput.CorrectionofthevideosignalassumesthattheOptotrakdataandultrasoundvideoaresynchronized,whichisaccomplishedbyusingtheacousticsignalsthataresimultaneouslyrecordedontotheVCRandOptotrakunitsfromthesamemicrophone.Thetwosignalsarefirstresampledtoacommonrate,thentime-alignedbasedoncross-correlation.Thelaststepinthesynchronizationisdown-samplingtheOptotraksignaltoa30-Hzframeratetomatchtheultrasoundvideorate,ifnecessary.Thetonguesurfaceisextractedbyusinga‘‘snakes-based’’(seebelow)proceduresimilartoonedescribedbyIskarous(2005).First,asearchwindowthatcon-tainsthetongueedgeandnootheredgesischosenbytheuserbyinteractivelymanipulatingthesizeandshapeofthewindowusingfivecontrolpoints.Unlessthereiscontactbetweenthetongueandotherstructures,itshould Figure3.Ultrasoundimagetakenwhileawaterbolusisbeingheldinthemouth,allowingtheimagingofthehardpalate.(a)Thewaterbolusisinthemouth,butthereisapocketofairbetweenitandthepalate,givingafalseimpressionofwherethepalateis.(b)Inaframetakenjustasthespeakerswallows,thepalateismoreclearlyoutlined. Whalenetal.:HaskinsUltrasoundSystem bepossibletoconstructawindowcontainingonlythetonguesurface.Thepointswithinthewindowwithhigh-estluminancegradientarethenautomaticallydetected,andB-splinesarethenfittothegradientdatabyusingaleastsquarescriterion(Blake&Isard,1998).Figure4showsasampletonguesurfacefromtheultrasoundandthesplinethatisfittedtoit.Theprocedureistheniter-ativelyappliedtotheframesinafilm.Rigidbodytransformationisthenperformedtolocatetheultrasoundimageofthetonguerelativetotherestofthevocaltract.Thecamera-centricOptotrakdataforeachspeechtrialgivethelocationinspaceofeachoftheIREDsontheprobeandonthehead-mountedglasses,whichestablishheadposition.Todeterminetherigidbodycoordinates(translationsandrotations)oftheheadandprobe,asetofMATLABprocedurespreviouslydevelopedforjawmotiondetectionareused(Ostryetal.,1997).Atwo-stepoptimizationprocedureisusedtofirstcorrectforheadmotionrelativetothecameraandthentospecifythemotionoftheprobeinahead-centeredcoordinatesystemforthatframe.Foreachultrasoundframe,therigidbodyrecon-structionandcorrectionproceduresdeterminesixnum-bersspecifyingthepositionandorientationoftheprobeforthatframe.Threeofthesespecifythevertical,lateral,andhorizontalposition,andtheotherthreespecifythepitch,roll,andyaw.Figure2showsthecoordinatesys-temused.Becausetheimagedpartsofthetongueandthepalatechangedirectlywiththemovementoftheprobeandthehead,thetongueedgeandpalatesplinesarethenrotatedandtranslatedtocorrectforthemotionoftheprobeandheadfromframetoframe.However,notalltherigidbodycoordinatesareusedforcorrection.Formidsagittalimaging,correctiontotheedgeisbasedonthreeoftherigidbodycoordinatesoftheprobe:1,verti-caldisplacement;2,anterior-posteriordisplacement;and3,pitch.Variationinthesethreenumberscanbecor-rectedbecausetheimageisstillinthemidsagittalplane.Theotherthreenumbersareusedtodeterminewhetherthemidsagittaldataarewithinourtolerancelevels(see Figure4.Splinefittingofthetonguesurface.Thewhitestripeistheairlayerabovethetongueedge. Figure5.Measurementsofaverageinter-splineerrorbetween0andthevaluewithvaryingamountsofrotationaroundtheverticalaxis(yaw),forthreetrialsforeachofthreevowels(/a/,/e/,and/o/). JournalofSpeech,Language,andHearingResearchVol.48June2005 below).Underourcurrentdatacollectionprocedureinwhichtheexperimenterholdstheprobe,departurefromthedesiredplanedoesoccasionallyoccur,butweareabletoexcludesegmentsofvideocollectedwhentheprobewasoutofalignmentbecausetheremainingthreerigidbodycoordinatesspecifythelackofalign-ment.However,iftherotationortranslationoutofthemidsagittalplaneisofasmallmagnitude,thedatamaystillbeusable.Wehaveconductedtrialsinwhichwehavesystematicallyrotatedandtranslatedtheprobewhilemeasuringtongueshapeandalsomeasuringtherigidbodycoordinatesoftheprobe.Wethenmeasuredthedifferenceintongueshapeofaheldarticulationastheprobepositionvariedanddeterminedthresholdsbeyondwhichthedatawereunusable.Specifically,bysettingthresholdvaluesonthelateralpositionofthetransceiverandrollandyawangles,deviantframescanbediscarded.Figure5showsanexampleofthresh-olddetermination.Thedatainthisfigurewerecol-lectedbyrotatingtheprobeintheverticalplane(yaw)whiletheparticipantwassaying/a/,/e/,and/o/.Whatwasplottedistheaveragedistance(inmillimeters)be-tweenthemidsagittalsliceandeveryout-of-planeslice.Iftheprobeisrotatedlessthanabout5,theaverageerrorisatmost0.7mm.Astheprobeisrotatedevenmore,theshapeofthetonguechangesgreatlyandthedistancebecomesquitelarge.Thisdataarethenusedtosetathresholdbeyondwhichdatashouldbediscarded—inthiscase,thethresholdisabout5indicatingthat0.7mmaveragedeviationorlessisacceptable.Todetermineifdataofthistypearere-peatable,weperformedthesameexperimentonthesameparticipantthreetimes.Ascanbeseenfromthefigure,thedatafromthethreetrialsarequalitativelysimilar.Inourtypicalexperiments,thespeakerper-formsthistaskattheendoftheexperiment,andthresholdsaresetbyaveragingtheminimalvaluesofdeviationforthethreevowels.Typically,lateralmotionofthetransducerisharmfulonlyifitisgreaterthan2–4mm.Pitchandrolloflessthanabout5providesacceptabledata.Figure6showsarangeoftokensofEnglishvowelsin(h)Vdwords(e.g.,heed,aid)aftertheextractedtonguesurfacehasbeenorientedtothehardpalate.WhileitisnotpossibletotracktheindividualvowelsinFig-ure6a,itdoesshowthetotalvowelspaceasusedbythisfemaletalker.Figure6bshowsthepointvowels/a/,/i/,and/u/.Figure6cshowsthepair/u/,highlightingthedifferenceinthetongueroot.Theimaginginthepharyngealregionisextensiveandallowsustoeval-uatethecontributionofthispartofthevocaltracttospeech.Ifwecanassumethattherearpharyngealwallisfixed,itisevenpossibletomeasurechangesinpharyngealdepthbynotingchangesinthelocationoftheanteriorpharyngealwall(i.e.,thetongueroot).Distancesbetweenthetongueandthehardstruc-turesofthevocaltractcanbecomputedwithreasonablecertaintyinthepalatalregion,becausethepalateisimagedwiththesamesystemasthetongue.Wearealso Figure6.TongueshapesforEnglishvowels.Anterioristotheright.Theheavylineatthetopofeachpanelisthepalatetrace.(a)Tonguesurfacesforall11Englishvowelsfrom1femalespeaker.(b)Tracesforthreepointvowels.(c)Tracesforthetenselaxpairu/ Whalenetal.:HaskinsUltrasoundSystem attemptingtoincludethevocaltractobtainedfromMRIresultsforthesamespeakers.AnexampleisgiveninFigure7.Itappearslikelythatpostureaffectstherelationshipbetweentheposteriorpharyngealwallandthehardpalate,soitmaybethatMRIwouldneedtobedoneforavarietyofpossibleposturesandthenmatchedtothepostureadoptedbythespeaker.Ifanacceptablefitcanbemade,itshouldbepossibletopredicttheposteriorpharyngealwallpositionfromtheanteriorpo-sition,muchaswewereabletopredictanteriorpha-ryngealtonguepositionfairlywellfromfleshpointsonthetonguebody(Whalen,Kang,Magen,Fulbright,&Gore,1999).Becausetheshapeofthesupralaryngealvocaltractiscriticalfordeterminingtheacoustictrans-ferfunction,thisanalysisholdsthepromiseofpro-vidingafairlycompletedescriptionofthestructuresinvolved.Thiscanbetestedwitharticulatorysynthesis(Rubin,Baer,&Mermelstein,1981;Rubinetal.,1996),andthatsynthesiscaninturnbeimprovedwiththiskindofdata(Iskarous,Goldstein,Whalen,Tiede,&Rubin,2003).However,changesinthepositionoftheanteriorpharyngealwall(i.e.,thetongueroot)shouldbe Figure7.Superpositionofareconstructedpalateonamagneticresonanceimageofthesamespeaker(takenseparately). Table1.Comparisonofvariousspeechmeasurementsystemsthatimagethetongue.movementfeaturesofpopulations2-dimensionalmagnetometryFleshpoints(usually4)200–500HzNoRestrictedReceiversaffectarticulationFairlybroad3-dimensionalmagnetometryFleshpoints(usually4)Usually200HzNoFreeReceiversaffectarticulationFairlybroadUltrasoundFull-length30–200HzYesNoFreeProbeslightlyimpingesonjawBroadUltrasoundwithheadholder30–200HzYesNoRestrictedSomeeffectonjawFairlybroadUltrasound/OptotrakFull-length30–200HzYesNoFreeProbeslightlyimpingesonjawFairlybroad(HOCUS)StaticMRIFull-length—YesYesRestrictedSupinepositionLimitedCine-MRIFull-length8–24HzYesYesRestrictedSupinepositionLimitedX-raymicrobeamFleshpoints(usually4–5)40–160HzNoFreeSomeeffectsonspeechFairlybroadSomeoftheX-rayandcomputedtomographysystemsnowinusearenotdescribed.Thereareavarietyofthesesystems,withdifferentparameters,makingitdifficulttoincludetheminthistable.HOCUS=HaskinsOpticallyCorrectedUltrasoundSystem;MRI=magneticresonanceimaging.Onlyafewspeakerswithalowgagreflexareabletotolerateapharyngealpelletorreceiver.Itispossibletoglueareceiverorsutureapellettotheundersideofthevelum,butthisisrarelydone.Theultrasoundimageextendsfromjustabovethehyoidbonetonearthetipofthetongue.Wheneverthesignalhitsair,itdisappears,soanyportionofthetonguetipthatisoverthesublingualcavitywillnotbeimaged.Similarly,retroflextongueshapesarenotwellimaged.Note,however,thatthecoverageofthetipmaybesimilartothatofmagnetometryandmicrobeam,becausethereceiversforthetipareplaced1cmposteriortotheactualtip;anexperimentisunderwaytoseehowoftenthispointisimagedbyultrasound.Theinternalcombinationsofsettingsresultinamachineinternalsamplingrateofanywherefrom30to200Hz.Onmostmachines,theimagemustberecordedonvideotape,whichrunsat30HzforNorthAmericanvideotape.Machineswithdigitalimagingcanrecordatthetruesamplingrate.WeismerandBunton(1999)foundthatonlyafewindividualshadnoticeableeffectsofthepelletsontheirproductionofonesamplesentence.Someothersubgroupshadtendenciestowardperceptibleeffects. JournalofSpeech,Language,andHearingResearchVol.48June2005 directlyinterpretable,assumingonlysmallchangesinposture,becausetheposteriorwallhasbeenshownnottomovesignificantlyduringspeech(Magen,Kang,Tiede,&Whalen,2003).ComparisonAmongSystemsThevariouswaysofmeasuringthetonguehavedifferentstrengthsandweaknesses,asoutlinedinTable1.Ifstaticimagingistheresearchgoal,thenMRIisbetterthanothertechniques,becauseitprovidesmoreextensivecoverageofthevocaltract,butforrun-ningspeech,ultrasoundorpoint-trackingdevicesaresuperior.(ImprovementsinacquiringmultipleMRim-agesfromasingleutterancereducethisdifference.)Ifthegoalisanimagingsituationmostsimilartonaturalspeechconditions,theHOCUSsystemisideal,becauseheadmovementisfree,thereisminimalpressureonthejaw,andtheparticipantisupright.Butifthegoalisthecollectionofvelocityandaccelerationdataonpar-ticularpointsofthetongue,point-trackingdevicesortaggedMRIareappropriate,whereasHOCUSandba-sicMRIarenotasuseful.Resolutionisoftenthoughttobeamajorcompar-isonpointbetweenultrasoundandpoint-trackingsystems,butadirectcomparisonoftheaccuracyofmagnetometerandultrasounddatafoundthatthetwoagreedonthepositionofthepointsofthetonguesurfacetowithin1.16mm(Kaburagi&Honda,1994a).GreatertemporalresolutionisanadvantageofX-raymicrobeamandelectromagneticarticulographysystemscomparedwithultrasoundsystemswithonlyavideo-recordedoutput,butwiththeadventofdigitalvideo,temporalresolutionceasestobealimitingfactoronultrasound.Themaindifferenceisintheobjectofmeasurement,tongueshapesforultrasoundandpointmotionsforpoint-trackingdevices.Itiscertainlythecasethatthepointsareeasiertoquantifyandtoworkwiththanshapes,butmethodsforquantifyingtonguemovementfromultra-soundimageshaveshownhighreliabilityandreproduci-bility(Akgul,Kambhamettu,&Stone,1999).Newmethodsforglobalquantificationoftongueshapeusingonlyafewparameters(Iskarousetal.,2001)alsopromisetoclosethequantificationgap.Further,therearepatternsinarticulatormovementthatcannotbeseeninthemovementofjustafewpointsonthetongue,suchasthepivotsbetweenadjacentsegments(Iskarous,inpress).Recentadvancesinultrasoundtechnologyhavemadeitausefultoolformeasuringspeecharticula-tion,especiallyforimagingofthetongueinrunningspeech.Oursystem,HOCUS,whichcombinesultrasoundwithopticaltracking,allowsforrelativelyunobtrusivemeasurementofmostofthevocaltract.Thevarioustechniquescomplementeachotherinwaysthatwillallowusinthecomingyearstomeasurethespeechar-ticulatorsmorecompletelythanhaseverbeenpossi-ble.TherelativeeaseofuseofHOCUSwithnormalspeakersshouldallowformoreefficientuseoflabo-ratorytime,anditshouldmakethemeasurementofspecialpopulations—personswithdysarthia,theelderly,andchildren—muchmorefeasibleaswell.AcknowledgmentsThisworkwassupportedbyNationalInstitutesofHealthGrantDC-02717toHaskinsLaboratories.WethankthecontributionoftheATRHumanInformationProcessingResearchLaboratoriestothebaseroutinesfortherigidbodyreconstruction.WethankVinceGraccoandtwoanonymousreviewersforhelpfulcomments.Abbs,J.H.,&Nadler,R.D.User’smanualfortheUniversityofWisconsinX-raymicrobeam.Madison:UniversityofWisconsin—Madison,WaismanResearchAkgul,Y.S.,Kambhamettu,C.,&Stone,M.L.Automaticextractionandtrackingofthetonguecontours.IEEETransactionsonMedicalImaging,18,Baer,T.,Gore,J.C.,Boyce,S.E.,&Nye,P.W.ApplicationofMRItotheanalysisofspeechproduction.MagneticResonanceImaging,5,Baer,T.,Gore,J.C.,Gracco,L.C.,&Nye,P.W.Analysisofvocaltractshapeanddimensionsusingmagneticresonanceimaging:Vowels.JournaloftheAcousticalSocietyofAmerica,90,Blake,A.,&Isard,M.Activecontours:Theapplicationoftechniquesfromgraphics,vision,controltheoryandstatisticstovisualtrackingofshapesinmotionLondon:Springer.Carmody,F.(1941).Anx-raystudyofpharyngealUniversityofCaliforniaPublicationsinModernPhilology,21(5),377–384.Demolin,D.,Metens,T.,&Soquet,A.(2000).RealtimeMRIandarticulatorycoordinationsinvowels.InP.Hoole(Ed.),5thSeminaronSpeechProduction:Modelsanddata(pp.93…96).Munich,Germany:Ludwig-Engwall,O.(2003).CombiningMRI,EMAandEPGmeasurementsinathree-dimensionaltonguemodel.SpeechCommunication,41,Fitch,W.T.,&Reby,D.(2001).Thedescendedlarynxisnotuniquelyhuman.ProceedingsoftheRoyalSocietyofLondonSeriesB,BiologicalSciences,268,Gick,B.(2002).Theuseofultrasoundforlinguisticphoneticfieldwork.JournaloftheInternationalPhoneticAssociation,32,Gick,B.,Iskarous,K.,Whalen,D.H.,&Goldstein,L.M.(2003).ConstraintsonvariationintheproductionofWhalenetal.:HaskinsUltrasoundSystem English/r/.InS.Palethorpe&M.Tabain(Eds.),Proceedingsofthe6thInternationalSeminaronSpeech(pp.73–78).Sydney,Australia:MacquarieHonorof,D.N.,Chang,C.Y.C.,Iskarous,K.,Tiede,M.K.,Ostry,D.,&Whalen,D.H.(2003,September).Mandarin/r/asagroovedapproximant:ReconstructedtongueshapedatafromMRI&OPTOTRAK-ultrasoundPaperpresentedatSocietasLinguisticaEuropea,Lyons,France.Hoole,P.,&Nguyen,N.(1997).Electromagneticarticu-lographyincoarticulationresearch.ForschungsberichtedesInstitutsfu¨rPhonetikundSprachlicheKommunikationder¨tMu¨nchen,35,Iskarous,K.(2005).Detectingtheedgeofthetongue:AClinicalLinguisticsandPhonetics,19,Iskarous,K.(inpress).Patternsoftonguemovement.JournalofPhoneticsIskarous,K.,Goldstein,L.M.,Whalen,D.H.,Tiede,M.K.,&Rubin,P.E.(2003).CASY:TheHaskinsConfigurableArticulatoryspeechsynthesizer.InD.Recasens,M.-J.Sole´,&J.Romero(Eds.),Proceedingsofthe15thInternationalCongressofPhoneticSciences(pp.185–188).Barcelona,Spain:UniversitatAutonomadeBarcelona.Iskarous,K.,Whalen,D.H.,&Mattingly,I.G.Modelingtongueshapeswithconicarcs.JournaloftheAcousticalSocietyofAmerica,110,Kaburagi,T.,&Honda,M.(1994a).Determinationofsagittaltongueshapefromthepositionsofpointsonthetonguesurface.JournaloftheAcousticalSocietyofAmerica,96,Kaburagi,T.,&Honda,M.(1994b).Anultrasonicmethodformonitoringtongueshapeandthepositionofafixed-pointonthetonguesurface.JournaloftheAcousticalSocietyofAmerica,95,Kelso,J.A.S.,Tuller,B.,Vatikiotis-Bateson,E.,&Fowler,C.A.(1984).Functionallyspecificarticulatorycooperationfollowingjawperturbationsduringspeech:Evidenceforcoordinativestructures.JournalofExperimentalPsychology:HumanPerceptionandPerformance,10,Kiritani,S.(1986).X-raymicrobeammethodforthemeasurementofarticulatorydynamics:Techniquesandresults.SpeechCommunication,45,Kiritani,S.,Itoh,K.,&Fujimura,O.(1975).Tongue-pellettrackingbyacomputer-controlledx-raymicrobeamsystem.JournaloftheAcousticalSocietyofAmerica,57,Lakshminarayanan,A.V.,Lee,S.,&McCutcheon,M.J.(1991).MRimagingofthevocaltractduringvowelproduction.JournalofMagneticResonanceImaging,1,Larsson,S.G.,Mancuso,A.,&Hanafee,W.Computed-tomographyofthetongueandflooroftheRadiology,143,Lindblom,B.E.,Lubker,J.,&Gay,T.(1979).Formantfrequenciesofsomefixed-mandiblevowelsandamodelofspeechmotorprogrammingbypredictivesimulation.JournalofPhonetics,7,¨fqvist,A.,&Gracco,V.(1994).Tonguebodykinematicsinvelarstopproduction:Influencesofconsonantvoicingandvowelcontext.Phonetica,51,Lundberg,A.J.,&Stone,M.L.(1999).Three-dimensionaltonguesurfacereconstruction:Practicalconsiderationsforultrasounddata.JournaloftheAcousticalSocietyofAmerica,106,Magen,H.S.,Kang,A.M.,Tiede,M.K.,&Whalen,D.H.(2003).Posteriorpharyngealwallpositionintheproductionofspeech.JournalofSpeech,Language,andHearingResearch,46,Morrish,K.A.,Stone,M.L.,Sonies,B.C.,Kurtz,D.,&Shawker,T.(1984).Characterizationoftongueshape.UltrasonicImaging,6,Munhall,K.G.,Ostry,D.J.,&Parush,A.Characteristicsofvelocityprofilesofspeechmovements.JournalofExperimentalPsychology:HumanPerceptionandPerformance,11,Munhall,K.G.,Vatikiotis-Bateson,E.,&Tohkura,Y.X-rayfilmdatabaseforspeechresearch[Videodisc].Kyoto,Japan:ATRLaboratories.Narayanan,S.,Nayak,K.,Lee,S.,Sethy,A.,&Byrd,D.(2004).Anapproachtoreal-timemagneticresonanceimagingforspeechproduction.JournaloftheAcousticalSocietyofAmerica,115,Niitsu,M.,Kumada,M.,Campeau,N.G.,Niimi,S.,Riederer,S.J.,&Itai,Y.(1994).Tonguedisplacement:Visualizationwithrapidtaggedmagnetization-preparedRadiology,191,hman,S.E.G.(1966).CoarticulationinVCVutterances:Spectographicmeasurements.JournaloftheAcousticalSocietyofAmerica,39,Ostry,D.J.,Keller,E.,&Parush,A.(1983).Similaritiesinthecontrolofspeecharticulatorsandthelimbs:Kinematicsoftonguedorsummovementinspeech.JournalofExperimentalPsychology:HumanPerceptionandPerformance,9,Ostry,D.J.,Vatikiotis-Bateson,E.,&Gribble,P.L.(1997).Anexaminationofthedegreesoffreedomofhumanjawmotioninspeechandmastication.JournalofSpeech,Language,andHearingResearch,40,Parush,A.,Ostry,D.J.,&Munhall,K.G.(1983).AkinematicstudyoflingualcoarticulationinVCVsequences.JournaloftheAcousticalSocietyofAmerica,74,1115–1125.Perkell,J.S.Physiologyofspeechproduction:Resultsandimplicationsofaquantitativecineradiographic.Cambridge,MA:MITPress.Perkell,J.S.,Cohen,M.H.,Svirsky,M.A.,MatthiesM.L.,Garabieta,I.,&Jackson,M.T.T.Electromagneticmidsagittalarticulometersystemsfortransducingspeecharticulatorymovements.JournaloftheAcousticalSocietyofAmerica,92,Perkell,J.S.,Zandipour,M.,Matthies,M.L.,&Lane,H.(2002).Economyofeffortindifferentspeakingconditions.I.Apreliminarystudyofintersubjectdifferencesandmodelingissues.JournaloftheAcousticalSocietyofAmerica,112,Pouplier,M.,&Goldstein,L.M.(2002).Asymmetriesinspeecherrors:Production,perceptionandthequestionofunderspecification.InT.A.Hall,B.Pompino-Marschall,JournalofSpeech,Language,andHearingResearchVol.48June2005 &M.Rochon(Eds.),Papersonphoneticsandphonology:Thearticulation,acousticsandperceptionofconsonants(pp.73–82).Berlin,Germany:ZAS.Rochette,C.Lesgroupesdeconsonnesenfranc¸ais´bec,Canada:LesPressesdel’Universite´Laval.Rokkaku,M.,Hashimoto,K.,Imaizumi,S.,Niimi,S.,&Kiritani,S.(1986).Measurementofthethree-dimensionalshapeofthevocaltractbasedonthemagneticresonanceimagingtechnique.AnnualBulletinRILP,20,Rubin,P.E.,Baer,T.,&Mermelstein,P.(1981).Anarticulatorysynthesizerforperceptualresearch.oftheAcousticalSocietyofAmerica,70,Rubin,P.E.,Saltzman,E.,Goldstein,L.,McGowan,R.,Tiede,M.,&Browman,C.(1996,May).CASYandextensionstothetask-dynamicmodel.Paperpresentedatthe1stESCATutorialandResearchWorkshoponSpeechProductionModelingand4thSpeechProductionSeminar,Autrans,France.Russell,G.O.Thevowel:Itsphysiologicalmechanismasshownbyx-ray.Columbus:OhioStateUniversityPress.¨nle,P.,Grabe,K.,Wenig,P.,Hohne,J.,Schrader,J.,&Conrad,B.(1987).Electromagneticarticulography:Useofalternatingmagneticfieldsfortrackingmovementsofmultiplepointsinsideandoutsidethevocaltract.andLanguage,31,Stark,J.,Ericsdotter,C.,Branderud,P.,Sundberg,J.,Lundberg,H.-J.,&Lander,J.(1999).TheAPEXmodelasatoolinthespecificationofspeakerspecificarticulatorybehavior.InJ.J.Ohala,Y.Hasegawa,M.Ohala,D.Granville,&A.C.Bailey(Eds.),ofthe14thInternationalCongressofPhoneticSciences(pp.2279–2282).SanFrancisco:UniversityofCalifornia,Stevens,K.N.,&Ohman,S.E.G.(1963).Cineradiographicstudiesofspeech.KTHSTL-QPSR,2,Stone,M.L.(1997).Laboratorytechniquesforinvestigatingspeecharticulation.InW.J.Hardcastle&J.Laver(Eds.),Thehandbookofphoneticsciences(pp.11–32).Oxford,England:Blackwell.Stone,M.L.,&Davis,E.P.(1995).Aheadandtransducersupportsystemformakingultrasoundimagesoftongue/jawmovement.JournaloftheAcousticalSocietyofAmerica,98,Stone,M.L.,Davis,E.P.,Douglas,A.S.,Aiver,M.N.,Gullapalli,R.,Levine,W.S.,etal.(2001).Modelingtonguesurfacecontoursfromcine-MRIimages.JournalofSpeech,Language,andHearingResearch,44,Stone,M.L.,Epstein,M.,&Iskarous,K.(2004).Func-tionalsegmentsintonguemovement.ClinicalLinguisticsandPhonetics,18,Stone,M.L.,Faber,A.,Raphael,L.J.,&Shawker,T.H.(1992).Cross-sectionaltongueshapeandlinguopalatalcontactpatternsin[s],[],and[l].JournalofPhonetics,Stone,M.L.,&Lundberg,A.(1996).Three-dimensionaltonguesurfaceshapesofEnglishconsonantsandvowels.JournaloftheAcousticalSocietyofAmerica,99,3728–3737.Stone,M.L.,Sonies,B.C.,Shawker,T.H.,Weiss,G.,&Nadel,L.(1983).Analysisofreal-timeultrasoundimagesoftongueconfigurationusingagrid-digitizingsystem.JournalofPhonetics,11,Stutley,J.,Cooke,J.,&Parsons,C.(1989).NormalCTanatomyofthetongue,floorofmouthandoropharynx.ClinicalRadiology,40,vanLieshout,P.H.H.M.,Alfonso,P.J.,Hulstijn,W.,&Peters,H.F.M.(1993).Electromagneticarticulography(EMA)instutteringresearch.Institutfu¨rPhonetikundsprachlicheKommunikationderUniversita¨tMu¨nchen—Forschungsberichte,31,Vatikiotis-Bateson,E.,&Ostry,D.J.(1995).Ananalysisofthedimensionalityofjawmotioninspeech.JournalofPhonetics,23,Weismer,G.,&Bunton,K.(1999).Influencesofpelletmarkersonspeechproductionbehavior:Acousticalandperceptualmeasures.JournaloftheAcousticalSocietyofAmerica,105,Weismer,G.,Yunusova,Y.,&Westbury,J.R.Interarticulatorcoordinationindysarthria:AnX-raymicrobeamstudy.JournalofSpeech,Language,andHearingResearch,46,Westbury,J.R.(1994a).Oncoordinatesystemsandtherepresentationofarticulatorymovements.JournaloftheAcousticalSocietyofAmerica,95,Westbury,J.R.(1994b).X-raymicrobeamspeechproductiondatabaseuser’shandbook[Softwaremanual].Madison:UniversityofWisconsin—Madison,WaismanResearchCenter.Whalen,D.H.,Kang,A.M.,Magen,H.,Fulbright,R.K.,&Gore,J.C.(1999).Predictingpharynxshapefromtonguepositionduringvowelproduction.JournalofSpeech,Language,andHearingResearch,42,Wood,S.X-rayandmodelstudiesofvowel(WorkingPaperNo.23).Lund,Sweden:LundUniversity,DepartmentofLinguistics.Wrench,A.A.,&Scobbie,J.M.(2003).CategorisingvocalisationofEnglish/l/usingEPG,EMAandultrasound.InS.Palethorpe&M.Tabain(Eds.),Proceedingsofthe6thInternationalSeminaronSpeech(pp.314–319).Sydney,Australia:MacquarieReceivedMay8,2004AcceptedOctober22,2004DOI:10.1044/1092-4388(2005/037)Contactauthor:D.H.Whalen,HaskinsLaboratories,300GeorgeStreet,NewHaven,CT06511.E-mail:whalen@haskins.yale.eduWhalenetal.:HaskinsUltrasoundSystem