12thInternationalSocietyforMusicInformationRetrievalConference(ISMIR20 - PDF document

12thInternationalSocietyforMusicInformationRetrievalConference(ISMIR2011) 353 ETHNOGRAPHICOBSERVATIONSOFMUSICOLOGISTSATTHEBRITISHLIBRARY:IMPLICATIONSFORMUSICINFORMATIONRETRIEVALMathieuBarthetCentreforDigitalMusicQueenMaryUniversityofLondonmathieu.barthet@eecs.qmul.ac.ukSimonDixonCentreforDigitalMusicQueenMaryUniversityofLondonsimon.dixon@eecs.qmul.ac.ukABSTRACTWithoutarichunderstandingofuserbehavioursandneeds,musicinformationretrieval(MIR)systemsmightnotbeide-allysuitedtotheirpotentialusers.Inthisstudy,wefol-lowedanethnographicmethodologytoelicitsomeofthestrategiesusedbymusicologiststoexploreanddocumentmusicalperformances,inordertoinvestigateifandhowtechnologiescouldenhancesuchaprocess.ObservationsofmusicologistsstudyinghistoricalrecordingsofclassicalmusicwereconductedattheBritishLibrary.Theobserva-tionsshowthatthemusicologistsalternatebetweenaclosedlisteningpractice,relyingexclusivelyonauralobservations,andamultimodallisteningpractice,wheretheyinteractwithvariousmusicrepresentationsandinformationsourcesus-ingdifferentmedia(e.g.metadataabouttherecordingsandperformers,soundvisualisations,scores,lyricsandperfor-mancevideos).Thespokenpartsofbroadcastrecordingsbroughthistorical/extra-musicalclueshelpingtounderstandmusicperformancepractices.Soundvisualisationandcom-putationalmethodsfosteredtheanalysisofspecicmusi-calexpressionpatterns.Wesuggestthatsoftwaredesignedformusicologistsshouldfacilitateswitchingbetweenclosedandmultimodallisteningmodes,interactionwithscoresandlyrics,andanalysisandannotationofspeechandmusicper-formanceusingcontent-basedMIRtechniques.1.INTRODUCTIONTheinterdisciplinaryresearchareaofmusicinformationre-trieval(MIR)hasdevelopedfromtwoneeds:managingin-creasingcollectionsofmusicmaterialindigitalform,andsolvingfundamentalproblemsrelatedtomusicanalysisandperception[1].Overthepastdecade,awidevarietyofMIRtechniquesandtoolshavebeendevelopedusingvar-ioustypesofmusicrepresentations(audio,symbolic,vi-Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprotorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationontherstpage.c 2011InternationalSocietyforMusicInformationRetrieval.sualandmetadata).However,asCunningham[2]pointsout,theyhaveoftenbeendesignedbasedonanecdotalevi-dence,intuitivefeelings,oraprioriassumptionsofuserbe-havioursandneeds.Withoutarichunderstandingofthelatter,systemsdesignedusingMIRresearchmightnotbeideallysuitedtotheirpotentialusers.Bridgingthegapbe-tweentheresearchlaboratoryandreal-worldsituationsisoneofthegoalsofthisstudy.Thisrequiresworkingwithspecicusergroupsinordertobetterunderstandtheiractiv-ityandhowtheyinteractwithtechnologies.Wefocusedonelicitingsomeofthestrategiesusedbymusicologiststoexploremusicaldocuments,andtheinter-actionswithmusic-relatedtechnologiesduringthisprocess.Bonardi[3]proposedinterestingsolutionstoimprovemusi-cologists'workstationsusingMIRtechnologiesbyexamin-ingtheirneedswhenanalysingthecontemporarycatalogueatIRCAM'sdigitallibrary.Hestatedthattheworkstationsshouldallowvariousrepresentationsofmusic(e.g.graphi-cal,sound,andsymbolic),listeningtorecordingswhilecon-sultingdifferentmusicaldocuments(`active'listening),andreading(e.g.thescore)andwritingusingthesamemedia.Inthisstudy,wesoughttoobtainevidencetotestthevalidityofsuchstatements,andwhethertheywouldberel-evantinadifferentcontext(setting,differenttypesofmu-sicologicalstudies,andmusicalrepertoires).Weconductedanethnographicstudybasedontheobservationofmusicol-ogistsworkingwithclassicalmusicrecordingsattheBritishLibraryinLondon.Theethnographicmethodisaqualita-tiveapproachbywhichndingsarenotinferredfromsta-tisticaltestsbutfromthedetailedanalysisofthebehavioursandactionsoftheparticipantsacrossalargenumberofeldobservations.Theoutcomesofourresearcharetwofold:theygiveinsightsonhowtoadaptorimproveexistingdigi-talmusictechnologiestotuserneedsbetter(e.g.MIRtech-niques,softwarefeatures,userinterfacedesign),andtheycanraiseideasforthedevelopmentofnewsystems.Theremainderofthepaperisstructuredasfollows.Sec-tion2presentsthesettingandmethodsoftheethnographicstudy.Sections3,4,and5aredevotedtothematicanaly-sesbasedontheobservations.WediscussthendingsinSection6andgiveaconclusioninSection7. OralSession5:UserStudies 354 2.SETTINGANDMETHODSViatheEdisonFellowshipscheme,theBritishLibrary(BL)encouragesmusicologicalstudiesdevotedtothehistoryofrecordingsofclassicalmusicandmusicinperformance,bycreatingtheconditionsforconcentrateduseoftheLibrary'srecordingscollectiontoscholarsselectedonayearlyba-sis.TheBL'ssoundarchivecountsmorethan3.5millionpublishedandunpublishedrecordingsofsoundsandmu-sicincludingmanyuniquehistoricitems.InanerawheretheLibrarydevelopstheaccessandanalysisofdigitisedrecordings,andgiventhecloselinkbetweenMIRandli-braryscience(e.g.representation,classication,metadata),theBLisagoodsettingforinvestigatingthepossiblerolesofmusic-relatedtechnologiesinmusicologicalresearch.WecontactedfourEdisonFellowswiththecollaborationoftheBL'smusicdepartmentstaff,andobtainedtheirconsenttoparticipateinthestudy.Thenamesusedintheeldnotespresentedthroughoutthearticlearepseudonyms.ThegroupwasformedoftwoBritishandtwoAmericanmales(averageage38).Theirprofessionalactivitiesincludedresearchandteachingpositions(PhDinmusicology,lecturerinmusic,singingteacher),aswellasperformance(pianist,singer).Twoofthemhadreceivedtraininginscienceandtechnol-ogy.Themusicalrepertoirestheystudiedwerevaried:earlymusic(e.g.medievaldance,vocalandconsortmusic),clas-sicalandromanticmusic(e.g.artsongs,operas,pianosolopieces),andcontemporarymusic(electronicmusic).Wechoseanethnographicmethodologyprimarilybasedonparticipantobservation[4].Oneoftheadvantagesofob-servingtheactionsofparticipantsperformedinaconcretesettingisthatitgivesaccesstowhatpeopledoandhow(behaviours)ratherthanwhatpeoplesay(attitudes),thelat-terbeingobtainedwithotherqualitativemethodssuchassurvey,questionnaire,orinterviews.Furthermore,stayingforarelativelylongperiodoftimeintheenvironmentofthegroupstudiedfostersthecollectionofrichdetailswhichwouldotherwisedemandahighdegreeofself-awarenessandagreatpowerofrecallforpeopletoreportoutofthecontextoftheactivity.Toachievethislevelofdetail,itisnecessarytofocusonasmallnumberofparticipants.Thistradeoffofquantityforqualityiscommonindisciplinesre-lyingonqualitativemethods(e.g.psychology)[4,5].Theobservationsweremadebyfocusing(i)onthepro-cessesunderlyingmusicologicalresearch,and(ii)onthere-lationshipswithmusic-relatedtechnologiesandtheirrolesduringsuchprocesses.TheobservationstookplaceinthemusicdepartmentoftheBLwheretheFellowshadare-serveddeskspaceattheirdisposal.Theywereconductedbyoneethnographerduringrepeatedvisits(twiceaweek,onaverage)overaperiodofthreemonths.Theobservationaldatawerecollectedbytakingeldnotesusinganotebookandpen.DuetotheregulationsoftheBritishLibraryandinordertominimizedisturbancetothestaff,video/audiorecordingswerenotused.Ancillarysourcesofinformationwerealsousedinadditiontotheobservations.Ethnographicinterviews[4]occurredduringtheresearchintheeldinor-dertoshedlightonspecictasksandtohaveadeeperunder-standingofthescopeofthestudiesoftheparticipant.Someoftheparticipants'ownworkingnoteswerealsoemployed,withtheirconsent.Thecollecteddatawereanalysedusingtheapproachproposedin[6],whichdrawsfrommethodsdevelopedbysociologistsfollowingthegroundedtheory:codingoftheeldnotes(identifyingandnamingspecican-alyticdimensionsandcategories),andanalysisbythemeswhichreectrecurrentorunderlyingpatternsofactivity.3.USEOFRECORDINGS3.1RetrievalandmetadataMetadatawereusedtofacilitatetheretrievalofrecordingsintheBL'scatalogue(byusingdetailssuchastherecordnum-ber,thelabel,ortheconductor'sname).Additionalmeta-datawerefetchedduringthelisteningprocess(seeSubsec-tion4.2),usingvarioussourcesofinformation:theknowl-edgeoftheLibrary'scurators,theweb,therecordings'car-riers,linernotes,oraccompanyingmanuscriptdocuments(e.g.apapercardsystemthatanoriginalcollectorhadkept).3.2FormatandplaybacktechnologiesTherecordingsalreadydigitizedwereimmediatelyaccessi-blethroughtheBritishLibrarySoundServerasMP3les.Whentherecordingswereuniqueorheldonfragilefor-mats(e.g.reeltoreeltapes),theFellowswereprovidedwithanalogcopiesoftherecordingsordigitizedversionsonau-dioCDs,orlesscommonly,VHStapes(PCM).Theana-logformatsincludedreeltoreeltapes,compactaudiocas-sette(K7),aswellaslong-playing(LP)and78rpmdiscs.Therecordingsheldonaphysicalsupportwereplayedus-ingdedicatedplaybackequipmentconnectedtoanampli-er.TheMP3lesfromtheSoundServerwereplayedfromthedesktopcomputerusingWindowsMediaPlayerClas-sic.Insomecases,theyalsolistenedtoandanalysedownedcommercialrecordingswiththeirlaptopsusingiTunestoplayrecordings,andSonicVisualiser1asaplayerandanal-ysistool(seeSubsection4.2).ForsomeoftheFellows,theformatwasnotanissuesincetheywereinterestedinthecontentoftherecordingsandnotthecarrieritself.Inthatcase,theywerenotbotheredbyuseofMP3lesratherthanuncompresseddigitalororiginalanalogrecordings.Onthecontrary,digitalrecordingswerepreferredbecausethenavi-gationinrecordingswasmadeeasierandquicker.However,otherspreferredtodealwithrecordingsintheiroriginalfor-mat(“Thereismorecontextwhenyouhavetheoriginal,thelabels,howitwasheldforinstance.WithmostMP3syoudolosesomething.Idowonderwhethersometimesyou'relosingthecoreproduct.”). 1http://www.sonicvisualiser.org/ 12thInternationalSocietyforMusicInformationRetrievalConference(ISMIR2011) 355 4.LISTENINGANDOBSERVING4.1ListeningpracticesThelisteningprocesslayatthecenterofthestudyoftherecordings.Themusicologistscommonlyalternatedtwodistinctbutcomplementarypracticesoflistening.Intherstlisteningpractice,theanalysisoftherecordingswasperformedexclusivelythroughauralobservations.Thesec-ondlisteningpracticewasmultimodalandcharacterisedbyaninteractivitywithmusical,textualorvisualdocumentsenrichingormodifyingtheauralobservations.4.1.1ClosedlisteningClosedlisteningwascharacterisedbyacarefulandfocusedlisteningtotherecordingwithoutusinganyothersourceofinformationthanthatprovidedbythesound:WilliamputhisheadphonesontolistentoTelemann'sConcertoinFfor3recorders,2oboes,2violins,andcontinuo,performedbytheEarlyMusicConsort.AfterstartingtherecordingintheCDplayer,hesatbackinhischair,closedhiseyes,andlis-tenedcarefullytothemusic.Amomentlater,Inoticedthathewastappingthebeatwithhisfoot.Thisexampleshowshowtheauralexperiencebecameaphysicalone(tappingthebeatwiththefoot)whileretrievinginformationaboutthetimingofthemusicalpiece(tempo).Intheclosedlisteningmode,themusicologistsdrewauralobservationsinvolvingperceptualandcognitiveaspects(arecollectionofthescore,forinstance).Eitherinparallelorshortlyafterthelisteningprocess,theywrotedowntheirauralobservationsbyhandorusingatexteditor.Typednoteshadtheadvantagethattheycouldbequeriedquicklybyusingkeywordssuchasthenameofacomposer.4.1.2MultimodallisteningAdifferentpracticewascharacterisedbytheuseofvari-ousmusic-relateddocuments(e.g.thebiographyofacom-poser,informationontherecording)andmusicrepresen-tations(e.g.scores,featurevisualisations)whilelistening.Thislisteningpracticecanbedescribedasanactivepro-cess[3],sinceitdoesnotjustconsistofreceivingmusicalinformation,butisonthecontrarybasedonasetofmul-timodalinteractionsbetweenthelistenersandmusicaldoc-uments.Theadvantagesofusingmultiplemodalitieswereanincreasedaccesstomeaning,uncoveringthecontextofarecordingandtheintentionsofcomposers,conductors,orperformers,andbetterunderstandingoftheperceptionofthemusic.Multimodallisteningwasperformedbyvaryingthemediaandtechnologiesusedtodocumentthemusicalrecordings.4.2Documentingthemusicrecordings4.2.1ContextualinformationInthemultimodallisteningpractice,themusicologistscom-monlyusedwebresourcestoseekseveraltypesofinforma-tionrelatedtotherecordings:contextual(ndingmetadataaboutamusicalpiece,forinstance),bibliographic(musicartists'websites,Wikipedia),aswellasvisualandicono-graphic(YouTubevideosweresometimesusedtouncovervisualaspectsofperformance,GoogleImageswasusedtoprovidepicturesofspecicmusicians).Suchresourceswerealsousedwithoutlisteningtotherecordings.4.2.2ScoresandlyricsTheonlinemusicsheetdatabasefromtheInternationalMu-sicScoreLibraryProject2wasoftenusedtoretrievepublicdomaineditionsofscores,whichareprovidedasscannedimagesinPDFformat.Someofthemusicologistsreadthescoreusingaprintedcopy,whileothersusedtheelectronicformatandfollowedthemusicwiththemousewhilelisten-ing.Whentheywereavailable,scoreswereusedbothinordertoretrievegeneralinformationsuchasthekeyofapiece,andmoredetailedinformationthroughacloseanaly-sisofthenotesandexpressivenotation.Singingwhileread-ingthescorewassometimesusedtondthescaleusedbythecomposer(e.g.Lydianmode).Scoresactedasarefer-enceagainstwhichtotestwhethertheintentionsofthecom-poserwererespectedbyperformers,asthefollowingnotesshow:“Seemsreallyconsistentwithmarkingsinthescore.Beautifullysung-singingthenotevaluesandgenerallythedynamicswrittenbySamuelColeridge-Taylor.”,“Ismuchfreerwiththeinterpretationofthescore.Interpolatesahighnoteattheendandchangesthemelodiclineattheendofthesong.”Inthecaseofvocalmusiccompositions,readingthelyricswhilelisteningalsohelpedtofollowthemusicalstructureandtounderstandtheexpression,asshowninthisnotedescribingthetimbreoftheperformer'stonesbyref-erencetothelyricsratherthanthepitch:“quiteshrillandshakyon`Awindcomesandletmebe',andmoremellowon`saiditslow'.”4.2.3Soundvisualisation,acousticalanalyses,andtime-stretchingMusicologistswithpreviousbackgroundinmusictechnolo-gies(comingeitherfromtheireducation,personaltraining,orfromcollaborationwithcomputerscientists)alsousedsoftware(SonicVisualiser)toanalyseandvisualisemusicrecordings.Thevisualisationofthewaveformwashelp-fultonavigatedigitalrecordingsbyjumpingbetweensec-tionsthathavedifferentdynamics(e.g.betweenaspokenpartandthestartofanorchestralpart,forinstance).Spec-trogramrepresentationswereusedtoanalysethesubtletiesofexpressiveeffectsuchasthevibrato:“IfI'mlookingat 2http://www.imslp.org/ OralSession5:UserStudies 356 awaveform[theonefromatone'spartial]andIcanseethereisvibratointhenote,Ihearitmuchbetter”.Suchacousticalanalyseshelpedtounderstandtheperceptualef-fectsexperiencedwhenlistening:“Thespectrogramshowsyouthattherealskilltoher[EmmaKirkby's]vibratouseisthatthenotestartswithveryveryminimalvibrato.Soyourmindisfedaveryaccuratepitch,beforethepitchisthendecoratedbyvibrato.Sothat'swhyyouhearitassuchapurevoice,becauseshe'salreadytoldyoutheinforma-tionaboutexactlywhatthenoteisbeforeitvibrates,soyourbrainsomehowkeepsonthatcentraltuningissueduringthevibration[...]Whereassingersthatimmediatelystartwithvibrato,youcanneverreallytellwhatthey'resinging.”Acousticalmeasurementswereperformedfromthespec-trogramrepresentations(ameasuretoolisprovidedinSonicVisualiser)inordertocharacterisethepropertiesofvibrato(frequency,andpitchextent).Thesemeasurementswereconductedinasystematicwayforvariousperformersbycomparinglongsustainednotes.Theresultingquantitativedatagavecluestounderstandornuanceauralobservationsmadeonvibratobyothermusicologists:“Theminimalvi-bratosoundsthatMunrowlisted[...]wereallfasterandshallowerthantheotherexamplesofvibrato.WhenthisiscombinedwithMunrow'sownexplicitdisapprovalofcon-stantvibrato,webegintounderstandthatheissuggestingapreferencefor`controlled'vibrato”.Thetime-stretchingtechniqueprovidedbySonicVisu-aliserwhichpreservestheoriginalpitchandtimbrewasalsousedtoproduceslowed-downversionsofnotesormusicalpassages.Theseslowed-downexcerptswereplayedwhilevisualisingscrollingspectrogramrepresentationsgivingthetimetotheearandtheeyetouncovernedetails:“IknewsomethingwasupthroughlisteningbutIcouldn'ttellwhatwasup,andthenwhenIvisualised...whenIsloweddown,moreofitmadesense,Irealisedthevibratowasnotconsis-tent,butIcouldn'tworkoutthatitstartedwithoutvibratowithoutthespectrogram”.Spectrogramanalysesandtime-stretchingwerealsousedtovalidateintuitionsobtainedwithauralobservationstoexplainthetechniqueandexpressionofapianist.Bylookingatthealignmentofthenotesonthespectrogramwhilelisteningtoaslowed-downpassage,sub-tledifferencesoftimingbetweenchordnotesplayedbytheleftandtherighthandswerenoticed.5.MUSICALFEATURES5.1InstrumentationandtuningDetailsabouttheinstrumentationwereretrievedinseveralways:fromtherecordings'metadata,fromtheannouncerinthecaseofbroadcastmusicprogrammes,orbyearwhenlisteningtoamusicalpiece.Thechoiceofinstrumentationwasanimportantaspectinthestudyofhistoricallyinformedperformancesofearlymusic(e.g.choiceofepochinstru-mentsratherthanmodernones),especiallysinceearlymu-sicscoresdonotindicateinstruments.Therecognitionofmodernversusepochinstrumentsinmusicalperformanceswasnotatrivialprocesstoperformaurally(“I'massumingthesearemoderninstruments,440etc.”).Similarly,iso-latingaspecicinstrumentamongstanensemble(e.g.theviolininastringensemble:“`Quanjevoyleduc'-mostat-tractiveinstrumentalpieceofcollectionbuthorridscratchystringplaying,ddle?”),orretrievingthenumberofmusi-ciansplayingapartinaspecicregister(“Twosopranos?”)werenoteasytasks.Thetuningoftheinstrumentswasalsousedtojudgemusicalinterpretations(e.g.“LamentodellaNymfaparticularlytellingwithtoomanyharpsichordsIfeel-eachoneslightlyoutoftune.”,“Tuningofviolinsnotgreatinsecondtrack”).5.2MusicalexpressionVariousmusicalfeaturescorrelatedtomusicalexpressionwererecurrentlyanalysed,including:dynamics(e.g.“Dellerusinggreatsweepingphraseswithmanydynamicnuances.”),timing(e.g.tappingthebeatwhilelisteningtoincreasethesensationofthetempo,measuringthedurationofaperfor-mance,detailedanalysesofpianists'handasynchroniesus-ingspectrograms),timbre(e.g.“Whenshesingssofter,shedoesn'thavethesamequality.Thenotessoundmellower.”),pitch(e.g.“TheKing'sSingers'stylehasn'tchangedmuchbutaltosoundisat!”),vocalstyle(e.g.“Overabundanceofrolled`R's-stylisticallyok,butabitobtrusiveinanoth-erwisebeautifulrendition.”),vibrato(e.g.“Dellerconsortstillhasalotofvibratointenor(s)butverygoodensem-blesinging.”),andphrasing(e.g.“Reminiscentofbaroquephrasingratherthanrenaissance.”).Musicalexpressionwasanalysedeitherbyconsideringaspecicperformer(e.g.thesingerDeller),orbyconsid-eringanensemble(e.g.theKing'sSingers,thetenors),byfocusingonthenotes,oronphrases,thelattershowingtheuseofdifferenttimescalesintheanalyses.Somefeaturesweremoredifculttodescribesolelybasedonauralobser-vationsthanothers.Ifdynamicsvariationsseemedtobeeasilyperceived,somevariationsofpitchandtimbreweremoredifculttodetectcondently(e.g.“Notsureitstaysintunetoowell,sinkingoverthewholeperf,lessthanasemi.”,“Maybebecauseofthechoiceofqualityfornotesofthesamepitchontwodifferentpieces,thevoicedoesn'tsoundthesame:itdoesn'tsoundasshrillasitdidontheG.MaybeduetothekeyD[.”).Oftentheexpertiseofthemusi-cologistsasperformerswasemployedtondcausalexpla-nationsofsoundeffectsbasedoninstrumentaltechniques:singerswereabletoassociatevocaltimbrevariationswiththevocaltechniqueusedtoproducethem(“Dellerseemstousechestvoiceforthesecond,lower,`Zion'.”),pianistswereabletodetecttimingeffectsbetweennotesbyfocusingonthehandtechnique(handasynchronyinchords)charac-teristicofthestyleoftheperformer.Musicalexpressionwasalsodescribedinacriticalwaybyusingaestheticjudgments 12thInternationalSocietyforMusicInformationRetrievalConference(ISMIR2011) 357 (“Sopranosoundisratherlovelyitmustbesaid”,“`des-olata'isquiteseasick”,“Ilovethebottomofhervoice”,“Beautiful-veryclearrendition”,“Veryrousingandexhil-aratingrenditionbyWebsterBooth”).6.DISCUSSION6.1VisualisationandcomputationalanalysisenrichtheempiricalevidenceEventhough,aseducatedandexpertlisteners,musicolo-gistswereabletoperceiveextremelynedetails,visuali-sationandcomputationalanalysisconveyedempiricalev-idencewhichhelpedthemtoconrmandproveauralob-servations(“Thetoolsononehand,Idon'tneedthem,Icoulddescribethat,ontheotherhandIcan'tproveit.Thistool[SonicVisualiser]isallowingmetoexpressthatinsomewayit[thending]isobjective.”).AsputforwardbyCook[7],computationalmethodsbringthepotentialformu-sicologytobepursuedasamoredata-richdiscipline.TheobservationsreportedinSubsection4.2.3showtheutilityofmultiplesourcesofinformationtoanalysemusicperfor-mancepractices.Visualisationsandquantitativedatare-trievedthroughsignalmeasurementswerehelpfulindis-cussing,interpreting,orprovinghypothesesaboutqualita-tivedatacollectedthroughauralobservations.Furthermore,theseanalysesenabledsystematiccomparisonofthemu-sicalexpressionofvariousperformersindifferentmusicalpieces(e.g.measurementoftherateandextentofthevi-bratoonlongsustainednotesbasedonspectrogramanaly-ses)andledtoexplanationsofexpressivetechniqueswhichcouldnotbereachedthroughauralobservationsalone(“Youcanonlyhearthepitchaspectofthevibratoasaneducatedlistenerwithnosoftwareortechnology.”).6.2Cross-modaleffectsexistbetweenauditoryandvisualfeedbackManyoftheexamplesgiveninSubsection4.2.3alsoshowthatthevisualisationandlisteningprocesses(eitherattheoriginalspeedorusingslowedplayback)affecteachother.Forexample,thespectrogramhelpstohearvibratomuchbetter,theslowedplaybackofatonehelpstouncoverthatthevibratoisnotconstant,whilethespectrogramaidsinunderstandingthatthevariationcomesfromthefactthatthenotestartswithoutvibrato.Hence,newempiricalevidenceemergesfromthecross-modaleffectsbetweenauditoryandvisualfeedback.VisualisationwasdescribedbyoneoftheFellowsasa“learningprocess”(“NowI'veseenthespec-trogram,Icanonlyhearit[thevibrato],it'stherenow...inmyunderstanding.”).However,cross-modaleffectsbe-tweenauditoryandvisualfeedbackalsoraiseaparadox:ifvisualisationbringstotheauralexperiencean“increasedemphasisonwhatyoucansee”,itconcomitantly“deem-phasiseswhatyoucan'tsee”.Thereforetheearmaydis-cardrelevantaspectswhentheeyefocusesonaspectrogramrepresentationwhilelistening.Afterperforminganalysesbasedonspectrograms,oneFellownoted“Icompletelyfor-gotaboutthebassoon,itfeelslikeitisunimportantnow,butIwasoncestruckbyit.”.Forthisreason,beingabletolistentoamusicalpieceatrstwithoutvisualswasdeemedtobeimportant,otherwisevisualisationmay“irreversiblyeditstuffoutofyourbrainthatyoucan'tsee”.Thedesign-ersandusersofmusicfeaturevisualisationsoftwareneedtobeawareofcross-modalinteractionswhichmightaffecttheobjectivityoftheirobservations[8].6.3SoftwareformusicologistsshouldsupportclosedandmultimodallisteningpracticesWesuggestthatsoftwaredesignedforassistingmusicolo-gistsintheiranalysesofrecordingsshouldbeinlinewiththeirlisteningpracticesbysupportingbothclosedandmul-timodallistening.Duetothecross-modaleffectsmentionedintheprevioussection,itwouldbehelpfulfortheuserin-terfacersttoprovideaclosedlisteningmodewithoutvisu-als,andthenofferthepossibilityofswitchingtoamoread-vancedlisteningmodeofferingmultimodalfeedback.Themultimodalmodeshouldlinkthemusicdocumentsandrep-resentationsusingaural,visual,textual,andsymbolicin-formation(seeSubsection4.2).Differentsoftwareoruserinterfacesmaybeneededtohandleprimary(e.g.scoresandsoundvisualisations)andsecondary(e.g.musicbiographies)informationsources.Onewayofprovidingtextualandvisualinformationre-latedtoarecording(e.g.metadata,pictures)isviasemanticwebtechnologies.Linkeddataofferpromisingwaystofa-cilitatetheretrievalofmetadatadescribingtherecordings(date,albumartcovers,etc.)andthemusicians(biogra-phies,photos,etc.).Inadditiontovisualisationsofacousticparameters(seeSubsections4.2.3and6.1),thevisualisationofscoresand/orlyricswithinthesoftwarewouldfacilitatetheanalysisofmusicrecordings.Semanticwebtechnolo-giesmayalsoprovidewaystoretrievescoresfromonlinedatabasesdirectlyfromtheaudioplayer.Scorescouldthenbeusedasareferencetocomputetheperformers'expres-sivedeviationsusingcontent-basedMIRtechniques.Thevisualisationofexpressivedeviationscouldhelpmusicolo-giststodeterminetheextenttowhichexpressivemarkingsinthescorearefollowedintheperformance(seethenotementionedearlier:“Seemsreallyconsistentwithmarkingsinthescore.”),andtocharacterisetheartisticintentionsoftheconductorand/orperformers.BasedontheobservationsreportedinSubsection4.2,thealignmentofscores,lyricsorothertime-basedmeta-datatoaudiorecordingscouldalsoaidperformanceprac-ticeanalysis,byfacilitatingmultimodallisteningandpro-vidingbetternavigationofaudiodocuments.Forannota-tionofrecordings,theinclusionoftexteditingfunction-alityintoanalysisandplaybacksoftwarewouldbeawel- OralSession5:UserStudies 358 comefeature,sincemusicologistsgenerallywritedownob-servationswhilelistening.Thiscouldindeedbeameanstoconnectnotesupwiththeactualpoint-in-timeofthemu-sicwhichwouldeasefurtherproof-readingorenrichmentofthenotes.Controllingtheaudioplayback,witheitherthekeyboardorwithatranscriptionfootpedal,wouldfacilitatetaskssuchasthetranscriptionofinterviewsfrombroadcastrecordingsincludingspeechandmusic,andavoidthecon-stantswitchesbetweenvariouscomputersoftwareordiffer-entdeviceswhicharetime-consuming(“It'ssoirritatingtranscribingfromacomputerlebecauseyou'realsotry-ingtowriteonthesamecomputer,soyouhavetokeepgo-ingintothatprogramtomovetherecordingbackabit,gobacktothewordprogramtotypeupthatsentencemoreac-curately.So[...]ifit'smyleonmyiPod,Icanstartandstopusingadifferentdevicethanthecomputer,orhereI'musingtheCDplayer.”).6.4Cancontent-basedMIRaidmusicologicalstudy?Severalareasofcontent-basedMIRarerelevantformusico-logicalpurposes.Forinstance,automaticspeech/musicseg-mentationwouldhelpthenavigationbetweenspokenandmusicpartsofdocumentariesandotherbroadcastmaterial.Speechrecognitionsoftwarewouldalsobeofconsiderablehelptoautomaticallytranscribeinterviews,enablingsearchofthenon-musicaudiosegmentsforconversationsaboutspecictopicsormusicians.Regardingtheanalysisofper-formancepractices,automaticsourceseparationtechniquescouldfacilitateseparateanalysisofthemusicalexpressionofdifferentperformersorgroupsofperformers(seeSec-tion5.2).Variationsoftimbrearemoredifculttoqualifyaurallythanothervariationssuchasintiming.ThereforeMIRtechniquesimprovingtimbrecharacterisation(e.g.atthenotelevel)andidenticationofinstrumentationorper-formerscouldhelpanswerquestionslike:“IsthatJanitausingsomevibratointhesolos?”7.CONCLUSIONInthispaper,wepresentedandanalysedethnographicob-servationsofmusicologistsstudyingclassicalmusicrecord-ings.Theobservedpatternsrevealedtheimportanceof:(i)thealternationofclosedandmultimodallisteningmodes;(ii)theuseofvisualisationandcomputationalmethodstoprovideempiricalevidenceaboutlisteners'impressions;(iii)scoresandlyricsactingasareferenceinperformanceanaly-sis;and(iv)websitesandspeechrecordingssupplyinghis-toricalandextra-musicalinformation.Thesendingsgivecluesregardinghowtoimprovesoft-waredesignedformusicologists.Suchsoftwareshouldbothsupportclosedandmultimodallistening,minimisingdis-tractionsandallowingtheusertodecideonthedisplayofanyfeaturevisualisationsduringlistening.Thefeaturesofinterestforcomputer-assistedmusicologyarethosecharac-terisingartisticchoicessuchasperformers'expressivein-tentions(e.g.tuning,temperament,timing,pitch,timbre,dynamics,articulationandvibrato),mostusefullydisplayedinconjunctionwithscoresandlyrics.Content-basedmeta-datasonicationshouldbehandledtofacilitatetheinterpre-tationofthefeatures(e.g.pitch).Interfacesmanagingtheretrievalofcontextualinformation(e.g.metadata,biogra-phies,articles,pictures)duringmultimodallisteningwouldbenetthehistoricalapproachtomusicology.Linkeddataoffersapromisingwaytoconnectsuchextra-musicalin-formationwiththerecordingsbyexploitingwebresourcessuchastheopenmusicencyclopediaMusicBrainz3.8.ACKNOWLEDGMENTSTheauthorswishtothanktheEdisonFellowsandtheBritishLibraryfortheirkindparticipationandhelpduringthisstudy.ThisstudywasconductedaspartoftheRCUKDigitalEcon-omyprojectEP/I001832/1,MusicologyfortheMasses4.9.REFERENCES[1]J.Futrelle,andJ.StephenDownie:“InterdisciplinaryCommunitiesandResearchIssuesinMusicInformationRetrieval”,ProceedingsoftheInternationalSymposiumonMusicInformationRetrieval,2001.[2]S.-J.Cunningham,N.Reeves,andM.Britland:“AnEthnographicStudyofMusicInformationSeeking:Im-plicationsfortheDesignofaMusicDigitalLibrary”,Proceedingsofthe2003JointConferenceonDigitalLi-braries(JCDL'03),2003.[3]A.Bonardi:“IRforContemporaryMusic:WhattheMusicologistsNeeds”,ProceedingsoftheInternationalSymposiumonMusicInformationRetrieval,2000.[4]G.Gobo:“DoingEthnography”,Sage,London,2008.[5]R.Collins:“TheoreticalSociology”,Harcourt,SanDiego,1988.[6]R.M.Emerson,R.I.Fretz,andL.L.Shaw:“WritingEthnographicFieldnotes”,TheUniversityofChicagoPress,Chicago,1995.[7]N.Cook:“ComputationalandComparativeMusi-cology”,inEmpiricalMusicology:Aims,Methods,Prospects,OxfordUniversityPress,NewYork,2004.[8]S.Dixon,W.GoeblandE.Cambouropoulos:“Percep-tualSmoothnessofTempoinExpressivelyPerformedMusic”,MusicPerception,Vol.23,No.3,pp.195–214,2006. 3http://musicbrainz.org/andhttp://linkedbrainz.c4dmpresents.org/content/linkedbrainz-summary4http://www.elec.qmul.ac.uk/digitalmusic/m4m/