MODELING RHYTHM SIMILARITY FOR ELECTRONIC DANCE MUSIC Maria Panteli University of Amsterdam - PDF document

assumptionscouldberelaxedtoanalyzeEDMwithternarydivisionsorno meter,orexpandedtoothermusicstyleswithsimilarstructure.ThecorrelationreportedinSection3.4iscomputedfromapreliminarysetofexperimentdata.Moreratingsarecur-rentlycollectedandaregressionanalysisandtuningofthemodelisconsideredinfuturework.5.CONCLUSIONAmodelofrhythmsimilarityforElectronicDanceMusichasbeenpresented.Themodelextractsrhythmicfeaturesfromaudiosegmentsandcomputessimilaritybycompar-ingtheirfeaturevectors.AmethodforrhythmicstreamdetectionisproposedthatestimatesthenumberandrangeoffrequencybandsfromthespectralrepresentationofeachsegmentratherthanaÞxeddivision.Featuresareextractedfromeachstream,anapproachshowntobeneÞttheanal-ysis.Similaritypredictionsofthemodelmatchperceptualratingswithacorrelationof0.7.FutureworkwillÞne-tunepredictionsbasedonaperceptualrhythmsimilaritymodel.6.REFERENCES[1]S.Bock,A.Arzt,K.Florian,andS.Markus.On-linereal-timeonsetdetectionwithrecurrentneuralnet-works.InInternationalConferenceonDigitalAudioEffects,2012.[2]M.J.Butler.UnlockingtheGroove.IndianaUniversityPress,BloomingtonandIndianapolis,2006.[3]E.Cambouropoulos.VoiceandStream:PerceptualandComputationalModelingofVoiceSeparation.Perception,26(1):75Ð94,2008.[4]D.Diakopoulos,O.Vallis,J.Hochenbaum,J.Murphy,andA.Kapur.21stCenturyElectronica:MIRTech-niquesforClassiÞcationandPerformance.In[5]S.Dixon,F.Gouyon,andG.Widmer.TowardsChar-acterisationofMusicviaRhythmicPatterns.In[6]A.EigenfeldtandP.Pasquier.EvolvingStructuresforElectronicDanceMusic.InGeneticandEvolutionaryComputationConference,2013.[7]J.FooteandS.Uchihashi.Thebeatspectrum:anewapproachtorhythmanalysis.In,2001.[8]J.T.Foote.Mediasegmentationusingself-similaritydecomposition.InElectronicImaging.InternationalSocietyforOpticsandPhotonics,2003.[9]D.Gartner.Tempoestimationofurbanmusicusingtatumgridnon-negativematrixfactorization.In[10]J.W.Gordon.TheperceptualattacktimeofmusicalTheJournaloftheAcousticalSocietyofAmer-,82(1):88Ð105,1987.[11]T.D.GrifÞthsandJ.D.Warren.WhatisanauditoryNatureReviewsNeuroscience,5(11):887Ð892,[12]C.Guastavino,F.Gomez,G.Toussaint,F.Maran-dola,andE.Gomez.MeasuringSimilaritybetweenFlamencoRhythmicPatterns.JournalofNewMusicResearch,38(2):129Ð138,June2009.[13]J.A.Hockman,M.E.P.Davies,andI.Fujinaga.OneintheJungle:DownbeatDetectioninHardcore,Jungle,andDrumandBass.In,2012.[14]A.Klapuri,A.J.Eronen,andJ.T.Astola.Analysisofthemeterofacousticmusicalsignals.IEEETrans-actionsonAudio,SpeechandLanguageProcessing14(1):342Ð355,January2006.[15]F.Krebs,S.Bock,andG.Widmer.Rhythmicpatternmodelingforbeatanddownbeattrackinginmusicalaudio.In,2013.[16]O.Lartillot,T.Eerola,P.Toiviainen,andJ.Fornari.Multi-featureModelingofPulseClarity:Design,Vali-dationandOptimization.In,2008.[17]O.LartillotandP.Toiviainen.AMatlabToolboxforMusicalFeatureExtractionFromAudio.IntionalConferenceonDigitalAudioEffects,2007.[18]H.C.Longuet-HigginsandC.S.Lee.TheRhyth-micInterpretationofMonophonicMusic.MusicPer-ception:AnInterdisciplinaryJournal,1(4):424Ð441,[19]A.Novello,M.M.F.McKinney,andA.Kohlrausch.PerceptualEvaluationofInter-songSimilarityinWest-ernPopularMusic.JournalofNewMusicResearch40(1):1Ð26,March2011.[20]J.PaulusandA.Klapuri.MeasuringtheSimilarityofRhythmicPatterns.In,2002.[21]B.Rocha,N.Bogaards,andA.Honingh.SegmentationandTimbreSimilarityinElectronicDanceMusic.InSoundandMusicComputingConference,2013.[22]E.D.Scheirer.Tempoandbeatanalysisofacousticmusicalsignals.TheJournaloftheAcousticalSocietyofAmerica,103(1):588Ð601,January1998.[23]M.R.Schroeder,B.S.Atal,andJ.L.Hall.Optimizingdigitalspeechcodersbyexploitingmaskingpropertiesofthehumanear.TheJournaloftheAcousticalSocietyofAmerica,pages1647Ð1652,1979.[24]L.M.Smith.Rhythmicsimilarityusingmetricalpro-Þlematching.InInternationalComputerMusicCon-ference,2010.[25]J.R.ZapataandE.Gomez.ComparativeEvaluationandCombinationofAudioTempoEstimationAp-proaches.InAudioEngineeringSocietyConference 15th International Society for Music Information Retrieval Conference (ISMIR 2014) 6 3.2OnsetDetectionEvaluationOnsetdetectionisevaluatedwithasetof25MIDIandcorrespondingaudioexcerpts,speciÞcallycreatedforthispurpose.Inthisapproach,onsetsaredetectedperstream,thereforeonsetannotationsshouldalsobeprovidedperstream.ForanumberofdifferentEDMrhythms,MIDIÞleswerecreatedwiththeconstraintthateachMIDIin-strumentperformsauniquerhythmicpatternthereforerep-resentsauniquestream,andwereconvertedtoaudio.TheonsetsestimatedfromtheaudiowerecomparedtotheannotationsoftheMIDIÞleusingtheevaluationmea-suresoftheMIREXOnsetDetectiontask.Forthis,nostreamalignmentisperformedbutratheronsetsfromallstreamsaregroupedtoasingleset.Forexcerpts,an-measureof,presicionof,andrecallofareobtainedwithatolerancewindowofms.Inaccura-ciesinonsetdetectionaredue(onaverage)todoubledthanmergedonsets,becauseusuallymorestreams(andhencemoreonsets)aredetected.3.3DownbeatDetectionEvaluationToevaluatethedownbeatthesubsetofsegmentsde-scribedinSection3.1wasused.Foreachsegmenttheannotateddownbeatwascomparedtotheestimatedonewithatolerancewindowofms.Anaccuracyofwasachieved.Downbeatdetectionwasalsoevaluatedatthebeat-level,i.e.,estimatingwhetherthedownbeatcor-respondstooneofthefourbeatsofthemeter(insteadofoff-beatpositions).Thisgaveanaccuracyof,mean-ingthatintheothercasesthedownbeatwasdetectedontheoff-beatpositions.ForsomeEDMtracksitwasobservedthathighdegreeofperiodicitycompensatesforawronglyestimateddownbeat.Theoverallresultsofthesimilaritypredictionsofthemodel(Section3.4)indicateonlyami-norincreasewhenthecorrect(annotated)downbeatsaretakenintoaccount.Itishenceconcludedthatthedown-beatdetectionalgorithmdoesnothavegreatinßuenceonthecurrentresultsofthemodel.3.4MappingModelPredictionstoPerceptualRatingsofSimilarityThemodelÕspredictionswereevaluatedwithperceptualratingsofrhythmsimilaritycollectedviaalisteningex-periment.PairwisecomparisonsofasmallsetofsegmentsrepresentingvariousrhythmicpatternsofEDMwerepre-sented.Subjectswereaskedtoratetheperceivedrhythmsimilarity,choosingfromafourpointscale,andreportalsotheconÞdenceoftheirrating.Fromapreliminarycollec-tionofexperimentdata,pairs(representingatotalofuniquemusicsegments)wereselectedforfurtheranalysis.Thesewereratedfromatotalofparticipants,withmeanyearsoldandstandarddeviation.Thetheparticipantsreceivedformalmusicaltraining,wasfamiliarwithEDMandhadexperienceasEDMmu-sician/producer.Theselectedpairswereratedbetweentimes,withallparticipantsreportingconÞdenceintheir www.MIREX.org rpfeatures -0.170.22attackcharacterization0.480.000.330.01metricaldistributionexcl.metricalproÞle0.690.00metricaldistributionincl.metricalproÞle0.700.00 Table1:PearsonÕscorrelation-valuesbetweenthemodelÕspredictionsandperceptualratingsofrhythmsim-ilarityfordifferentsetsoffeatures.rating,andallratingsbeingconsistent,i.e.,ratedsimilaritywasnotdeviatingmorethanpointscale.Themeanoftheratingswasutilizedasthegroundtruthratingperpair.Foreachpair,similaritycanbecalculatedviaapplyingadistancemetrictothefeaturevectorsoftheunderlyingsegments.Inthispreliminaryanalysis,thecosinedistancewasconsidered.PearsonÕscorrelationwasusedtocomparetheannotatedandpredictedratingsofsimilarity.ThiswasappliedfordifferentsetsoffeaturesasindicatedinTable1.Amaximumcorrelationofwasachievedwhenallfeatureswerepresented.Thenon-zerocorrelationhypoth-esiswasnotrejected()fortheattackcharacter-izationfeaturesindicatingnon-signiÞcantcorrelationwiththe(currentsetof)perceptualratings.Theperiodicityfea-turesarecorrelatedwith,showingastronglinkwithperceptualrhythmsimilarity.Themetricaldistribu-tionfeaturesindicateacorrelationincreaseofthemetricalproÞleisincludedinthefeaturevector.ThisisinagreementwiththeÞndingof[24].Asanalternativeevaluationmeasure,themodelÕspre-dictionsandperceptualratingsweretransformedtoabi-naryscale(i.e.,beingdissimilarandbeingsimilar)andtheiroutputwascompared.ThemodelÕspredictionsmatchedtheperceptualratingswithanaccuracyofHencethemodelmatchestheperceptualsimilarityratingsatnotonlyrelative(i.e.,PearsonÕscorrelation)butalsoab-soluteway,whenabinaryscalesimilarityisconsidered.4.DISCUSSIONANDFUTUREWORKIntheevaluationofthemodel,thefollowingconsidera-tionsaremade.HighcorrelationofwasachievedwhenthemetricalproÞle,outputperstream,wasaddedtothefeaturevector.Analternativeexperimenttestedthecor-relationwhenconsideringthemetricalproÞleasawhole,i.e.,asasumacrossallstreams.Thisgaveacorrelationofindicatingtheimportanceofstreamseparationandhencetheadvantageofthemodeltoaccountforthis.Amaximumcorrelationofwasreported,takingintoaccountthedownbeatdetectionbeingofthecasescorrect.AlthoughregularityinEDMsometimescompen-satesforthis,modelÕspredictionscanbeimprovedwithamorerobustdownbeatdetection.Featuresofperiodicity(Section2.2.2)andmetricaldis-tribution(Section2.2.3)wereextractedassuminga ter,and-thnoteresolutionthroughoutthesegment.ThisisgenerallytrueforEDM,butexceptionsdoexist[2].The 15th International Society for Music Information Retrieval Conference (ISMIR 2014) 5 0 0.5 1 1.5 2 2.5 !0.6 !0.4 !0.2 0 0.2 0.4 0.6 0.8 1 1.2 Lag (s)Normalized amplitude 1 Beat1 Bar Figure4:Autocorrelationofonsetsindicatinghighperi-odicitiesof1barand1beatduration. !!! 0 10 20 30 time (s)Stream 1 0 5 10 15 Stream 2 0 10 20 30 Onset curve (Envelope) (half!wave rectified)Stream 3 Bar 2Bar 3Bar 4Bar 1 Downbeat ! "#$!%! "#$!& ! !!!!!!!!!! !!!!!!!!!! '($)#*!% +!+!+!%!+!+!+!%!+!+!+!%!+!+ +!+!+!%!+!+!+!%!+!+!+!%!+!+ '($)#*! +!+!+!+!%!+!+!+!+!+!+!+!%!+!+ +!+!+!+!%!+!+!+!+!+!+!+!%!+!+ '($)#*!, +!+!%!+!+!+!%!+!+!+!%!+!++!%!+! +!+!%!+!+!+!%!+!+!+!%!+!++!%!+ '($)#*!- %!%!%!%!%!%!%!%!%!%!%!%!%!%!%!% %!%!%!%!%!%!%!%!%!%!%!%!%!%!%!% ! "#$!%! "#$!& "#$!,! "#$!- ! !!!!!!!!!! !!!!!!!!!! !!!!!!!!!! !!!!!!!!!! '($)#*!, %!+!%!+!%!+!%!+!%!+!%!+!%!+!%!+! %!+!%!+!%!+!%!+!%!+!%!+!%!+!%!+! %!+!%!+!%!+!%!+!%!+!%!+!%!+!%!+! %!+!%!+!%!+!%!+!%!+!%!+!%!+!%!+! '($)#*!& +!+!+!+%!+!+!+!+!+!+!+!%!+!+!+! +!+!+!+!%!+!+!+!+!+!+!+!%!+!+!+! +!+!+!+!%!+!+!+!+!+!+!+!%!+!+!+! +!+!+!+!%!+!+!+!+!+!+!+!%!+!+!+! '($)#*!% %!+!+!+!+!+!+!+!%!+!+!+!+!+!+!+! %!+!+!+!+!+!+!+!%!+!+!+!+!+!+!+! %!+!+!+!+!+!+!+!%!+!+!+!+!+!+!+! %!+!+!+!+!+!+!+!%!+!+!+!+!+!+!+! Figure5:MetricalproÞleoftherhythminFigure1assum-ingforsimplicitya-barlengthandconstantamplitude.Þleiscomputedandthefollowingfeaturesareextracted.FeaturesarecomputedperstreamandaveragedacrossallMeasuresthestrengthoftheeventslyingontheweaklocationsofthemeter.Thesyncopationmodelof[18]isusedwithadaptationtoaccountfortheamplitude(onsetstrength)ofthesyncopatednote.Threemeasuresofsyncopationareconsideredthatapplyhierarchicalweightswith,respectively,sixteenthnote,eighthnote,andquarternoteresolution.DenotestheratioofthenumberofonsetsinthesecondhalfofthepatternthatappearinexactlythesamepositionintheÞrsthalfofthepattern[6].Istheratioofthenumberofonsetsoverthepossibletotalnumberofonsetsofthepattern(inthiscaseMeasurestheonsetsÕstrengthofthepattern.ItdescribestheratioofthesumofonsetsÕstrengthoverthemaximumstrengthmultipliedbythepossibletotalnumberofonsets(inthiscaseCentreofGravity:Denotesthepositioninthepatternwherethemostandstrongestonsetsoccur(i.e.,indicateswhethermostonsetsappearatthebeginningorattheendofthepatternetc.).Asidefromthesefeatures,themetricalproÞle(cf.Fig-ure5)isalsoaddedtotheÞnalfeaturevector.Thiswasfoundtoimproveresultsin[24].Inthecurrentapproach,themetricalproÞleisprovidedperstream,restrictedtoatotalofstreams,andoutputintheÞnalfeaturevectorinorderoflowtohighfrequencycontentstreams.2.2.4DownbeatDetectionThedownbeatdetectionalgorithmusesinformationfromthemetricalstructureandmusicalheuristics.Twoassump-tionsaremade:Assumption1:Strongbeatsofthemeteraremorelikelytobeemphasizedacrossallrhythmicstreams.Assumption2:Thedownbeatisoftenintroducedbyaninstrumentinthelowfrequencies,i.e.abassorakickdrum[2,13].Consideringtheabove,theonsetsperstreamarequan-tizedassuminga meter,-thnoteresolution,andasetofdownbeatcandidates(inthiscasetheonsetsthatliewithinonebarlengthcountingfromthebeginningoftheseg-ment).Foreachdownbeatcandidate,hierarchicalweights[18]thatemphasizethestrongbeatsofthemeterasindi-catedbyAssumption1,areappliedtothequantizedpat-terns.Note,thereisonepatternforeachrhythmicstream.Thepatternsarethensummedbyapplyingmoreweighttothepatternofthelow-frequencystreamasindicatedbyAs-sumption2.Finally,thecandidatewhosequantizedpatternwasweightedmost,ischosenasthedownbeat.3.EVALUATIONOneofthegreatestchallengesofmusicsimilarityevalu-ationisthedeÞnitionofagroundtruth.Insomecases,objectiveevaluationispossible,whereagroundtruthisde-ÞnedonaquantiÞablecriterion,i.e.,rhythmsfromapartic-ulargenrearesimilar[5].Inothercases,musicsimilarityisconsideredtobeinßuencedbytheperceptionofthelis-tenerandhencesubjectiveevaluationismoresuitable[19].Objectiveevaluationinthecurrentstudyisnotpreferablesincedifferentrhythmsdonotnecessarilyconformtodif-ferentgenresorsubgenres.Thereforeasubjectiveeval-uationisusedwherepredictionsofrhythmsimilarityarecomparedtoperceptualratingscollectedviaalisteningex-periment(cf.Section3.4).Detailsoftheevaluationofrhythmicstream,onset,anddownbeatdetectionarepro-videdinSections3.1-3.3.Asubsetoftheannotationsusedintheevaluationofthelatterisavailableonline3.1RhythmicStreamsEvaluationThenumberofstreamsisevaluatedwithperceptualanno-tations.Forthis,asubsetofsongsfromatotalofartists(songsperartist)fromavarietyofEDMgenresandsubgenreswasselected.Foreachsong,segmentationwasappliedusingthealgorithmof[21]andacharacteristicsegmentwasselected.Foursubjectswereaskedtoevalu-atethenumberofrhythmicstreamstheyperceiveineachsegment,choosingbetween,whererhythmicstreamwasdeÞnedasastreamofuniquerhythm.Forofthesegments,thesubjectsÕresponsesÕstandarddeviationwassigniÞcantlysmall.Theestimatednumberofrhythmicstreamsmatchedthemeanofthesub-jectÕsresponsedistributionwithanaccuracyof AlthoughsomerhythmicpatternsarecharacteristictoanEDMgenreorsubgenre,itisnotgenerallytruethattheseareuniqueandinvariant. 15th International Society for Music Information Retrieval Conference (ISMIR 2014) 4 ! 16 18 20 22 24 26 28 5 10 15 20 Bark!Spectrumtime axis (in s.)Bark bands "#$% &'()!*'+,- (a)Bark-bandspectrogram. ! 5 10 15 20 20 15 10 5 "#$%!&#'() "#$%!&#'() (b)Self-similaritymatrix. ! 2 4 6 8 10 12 14 16 18 20 22 24 10 15 20 Novelty ValueBark BandsNovelty Curve "#$%&'!( "#$%&'!) "#$%&'!* "#$%&'!+ ,-.%/#0!.&/1% 2&$3!4&567 (c)Noveltycurve.Figure3:Detectionofrhyhmicstreamsusingthenoveltyapproach;Þrstabark-bandspectrogramiscomputed,thenitsself-similaritymatrix,andthenthenovelty[7]isappliedwherethenoveltypeaksdeÞnethestreamboundaries.2.2FeatureExtractionTheonsetsineachstreamrepresenttherhythmicelementsofthesignal.Tomodeltheunderlyingrhythm,featuresareextractedfromeachstream,basedonthreeattributes,namely,characterizationofattack,periodicity,andmetri-caldistributionofonsets.Thesearecombinedtoafeaturevectorthatservesformeasuringinter-segmentsimilarity.Thesectionsbelowdescribethefeatureextractionprocessindetail.2.2.1AttackCharacterizationTodistinguishbetweenpercussiveandnon-percussivepat-terns,featuresareextractedthatcharacterizetheattackpha-seoftheonsets.Inparticular,theattacktimeandattackslopeareconsidered,amongother,essentialinmodelingtheperceivedattacktime[10].Theattackslopewasalsousedinmodelingpulseclarity[16].Ingeneral,onsetsfrompercussivesoundshaveashortattacktimeandsteepattackslope,whereasnon-percussivesoundshavelongerattacktimeandgraduallyincreasingattackslope.Forallonsetsinallstreams,theattacktimeandat-tackslopeisextractedandsplitintwoclusters;theÔslowÕ(non-percussive)andÔfastÕ(percussive)attackphaseon-sets.Here,itisassumedthatbothpercussiveandnon-percussiveonsetscanbepresentinagivensegment,hencesplittingintwoclustersissuperiorto,e.g.,computingtheaverage.Themeanandstandarddeviationofthetwoclus-tersoftheattacktimeandattackslope(atotalofisoutputtothefeaturevector.2.2.2PeriodicityOneofthemostcharacteristicstyleelementsinthemusicalstructureofEDMisrepetition;theloop,andconsequentlytherhythmicsequence(s),arerepeatingpatterns.Toana-lyzethis,theperiodicityoftheonsetdetectionfunctionperstreamiscomputedviaautocorrelationandsummedacrossallstreams.Themaximumdelaytakenintoaccountispro-portionaltothebarduration.Thisiscalculatedassumingasteadytempoand meterthroughouttheEDMtrack[2].Thetempoestimationalgorithmof[21]isused.Fromtheautocorrelationcurve(cf.Figure4),atotaloffeaturesareextracted:Lagdurationofmaximumautocorrelation:Thelo-cation(intime)ofthesecondhighestpeak(theÞrstbeingatlag0)oftheautocorrelationcurvenormalizedbythebarduration.Itmeasureswhetherthestrongestperiodicityoc-cursineverybar(i.e.featurevalue=1),oreveryhalfbar(i.e.featurevalue=0.5)etc.Amplitudeofmaximumautocorrelation:Theam-plitudeofthesecondhighestpeakoftheautocorrelationcurvenormalizedbytheamplitudeofthepeakatlag0.Itmeasureswhetherthepatternisrepeatedinexactlythesameway(i.e.featurevalue=1)orsomewhatinasimilarway(i.e.featurevalue)etc.Harmonicityofpeaks:Thisistheharmonicityasde-Þnedin[16]withadaptationtothereferencelagcor-respondingtothebeatdurationandadditionalweightingoftheharmonicityvaluebythetotalnumberofpeaksoftheautocorrelationcurve.Thisfeaturemeasureswhetherrhythmicperiodicitiesoccurinharmonicrelationtothebeat(i.e.featurevalue=1)orinharmonic(i.e.featurevalue=0).Measureswhethertheautocorrelationcurveissmoothorspikyandissuitablefordistinguishingbe-tweenperiodicpatterns(i.e.featurevalue=0),andnon-periodic(i.e.featurevalue=1).Entropy:AnothermeasureoftheÔpeakinessÕofauto-correlation[16],suitablefordistinguishingbetweenÔclearÕrepetitions(i.e.distributionwithnarrowpeaksandhencefeaturevaluecloseto0)andunclearrepetitions(i.e.widepeaksandhencefeaturevalueincreased).2.2.3MetricalDistributionTomodelthemetricalaspectsoftherhythmicpattern,themetricalproÞle[24]isextracted.Forthis,thedownbeatisdetectedasdescribedinSection2.2.4,onsetsperstreamarequantizedassuminga meterand-thnoteresolu-tion[2],andthepatterniscollapsedtoatotalofbars.ThelatterisinagreementwiththelengthofamusicalphraseinEDMbeingusuallyinmultiplesof,i.e.,4-bar,8-bar,or16-barphrase[2].ThemetricalproÞleofagivenstreamisthuspresentedasavectorofbins(sixteenthnotesperbeat)withrealvaluesrangingbe-tween0(noonset)to1(maximumonsetstrength)asshowninFigure5.Foreachrhythmicstream,ametricalpro- 15th International Society for Music Information Retrieval Conference (ISMIR 2014) 3 feature extraction stream # 2 segmentation feature vector similarity metrical distribution rhythmic streamsdetection periodicity attackcharacterization onset detectionfeature extraction stream # 1stream # 3 Figure2:Overviewofmethodology.percussiveornon-percussiveinstruments.Althoughthisistypicallyviewedasatimbreattribute,thepercussive-nessofasoundisexpectedtoinßuencetheperceptionofrhythm[16].b)TherepetitionofrhythmicsequencesofthepatternaredescribedbyevaluatingcharacteristicsofdifferentlevelsofonsetsÕperiodicity.c)Themetricalstructureofthepatternischaracterizedviafeaturesex-tractedfromthemetricalproÞle[24]ofonsets.Basedontheabove,afeaturevectorisextractedforeachsegmentandisusedtomeasurerhythmsimilarity.Inter-segmentsimilarityisevaluatedwithperceptualratingscollectedviaaspeciÞcallydesignedexperiment.AnoverviewofthemethodologyisshowninFigure2anddetailsforeachstepareprovidedinthesectionsbelow.PartofthealgorithmisimplementedusingtheMIRToolbox[17].2.1RhythmicStreamsSeveralinstrumentscontributetotherhythmicpatternofanEDMtrack.Mosttypicalexamplesincludecombina-tionsofbassdrum,snareandhi-hat(eg.Figure1).Thisismainlyafunctionalratherthanastrictlyinstrumentaldi-vision,andinEDMoneÞndsvariousinstrumentsoundstotaketheroleofbass,snareandhi-hat.Indescribingrhythm,itisessentialtodistinguishbetweenthesesourcessinceeachcontributesdifferentlytorhythmperception[11].Followingthis,[15,24]describerhythmicpatternsoflatindancemusicintwopreÞxedfrequencybands(lowandhighfrequencies),and[9]representsdrumpatternsastwocomponents,thebassandsnaredrumpattern,calculatedvianon-negativematrixfactorizationofthespectrogram.In[20],rhythmiceventsaresplitbasedontheirperceivedloudnessandbrightness,wherethelatterisdeÞnedasafunctionofthespectralcentroid.Inthecurrentstudy,rhythmicstreamsareextractedwithrespecttothefrequencydomainandloudnesspattern.Inparticular,theShortTimeFourierTransformofthesig-naliscomputedandlogarithmicmagnitudespectraareas-signedtobarkbands,resultingintoatotalof24bandsforkHzsamplingrate.Synchronousmaskingismod-eledusingthespreadingfunctionof[23],andtemporalmaskingismodeledwithasmoothingwindowofThisrepresentationishereafterreferredtoasloudnessen-velopeanddenotedbyforbarkbands,...,self-similaritymatrixiscomputedfromthis24-bandrep-resentationindicatingthebandsthatexhibitsimilarloud-nesspattern.Thenoveltyapproachof[8]isappliedtosimilaritymatrixtodetectadjacentbandsthatshouldbegroupedtothesamerhythmicstream.ThepeakofthenoveltycurvedeÞnethenumberofthebarkbandthatmarksthebeginningofanewstream,i.e.,if,...,,...,Ifortotalnumberof,thenstreamconsistsofbarkbandsgivenby,,pi,pi+1#1]}fori=1,...,IIpI,24]}fori=I.(1)Anupperlimitofstreamsisconsideredbasedontheap-proachof[22]thatusesatotalofbandsforonsetdetec-tionand[14]thatsuggestsatotalofthreeorfourbandsformeteranalysis.Thenotionofrhythmicstreamhereissimilartotheno-tionofÔaccentbandÕin[14]withthedifferencethateachrhythmicstreamisformedonavariablenumberofadja-centbarkbands.Detectingarhythmicstreamdoesnotnecessarilyimplyseparatingtheinstruments,sinceiftwoinstrumentsplaythesamerhythmtheyshouldbegroupedtothesamerhythmicstream.Theproposedapproachdoesnotdistinguishinstrumentsthatlieinthesamebarkband.Theadvantageisthatthenumberofstreamsandthefre-quencyrangeforeachstreamdonotneedtobepredeter-minedbutareratherestimatedfromthespectralrepresen-tationofeachsong.ThisbeneÞtstheanalysisofelectronicdancemusicbynotimposinganyconstraintsonthepossi-bleinstrumentsoundsthatcontributetothecharacteristicrhythmicpattern.2.1.1OnsetDetectionToextractonsetcandidates,theloudnessenvelopeperbarkbandanditsderivativearenormalizedandsummedwithmoreweightonloudnessthanitsderivative,i.e.,)=(1isthenormalizedloudnessenvelopenormalizedderivativeof,...,Ntheframenum-berforatotalofframes,andtheweightingfac-tor.ThisissimilartotheapproachdescribedbyEquationin[14]withreduced,andiscomputedpriorsummationtothedifferentstreamsassuggestedin[14,22].Onsetsaredetectedviapeakextractionwithineachstream,wherethe(rhythmic)contentofstreamisdeÞnedasasinEquation1andasinEquation2.Thisonsetdetectionapproachincorporatessimilarmethodolog-icalconceptswiththepositivelyevaluatedalgorithmsforthetaskofaudioonsetdetection[1]inMIREX2012,andtempoestimation[14]inthereviewof[25]. 15th International Society for Music Information Retrieval Conference (ISMIR 2014) 2 MODELINGRHYTHMSIMILARITYFORELECTRONICDANCEMUSICMariaPanteliUniversityofAmsterdam,Amsterdam,NetherlandsNielsBogaardsElephantcandy,Amsterdam,NetherlandsAlineHoninghUniversityofAmsterdam,Amsterdam,NetherlandsABSTRACTAmodelforrhythmsimilarityinelectronicdancemusic(EDM)ispresentedinthispaper.RhythminEDMisbuiltontheconceptofaÔloopÕ,arepeatingsequencetypicallyassociatedwithafour-measurepercussivepattern.Thepresentedmodelcalculatesrhythmsimilaritybetweenseg-mentsofEDMinthefollowingsteps.1)Eachsegmentissplitindifferentperceptualrhythmicstreams.2)Eachstreamischaracterizedbyanumberofattributes,mostno-tably:attackphaseofonsets,periodicityofrhythmicel-ements,andmetricaldistribution.3)Theseattributesarecombinedintoonefeaturevectorforeverysegment,af-terwhichthesimilaritybetweensegmentscanbecalcu-lated.Thestagesofstreamsplitting,onsetdetectionanddownbeatdetectionhavebeenevaluatedindividually,andalisteningexperimentwasconductedtoevaluatetheover-allperformanceofthemodelwithperceptualratingsofrhythmsimilarity.1.INTRODUCTIONMusicsimilarityhasattractedresearchfrommultidisci-plinarydomainsincludingtasksofmusicinformationre-trievalandmusicperceptionandcognition.Especiallyforrhythm,studiesexistonidentifyingandquantifyingrhythmproperties[16,18],aswellasestablishingrhythmsimilar-itymetrics[12].Inthispaper,rhythmsimilarityisstudiedwithafocusonElectronicDanceMusic(EDM),agenrewithvariousanddistinctrhythms[2].EDMisanumbrellatermconsistingoftheÔfourontheßoorÕgenressuchastechno,house,trance,andtheÔbreakbeat-drivenÕgenressuchasjungle,drumÔnÕbass,breaksetc.Ingeneral,fourontheßoorgenresarecharac-terizedbyafour-beatsteadybass-drumpatternwhereasbreakbeat-drivenexploitirregularitybyemphasizingthemetricallyweaklocations[2].However,rhythminEDMexhibitsmultipletypesofsubtlevariationsandembellish-ments.Thegoalofthepresentstudyistodeveloparhythmsimilaritymodelthatcapturestheseembellishmentsandal-lowsforaÞneinter-songrhythmsimilarity. MariaPanteli,NielsBogaards,AlineHoningh.LicensedunderaCreativeCommonsAttribution4.0InternationalLi-cense(CCBY4.0).Attribution:MariaPanteli,NielsBogaards,AlineHoningh.ÒModelingrhythmsimilarityforelectronicdancemusicÓ,15thInternationalSocietyforMusicInformationRetrievalConference,2014. /0$-$'0( 04&!"#$"% )0+$&50%%0(&6(+$7*%8($-.& & 9:;:9= �-++&?7*% ;:9= @(-78&?7*%A&"-(?,.-B+& =:C:99:9; "-$&F0B8(&07&,.0+8?GA&-.+0& 1.. "-$&F,.0+8?G & Figure1:Exampleofacommon(even)EDMrhythm[2].Themodelfocusesoncontent-basedanalysisofaudiorecordings.Alargeanddiverseliteraturedealswiththechallengesofaudiorhythmsimilarity.Theseinclude,a-mongstother,approachestoonsetdetection[1],tempoes-timation[9,25],rhythmicrepresentations[15,24],andfea-tureextractionforautomaticrhythmicpatterndescriptionandgenreclassiÞcation[5,12,20].SpeciÞctoEDM,[4]studyrhythmicandtimbrefeaturesforautomaticgenreclassiÞcation,and[6]investigatetemporalandstructuralfeaturesformusicgeneration.Inthispaper,analgorithmforrhythmsimilaritybasedonEDMcharacteristicsandperceptualrhythmattributesispresented.Themethodologyforextractingrhythmicele-mentsfromanaudiosegmentandasummaryofthefea-turesextractedisprovided.Thestepsofthealgorithmareevaluatedindividually.Similaritypredictionsofthemodelarecomparedtoperceptualratingsandfurtherconsidera-tionsarediscussed.2.METHODOLOGYStructuralchangesinanEDMtracktypicallyconsistofanevolutionoftimbreandrhythmasopposedtoaverse-chorusdivision.SegmentationisÞrstlyperformedtosplitthesignalintomeaningfulexcerpts.Thealgorithmdevel-opedin[21]isused,whichsegmentstheaudiosignalbasedontimbrefeatures(sincetimbreisimportantinEDMstruc-ture[2])andmusicalheuristics.EDMrhythmisexpressedviatheÔloopÕ,arepeatingpatternassociatedwithaparticular(oftenpercussive)in-strumentorinstruments[2].Rhythminformationcanbeextractedbyevaluatingcharacteristicsoftheloop:First,therhythmicpatternisoftenpresentedasacombinationofinstrumentsounds(eg.Figure1),thusexhibitingacertainÔrhythmpolyphonyÕ[3].Toanalyzethis,thesignalissplitintotheso-calledrhythmicstreams.Then,todescribetheunderlyingrhythm,featuresareextractedforeachstreambasedonthreeattributes:a)Theattackphaseoftheon-setsisconsideredtodescribeifthepatternisperformedon 15th International Society for Music Information Retrieval Conference (ISMIR 2014) 1 assumptionscouldberelaxedtoanalyzeEDMwithternarydivisionsorno meter,orexpandedtoothermusicstyleswithsimilarstructure.ThecorrelationreportedinSection3.4iscomputedfromapreliminarysetofexperimentdata.Moreratingsarecur-rentlycollectedandaregressionanalysisandtuningofthemodelisconsideredinfuturework.5.CONCLUSIONAmodelofrhythmsimilarityforElectronicDanceMusichasbeenpresented.Themodelextractsrhythmicfeaturesfromaudiosegmentsandcomputessimilaritybycompar-ingtheirfeaturevectors.AmethodforrhythmicstreamdetectionisproposedthatestimatesthenumberandrangeoffrequencybandsfromthespectralrepresentationofeachsegmentratherthanaÞxeddivision.Featuresareextractedfromeachstream,anapproachshowntobeneÞttheanal-ysis.Similaritypredictionsofthemodelmatchperceptualratingswithacorrelationof0.7.FutureworkwillÞne-tunepredictionsbasedonaperceptualrhythmsimilaritymodel.6.REFERENCES[1]S.Bock,A.Arzt,K.Florian,andS.Markus.On-linereal-timeonsetdetectionwithrecurrentneuralnet-works.InInternationalConferenceonDigitalAudioEffects,2012.[2]M.J.Butler.UnlockingtheGroove.IndianaUniversityPress,BloomingtonandIndianapolis,2006.[3]E.Cambouropoulos.VoiceandStream:PerceptualandComputationalModelingofVoiceSeparation.Perception,26(1):75Ð94,2008.[4]D.Diakopoulos,O.Vallis,J.Hochenbaum,J.Murphy,andA.Kapur.21stCenturyElectronica:MIRTech-niquesforClassiÞcationandPerformance.In[5]S.Dixon,F.Gouyon,andG.Widmer.TowardsChar-acterisationofMusicviaRhythmicPatterns.In[6]A.EigenfeldtandP.Pasquier.EvolvingStructuresforElectronicDanceMusic.InGeneticandEvolutionaryComputationConference,2013.[7]J.FooteandS.Uchihashi.Thebeatspectrum:anewapproachtorhythmanalysis.In,2001.[8]J.T.Foote.Mediasegmentationusingself-similaritydecomposition.InElectronicImaging.InternationalSocietyforOpticsandPhotonics,2003.[9]D.Gartner.Tempoestimationofurbanmusicusingtatumgridnon-negativematrixfactorization.In[10]J.W.Gordon.TheperceptualattacktimeofmusicalTheJournaloftheAcousticalSocietyofAmer-,82(1):88Ð105,1987.[11]T.D.GrifÞthsandJ.D.Warren.WhatisanauditoryNatureReviewsNeuroscience,5(11):887Ð892,[12]C.Guastavino,F.Gomez,G.Toussaint,F.Maran-dola,andE.Gomez.MeasuringSimilaritybetweenFlamencoRhythmicPatterns.JournalofNewMusicResearch,38(2):129Ð138,June2009.[13]J.A.Hockman,M.E.P.Davies,andI.Fujinaga.OneintheJungle:DownbeatDetectioninHardcore,Jungle,andDrumandBass.In,2012.[14]A.Klapuri,A.J.Eronen,andJ.T.Astola.Analysisofthemeterofacousticmusicalsignals.IEEETrans-actionsonAudio,SpeechandLanguageProcessing14(1):342Ð355,January2006.[15]F.Krebs,S.Bock,andG.Widmer.Rhythmicpatternmodelingforbeatanddownbeattrackinginmusicalaudio.In,2013.[16]O.Lartillot,T.Eerola,P.Toiviainen,andJ.Fornari.Multi-featureModelingofPulseClarity:Design,Vali-dationandOptimization.In,2008.[17]O.LartillotandP.Toiviainen.AMatlabToolboxforMusicalFeatureExtractionFromAudio.IntionalConferenceonDigitalAudioEffects,2007.[18]H.C.Longuet-HigginsandC.S.Lee.TheRhyth-micInterpretationofMonophonicMusic.MusicPer-ception:AnInterdisciplinaryJournal,1(4):424Ð441,[19]A.Novello,M.M.F.McKinney,andA.Kohlrausch.PerceptualEvaluationofInter-songSimilarityinWest-ernPopularMusic.JournalofNewMusicResearch40(1):1Ð26,March2011.[20]J.PaulusandA.Klapuri.MeasuringtheSimilarityofRhythmicPatterns.In,2002.[21]B.Rocha,N.Bogaards,andA.Honingh.SegmentationandTimbreSimilarityinElectronicDanceMusic.InSoundandMusicComputingConference,2013.[22]E.D.Scheirer.Tempoandbeatanalysisofacousticmusicalsignals.TheJournaloftheAcousticalSocietyofAmerica,103(1):588Ð601,January1998.[23]M.R.Schroeder,B.S.Atal,andJ.L.Hall.Optimizingdigitalspeechcodersbyexploitingmaskingpropertiesofthehumanear.TheJournaloftheAcousticalSocietyofAmerica,pages1647Ð1652,1979.[24]L.M.Smith.Rhythmicsimilarityusingmetricalpro-Þlematching.InInternationalComputerMusicCon-ference,2010.[25]J.R.ZapataandE.Gomez.ComparativeEvaluationandCombinationofAudioTempoEstimationAp-proaches.InAudioEngineeringSocietyConference 15th International Society for Music Information Retrieval Conference (ISMIR 2014) 542 3.2OnsetDetectionEvaluationOnsetdetectionisevaluatedwithasetof25MIDIandcorrespondingaudioexcerpts,speciÞcallycreatedforthispurpose.Inthisapproach,onsetsaredetectedperstream,thereforeonsetannotationsshouldalsobeprovidedperstream.ForanumberofdifferentEDMrhythms,MIDIÞleswerecreatedwiththeconstraintthateachMIDIin-strumentperformsauniquerhythmicpatternthereforerep-resentsauniquestream,andwereconvertedtoaudio.TheonsetsestimatedfromtheaudiowerecomparedtotheannotationsoftheMIDIÞleusingtheevaluationmea-suresoftheMIREXOnsetDetectiontask.Forthis,nostreamalignmentisperformedbutratheronsetsfromallstreamsaregroupedtoasingleset.Forexcerpts,an-measureof,presicionof,andrecallofareobtainedwithatolerancewindowofms.Inaccura-ciesinonsetdetectionaredue(onaverage)todoubledthanmergedonsets,becauseusuallymorestreams(andhencemoreonsets)aredetected.3.3DownbeatDetectionEvaluationToevaluatethedownbeatthesubsetofsegmentsde-scribedinSection3.1wasused.Foreachsegmenttheannotateddownbeatwascomparedtotheestimatedonewithatolerancewindowofms.Anaccuracyofwasachieved.Downbeatdetectionwasalsoevaluatedatthebeat-level,i.e.,estimatingwhetherthedownbeatcor-respondstooneofthefourbeatsofthemeter(insteadofoff-beatpositions).Thisgaveanaccuracyof,mean-ingthatintheothercasesthedownbeatwasdetectedontheoff-beatpositions.ForsomeEDMtracksitwasobservedthathighdegreeofperiodicitycompensatesforawronglyestimateddownbeat.Theoverallresultsofthesimilaritypredictionsofthemodel(Section3.4)indicateonlyami-norincreasewhenthecorrect(annotated)downbeatsaretakenintoaccount.Itishenceconcludedthatthedown-beatdetectionalgorithmdoesnothavegreatinßuenceonthecurrentresultsofthemodel.3.4MappingModelPredictionstoPerceptualRatingsofSimilarityThemodelÕspredictionswereevaluatedwithperceptualratingsofrhythmsimilaritycollectedviaalisteningex-periment.PairwisecomparisonsofasmallsetofsegmentsrepresentingvariousrhythmicpatternsofEDMwerepre-sented.Subjectswereaskedtoratetheperceivedrhythmsimilarity,choosingfromafourpointscale,andreportalsotheconÞdenceoftheirrating.Fromapreliminarycollec-tionofexperimentdata,pairs(representingatotalofuniquemusicsegments)wereselectedforfurtheranalysis.Thesewereratedfromatotalofparticipants,withmeanyearsoldandstandarddeviation.Thetheparticipantsreceivedformalmusicaltraining,wasfamiliarwithEDMandhadexperienceasEDMmu-sician/producer.Theselectedpairswereratedbetweentimes,withallparticipantsreportingconÞdenceintheir www.MIREX.org rpfeatures -0.170.22attackcharacterization0.480.000.330.01metricaldistributionexcl.metricalproÞle0.690.00metricaldistributionincl.metricalproÞle0.700.00 Table1:PearsonÕscorrelation-valuesbetweenthemodelÕspredictionsandperceptualratingsofrhythmsim-ilarityfordifferentsetsoffeatures.rating,andallratingsbeingconsistent,i.e.,ratedsimilaritywasnotdeviatingmorethanpointscale.Themeanoftheratingswasutilizedasthegroundtruthratingperpair.Foreachpair,similaritycanbecalculatedviaapplyingadistancemetrictothefeaturevectorsoftheunderlyingsegments.Inthispreliminaryanalysis,thecosinedistancewasconsidered.PearsonÕscorrelationwasusedtocomparetheannotatedandpredictedratingsofsimilarity.ThiswasappliedfordifferentsetsoffeaturesasindicatedinTable1.Amaximumcorrelationofwasachievedwhenallfeatureswerepresented.Thenon-zerocorrelationhypoth-esiswasnotrejected()fortheattackcharacter-izationfeaturesindicatingnon-signiÞcantcorrelationwiththe(currentsetof)perceptualratings.Theperiodicityfea-turesarecorrelatedwith,showingastronglinkwithperceptualrhythmsimilarity.Themetricaldistribu-tionfeaturesindicateacorrelationincreaseofthemetricalproÞleisincludedinthefeaturevector.ThisisinagreementwiththeÞndingof[24].Asanalternativeevaluationmeasure,themodelÕspre-dictionsandperceptualratingsweretransformedtoabi-naryscale(i.e.,beingdissimilarandbeingsimilar)andtheiroutputwascompared.ThemodelÕspredictionsmatchedtheperceptualratingswithanaccuracyofHencethemodelmatchestheperceptualsimilarityratingsatnotonlyrelative(i.e.,PearsonÕscorrelation)butalsoab-soluteway,whenabinaryscalesimilarityisconsidered.4.DISCUSSIONANDFUTUREWORKIntheevaluationofthemodel,thefollowingconsidera-tionsaremade.HighcorrelationofwasachievedwhenthemetricalproÞle,outputperstream,wasaddedtothefeaturevector.Analternativeexperimenttestedthecor-relationwhenconsideringthemetricalproÞleasawhole,i.e.,asasumacrossallstreams.Thisgaveacorrelationofindicatingtheimportanceofstreamseparationandhencetheadvantageofthemodeltoaccountforthis.Amaximumcorrelationofwasreported,takingintoaccountthedownbeatdetectionbeingofthecasescorrect.AlthoughregularityinEDMsometimescompen-satesforthis,modelÕspredictionscanbeimprovedwithamorerobustdownbeatdetection.Featuresofperiodicity(Section2.2.2)andmetricaldis-tribution(Section2.2.3)wereextractedassuminga ter,and-thnoteresolutionthroughoutthesegment.ThisisgenerallytrueforEDM,butexceptionsdoexist[2].The 15th International Society for Music Information Retrieval Conference (ISMIR 2014) 541 0 0.5 1 1.5 2 2.5 !0.6 !0.4 !0.2 0 0.2 0.4 0.6 0.8 1 1.2 Lag (s)Normalized amplitude 1 Beat1 Bar Figure4:Autocorrelationofonsetsindicatinghighperi-odicitiesof1barand1beatduration. !!! 0 10 20 30 time (s)Stream 1 0 5 10 15 Stream 2 0 10 20 30 Onset curve (Envelope) (half!wave rectified)Stream 3 Bar 2Bar 3Bar 4Bar 1 Downbeat ! "#$!%! "#$!& ! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! '($)#*!% %!+!+!+!%!+!+!+!%!+!+!+!%!+!+ %!+!+!+!%!+!+!+!%!+!+!+!%!+!+ '($)#*!&! +!+!+!+!%!+!+!+!+!+!+!+!%!+!+ +!+!+!+!%!+!+!+!+!+!+!+!%!+!+ '($)#*!, +!+!%!+!+!+!%!+!+!+!%!+!++!%!+! +!+!%!+!+!+!%!+!+!+!%!+!++!%!+ '($)#*!- %!%!%!%!%!%!%!%!%!%!%!%!%!%!%!% %!%!%!%!%!%!%!%!%!%!%!%!%!%!%!% ! "#$!%! "#$!& "#$!,! "#$!- ! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! '($)#*!, %!+!%!+!%!+!%!+!%!+!%!+!%!+!%!+! %!+!%!+!%!+!%!+!%!+!%!+!%!+!%!+! %!+!%!+!%!+!%!+!%!+!%!+!%!+!%!+! %!+!%!+!%!+!%!+!%!+!%!+!%!+!%!+! '($)#*!& +!+!+!+%!+!+!+!+!+!+!+!%!+!+!+! +!+!+!+!%!+!+!+!+!+!+!+!%!+!+!+! +!+!+!+!%!+!+!+!+!+!+!+!%!+!+!+! +!+!+!+!%!+!+!+!+!+!+!+!%!+!+!+! '($)#*!% %!+!+!+!+!+!+!+!%!+!+!+!+!+!+!+! %!+!+!+!+!+!+!+!%!+!+!+!+!+!+!+! %!+!+!+!+!+!+!+!%!+!+!+!+!+!+!+! %!+!+!+!+!+!+!+!%!+!+!+!+!+!+!+! Figure5:MetricalproÞleoftherhythminFigure1assum-ingforsimplicitya-barlengthandconstantamplitude.Þleiscomputedandthefollowingfeaturesareextracted.FeaturesarecomputedperstreamandaveragedacrossallMeasuresthestrengthoftheeventslyingontheweaklocationsofthemeter.Thesyncopationmodelof[18]isusedwithadaptationtoaccountfortheamplitude(onsetstrength)ofthesyncopatednote.Threemeasuresofsyncopationareconsideredthatapplyhierarchicalweightswith,respectively,sixteenthnote,eighthnote,andquarternoteresolution.DenotestheratioofthenumberofonsetsinthesecondhalfofthepatternthatappearinexactlythesamepositionintheÞrsthalfofthepattern[6].Istheratioofthenumberofonsetsoverthepossibletotalnumberofonsetsofthepattern(inthiscaseMeasurestheonsetsÕstrengthofthepattern.ItdescribestheratioofthesumofonsetsÕstrengthoverthemaximumstrengthmultipliedbythepossibletotalnumberofonsets(inthiscaseCentreofGravity:Denotesthepositioninthepatternwherethemostandstrongestonsetsoccur(i.e.,indicateswhethermostonsetsappearatthebeginningorattheendofthepatternetc.).Asidefromthesefeatures,themetricalproÞle(cf.Fig-ure5)isalsoaddedtotheÞnalfeaturevector.Thiswasfoundtoimproveresultsin[24].Inthecurrentapproach,themetricalproÞleisprovidedperstream,restrictedtoatotalofstreams,andoutputintheÞnalfeaturevectorinorderoflowtohighfrequencycontentstreams.2.2.4DownbeatDetectionThedownbeatdetectionalgorithmusesinformationfromthemetricalstructureandmusicalheuristics.Twoassump-tionsaremade:Assumption1:Strongbeatsofthemeteraremorelikelytobeemphasizedacrossallrhythmicstreams.Assumption2:Thedownbeatisoftenintroducedbyaninstrumentinthelowfrequencies,i.e.abassorakickdrum[2,13].Consideringtheabove,theonsetsperstreamarequan-tizedassuminga meter,-thnoteresolution,andasetofdownbeatcandidates(inthiscasetheonsetsthatliewithinonebarlengthcountingfromthebeginningoftheseg-ment).Foreachdownbeatcandidate,hierarchicalweights[18]thatemphasizethestrongbeatsofthemeterasindi-catedbyAssumption1,areappliedtothequantizedpat-terns.Note,thereisonepatternforeachrhythmicstream.Thepatternsarethensummedbyapplyingmoreweighttothepatternofthelow-frequencystreamasindicatedbyAs-sumption2.Finally,thecandidatewhosequantizedpatternwasweightedmost,ischosenasthedownbeat.3.EVALUATIONOneofthegreatestchallengesofmusicsimilarityevalu-ationisthedeÞnitionofagroundtruth.Insomecases,objectiveevaluationispossible,whereagroundtruthisde-ÞnedonaquantiÞablecriterion,i.e.,rhythmsfromapartic-ulargenrearesimilar[5].Inothercases,musicsimilarityisconsideredtobeinßuencedbytheperceptionofthelis-tenerandhencesubjectiveevaluationismoresuitable[19].Objectiveevaluationinthecurrentstudyisnotpreferablesincedifferentrhythmsdonotnecessarilyconformtodif-ferentgenresorsubgenres.Thereforeasubjectiveeval-uationisusedwherepredictionsofrhythmsimilarityarecomparedtoperceptualratingscollectedviaalisteningex-periment(cf.Section3.4).Detailsoftheevaluationofrhythmicstream,onset,anddownbeatdetectionarepro-videdinSections3.1-3.3.Asubsetoftheannotationsusedintheevaluationofthelatterisavailableonline3.1RhythmicStreamsEvaluationThenumberofstreamsisevaluatedwithperceptualanno-tations.Forthis,asubsetofsongsfromatotalofartists(songsperartist)fromavarietyofEDMgenresandsubgenreswasselected.Foreachsong,segmentationwasappliedusingthealgorithmof[21]andacharacteristicsegmentwasselected.Foursubjectswereaskedtoevalu-atethenumberofrhythmicstreamstheyperceiveineachsegment,choosingbetween,whererhythmicstreamwasdeÞnedasastreamofuniquerhythm.Forofthesegments,thesubjectsÕresponsesÕstandarddeviationwassigniÞcantlysmall.Theestimatednumberofrhythmicstreamsmatchedthemeanofthesub-jectÕsresponsedistributionwithanaccuracyof AlthoughsomerhythmicpatternsarecharacteristictoanEDMgenreorsubgenre,itisnotgenerallytruethattheseareuniqueandinvariant. 15th International Society for Music Information Retrieval Conference (ISMIR 2014) 540 ! 16 18 20 22 24 26 28 5 10 15 20 Bark!Spectrumtime axis (in s.)Bark bands "#$% &'()!*'+,- (a)Bark-bandspectrogram. ! 5 10 15 20 20 15 10 5 "#$%!&#'() "#$%!&#'() (b)Self-similaritymatrix. ! 2 4 6 8 10 12 14 16 18 20 22 24 10 15 20 Novelty ValueBark BandsNovelty Curve "#$%&'!( "#$%&'!) "#$%&'!* "#$%&'!+ ,-.%/#0!.&/1% 2&$3!4&567 (c)Noveltycurve.Figure3:Detectionofrhyhmicstreamsusingthenoveltyapproach;Þrstabark-bandspectrogramiscomputed,thenitsself-similaritymatrix,andthenthenovelty[7]isappliedwherethenoveltypeaksdeÞnethestreamboundaries.2.2FeatureExtractionTheonsetsineachstreamrepresenttherhythmicelementsofthesignal.Tomodeltheunderlyingrhythm,featuresareextractedfromeachstream,basedonthreeattributes,namely,characterizationofattack,periodicity,andmetri-caldistributionofonsets.Thesearecombinedtoafeaturevectorthatservesformeasuringinter-segmentsimilarity.Thesectionsbelowdescribethefeatureextractionprocessindetail.2.2.1AttackCharacterizationTodistinguishbetweenpercussiveandnon-percussivepat-terns,featuresareextractedthatcharacterizetheattackpha-seoftheonsets.Inparticular,theattacktimeandattackslopeareconsidered,amongother,essentialinmodelingtheperceivedattacktime[10].Theattackslopewasalsousedinmodelingpulseclarity[16].Ingeneral,onsetsfrompercussivesoundshaveashortattacktimeandsteepattackslope,whereasnon-percussivesoundshavelongerattacktimeandgraduallyincreasingattackslope.Forallonsetsinallstreams,theattacktimeandat-tackslopeisextractedandsplitintwoclusters;theÔslowÕ(non-percussive)andÔfastÕ(percussive)attackphaseon-sets.Here,itisassumedthatbothpercussiveandnon-percussiveonsetscanbepresentinagivensegment,hencesplittingintwoclustersissuperiorto,e.g.,computingtheaverage.Themeanandstandarddeviationofthetwoclus-tersoftheattacktimeandattackslope(atotalofisoutputtothefeaturevector.2.2.2PeriodicityOneofthemostcharacteristicstyleelementsinthemusicalstructureofEDMisrepetition;theloop,andconsequentlytherhythmicsequence(s),arerepeatingpatterns.Toana-lyzethis,theperiodicityoftheonsetdetectionfunctionperstreamiscomputedviaautocorrelationandsummedacrossallstreams.Themaximumdelaytakenintoaccountispro-portionaltothebarduration.Thisiscalculatedassumingasteadytempoand meterthroughouttheEDMtrack[2].Thetempoestimationalgorithmof[21]isused.Fromtheautocorrelationcurve(cf.Figure4),atotaloffeaturesareextracted:Lagdurationofmaximumautocorrelation:Thelo-cation(intime)ofthesecondhighestpeak(theÞrstbeingatlag0)oftheautocorrelationcurvenormalizedbythebarduration.Itmeasureswhetherthestrongestperiodicityoc-cursineverybar(i.e.featurevalue=1),oreveryhalfbar(i.e.featurevalue=0.5)etc.Amplitudeofmaximumautocorrelation:Theam-plitudeofthesecondhighestpeakoftheautocorrelationcurvenormalizedbytheamplitudeofthepeakatlag0.Itmeasureswhetherthepatternisrepeatedinexactlythesameway(i.e.featurevalue=1)orsomewhatinasimilarway(i.e.featurevalue)etc.Harmonicityofpeaks:Thisistheharmonicityasde-Þnedin[16]withadaptationtothereferencelagcor-respondingtothebeatdurationandadditionalweightingoftheharmonicityvaluebythetotalnumberofpeaksoftheautocorrelationcurve.Thisfeaturemeasureswhetherrhythmicperiodicitiesoccurinharmonicrelationtothebeat(i.e.featurevalue=1)orinharmonic(i.e.featurevalue=0).Measureswhethertheautocorrelationcurveissmoothorspikyandissuitablefordistinguishingbe-tweenperiodicpatterns(i.e.featurevalue=0),andnon-periodic(i.e.featurevalue=1).Entropy:AnothermeasureoftheÔpeakinessÕofauto-correlation[16],suitablefordistinguishingbetweenÔclearÕrepetitions(i.e.distributionwithnarrowpeaksandhencefeaturevaluecloseto0)andunclearrepetitions(i.e.widepeaksandhencefeaturevalueincreased).2.2.3MetricalDistributionTomodelthemetricalaspectsoftherhythmicpattern,themetricalproÞle[24]isextracted.Forthis,thedownbeatisdetectedasdescribedinSection2.2.4,onsetsperstreamarequantizedassuminga meterand-thnoteresolu-tion[2],andthepatterniscollapsedtoatotalofbars.ThelatterisinagreementwiththelengthofamusicalphraseinEDMbeingusuallyinmultiplesof,i.e.,4-bar,8-bar,or16-barphrase[2].ThemetricalproÞleofagivenstreamisthuspresentedasavectorofbins(sixteenthnotesperbeat)withrealvaluesrangingbe-tween0(noonset)to1(maximumonsetstrength)asshowninFigure5.Foreachrhythmicstream,ametricalpro- 15th International Society for Music Information Retrieval Conference (ISMIR 2014) 539 feature extraction stream # 2 segmentation feature vector similarity metrical distribution rhythmic streamsdetection periodicity attackcharacterization onset detectionfeature extraction stream # 1stream # 3 Figure2:Overviewofmethodology.percussiveornon-percussiveinstruments.Althoughthisistypicallyviewedasatimbreattribute,thepercussive-nessofasoundisexpectedtoinßuencetheperceptionofrhythm[16].b)TherepetitionofrhythmicsequencesofthepatternaredescribedbyevaluatingcharacteristicsofdifferentlevelsofonsetsÕperiodicity.c)Themetricalstructureofthepatternischaracterizedviafeaturesex-tractedfromthemetricalproÞle[24]ofonsets.Basedontheabove,afeaturevectorisextractedforeachsegmentandisusedtomeasurerhythmsimilarity.Inter-segmentsimilarityisevaluatedwithperceptualratingscollectedviaaspeciÞcallydesignedexperiment.AnoverviewofthemethodologyisshowninFigure2anddetailsforeachstepareprovidedinthesectionsbelow.PartofthealgorithmisimplementedusingtheMIRToolbox[17].2.1RhythmicStreamsSeveralinstrumentscontributetotherhythmicpatternofanEDMtrack.Mosttypicalexamplesincludecombina-tionsofbassdrum,snareandhi-hat(eg.Figure1).Thisismainlyafunctionalratherthanastrictlyinstrumentaldi-vision,andinEDMoneÞndsvariousinstrumentsoundstotaketheroleofbass,snareandhi-hat.Indescribingrhythm,itisessentialtodistinguishbetweenthesesourcessinceeachcontributesdifferentlytorhythmperception[11].Followingthis,[15,24]describerhythmicpatternsoflatindancemusicintwopreÞxedfrequencybands(lowandhighfrequencies),and[9]representsdrumpatternsastwocomponents,thebassandsnaredrumpattern,calculatedvianon-negativematrixfactorizationofthespectrogram.In[20],rhythmiceventsaresplitbasedontheirperceivedloudnessandbrightness,wherethelatterisdeÞnedasafunctionofthespectralcentroid.Inthecurrentstudy,rhythmicstreamsareextractedwithrespecttothefrequencydomainandloudnesspattern.Inparticular,theShortTimeFourierTransformofthesig-naliscomputedandlogarithmicmagnitudespectraareas-signedtobarkbands,resultingintoatotalof24bandsforkHzsamplingrate.Synchronousmaskingismod-eledusingthespreadingfunctionof[23],andtemporalmaskingismodeledwithasmoothingwindowofThisrepresentationishereafterreferredtoasloudnessen-velopeanddenotedbyforbarkbands,...,self-similaritymatrixiscomputedfromthis24-bandrep-resentationindicatingthebandsthatexhibitsimilarloud-nesspattern.Thenoveltyapproachof[8]isappliedtosimilaritymatrixtodetectadjacentbandsthatshouldbegroupedtothesamerhythmicstream.ThepeakofthenoveltycurvedeÞnethenumberofthebarkbandthatmarksthebeginningofanewstream,i.e.,if,...,,...,Ifortotalnumberof,thenstreamconsistsofbarkbandsgivenby,,pi,pi+1#1]}fori=1,...,IIpI,24]}fori=I.(1)Anupperlimitofstreamsisconsideredbasedontheap-proachof[22]thatusesatotalofbandsforonsetdetec-tionand[14]thatsuggestsatotalofthreeorfourbandsformeteranalysis.Thenotionofrhythmicstreamhereissimilartotheno-tionofÔaccentbandÕin[14]withthedifferencethateachrhythmicstreamisformedonavariablenumberofadja-centbarkbands.Detectingarhythmicstreamdoesnotnecessarilyimplyseparatingtheinstruments,sinceiftwoinstrumentsplaythesamerhythmtheyshouldbegroupedtothesamerhythmicstream.Theproposedapproachdoesnotdistinguishinstrumentsthatlieinthesamebarkband.Theadvantageisthatthenumberofstreamsandthefre-quencyrangeforeachstreamdonotneedtobepredeter-minedbutareratherestimatedfromthespectralrepresen-tationofeachsong.ThisbeneÞtstheanalysisofelectronicdancemusicbynotimposinganyconstraintsonthepossi-bleinstrumentsoundsthatcontributetothecharacteristicrhythmicpattern.2.1.1OnsetDetectionToextractonsetcandidates,theloudnessenvelopeperbarkbandanditsderivativearenormalizedandsummedwithmoreweightonloudnessthanitsderivative,i.e.,)=(1isthenormalizedloudnessenvelopenormalizedderivativeof,...,Ntheframenum-berforatotalofframes,andtheweightingfac-tor.ThisissimilartotheapproachdescribedbyEquationin[14]withreduced,andiscomputedpriorsummationtothedifferentstreamsassuggestedin[14,22].Onsetsaredetectedviapeakextractionwithineachstream,wherethe(rhythmic)contentofstreamisdeÞnedasasinEquation1andasinEquation2.Thisonsetdetectionapproachincorporatessimilarmethodolog-icalconceptswiththepositivelyevaluatedalgorithmsforthetaskofaudioonsetdetection[1]inMIREX2012,andtempoestimation[14]inthereviewof[25]. 15th International Society for Music Information Retrieval Conference (ISMIR 2014) 538 MODELINGRHYTHMSIMILARITYFORELECTRONICDANCEMUSICMariaPanteliUniversityofAmsterdam,Amsterdam,NetherlandsNielsBogaardsElephantcandy,Amsterdam,NetherlandsAlineHoninghUniversityofAmsterdam,Amsterdam,NetherlandsABSTRACTAmodelforrhythmsimilarityinelectronicdancemusic(EDM)ispresentedinthispaper.RhythminEDMisbuiltontheconceptofaÔloopÕ,arepeatingsequencetypicallyassociatedwithafour-measurepercussivepattern.Thepresentedmodelcalculatesrhythmsimilaritybetweenseg-mentsofEDMinthefollowingsteps.1)Eachsegmentissplitindifferentperceptualrhythmicstreams.2)Eachstreamischaracterizedbyanumberofattributes,mostno-tably:attackphaseofonsets,periodicityofrhythmicel-ements,andmetricaldistribution.3)Theseattributesarecombinedintoonefeaturevectorforeverysegment,af-terwhichthesimilaritybetweensegmentscanbecalcu-lated.Thestagesofstreamsplitting,onsetdetectionanddownbeatdetectionhavebeenevaluatedindividually,andalisteningexperimentwasconductedtoevaluatetheover-allperformanceofthemodelwithperceptualratingsofrhythmsimilarity.1.INTRODUCTIONMusicsimilarityhasattractedresearchfrommultidisci-plinarydomainsincludingtasksofmusicinformationre-trievalandmusicperceptionandcognition.Especiallyforrhythm,studiesexistonidentifyingandquantifyingrhythmproperties[16,18],aswellasestablishingrhythmsimilar-itymetrics[12].Inthispaper,rhythmsimilarityisstudiedwithafocusonElectronicDanceMusic(EDM),agenrewithvariousanddistinctrhythms[2].EDMisanumbrellatermconsistingoftheÔfourontheßoorÕgenressuchastechno,house,trance,andtheÔbreakbeat-drivenÕgenressuchasjungle,drumÔnÕbass,breaksetc.Ingeneral,fourontheßoorgenresarecharac-terizedbyafour-beatsteadybass-drumpatternwhereasbreakbeat-drivenexploitirregularitybyemphasizingthemetricallyweaklocations[2].However,rhythminEDMexhibitsmultipletypesofsubtlevariationsandembellish-ments.Thegoalofthepresentstudyistodeveloparhythmsimilaritymodelthatcapturestheseembellishmentsandal-lowsforaÞneinter-songrhythmsimilarity. MariaPanteli,NielsBogaards,AlineHoningh.LicensedunderaCreativeCommonsAttribution4.0InternationalLi-cense(CCBY4.0).Attribution:MariaPanteli,NielsBogaards,AlineHoningh.ÒModelingrhythmsimilarityforelectronicdancemusicÓ,15thInternationalSocietyforMusicInformationRetrievalConference,2014. /0$-$'0( 04&!"#$"% )0+$&50%%0(&6(+$7*%8($-.& & 9:;:9= �-++&?7*% ;:9= @(-78&?7*%A&"-(?,.-B+& =:C:99:9; "-$&F0B8(&07&,.0+8?GA&-.+0& 1..& "-$&F,.0+8?G & Figure1:Exampleofacommon(even)EDMrhythm[2].Themodelfocusesoncontent-basedanalysisofaudiorecordings.Alargeanddiverseliteraturedealswiththechallengesofaudiorhythmsimilarity.Theseinclude,a-mongstother,approachestoonsetdetection[1],tempoes-timation[9,25],rhythmicrepresentations[15,24],andfea-tureextractionforautomaticrhythmicpatterndescriptionandgenreclassiÞcation[5,12,20].SpeciÞctoEDM,[4]studyrhythmicandtimbrefeaturesforautomaticgenreclassiÞcation,and[6]investigatetemporalandstructuralfeaturesformusicgeneration.Inthispaper,analgorithmforrhythmsimilaritybasedonEDMcharacteristicsandperceptualrhythmattributesispresented.Themethodologyforextractingrhythmicele-mentsfromanaudiosegmentandasummaryofthefea-turesextractedisprovided.Thestepsofthealgorithmareevaluatedindividually.Similaritypredictionsofthemodelarecomparedtoperceptualratingsandfurtherconsidera-tionsarediscussed.2.METHODOLOGYStructuralchangesinanEDMtracktypicallyconsistofanevolutionoftimbreandrhythmasopposedtoaverse-chorusdivision.SegmentationisÞrstlyperformedtosplitthesignalintomeaningfulexcerpts.Thealgorithmdevel-opedin[21]isused,whichsegmentstheaudiosignalbasedontimbrefeatures(sincetimbreisimportantinEDMstruc-ture[2])andmusicalheuristics.EDMrhythmisexpressedviatheÔloopÕ,arepeatingpatternassociatedwithaparticular(oftenpercussive)in-strumentorinstruments[2].Rhythminformationcanbeextractedbyevaluatingcharacteristicsoftheloop:First,therhythmicpatternisoftenpresentedasacombinationofinstrumentsounds(eg.Figure1),thusexhibitingacertainÔrhythmpolyphonyÕ[3].Toanalyzethis,thesignalissplitintotheso-calledrhythmicstreams.Then,todescribetheunderlyingrhythm,featuresareextractedforeachstreambasedonthreeattributes:a)Theattackphaseoftheon-setsisconsideredtodescribeifthepatternisperformedon 15th International Society for Music Information Retrieval Conference (ISMIR 2014) 537