Zhiyao Duan Jinyu Han and Bryan Pardo EECS Dept Northwestern Univ Interactive Audio Lab httpmusiccsnorthwesternedu For presentation in ICASSP 2010 Dallas Texas USA Multipitch Estimation amp Tracking Task ID: 784390
Download The PPT/PDF document "Song-level Multi-pitch Tracking by Heavi..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Song-level Multi-pitch Tracking by Heavily Constrained Clustering
Zhiyao Duan, Jinyu Han and Bryan PardoEECS Dept., Northwestern Univ.Interactive Audio Lab, http://music.cs.northwestern.eduFor presentation in ICASSP 2010, Dallas, Texas, USA.
Slide2Multi-pitch Estimation & Tracking Task
Given polyphonic music played by several monophonic harmonic instruments (Num known) Estimate a pitch trajectory for each instrumentNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu2
Slide3Potential ApplicationsAutomatic music transcription
Harmonic source separationOther applicationsMelody-based music searchChord recognitionSource localizationMusic education……Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu3
Slide4The 2-stage Standard Approach
Stage 1: Multi-pitch Estimation (MPE): estimate pitches in each single time frameZ. Duan, B. Pardo and C. Zhang. , “Multiple Fundamental Frequency Estimation by Modeling Spectral Peaks and Non-peak Regions”, IEEE Trans. Audio Speech Language Process., in press.Stage 2: Multi-pitch Tracking (MPT): connect pitch estimates across frames into pitch trajectories4
…
Time
Frequency
Slide5State of the Art of MPTWhat existing MPT methods do
Form short pitch trajectories within a note, (note-level) according to local time-frequency proximity of pitch estimatesOur contributionForm long pitch trajectories through multiple notes (song-level) using a new constrained clustering algorithmNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu5
Slide6Try Clustering by TimbreEach trajectory is a cluster of pitch estimates
One cluster per instrumentClustering principle: maintain timbre consistency in each clusterNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu?
Slide7Timbre Feature of Pitch EstimatesHarmonic structure:
relative amplitudes of first 50 harmonics
Time
Frequency
Slide8Minimize This Objective FunctionNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu
A partitioninto K clustersThe 50-d harmonicstructure of i-thpitch estimateNumber ofClustersCenter of k-th clusterFor all pitch estimates in k-th cluster
Slide9Objective Function Is Not EnoughNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu
Slide10Add Pitch-locality ConstraintsMust-link
: pitch estimates close in both time and frequency should be in the same clusterCannot-link: simultaneous pitches should not be in the same cluster (only for monophonic instruments)10
Time
Frequency
Slide11Properties of Our ProblemObjective: timbre consistency
Constraints: pitch localityPrevious constrained clustering algorithms do not apply due to the following properties:Inconsistent constraints: pitch estimates sometimes erroneous may make constraints unsatisfiable Heavily constrained: nearly every pitch estimate is involved in at least one constraintNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu
Slide12The Proposed Clustering Algorithm
: clustering in n-th iteration; : {all constraints satisfied by } ; 1. Start from an initial clustering , which satisfies , a subset of all constraints; n=1; 2. Find a new clustering that decreases the objective and also satisfies ; 3. = {all constraints satisfied by } ; 4. Repeat 2-4 until the objective (nearly) cannot be decreased; Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu
Slide13Initial ClusteringTrivial one
: a random partition : constraints satisfied by , may be emptyA more informative one for MPT : label pitches according to pitch order in each frame: highest, second-highest, third.., fourth… : will contain all cannot-links
…
Time
Frequency
…
Time
Frequency
Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu
Slide141. Satisfy current constraints2. Decrease the objective function
: satisfied cannot-link : unsatisfied cannot-link : satisfied must-link : unsatisfied cannot-linkSwap set: A connected subgraph between two clusters. Traverse all swap sets until finding a new clustering that decreases the objective function4237
8
3
1
5
6
Find A New Clustering
Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu
4
2
3
7
8
3
1
5
6
4
2
3
7
8
3
1
5
6
Slide15Algorithm ReviewNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu
: partition of points into clusters : feasible solution space under constraints
Slide16Experiments
Data set10 J.S. Bach chorales (quartets, played by violin, clarinet, saxophone and bassoon)Each instrument is recorded individually, then mixedGround-truth pitch trajectoriesUse YIN on monophonic tracks before mixingInput pitch estimatesOur previous work in [1]Input accuracy: 70.0+-3.1% [1] Zhiyao Duan, Bryan Pardo and Changshui Zhang, “Multiple Fundamental Frequency Estimation by Modeling Spectral Peaks and Non-peak Regions”, IEEE Trans. Audio Speech Language Process., in press.16Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu
Slide17Overall Multi-pitch Tracking Results
Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.eduMean % of correct pitch estimates
Slide18Among Correctly Estimated Pitches
Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu
Slide19An Example
Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu
Slide20An Example
Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu
Slide21ConclusionFormulate the song-level Multi-pitch Tracking problem as a constrained clustering problem
Objective: timbre consistencyConstraints: pitch localityExisting constrained clustering algorithms do not apply due to problem propertiesPropose a new constrained clustering algorithmExperimental results are promisingNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu
Slide22Thanks you!22
Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu