Song-level Multi-pitch Tracking by Heavily Constrained Clus - PowerPoint Presentation

Slide1

Song-level Multi-pitch Tracking by Heavily Constrained Clustering

Zhiyao Duan, Jinyu Han and Bryan PardoEECS Dept., Northwestern Univ.Interactive Audio Lab, http://music.cs.northwestern.eduFor presentation in ICASSP 2010, Dallas, Texas, USA.Slide2

Multi-pitch Estimation & Tracking Task

Given polyphonic music played by several monophonic harmonic instruments (Num known) Estimate a pitch trajectory for each instrumentNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu2Slide3

Potential ApplicationsAutomatic music transcription

Harmonic source separationOther applicationsMelody-based music searchChord recognitionSource localizationMusic education……Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu3Slide4

The 2-stage Standard Approach

Stage 1: Multi-pitch Estimation (MPE): estimate pitches in each single time frameZ. Duan, B. Pardo and C. Zhang. , “Multiple Fundamental Frequency Estimation by Modeling Spectral Peaks and Non-peak Regions”, IEEE Trans. Audio Speech Language Process., in press.Stage 2: Multi-pitch Tracking (MPT): connect pitch estimates across frames into pitch trajectories4

…

Time

FrequencySlide5

State of the Art of MPTWhat existing MPT methods do

Form short pitch trajectories within a note, (note-level) according to local time-frequency proximity of pitch estimatesOur contributionForm long pitch trajectories through multiple notes (song-level) using a new constrained clustering algorithmNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu5Slide6

Try Clustering by TimbreEach trajectory is a cluster of pitch estimates

One cluster per instrumentClustering principle: maintain timbre consistency in each clusterNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu?Slide7

Timbre Feature of Pitch EstimatesHarmonic structure:

relative amplitudes of first 50 harmonics

Time

FrequencySlide8

Minimize This Objective FunctionNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu

A partitioninto K clustersThe 50-d harmonicstructure of i-thpitch estimateNumber ofClustersCenter of k-th clusterFor all pitch estimates in k-th clusterSlide9

Objective Function Is Not EnoughNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.eduSlide10

Add Pitch-locality ConstraintsMust-link

: pitch estimates close in both time and frequency should be in the same clusterCannot-link: simultaneous pitches should not be in the same cluster (only for monophonic instruments)10

Time

FrequencySlide11

Properties of Our ProblemObjective: timbre consistency

Constraints: pitch localityPrevious constrained clustering algorithms do not apply due to the following properties:Inconsistent constraints: pitch estimates sometimes erroneous may make constraints unsatisfiable Heavily constrained: nearly every pitch estimate is involved in at least one constraintNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.eduSlide12

The Proposed Clustering Algorithm

: clustering in n-th iteration; : {all constraints satisfied by } ; 1. Start from an initial clustering , which satisfies , a subset of all constraints; n=1; 2. Find a new clustering that decreases the objective and also satisfies ; 3. = {all constraints satisfied by } ; 4. Repeat 2-4 until the objective (nearly) cannot be decreased; Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.eduSlide13

Initial ClusteringTrivial one

: a random partition : constraints satisfied by , may be emptyA more informative one for MPT : label pitches according to pitch order in each frame: highest, second-highest, third.., fourth… : will contain all cannot-links

…

Time

Frequency

…

Time

Frequency

Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.eduSlide14

1. Satisfy current constraints2. Decrease the objective function

: satisfied cannot-link : unsatisfied cannot-link : satisfied must-link : unsatisfied cannot-linkSwap set: A connected subgraph between two clusters. Traverse all swap sets until finding a new clustering that decreases the objective function4237

8

3

1

5

6

Find A New Clustering

Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu

4

2

3

7

8

3

1

5

6

4

2

3

7

8

3

1

5

6Slide15

Algorithm ReviewNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu

: partition of points into clusters : feasible solution space under constraints Slide16

Experiments

Data set10 J.S. Bach chorales (quartets, played by violin, clarinet, saxophone and bassoon)Each instrument is recorded individually, then mixedGround-truth pitch trajectoriesUse YIN on monophonic tracks before mixingInput pitch estimatesOur previous work in [1]Input accuracy: 70.0+-3.1% [1] Zhiyao Duan, Bryan Pardo and Changshui Zhang, “Multiple Fundamental Frequency Estimation by Modeling Spectral Peaks and Non-peak Regions”, IEEE Trans. Audio Speech Language Process., in press.16Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.eduSlide17

Overall Multi-pitch Tracking Results

Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.eduMean % of correct pitch estimatesSlide18

Among Correctly Estimated Pitches

Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.eduSlide19

An Example

Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.eduSlide20

An Example

Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.eduSlide21

ConclusionFormulate the song-level Multi-pitch Tracking problem as a constrained clustering problem

Objective: timbre consistencyConstraints: pitch localityExisting constrained clustering algorithms do not apply due to problem propertiesPropose a new constrained clustering algorithmExperimental results are promisingNorthwestern University, Interactive Audio Lab. http://music.cs.northwestern.eduSlide22

Thanks you!22

Northwestern University, Interactive Audio Lab. http://music.cs.northwestern.edu