Harmonic Shape Analysis PowerPoint Presentation

Harmonic Shape Analysisfrom Fourier to Wavelets

Ming

Zhong

2012.9

Overview (1)

Harmonic analysis basicsRepresent signals as the linear combination of basic overlapping, wave-like functionsNatural domain (space/time)frequency domainWide applicationssignal analysis/processingaudio, images, radio waves, seismic waves …smoothing, enhancing, equalization, denoising …reconstruction, approximation, compression … solving PDEsnumerical computation (FFT)Fourier analysisWavelet analysis

frequency

coefficients

edited coefficients

s

ignals

processed signals

space/time

transform / analysis / decomposition

inverse transform / synthesis / reconstruction

Harmonic analysis pipeline

Overview (2)

Motivations

for

extending harmonic

analysis

to 3D shapes

Signal processing on shapes

v

ertex coordinates, normals, texture seen as functions on surface

Deformable shape analysis

Manifold Fourier bases are “shape aware”

shape

classification, symmetry detection, segmentation, matching,

etc

Diffusion geometry representation

Presentation Outline

Manifold Fourier

A

nalysis : Theories

Manifold Fourier

A

nalysis : Applications

Spectral Geometry

P

rocessing

Laplacian-based shape analysis

Diffusion-based shape analysis

Manifold Wavelets

Preliminary Work in Deformable

S

hape

A

nalysis

High-order shape matching

Hierarchical shape registration

Bag-of-Feature-Graph shape retrieval

Conclusion

Manifold Fourier Analysis: Theoriesclassical Fourier analysis (1)

: space of periodic square-integrable functions on real lineBounded functions: Inner product: Fourier basis: Global supportOscillatingDistinct frequenciesOrthonormal basis of Ahas unique decomposition on

Fourier basis functions of different frequencies

Manifold Fourier Analysis: Theoriesclassical Fourier analysis (2)

Fourier transformTransform from to Fourier space-th Fourier coefficient:Inner product in Fourier spaceReconstruction (Fourier series)Parseval’s equation Preservation of energy

Approximating functions using Fourier series

Manifold Fourier Analysis: Theories manifold Fourier analysis (1)

Space of real valued functions defined on a 3D shapeSurface viewed as a compact Riemannian manifold with metric is a closed manifold or a manifold with Dirichlet boundaryInner product of What is the Fourier basis of ?

Origins of classical Fourier basis

Helmholtz equation on real line: 1D Laplace operator: -th eigenfunction of Laplacian : -th eigenvalue, square of the frequency

Origins of manifold Fourier basis

Helmholtz equation on manifold : Laplace-Beltrami operator on : -th eigenfunction of , manifold harmonic basis: -th eigenvalue, square of shape frequency

Manifold Fourier Analysis: Theories

manifold Fourier analysis (2)

Manifold Fourier Analysis: Theories manifold Fourier analysis (3)

Laplace-Beltrami operator Self-adjointsemi-positive definiteAdmitting an orthonormal eigensystem by solving the Dirichlet problemEigenvalues constitute a real non-negative sequenceEigenfunctions form an orthonormal basis of

Visualized manifold harmonic basis functions of different frequencies(value increases from blue to red)

Manifold Fourier Analysis: Theories manifold Fourier analysis (4)

Manifold Fourier transform (manifold harmonic transform)Scalar function defined on manifold ()Manifold Fourier coefficientManifold Fourier series (reconstruction)

Manifold Fourier Analysis: Theories classical and manifold Fourier comparisons

classical

Fourier analysisManifold Fourier analysisdomaininner productLaplacian equationfrequencyFourier basis (Fourier transformFourier series

classical

Fourier analysis

Manifold Fourier analysis

domain

inner product

Laplacian equation

frequency

Fourier transform

Fourier series

Manifold Fourier Analysis: Theories discrete settings (1)

Discrete Laplace-Beltrami operatorDiscrete mesh Cotangent formula [Belkin09]Area matrix , is the Voronoi cell size around point Weight matrix Laplacian matrix On general meshes, . So is not symmetric.

Manifold Fourier Analysis: Theories discrete settings (2)

Generalized eigenvalue problemEigenvectors: Orthogonal w.r.t. -inner productIf , -inner product becomes the standard dot product of vectors.Fourier decomposition of

Presentation Outline

Manifold Fourier

A

nalysis : Theories

Manifold Fourier

A

nalysis : Applications

Spectral Geometry

P

rocessing

Laplacian-based shape analysis

Diffusion-based shape analysis

Manifold Wavelets

Preliminary Work in Deformable

S

hape

A

nalysis

High-order shape matching

Hierarchical shape registration

Bag-of-Feature-Graph shape retrieval

Conclusion

Manifold Fourier Analysis : Applicationsspectral geometry processing (1)

Manifold Fourier analysis on the mesh geometryCartesian coordinates of vertices Manifold Fourier transform of coordinatesReconstructionLevel of detail control

Horse model reconstructed using indicated number of eigenvectors.[Zhang et al., 2010]

Manifold Fourier Analysis : Applicationsspectral geometry processing (1)

Mesh Filtering: Frequency space filter/transfer functionMesh smoothing / exaggeration [Vallet08]Pose transfer / detail coating / shape blending [Vallet08, Lipman04]Other applicationsMesh compression [Karni00, Sorkine03]Mesh watermarking [Ohbuchi01]

Effect of mesh filters [Vallet and Lévy, 2008]

From left to right: original shape; smoothing; detail enhancing; band exaggeration.

Pose transfer filters [Vallet and Lévy, 2008]

(C) Is generated by copying the first 5 Fourier coefficients of (A) to (B)

Geometry detail coating [Sorkine 2005]

Manifold Fourier Analysis : ApplicationsLaplacian-based shape analysis

The Laplacian eigenfunctions and eigenvalues areIntrinsic to shape geometryInvariant to rigid deformations and isometric deformationsGlobally shape awareStableLaplacian embeddingLaplacian-spectra (eigenvalues of the Laplace-Beltrami operator)Global Point Signature Shape analysis applicationsShape classification [Reuter06, Rustamov07]Shape segmentation [Reuter09, Liu07,Goes08]Intrinsic symmetry detection [Ovsjanikov08]Mesh sequencing [Isenburg01]

An eigenfunction of the Laplace-Beltrami operator computed on different deformations, showing isomteric invariance

Shape classification using the Laplacian eigenvalues. Plot by 2D MDS. [Reuter et al., 2006]

Intrinsic symmetry detection using Global Point Signature embedding [Ovsjanikov et at., 2008]

Shape segmentation using nodal sets. The nodal sets are the zero sets of Laplacian-eigenfunctions.[Reuter et al., 2009]

Diffusion-Based Shape Analysisheat diffusion and Fourier series (1)

Heat diffusion in a rod of length Solution represented by Fourier series

Diffusion-Based Shape Analysisheat diffusion and Fourier series (2)

Heat diffusion on manifold Heat kernel fundamental solution to heat equationImpulse response Both heat kernel and general solution are represented by manifold Fourier series

Diffusion-Based Shape Analysisheat kernel (1)

Physical interpretation of heat kernel :Heat distribution at time and point given that an initial unit of heat energy is placed at point at Transition density function of Brownian motion on the manifoldsmall t: fast decay of in space, slow decay in frequencylarge t: slow decay of in space, fast decay in frequencyHeat kernel embeddingHeat kernel signatureHeat diffusion distanceHeat Kernel Mapping (Single anchor, multiple timescales)Heat Kernel Coordinates (Multiple anchors, single timescale)

Diffusion-Based Shape Analysisheat kernel (2)

Why use heat kernel for deformable shape analysis?IntrinsicInvariant to rigid and isometric deformations.InformativeGlobally shape-awareDetermines a shape up to isometry.Multi-scaleFor small , only reflects local properties of the shape around . For large values of , carry global structure of the shape from the point of view of Heat kernel enables more flexible analysis by controlling the timescales.Local objectsContains both local and global information. Feasible for analyzing partial deformable shape.RobustRepresented as the aggregate of eigenfunctions.Insensitive to perturbations, including topological change.

Diffusion-Based Shape Analysisheat kernel (3)

Heat Kernel applications

Feature detection [Sun09

,

Gebal09]

Shape

matching [

Ovsjanikov10]

Shape retrieval [Bronstein11]

Normalized HKS of the 4 marked points[Sun et al., 2009]

Feature detection and classification by HKS[Sun et al., 2009]

Two shapes have very different eigenfunctions (Left),while their HKSs (Right) are very close, esp. when t is small

Diffusion-Based Shape Analysisdiffusion kernels (1)

Heat kernel can be generalized to general diffusion kernelsDiffusion kernel function should satisfySymmetry: Positive preserving: Positive-semidefiniteness: Square integrability: Conservation: can be interpreted as transition probability from x to y

Diffusion-Based Shape Analysisdiffusion kernels (2)

Any diffusion kernel on can be represented by the manifold harmonics characterized by the real valued transfer function satisfiesPositive semi-definiteness: Square integrability: Conservation:

Diffusion-Based Shape Analysisdiffusion kernels (3)

Multi-scaleDefine transfer function as Diffusion kernel has the scale parameter Scale parameter can be interpreted as diffusion time or length of random walkDiffusion distanceDiffusion kernel signature

Diffusion-Based Shape Analysisdiffusion kernels (4)

Problem with heat kernel High frequency information (corresponding to with large ’s) are always suppressedUndesirable for high-precision analysis where details matter.Wave Kernel SignaturePhysical interpretation: average probability that a particle of energy is at point Energy distributionSignature vector is characterized by a collection of band-pass filters; large energies (highly oscillatory particles) more influenced by local geometry; small energies more influenced by global geometry.

Comparison of scaled-HKS and WKS of two different points[Aubry et al., 2011]

Diffusion-based Shape Analysisdiffusion kernels (5)

Learning diffusion kernelTrain the optimal transfer function based on specific dataset and applications

Transfer functions of heat kernel

Transfer functions of wave kernel

Transfer functions of a learned diffusion

kerne

Presentation Outline

Manifold Fourier

A

nalysis : Theories

Manifold Fourier

A

nalysis : Applications

Spectral Geometry

P

rocessing

Laplacian-based shape analysis

Diffusion-based shape analysis

Manifold Wavelets

Preliminary Work in Deformable

S

hape

A

nalysis

High-order shape matching

Hierarchical shape registration

Bag-of-Feature-Graph shape retrieval

Conclusion

Wave-like oscillationStart and end at zero; local supportA family of wavelet are generated through scaling and translation from a single mother wavelet

Manifold

Waveletswavelet basics

Manifold Waveletscomparison of Fourier basis and wavelets

Fourier basis functions

wavelet functionsLocalized in frequency but have global support in space/timeLocalized both in frequency and space/timeEach Fourier harmonic has an exclusive frequencyMultiple wavelets localized at different locations represent the information of a single frequency. Generally, we use fewer wavelets at lower-frequency side and more wavelets at higher-frequency side.A family of Fourier harmonics are orthogonal to each other, forming a complete orthonormal basisA family of wavelet functions form a frame, but are not necessarily orthogonal

Fourier basis functions

wavelet functions

Localized

in frequency but have global support in space/time

Localized

both in frequency and space/time

Multiple wavelets localized at different locations represent the information of a single frequency.

Generally,

we use fewer wavelets at

lower-frequency

side

and

more wavelets at higher-frequency

side.

A family of Fourier

harmonics are orthogonal to each other, forming

a complete

orthonormal basis

A

family of wavelet

functions form a frame, but

are not necessarily

orthogonal

Distributions of Fourier basis and wavelet functions in space-frequency

Manifold Waveletsclassical wavelet analysis

Mother wavelet: Square integrable: Scaled by and translated by Continuous wavelet transform (CWT)Admissibility conditionInverse CWT

Manifold Waveletssubdivision wavelets [Lounsbery et al., 1997]

Based on subdivision connectivityApplied in surface and surface data approximation, hierarchical editing

Approximating color as a function over the

sphere using subdivision wavelets. [

Lounsbery

et al., 1997]

Manifold Waveletsdiffusion wavelets [Coifman and Maggioni, 2006]

Dyadic powers of a diffusion operator to expand nested subspaceLifting schemeBottom-up fashion Slow computation

Manifold Waveletsspectral manifold wavelets

Motivation: Scaling of mother wavelet cannot be explicitly expressed on manifold domain, but it can be easily defined on the frequency domainManifold: Laplace-Beltrami operator: Manifold Fourier transform

Manifold Waveletsspectral manifold wavelets

Operator valued function Fourier transform of function generator function/transfer functionFourier transform of operator

Manifold Waveletsspectral manifold wavelets

Set Dilate by scaling Fourier coefficientsScale by and localized at Bivariate kernel Wavelet functions Fourier transform

Manifold WaveletsSpectral Manifold Wavelets

Wavelet TransformInverse TransformAdmissibility condition

Manifold Waveletsspectral manifold wavelets

Examples: Mexican hat waveletTransfer function: Wavelet functions: For a larger scale, the Mexican hat wavelet has a wider windows in space, but a narrower window in frequency.

1D Mexican hat

Manifold Mexican hat wavelets in frequency domain

Presentation Outline

Manifold Fourier

A

nalysis : Theories

Manifold Fourier

A

nalysis : Applications

Spectral Geometry

P

rocessing

Laplacian-based shape analysis

Diffusion-based shape analysis

Manifold Wavelets

Preliminary Work in Deformable

S

hape

A

nalysis

High-order shape matching

Hierarchical shape registration

Bag-of-Feature-Graph shape retrieval

Conclusion

High-order Shape Matchingoverview

Partial deformable feature matchingDiffusion-driven distanceTensor-based high order graph matchingTwo-level matching hierarchy

T.

Hou

,

M.

Zhong

, H. Qin, "Diffusion-Driven High-Order Matching of Partial Deformable Shapes", ICPR’12 (accepted)

High-order Shape MatchingDiffusion-driven metric

On manifold , the diffusion distance between point and is given by Heat kernel When , converges to the geodesic between and

High-order Shape Matchingheat kernel tensor (1)

Two overlapping partial shapes (size ) and (size )A pair denotes a candidate match with and Assignment matrix Consider a tuple of three candidate matches for improved geometric compatibilityIf both and form triangles through diffusion distance Embed three-point tuple into a space by the three angles of the triangle the tuple formsDistance between and Affinity of

High-order Shape Matchingheat kernel tensor (2)

High-order score of assignment if matched to Heat kernel tensor: Tensor power iteration [Duchenne et al., 2009]

HKT for shape matching.

High-order Shape Matching2-level matching hierarchy

Segment shapes using low-frequency eigenfunctionsCluster subgraph centered at the local extreama of low-frequency eigenfunctionsFirst match the cluster centers, then match each subgraphMuch improved time performance

Matching results with noise (left) and holes (right)

Matching features on similar objects

Hierarchical Shape Registrationpipeline

Find and match initial features.

Construct multi-resolution structures of the two shapes in question.Register points in the coarsest level using the matched features as seeds and anchors points for local parameterization.Select additional features from newly registered points to expand the set of matched features.Map registered points to the higher level.Register points in the current level with the help of the expanded set of matched features

M.

Zhong

,

T.

Hou

,

H.

Qin, "A Hierarchical Approach to High-Quality Partial

Shape Registration

",

ICPR’12 (accepted)

(a) Initial feature correspondences; (b) Coarse registration result (Third level); (c) Expanded feature correspondences (Third level); (d) Final registration result.

Hierarchical Shape Registrationkey points

Feature driven. Features are used for

Seeding points for dense registration

Anchor points for local parameterization

Diffusion embedding

Use heat kernel coordinates to compare distances when searching for best matching candidates

Progressive registration

Registrations are propagated from already registered points to their neighbors, achieving great time performance and geometric compatibility

Hierarchical

Coarse-to-fine registration

Feature sets are augmented in each level for more accurate HKC embedding

Comparison of multi-resolution (Left) and single-resolution (Right) shape registrations

Bag-of-Feature-Graph Shape Retrievaloverview

Inspired by Bag-of-Words (BoW) and Spatial-Sensitive Bag-of-Words (SS-BoW)Feature-based representationExplicitly encode both spatial information and geometric word distribution among featuresGreat time performance

T.

Hou

,

X.

Hou

,

M.

Zhong

,

H.

Qin, "Bag-of-Feature-Graphs: A New Paradigm

for Non-rigid

Shape Retrieval

", ICPR’02

Bag-of-Feature-Graph Shape Retrievalformulation(1)

HKS point descriptor A vector of HKS sampled at different values of tGeometric word Representative HKS vector clustered in the HKS descriptor space by k-means algorithmVocabulary Similarity of point and word

Bag-of-Feature-Graph Shape Retrieval

formulation(2)

Feature set F of shape MFeature graph associated with the i-th geometric wordK(t,x,y): heat kernelBag-of-Feature-Graph representation of M

Bag-of-Feature-Graph Shape Retrievalformulation (3)

Bag-of-Feature-Graph descriptorChoose the sixe largest eigenvalues of each graph matrix, denoted by vector Shape distanceRetrieval by approximate nearest neighbor (ANN) search

Non-rigid shapes and their

BoFG

descriptors

Conclusion

Manifold Fourier analysis

Spectral geometry processing

Laplacian-based shape analysis

Heat diffusion on manifold and diffusion based shape analysis

Manifold wavelets

Preliminary research on deformable shape analysis

Shape matching

Shape registration

Shape retrieval

Future work

Shape correspondenceIncorporating the similarity of point descriptors when evaluating candidates for best matchesExperiment new diffusion kernels for more accurate local parameterizationHeat kernel not accurate enough for high-precision, small scale analysisApplications such as space-time model completion

Future work

Spectral manifold waveletEfficient computation of reconstructionWavelets design via the transfer function Multi-scale feature detectionShape approximatingAntistrophic mesh filtering

Thank you!

Questions?

Presentation Outline

Manifold Fourier

A

nalysis : Theories

Manifold Fourier

A

nalysis : Applications

Spectral Geometry

P

rocessing

Laplacian-based shape analysis

Diffusion-based shape analysis

Manifold Wavelets

Preliminary Work in Deformable

S

hape

A

nalysis

High-order shape matching

Hierarchical shape registration

Bag-of-Feature-Graph shape retrieval

Conclusion