Latent Tree Models Part II: Definition and Properties - PowerPoint Presentation

1. Latent Tree ModelsPart II: Definition and PropertiesNevin L. ZhangDept. of Computer Science & EngineeringThe Hong Kong Univ. of Sci. & Tech.http://www.cse.ust.hk/~lzhangAAAI 2014 Tutorial

2. Part II: Concept and PropertiesLatent Tree ModelsDefinitionRelationship with finite mixture modelsRelationship with phylogenetic treesBasic Properties

3. Basic Latent Tree Models (LTM)Bayesian networkAll variables are discrete Structure is a rooted treeLeaf nodes are observed (manifest variables)Internal nodes are not observed (latent variables)Parameters:P(Y1), P(Y2|Y1),P(X1|Y2), P(X2|Y2), … Semantics:Also known as Hierarchical latent class (HLC) models, HLC models (Zhang. JMLR 2004)

4. Marginalizing out the latent variables in , we get a joint distribution over the observed variables .In comparison with Bayesian network without latent variables, LTM: Is computationally very simple to work with.Represent complex relationships among manifest variables.What does the structure look like without the latent variables?Joint Distribution over Observed Variables

5. Pouch Latent Tree Models (PLTM)An extension of basic LTM Rooted treeInternal nodes represent discrete latent variablesEach leaf node consists of one or more continuous observed variable, called a pouch.(Poon et al. ICML 2010)

6. More General Latent Variable Tree ModelsSome internal nodes can be observedInternal nodes can be continuousForestPrimary focus of this tutorial: the basic LTM(Choi et al. JMLR 2011)

7. Part II: Concept and PropertiesLatent Tree ModelsDefinitionRelationship with finite mixture modelsRelationship with phylogenetic treesBasic Properties

8. Finite Mixture Models (FMM)Gaussian Mixture Models (GMM): Continuous attributes Graphical model

9. Finite Mixture Models (FMM)GMM with independence assumptionBlock diagonal co-variable matrixGraphical Model

10. Finite Mixture ModelsLatent class models (LCM): Discrete attributes Distribution for cluster k: Product multinomial distribution: All FMMsOne latent variableYielding one partition of dataGraphical Model

11. From FMMs to LTMsStart with several GMMs, Each based on a distinct subset of attributesEach partitions data from a certain perspective.Different partitions are independent of each otherLink them up to form a tree modelGet Pouch LTMConsider different perspectives in a single modelMultiple partitions of data that are correlated.

12. From FMMs to LTMsStart with several LCMs, Each based on a distinct subset of attributesEach partitions data from a certain perspective.Different partitions are independent of each otherLink them up to form a tree modelGet LTMConsider different perspectives in a single modelMultiple partitions of data that are correlated.Summary: An LTM can be viewed as a collections of FMMs, with their latent variables linked up to form a tree structure.

13. Part II: Concept and PropertiesLatent Tree ModelsDefinitionRelationship with finite mixture modelsRelationship with phylogenetic treesBasic Properties

14. Phylogenetic treesTAXA (sequences) identify speciesEdge lengths represent evolution timeUsually, bifurcating tree topologyDurbin, et al. (1998). Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids. Cambridge University Press.

15. Probabilistic Models of EvolutionTwo assumptions There are only substitutions, no insertions/deletions (aligned)One-to-one correspondence between sites in different sequencesEach site evolves independently and identicallyP(x|y, t) = Pi=1 to m P(x(i) | y(i), t)m is sequence lengthP(x(i)|y(i), t) Jukes-Cantor (Character Evolution) Model [1969]Rate of substitution a

16. Phylogenetic Trees are Special LTMsWhen focus on one site, phylogenetic trees are special latent tree modelsThe structure is a binary tree The variables share the same state space.Each conditional distribution is characterized by only one parameters, i.e., the length of the corresponding edge

17. Hidden Markov models are also special latent tree modelsAll latent variables share the same state space.All observed variables share the same state space.P(yt |st ) and P(st+1 | st ) are the same for different t ’s.Hidden Markov Models

18. Part II: Concept and Basic PropertiesLatent Tree ModelsDefinitionRelationship with finite mixture modelsRelationship with phylogenetic treesBasic Properties

19. So far, a model consists ofObserved and latent variablesConnections among the variablesProbability valuesFor the rest of Part II, a model consists ofObserved and latent variablesConnections among the variablesProbability parametersTwo Concepts of Models

20. Model Inclusion

21. If m includes m’ and vice versa, then they are marginally equivalent. If they also have the same number of free parameters, then they are equivalent.It is not possible to distinguish between equivalent models based on data.Model Equivalence

22. Root Walking

23. Root Walking Example Root walks to X2; Root walks to X3

24. Theorem: Root walking leads to equivalent latent tree models.Root Walking(Zhang, JMLR 2004)Special case of covered arc reversal in general Bayesian network, Chickering, D. M. (1995). A transformational characterization of equivalentBayesian network structures. UAI.

25. Edge orientations in latent tree models are not identifiable.Technically, better to start with alternative definition of LTM:A latent tree model (LTM) is a Markov random field over an undirected tree, or tree-structured Markov network where variables at leaf nodes are observed and variables at internal nodes are hidden.Implication

26. For technical convenience, we often root an LTM at one of its latent nodes and regard it as a directed graphical model.Rooting the model at different latent nodes lead to equivalent directed models.This is why we introduced LTM as directed models.Implication

27. Regularity|X|: Cardinality of variable X, i.e., the number of states.

28. Can focus on regular models onlyIrregular models can be made regular Regularized models better than irregular modelsTheorem: The set of all regular models for a given set of observed variables is finite.Regularity(Zhang, JMLR 2004)