ReviewconditionalindependenceTworandomvariablesandYareindependentwrittenYipXpXjYyforallyIfYthenYgivesusnoinformationaboutandYareconditionallyindependentgivenZwrittenYjZipXjZ ID: 111736
Download Pdf The PPT/PDF document "AShortCourseonGraphicalModels3.TheJuncti..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
AShortCourseonGraphicalModels3.TheJunctionTreeAlgorithmsMarkPaskinmark@paskin.org1 Review:conditionalindependenceTworandomvariablesandYareindependent(written??Y)ipX()=pXjY(;y)forallyIf??YthenYgivesusnoinformationabout.andYareconditionallyindependentgivenZ(written??YjZ)ipXjZ(;z)=pXjYZ(;y;z)forallyandzIf??YjZthenYgivesusnonewinformationaboutonceweknowZ.Wecanobtaincompact,factorizedrepresentationsofdensitiesbyusingthechainruleincombinationwithconditionalindependenceassumptions.TheVariableEliminationalgorithmusesthedistributivityofover+toperforminferenceecientlyinfactorizeddensities.2 Review:graphicalmodelsBayesiannetworkundirectedgraphicalmodelABjACjADjBEjCFjBE1 A AB AC BD CE BEF-separation!cond.indep.graphseparation!cond.indep.MoralizationconvertsaBayesiannetworkintoanundirectedgraphicalmodel(butitdoesnotpreservealloftheconditionalindependenceproperties).3 AnotationforsetsofrandomvariablesItishelpfulwhenworkingwithlarge,complexmodelstohaveagoodnotationforsetsofrandomvariables.Let=(i:i2V)beavectorrandomvariablewithdensityp.ForeachAV,letA4=(i:i2A).ForA;BV,letpA4=pXAandpAjB4=pXAjXB.Example.IfV=fa;b;cgandA=fa;cgthen=2664abc3775andA=24ac35wherea,b,andcarerandomvariables.4 AnotationforassignmentsWealsoneedanotationfordealing\rexiblywithfunctionsofmanyarguments.AnassignmenttoAisasetofindex-valuepairsu=f(i;xi):i2Ag,oneperindexi2A,wherexiisintherangeofi.LetXAbethesetofassignmentstoA(withX4=XV).Buildingnewassignmentsfromgivenassignments:{GivenassignmentsuandvtodisjointsubsetsAandB,respectively,theirunionu[visanassignmenttoA[B.{IfuisanassignmenttoAthentherestrictionofutoBVisuB4=f(i;xi)2u:i2Bg,anassignmenttoA\B.Ifu=f(i;xi):i2Agisanassignmentandisafunction,then(u)4=(xi:i2A)5 Examplesoftheassignmentnotation1.IfpisthejointdensityofthenthemarginaldensityofAispA(v)=Xu2XAp(v[u);v2XAwhereA=VnAisthecomplementofA.2.Ifptakestheformofanormalizedproductofpotentials,wecanwriteitasp(u)=1ZYC2C C(uC);u2XwhereCisasetofsubsetsofV,andeach CisapotentialfunctionthatdependsonlyuponC.TheMarkovgraphofphascliquesetC.6 Review:theinferenceproblemInput:{avectorrandomvariable=(i:i2V);{ajointdensityforoftheformp(u)=1ZYC2C C(uC){anevidenceassignmentwtoE;and{somequeryvariablesQ.Output:pQjE(;w),theconditionaldensityofQgiventheevidencew.7 DealingwithevidenceFromthedenitionofconditionalprobability,wehave:pEjE(u;w)=p(u[w)pE(w)=1ZQC2C C(uC[wC)pE(w)ForxedevidencewonE,thisisanothernormalizedproductofpotentials:pEjw(u)=1Z0YC02C0 C0(uC0)whereZ04=ZpE(w),C04=CnE,and C0(u)4= C(u[wC).Thus,todealwithevidence,wesimplyinstantiateitinallcliquepotentials.8 ThereformulatedinferenceproblemGivenajointdensityfor=(i:i2V)oftheformp(u)=1ZYC2C C(uC)computethemarginaldensityofQ:pQ(v)=Xu2XQp(v[u)=Xu2XQ1ZYC2C C(vC[uC)9 Review:VariableEliminationForeachi2,pushinthesumoveriandcomputeit:pQ(v)=1ZXu2XQYC2C C(vC[uC)=1ZXu2XQiXw2XiYC2C C(vC[uC[wC)=1ZXu2XQiYC2Ci62C C(vC[uC)Xw2XiYC2Ci2C C(vC[uC[w)=1ZXu2XQiYC2Ci62C C(vC[uC) Ei(vEi[uEi)ThiscreatesaneweliminationcliqueEi=SC2Ci2CCnfig.AttheendwehavepQ=1Z QandwenormalizetoobtainpQ(andZ).10 FromVariableEliminationtothejunctiontreealgorithmsVariableEliminationisquerysensitive:wemustspecifythequeryvariablesinadvance.Thismeanseachtimewerunadierentquery,wemustre-runtheentirealgorithm.ThejunctiontreealgorithmsgeneralizeVariableEliminationtoavoidthis;theycompilethedensityintoadatastructurethatsupportsthesimultaneousexecutionofalargeclassofqueries.11 Junctiontreesb, e, fGTAclustergraphTisajunctiontreeforGifithasthesethreeproperties:1.singlyconnected:thereisexactlyonepathbetweeneachpairofclusters.2.covering:foreachcliqueAofGthereissomeclusterCsuchthatAC.3.runningintersection:foreachpairofclustersBandCthatcontaini,eachclusterontheuniquepathbetweenBandCalsocontainsi.12 BuildingjunctiontreesTobuildajunctiontree:1.ChooseanorderingofthenodesanduseNodeEliminationtoobtainasetofeliminationcliques.2.Buildacompleteclustergraphoverthemaximaleliminationcliques.3.WeighteachedgefB;CgbyjB\Cjandcomputeamaximum-weightspanningtree.ThisspanningtreeisajunctiontreeforG(seeCowelletal.,1999).Dierentjunctiontreesareobtainedwithdierenteliminationordersanddierentmaximum-weightspanningtrees.FindingthejunctiontreewiththesmallestclustersisanNP-hardproblem.13 Anexampleofbuildingjunctiontrees1.Computetheeliminationcliques(theorderhereisf;d;e;c;b;a).cbeacbab2.Formthecompleteclustergraphoverthemaximaleliminationcliquesandndamaximum-weightspanningtree.b, e, fb, e, f14 DecomposabledensitiesAfactorizeddensityp(u)=1ZYC2C C(uC)isdecomposableifthereisajunctiontreewithclustersetC.Toconvertafactorizeddensityptoadecomposabledensity:1.BuildajunctiontreeTfortheMarkovgraphofp.2.Createapotential CforeachclusterCofTandinitializeittounity.3.Multiplyeachpotential ofpintotheclusterpotentialofoneclusterthatcoversitsvariables.Note:thisispossibleonlybecauseofthecoveringproperty.15 ThejunctiontreeinferencealgorithmsThejunctiontreealgorithmstakeasinputadecomposabledensityanditsjunctiontree.Theyhavethesamedistributedstructure:Eachclusterstartsoutknowingonlyitslocalpotentialanditsneighbors.Eachclustersendsonemessage(potentialfunction)toeachneighbor.Bycombiningitslocalpotentialwiththemessagesitreceives,eachclusterisabletocomputethemarginaldensityofitsvariables.16 ThemessagepassingprotocolThejunctiontreealgorithmsobeythemessagepassingprotocol:ClusterBisallowedtosendamessagetoaneighborConlyafterithasreceivedmessagesfromallneighborsexceptC.OneadmissiblescheduleisobtainedbychoosingoneclusterRtobetheroot,sothejunctiontreeisdirected.ExecuteCollect(R)andthenDistribute(R):1.Collect(C):ForeachchildBofC,recursivelycallCollect(B)andthenpassamessagefromBtoC.2.Distribute(C):ForeachchildBofC,passamessagetoBandthenrecursivelycallDistribute(B).ISTRIBUTE17 TheShafer{ShenoyAlgorithmThemessagesentfromBtoCisdenedasBC(u)4=Xv2XBC B(u[v)Y(A;B)2EA=CAB(uA[vA)Procedurally,clusterBcomputestheproductofitslocalpotential BandthemessagesfromallclustersexceptC,marginalizesoutallvariablesthatarenotinC,andthensendstheresulttoC.Note:BCiswell-denedbecausethejunctiontreeissinglyconnected.TheclusterbeliefatCisdenedasC(u)4= C(u)Y(B;C)2EBC(uB)Thisistheproductofthecluster'slocalpotentialandthemessagesreceivedfromallofitsneighbors.WewillshowthatC/pC.18 Correctness:Shafer{ShenoyisVariableEliminationinalldirectionsatonceTheclusterbeliefCiscomputedbyalternatinglymultiplyingclusterpotentialstogetherandsummingoutvariables.ThiscomputationisofthesamebasicformasVariableElimination.ToprovethatC/pC,wemustprovethatnosumis\pushedintoofar".Thisfollowsdirectlyfromtherunningintersectionproperty:Cclusters containing ii is summed out whencomputing this messagemessages with imessages without ithe running intersection propertyguarantees the clusters containing iconstitute a connected subgraph19 ThehuginAlgorithmGiveeachclusterCandeachseparatorSapotentialfunctionoveritsvariables.Initialize:C(u)= C(u)S(u)=1TopassamessagefromBtoCoverseparatorS,updateS(u)=Xv2XBSB(u[v)C(u)=C(u)S(uS)S(uS)Afterallmessageshavebeenpassed,C/pCforallclustersC.20 Correctness:huginisatime-ecientversionofShafer{ShenoyEachtimetheShafer{Shenoyalgorithmsendsamessageorcomputesitsclusterbelief,itmultipliestogethermessages.Toavoidperformingthesemultiplicationsrepeatedly,thehuginalgorithmcachesinCtherunningproductof Candthemessagesreceivedsofar.WhenBsendsamessagetoC,itdividesoutthemessageCsenttoBfromthisrunningproduct.21 Summary:thejunctiontreealgorithmsCompiletime:1.BuildthejunctiontreeT:(a)ObtainasetofmaximaleliminationcliqueswithNodeElimination.(b)Buildaweighted,completeclustergraphoverthesecliques.(c)ChooseTtobeamaximum-weightspanningtree.2.MakethedensitydecomposablewithrespecttoT.Runtime:1.Instantiateevidenceinthepotentialsofthedensity.2.Passmessagesaccordingtothemessagepassingprotocol.3.Normalizetheclusterbeliefs/potentialstoobtainconditionaldensities.22 ComplexityofjunctiontreealgorithmsJunctiontreealgorithmsrepresent,multiply,andmarginalizepotentials:tabularGaussianstoring CO(kjCj)O(jCj2)computing B[C= B CO(kjB[Cj)O(jB[Cj2)computing CnB(u)=Pv2XB C(u[v)O(kjCj)O(jBj3jCj2)ThenumberofclustersinajunctiontreeandthereforethenumberofmessagescomputedisO(jVj).Thus,thetimeandspacecomplexityisdominatedbythesizeofthelargestclusterinthejunctiontree,orthewidthofthejunctiontree:{Intabulardensities,thecomplexityisexponentialinthewidth.{InGaussiandensities,thecomplexityiscubicinthewidth.23 GeneralizedDistributiveLawThegeneralproblemsolvedbythejunctiontreealgorithmsisthesum-of-productsproblem:computepQ(v)/Xu2XQYC2C C(vC[uC)Thepropertyusedbythejunctiontreealgorithmsisthedistributivityofover+;moregenerally,weneedacommutativesemiring:[0;1)(+;0)(;1)sum-product[0;1)(max;0)(;1)max-product( 1;1](min;1)(+;0)min-sumfT;Fg(_;F)(^;T)BooleanManyotherproblemsareofthisform,includingmaximumaposterioriinference,theHadamardtransform,andmatrixchainmultiplication.24 SummaryThejunctiontreealgorithmsgeneralizeVariableEliminationtotheecient,simultaneousexecutionofalargeclassofqueries.Thealgorithmstaketheformofmessagepassingonagraphcalledajunctiontree,whosenodesareclusters,orsets,ofvariables.Eachclusterstartswithonepotentialofthefactorizeddensity.Bycombiningthispotentialwiththepotentialsitreceivesfromitsneighbors,itcancomputethemarginaloveritsvariables.TwojunctiontreealgorithmsaretheShafer{Shenoyalgorithmandthehuginalgorithm,whichavoidsrepeatedmultiplications.Thecomplexityofthealgorithmsscaleswiththewidthofthejunctiontree.Thealgorithmscanbegeneralizedtosolveotherproblemsbyusingothercommutativesemirings.25