ReviewconditionalindependenceTworandomvariablesandYareindependentwrittenYi pXpXjYyforallyIfYthenYgivesusnoinformationaboutandYareconditionallyindependentgivenZwrittenYjZi pXjZ ID: 111736
Download Pdf The PPT/PDF document "AShortCourseonGraphicalModels3.TheJuncti..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
AShortCourseonGraphicalModels3.TheJunctionTreeAlgorithmsMarkPaskinmark@paskin.org1 Review:conditionalindependenceTworandomvariablesandYareindependent(written??Y)i pX()=pXjY(;y)forallyIf??YthenYgivesusnoinformationabout.andYareconditionallyindependentgivenZ(written??YjZ)i pXjZ(;z)=pXjYZ(;y;z)forallyandzIf??YjZthenYgivesusnonewinformationaboutonceweknowZ.Wecanobtaincompact,factorizedrepresentationsofdensitiesbyusingthechainruleincombinationwithconditionalindependenceassumptions.TheVariableEliminationalgorithmusesthedistributivityofover+toperforminferenceecientlyinfactorizeddensities.2 Review:graphicalmodelsBayesiannetworkundirectedgraphicalmodelABjACjADjBEjCFjBE1 A AB AC BD CE BEF-separation!cond.indep.graphseparation!cond.indep.MoralizationconvertsaBayesiannetworkintoanundirectedgraphicalmodel(butitdoesnotpreservealloftheconditionalindependenceproperties).3 AnotationforsetsofrandomvariablesItishelpfulwhenworkingwithlarge,complexmodelstohaveagoodnotationforsetsofrandomvariables.Let=(i:i2V)beavectorrandomvariablewithdensityp.ForeachAV,letA4=(i:i2A).ForA;BV,letpA4=pXAandpAjB4=pXAjXB.Example.IfV=fa;b;cgandA=fa;cgthen=2664abc3775andA=24ac35wherea,b,andcarerandomvariables.4 AnotationforassignmentsWealsoneedanotationfordealing\rexiblywithfunctionsofmanyarguments.AnassignmenttoAisasetofindex-valuepairsu=f(i;xi):i2Ag,oneperindexi2A,wherexiisintherangeofi.LetXAbethesetofassignmentstoA(withX4=XV).Buildingnewassignmentsfromgivenassignments:{GivenassignmentsuandvtodisjointsubsetsAandB,respectively,theirunionu[visanassignmenttoA[B.{IfuisanassignmenttoAthentherestrictionofutoBVisuB4=f(i;xi)2u:i2Bg,anassignmenttoA\B.Ifu=f(i;xi):i2Agisanassignmentandisafunction,then(u)4=(xi:i2A)5 Examplesoftheassignmentnotation1.IfpisthejointdensityofthenthemarginaldensityofAispA(v)=Xu2XAp(v[u);v2XAwhereA=VnAisthecomplementofA.2.Ifptakestheformofanormalizedproductofpotentials,wecanwriteitasp(u)=1ZYC2C C(uC);u2XwhereCisasetofsubsetsofV,andeach CisapotentialfunctionthatdependsonlyuponC.TheMarkovgraphofphascliquesetC.6 Review:theinferenceproblemInput:{avectorrandomvariable=(i:i2V);{ajointdensityforoftheformp(u)=1ZYC2C C(uC){anevidenceassignmentwtoE;and{somequeryvariablesQ.Output:pQjE(;w),theconditionaldensityofQgiventheevidencew.7 DealingwithevidenceFromthede nitionofconditionalprobability,wehave:pEjE(u;w)=p(u[w)pE(w)=1ZQC2C C(uC[wC)pE(w)For xedevidencewonE,thisisanothernormalizedproductofpotentials:pEjw(u)=1Z0YC02C0 C0(uC0)whereZ04=ZpE(w),C04=CnE,and C0(u)4= C(u[wC).Thus,todealwithevidence,wesimplyinstantiateitinallcliquepotentials.8 ThereformulatedinferenceproblemGivenajointdensityfor=(i:i2V)oftheformp(u)=1ZYC2C C(uC)computethemarginaldensityofQ:pQ(v)=Xu2XQp(v[u)=Xu2XQ1ZYC2C C(vC[uC)9 Review:VariableEliminationForeachi2,pushinthesumoveriandcomputeit:pQ(v)=1ZXu2XQYC2C C(vC[uC)=1ZXu2XQiXw2XiYC2C C(vC[uC[wC)=1ZXu2XQiYC2Ci62C C(vC[uC)Xw2XiYC2Ci2C C(vC[uC[w)=1ZXu2XQiYC2Ci62C C(vC[uC) Ei(vEi[uEi)ThiscreatesaneweliminationcliqueEi=SC2Ci2CCnfig.AttheendwehavepQ=1Z QandwenormalizetoobtainpQ(andZ).10 FromVariableEliminationtothejunctiontreealgorithmsVariableEliminationisquerysensitive:wemustspecifythequeryvariablesinadvance.Thismeanseachtimewerunadi erentquery,wemustre-runtheentirealgorithm.ThejunctiontreealgorithmsgeneralizeVariableEliminationtoavoidthis;theycompilethedensityintoadatastructurethatsupportsthesimultaneousexecutionofalargeclassofqueries.11 Junctiontreesb, e, fGTAclustergraphTisajunctiontreeforGifithasthesethreeproperties:1.singlyconnected:thereisexactlyonepathbetweeneachpairofclusters.2.covering:foreachcliqueAofGthereissomeclusterCsuchthatAC.3.runningintersection:foreachpairofclustersBandCthatcontaini,eachclusterontheuniquepathbetweenBandCalsocontainsi.12 BuildingjunctiontreesTobuildajunctiontree:1.ChooseanorderingofthenodesanduseNodeEliminationtoobtainasetofeliminationcliques.2.Buildacompleteclustergraphoverthemaximaleliminationcliques.3.WeighteachedgefB;CgbyjB\Cjandcomputeamaximum-weightspanningtree.ThisspanningtreeisajunctiontreeforG(seeCowelletal.,1999).Di erentjunctiontreesareobtainedwithdi erenteliminationordersanddi erentmaximum-weightspanningtrees.FindingthejunctiontreewiththesmallestclustersisanNP-hardproblem.13 Anexampleofbuildingjunctiontrees1.Computetheeliminationcliques(theorderhereisf;d;e;c;b;a).cbeacbab2.Formthecompleteclustergraphoverthemaximaleliminationcliquesand ndamaximum-weightspanningtree.b, e, fb, e, f14 DecomposabledensitiesAfactorizeddensityp(u)=1ZYC2C C(uC)isdecomposableifthereisajunctiontreewithclustersetC.Toconvertafactorizeddensityptoadecomposabledensity:1.BuildajunctiontreeTfortheMarkovgraphofp.2.Createapotential CforeachclusterCofTandinitializeittounity.3.Multiplyeachpotential ofpintotheclusterpotentialofoneclusterthatcoversitsvariables.Note:thisispossibleonlybecauseofthecoveringproperty.15 ThejunctiontreeinferencealgorithmsThejunctiontreealgorithmstakeasinputadecomposabledensityanditsjunctiontree.Theyhavethesamedistributedstructure:Eachclusterstartsoutknowingonlyitslocalpotentialanditsneighbors.Eachclustersendsonemessage(potentialfunction)toeachneighbor.Bycombiningitslocalpotentialwiththemessagesitreceives,eachclusterisabletocomputethemarginaldensityofitsvariables.16 ThemessagepassingprotocolThejunctiontreealgorithmsobeythemessagepassingprotocol:ClusterBisallowedtosendamessagetoaneighborConlyafterithasreceivedmessagesfromallneighborsexceptC.OneadmissiblescheduleisobtainedbychoosingoneclusterRtobetheroot,sothejunctiontreeisdirected.ExecuteCollect(R)andthenDistribute(R):1.Collect(C):ForeachchildBofC,recursivelycallCollect(B)andthenpassamessagefromBtoC.2.Distribute(C):ForeachchildBofC,passamessagetoBandthenrecursivelycallDistribute(B).ISTRIBUTE17 TheShafer{ShenoyAlgorithmThemessagesentfromBtoCisde nedasBC(u)4=Xv2XBC B(u[v)Y(A;B)2EA=CAB(uA[vA)Procedurally,clusterBcomputestheproductofitslocalpotential BandthemessagesfromallclustersexceptC,marginalizesoutallvariablesthatarenotinC,andthensendstheresulttoC.Note:BCiswell-de nedbecausethejunctiontreeissinglyconnected.TheclusterbeliefatCisde nedas C(u)4= C(u)Y(B;C)2EBC(uB)Thisistheproductofthecluster'slocalpotentialandthemessagesreceivedfromallofitsneighbors.Wewillshowthat C/pC.18 Correctness:Shafer{ShenoyisVariableEliminationinalldirectionsatonceTheclusterbelief Ciscomputedbyalternatinglymultiplyingclusterpotentialstogetherandsummingoutvariables.ThiscomputationisofthesamebasicformasVariableElimination.Toprovethat C/pC,wemustprovethatnosumis\pushedintoofar".Thisfollowsdirectlyfromtherunningintersectionproperty:Cclusters containing ii is summed out whencomputing this messagemessages with imessages without ithe running intersection propertyguarantees the clusters containing iconstitute a connected subgraph19 ThehuginAlgorithmGiveeachclusterCandeachseparatorSapotentialfunctionoveritsvariables.Initialize: C(u)= C(u) S(u)=1TopassamessagefromBtoCoverseparatorS,update S(u)=Xv2XBS B(u[v) C(u)= C(u) S(uS) S(uS)Afterallmessageshavebeenpassed, C/pCforallclustersC.20 Correctness:huginisatime-ecientversionofShafer{ShenoyEachtimetheShafer{Shenoyalgorithmsendsamessageorcomputesitsclusterbelief,itmultipliestogethermessages.Toavoidperformingthesemultiplicationsrepeatedly,thehuginalgorithmcachesin Ctherunningproductof Candthemessagesreceivedsofar.WhenBsendsamessagetoC,itdividesoutthemessageCsenttoBfromthisrunningproduct.21 Summary:thejunctiontreealgorithmsCompiletime:1.BuildthejunctiontreeT:(a)ObtainasetofmaximaleliminationcliqueswithNodeElimination.(b)Buildaweighted,completeclustergraphoverthesecliques.(c)ChooseTtobeamaximum-weightspanningtree.2.MakethedensitydecomposablewithrespecttoT.Runtime:1.Instantiateevidenceinthepotentialsofthedensity.2.Passmessagesaccordingtothemessagepassingprotocol.3.Normalizetheclusterbeliefs/potentialstoobtainconditionaldensities.22 ComplexityofjunctiontreealgorithmsJunctiontreealgorithmsrepresent,multiply,andmarginalizepotentials:tabularGaussianstoring CO(kjCj)O(jCj2)computing B[C= B CO(kjB[Cj)O(jB[Cj2)computing CnB(u)=Pv2XB C(u[v)O(kjCj)O(jBj3jCj2)ThenumberofclustersinajunctiontreeandthereforethenumberofmessagescomputedisO(jVj).Thus,thetimeandspacecomplexityisdominatedbythesizeofthelargestclusterinthejunctiontree,orthewidthofthejunctiontree:{Intabulardensities,thecomplexityisexponentialinthewidth.{InGaussiandensities,thecomplexityiscubicinthewidth.23 GeneralizedDistributiveLawThegeneralproblemsolvedbythejunctiontreealgorithmsisthesum-of-productsproblem:computepQ(v)/Xu2XQYC2C C(vC[uC)Thepropertyusedbythejunctiontreealgorithmsisthedistributivityofover+;moregenerally,weneedacommutativesemiring:[0;1)(+;0)(;1)sum-product[0;1)(max;0)(;1)max-product( 1;1](min;1)(+;0)min-sumfT;Fg(_;F)(^;T)BooleanManyotherproblemsareofthisform,includingmaximumaposterioriinference,theHadamardtransform,andmatrixchainmultiplication.24 SummaryThejunctiontreealgorithmsgeneralizeVariableEliminationtotheecient,simultaneousexecutionofalargeclassofqueries.Thealgorithmstaketheformofmessagepassingonagraphcalledajunctiontree,whosenodesareclusters,orsets,ofvariables.Eachclusterstartswithonepotentialofthefactorizeddensity.Bycombiningthispotentialwiththepotentialsitreceivesfromitsneighbors,itcancomputethemarginaloveritsvariables.TwojunctiontreealgorithmsaretheShafer{Shenoyalgorithmandthehuginalgorithm,whichavoidsrepeatedmultiplications.Thecomplexityofthealgorithmsscaleswiththewidthofthejunctiontree.Thealgorithmscanbegeneralizedtosolveotherproblemsbyusingothercommutativesemirings.25