Vp TConsiderafeaturefspeciedbyhowmuchitappliestoeachconceptAfgivestheweightedsumoftheconceptsthathavethatfeaturecspeciedbyhowmucheachfeatureappliestoallthoseconceptsTheSVDallowsustowr ID: 164305
Download Pdf The PPT/PDF document "whatgraphconnectionsarerequired?whataret..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
whatgraphconnectionsarerequired?whataretheproperin-ferencerules?).Again,a“scruffy”approachtradesprecisionforbreadthandrobustness.Blending(Havasietal.2009)extendsthefactoredin-ferenceofAnalogySpacetodatafrommultipledomains:commonsenseconcepts,domain-specializedconcepts(andfeatures),andevennon-linguisticentities.Ithasbeenshowntobeusefulformanyapplications,includingopinionmin-ing(Speeretal.2010),wordsensedisambiguation(Havasi2009),andgoal-orientedcodesearch(ArnoldandLieberman2010).Wheredatafrommultipledomainsreferstothesamesetofentities(conceptsorfeatures),therepresentationofthoseentitiesaccountsforeachdataset.Sowhenthefactoredrepresentationsareusedtomakeinferences,thoseinferencesareinuencedbyalltheinputdatasets.HowtoFactorOneorMoreMatricesSingularValueDecompositionOnecommonprocessforfactoringthematrixofassertionsAisthesingularvaluedecomposition(SVD),whichexpressesAintermsofitssingularvaluesandsingularvectors.IfA~v=~uandAT~u=~v,forunitvectors~iand~j,thenisasingularvalueofAand~uand~varethecorrespondingsingularvectors.Wecansolveforthesingularvectorsbysubstitution:tosolvefor~u,multiplyontheleftbyA:AAT~u=A~v=A~u=2~u:Similarly,ATA~v=2~v.Soweseethat~uisaneigen-vectorofAAT,withaneigenvalueof2.Arrangethesingularvectors~ui,~viinthecolumnsofmatricesU=[~u1j~u2jj~un]andV=[~v1j~v2jj~vn],andarrangethesingularvaluesialongthediagonalof.Orderthembydecreasingandstopafterksingularvectors.NowwecanwritethesingularvaluedecompositionofA:AUVT.(Wecanseethatthisdecompositionisconsistentwiththegen-eralfactoringdenitionbywritingA(Up )(Vp )T.)Considerafeature~f,speciedbyhowmuchitappliestoeachconcept.A~fgivestheweightedsumoftheconceptsthathavethatfeature,~c,speciedbyhowmucheachfeatureappliestoallthoseconcepts.TheSVDallowsustowriteA~fUVT~f,which,readright-to-left,givesanintuitiveunderstandingoftheSVD.First,VT~fgiveshowmucheachsingularvector~viisassociatedwith~f.Then,scalesbytheprominenceofeachsingularvector.Finally,Uexpresseshowmucheachconceptisassociatedwitheachsingularvector.ABregmandivergence(SinghandGordon2008)measurestheerror,underagivenfunctionF,ofusing^AinsteadofA:DF(^AjA)=XijF(aij)�F0(aij)^aij+F(F0(aij));whereF(m)istheconvexdual(Legendretransform)ofFandgivesthenegativeofthey-interceptofthelinetangenttoFwithslopem,thusensuringthattheminimumdivergenceis0.ItcanbeshownthattheSVDminimizestheBregmandivergencefor^A=UVTundersquaredlossF(x)=x2:Dx2(^AjA)=Xij(aij�^aij)2:FactoringTwoMatricesUsingOneBigMatrixSupposewehavetwodatasources,e.g.,ConceptNetandanontologylikeWordNetorsomedomain-specicknowledgebase.Iftheyhaveconceptsorfeaturesincommon,ortheirconceptsandfeaturescanbealigned,theBlendingtechniqueprovidesawaytoanalyzethemtogether.Itconstructsasim-pliedviewoftheworld,similartothatofAnalogySpace,whereatleastoneofthe“lenses”(principalcomponents)con-siderscharacteristicsfrombothdatasets.Theresultcaninferthatifaconcepthascertaincharacteristicsinonedomainthatitmayhaverelatedcharacteristicsinanotherdomain.BlendingworksbyconstructingasinglelargematrixAcomposedofweightedsumsofeachdataset,alignedandzero-paddedsothattherowandcolumnlabelsmatch.1Reorderingtherowsorcolumnsofamatrix,oraddingrowsorcolumnsofallzeros,merelyreordersoraddszerorowstotheUandVmatricesintheSVDofeachinputmatrix.Thetechniqueworksforanarbitrarynumberofdatasetsinavarietyofcongurations,evenoverlapping.Forthispre-sentation,however,weconsideronlyasimpleblendoftwomatrices,XandY.Supposethattheyoverlapinconceptsbutnotfeatures,andthatthistimeconceptsareplacedincolumns.Sincethefeaturesaredisjoint,wecanwritetheweightedsummatrixasA=(1� )X Y.WenowfactorA(usingtheSVD,forexample):A=(1� )X YUXUYVT=UXVTUYVT:ThisdecompositionsaysthatweareforcedtousethesameVtofactorbothXandY.IfwehadfactoredeitherofXorYalone,wewouldhavegottenaVoptimizedforreconstructingthecorrespondingmatrix.Butinstead,Vmustaccount(tosomedegree,controlledby )forbothmatrices,inordertomakeareasonablereconstruction.BlendingSpreadsOuttheLoss,LikeCMFWhatproblemisBlendingsolving?TheSVDandmanyrelatedalgorithmsminimizesomeBregmandivergence:theerror,withrespecttoagivenfunctionF,ofusing^A=UVTinsteadofA.Inthisblendedsetup,wehaveDUXVTUYVTj(1� )X Y:Ifthedivergencefunctionisseparablebyelement(whichitisfortheSVD'ssquared-errorloss),wecanwritethedivergenceasinsteadD(UXVTj(1� )X)+D(UYVTj Y):Sothefactorizationminimizesaweightedlossoverbothmatrices.Thus,BlendingisaspecialcaseofCollectiveMa-trixFactorization(SinghandGordon2008),afactorization 1Sometimesthelabelsoftheinputdatamaynotmatchexactly;forexample,WordNetdistinguisheswordsenses,whereasConcept-Netdoesnot.Insuchacase,optionsincludetreatingConceptNetdataasapplyingtoallwordsensesequally,ignoringwordsensedistinctions,orusingtheBridgeBlendingtechniquediscussedlater. plotwillbevisiblelaterasFigure1).Whyisthisthecase?Whatoccurswhensingularvaluesapproacheachother?Inthissection,weusethemathematicaltechniquesdevelopedabovetoshowthatastheveeringpointisapproached,theresultingsingularvectoractuallyrotatesbetweenthetwosingularvectors,permittingcross-domaininference.Weconsiderasimpliedprobleminordertostudytheveeringeffectinisolation.Considertworank-1matricesX=~uX~vTXandB= ~uY~vTY,where~ufX;Ygand~vfX;Ygareunit-magnitudecolumnvectorswithmatchedlabels.(Inthenumericalexamplesshownlater,thesevectorsaretherstsingularvectorsoftheIsAandAtLocationportionsofConceptNet.)Theresultwillbeindependentofabsolutemag-nitude; representstherelativemagnitudeofYcomparedtoX.X+Ycouldbeuptorank2;whatareitssingularvaluesandvectors?Followingthederivationabove,westartbywritingX+Y=~uX~vTX+ ~uY~vTY=[~uX~uY]100 [~vX~vY]T:(2)Nowwewishtondtheparallelandorthogonalcompo-nentsof~uYwithrespectto~uX,whichisjustaprojection.TheparallelcomponentwasUUTAfromthegeneralderiva-tionabove,butforasinglecolumnvectoritsimpliesto~uX~uTX~uY,whichisjust~ak=~uX(~uX~uY)=~uXcosU,whereUistheanglebetweenthetwovectors.Assumingthat0jcosUj1,oneperpendicularcomponentexists,givenby~a?=~uY�~uXcosU.Since~a?istheoppo-sitesideofarighttrianglewith~ak,itsmagnitudeissinU.Wethennormalize~a?toget^a?=~uY�~uXcosU sinU.Then~uY=~uXcosU+^a?sinU,and[~uX~uY]=[~uX^a?]1cosU0sinU:Thecorrespondingderivationappliedto~vYyields[~vX~vY]=~vX^b?1cosV0sinV:Wecannowwriteequation2asX+Y=[~uX~uY]100 [~vX~vY]T=[~uX^a?]K~vX^b?T;wheretheinnertermKisgivenbyK=1cosU0sinU100 1cosV0sinVT1cosU0sinU10011 cosV0 sinVT=1000+cosUsinU cosV sinVT:Onecommoncaseisblendingamatrixwhereonlyeithertherowsorcolumnsoverlap.Ifthecolumnsdonotoverlap, Figure1:Singularvaluesvs. forseveralvaluesofUthen~vXisorthogonalto~vY,soV==2(socosV=0andsinV=1)andKbecomes:K=1cosU0sinU100 I=1 cosU0 sinU:WediagonalizeKbycomputingitsSVD:K=UKKVTK.Acomputeralgebrasystemwasusedtondthatthetwosingularvaluesare1=2r 2+2 22q 1+2 2+ 4�4 2sin2U;whichisplottedforaseveralvaluesofUinFigure1;themiddlevalueistheanglebetweentheIsAandAtLocationvectorsinConceptNet.Thus,theSVDofarank-1blendoverlappingincolumnsonly,intermsoftheanglebetweentherowvectors,is~uX~vTX+ ~uY~vTY=([~uXj^a?]UK)Kh~vXj^b?iVKTAsymbolicsolutionforthesingularvectorsisnotasstraightforward,butanintuitioncanbeobtainedbyplot-tinghowtheunitcircleisaffectedbyK,rememberingthatUkandVk,beingorthogonal,actasrotationsabouttheorigin.Figure2showsK~cforj~cj=1.SinceVTK~conlyrotatesabouttheorigin,itdoesnotchangethecircle.Thenstretchesthecircleintoanellipsealongthex-andy-axes,whichcorrespondintuitivelytotheXandYmatrices.Finally,UKrotatesthatellipsesothatitsaxesalignwiththesingularvec-tors~uk1and~uk2,whichareplotted(scaledbytheirsingularvalues)inthegure.When =0,Y=0;whenU=0,Yliesentirelywithinthecolumn-spaceofX(inthiscaseitisascalarmultipleofX).Ineithercase,theellipseissquashedontothex-axis,indicatingthatonlythesingularvectoroftheresultisthesingularvectorofX.Asthemagnitude ofthesecondmatrixincreases,ortheangleUbetweenthetwomatricesincreases,therstsingularvectorbeginstorotateoffofthex-axis,indicatingthatitisnowalinearcombination Figure2:K~cforj~cj=1,representingUk,forseveralvaluesof andU.x-axisrepresentsthefactorof~uX;y-axisrepresentsthefactorof^a?.ofthesingularvectorsofbothXandY.However,onceU==2,i.e.,theinputvectorsbecomeorthogonal,theresultisacircle,meaningthattheoriginalsingularvectorspassedthrougheitherunchangedormerelyreordered.Astherstsingularvectorrotates,thesecondalsonecessarilyrotatesinordertoremainorthogonaltotherst,sonowbothvectorshavesupportfrombothXandY.Incidentally,thismeansthateventhoughthefeaturesdidnotoverlapatrst,therotatedUnowprojectsconceptsinonematrixintospacethathasfeaturesinbothmatrices.ThismeansthatbyprojectingavectorcontainingonlyfeaturesfromXintothespaceandtruncatingittoitsinitialsingularvalue,weendupwithavectorthatmapstofeaturesinbothXandY;thisiscross-domaininference.Soweseethattomaximizeinteractionofonepairofsin-gularvalues,therelativemagnitudesshouldbeequal.Thus,(Havasietal.2009)usedthisasaheuristicforblendingmorecomplexmatrices:maketheirrstsingularvaluesequal.Onepossibility,asyetunexploredempirically,istoengineerinteractionsbetweenmultiplesingularvalues;thiscanbeaccomplishedusingthe“rank-1”formulationoftheSVD,X+YkXi=0Xi~uXiT~vXi+Yi~uYiT~vYi;bysettingthesapriori.BridgeBlendingBlendingonlyworkswhenpairsofdatasetsoverlapinei-thertheirrowsortheircolumns.ConsiderthelayoutofFigure3a,2forexample:wehave(hypothetically)common-senseknowledgeinEnglishandFrench,butwithoutknow-ingwhichEnglishconceptsorfeaturescorrespondtowhichFrenchconceptsorfeatures,wehavenowayreasoningjointly 2Inpractice,commonsensedatamatricesareabout5timesaswideastall,sincemostconceptsparticipateinseveralfeatures. (a)Nooverlap=noblend-ing (b)Transposingoneofthematricesallowsforaconcept-to-conceptbridgeFigure3:BridgeBlending:Connectingdatathatdonotoverlapoverthem.Thesimilarity(dotproduct)betweenanyEnglishconcept/featureandanyFrenchconcept/featureinsuchalay-outisexactly0.Infact,it'sreadilyshownthatunlessthetwomatricesshareasingularvalueexactly,noneoftheaxeswillcontainbothEnglishandFrenchconcepts.Rather,thesetofsingularvectorsofthe“blend”willbesimplytheunionofthesingularvaluesoftheEnglishandFrenchmatricesalone,paddedwithzeros.Boring.SohowcanwereasonjointlyoverbothEnglishandFrench?Weneedtoaddanotherdataset,calleda“bridge,”toconnectEnglishandFrench.Itcouldlloneofthemissingoff-diagonalentriesinFigure3,butwithwhat?WewouldneeddataabouteitherFrenchfeaturesaboutEnglishcon-cepts,orEnglishfeaturesaboutFrenchconcepts.Wedonothavethatdatadirectly,thoughwecouldpossiblyinferitfromtheEnglishandFrenchcommonsensedata.Morereadilyavailableisabilingualdictionary,connectingEnglishcon-ceptstoFrenchconceptsandviceversa.WecouldtransformthatintoamatrixofEnglishconceptsbyFrenchconcepts.ThebridgedatacouldtintotheblendifwetransposedoneoftheConceptNetmatrices,asinFigure3b.Thecanonicalencodingofcommonsensedataisconceptsonrowsandfeaturesoncolumns;willthetransposedarrange-mentstillyieldmeaningfulresults?TransposingamatrixjustreversestherolesofUandVinitsSVD,sotransposingasinglematrixdoesnoharm.ButwemightworrythatthefactthattheEnglishandFrenchconcepts/featuresareondifferentsidesofthematrixkeepsthemfrombeingmeaningfullyre-lated.Thissectiongivesafewstepstowardsamathematicaldemonstrationthatcross-domaininferenceoccursinbridgedblendingingeneralandinthetransposedarrangementinparticular.Acompletemathematicaltreatmentawaitsafu-turepublication,butapplicationsincludingProcedureSpace(ArnoldandLieberman2010)haveempiricallydemonstratedtheeffectivenessofthetechnique.LetXbetheEnglishConceptNetwithconceptsxi,andYbetheFrenchConceptNetwithconceptsyi.WethenencodethebilingualdictionaryintoamatrixB,whereB(i;j)givesthesimilaritybetweenconceptsxiandyj.WenowarraythetwodatasetsXandYalongwiththebridgedatasetBinthetransposedbridgeblendlayout: C=XBYTUVTIntuitively,wesuspectthatthebridgedatasetwillcausetheeigenconceptsandeigenfeaturestobecomposedofitemsfrombothXandYT.Ifthisworks,thenwewillbeabletodeterminewhatFrenchconceptsapplytoanEnglishfeature,oraskaboutthetranslationofdifferentsensesofawordbasedonprojectingdifferentcombinationsoffeatures,allbycomputingmatrix-by-vectorproducts.Toseeifitworks,considerasimpliedcasethatthebridgedataisaweightedidentitymatrix,i.e.,everyxicorrespondstoexactlyoneyjwithconstantweight.ThissetuprequiresthatthenumberofrowsofXequalthenumberofrowsofY.Thoughrealisticbridgeblendsbreakbothoftheserules,thissetupisstillarepresentativeidealization.Thetransposedbridgelayoutwithidentitybridgingis:C=X I0YT:IfYT=0,blendingwiththeconstant-weightidentitiesadds 2toeacheigenvaluewithoutchangingtheeigenvectors.ToanalyzethecontributionofYT,westartbycomputingtherow-rowdotproducts:CCT=X I0YTXT0 IY=XXT+ 2I Y BYTY:IfXXT~u=~u(i.e.,~uisaneigenvectorofXXT),thenCCT~u0=XXT+ 2I Y YTYTY~u0=XXT~u+ 2I~u YT~u=(+ 2)~u YT~u:Soaslongas~uisnotinthenullspaceofYT,novectorwithzerosupportintheYTdomaincouldbeaneigenvector.Sotheeigenconceptsofthebridge-blendeddatamustbedeterminedbybothmatrices.ExchangingtherolesofXandYT,thesameargumentshowsthateigenfeaturesalsomusthavecross-domainsupport.BridgeBlendingisAlsoCMFConsiderageneralbridgeblendlayout:XYZUXYU0ZVX0VYZT=UXYVTX0UXYVTYZU0ZVTX0U0ZVTYZThelossonceagainfactorsintoseparatetermsforeachrela-tion,showingthatbridgeblendingisalsoakindofCMF:D(^AjA)=D(UXYVTX0jX)+D(UXYVTYZjY)+D(U0ZVTX0j0)+D(U0ZVTYZjZ)WeobservethatthefactorVYZtiestogetherthefactorizationofXandZthroughthebridgedataY;withoutapenaltyforreconstructingYpoorly,thefactorizationsofXandZwouldbeindependent.Notethatonecomponentofthelossispredictingnon-zerointhezerobottom-leftcorner.Ifinfactweknowthatregiontobenon-zero,wecoulduseaweightedlossfunction,whichCMFpermits.ConclusionFactoredinference,asinAnalogySpace,isausefultoolforapproximatereasoningovernoisyknowledgebaseslikeCon-ceptNet.WehaveshownthatBlendingisakindofCollectiveMatrixFactorization(CMF)inthatthefactorizationspreadsoutthepredictionlossbetweeneachdataset.Wealsoshowedthatblendingadditionaldatacausesthesingularvectorstorotatebetweenvectorsindifferentdomains,whichenablescross-domaininference.Inasimpliedexample,wejustiedpreviousevidencethatthemaximuminteractionoccurswhenthemagnitudes(asdenedbythelargestsingularvalues)ofthetwomatricesareequal.Finally,wedescribedandjusti-edBridgeBlending,bothnormalandtransposed,whichisakindofCMFthatcanconnectmorethan2datasets.AcknowledgementsWegratefullyacknowledgehelpfulcritiqueaboutconcepts,math,andexplanationsfromJasonAlonso,CatherineHavasi,andRobSpeer.WealsothankWeikePanofHongKongUniversityofScienceandTechnologyforbringingCollectiveMatrixFactorizationtoourattention.ReferencesArnold,K.C.,andLieberman,H.2010.Managingambigu-ityinprogrammingbyndingunambiguousexamples.InOnward!2010(toappear).Brand,M.2006.Fastlow-rankmodicationsofthethinsingularvaluedecomposition.LinearAlgebraanditsAppli-cations415(1):20–30.SpecialIssueonLargeScaleLinearandNonlinearEigenvalueProblems.Havasi,C.;Speer,R.;Pustejovsky,J.;andLieberman,H.2009.DigitalIntuition:Applyingcommonsenseusingdi-mensionalityreduction.IEEEIntelligentSystems.Havasi,C.;Speer,R.;andAlonso,J.2007.ConceptNet3:aexible,multilingualsemanticnetworkforcommonsenseknowledge.InRecentAdvancesinNaturalLanguageProcessing.Havasi,C.2009.DiscoveringSemanticRelationsUsingSingularValueDecompositionBasedTechniques.Ph.D.Dis-sertation,BrandeisUniversity.Singh,A.P.,andGordon,G.J.2008.Relationallearningviacollectivematrixfactorization.InKDD'08:Proceedingofthe14thACMSIGKDDinternationalconferenceonKnowl-edgediscoveryanddatamining,650–658.NewYork,NY,USA:ACM.Speer,R.H.;Havasi,C.;Treadway,K.N.;andLieberman,H.2010.Findingyourwayinamulti-dimensionalsemanticspacewithLuminoso.InIUI'10:Proceedingofthe14thinternationalconferenceonIntelligentuserinterfaces,385–388.NewYork,NY,USA:ACM.Speer,R.;Havasi,C.;andLieberman,H.2008.Analogy-Space:Reducingthedimensionalityofcommonsenseknowl-edge.ProceedingsofAAAI2008.