refertothesameevent.TheimportanceofeventcoreferencewasrecognizedintheM - PDF document

refertothesameevent.TheimportanceofeventcoreferencewasrecognizedintheMUCinformationextractiontasksinwhichitwaskeytoidentifysce-nariosrelatedtothesameevent(Humphreysetal.,1997).Recentworkintextunderstandinghasnotfocusedonthisissue,butitmustbetackledinasuc-cessfulcontradictionsystem.Oursystemincludeseventcoreference,andwepresenttherstdetailedexaminationofcontradictiondetectionperformance,onthebasisofourtypology.2RelatedworkLittleworkhasbeendoneoncontradictiondetec-tion.ThePASCALRecognizingTextualEntailment(RTE)Challenges(Daganetal.,2006;Bar-Haimetal.,2006;Giampiccoloetal.,2007)focusedontextualinferenceinanydomain.Condoravdietal.(2003)rstrecognizedtheimportanceofhandlingentailmentandcontradictionfortextunderstanding,buttheyrelyonastrictlogicaldenitionofthesephenomenaanddonotreportempiricalresults.Toourknowledge,Harabagiuetal.(2006)providetherstempiricalresultsforcontradictiondetection,buttheyfocusonspecickindsofcontradiction:thosefeaturingnegationandthoseformedbyparaphrases.Theyconstructedtwocorporaforevaluatingtheirsystem.OnewascreatedbyovertlynegatingeachentailmentintheRTE2data,producingabal-anceddataset(LCC negation).Toavoidovertrain-ing,negativemarkerswerealsoaddedtoeachnon-entailment,ensuringthattheydidnotcreatecon-tradictions.Theotherwasproducedbyparaphras-ingthehypothesissentencesfromLCC negation,re-movingthenegation(LCC paraphrase):Ahungerstrikewasnotattempted!Ahungerstrikewascalledoff.Theyachievedverygoodperformance:accuraciesof75.63%onLCC negationand62.55%onLCC paraphrase.Yet,contradictionsarenotlim-itedtotheseconstructions;tobepracticallyuseful,anysystemmustprovidebroadercoverage.3Contradictions3.1Whatisacontradiction?Onestandardistoadoptastrictlogicaldenitionofcontradiction:sentencesAandBarecontradictoryifthereisnopossibleworldinwhichAandBarebothtrue.However,forcontradictiondetectiontobeuseful,alooserdenitionthatmorecloselymatcheshumanintuitionsisnecessary;contradictionoccurswhentwosentencesareextremelyunlikelytobetruesimultaneously.PairssuchasSallysoldaboattoJohnandJohnsoldaboattoSallyaretaggedascon-tradictoryeventhoughitcouldbethateachsoldaboattotheother.Thisdenitioncapturesintuitionsofincompatiblity,andperfectlytsapplicationsthatseektohighlightdiscrepanciesindescriptionsofthesameevent.Examplesofcontradictionaregivenintable1.Fortextstobecontradictory,theymustin-volvethesameevent.Twophenomenamustbecon-sideredinthisdetermination:impliedcoreferenceandembeddedtexts.Givenlimitedcontext,whethertwoentitiesarecoreferentmaybeprobableratherthancertain.Tomatchhumanintuitions,compatiblenounphrasesbetweensentencesareassumedtobecoreferentintheabsenceofclearcountervailingev-idence.Inthefollowingexample,itisnotnecessarythatthewomanintherstandsecondsentencesisthesame,butonewouldlikelyassumeitisifthetwosentencesappearedtogether:(1)PassionssurroundingGermany'snalmatchturnedviolentwhenawomanstabbedherpartnerbecauseshedidn'twanttowatchthegame.(2)Awomanpassionatelywantedtowatchthegame.Wealsomarkascontradictionspairsreportingcon-tradictorystatements.Thefollowingsentencesrefertothesameevent(deMenezesinasubwaystation),anddisplayincompatibleviewsofthisevent:(1)EyewitnessessaiddeMenezeshadjumpedovertheturnstileatStockwellsubwaystation.(2)ThedocumentsleakedtoITVNewssuggestthatMenezeswalkedcasuallyintothesubwaystation.Thisexamplecontainsan“embeddedcontradic-tion.”ContrarytoZaenenetal.(2005),wearguethatrecognizingembeddedcontradictionsisimpor-tantfortheapplicationofacontradictiondetectionsystem:ifJohnthinksthatheisincompetent,andhisbossbelievesthatJohnisnotbeinggivenachance,onewouldliketodetectthatthetargetedinformationinthetwosentencesiscontradictory,eventhoughthetwosentencescanbetruesimultaneously.3.2TypologyofcontradictionsContradictionsmayarisefromanumberofdifferentconstructions,someovertandothersthatarecom- Data#contradictions#totalpairs RTE1 dev148287RTE1 dev255280RTE1 test149800RTE2 dev111800RTE3 dev80800RTE3 test72800 Table2:NumberofcontradictionsintheRTEdatasets.Inbothcases,therstsentencediscussesoneen-tity(CFAP,TheChannelTunnel)witharelationship(purchase,stretch)tootherentities.Thesecondsen-tencepositsasimilarrelationshipthatincludesoneoftheentitiesinvolvedintheoriginalrelationshipaswellasanentitythatwasnotinvolved.However,differentoutcomesresultbecauseatunnelconnectsonlytwouniquelocationswhereasmorethanoneentitymaypurchasefood.Thesefrequentinterac-tionsbetweenworld-knowledgeandstructuremakeithardtoensurethatanyparticularinstanceofstruc-turalmismatchisacontradiction.3.3ContradictioncorporaFollowingtheguidelinesabove,weannotatedtheRTEdatasetsforcontradiction.Thesedatasetscon-tainpairsconsistingofashorttextandaone-sentencehypothesis.Table2givesthenumberofcontradictionsineachdataset.TheRTEdatasetsarebalancedbetweenentailmentsandnon-entailments,andeveninthesedatasetstargetinginference,therearefewcontradictions.Usingourguidelines,RTE3 testwasannotatedbyNISTaspartoftheRTE3Pilottaskinwhichsystemsmadea3-wayde-cisionastowhetherpairsofsentenceswereentailed,contradictory,orneither(Voorhees,2008).1OurannotationsandthoseofNISTwereper-formedontheoriginalRTEdatasets,contrarytoHarabagiuetal.(2006).Becausetheircorporaareconstructedusingnegationandparaphrase,theyareunlikelytocoveralltypesofcontradictionsinsec-tion3.2.Wemighthypothesizethatrewritingex-plicitnegationscommonlyoccursviathesubstitu-tionofantonyms.Imagine,e.g.:H:Billhasnishedhismath. 1Informationaboutthistaskaswellasdatacanbefoundathttp://nlp.stanford.edu/RTE3-pilot/. TypeRTEsets`Real'corpus 1Antonym15.09.2Negation8.817.6Numeric8.829.0 2Factive/Modal5.06.9Structure16.33.1Lexical18.821.4WK27.513.0 Table3:PercentagesofcontradictiontypesintheRTE3 devdatasetandtherealcontradictioncorpus.Neg-H:Billhasn'tnishedhismath.Para-Neg-H:Billisstillworkingonhismath.Therewritinginboththenegatedandthepara-phrasedcorporaislikelytoleaveoneinthespaceof`easy'contradictionsandaddressesfewerthan30%ofcontradictions(table3).WecontactedtheLCCauthorstoobtaintheirdatasets,buttheywereunabletomakethemavailabletous.Thus,wesimulatedtheLCC negationcorpus,addingnegativemarkerstotheRTE2testdata(Neg test),andtoadevelopmentset(Neg dev)constructedbyrandomlysampling50pairsofentailmentsand50pairsofnon-entailmentsfromtheRTE2developmentset.SincetheRTEdatasetswereconstructedfortex-tualinference,thesecorporadonotreect`real-life'contradictions.Wethereforecollectedcontradic-tions`inthewild.'Theresultingcorpuscontains131contradictorypairs:19fromnewswire,mainlylookingatrelatedarticlesinGoogleNews,51fromWikipedia,10fromtheLexisNexisdatabase,and51fromthedatapreparedbyLDCforthedistillationtaskoftheDARPAGALEprogram.Despitetheran-domnessofthecollection,wearguethatthiscorpusbestreectsnaturallyoccurringcontradictions.2Table3givesthedistributionofcontradictiontypesforRTE3 devandtherealcontradictioncor-pus.Globally,weseethatcontradictionsincategory(2)occurfrequentlyanddominatetheRTEdevelop-mentset.Intherealcontradictioncorpus,thereisamuchhigherrateofthenegation,numericandlex-icalcontradictions.Thissupportstheintuitionthatintherealworld,contradictionsprimarilyoccurfortworeasons:informationisupdatedasknowledge 2Ourcorpora—thesimulationoftheLLC negationcorpus,theRTEdatasetsandtherealcontradictions—areavailableathttp://nlp.stanford.edu/projects/contradiction. StrategyPrecisionRecall Nolter55.1032.93Root61.3632.93Root+topic61.9031.71 Table4:PrecisionandrecallforcontradictiondetectiononRTE3 devusingdifferentlteringstrategies.woundedinabombing,targetingtheirconvoynearBeiji,150milesnorthofBaghdad.H:ThreeIraqisoldiersalsodiedSaturdaywhentheirconvoywasattackedbygunmennearAdhaim.Itseemsthattherealworldfrequencyofeventsneedstobetakenintoaccount.Inthiscase,attacksinIraqareunfortunatelyfrequentenoughtoassertthatitisunlikelythatthetwosentencespresentmis-matchinginformation(i.e.,differentlocation)aboutthesameevent.Butcomparethefollowingexample:T:PresidentKennedywasassassinatedinTexas.H:Kennedy'smurderoccurredinWashington.Thetwosentencesrefertooneuniqueevent,andthelocationmismatchrendersthemcontradictory.4.4ExtractionofcontradictionfeaturesInthenalstage,weextractcontradictionfeaturesonwhichweapplylogisticregressiontoclassifythepairascontradictoryornot.Thefeatureweightsarehand-set,guidedbylinguisticintuition.5FeaturesforcontradictiondetectionInthissection,wedeneeachofthefeaturesetsusedtocapturesalientpatternsofcontradiction.Polarityfeatures.Polaritydifferencebetweenthetextandhypothesisisoftenagoodindicatorofcon-tradiction,providedthereisagoodalignment(seeexample2intable1).Thepolarityfeaturescap-turethepresence(orabsence)oflinguisticmark-ersofnegativepolaritycontexts.Thesemarkersarescopedsuchthatwordsareconsiderednegatediftheyhaveanegationdependencyinthegraphorareanexplicitlinguisticmarkerofnegation(e.g.,sim-plenegation(not),downward-monotonequantiers(no,few),orrestrictingprepositions).Ifonewordisnegatedandtheotherisnot,wemayhaveapolaritydifference.Thisdifferenceisconrmedbycheckingthatthewordsarenotantonymsandthattheylackunalignedprepositionsorothercontextthatsuggeststheydonotrefertothesamething.Insomecases,negationsarepropagatedontothegovernor,whichallowsonetoseethatnobulletpenetratedandabul-letdidnotpenetratehavethesamepolarity.Number,dateandtimefeatures.Numericmis-matchescanindicatecontradiction(example3intable1).Thenumericfeaturesrecognize(mis-)matchesbetweennumbers,dates,andtimes.Wenormalizedateandtimeexpressions,andrep-resentnumbersasranges.Thisincludesexpressionmatching(e.g.,over100and200isnotamismatch).Alignednumbersaremarkedasmismatcheswhentheyareincompatibleandsurroundingwordsmatchwell,indicatingthenumbersrefertothesameentity.Antonymyfeatures.Alignedantonymsareaverygoodcueforcontradiction.OurlistofantonymsandcontrastingwordscomesfromWordNet,fromwhichweextractwordswithdirectantonymylinksandexpandthelistbyaddingwordsfromthesamesynsetastheantonyms.WealsouseoppositionalverbsfromVerbOcean.Wecheckwhetheranalignedpairofwordsappearsinthelist,aswellascheckingforcommonantonymprexes(e.g.,anti,un).Thepolarityofthecontextisusedtodetermineiftheantonymscreateacontradiction.Structuralfeatures.Thesefeaturesaimtodeter-minewhetherthesyntacticstructuresofthetextandhypothesiscreatecontradictorystatements.Forex-ample,wecomparethesubjectsandobjectsforeachalignedverb.Ifthesubjectinthetextoverlapswiththeobjectinthehypothesis,wendevidenceforacontradiction.Considerexample6intable1.Inthetext,thesubjectofsucceedisJacquesSanterwhileinthehypothesis,Santeristheobjectofsucceed,suggestingthatthetwosentencesareincompatible.Factivityfeatures.Thecontextinwhichaverbphraseisembeddedmaygiverisetocontradiction,asinexample5(table1).Negationinuencessomefactivitypatterns:Billforgottotakehiswalletcon-tradictsBilltookhiswalletwhileBilldidnotforgettotakehiswalletdoesnotcontradictBilltookhiswallet.Foreachtext/hypothesispair,wecheckthe(grand)parentofthetextwordalignedtothehypoth-esisverb,andgenerateafeaturebasedonitsfactiv- TypeRTE3 devRTE3 test 1Antonym25.0(3/12)42.9(3/7)Negation71.4(5/7)60.0(3/5)Numeric71.4(5/7)28.6(2/7) 2Factive/Modal25.0(1/4)10.0(1/10)Structure46.2(6/13)21.1(4/19)Lexical13.3(2/15)0.0(0/12)WK18.2(4/22)8.3(1/12) Table6:Recallbycontradictiontype.7ErroranalysisanddiscussionOnesignicantissueincontradictiondetectionislackoffeaturegeneralization.Thisproblemises-peciallyapparentforitemsincategory(2)requiringlexicalandworldknowledge,whichprovedtobethemostdifcultcontradictionstodetectonabroadscale.Whileweareabletondcertainspecicre-lationshipsinthedevelopmentsets,thesefeaturesattainedonlylimitedcoverage.Manycontradictionsinthiscategoryrequiremultipleinferencesandre-mainbeyondourcapabilities:T:TheAuburnHighSchoolAthleticHallofFamere-centlyintroduceditsClassof2005whichincludes10members.H:TheAuburnHighSchoolAthleticHallofFamehastenmembers.Ofthetypesofcontradictionsincategory(2),wearebestataddressingthoseformedviastructuraldiffer-encesandfactive/modalconstructionsasshownintable6.Forinstance,wedetectexamples5and6intable1.However,creatingfeatureswithsufcientprecisionisanissueforthesetypesofcontradic-tions.Intuitively,twosentencesthathavealignedverbswiththesamesubjectanddifferentobjects(orviceversa)arecontradictory.Thisindeedindicatesacontradiction55%ofthetimeonourdevelopmentsets,butthisisnothighenoughprecisiongiventherarityofcontradictions.Anothertypeofcontradictionwhereprecisionfal-tersisnumericmismatch.Weobtainhighrecallforthistype(table6),asitisrelativelysimpletodeter-mineiftwonumbersarecompatible,buthighpreci-sionisdifculttoachieveduetodifferencesinwhatnumbersmaymean.Consider:T:NikeInc.saidthatitsprotgrew32percent,asthecompanypostedbroadgainsinsalesandorders.H:Nikesaidordersforfootweartotaled$4.9billion,includinga12percentincreaseinU.S.orders.Oursystemdetectsamismatchbetween32percentand12percent,ignoringthefactthatonereferstoprotandtheothertoorders.Accountingforcon-textrequiresextensivetextcomprehension;itisnotenoughtosimplylookatwhetherthetwonumbersareheadedbysimilarwords(grewandincrease).Thisemphasizesthefactthatmismatchinginforma-tionisnotsufcienttoindicatecontradiction.Asdemonstratedbyour63%accuracyonNeg test,wearereasonablygoodatdetectingnega-tionandcorrectlyascertainingwhetheritisasymp-tomofcontradiction.Similarly,wehandlesinglewordantonymywithhighprecision(78.9%).Never-theless,Harabagiuetal.'sperformancedemonstratesthatfurtherimprovementonthesetypesispossible;indeed,theyusemoresophisticatedtechniquestoextractoppositionaltermsanddetectpolaritydiffer-ences.Thus,detectingcategory(1)contradictionsisfeasiblewithcurrentsystems.WhilethesecontradictionsareonlyathirdofthoseintheRTEdatasets,detectingsuchcontra-dictionsaccuratelywouldsolvehalfoftheprob-lemsfoundintherealcorpus.Thissuggeststhatwemaybeabletogainsufcienttractiononcontra-dictiondetectionforrealworldapplications.Evenso,category(2)contradictionsmustbetargetedtodetectmanyofthemostinterestingexamplesandtosolvetheentireproblemofcontradictiondetection.Sometypesofthesecontradictions,suchaslexi-calandworldknowledge,arecurrentlybeyondourgrasp,butwehavedemonstratedthatprogressmaybemadeonthestructureandfactive/modaltypes.Despitebeingrare,contradictionisfoundationalintextcomprehension.Ourdetailedinvestigationdemonstrateswhichaspectsofitcanberesolvedandwherefurtherresearchmustbedirected.AcknowledgmentsThispaperisbasedonworkfundedinpartbytheDefenseAdvancedResearchProjectsAgencythroughIBMandbytheDisruptiveTechnologyOfce(DTO)PhaseIIIProgramforAdvancedQuestionAnsweringforIntelligence(AQUAINT)throughBroadAgencyAnnouncement(BAA)N61339-06-R-0034.