pittedu Janyce Wiebe Department of Computer Science University of Pittsburgh Pittsburgh PA 15260 wiebecspittedu Paul Hoffmann Intelligent Systems Program University of Pittsburgh Pittsburgh PA 15260 hoffmanpcspittedu Abstract This paper presents a ne ID: 26513
Download Pdf The PPT/PDF document "Recognizing Contextual Polarity in Phras..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
noonethinksthatit'sgood).Inaddition,certainphrasesthatcontainnegationwordsintensifyratherthanchangepolarity(e.g.,notonlygoodbutamaz-ing).Contextualpolaritymayalsobeinuencedbymodality(e.g.,whetherthepropositionisassertedtobereal(realis)ornotreal(irrealis)–noreasonatalltobelieveisirrealis,forexample);wordsense(e.g.,EnvironmentalTrustversusHehaswonthepeo-ple'strust);thesyntacticroleofawordinthesen-tence(e.g.,pollutersareversustheyarepolluters);anddiminisherssuchaslittle(e.g.,littletruth,lit-tlethreat).(See(PolanyaandZaenen,2004)foramoredetaileddiscussionofcontextualpolarityin-uencers.)Thispaperpresentsnewexperimentsinautomat-icallydistinguishingpriorandcontextualpolarity.Beginningwithalargestableofcluesmarkedwithpriorpolarity,weidentifythecontextualpolarityofthephrasesthatcontaininstancesofthosecluesinthecorpus.Weuseatwo-stepprocessthatemploysmachinelearningandavarietyoffeatures.Therststepclassieseachphrasecontainingaclueasneutralorpolar.Thesecondsteptakesallphrasesmarkedinsteponeaspolaranddisambiguatestheircontextualpolarity(positive,negative,both,orneu-tral).Withthisapproach,thesystemisabletoauto-maticallyidentifythecontextualpolarityforalargesubsetofsentimentexpressions,achievingresultsthataresignicantlybetterthanbaseline.Inaddi-tion,wedescribenewmanualannotationsofcontex-tualpolarityandasuccessfulinter-annotatoragree-mentstudy.2ManualAnnotationSchemeTocreateacorpusfortheexperimentsbelow,weaddedcontextualpolarityjudgmentstoexistingan-notationsintheMulti-perspectiveQuestionAnswer-ing(MPQA)OpinionCorpus1,namelytothean-notationsofsubjectiveexpressions2.Asubjectiveexpressionisanywordorphraseusedtoexpressanopinion,emotion,evaluation,stance,speculation, 1TheMPQACorpusisdescribedin(Wiebeetal.,2005)andavailableatnrrc.mitre.org/NRRC/publications.htm.2IntheMPQACorpus,subjectiveexpressionsaredirectsubjectiveexpressionswithnon-neutralexpressionintensity,plusalltheexpressivesubjectiveelements.Pleasesee(Wiebeetal.,2005)formoredetailsontheexistingannotationsintheMPQACorpus.etc.Ageneralcoveringtermforsuchstatesispri-vatestate(Quirketal.,1985).IntheMPQACor-pus,subjectiveexpressionsofvaryinglengthsaremarked,fromsinglewordstolongphrases.Forthiswork,ourfocusisonsentimentexpres-sions–positiveandnegativeexpressionsofemo-tions,evaluations,andstances.Asthesearetypesofsubjectiveexpressions,tocreatethecorpus,wejustneededtomanuallyannotatetheexistingsubjectiveexpressionswiththeircontextualpolarity.Inparticular,wedevelopedanannotationscheme3formarkingthecontextualpolarityofsub-jectiveexpressions.Annotatorswereinstructedtotagthepolarityofsubjectiveexpressionsaspositive,negative,both,orneutral.Thepositivetagisforpositiveemotions(I'mhappy),evaluations(Greatidea!),andstances(Shesupportsthebill).Theneg-ativetagisfornegativeemotions(I'msad),eval-uations(Badidea!),andstances(She'sagainstthebill).Thebothtagisappliedtosentimentexpres-sionsthathavebothpositiveandnegativepolarity.Theneutraltagisusedforallothersubjectiveex-pressions:thosethatexpressadifferenttypeofsub-jectivitysuchasspeculation,andthosethatdonothavepositiveornegativepolarity.Belowareexamplesofcontextualpolarityanno-tations.Thetagsareinboldface,andthesubjectiveexpressionswiththegiventagsareunderlined.(5)Thousandsofcoupsupporterscelebrated (posi-tive)overnight,wavingags,blowingwhistles...(6)ThecriteriasetbyRicearethefollowing:thethreecountriesinquestionarerepressive (nega-tive)andgravehumanrightsviolators (negative)...(7)Besides,politiciansrefertogoodandevil (both)onlyforpurposesofintimidationandexaggeration.(8)Jeromesaysthehospitalfeels (neutral)nodif-ferentthanahospitalinthestates.Theannotatorswereaskedtojudgethecontex-tualpolarityofthesentimentthatisultimatelybe-ingconveyedbythesubjectiveexpression,i.e.,oncethesentencehasbeenfullyinterpreted.Thus,thesubjectiveexpression,theyhavenotsucceeded,and 3Theannotationinstructionsareavailableathttp://www.cs.pitt.edu/˜twilson. 6ExperimentsThegoaloftheexperimentsdescribedbelowistoclassifythecontextualpolarityoftheexpressionsthatcontaininstancesofthesubjectivitycluesinourlexicon.Whatthesystemspecicallydoesisgiveeachclueinstanceitsownlabel.Notethatthesystemdoesnottrytoidentifyexpressionbound-aries.Doingsomightimproveperformanceandisapromisingavenueforfutureresearch.6.1DenitionoftheGoldStandardWedenethegoldstandardusedtotrainandtestthesystemintermsofthemanualannotationsdescribedinSection2.Thegoldstandardclassofaclueinstancethatisnotinasubjectiveexpressionisneutral:sincetheclueisnoteveninasubjectiveexpression,itisnotcontainedinasentimentexpression.Otherwise,ifaclueinstanceappearsinjustonesubjectiveexpression(orinmultiplesubjectiveex-pressionswiththesamecontextualpolarity),thentheclassassignedtotheclueinstanceistheclassofthesubjectiveexpression(s).Ifaclueappearsinatleastonepositiveandonenegativesubjectiveexpression(orinasubjectiveexpressionmarkedasboth),thenitsclassisboth.Ifitisinamixtureofnegativeandneutralsubjectiveexpressions,itsclassisnegative;ifitisinamixtureofpositiveandneu-tralsubjectiveexpressions,itsclassispositive.6.2PerformanceofaPrior-PolarityClassierAnimportantquestionishowusefulpriorpolarityaloneisforidentifyingcontextualpolarity.Toan-swerthisquestion,wecreateaclassierthatsim-plyassumesthatthecontextualpolarityofacluein-stanceisthesameastheclue'spriorpolarity,andweexploretheclassier'sperformanceonthedevelop-mentset.Thissimpleclassierhasanaccuracyof48%.FromtheconfusionmatrixgiveninTable2,weseethat76%oftheerrorsresultfromwordswithnon-neutralpriorpolarityappearinginphraseswithneu-tralcontextualpolarity.6.3ContextualPolarityDisambiguationThefactthatwordswithnon-neutralpriorpolaritysofrequentlyappearinneutralcontextsledustoPrior-PolarityClassier NeutPosNegBothTotal Neut 798 784 698 4 2284 Pos 81 371 40 0 492 Gold Neg 149 181 622 0 952 Both 4 11 13 5 33 Total 1032 1347 1373 9 3761 Table2:Confusionmatrixfortheprior-polarityclassieronthedevelopmentset.adoptatwo-stepapproachtocontextualpolaritydis-ambiguation.Fortherststep,weconcentrateonwhetherclueinstancesareneutralorpolarincontext(wherepolarincontextreferstohavingacontextualpolaritythatispositive,negativeorboth).Forthesecondstep,wetakeallclueinstancesmarkedaspolarinstepone,andfocusonidentifyingtheircon-textualpolarity.Forbothsteps,wedevelopclassi-ersusingtheBoosTexterAdaBoost.HM(SchapireandSinger,2000)machinelearningalgorithmwith5000roundsofboosting.Theclassiersareevalu-atedin10-foldcross-validationexperiments.6.3.1Neutral-PolarClassicationTheneutral-polarclassieruses28features,listedinTable3.WordFeatures:Wordcontextisabagofthreewordtokens:thepreviousword,theworditself,andthenextword.Thepriorpolarityandreliabilityclassareindicatedinthelexicon.ModicationFeatures:Thesearebinaryrela-tionshipfeatures.Therstfourinvolverelationshipswiththewordimmediatelybeforeorafter:ifthewordisanounprecededbyanadjective,ifthepre-cedingwordisanadverbotherthannot,ifthepre-cedingwordisanintensier,andiftheworditselfisanintensier.Awordisconsideredanintensierifitappearsinalistofintensiersandifitprecedesawordoftheappropriatepart-of-speech(e.g.,anin-tensieradjectivemustcomebeforeanoun).Themodifyfeaturesinvolvethedependencyparsetreeforthesentence,obtainedbyrstparsingthesentence(Collins,1997)andthenconvertingthetreeintoitsdependencyrepresentation(XiaandPalmer,2001).Inadependencyrepresentation,everynodeinthetreestructureisasurfaceword(i.e.,therearenoabstractnodessuchasNPorVP).Theedgebe-tweenaparentandachildspeciesthegrammaticalrelationshipbetweenthetwowords.Figure1shows Acc PolarRecPolarPrecPolarF NeutRecNeutPrecNeutF wordtoken 73.6 45.372.255.7 89.974.081.2 word+priorpol 74.2 54.368.660.6 85.776.480.7 28features 75.9 56.871.663.4 87.077.782.1 Table4:ResultsforStep1Neutral-PolarClassication Positive Negative Both Neutral Acc RecPrecF RecPrecF RecPrecF RecPrecF wordtoken 61.7 59.363.461.2 83.964.773.1 9.235.214.6 30.250.137.7 word+priorpol 63.0 69.455.361.6 80.471.275.5 9.235.214.6 33.551.840.7 10features 65.7 67.163.365.1 82.172.977.2 11.228.416.1 41.452.446.2 Table5:ResultsforStep2PolarityClassication. Experiment FeaturesRemoved AB1negated,negatedsubject AB2modiespolarity,modiedbypolarity AB3conjpolarity AB4general,negative,andpositivepolarityshifters Table7:Featuresforpolarityclassicationtaketurnsdoingbetterorworseforprecisionandrecall.Usingjustthewordtoken,positivepreci-sionisslightlyhigherthanforthe10-featureclas-sier,butpositiverecallis11.6%lower.Addthepriorpolarity,andpositiverecallimproves,butattheexpenseofprecision,whichis12.6%lowerthanforthe10-featureclassier.Theresultsfornegativeexpressionsaresimilar.Theword-tokenclassierdoeswellonnegativerecallbutpoorlyonnegativeprecision.Whenpriorpolarityisadded,negativerecallimprovesbutnegativeprecisiondrops.Itisonlywiththeadditionofthepolarityfeaturesthatweachievebothhigherprecisionsandhigherrecalls.Toexplorehowmuchthevariouspolarityfeaturescontributetotheperformanceofthepolarityclassi-er,weperformfourexperiments.Ineachexperi-ment,adifferentsetofpolarityfeaturesisexcluded,andthepolarityclassierisretrainedandevaluated.Table7liststhefeaturesthatareremovedforeachexperiment.TheonlysignicantdifferenceinperformanceintheseexperimentsisneutralF-measurewhenthemodicationfeatures(AB2)areremoved.Theseablationexperimentsshowthatthecombinationoffeaturesisneededtoachievesignicantresultsoverbaselineforpolarityclassication.7RelatedWorkMuchworkonsentimentanalysisclassiesdocu-mentsbytheiroverallsentiment,forexampledeter-miningwhetherareviewispositiveornegative(e.g.,(Turney,2002;Daveetal.,2003;PangandLee,2004;Beinekeetal.,2004)).Incontrast,ourex-perimentsclassifyindividualwordsandphrases.Anumberofresearchershaveexploredlearningwordsandphraseswithpriorpositiveornegativepolarity(anothertermissemanticorientation)(e.g.,(Hatzi-vassiloglouandMcKeown,1997;KampsandMarx,2002;Turney,2002)).Incontrast,webeginwithalexiconofwordswithestablishedpriorpolarities,andidentifythecontextualpolarityofphrasesinwhichinstancesofthosewordsappearinthecor-pus.Tomaketherelationshipbetweenthattaskandoursclearer,notethatsomewordlistsusedtoevaluatemethodsforrecognizingpriorpolarityareincludedinourprior-polaritylexicon(GeneralIn-quirerlists(General-Inquirer,2000)usedforevalu-ationbyTurney,andlistsofmanuallyidentiedpos-itiveandnegativeadjectives,usedforevaluationbyHatzivassiloglouandMcKeown).Someresearchclassiesthesentimentsofsen-tences.YuandHatzivassiloglou(2003),KimandHovy(2004),HuandLiu(2004),andGrefenstetteetal.(2001)4allbeginbyrstcreatingprior-polaritylexicons.YuandHatzivassiloglouthenassignasen-timenttoasentencebyaveragingthepriorsemanticorientationsofinstancesoflexiconwordsinthesen-tence.Thus,theydonotidentifythecontextualpo-larityofindividualphrasescontainingclues,aswe 4In(Grefenstetteetal.,2001),theunitsthatareclassiedarexedwindowsaroundnamedentitiesratherthansentences.