LING287CS424PStanfordPottsSentimentlexicons33MicroWNOpCerinietal2007Documentationanddownloadhttpwwwunipvitwnop PosNegSynset 110truea2reala4210illustriousa1famousa13050real ID: 515934
Download Pdf The PPT/PDF document "LING287/CS424P,Stanford(Potts)Sentimentl..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
LING287/CS424P,Stanford(Potts)Sentimentlexicons3.2WordNet(Miller1995;Fellbaum1998)Download/documentation:http://wordnet.princeton.edu/Webinterface:http://wordnetweb.princeton.edu/perl/webwnAPIs:http://wordnet.princeton.edu/wordnet/related-projects/•idle,a–Synset:idlea01Denition:notinactionoratworkExamples:fanidlelaborer,idledrifters,theidlerich,anidlemindgAlsosees:fineectivea01,unemployeda01gSimilartos:funengageds01,...,bone-idles01g...Lemma:idlea01idle·Antonyms:fbusya01busyg·Derivationallyrelatedforms:ffaineancen01idlenessg·Pertainyms:fg·...–Synset:baselesss01Denition:withoutabasisinreasonorfactExamples:fbaselessgossip,unfoundedsuspicions,...,unwarrantedjealousygAlsosees:fgSimilartos:funsupporteda01g...Lemma:baselesss01baseless·Antonyms:fg·Derivationallyrelatedforms:fg·Pertainyms:fg·...Lemma:baselesss01groundless·...–...Table3:Example—fromstringstolexicalstructure.FieldsmarkedwiththeemptysethappentobeemptyforthisexamplebutcanbelledInWordNet3.0(Miller2009),thestringidlewithcategoryahas7synsets.Collectively,thosesynsetshave18lemmas.Eachsynsetandeachlemmacanreachothersynsets(lemmas)viarelationslikethoseexempliedabove...(seesec.5.1).OpenThesaurus(German):http://www.openthesaurus.de/Moreglobally:http://www.globalwordnet.org/(withfreedownloadsforatleastArabic,Danish,French,Hindi,Russian,Tamil)3 LING287/CS424P,Stanford(Potts)Sentimentlexicons3.3Micro-WNOp(Cerinietal.2007)Documentationanddownload:http://www.unipv.it/wnop PosNegSynset 110truea2reala4210illustriousa1famousa1...30.50reala6tangiblea240.250existenta2reala150.1250.125reala2...11000demandv6 (a)`Common':Fiveevaluatorsworkingtogether,110synsets. Evaluator1Evaluator2Evaluator3Pos1Neg1Pos2Neg2Pos3Neg3Synset 1101010gooda15wella2210100.750sweet-smellinga1perfumeda2...3101010gooda23unspoilta1unspoileda140.500.2500.250hota16...4960.50100.50healv3bring_aroundv2curev1 (b)`Group1':Threeevaluatorsworkingseparately,496synsets.Completeagreementon197(40%).Polarityagreement(sign(Pos1�Neg1)=sign(Pos2�Neg2)=sign(Pos3�Neg3))on387(78%). Evaluator1Evaluator2Pos1Neg1Pos2Neg2Synset 10101forlorna1godforsakena2lorna1desolatea220101rottena231010intimatea2cozya2informala440000federala1...4990000termv1 (c)`Group2':Twoevaluatorsworkingseparately,499synsets.Completeagreementon395(79%).Polarityagreement(sign(Pos1�Neg1)=sign(Pos2�Neg2))on471(90.4%).Table4:ThethreegroupsoftheMicro-WNOp.ThestringslistedunderSynsetidentifylemmas.4 LING287/CS424P,Stanford(Potts)Sentimentlexicons PosNegObjTotal a(adj)624550157n(noun)3058273361r(adv)402428v(verb)273693156 Total330139440 702 (a)Bysynset. PosNegObjTotal a15113589375n83144663890r905362v87112233432 Total3303911,038 1,759 (b)Bylemma. PosNegObjTotal a10511362280n66107483656r703744v6890178336 Total246310760 1,316 (c)Byhstring,tagipair. (d)Thedistributionofscores,bylemma.Objectivityvaluesare1�(Pos+Neg),soascoreof1theremeansthatbothPositiveandNegativewere0.Relativelyfewofthesynsetsreceivednon-0/1ratings.Table5:Micro-WNOplimitedtothe702synsetsonwhichallannotatorsagreedexactlyonallvalues(tab.5(a)).Posmeansthatthepositivescorewashigher,Negthatthenegativescorewashigher,andObjthatthetwoscoreswerethesame(eveniftheyweregreaterthan0).3.4POStags:relatingHarvardInquirerandWordNet/Micro-WNOpmatches`noun'(case-insensitive)7!nmatches`verb'or`supv'(case-insensitive)7!vmatches`modif'(case-insensitive)7!ais`LY'ormatches`LY'(case-sensitive)7!rmatchesnoneoftheabovenochangeTable6:PartialmappingfromInquirercategoriestoWordNetcategories.Themappingispartialfortworeasons.First,Inquirerincludesdeterminers,prepositions,andotherpartsofspeechnotfoundinWordNet(andhencenotrepresentedatallinMicro-WNOp).Second,evensettingthoseclassesaside,Inquirerparts-of-speecharehighlysubcategorizedanddon'tnecessarilyfallintothefour-wayWordNettypology.5 LING287/CS424P,Stanford(Potts)Sentimentlexicons4AfewnotesonassessmentPredictedPosNegObjPos 15 10100ObservedNeg10 15 10Obj10100 1,000 sum_diagonal sum_all_cells (a)Accuracy.Typicallyappropriateonlyifthecategoriesareequalinsize.PredictedPosNegObjPos 15 10 100ObservedNeg 10 15 10Obj 10100 1,000 true_positive true_positive+false_positive (b)Precision:thecorrectguessespenalizedbythenumberofincorrectguesses.YoucanoftengethighprecisionforacategoryCbyrarelyguessingC,butthiswillruinyourrecall.PredictedPosNegObjPos 15 10 100ObservedNeg10 15 10Obj 10 100 1,000 true_positive true_positive+false_negative (c)Recall:thecorrectguessespenalizedbythenum-berofmisseditems.YoucanoftengethighrecallforacategoryCbyalwaysguessingC,butthiswillruinyourprecision.Table9:Confusionmatrices.Thesevaluesshouldallbeinterpretedasboundedby[0,1].(Ifyouaretemptedtodivideby0,return0.)Denition1(F1).2precision*recall precision+recallNote:F1givesequalweighttoprecisionandrecall.However,dependingonwhatyou'restudying,youmightvalueonemorethantheother.Denition2(Macroaverage).Averageprecision/recall/F1overalltheclassesofinterest.Denition3(Microaverage).Sumcorrespondingcellstocreatea22confusionmatrix,andcalculateprecision=recall=F1intermsofthatnewmatrix.PosNegObjPos1510100Neg101510Obj101001,000)264PosOtherPos15110Other201,125NegOtherNeg1520Other1101,125ObjOtherObj1,000110Other11050375)YesNoYes1,030240No2402,300Note:Beforeusingthesesummarymeasures,lookatthedistributionofyourby-categorynumberstomakesurethatyou'renotlosingimportantstructure.Formore:ManningandSchütze1999:§8.1;Manningetal.2009:§8.3,13.6.7 LING287/CS424P,Stanford(Potts)Sentimentlexicons5WordNet-basedapproaches5.1Simplesense/sentimentpropagationHypothesis:Sentimentisconstantthroughoutregionsoflexicallyrelateditems.Thus,sentimentpropertiesofhand-builtseed-setswillbepreservedaswefollowWordNetrelationsoutfromthem.5.1.1Algorithm(Valituttietal.2004;EsuliandSebastiani2006;sec.5.3forprecedents)WORDNETSENSEPROPAGATE(S,iter)InputS:alistofsynsets.Forexample: fbrilliants01,nwin01g,fsadlyr01,grossa01g iter:thenumberofiterationsOutputT:LENGTH(S)1+itersynsetmatrix:0BB@S[1]iter-thpropagationofS[1]......S[LENGTH(S)]iter-thpropagationofS[n]1CCA1initializeT:aLENGTH(S)1+itermatrixsuchthatT[i][1]=S[i]for1¶i¶1+iter2fori 1toiter3forj 1toLENGTH(S)4newSame SAMEPOLARITY(T[j][i])5others SLENGTH(S)k=1T[k][i]fork6=jTheotherseed-setsinthiscolumn.6newDiff OTHERPOLARITY(others)Fortheexperiments,I rstcalculateallthepropagationsetsandtheneliminatetheirpairwiseintersectionfromeach,toensurenooverlap.7T[j][i+1] (newSame[newDiff)8returnTSAMEPOLARITY(synsets)1newsynsets fg2fors2synsetsSynset-levelrelations.3newsynsets newsynsets[fsg[AlsoSees(s)[SimilarTos(s)4forlemma2Lemmas(s)Lemma-levelrelations.5foraltLemma2(DerivationallyRelatedForms(lemma)[Pertainyms(lemma))6newsynsets newsynsets[fSynset(altLemma)g7returnnewsynsetsOTHERPOLARITY(synsets)1newsynsets fg2fors2synsets3forlemma2Lemmas(s)Lemma-levelrelations.4foraltLemma2Antonyms(lemma)5newsynsets newsynsets[fSynset(altLemma)g6returnnewsynsets8 LING287/CS424P,Stanford(Potts)Sentimentlexicons FreevariablesTheseed-sets,theWordNetrelationscalledinSAMEPOLARITYandOTHERPOLARITY,thenumberofiterations,thedecisiontoremoveoverlap. 5.1.2ExperimentFormulateseed-setsbyhand,usingintuitions.Usethealgorithmtoderivepositive,negative,andobjectivesetsofhstring,posipairs.positiveexcellent,good,nice,positive,fortunate,correct,superiornegativenasty,bad,poor,negative,unfortunate,wrong,inferiorobjectiveadministrative,nancial,geographic,constitute,analogy,ponder,material,public,department,measurement,visualTable10:Seed-sets. Figure1:Growthinthesizeofthesetsasthenumberofiterationsgrows. Figure2:Percentageofitemsfromthegoldstandardthatareunreachedbypropagation.9 LING287/CS424P,Stanford(Potts)Sentimentlexicons (a)F1usingdef.4,option(i):“Scoreonlyhstring,posipairsthatareintheintersectionofInquirerWN(Micro-WNOp)withtheunionofthethreederivedsets.” (b)F1usingdef.4,option(ii):“Treatallitemsnotreachedbythepropagationalgorithmasmembersofobjective.” (c)F1usingdef.4,option(iii):“Createacategory`missing'topenalizeeffectivenessmeasuresfortheothercategories.”Figure3:UsingF1toassessthepropagationalgorithmagainstInquirerandMicro-WNOp.Thethreescoringoptionsemphasizedifferentaspectsofthealgorithm.11 LING287/CS424P,Stanford(Potts)Sentimentlexicons5.2.2G08resultsInourexperiments,theoriginalseed-setcontained20neg-ativeand47positivewordsthatwereselectedbyhandtomaximizedomaincoverage,aswellas293neutralwordsthatlargelyconsistofstopwords.[...]Runningthealgorithmresultedinanexpandedsentimentlexiconof5,705positiveand6,605negativewords,someofwhichareshowninTable2withtheirnalscores.Adjectivesformnearly90percentoftheinducedvocabulary,followedbyverbs,nounsandnallyadverbs. G08donotstophere.Theygoontousetheirlexiconforclassicationandsummarization.We'llreturntothoseportionsofthepaperinthenexttwoclasses.5.2.3OurexperimentMicro-WNOpprovidesthesetofseed-sets:weusetheentiresetofpositive,negative,andobjectivesetsasdenedintab.5(c).Theideaisthatwewillusethisalgorithm,nottolearnnewsentimentlexicons,butrathertolearnthestrengthrelationshipsbetweenwordsofthesamepolarity. Seed-setAfterpropagation P2461,021N310978M760154,585 Total1,316156,584 Table11:Experimentalresults.Thecombinedseeds-setsaretheentireMicro-WNOpcorpus.ThetotalafterpropagationisthenumberoflemmasinWordNet3.0.ThegaininthepositiveandnegativecategoriesisnotaslargeasG08saw.ThisisalmostcertainlyduetothefactthattheMicro-WNOpcorpusdoesnotcoverasmuchofthedomainastheirseed-setsdid. Figure4:Distributionofnon-0scoresinthenallexicon.Thevastmajorityarecloseto0.Thelargestis185.12(valour)andthesmallestis�61.1(rophy,`streetnamesforunitrazepan'(!?)).13 LING287/CS424P,Stanford(Potts)Sentimentlexicons5.3SimilarapproachesandideasHuandLiu2004Oneofthersttousethiskindofpropagationforsentiment.Theiralgorithmisverysimilartotheoneinsec.5.1.1,thoughitsinputsandoutputsarestringsratherthansynsetsanditrestrictsattentiontoadjectives(positiveandnegative).KimandHovy2006SensepropagationoverWordNetfromsmallseed-sets,dealingwithoverlapviamaximumlikelihood,whichalsodeliverssentimentscores/rankingsforwords(strings).Godboleetal.2007WordNetpropagationrelativetogeneralweblogtopiccategories,withscoresbasedondistancefromseed-setwords,takingintoaccountthedangersofmixedpaths. AndreevskaiaandBergler2006PropagationusingalimitedsetofbasicWordNetrelations,butwithcre-ativeuseoftheglossestofurtherexpandthelexicon.ThewordlistsofHatzivassiloglouandMcKeown1997(seesec.6.2)areusedtoformulatetheseed-sets,andtheHarvardInquireristhegoldstandard. RaoandRavichandran2009LabelpropagationusingWordNetsupplementedwithotherhand-buildresources.EvaluationispartlywithHarvardInquirer.TheapproachisextendedtoHindiandFrenchwithstrongresults.SentiWordNet(EsuliandSebastiani2006;Baccianellaetal.2010)Builtbyrstpropagatingoutfromthepositiveandnegativeseed-setsintab.10andthenbuildingaclassierfromtheresultingsynsets.Thefeaturesintheclassierarenotthesynsetsthemselves,butratherthesynsetsoftheirdenitions,asprovidedbythePrincetonAnnotatedGlossCorpus:2verygood;ofthehighestquality;“madeanexcellentspeech”[...]very3very4good1good3good4high3high4quality1quality3make2made3excellent3speech1http://sentiwordnet.isti.cnr.it/ 2http://wordnet.princeton.edu/glosstag.shtml15 LING287/CS424P,Stanford(Potts)Sentimentlexicons6Distributionalapproaches6.1Web-basedpropagationHypothesis:Thebasichypothesisofsec.5.2iscorrect,andwecanmovepasttheconnesofWordNetbyrelatingdistributionalsimilaritytosentiment.6.1.1Algorithm(Velikovichetal.2010) •Gisacosinesimilaritygraph•PandNareseed-sets• isathreshold:sentimentscoresbelowitwillberoundedto0•Tisthenumberofiterations.Denition6(Cosinesimilarity).LetAandBbevec-torsofco-occurrencescountsforwordswAandwBdenedoveraxed,orderedvocabulary(rowsintab.13(b))oflengthn.Theircosinesimilarity:cosim(wA,wB)def=nPi=1(AiBi) r nPi=1A2ir nPi=1B2i6.1.2Examplesuperbamazingsuperbmoviesuperbmoviesuperbmoviesuperbmoviesuperbmovieamazingmovieamazingmoviecoolsuperb(a)Corpus.0BBBBB@amazingcoolmoviesuperbamazing0021cool0001movie2005superb11501CCCCCA(b)Co-occurrencematrix.0BBBBB@amazingcoolmoviesuperbamazing1.00.450.420.86cool0.451.00.930.0movie0.420.931.00.07superb0.860.00.071.01CCCCCA(c)Cosinesimilaritymatrix.Table13:Examplecorpusandsimilaritymeasures.Fig.5continuestheexample16 LING287/CS424P,Stanford(Potts)Sentimentlexicons (a)Cosinesimilaritygraphforthepositivesub-graphofasmallcorpus.0-valuededgesnotrepresented. (b)Initialmatrix 0withseed-setP=fsuperbg. (c)Matrix 1.AtthispointF=fsuperbg. (d)Matrix 2.Fhasexpandedtoallthenodesing.5(a)reachablefromsuperbinonestep.coolandmoviebenetfromtheirclosenesstoamazing,whichisclosetotheseed-word.Yourvaluesmightbedifferentifyoumodify inplaceasinline6ofthealgorithm,whichinvolvessome nvaluesin ncalculations.Figure5:Theinnerloop(lines3-6)ofthewebpropagationalgorithmwithPfsuperbg.17 LING287/CS424P,Stanford(Potts)Sentimentlexicons6.2.4ThebutruleH&M97“conjunctionsusingbutexhibittheoppositepattern,usuallyinvolvingadjectivesofdif-ferentorientations.Thus,arevisedbutstillsimplerulepredictsadifferent-orientationlinkifthetwoadjectiveshavebeenseeninabutconjunction,andasame-orientationlinkotherwise,assumingthetwoadjectiveswereseenconnectedbyatleastoneconjunction.” Accuracy Theirdata81.81%Ourdata74% (a)Accuracy.Inaddition,Ipickedran-domlybalancedsubgroupswith20,000coordinationtokensineachcategory.Thebaselineisthen50%,buttheaver-ageaccuracyofthebut-ruleover10runswas67%.PredictedDifferentSameDifferent12,20116,170Same10,78766,265(b)Confusionmatrix. PrecisionRecallF1 Different53%43%47%Same47%86%61% (c)Effectiveness(ourdata)Table17:Assessingthebut-rule.6.2.5LogisticregressionCoefcientEstimateStandarderror 0Intercept1.560.01 1Coord=butthecoordinatorisbut�0.850.04 2but-ruleasdescribedinsec.6.2.4�1.370.02 3Conj1Negatedtherstconjuncthasanexternalnegation1.040.15 4Conj1Negatedthesecondconjuncthasanexternalnegation0.620.07 5StemsMatchatleastoneadjiscomplexandthestemsmatch�4.050.37 6Conj1Negated:Conj2Negatedinteraction:bothconjunctsnegatedexternally�3.350.29 Table18:Coefcientestimateswithstandarderrors.Allofthefeaturesarebinary:1ifthedescriptionistrue,else0.Anegativecoefcentindicatesthatthefeaturebiasesinfavorofdifferentorientationfortheadjectives,whereasapositivecoefcientindicatesabiasforsameorientation.Thereareclearlyotherfeaturesandinteractionsthatcouldbetried—datahomework1,problem3,asksyoutostartexploringotheroptions.20 LING287/CS424P,Stanford(Potts)SentimentlexiconsPr(same)=logit�10BBBBBBBBB@1.56+�0.851+�1.371+1.041+0.621+�4.050+�3.35(11)+1CCCCCCCCCA=0.09(a)notout.stand.ingbutnotaw.fulPr(same)=logit�10BBBBBBBBB@1.51+�0.850+1.370+1.040+0.620+�4.050+�3.35(00)+1CCCCCCCCCA=0.82(b)healthyandrobustTable19:ExamplesDenition7(Logisticregressionpredictions).ThepredictioniscorrectifthepolaritydiffersandPr(same)¶0.5orifthepolarityisthesameandPr(same)�0.5PredictedDifferentSameEmpiricalDifferent3,08219,702Same1,62770,376(a)Confusionmatrix. PrecisionRecallF1 Different65%14%22%Same78%98%87% (b)Effectiveness.Table20:Interimresults:Theabovemodelcorrectlyclassies77%ofthesamples,usingdef.7.6.2.6DissimilaritymeasureDenition8(Dissimilarity).disapartialfunctionfrom(wordswords)into[0,1]d(x,y)=8:1.0�themedianttedvalueof(x,y)ifavailable,else1.0�themedianttedvalueof(y,x)ifavailable,else0.5Note:Thettedmodelwilldelivermultiplevaluesforaadjectivepairsthatappearindifferentenvironments.AsfarasIcantell,H&M97don'tspecifyhowtomovefromthesevaluestoauniqueone.Ichosethemedianbecauseitseemstohaveagoodchanceofbeingintherightareaincaseswheretherearemultiplevalues,somesmallandsomelarge.Note:Thisisn'tadistancemeasurebecauseitfailstosatisfythetriangleinequality(thesumofthelengthsofanytwosidesofatriangleisgreaterthanthelengthofthethird).Asaresult,itisn'tappropriateforuseintheobjectivefunctionsforcentroidclusteringalgorithmslikek-means(HatzivassiloglouandMcKeown1993).21 LING287/CS424P,Stanford(Potts)Sentimentlexicons (1�0.4)+(1�0.9)+(1�0.3) 3+(1�0.1)+(1�0.9)+(1�0.01) 3=1.13 (1�0.4)+(1�0.9)+(1�0.3)+(1�0.8)+(1�0.9)+(1�0.2) 4+1�0.9 2=0.675Figure6:Exchangeofexcellent.6.2.8ClusterlabelingH&M97“Theclusteringalgorithmseparateseachcomponentofthegraphintotwogroupsofadjectives,butdoesnotactuallylabeltheadjectivesaspositiveornegative.[...]Wecomputetheaveragefrequencyofthewordsineachgroup,expectingthegroupwithhigheraveragefrequencytocontainthepositiveterms.”HereThefrequenciesforunigramsintheGoogleN-gramscorpus(BrantsandFranz2006).23 LING287/CS424P,Stanford(Potts)Sentimentlexicons FreevariablesThechoiceofclusteringalgorithms,theinherentrandomnessintheini-tialpartition,theorderinwhichthewordsaregonethrough(whichcanleadtodifferentlocaloptima),thenatureofthedictionaryd,whichisdeterminedbythefeaturesinthelogisticregression,thefrequencymeasurethatdeterminesclasslabels. 6.2.9Assessment Table21:H&M97'sresults. Figure7:Assessmentofthecoordinationexperimentwithtwoslightlydifferentmodelts.Theclusterlabelingprocedurewaseffective,inthatnotrialhadunder50%accuracy,whichwouldhavebeenasignthatwegotthelabelsbackwards.Dropsineffectivenessseemcorrelatedwithimbalancesintheinitialrandomsplit,asweseewithtrial6ontheleftandtrial9ontheright.24 LING287/CS424P,Stanford(Potts)SentimentlexiconsReferencesAndreevskaia,AlinaandSabineBergler.2006.MiningWordNetforafuzzysentiment:SentimenttagextractionfromWordNetglosses.InProceedingsoftheEuropeanChapteroftheAssociationforComputationalLinguistics(EACL).Baccianella,Stefano;AndreaEsuli;andFabrizioSebastiani.2010.SentiWordNet3.0:Anen-hancedlexicalresourceforsentimentanalysisandopinionmining.InProceedingsoftheSeventhConferenceonInternationalLanguageResourcesandEvaluation,2200–2204.EuropeanLanguageResourcesAssociation.Blair-Goldensohn,Sasha;KerryHannan;RyanMcDonald;TylerNeylon;GeorgeA.Reis;andJeffReynar.2008.Buildingasentimentsummarizerforlocalservicereviews.InWWWWorkshoponNLPintheInformationExplosionEra(NLPIX).Beijing,China.Brants,ThorstenandAlexFranz.2006.Web1T5-gramversion1.LinguisticDataConsortium,Philadelphia.Cerini,S.;V.Compagnoni;A.Demontis;M.Formentelli;andG.Gandini.2007.Micro-WNOp:Agoldstandardfortheevaluationofautomaticallycompiledlexicalresourcesforopinionmin-ing.InAndreaSansò,ed.,LanguageResourcesandLinguisticTheory:Typology,SecondLanguageAcquisition,EnglishLinguistics.Milan:FrancoAngeliEditore.Esuli,AndreaandFabrizioSebastiani.2006.SentiWordNet:Apubliclyavailablelexicalresourceforopinionmining.InProceedingsofthe5thConferenceonLanguageResourcesandEvaluation,417–422.Genova.Fellbaum,Christiane,ed.1998.WordNet:AnElectronicDatabase.Cambridge,MA:MITPress.Godbole,Namrata;ManjunathSrinivasaiah;andStevenSkiena.2007.Large-scalesentimentanal-ysisfornewsandblogs.InProceedingsoftheInternationalConferenceonWeblogsandSocialMedia.Hatzivassiloglou,VasileiosandKathleenR.McKeown.1993.Towardstheautomaticidenticationofadjectivalscales:Clusteringadjectivesaccordingtomeaning.InProceedingsofthe31stAnnualMeetingoftheAssociationforComputationalLinguistics,172–182.ACL.Hatzivassiloglou,VasileiosandKathleenR.McKeown.1997.Predictingthesemanticorientationofadjectives.InProceedingsofthe35thAnnualMeetingoftheACLandthe8thConferenceoftheEuropeanChapteroftheACL,174–181.ACL.Hu,MinqingandBingLiu.2004.Miningandsummarizingcustomerreviews.InProceedingsofthe10thACMSIGKDDInternationalConferenceonKnowledgeDiscoveryandDataMining,168–177.ACL.Kim,Soo-MinandEduardHHovy.2006.Identifyingandanalyzingjudgmentopinions.InRobertC.Moore;JeffA.Bilmes;JenniferChu-Carroll;andMarkSanderson,eds.,ProceedingsoftheHu-manLanguageTechnologyConferenceoftheNorthAmericanChapteroftheAssociationofCompu-tationalLinguistics,200–207.ACL.26