edu sarakathy cscolumbiaedu Abstract We introduce a new corpus of sentencelevel agreement and disagreement annotations over LiveJournal and Wikipedia threads This is the rst agreement corpus to offer fulldocument annotations for threaded discussions ID: 77584
Download Pdf The PPT/PDF document "Annotating Agreement and Disagreement in..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
c1ThereseemstobeamuchbetterlistattheNationalCancerInstitutethantheonewe'vegot.Ittiesmuchbetterto theactualpublication(thesame11sections,inthesameorder).I'dliketoreplacethatsectioninthisarticle.Any objections? c2Notaproblem.Perhapswecanalsoinserttherelativeincidenceaspublishedinthismonth'swikiBloodjournal c3I'vemadetheupdate.I'veincludedtemplatelinkstoasourcethatsupportslookingupinformationby ICD-Ocode. c4CanArcadiantellmewhyhe/sheincludedtheleukemiaclassicationtothislymphomapage?Itisnoteven listedintheWikipedialeukemiapage!IvotefordividingtheWHOclassicationinto4partsin4distinctpages: leukemia,lymphoma,histocyticandmastocyticneoplasms.LetmeknowwhatyouthinkbeforeIdeletethem. c5Emmanuelm,aren'tyouthepersonwhoaddedthoseothercategorieson6July2005? c6Arcadian,IaddedonlythelymphomaportionoftheWHOclassication.Youaddedtheleukemiason Dec29th.Wouldyoumindmovingtheleukemiaportiontotheleukemiapage c7Oh,andpleasenotethatIwouldbeverycomfortablewitha“cross-coverage”oflymphocyticleukemiasin bothpages.Mycommentisreallyaboutmyeloid,histiocyticandmastcellneoplasmswhosharenoreal relationshipwithlymphomas. c8Tosimplifythediscussion,Ihaverestoredthatsectiontoyourversion.Youmaymakeanyfurtheredits, andIwillhavenoobjection. Table1:ExamplesofagreementanddisagreementinaWikipediadiscussionforum.DirectResponse:c2!c1,c6!c5,c8!c7,c6 c1Iwantthisjacket.Because100%silkissopractical,especiallyinahouseholdwithcats.butit'sso,Idon'tknow–raggedylooking!That'sawesome! c2Thatjacketisgorgeous.Impractical,waytooexpensiveforthelook,andprettymuchgorgeous.guh. c3Iknoooooow,andyou'renothelping.:) c4Monday!WHEE!Itisabitraggedylooking.Ithinkit'sbecauseoftheties. c5Wow,thatjacketlooksreallynice...IwishIcouldaffordit! Table2:ExamplesofaagreementinaLiveJournalweblog.DirectResponse:c2!c1,c5!c1,DirectParaphrase:c4!c1,IndirectParaphrase:c5!c23.AnnotationGuidelinesAthreadconsistsofasetofpostsorganizedinatree.Weusestandardterminologytorefertothestructureofthistree(soeveryposthasasingle“parent”towhichitreplies,andallnodesdescendfromasingle“root”).Eachpostismarkedwithatimestampandanauthor,astring(its“body”).Agreementannotationisperformedonpairsofsentencesfs;tg,whereeachsentenceisasubstringofthebodyofapost.sisreferredtoasthe“antecedentsentence”,andtasthe“reactionsentence.”Theantecedentsentenceandreac-tionsentenceoccurindifferentpostswrittenbydifferentauthors.Annotationsareimplicitlydirectedfromreactiontoantecedent;thereactionisalwaysthesentencefromthepostwiththelatertimestamp.Annotationsbetweenpairsofpostswiththesameauthorareforbidden.Eachpairisalsoannotatedwithatype.TypeEachsentencepaircanbeofeithertypeagreementordis-agreement.Twosentencesareinagreementwhentheypro-videevidencethattheirauthorsbelievethesamefactoropinion,andindisagreementotherwise.ModeModeindicatesthemannerinwhichagreementordisagree-mentisexpressed.Broadly,apairofpostsareina“direct”relationshipifoneisanancestoroftheother,andindirectotherwise;theyareina“response”relationshipifoneex-plicitlyacknowledgesaclaimmadeintheother,and“para-phrase”otherwise.Morespecically:Directresponse:Thereactionauthorexplicitlystatesthattheyareinagreementordisagreement,e.g.bysaying“Iagree”or“No,that'snottrue.”Anagree-ment/disagreementisonlyadirectresponseifitisadi-rectreplytoitsclosestancestor,i.e.itsparent.Forex-ample,inTable2,thereactionsentence“Iknooooow.”inc3isadirectresponsetotheantecedentsentence“Thatjacketisgorgeous.”inc2.InTable1,thereac-tionsentence“Arcadian,IaddedonlythelymphomaportionoftheWHOclassication.”inc6isadirectdisagreementtothesentence“Emmanuelm,arentyouthepersonwhoaddedthoseothercategorieson6July2005?”inc5.Directparaphrase:Thereactionauthorrestatesaclaimmadeinanancestorpost.Anagreement/disagreementisonlyadirectparaphraseifitisadirectrewordingofitsclosestancestor,i.e.itsparent.Forexample,inTable2,thesentence“Itisabitraggedylooking.”inc4isadirectparaphraseofthesentence“butit'sso,Idon'tknow–raggedylooking!”ofitsparent,c1.Indirectresponse:Thereactionisadirectresponsetoaclaim,butthepostdoesnotdescendfromthesource.Thisoftenoccurswhentheauthorpressedthe“reply” AnnotatingAgreementandDisagreementinThreadedDiscussionJacobAndreas,SaraRosenthal,KathyMcKeownColumbiaUniversitysara,kathyWeintroduceanewcorpusofsentence-levelagreementanddisagreementannotationsoverLiveJournalandWikipediathreads.Thisistherstagreementcorpustoofferfull-documentannotationsforthreadeddiscussions.Weprovideamethodologyforcodingresponsesaswellasanimplementedtoolwithaninterfacethatfacilitatesannotationofaspecicresponsewhileviewingthefullcontextofthethread.Boththeresultsofanannotatorquestionnaireandhighinter-annotatoragreementstatisticsindicatethattheannotationscollectedareofhighquality.Keywords:annotation,agreement,onlinediscussion1.IntroductionTheidenticationofagreementanddisagreementamongparticipantsinadiscussionhasbeenwidelystudied.Theproblem,however,suffersfromapaucityofdata.Inor-dertodevelopsystemsthatcanrecognizewhenadialogparticipantagreesordisagreeswithapreviousspeaker,cor-porathatcontaingoldstandardannotationsidentifyingwhoagrees(ordisagrees)withwhomareneeded.Atpresent,onlyasmallnumberofcorporacoveringalimitedrangeofdiscussionformatsexist.Asisusuallythecase,itisdif-culttoextendsystemstrainedonthesecorporatonewtasksordifferently-formatteddata,particularlywhendiscussionsinthetargetdatahaveafundamentallydifferentstructurefromthatoftheavailabletrainingdata.Inparticular,almostallexistingcorporacoverlineardis-cussions,typicallytranscriptionsofmeetings,inwhichthesentencesorutterancesinthediscussionproceedinasin-glechainwithadeniteorder.Suchcorpora,whilegoodformodelingdiscussionsthattakeplaceinrealtime,areill-suitedtolearningpatternsfoundinconversationstakingplaceonInternetmessageboardsorblogs.Inonlinefo-rums,discussionsaretypicallystructuredasthreads,tree-shapedstructuresinwhichmultiplepostscansharethesameparent.Infact,itisoftenthecasethatasinglepostmayelicitmanycomments,whichcaneitherrespondtotheinitialposterortooneofthecommentsonthepost.Whilethereisadeniteunderlyingtemporalordering(thesequenceinwhichpostswerecreated),themostimportantfeatureisasetofexplicitedgesbetweenpostswhichex-plainwhoisrespondingtowhom.Inthispaper,weintroduceacorpusofagreementanddisagreementannotationsontwodifferentonlinesources:Wikipediadiscussionforums,andLiveJournalweblogs.Thiscorpus,whichwebelievetobetherstofitskind,providesasourceofagreementanddisagreementdatabothforanewsource(theInternet)andanewdocumentstruc-ture(threadeddiscussion).Wepresenttheannotationguidelinesthatwereusedtocre-atethiscorpus,andstatisticsabouttheannotationsthatwerecollected.Wealsodiscusstheprocessbywhichthecorpuswascreated,highlightingtheannotationtoolwecre-atedforthistaskandtheresultsofquestionnairesthatwerepresentedtotheannotators.2.RelatedWorkVariouscorporaandannotationtoolsformorerestrictedagreement/disagreementannotationtasksexist.DAMSL(AllenandCore,1997)isadialogactannotationschemeandtoolusedtoannotatevariouskindsofcommunica-tion,includingagreement,forspeechtranscripts.TheICSImeetingcorpus(Janinetal.,2003;Shribergetal.,2004)hassimilarlybeenannotated(Galley,2007)foragreementandotherdialogacts.Asdiscussed,theselineartranscriptsdonotgeneralizewelltothreadeddocuments.PerhapsthemostsimilarworktooursisthatofAbbottetal(2011);theyidentifydisagreementinpoliticalblogsonpairsusingonlylexicalfeatures.Incontrast,ourannotationtoolexploresseveralformsofagreementanddisagreementandaskstheannotatortotakeintoaccountthecontextofthephrasesbyprovidingtheentiredocument(whichwasanoptionalfeatureintheirannotation).Inotherrelatedwork,Benderetal(2011)annotateWikipediadiscussionforumsforpositiveandnegativealignmentmoveswhichexpressagreementanddisagreementrespectivelybetweenthesourceandtarget.Theirannotationincludespraise,doubt,andsarcasminadditiontoexplicitagreementanddisagreement.Theydidnothaveanannotationtool,butsimplyhadtheannotatorsannotatethedocumentsdirectly.Ourannotationtoolcouldbeeasilymodiedfortheirap-Thisprojectisdirectlymotivatedbythelackofadequatedatafortraininganagreementmoduleinalargerprojectaimedatidentifyinginuentialparticipantsandsubgroupformationinonlinemessageboards;weexpectthatagree-mentclassierstrainedonthisdatawillbeusefulforawidevarietyofhigher-leveldiscourseanalysistaskslikeours.Thereisalreadyagreatdealofworkontheprob-lemoflabelingindividualutterancesina(linear)meetingtranscript;wenoteinparticulartheworkof(Galleyetal.,2004),whichfocusedonidentifyingadjacencypairs,andasimilarpaperby(Hillardetal.,2003),whichstudiedthesametaskusingareducedfeatureset.Morerecentworkonagreement/disagreementdetectionincludes(Hahnetal.,2006;GermesinandWilson,2009;Wangetal.,2011).Wehopethatthiscorpuswillenableageneralizationof(Galleyetal.,2004)'sworktootherdocumentstructures. 818 ThereseemstobeamuchbetterlistattheNationalCancerInstitutethantheonewe'vegot.Ittiesmuchbetterto theactualpublication(thesame11sections,inthesameorder).I'dliketoreplacethatsectioninthisarticle.Any objections? Notaproblem.Perhapswecanalsoinserttherelativeincidenceaspublishedinthismonth'swikiBloodjournal I'vemadetheupdate.I'veincludedtemplatelinkstoasourcethatsupportslookingupinformationby ICD-Ocode. CanArcadiantellmewhyhe/sheincludedtheleukemiaclassicationtothislymphomapage?Itisnoteven listedintheWikipedialeukemiapage!IvotefordividingtheWHOclassicationinto4partsin4distinctpages: leukemia,lymphoma,histocyticandmastocyticneoplasms.LetmeknowwhatyouthinkbeforeIdeletethem. Emmanuelm,aren'tyouthepersonwhoaddedthoseothercategorieson6July2005? Arcadian,IaddedonlythelymphomaportionoftheWHOclassication.Youaddedtheleukemiason Dec29th.Wouldyoumindmovingtheleukemiaportiontotheleukemiapage Oh,andpleasenotethatIwouldbeverycomfortablewitha“cross-coverage”oflymphocyticleukemiasin bothpages.Mycommentisreallyaboutmyeloid,histiocyticandmastcellneoplasmswhosharenoreal relationshipwithlymphomas. Tosimplifythediscussion,Ihaverestoredthatsectiontoyourversion.Youmaymakeanyfurtheredits, andIwillhavenoobjection. Table1:ExamplesofagreementanddisagreementinaWikipediadiscussionforum.DirectResponse:c,c Iwantthisjacket.Because100%silkissopractical,especiallyinahouseholdwithcats.butit'sso,Idon'tknow–raggedylooking!That'sawesome! Thatjacketisgorgeous.Impractical,waytooexpensiveforthelook,andprettymuchgorgeous. Iknoooooow,andyou'renothelping.:) Monday!WHEE!Itisabitraggedylooking.Ithinkit'sbecauseoftheties. Wow,thatjacketlooksreallynice...IwishIcouldaffordit! Table2:ExamplesofaagreementinaLiveJournalweblog.DirectResponse:c,c,DirectParaphrase:c,IndirectParaphrase:c3.AnnotationGuidelinesAthreadconsistsofasetofpostsorganizedinatree.Weusestandardterminologytorefertothestructureofthistree(soeveryposthasasingle“parent”towhichitreplies,andallnodesdescendfromasingle“root”).Eachpostismarkedwithatimestampandanauthor,astring(itsAgreementannotationisperformedonpairsofsentencess;t,whereeachsentenceisasubstringofthebodyofaisreferredtoasthe“antecedentsentence”,andthe“reactionsentence.”Theantecedentsentenceandreac-tionsentenceoccurindifferentpostswrittenbydifferentauthors.Annotationsareimplicitlydirectedfromreactiontoantecedent;thereactionisalwaysthesentencefromthepostwiththelatertimestamp.Annotationsbetweenpairsofpostswiththesameauthorareforbidden.Eachpairisalsoannotatedwithatype.TypeEachsentencepaircanbeofeithertypeagreementordis-agreement.Twosentencesareinagreementwhentheypro-videevidencethattheirauthorsbelievethesamefactoropinion,andindisagreementotherwise.Modeindicatesthemannerinwhichagreementordisagree-mentisexpressed.Broadly,apairofpostsareina“direct”relationshipifoneisanancestoroftheother,andindirectotherwise;theyareina“response”relationshipifoneex-plicitlyacknowledgesaclaimmadeintheother,and“para-phrase”otherwise.Morespecically:Directresponse:Thereactionauthorexplicitlystatesthattheyareinagreementordisagreement,e.g.bysaying“Iagree”or“No,that'snottrue.”Anagree-ment/disagreementisonlyadirectresponseifitisadi-rectreplytoitsclosestancestor,i.e.itsparent.Forex-ample,inTable2,thereactionsentence“Iknooooow.”incisadirectresponsetotheantecedentsentence“Thatjacketisgorgeous.”inc.InTable1,thereac-tionsentence“Arcadian,IaddedonlythelymphomaportionoftheWHOclassication.”incisadirectdisagreementtothesentence“Emmanuelm,arentyouthepersonwhoaddedthoseothercategorieson6JulyincDirectparaphrase:Thereactionauthorrestatesaclaimmadeinanancestorpost.Anagreement/disagreementisonlyadirectparaphraseifitisadirectrewordingofitsclosestancestor,i.e.itsparent.Forexample,inTable2,thesentence“Itisabitraggedylooking.”isadirectparaphraseofthesentence“butit'sso,Idon'tknow–raggedylooking!”ofitsparent,cIndirectresponse:Thereactionisadirectresponsetoaclaim,butthepostdoesnotdescendfromthesource.Thisoftenoccurswhentheauthorpressedthe“reply” 819 Figure1:Schematicoftheannotationtool:Theleftsideshowsthecontrolsusedfornavigationandtherightdis-playsthecurrentthread.buttononapostotherthantheonetheywereattempt-ingtorespondto(thiswouldbethecaseif,forexam-ple,cdescendedfromcinsteadofcabove).Or,perhapsitisintendedtoanswermorethanoneprevi-ouspost.Thereactionofanindirectresponseshouldbethesinglesentencewrittenclosestintimetoitsan-Indirectparaphrase:Thereactionrestatesaclaimmadeinapostthatisearlierintime,butnotanancestorofthepost.Thereactionofanindirectparaphrasean-notationbethesinglesentencewrittenclosestintimetoitsantecedent.Forexample,inTable2,cisanindirectparaphraseofc4.TheAnnotationProcessInrecentyearstherehasbeenanincreasinglypopulartrendtouseAmazon'sMechanicalTurktolabeldata.Studieshaveshown(Snowetal.,2008)thatMechanicalTurkusersareabletoproducehighqualitydatathatiscomparabletoexpertannotatorsforsimplelabelingtasksthatcanbecompletedinafewsecondssuchasaffectivetextanaly-sisandwordsensedisambiguation.However,othershaveshown(Callison-BurchandDredze,2010)thattheanno-tationsareconsiderablylessreliablefortasksrequiringasubstantialamountofreadingorinvolvingcomplicatedan-notationschemes.Ourannotationtaskwasdifcultforsev-eralreasons;observationoftheentirethreadwasneces-sarytoannotateeachedgeandtheannotationsthemselveswerefairlyinvolved.Therefore,wedecidedtorelyontwotrainedannotatorsratherthanalargenumberofuntrainedBothannotatorswereundergraduates,andneitherhadanypreviousexperiencewithNLPannotationtasks.Theyweretrainedtousetheweb-basedannotationtooldescribedinthefollowingsectionforapproximatelyonehour;theythenannotatedtheremainderofthecorpusontheirown.4.1.TheAnnotationToolTheweb-basedannotationtool(Fig.1and2)isusedtopro-videasimpleandeasywaytoannotatethreads.Theinter-faceconsistsoftwoparts;thelefthand-sidewhichcontains Field Value DocumentID 5 Annotator JohnDoe AntecedentID 13 ReactionID 11 Antecedent Itdoesseemheavilycensored Reaction Um,Ican`thelpbutnoticethatthisarticleseemsheavilycensored. Type agreement Mode indirectparaphrase Table3:Sampleannotatoroutputcontrolstonavigatethroughthreadsandaddagreements,andtheright-handsidewhichdisplaysthecurrentthread.Thedocumentisdisplayedusingitsthreadstructureindi-catedbybothindentationandboxesnestingchildrenundertheirparentsasshowninFigure1.Toclearlydifferentiatebetweenpossiblesentences,eachsentenceisdisplayedonitsownline.Annotatorsbeginbyselectingtheindividualsentencesfromtheantecedentandreactionwhichprovideevidenceofagreementordisagreement,andthenmarktypeandmodeusingthepostcontrolsontheleft-handside.(Whilethere-sponse/paraphraseannotationmustbeencodedbyhand,thedirect/indirectdistinctionisinferredautomaticallyfromthedocumentstructure.)Eachpostpairthatisaddedappearsasasavedpostontheleft-handsideofthetooldirectlybe-lowthepostcontrols.Savedpostscanberemovediftheyweremistakenlyadded.Figure2showstheannotationtoolinuse.ThesystemautomaticallypreventsusersfromannotatingtheforbiddencasesmentionedinSection3.,suchasthean-tecedentandreactionsentencesbeingwrittenbythesameauthor.Italsoautomaticallydeterminestheantecedentandreactionoftheannotationbasedonthetimestampsofthetwopostsinvolved.TheannotationtooloutputsaCSVle(Table3)encodingtheannotatorID,thedocumentIDandthepoststructureasaJSONarraywithentriesforeachannotatedarc.5.UserStudiesAftercompletingtheirportionofthetask,annotatorswereaskedtolloutabriefquestionnairedescribingtheirex-perience(Fig.3).Theyreportedthattheannotationtoolwas“easytouse”and“effective”,thattheannotationtaskwas“interesting”,andthattherewere“norealchallenges”inannotating.Theyreportedthatbetweenthetwogenres,theLiveJournalentrieswerebothconceptuallyeasiertoan-notateandrequiredlesstime,primarilybecausethepostswereshorterinlength.AnnotatorsreportedthatLiveJour-nalentriesrequiredanaverage2to10minutestoanno-tate,whileWikipediadiscussionsrequired10to20min-utes.Annotatorsweredividedintheiropinionsonwhetheragreementsordisagreements,anddirectorindirectwereeasiertoidentifyindicatingthatitisamatterofpersonalTheseresponseshavegivenuscondencethattheannota-tiontoolsucceededinitspurpose(ofsimplifyingthedata 820 Figure2:Screenshotoftheannotationtoolinuse.1.Didyoundthetooleasytouse?2.Whatchallengesdidyouencounterwhenusingthetool?3.WeretheLiveJournalorWikipediadiscussionseasierto4.WeretheLiveJournalorWikipediadiscussionsfasterto5.Didyouhaveanypreviousexperiencewithannotation?6.Whatwasthelearningcurveassociatedwiththetask?7.Onaverage,howlongdidittakeyoutocompleteasingleLiveJournaldiscussion?AWikipediadiscussion?8.Wasiteasiertondagreementsordisagreements?9.Wasiteasiertonddirectorindirectagree-10.Whatisyourgeneralopinionaboutthetask?11.Isthereanythingelsethatyouwouldliketoletusknowaboutwithregardtothisannotationtask? Figure3:Annotatorquestionnairecollectionprocessforthiscorpus),andthatitwillbeeasytofurtherexpandthecorpusifwerequireadditionaldata.6.CorpusDocumentsinthecorpuscamefromtwosources:WikipediaandLiveJournal.Wikipediais,afree,collaboratively-editedencyclopediawhichrecordsconver-sationsamongeditorsaboutpagecontent,andLiveJour-nalis,ajournalingwebsitewhichallowsthreadeddiscus-sionabouteachposting.Inordertoensurethatthecor-puscontainsdocumentsinwhichsubstantialconversationtakesplace,bothofthesesourceswereinitiallylteredforIndirectDirectTot. LiveJournalagreement143236379disagreement146680total157302459 Wikipediaagreement30111141disagreement38172210total68283351Table4:Thenumberordirectandindirectresponsesinthethreadswhere#ofposts #ofparticipantsAtthetimeofpublication,oneannotatorhadlabeled92documentsandtheother109.Intotal,118uniquedocumentswerelabeled;the83documentsannotatedincommonwereusedtodetermineaninter-annotatoragree-mentstatistic.Restrictedjusttothethree-classagree-ment/disagreement/nonelabelingtaskonalledges,theirannotationshadCohen's=0,indicatingsub-stantialagreement.Consideringthemoregranularve-classlabelingtaskthatdistinguishesbetweenagreement-response,agreement-paraphrase,disagreement-responseanddisagreement-paraphrase,wehave=0,alsoindi-catingsubstantialagreement.Table4showsthebreakdownofdirectvs.indirectresponsesinthecorpus.Ingeneral,weobservedthatagreaterpercentageofthepostsinLiveJournalentriesparticipatedinagree-ment/disagreementrelationswhileWikipediaarticlestendedtohaveahigherdisagreement/agreementratio.7.ConclusionWehaveintroducedanewcorpusofagreementanddis-agreementannotationsoverthreadedonlinediscussions. 821 Inadditiontotheresultingcorpus,wehavealsopro-videdamethodologyforlabelingonlinediscussionswhichmakesuseofthreadstructureanddistinguisheswhetheragreement/disagreementwasdirectlystatedorconveyedbymeansofaparaphraseoftheoriginalpost.Thecorpuswascollectedusinganeasy-to-useannotationtoolbyapairoftrainedannotators.Inter-annotatoragreementshowedsub-stantialagreementforboththethree-classandve-classla-belingtasks,withKappaof.67andabove.Theannotationprocessisongoing,andweplantoreleasethecompletecorpusatalatertime.8.AcknowledgementsThisresearchwasfundedbytheOfceoftheDirectorofNationalIntelligence(ODNI),IntelligenceAdvancedRe-searchProjectsActivity(IARPA),throughtheU.S.ArmyResearchLab.Allstatementsoffact,opinionorconclu-sionscontainedhereinarethoseoftheauthorsandshouldnotbeconstruedasrepresentingtheofcialviewsorpoli-ciesofIARPA,theODNIortheU.S.Government.9.ReferencesRobAbbott,MarilynWalker,PranavAnand,JeanE.FoxTree,RobesonBowmani,andJosephKing.2011.Howcanyousaysuchthings?!?:Recognizingdisagree-mentininformalpoliticalargument.InProceedingsoftheWorkshoponLanguageinSocialMedia(LSM2011)pages2–11,Portland,Oregon,June.AssociationforComputationalLinguistics.JamesAllenandMarkCore.1997.DraftofDAMSL:Dialogactmarkupinseverallayers.UnpublishedEmilyM.Bender,JonathanT.Morgan,MeghanOxley,MarkZachry,BrianHutchinson,AlexMarin,BinZhang,andMariOstendorf.2011.Annotatingsocialacts:au-thorityclaimsandalignmentmovesinwikipediatalkpages.InProceedingsoftheWorkshoponLanguagesinSocialMedia,LSM'11,pages48–57,Stroudsburg,PA,USA.AssociationforComputationalLinguistics.ChrisCallison-BurchandMarkDredze.2010.Creatingspeechandlanguagedatawithamazon'smechanicalProceedingsoftheNAACLHLT2010WorkshoponCreatingSpeechandLanguageDatawithAmazonsMechanicalTurkLosAngeles,(June):1–12.MichelGalley,McKeown,Kathleen,JuliaHirschberg,andElizabethShriberg.2004.Identifyingagreementanddisagreementinconversationalspeech:Useofbayesiannetworkstomodelpragmaticdependencies.InProceed-ingsoftheACLMichelGalley.2007.Incorporatingdiscourseandsyntac-ticdependenciesintoprobabilisticmodelsforsumma-rizationofmultipartyspeech.Ph.D.thesis,ColumbiaUniversity,NewYork,NY,USA.Adviser-Mckeown,,KathleenR.SebastianGermesinandTheresaWilson.2009.Agree-mentdetectioninmultipartyconversation.InMLMI'09,November.SangyunHahn,RichardLadner,andMariOstendorf.2006.Agreement/disagreementclassication:exploitingunla-beleddatausingcontrastclassiers.InInHLT/NAACLDustinHillard,MariOstendorf,andElizabethShriberg.2003.Detectionofagreementvs.disagreementinmeet-ings:Trainingwithunlabeleddata.InProceedingsoftheHLT-NAACLConferenceAdamJanin,DonBaron,JaneEdwards,DanEllis,DavidGelbart,NelsonMorgan,BarbaraPeskin,ThiloPfau,ElizabethShriberg,AndreasStolcke,andChuckWoot-ers.2003.Theicsimeetingcorpus.pages364–367.ElizabethShriberg,RajDhillon,SonaliBhagat,JeremyAng,andHannahCarvey.2004.Theicsimeetingrecorderdialogact(mrda)corpus.InMichaelStrubeandCandySidner,editors,Proceedingsofthe5thSIG-dialWorkshoponDiscourseandDialogue,pages97–100,Cambridge,Massachusetts,USA,April30-May1.AssociationforComputationalLinguistics.RionSnow,BrendanO'Connor,DanielJurafsky,andAn-drewY.Ng.2008.Cheapandfast—butisitgood?:eval-uatingnon-expertannotationsfornaturallanguagetasks.ProceedingsofEMNLP,pages254–263.WenWang,SibelYaman,KristinPrecoda,ColleenRichey,andGeoffreyRaymond.2011.DetectionofAgreementandDisagreementinBroadcastConversations.InPro-ceedingsofthe49thAnnualMeetingoftheAssociationforComputationalLinguistics:HumanLanguageTech-nologies,pages374–378,Portland,Oregon,USA,June.AssociationforComputationalLinguistics. 822