Q1 Q2 Q3 Q4 Q5 Q6 Q7 S1 45 45 28 40 38 25 42 S2 47 48 15 42 46 22 45 S3 52 51 20 44 46 19 47 S4 49 53 22 47 49 21 47 S5 45 42 19 43 44 28 45 AscanbenotedfromTable1th ID: 185217
Download Pdf The PPT/PDF document "humansdonotagreeonwhatisagoodsummaryofat..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
humansdonotagreeonwhatisagoodsummaryofatext(LinandHovy,2002;HasselandDalianis,2005;Jingetal.,1998),whichmeansthatthereisprobablynotonesinglebestsummary.Theresultspresentedherealsopointtowardstextshavingalimitonimportantsentencesthatshouldbeincludedinsummaries.Somethingthathastobefurtherinvestigated.4.EvaluationEvaluationofthegoldstandardswasconductedbyhavingsubjectsreadthesummariesandansweraquestionnaireonthequalityofthesummary.Thequestionnairesusedsix-pointLikertitemsandcomprisedthefollowingitemsonthesummary:[Q1]...hasagoodlengthtogiveanideaonthecontentintheoriginaltext,[Q2]...isexperiencedtobein-formationrich,[Q3]...isexperiencedasstrenuoustoread,[Q4]...givesagoodideaonwhatiswrittenintheorigi-naldocument,[Q5]...givesagoodunderstandingofthecontentoftheoriginaldocument.[Q6]...isexperiencedasmissingrelevantinformationfromtheoriginaldocument,and[Q7]...isexperiencedasagoodcomplementtotheoriginaldocument.Thesubjectsforourevaluationwhere10studentsand6professionaladministratorsattheSwedishSocialInsuranceAdministration.Allsubjectsreadthesummarybutdidnothavetheorigi-naltextathand,tomoreresemblefutureuseofthesystem.Discoursecoherenceforextractionbasedsummariesis,ofcourse,aproblem.Ourevaluatorswerenotinstructedtodisregarddiscoursecoherencesincethisisafactorwhichhastobeaccountedforwhencreatingtextsofthissort.TheresultsfromthestudentevaluationsarepresentedinTable1.Notethat,astheitemsarestated,ahighscoreisconsideredpositiveonQ1,Q2,Q4,Q5andQ7whereasaslowscoreonQ3andQ6isconsideredpositive.Notealsothatthequestionsthemselvesareintertwinedandhenceactassomesortofcontrolquestionstoeachotherinordertoassurethatthedatagivenbytheparticipantsintheques-tionnaireiscorrect.Table1:Meanfromthestudents'responses Q1 Q2 Q3 Q4 Q5 Q6 Q7 S1 4,5 4,5 2,8 4,0 3,8 2,5 4,2 S2 4,7 4,8 1,5 4,2 4,6 2,2 4,5 S3 5,2 5,1 2,0 4,4 4,6 1,9 4,7 S4 4,9 5,3 2,2 4,7 4,9 2,1 4,7 S5 4,5 4,2 1,9 4,3 4,4 2,8 4,5 AscanbenotedfromTable1theevaluatorsgivepositiveopinionsonallitems.Table2:Meanfromtheprofessionals'responses Q1 Q2 Q3 Q4 Q5 Q6 Q7 S1 4,0 4,2 4,0 4,2 4,2 2,5 4,2 S2 4,7 4,5 2,8 4,3 4,2 2,3 4,3 S3 4,5 4,5 3,0 4,5 4,7 2.2 4,8 S4 4,5 4,7 2,2 4,7 4,7 1,7 5,0 S5 4,5 4,0 3,5 4,3 4,5 1,8 4,0 Theresultsfromtheprofessionaladministrators'an-swerstothequestionnaires,Table2,alsodemonstrateposi-tiveopinionsonallitems,butQ3.Theprofessionaladmin-istratorsareindifferentregardinghowhardthetextsaretoread.Infact,twosubjectsrankthemasratherhardtoread.Notableisthatthestudentsandprofessionaladministra-torsprovideverysimilaranswerstomostofthequestion-naires.Theyallconsiderthetexttobeinformative,Q2,andhavinganappropriatelength,Q1.Theyalso,allthinkthatthetextsprovideagoodideaonwhatwasintheoriginaltext,Q4andQ5.Furthermore,thesubjectsdonotthinkthatthetextsmissrelevantinformation.5.SummaryWehaveusedthepyramidmethodtocreateextractionbasedsummariesofgovernmentaltexts.Thesummariesareevaluatedbybothnovices(students)andprofessionals(ad-ministratorsatthelocalgovernmentalagency)andtheeval-uationsshowthatthesummariesareinformativeandeasytoread.Ourresultsareinlinewithpreviousresearch(Nenkova,2006)whichstatesthatvehumansummarisersareenoughtoproduceagoldstandard.Itcanbefurtherstatedthatthepyramidmethodthennotonlycanbeusedinordertocre-ategoldstandardsfromabstractsummariesbutalsofromextractionbasedsummaries.AcknowledgementsThisresearchisnancedbySantaAnnaITResearchInsti-tuteAB.WearegratefultoourevaluatorsandespeciallythestaffattheSwedishSocialInsuranceAdministration.6.ReferencesHansVanHalterenandSimoneTeufel.2003.Examiningtheconsensusbetweenhumansummaries:Initial.InInHLT-NAACLDUCWorkshop,pages57–64.MartinHasselandHerculesDalianis.2005.GenerationofReferenceSummaries.InProceedingsof2ndLanguage&TechnologyConference:HumanLanguageTechnolo-giesasaChallengeforComputerScienceandLinguis-tics,Poznan,Poland,April21-23.HJing,RBarzilay,KMcKeown,andMElhadad.1998.Summarizationevaluationmethods:Experimentsandanalysis.AAAISymposiumonIntelligentSummariza-tion,Jan.ArneJ¨onsson,MimiAxelsson,EricaBergenholm,BertilCarlsson,GroDahlbom,P¨arGustavsson,JonasRybing,andChristianSmith.2008.Skimreadingofaudioinfor-mation.InProceedingsoftheThesecondSwedishLan-guageTechnologyConference(SLTC-08),Stockholm,Sweden.Chin-YewLinandEduardHovy.2002.Manualandau-tomaticevaluationofsummaries.InProceedingsoftheACL-02WorkshoponAutomaticSummarization,pages45–51,Morristown,NJ,USA.AssociationforComputa-tionalLinguistics.AniNenkova.2006.Understandingtheprocessofmulti-documentsummarization:Contentselection,rewritingandevaluation.Ph.D.thesis,DigitalCom-mons@Columbia,January01.