Factors Affecting Reliable Word Sense Annotation Susan Windisch Brown Travis Rood and Martha Palmer University of Colorado at Boulder Annotators in their little nests agree And tis a shameful sight ID: 612010
Download Presentation The PPT/PDF document "Number or Nuance:" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Number or Nuance: Factors Affecting Reliable Word Sense Annotation
Susan Windisch Brown, Travis Rood, and Martha PalmerUniversity of Colorado at BoulderSlide2
Annotators in their little nests agree;And ‘tis a shameful sight,When taggers on one projectFall out, and chide, and fight.
—[adapted from] Isaac WattsSlide3
Automatic word sense disambiguation
Lexical ambiguity is a significant problem in natural language processing (NLP) applications (Agirre & Edmonds, 2006)Text summarizationQuestion answeringWSD systems might help
Several studies show benefits for NLP tasks
(Sanderson, 2000;
Stokoe
, 2003;
Carpuat
and Wu, 2007; Chan, Ng and Chiang, 2007)But only with higher system accuracy (90%+)
3Slide4
Annotation reliability affects system accuracy
4WSD systemSystem Performance
Inter-annotator agreement
Sense Inventory
SensEval2
62.5%
70%
WordNet
Chen et al.
(2007)
82%
89%OntoNotes Palmer (2008)90%94%PropBank
Slide5
Senses for the verb control
5WordNetOntoNotes
1. exercise authoritative control or power over
1. exercise power or influence over; hold within limits
2. control (others or
oneself
) or influence skillfully
3. handle and cause to function
4. lessen the intensity of; temper
5. check or regulate (a scientific experiment) by conducting a parallel experiment
2. verify something by comparing to a standard
6. verify by using a duplicate register for comparison
7. be careful or certain to do something
8. have a firm understanding ofSlide6
Possible factors affecting the reliability of word sense annotation
6Fine-grained senses result in many senses per word, creating a heavy cognitive load on annotators, making accurate and consistent tagging difficultFine-grained senses are not distinct enough to reliably discriminate between Slide7
Requirements to compare fine-grained and coarse-grained annotation
7 Annotation of the same words on the same corpus instancesSense inventories differing only in sense granularity
Previous work
(Ng et al., 1999; Edmonds & Cotton, 2001;
Navigli
et al. 2007)Slide8
3 experiments8
40 verbsNumber of senses : 2-26Sense granularity: WordNet vs. OntoNotesExp. 1: confirm difference in reliability between fine- and coarse-grained annotation; vary granularity and number of sensesExp. 2: hold granularity constant; vary number of sensesExp. 3: hold number constant; vary granularitySlide9
Experiment 1Compare fine-grained sense inventory to coarse70
instances for each verb from the ON corpusAnnotated with WN senses by multiple pairs of annotatorsAnnotated with ON senses by multiple pairs of annotatorsCompare the ON ITAs to the WN ITAs9
Ave.
n
umber of senses
Granularity
OntoNotes
6.2
Coarse
WN
14.6
FineSlide10
Results10Slide11
ResultsCoarse-grained ON annotations had higher ITAs than fine-grained WN annotations
Number of sensesNo significant effect (t(79) = -1.28, p = .206). Sense nuance Yes, a significant effect (t(79) = 10.39, p < .0001).With number of senses held constant, coarse-grained annotation is 16.2 percentage points higher than fine-grained.
11Slide12
Experiment 2: Number of sensesHold sense granularity constant; vary # of senses
2 pairs of annotators, using fine-grained WN sensesFirst pair uses full set of WN senses for a wordSecond pair uses a restricted set on instances that we know should fit one of those senses12
Ave.
n
umber of senses
Granularity
WN
Full set
14.6
Fine
WN Restricted set
5.6FineSlide13
13
OntoNotes grouped sense B
OntoNotes grouped sense C
OntoNotes grouped sense A
WN 3 7 8
13 14
WN 9 10
WN 1 2 4 5
6 11 12Slide14
"Then I just bought plywood, drew the pieces on it and cut them out."
1. ----------------2. ----------------3. ----------------4. ----------------5. ----------------6. ----------------7. ----------------8. ----------------9. ----------------10. ----------------
11. ----------------
12. ----------------
13. ----------------
14. ----------------
3. ----------------
7. ----------------
8. ----------------
13. ----------------
14. ----------------
14Full set of WN sensesRestricted set of WN sensesSlide15
Results15Slide16
Experiment 3Number of senses controlled; vary sense granularity
Compare the ITAs for the ON tagging with the restricted-set WN tagging16
Ave.
n
umber of senses
Granularity
OntoNotes
6.2
Coarse
WN Restricted set
5.6
FineSlide17
Results17Slide18
Conclusion
Number of senses annotators must choose between: never a significant factorGranularity of the senses: a significant factor, with fine-grained senses leading to lower ITAsPoor reliability of fine-grained word sense annotation cannot be improved by reducing the cognitive load on annotators.Annotators cannot reliably discriminate between nuanced sense distinctions.
18Slide19
Acknowledgements
19 We gratefully acknowledge the efforts of all of the annotators and the support of the National Science Foundation Grants NSF-0415923, Word Sense Disambiguation and CISE-CRI-0551615, Towards a Comprehensive Linguistic Annotation and CISE-CRI 0709167, as well as a grant from the Defense Advanced Research Projects Agency (DARPA/IPTO) under the GALE program, DARPA/CMO Contract No. HR0011-06-C-0022, a subcontract from BBN, Inc.Slide20
Restricted set annotation20
Use the adjudicated ON data to determine the ON sense for each instance.Use instances from experiment1 that were labeled with one selected ON sense (35 instances).Each restricted-set annotator saw only the WN senses that were clustered to form the appropriate ON sense.Compare to the full set annotation for those instances.