Amanda Hicks University of Florida aehicksufledu 6 th Annual CTSOG Workshop Ann Arbor MI 1 Overview Overview of FACTS Interannotator Agreement scores The difficult t ask Can ontologists and philosophers do it better ID: 814445
Download The PPT/PDF document "Measuring Interannotator Agreement in th..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Measuring Interannotator Agreement in the Florida Annotated Corpus for Translational Science - The difficult ontological task
Amanda HicksUniversity of Floridaaehicks@ufl.edu6th Annual CTSOG Workshop, Ann Arbor MI
1
Slide2OverviewOverview of FACTS
Interannotator Agreement scoresThe difficult taskCan ontologists and philosophers do it better?2
Slide3The Big Goal
Comparatively evaluate the adequacy of ontologies for extracting patient-level data from unstructured text.Create a gold standard, ontologically annotated corpus for clinical and translational science.Annotate with multiple – and, as far as possible, competing –ontologies
3
Slide4The Florida Annotated Corpus for Translational Science (FACTS)
Currently consists of 20 annotated case reports on hypertensionFull textFreely available through PubMedEnglishWithin last 6 yearsStratified by race, ethnicity, gender and age (<18 or 18+)Annotated with VSO, in the process of annotating with DOID
Can be extended to other domains or document types
4
Slide5The Annotation Tasks
Identify the assertions about a person mentioned in the corpus Annotate the entities referred to in those assertions with ontology classesAnnotate the relations between individuals thereby representing the full assertion
c
urrent
tasks
5
Slide6The Annotation Process
Follows the annotation procedure for the CRAFT corpusTwo primary annotators annotated case reports with classes from the Vital Sign Ontology using BRATOne medical student, one public health specialist with training in nursingPrimary annotations were sent to the lead annotator, who reviewed discrepanciesDiffs were discussed at weekly meetings and consensus achieved, producing the gold standard.Annotation guidelines were used and revised during the annotation process.
6
Slide7Interannotator Agreement Was Low
f-measure
exact matches only
f-measure
exact and partial matches
Hypertension
1
1
st
set of10 case reports
0.50
0.54
Hypertension
2
2
nd
set of 10 case reports
0.60
0.69
Full Corpus0.570.60
The CRAFT corpus achieves ~.90 f-score consistently.
7
Slide8What happened?
One annotator had more training and experience than the other.However, IAA on Hypertension 2 is still quite low .06-.69.Our annotators performed two tasks, unlike CRAFT annotators.Identify the instance level assertions about an individual person mentioned in the corpus Annotate the entities referred to in those assertions with ontology classes
8
Slide9What is the major source of disagreement?
f-measure
exact matches only
f-measure
exact and partial matches
f-measure
agreement of classes on matched spans only
Hypertension 1
0.50
0.54
0.93
Hypertension 2
0.60
0.69
0.87
Full Corpus
0.57
0.60
0.90
When the primary annotators agree on the span, they tend to agree on the class.
This suggests that the difficult task is determining whether a token expresses an instance level assertion.
9
Slide10Easy Cases
"A 56-year-old man suddenly developed dyspnea after resection of choroidal melanoma”"Previous studies noted a significant association between melanoma and endothelin (ET)-1.”
Sato K,
Saji
T, Kaneko T, Takahashi K,
Sugi
K. Unexpected pulmonary hypertensive crisis after surgery for ocular malignant melanoma. Life Sci. 2014;118(2):420-3.
Epub
2014/03/19.
doi
: 10.1016/j.lfs.2014.03.004. PubMed PMID: 24632478.
10
Slide11Difficult cases
"First, a massive amount of ET-1, which is a proliferation factor in malignant melanoma, was released due to mechanical stimulation from endoresection, a procedure in which the tumor is cut into very small fragments and aspirated (Fig. 6).""As this is the first report of pulmonary hypertension
after endoresection, it might be useful to determine the differences between our patient and other patients treated with endoresection.”
Sato K,
Saji
T, Kaneko T, Takahashi K,
Sugi
K. Unexpected pulmonary hypertensive crisis after surgery for ocular malignant melanoma. Life Sci. 2014;118(2):420-3.
Epub
2014/03/19.
doi
: 10.1016/j.lfs.2014.03.004. PubMed PMID: 24632478
.
We
decided that that 'pulmonary hypertension' denotes a particular, but it is not clear to me that this is correct.
Which terms denote individuals and which do not? The subordinate clauses make this an interesting case.
11
Slide12Next steps
Can ontologists and philosophers agree more than the specialist annotators on the hard task?We will have working ontologists annotate a sample set of case reports for instance level statements.We will have philosophy graduate students annotate a sample set of case reports for instance level statements.12
Slide13Acknowlegments
Selja Seppälä, University of CorkBill Hogan, University of FloridaCarl Pepine, University of FloridaNathan Boire, University of Florida
Chloe Herring, University of Florida
This work was supported in part by the NIH/NCATS Clinical and Translational Science Award to the University of Florida UL1 TR000064. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the NCTE
.
13