Torsten Zesch Iryna Gurevych Ubiquitous Knowledge Processing Lab Technische Universität Darmstadt Omer Levy Ido Dagan Natural Language Processing Lab Bar Ilan University ID: 719390
Download Presentation The PPT/PDF document "Recognizing Partial Textual Entailment" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Recognizing Partial Textual Entailment
Torsten Zesch Iryna GurevychUbiquitous Knowledge Processing LabTechnische Universität Darmstadt
Omer Levy Ido DaganNatural Language Processing LabBar-Ilan University
1Slide2
Textual Entailment
2Slide3
T: muscles move bones
3Slide4
T: muscles move bones
H: muscles generate movement4Slide5
T: muscles move bones
H: muscles generate movement5Slide6
T: muscles move bones
H: the muscles’ main job is to move bones?
6Slide7
T: muscles move bones
H: the muscles’ main job is to move bones
7Slide8
Complete Textual Entailment
8Slide9
Complete Textual Entailment
9Slide10
Complete Textual Entailment
. Partial .
10Slide11
Complete Textual Entailment
. Partial .
(Nielsen et al, 2009)11Slide12
T: muscles move bones
H: the muscles’ main job is to move bones12Slide13
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones13Slide14
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones
14Slide15
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones
15Student Response AnalysisSlide16
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones
16Student Response AnalysisSummarizationInformation ExtractionSlide17
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones
Facets17Slide18
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones (main job, muscles
) (muscles, move) (move, bones) Facets
18Slide19
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones (main job, muscles
) (muscles, move) (move, bones) Facets
19Slide20
T: muscles move bones
(main job of muscles) (muscles move) (bones are moved) H: the muscles’ main job is to move bones (main job, muscles) (
muscles, move) (move, bones) Facets20Slide21
Facet: a pair of terms
and their implied semantic relation.21Slide22
Facet: a pair of
terms and their implied semantic relation.(move, bones)
22Slide23
Facet: a pair of
terms and their implied semantic relation.
(move, bones)… is to move bonestheme23Slide24
Recognizing
Faceted Entailment24Slide25
T: muscles move bones
H: the muscles’ main job is to move bonesRecognizing Faceted
Entailment25Slide26
T: muscles move bones
H: the muscles’ main job is to move bonesRecognizing Faceted
Entailment26Slide27
T: muscles move bones
H: the muscles’ main job is to move bones
Recognizing Faceted Entailment27Slide28
T: muscles move bones
H: the muscles’ main job is to move bones
Recognizing Faceted Entailment28Slide29
What did we do?
29Slide30
Approach
: Leverage Existing Methods30Slide31
Approach: Leverage Existing Methods
31Slide32
Exact Match
P: 96%, R: 30%penicillin cures pneumonia
(penicillin, cure) Lexical Inference (penicillin, cure) pneumonia is treated by penicillinLexical-Syntactic Inference Approach: Leverage Existing Methods32Slide33
Exact Match
P: 96%, R: 30%penicillin cures pneumonia
(penicillin, cure) Lexical Inference (penicillin, cure) pneumonia is treated by penicillinLexical-Syntactic Inference Approach: Leverage Existing Methods33Slide34
Exact Match
penicillin cures pneumonia (penicillin, cure) Lexical Inference WordNetpneumonia is
treated by penicillin (penicillin, cure) Lexical-Syntactic Inference Approach: Leverage Existing Methods34Slide35
Exact Match
penicillin cures pneumonia (penicillin, cure) Lexical Inference WordNetpneumonia is
treated by penicillin (penicillin, cure) Lexical-Syntactic Inference Approach: Leverage Existing Methods35Slide36
Exact Match
penicillin cures pneumonia (penicillin, cure) Lexical Inferencepneumonia is treated by penicillin (penicillin, cure)
Lexical-Syntactic Inference Approach: Leverage Existing Methodspenicillin cures pneumonia pneumonia is treated by penicillin36Slide37
Bar-
Ilan University Textual Entailment Engine (Stern and Dagan, 2011) 37Slide38
BIUTEE
38Slide39
BIUTEE
pneumonia
curespenicillin39 T HSlide40
BIUTEE
pneumonia
curespenicillinpenicillin
treated
pneumonia
is
by
40
T HSlide41
BIUTEE
pneumonia
curespenicillinpenicillin
treated
pneumonia
is
by
41
T HSlide42
42
pneumonia
curespenicillinSlide43
43
pneumonia
curespenicillinpneumonia
treats
penicillin
cure
treat
(lexical)
Slide44
44
pneumonia
curespenicillinpenicillin
treated
pneumonia
is
by
pneumonia
treats
penicillin
cure
treat
(lexical)
active
passive
(syntactic)
Slide45
Facets to Dependencies
pneumonia
isby45treated
penicillinSlide46
Facets to Dependencies
penicillin
pneumoniaisby46
treatedSlide47
Baseline
BaseLexBaseSynMajority
Decision Mechanisms Exact Match Exact Match OR Lexical Inference Exact Match OR Syntactic Inference Lexical Inference Exact Match OR AND Syntactic Inference47Slide48
Baseline
BaseLexBaseSynMajority
Decision Mechanisms Exact Match Exact Match OR Lexical Inference Exact Match OR Syntactic Inference Lexical Inference Exact Match OR AND Syntactic Inference48Slide49
Baseline
BaseLexBaseSynMajority
Decision Mechanisms Exact Match Exact Match OR WordNet Exact Match OR BIUTEE Lexical Inference Exact Match OR AND Syntactic Inference49Slide50
Baseline
BaseLexBaseSynMajority
Decision Mechanisms Exact Match Exact Match OR WordNet Exact Match OR BIUTEE WordNet Exact Match OR AND BIUTEE50Slide51
SemEval 2013 Task 7
51Slide52
Student Response Analysis
52Slide53
What is the muscles’ main job?
Muscles move bones.The muscles’ main job is to move bones.
Question: Student Answer: Reference Answer:53Slide54
What is the muscles’ main job?
Muscles move bones.The muscles’ main job is to move bones.
Question: Student Answer: Reference Answer:54Slide55
What is the muscles’ main job?
Muscles move bones.The muscles’ main job is to move bones.
Question: Student Answer: Reference Answer:55Slide56
What is the muscles’ main job?
Muscles move bones.The muscles’ main job is to move bones.
Question: Text: Hypothesis:56Slide57
What is the muscles’ main job?
Muscles move bones.The muscles’ main job is to move bones.
Question: Text: Hypothesis:57Slide58
SciEntsBank
(Djikovska et al, 2012)15 Domains
5,106 Student Responses16,263 FacetsDataset58Slide59
SciEntsBank
(Djikovska et al, 2012)15 Domains
5,106 Student Responses16,263 FacetsDataset59Slide60
SciEntsBank
(Djikovska et al, 2012)15 Domains5,106 Student Responses
16,263 FacetsDataset60Slide61
SciEntsBank
(Djikovska et al, 2012)15 Domains13,145 Train Facets
16,263 Test FacetsDataset61Slide62
SciEntsBank
(Djikovska et al, 2012)15 Domains13,145 Train Facets16,263 Test Facets
Dataset62Slide63
Unseen Answers 9%
Unseen Questions 12%Unseen Domains 79%
Scenarios63Slide64
Unseen Answers 9%
Unseen Questions 12%Unseen Domains 79%
Scenarios64Slide65
Unseen Answers 9%
Unseen Questions 12%Unseen Domains 79%
Scenarios65Slide66
Unseen Answers 9%
Unseen Questions 12%Unseen Domains 79%
Scenarios66Slide67
Results
67Slide68
Performance (Weighted F1)
68Slide69
Summary
69Slide70
Problem: Need fine-grained entailment
Solution: Partial Textual Entailment Approach: Leverage existing methods Results: Significant improvement
70Slide71
Problem: Need fine-grained entailment
Solution: Partial Textual Entailment Approach: Leverage existing methods Results: Significant improvement
71Slide72
Problem: Need fine-grained entailment
Solution: Partial Textual Entailment Approach: Leverage existing methods Results: Significant improvement72Slide73
Problem: Need fine-grained entailment
Solution: Partial Textual Entailment Approach: Leverage existing methods Results: Significant improvement73Slide74
Future
: Automatic Facet Extraction74Slide75
Future: Automatic
Facet Extraction75Slide76
Partial Textual Entailment is
interesting76Slide77
Partial Textual Entailment is
useful77Slide78
Partial Textual Entailment is
practical78Slide79
Partial Textual Entailment is
interesting79Slide80
Questions?
80