Omer Levy Ido Dagan Jacob Goldberger Bar Ilan University Israel Open IE Extracts propositions from text which makes aspirin relieve headaches No supervision No predefined schema ID: 719388
Download Presentation The PPT/PDF document "Focused Entailment Graphs for Open IE Pr..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Focused Entailment Graphs for Open IE Propositions
Omer Levy Ido Dagan Jacob GoldbergerBar-Ilan University, IsraelSlide2
Open IE
Extracts propositions from text“…which makes aspirin relieve headaches.”
No supervision
No pre-defined schema
Slide3
What’s missing in Open IE?
StructureOpen IE does not consolidate natural language expressionsrelieve
headache
treat
headache Slide4
Adding Structure to Open IE
Which structure?Build a graph of Open IE propositions and their semantic relationsSlide5
Adding Structure to Open IE
Which structure?Build a graph of Open IE propositions and their entailment
relationsWhy entailment?Merges paraphrases into
mutual entailment cliques
aspirin
relieves headache aspirin treats headacheOrganizes information
hierarchically
from
specific to general
aspirin
relieves
headache painkiller relieves headache
Slide6
aspirin, eliminate, headache
aspirin, cure, headache
headache, control with, aspirin
drug, relieve, headache
drug, treat, headache
analgesic, banish, headache
headache, respond to, painkiller
headache, treat with, caffeine
coffee, help, headache
tea, soothe, headache
Original Open IE OutputSlide7
aspirin, eliminate, headache
aspirin, cure, headache
headache, control with, aspirin
drug, relieve, headache
drug, treat, headache
analgesic, banish, headache
headache, respond to, painkiller
headache, treat with, caffeine
coffee, help, headache
tea, soothe, headache
Consolidated
Open IE OutputSlide8
Semantic Applications
Example: Structured Queries“What relieves headaches?”Slide9
Semantic Applications
Example: Structured Queries“What relieves headaches?”
Slide10
aspirin, eliminate, headache
aspirin, cure, headache
headache, control with, aspirin
drug, relieve, headache
drug, treat, headache
analgesic, banish, headache
headache, respond to, painkiller
headache, treat with, caffeine
coffee, help, headache
tea, soothe, headache
Structured Query:
Slide11
aspirin
, eliminate, headache
aspirin
, cure, headache
headache, control with,
aspirin
drug
, relieve, headache
drug
, treat, headache
analgesic
, banish, headache
headache, respond to,
painkiller
headache, treat with,
caffeine
coffee
, help, headache
tea
, soothe, headache
Structured Query:
Slide12
aspirin
drug
analgesic
painkiller
caffeine
coffee
tea
Structured Query:
Slide13
Our Contributions
Structuring Open IE with Proposition Entailment GraphsDataset: 30 gold-standard graphs, 1.5 million entailment annotationsAlgorithm for constructing Focused
Proposition Entailment GraphsAnalysis: Predicate entailment is not quite what we thoughtSlide14
Proposition Entailment GraphsSlide15
Related Work: Predicate
Entailment GraphsBerant et al. (2010,2011,2012)We extend Berant et al.’s work from predicates to propositionsSlide16
Focused Proposition Entailment Graphs
Nodes: Open IE propositionsEdges: Textual EntailmentSlide17
Focused Proposition Entailment Graphs
Assumptions: Binary Propositions and Common TopicBinary Propositions
Focused on a common topic
Slide18
Focused Proposition Entailment Graphs
Assumptions: Binary Propositions and Common TopicBinary Propositions
Focused on a common topic
Slide19
aspirin, eliminate, headache
aspirin, cure, headache
headache, control with, aspirin
drug, relieve, headache
drug, treat, headache
analgesic, banish, headache
headache, respond to, painkiller
headache, treat with, caffeine
coffee, help, headache
tea, soothe, headacheSlide20
aspirin, eliminate,
headache
aspirin, cure,
headache
headache
, control with, aspirin
drug, relieve,
headache
drug, treat,
headache
analgesic, banish,
headache
headache
, respond to, painkiller
headache
, treat with, caffeine
coffee, help,
headache
tea, soothe,
headacheSlide21
Focused Proposition Entailment Graphs
Edges: Textual Entailment
Proposition
Entailment
Simpler than sentence-level entailment
More complicated than lexical
entailment
Enables investigation of inference phenomena in an isolated manner
Slide22
Constructing Proposition Entailment Graphs
Task Definition:
Given a set of propositions
,
find all their entailment edges. Slide23
DatasetSlide24
Dataset: High-Quality Open IE Propositions
Google’s Syntactic N-gramsBased on millions of booksFilter for subject-verb-objectIncluding prepositional objects and passive
Result: 68 million high-quality propositionsSlide25
Dataset: Annotating Entailment Graphs
Select 30 healthcare topicsantibiotic, caffeine, insomnia, scurvy, …Collect a set of propositions focused on each topic
Manually clean noisy extractionsRetaining 200 propositions per graph (average)
Efficiently annotate entailment
1.5 million entailment judgments
Slide26
AlgorithmSlide27
How do we recognize proposition entailment?
.
?
Slide28
How do we recognize proposition entailment?
.
?
Slide29
How do we recognize proposition entailment?
.
Observation:
propositions entail
their lexical components entail
Slide30
How do we recognize proposition entailment?
.
Observation:
propositions entail
their lexical components entail
Slide31
How do we recognize proposition entailment?
.
Proposition entailment is reduced to
lexical entailment
in context
Slide32
Lexical Entailment
(Logistic)
Lexical Entailment
Lexical Entailment Features
Slide33
Lexical Entailment
(Logistic)
Lexical Entailment
Features
WordNet
Relations
UMLS
Distributional Similarity
String Edit Distance
Lexical Entailment Features
SupervisionSlide34
From Lexical to Proposition Entailment
Lexical Entailment(Logistic)
Lexical Entailment Features
SupervisionSlide35
Argument Entailment
(Logistic)
Predicate Entailment
(Logistic)
From Lexical to Proposition Entailment
Argument Entailment Features
Predicate Entailment Features
Supervision
SupervisionSlide36
Argument Entailment
(Logistic)
Predicate Entailment
(Logistic)
From Lexical to Proposition Entailment
Argument Entailment Features
Predicate Entailment Features
Supervision
Supervision
Proposition Entailment
(Conjunction)
Slide37
Following Snow (2005),
Berant
(2012)
Argument Entailment
(Logistic)
Predicate Entailment
(Logistic)
Distant Supervision (
WordNet
)?
Argument Entailment Features
Predicate Entailment Features
WordNet
WordNet
Proposition Entailment
(Conjunction)
Slide38
Argument Entailment
(Logistic)
Proposition Entailment(Conjunction)
Predicate Entailment
(Logistic)
Direct Supervision (30 Annotated Graphs)
Argument Entailment Features
Predicate Entailment Features
Annotated Graphs
Slide39
Proposition Entailment
(Conjunction)
Direct Supervision (30 Annotated Graphs)
Argument Entailment Features
Predicate Entailment Features
Hidden Layer
Annotated GraphsSlide40
Flat Model
Argument Entailment Features
Proposition Entailment
(Logistic)
Predicate Entailment Features
Annotated GraphsSlide41
Compared Methods
Component-Level Distant Supervision (WordNet)Predicates & ArgumentsPredicates OnlyArguments OnlyProposition-Level Direct Supervision (30 Annotated Graphs)
Hierarchical (our method)FlatAll methods used
Berant
et al.’s Global Optimization methodSlide42
ResultsSlide43
Direct Supervision: Flat vs Hierarchical
Hierarchal model performs better than flat modelBetter to model predicate and argument entailment separatelySlide44
Distant vs Direct Supervision
Direct supervision is betterAlthough WordNet provides more training examplesSlide45
Predicate Entailment with Distant Supervision
Ignoring predicates improves distant supervision baselinesSlide46
Are
WordNet relations capturing real-world predicate entailments?Slide47
Predicate Entailment vs WordNet Relations
Over a predicate inference subset, how many predicate entailments are covered by WordNet?Positive
indicatorssynonyms, hypernyms, entailmentSlide48
Why isn’t
WordNet
capturing predicate entailment?
Predicate Entailment vs
WordNet
Relations
Over
a predicate inference
subset, how many predicate entailments are covered by
WordNet
?
Positive
indicators
synonyms, hypernyms, entailment
Negative
Indicators
antonyms, hyponyms, cohyponymsSlide49
Predicate Entailment is Context-Sensitive
The words do not necessarily entail,but the situations
do.
Slide50
Predicate Entailment is Context-Sensitive
The words do not necessarily entail,but the situations
do.
Slide51
Investigating Context-Sensitive
EntailmentRecent work on context-sensitive lexical inferencee.g. (Melamud et al., 2013)
Previous datasetsLexical substitution (McCarthy and Navigli, 2007)Predicate inference (Zeichner
et al., 2012)
We offer a
new dataset of real-world lexical entailments in context!Sample: synthetic vs naturally occurringSize: several thousands vs 1.5 millionSlide52
ConclusionSlide53
Conclusion
Structuring Open IE with Proposition Entailment GraphsAlgorithm for constructing Focused Proposition Entailment GraphsAnalysis: Predicate entailment is extremely context-sensitive
Dataset: 1.5 million proposition entailment decisions
Thank you for listening!Slide54
Next Steps
Predicate entailment in context is an open problemImprove coverage of argument entailmentInvestigate more complex proposition and graph
structuresThank you for listening!Slide55
Berant et al.’s Method
Local EstimationFor each pair of predicates
, what is the probability that
?
Use local classifier trained with
distant supervision
(
WordNet
)
Distributional similarity features
Global Optimization
Select the most probable
transitive
entailment graph given:
Probabilities from local estimation
Transitivity constraints
WordNet
constraints
Slide56
Berant et al.’s Method
affect
treat
cure
trigger
A set of predicatesSlide57
Local estimation
of entailment probabilities
Berant et al.’s Method
affect
treat
cure
triggerSlide58
Global optimization
of entailment edges
Berant et al.’s Method
affect
treat
cure
triggerSlide59
From Predicates to Propositions
Local EstimationFor each pair of predicates
, what is the probability that
?
Use local classifier trained with
distant supervision
(
WordNet
)
Distributional similarity features
Global Optimization
Select the most probable
transitive
entailment graph given:
Probabilities from local estimation
Transitivity constraints
WordNet
constraints
Slide60
From Predicates to Propositions
Local EstimationFor each pair of propositions
, what is the probability that
?
Use local classifier trained with
distant supervision
(
WordNet
)
Distributional similarity features
WordNet
features
Global Optimization
Select the most probable
transitive
entailment graph given:
Probabilities from local estimation
Transitivity constraints
WordNet
constraints
Slide61
From Predicates to Propositions
Local EstimationFor each pair of propositions
, what is the probability that
?
Use local classifier trained with
distant supervision
(
WordNet
)
Distributional similarity features
WordNet
features
Global Optimization
Select the most probable
transitive
entailment graph given:
Probabilities from local estimation
Transitivity constraints
WordNet
constraints
Slide62
From Predicates to Propositions
Local EstimationFor each pair of propositions
, what is the probability that
?
Use local classifier trained with
distant supervision
(
WordNet
)
Distributional similarity features
WordNet
features
Global Optimization
Select the most probable
transitive
entailment graph given:
Probabilities from local estimation
Transitivity constraints
WordNet
constraints
Infer proposition entailment from lexical features
?Slide63
Component Entailment Conjunction (CEC)
Local EstimationFor each pair of propositions
, what is the probability that
?
Use local classifier trained with
distant supervision
(
WordNet
)
Distributional similarity features
WordNet
features
Global Optimization
Select the most probable
transitive
entailment graph given:
Probabilities from local estimation
Transitivity constraints
WordNet
constraints
Slide64
Component Entailment Conjunction (CEC)
Local EstimationFor each pair of propositions
, what is the probability that
?
Use local classifier trained with
direct supervision
(30 annotated graphs)
Distributional similarity features
WordNet
features
Global Optimization
Select the most probable
transitive
entailment graph given:
Probabilities from local estimation
Transitivity constraints
WordNet
constraints
Slide65
Component Entailment Conjunction (CEC)
Learn component-level classifiers from proposition-level supervisionExpectation Maximization (EM)E-Step: Estimate component-level labels from proposition-level label
M-Step: Use estimates as “soft” labels to train component weightsSlide66
Component Entailment Conjunction (CEC)
Argument Entailment Features
Argument Entailment
(Logistic)
Proposition Entailment
(Conjunction)
Predicate Entailment Features
Predicate Entailment
(Logistic)
Slide67
Component Entailment Conjunction (CEC)
Argument Entailment Features
Argument Entailment
(Logistic)
Proposition Entailment
(Conjunction)
Predicate Entailment Features
Predicate Entailment
(Logistic)
Slide68
How do we learn the weights?
Learn lexical classifiers with distant supervision (WordNet)Berant
et al.Snow et al.Doesn’t work well in practice!Learn
lexical classifiers
with
direct supervision (30 annotated graphs)Propagate proposition-level supervision to lexical features with EMSlide69
Creating a Predicate Entailment Dataset
Get lexical inferences (in context) from proposition-level annotationsIf aligned argument are equal, then predicates determine entailment
Slide70
Creating a Predicate Entailment Dataset
Get lexical inferences (in context) from proposition-level annotationsIf aligned argument are equal, then predicates determine entailment
Slide71
Predicate Entailment: Syntactic Glue?
If both arguments are identical, predicates will entail 80% of the time
Slide72
Argument Entailment
WordNetPrecision:
90% / Recall: 40%Lacks coverageCommon-sense:
Causality:
Distributional Similarity
Precision
:
27%
/ Recall: 40
%
Precision: 50% / Recall:
3%
Captures similarity, not entailment
Slide73
Open IE does not consolidate information
X
relieve headache
X
treat
headacheaspirinmedicine
peppermint
caffeine
intravenous magnesium sulfate
this combination medication
stress
naratriptan
the neck
the drug