Peter Clark Allen Institute for Artificial Intelligence AI2 Mission achieve scientific breakthroughs by constructing AI systems with reasoning learning and reading capabilities 2 Overall Goals ID: 538655
Download Presentation The PPT/PDF document "From Textual Entailment to Knowledgeable..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
From Textual Entailment to Knowledgeable Machines
Peter Clark
Allen Institute for Artificial Intelligence (AI2)Slide2
Mission:
achieve scientific breakthroughs by constructing AI systems with reasoning, learning, and reading capabilities.
2Slide3
Overall Goals
Vision: The Digital Aristotle
L
arge
volumes of general and scientific knowledge, stored in a "computable“ form that supports reasoning and explanation
.
Intermediate Focus:
Pass
science exams
as written
Heavy emphasis on semi-automated knowledge acquisition
Textual entailment at every step
≈ the “modus ponens” of reasoning
“Explainable Reasoning”Slide4Slide5
The Task
Current focus:
4th grade, multiple choice science questions
Wide variety of question typesRequires general, lexical, and scientific knowledgeSlide6
The 4th
Grade NY Regents’ Science Exam
What types of questions are there?
What would it take to answer them?Slide7
The 4th
Grade NY Regents’ Science Exam
What types of questions are there?
What would it take to answer them?Slide8
Multiple Choice to Textual Entailment
Q
A
: A potato is a fruit?QB
: An onion is a fruit?QC: A carrot is a fruit?QD: A pumpkin is a fruit?
1. Convert to 4 true/false questions
2
. Convert each true/false question to an entailment problem
T: potato
H: fruit?
H: potato is a fruit?
entails?
T:
entails?
OR:
(for X
isa
Y questions, and questions with a setup)Slide9
Multiple Choice to Textual Entailment
Q
A
: A potato is a fruit?QB
: An onion is a fruit?QC: A carrot is a fruit?QD: A pumpkin is a fruit?
Confidence?
0.04
0.12
0.21
0.64
1. Convert to 4 true/false questions
2
. Convert each true/false question to an entailment problem
T: potatoH: fruit?H: potato is a fruit?entails?
T:
entails?
OR:
(for X
isa
Y questions, and questions with a setup)
Answer is D
3. Pick highest confidence answerSlide10
The 4th
Grade NY Regents’ Science Exam
What types of questions are there?
What would it take to answer them?
“Basic”Slide11
1. Taxonomic
S
imple lexical entailmente.g., T:“sleet” H:“precipitation”
Several good sources of simple “isa” knowledgeWordNet,
Cyc, WikipediaIs a basic operation for more complex entailment tasks
entails?Slide12
2. Definitions
Search for best entailing definition
erosion
: The process of
being
eroded by wind, water, or other natural agents.erosion: The wearing away of rocks and other deposits on the earth's surface
…
erosion
: The gradual wearing away of land surface materials, especially rocks,
…
Dictionary ResourcesSlide13
2. Definitions
Search for best entailing definition
erosion
: The process of
being
eroded by wind, water, or other natural agents.erosion: The wearing away of rocks and other deposits on the earth's surface
…
erosion
: The gradual wearing away of land surface materials, especially rocks,
…
Dictionary Resources
T: The
gradual wearing away of land surface materials
, especially rocks, sediments, and soils, by the action of water, wind, or a glacier
H: The
movement
of soil
by wind or
water
entails?Slide14
T: The
gradual wearing away of land surface materials
, especially rocks,
sediments, and soils, by the action of water, wind
, or a glacier
H: The movement of soil by wind or water
entails?
# words
in common
weighted
∑ #words
# with
hypernyms
∑ bi-gramspara-
phrases
…
p(H|T)
3.0
4.31
3.45
2.0
1.20
?
2
.0
1.23
2.12
5.0
1.98
TRAINING DATA
H TRUE
6
.0
4.31
3.45
0.0
0
.20
H FALSE
1
.0
5.43
1.11
1.0
0.24
H FALSE
3.0
1.12
3.45
2.0
1.76
H TRUE
p(H|T)
= 0.76Slide15
2. Definitions
Search for best entailing definition
erosion
: The process of
being
eroded by wind, water, or otheragents.
erosion
: The wearing away of rocks and other deposits on the
earth …
erosion
: The gradual wearing away of land surface
materials, …
friction: The rubbing of surfaces against each otherfriction: a resistance encountered when one body moves relative to another body with which it is in
contactfriction: surface resistance to relative motion, as of a body slidingDictionary Resources0.540.430.760.210.110.13Answer is erosion (C)Slide16
3. within-question entailments
T: A girl eating an apple
Q
B
: A girl eating an apple is an example of an organism taking in nutrients?
H: an organism taking in nutrients
entails?Slide17
3. within-question entailments
T:
A girl
eating an apple
QB: A girl eating an apple
is an example of an organism taking in nutrients?
H
:
an organism
taking in
nutrients
p(H|T) = 0.91Slide18
The 4th
Grade NY Regents’ Science Exam
What types of questions are there?
What would it take to answer them?
“Entailment
from
Corpus”Slide19
4. Entailment from a corpus
H
A
: The flower is the part of a plant that produces the seedsH
B: The leaves are the part of a plant that produces the seedsHC
: The stem is the part of a plant that produces the seedsHD: The roots are the part of a plant that produces the seeds
T:
H
A
? H
B
? H
C
? H
D
?Slide20
4. Entailment from a corpus
H
A
: The flower
is the part of a plant that produces the
seeds HB: The leaves are the part of a plant that produces the seedsH
C
: The stem is
the part of a plant that produces the seeds
H
D
: The roots are the
part of a plant that produces the seeds
T:…Plants can grow from a seed into a flower, tree, or bush.Plants reproduce by producing flowers and fruits that have seeds.The seeds then grow into plants.…
Entailment
Confidence?
0.84
0.12
0.21
0.04Slide21
The 4th
Grade NY Regents’ Science Exam
What types of questions are there?
What would it take to answer them?
“Models”
“Diagrams”Slide22
5. Computational Models
Requires a specific computation over a representation
baby shake rattle
rattle make noise
movement
mechanical energy
sound
sound energy
(C) Mechanical EnergySlide23
Performance (excluding diagrams)
Works okay… ~55% score (vs. 25% random guessing)Slide24
But…
If horses are kept inside in a barn, they require regular
daily exercise
for their physical
health
and mental well-being.?
Carrots
can also be used alone or with
fruits
in jam and preserves.
System answer:
(B) System answer: (C) Slide25
System answer:
(A)
Graders
are commonly used in the construction and maintenance of dirt roads and
gravel
roads
In experiments in which
statoliths
were replaced with
metal
shavings, researchers "tricked" crayfish into swimming upside down by using
magnets
to pull the shavings to the upper end of the
statocysts
located at the base of their antennae.
?
System answer:
(D)
Slide26
What is going on?
Largely
“smart guessing” based on word associationsBUT:
doesn’t give us meaningful explanationsdoesn’t get us closer to machine readingperformance max’es out at ~60%
What is missing?Richer representation of meaningin the questionin the corpusSlide27
Entailment
?
?- has-part(
ribosome,?x
).
Text
Logic
Query
logical
entailment
“textual”
entailmentSlide28
Entailment
?
?- has-part(
ribosome,?x
).
Text
Logic
Query
logical
entailment
“textual”
entailmentSlide29
Textual Entailment
?
?- has-part(
ribosome,?x
).
Text
Logic
Query
Semi-
Formal
logical
entailment
“textual”
entailment
Bag of words
N-grams
Parse trees
Dependency trees
…
?
Textual Entailment =
The Science of Semi-Formal
RepresentationsSlide30
Semi-formal representations
Lexical
– good baselineDependency trees
– precise but complex to manipulateFull logic – very hard to translate intoSVO units (“depth 1 parses”, “tuples”)?
top-level: syntactic structurelower level: phrasal
These simple propositions = a basic “unit of meaning”Inference is a mixture of structural and phrasal matching
The vibrations from sound move tiny bones in our ears.
(the vibrations from sound, move, tiny bones in our ears)
subject
verb
object
[
pps
]Text
Repn
.Slide31
Sentences may contain one or more related units
Some animals grow thick fur in winter to stay warm.
P1
P2
(some animals, grow, thick fur, in winter
)
(
some animals, stay, , warm)
Types of relations between units:
P1 AND P2
P1 IMPLIES P2
P1 EFFECT P2
P1 CAUSES P2
P1 PURPOSE P2
EFFECTSlide32
Can define extraction patterns for these units
S V O “to help”
V O
P1
EFFECT
P2
Some animals grow thick fur in winter to
help maintain body heat
(
some animals, grow, thick fur, in winter)
EFFECT
(some animals, maintain, body heat)Slide33
Pattern-based Extraction
Can transduce text into this form
P1
P2 AND P3
P4
P5 CAUSES P6
P7 ENABLES P8 AND P9
P10
P11
P12 IMPLIES P13
P14 CAUSES P15
P16
S V O “to help” S V O
S V O “in order to” V OS V “cause” S V O… … …Slide34
Pattern-based Extraction
S V O “to help” S V O
S V O “in order to” V OS V “cause” S V O
… … …
(gravity, pull) CAUSE (objects, fall, , towards Earth)(scientists, using, a model) EFFECT (scientists, understand, , better)(, dividing, a single cell) EFFECT (, form, two daughter cells)(Animals, use, saturated fatty acids) EFFECT (Animals, store, energy)
(humans, get, regular rest) EFFECT (humans, be, healthy)(Fish, have, fins) EFFECT (fins, move, )(animals, move, to warmer climate) EFFECT (animals, avoid,
change
in seasons)
(cactus, hold, water) EFFECT (cactus, survive, in the desert
)
… … … …
Can transduce text into this formSlide35
The
child
exhibits the
facial
features
characteristic of this disorder.
H
A
A
facial
scar is a
characteristic
that a human offspring
can inherit?TSystem Answer: (A)Slide36
Some traits that can be inherited are color of hair, color of skin, color of eyes, and height.
H
B
Blue eyes is
a characteristic that a human offspring can
inherit?
T
System Answer: (B)
(human offspring,
can inherit
, the characteristic of
blue eyes
)
(, can inherit, color of hair)(, can inherit, color of skin)(, can inherit, color of eyes)(, can inherit, height)
Slide37
Where does this break down?
Quality of extractions is low
Sentence-level
units are too small, context independent
(, create, a diagram) EFFECT (, show, the cells of multicellular organisms
may be organized at different levels)(, Say, thanks to the authors) EFFECT (, access, a customizable version of this book)(All cells, are, small, very)
EFFECT
(one or more cells, need, )
Need larger-sized structures
spanning several sentencesSlide38
The Main Points
Textual entailment is
less about “matching text”more about the
science of semi-formal representationsThose representationsexplicate the “world knowledge” the text encodes
can be sharable “knowledge resources” in their own righttake us a step closer to “knowledgeable machines”
Thank you!
?
Query
Semi-
Formal
Text