From Textual Entailment to Knowledgeable Machines - PowerPoint Presentation

Slide1

From Textual Entailment to Knowledgeable Machines

Peter Clark

Allen Institute for Artificial Intelligence (AI2)Slide2

Mission:

achieve scientific breakthroughs by constructing AI systems with reasoning, learning, and reading capabilities.

2Slide3

Overall Goals

Vision: The Digital Aristotle

L

arge

volumes of general and scientific knowledge, stored in a "computable“ form that supports reasoning and explanation

.

Intermediate Focus:

Pass

science exams

as written

Heavy emphasis on semi-automated knowledge acquisition

Textual entailment at every step

≈ the “modus ponens” of reasoning

“Explainable Reasoning”Slide4
Slide5

The Task

Current focus:

4th grade, multiple choice science questions

Wide variety of question typesRequires general, lexical, and scientific knowledgeSlide6

The 4th

Grade NY Regents’ Science Exam

What types of questions are there?

What would it take to answer them?Slide7

The 4th

Grade NY Regents’ Science Exam

What types of questions are there?

What would it take to answer them?Slide8

Multiple Choice to Textual Entailment

Q

A

: A potato is a fruit?QB

: An onion is a fruit?QC: A carrot is a fruit?QD: A pumpkin is a fruit?

1. Convert to 4 true/false questions

2

. Convert each true/false question to an entailment problem

T: potato

H: fruit?

H: potato is a fruit?

entails?

T:

entails?

OR:

(for X

isa

Y questions, and questions with a setup)Slide9

Multiple Choice to Textual Entailment

Q

A

: A potato is a fruit?QB

: An onion is a fruit?QC: A carrot is a fruit?QD: A pumpkin is a fruit?

Confidence?

0.04

0.12

0.21

0.64

1. Convert to 4 true/false questions

2

. Convert each true/false question to an entailment problem

T: potatoH: fruit?H: potato is a fruit?entails?

T:

entails?

OR:

(for X

isa

Y questions, and questions with a setup)

Answer is D

3. Pick highest confidence answerSlide10

The 4th

Grade NY Regents’ Science Exam

What types of questions are there?

What would it take to answer them?

“Basic”Slide11

1. Taxonomic

S

imple lexical entailmente.g., T:“sleet” H:“precipitation”

Several good sources of simple “isa” knowledgeWordNet,

Cyc, WikipediaIs a basic operation for more complex entailment tasks

entails?Slide12

2. Definitions

Search for best entailing definition

erosion

: The process of

being

eroded by wind, water, or other natural agents.erosion: The wearing away of rocks and other deposits on the earth's surface

…

erosion

: The gradual wearing away of land surface materials, especially rocks,

…

Dictionary ResourcesSlide13

2. Definitions

Search for best entailing definition

erosion

: The process of

being

eroded by wind, water, or other natural agents.erosion: The wearing away of rocks and other deposits on the earth's surface

…

erosion

: The gradual wearing away of land surface materials, especially rocks,

…

Dictionary Resources

T: The

gradual wearing away of land surface materials

, especially rocks, sediments, and soils, by the action of water, wind, or a glacier

H: The

movement

of soil

by wind or

water

entails?Slide14

T: The

gradual wearing away of land surface materials

, especially rocks,

sediments, and soils, by the action of water, wind

, or a glacier

H: The movement of soil by wind or water

entails?

# words

in common

weighted

∑ #words

# with

hypernyms

∑ bi-gramspara-

phrases

…

p(H|T)

3.0

4.31

3.45

2.0

1.20

?

2

.0

1.23

2.12

5.0

1.98

TRAINING DATA

H TRUE

6

.0

4.31

3.45

0.0

0

.20

H FALSE

1

.0

5.43

1.11

1.0

0.24

H FALSE

3.0

1.12

3.45

2.0

1.76

H TRUE

p(H|T)

= 0.76Slide15

2. Definitions

Search for best entailing definition

erosion

: The process of

being

eroded by wind, water, or otheragents.

erosion

: The wearing away of rocks and other deposits on the

earth …

erosion

: The gradual wearing away of land surface

materials, …

friction: The rubbing of surfaces against each otherfriction: a resistance encountered when one body moves relative to another body with which it is in

contactfriction: surface resistance to relative motion, as of a body slidingDictionary Resources0.540.430.760.210.110.13Answer is erosion (C)Slide16

3. within-question entailments

T: A girl eating an apple

Q

B

: A girl eating an apple is an example of an organism taking in nutrients?

H: an organism taking in nutrients

entails?Slide17

3. within-question entailments

T:

A girl

eating an apple

QB: A girl eating an apple

is an example of an organism taking in nutrients?

H

:

an organism

taking in

nutrients

p(H|T) = 0.91Slide18

The 4th

Grade NY Regents’ Science Exam

What types of questions are there?

What would it take to answer them?

“Entailment

from

Corpus”Slide19

4. Entailment from a corpus

H

A

: The flower is the part of a plant that produces the seedsH

B: The leaves are the part of a plant that produces the seedsHC

: The stem is the part of a plant that produces the seedsHD: The roots are the part of a plant that produces the seeds

T:

H

A

? H

B

? H

C

? H

D

?Slide20

4. Entailment from a corpus

H

A

: The flower

is the part of a plant that produces the

seeds HB: The leaves are the part of a plant that produces the seedsH

C

: The stem is

the part of a plant that produces the seeds

H

D

: The roots are the

part of a plant that produces the seeds

T:…Plants can grow from a seed into a flower, tree, or bush.Plants reproduce by producing flowers and fruits that have seeds.The seeds then grow into plants.…

Entailment

Confidence?

0.84

0.12

0.21

0.04Slide21

The 4th

Grade NY Regents’ Science Exam

What types of questions are there?

What would it take to answer them?

“Models”

“Diagrams”Slide22

5. Computational Models

Requires a specific computation over a representation

baby shake rattle

rattle make noise

movement

mechanical energy

sound

sound energy

(C) Mechanical EnergySlide23

Performance (excluding diagrams)

Works okay… ~55% score (vs. 25% random guessing)Slide24

But…

If horses are kept inside in a barn, they require regular

daily exercise

for their physical

health

and mental well-being.?

Carrots

can also be used alone or with

fruits

in jam and preserves.



System answer:

(B) System answer: (C) Slide25

System answer:

(A)



 

Graders

are commonly used in the construction and maintenance of dirt roads and

gravel

roads

In experiments in which

statoliths

were replaced with

metal

shavings, researchers "tricked" crayfish into swimming upside down by using

magnets

to pull the shavings to the upper end of the

statocysts

located at the base of their antennae.

?

System answer:

(D)

Slide26

What is going on?

Largely

“smart guessing” based on word associationsBUT:

doesn’t give us meaningful explanationsdoesn’t get us closer to machine readingperformance max’es out at ~60%

What is missing?Richer representation of meaningin the questionin the corpusSlide27

Entailment

?

?- has-part(

ribosome,?x

).

Text

Logic

Query

logical

entailment

“textual”

entailmentSlide28

Entailment

?

?- has-part(

ribosome,?x

).

Text

Logic

Query

logical

entailment

“textual”

entailmentSlide29

Textual Entailment

?

?- has-part(

ribosome,?x

).

Text

Logic

Query

Semi-

Formal

logical

entailment

“textual”

entailment

Bag of words

N-grams

Parse trees

Dependency trees

…

?

Textual Entailment =

The Science of Semi-Formal

RepresentationsSlide30

Semi-formal representations

Lexical

– good baselineDependency trees

– precise but complex to manipulateFull logic – very hard to translate intoSVO units (“depth 1 parses”, “tuples”)?

top-level: syntactic structurelower level: phrasal

These simple propositions = a basic “unit of meaning”Inference is a mixture of structural and phrasal matching

The vibrations from sound move tiny bones in our ears.

(the vibrations from sound, move, tiny bones in our ears)

subject

verb

object

[

pps

]Text

Repn

.Slide31

Sentences may contain one or more related units

Some animals grow thick fur in winter to stay warm.

P1

P2

(some animals, grow, thick fur, in winter

)

(

some animals, stay, , warm)

Types of relations between units:

P1 AND P2

P1 IMPLIES P2

P1 EFFECT P2

P1 CAUSES P2

P1 PURPOSE P2

EFFECTSlide32

Can define extraction patterns for these units

S V O “to help”

V O

P1

EFFECT

P2

Some animals grow thick fur in winter to

help maintain body heat

(

some animals, grow, thick fur, in winter)

EFFECT

(some animals, maintain, body heat)Slide33

Pattern-based Extraction

Can transduce text into this form

P1

P2 AND P3

P4

P5 CAUSES P6

P7 ENABLES P8 AND P9

P10

P11

P12 IMPLIES P13

P14 CAUSES P15

P16

S V O “to help” S V O

S V O “in order to” V OS V “cause” S V O… … …Slide34

Pattern-based Extraction

S V O “to help” S V O

S V O “in order to” V OS V “cause” S V O

… … …

(gravity, pull) CAUSE (objects, fall, , towards Earth)(scientists, using, a model) EFFECT (scientists, understand, , better)(, dividing, a single cell) EFFECT (, form, two daughter cells)(Animals, use, saturated fatty acids) EFFECT (Animals, store, energy)

(humans, get, regular rest) EFFECT (humans, be, healthy)(Fish, have, fins) EFFECT (fins, move, )(animals, move, to warmer climate) EFFECT (animals, avoid,

change

in seasons)

(cactus, hold, water) EFFECT (cactus, survive, in the desert

)

… … … …

Can transduce text into this formSlide35

The

child

exhibits the

facial

features

characteristic of this disorder.

H

A

A

facial

scar is a

characteristic

that a human offspring

can inherit?TSystem Answer: (A)Slide36

Some traits that can be inherited are color of hair, color of skin, color of eyes, and height.

H

B

Blue eyes is

a characteristic that a human offspring can

inherit?

T

System Answer: (B)

(human offspring,

can inherit

, the characteristic of

blue eyes

)

(, can inherit, color of hair)(, can inherit, color of skin)(, can inherit, color of eyes)(, can inherit, height)

Slide37

Where does this break down?

Quality of extractions is low

Sentence-level

units are too small, context independent

(, create, a diagram) EFFECT (, show, the cells of multicellular organisms

may be organized at different levels)(, Say, thanks to the authors) EFFECT (, access, a customizable version of this book)(All cells, are, small, very)

EFFECT

(one or more cells, need, )

Need larger-sized structures

spanning several sentencesSlide38

The Main Points

Textual entailment is

less about “matching text”more about the

science of semi-formal representationsThose representationsexplicate the “world knowledge” the text encodes

can be sharable “knowledge resources” in their own righttake us a step closer to “knowledgeable machines”

Thank you!

?

Query

Semi-

Formal

Text