CSC 594 Topics in AI –

CSC 594 Topics in AI –Natural Language Processing

Spring 201814. Context-Free Parsing(Some slides adapted from Jurafsky & Martin)

ParsingParsing with CFGs refers to the task of assigning proper trees to input strings

Proper here means a tree that covers all and only the elements of the input and has an S at the topIt doesn’t mean that the system can select the correct tree from among all the possible trees

Speech and Language Processing -

Jurafsky

and Martin

2

For Now

Let’s assume…You have all the words for a sentence already in some bufferThe input is not POS tagged prior to parsingWe won’t worry about morphological analysisAll the words are knownThese are all problematic in various ways, and would have to be addressed in real applications.

Speech and Language Processing -

Jurafsky

and Martin

3

Search FrameworkIt’s productive to think about parsing as a form of search…

A search through the space of possible trees given an input sentence and grammarThis framework suggests that heuristic search methods and/or dynamic programming methods might be applicableIt also suggests that notions such as the direction of the search might be useful

Speech and Language Processing -

Jurafsky

and Martin

4

Top-Down SearchSince we’

re trying to find trees rooted with an S (Sentences), why not start with the rules that give us an S.Then we can work our way down from there to the words.

Speech and Language Processing -

Jurafsky

and Martin

5

Top Down Space

Speech and Language Processing -

Jurafsky

and Martin

6

Bottom-Up ParsingOf course, we also want trees that cover the input words. So we might also start with trees that link up with the words in the right way.

Then work your way up from there to larger and larger trees.

Speech and Language Processing -

Jurafsky

and Martin

7

Bottom-Up Search

Speech and Language Processing -

Jurafsky

and Martin

8

Bottom-Up Search

Speech and Language Processing -

Jurafsky

and Martin

9

Bottom-Up Search

Speech and Language Processing -

Jurafsky

and Martin

10

Bottom-Up Search

Speech and Language Processing -

Jurafsky

and Martin

11

Bottom-Up Search

Speech and Language Processing -

Jurafsky

and Martin

12

Top-Down and Bottom-UpTop-down

Only searches for trees that can be answers (i.e. S’s)But also suggests trees that are not consistent with any of the wordsBottom-upOnly forms trees consistent with the wordsBut suggests trees that make no sense globally

Speech and Language Processing -

Jurafsky

and Martin

13

ControlOf course, in both cases we left out how to keep track of the search space and how to make choices

Which node to try to expand nextWhich grammar rule to use to expand a nodeOne approach is called backtracking.Make a choice, if it works out then fineIf not then back up and make a different choiceSame as with ND-Recognize

Speech and Language Processing -

Jurafsky

and Martin

14

Problems

Even with the best filtering, backtracking methods are doomed because of two inter-related problemsAmbiguity and search control (choice)Shared sub-problems

Speech and Language Processing -

Jurafsky

and Martin

15

Ambiguity

Speech and Language Processing -

Jurafsky

and Martin

16

Shared Sub-ProblemsNo matter what kind of search (top-down or bottom-up or mixed) that we choose...

We can’t afford to redo work we’ve already done.Without some help naïve backtracking will lead to such duplicated work.

Speech and Language Processing -

Jurafsky

and Martin

17

Shared Sub-Problems

Consider“A flight from Indianapolis to Houston on TWA”

Speech and Language Processing -

Jurafsky

and Martin

18

Sample L1 Grammar

Speech and Language Processing -

Jurafsky

and Martin

19

Shared Sub-ProblemsAssume a top-down parse that has already expanded the

NP rule (dealing with the Det) Now its making choices among the various Nominal rulesIn particular, between these twoNominal -> NounNominal -> Nominal PPStatically choosing the rules in this order leads to the following bad behavior...

Speech and Language Processing -

Jurafsky

and Martin

20

Shared Sub-Problems

Speech and Language Processing -

Jurafsky

and Martin

21

Shared Sub-Problems

Speech and Language Processing -

Jurafsky

and Martin

22

Shared Sub-Problems

Speech and Language Processing -

Jurafsky

and Martin

23

Shared Sub-Problems

Speech and Language Processing -

Jurafsky

and Martin

24

Dynamic Programming

DP search methods fill tables with partial results and therebyAvoid doing avoidable repeated workSolve exponential problems in polynomial time (ok, not really)Efficiently store ambiguous structures with shared sub-parts.We’ll cover one approach that corresponds to a bottom-up strategyCKY

Speech and Language Processing -

Jurafsky

and Martin

25

CKY Parsing

First we’ll limit our grammar to epsilon-free, binary rules (more on this later)Consider the rule A  BCIf there is an A somewhere in the input generated by this rule then there must be a B followed by a

C

in the input.

If the

A spans from i to j in the input then there must be some k st. i<k<jIn other words, the B splits from the C someplace after the i and before the j.

Speech and Language Processing -

Jurafsky

and Martin

26

Problem

What if your grammar isn’t binary?As in the case of the TreeBank grammar?Convert it to binary… any arbitrary CFG can be rewritten into Chomsky-Normal Form automatically.What does this mean?The resulting grammar accepts (and rejects) the same set of strings as the original grammar.But the resulting derivations (trees) are different.

Speech and Language Processing -

Jurafsky

and Martin

27

ProblemMore specifically, we want our rules to be of the form

A  B COrA  w

That is, rules can expand to either 2 non-terminals or to a single terminal.

Speech and Language Processing -

Jurafsky

and Martin

28

Binarization Intuition

Eliminate chains of unit productions.Introduce new intermediate non-terminals into the grammar that distribute rules with length > 2 over several rules. So… S  A B C turns into S



X C

and

X  A BWhere X is a symbol that doesn’t occur anywhere else in the grammar.

Speech and Language Processing -

Jurafsky

and Martin

29

Sample L1 Grammar

Speech and Language Processing -

Jurafsky

and Martin

30

CNF Conversion

Speech and Language Processing -

Jurafsky

and Martin

31

CKY

Since each LHS non-terminal consists of exactly two RHS non-terminals, a 2-dimensional table can be used to record constituents found.Each cell [i,j] contains the set of (LHS) non-terminals that span i through j in the sentence.

0

Book

1

that 2 flight 3So a non-terminal spanning an entire sentence will sit in cell [0, n].Then for a sentence of length n, the algorithm uses only the upper-triangular portion of an (n+1) x (n+1) table.

Speech and Language Processing -

Jurafsky

and Martin

32

CKYFor a rule like

A  B C we should look for a B in [i,k] and a

C

in

[

k,j].In other words, if we think there might be an A spanning i,j in the input… AND A  B C is a rule in the grammar THENThere must be a B in [i,k] and a

C

in

[

k,j

]

for some k such that

i

<k<j

Speech and Language Processing -

Jurafsky

and Martin

33

CKYSo to fill the table loop over the cells [

i,j] values in some systematic wayThen for each cell, loop over the appropriate k values to search for things to add.Add all the derivations that are possible for each [i,j] for each k

Speech and Language Processing -

Jurafsky

and Martin

34

CKY Table

Speech and Language Processing -

Jurafsky

and Martin

35

Example

Speech and Language Processing -

Jurafsky

and Martin

36

CKY Algorithm

Looping over columnsFilling

bottom

cell

Filling row

i in column jLooping over the possible split locations between i and j.Check the grammar for rules that link the constituents in [i,k] with those in [k,j]. For each rule found store the LHS of the rule in cell [i,j].

Speech and Language Processing -

Jurafsky

and Martin

37

Example

Filling column 5

Speech and Language Processing -

Jurafsky

and Martin

38

Example

Speech and Language Processing -

Jurafsky

and Martin

39

Example

Speech and Language Processing -

Jurafsky

and Martin

40

Example

Speech and Language Processing -

Jurafsky

and Martin

41

Example

Speech and Language Processing -

Jurafsky

and Martin

42

ExampleSince there’

s an S in [0,5] we have a valid parse.Are we done? We sort of left something out of the algorithm.

Speech and Language Processing -

Jurafsky

and Martin

43

CKY Notes

Since it’s bottom up, CKY hallucinates a lot of silly constituents.Segments that by themselves are constituents but cannot really occur in the context in which they are being suggested.To avoid this we can switch to a top-down control strategyOr we can add some kind of filtering that blocks constituents where they can not happen in a final analysis.

Speech and Language Processing -

Jurafsky

and Martin

44

CKY NotesWe arranged the loops to fill the table a column at a time, from left to right, bottom to top.

This assures us that whenever we’re filling a cell, the parts needed to fill it are already in the table (to the left and below)It’s somewhat natural in that it processes the input a left to right a word at a timeKnown as onlineCan you think of an alternative strategy?

Speech and Language Processing -

Jurafsky

and Martin

45

Note

An alternative is to fill a diagonal at a time.That still satisfies our requirement that the component parts of each constituent/cell will already be available when it is filled in.Filling a diagonal at a time corresponds naturally to what task?

Speech and Language Processing -

Jurafsky

and Martin

46