Embodied construction grammar - PowerPoint Presentation

Slide1

Embodied construction grammar

CSCTR Session 8

Dana

Retov

áSlide2

group at UC Berkeley

&

Uni of HawaiiNancy ChangBenjamin BergenJerome Feldman, …General assumptionSemantic relations could be extracted from language input“In its communicative function, language is a set of tools with which we attempt to guide another mind to create within itself a mental representation that approximates one we have.” (Delancey 1997)

NTLSlide3

Listener and speaker have to share enough experienceLanguage can be expressed by a discrete set of parameters and by semantic relations among entities and actions.

How these relations are encoded in the sequences of letters and sounds?

LanguageSlide4

A word that conveys some meaning

“in, on, through”

Word order“red fire engine” vs. “fire engine red”Some change in a base word -”ed” ending for the past tenseSystematic change in spelling (“car”-> “cars”)

Converting a verb to a noun (“evoke”->”evocation”)

3 mechanisms for conveying a semantic relationSlide5

Context Free GrammarSlide6

Analysis of simple sentence by CFGSlide7

S -> VP NPVP.person

<->

NP.personVP.gender <-> NP.genderVP.number <-> NP.numberSolutionSlide8

Context

The meaning of

indexicals“here”, “now”Referents of expressions“they”, “that question”Ambiguous sentences“Harry waked into the café with the singer”MetaphorsIntonation (e.g. stress, irony,…)

“HARRY walked into the café.”“Harry WALKED into the café.”Gestures

What CFG cannot cover?Slide9

Language understandingSlide10

Meanings reside in words

Each word has multiple fixed meanings –

word sensesRules of grammar are devoid of meaning and only specify which combinations of words are allowedMeaning of any combination of words can be determined by first detecting which sense of each word is involved and then using the appropriate rule for each word sense.“stone lion”Should each animal name like “lion” have another word sense covering animal-shaped objectsTraditional theorySlide11

Each word activates alternative meaning

subnetwork

These subnetworks themselves are linked to other circuits representing the semantics of words and frames that are active in the current context.The meaning of a word in context is captured by the joint activity of all of the relevant circuitryNTL – alternative theorySlide12

To write down rules of grammar that are understandable by people and computer programs and that also characterize the way our brains actually process language

The job of grammar is to specify which semantic schemas are being evoked, how they are parameterized and how they are liked together in the semantic specification.

To formalize cognitive linguisticsGoal of NTL’s embodied grammarSlide13

Construction = pairing of linguistic form and meaning

All levels of linguistic form (prefixes, words, phrases, sentences, stories, etc.) can be represented as mapping from some regularities of form to some semantic relations in the semantic specification

“embodied”Semantic part of a construction is composed of various kinds of embodied schemasImageForce dynamicactionEmbodied construction grammarSlide14

Simulation-based language understanding

Analysis Process

Semantic

Specification

“Harry walked into the cafe.”

Utterance

CAFE

Simulation

Belief State

General Knowledge

Constructions

construction

W

ALKED

form

self

f

.phon



[wakt]

meaning :

Walk-Action

constraints

self

m

.time

before

Context.speech-time

self

m

..aspect



encapsulatedSlide15

“Harry strolled to Berkeley”Individual word

simplest construction (lexical)

Lexical construction To |From subcase of Spatial Preposition evokes SPG as s form “to” |“from”

meaning Trajector-Landmark

lm <-> s.goal |lm <-> s.source

traj <-> s.traj

ExampleSlide16

Construction

Spatial PP

subcase of Destination constituents r: Spatial Preposition base: NP form r < base

meaning

r.lm <-> baseIn CFG: Spatial PP -> Spatial Preposition NP

Spatial Prepositional PhrasesSlide17

SemSpec

for “Harry strolled into Berkley”Slide18

Lexical construction Harry

subcase of NP form “Harry” meaning Referent Schema type <-> person gender <-> male

count <-> one specificity <-> known

resolved <-> harry2“Harry”Slide19

SemSpec

for “Harry strolled into Berkeley”Slide20

Lexical construction Strolled

subcase of Motion Verb, Regular Past form “stroll+ed” meaning WalkX speed <-> slow

tense <-> past aspect <-> completed

“Strolled”Slide21

Only single parameter controls the rate of moving one leg after the other

Leg moves only after the other is stable

As opposed to runningWalkX schemaSlide22

SemSpec

for “Harry strolled into Berkeley”Slide23

Lexical construction Strolled

subcase of Motion Verb, Regular Past form “stroll+ed” meaning WalkX speed <-> slow

tense <-> past aspect <-> completed

“Strolled”Slide24

SemSpec

for “Harry strolled into Berkley”Slide25

Construction

Self-Directed Motion

subcase of Motion Clause constituents movA: NP actV: Motion Verb locPP

: Spatial PP form mover < action < direction

meaning Self-Motion Schema mover <-> movA

action <-> actV direction <->

locPPSelf-directed motionSlide26

SemSpec

for “Harry strolled into Berkley”Slide27

ECG’s formalized schemas are just a way of writing down hypothesized neural connections and bindings.

These schemas are connected to semantic specification (

SemSpec).The simulation semantics process uses SemSpec and other activated knowledge to achieve conceptual integration and the resulting inferencesWhat is the difference between

ECG and other formal notations of gramar rules?Slide28

Normally “sneeze” is intransitive

Traditional grammar would suggest separate word sense for sneeze as a transitive verb

ECG would need caused motion constructionConstruction Caused Motion subcase of Motion Clause constituents

causer: Agent action: Motion

trajector: Movable object direction:

SpatialSpec form causer < action < trajector < direction

meaning Caused Motion Schema causer <-> action.actor

direction <-> action.location

“She sneezed the tissue off the table”Slide29

In traditional view “opened” refers to one sense of beer while “drank” to another

“Beer” sometimes stands for a “container of beer”

In ECG we use measure phrase constructionConstruction Measure NP subcase of NP constituents

m: Measure NP “of”

s: Substance NP form m < “of” < s meaning

Containment Schema vessel <-> m contents <-> s

“She opened and drank an expensive large beer”Slide30

Schema

Construction

MapmetaphorsMental spaceCan formalize “Josh said that Harry strolled to Berkeley”Talking about other times, places, other people’s thoughts, etc.

4 basic formal structures to formalize cognitive linguisticsSlide31

Computer understanding systemsNarayanan (1997)

Analysis of metaphors in news articles

Used pre-processed semanticsBryant (2004)Program that derives semantic relations that underlie English sentencesLater Bryant, Narayanan and Sinha combined the two modelsUse of ECGSlide32

Human processing:

What can ECG tell us about natural intelligence?

Garden-path sentences“The horse raced past the barn fell”Narayanan et al. 1988 – computer model that gives detailed predictions of how various factors (frequency of individual words, likelihood that they appear in certain constructions, etc.) interact in determining the difficulty of a garden-path situation.“The witness examined by the lawyer turned out to be unreliable”“The evidence examined by the lawyer turned out to be unreliable”

Chang (2006)Model how children learn grammar

Use of ECGSlide33

Learning constructionsSlide34

First wordsSlide35

Image schemasTopological

E.g. a container

OrientationalE.g. “in front of”Force-dynamicE.g. “against”Reference object and smaller objectLandmark and trajectorUnderstanding prepositionsSlide36

English

ON

AROUND

OVER

IN

Bowerman & PedersonSlide37

Dutch

Bowerman & Pederson

ANN

OM

BOVEN

IN

OPSlide38

Chinese

Bowerman & Pederson

SHANG

ZHOU

LISlide39

“Into” binds

inside

to a goalSlide40

Language and thought

“El jam

ón prueba salado“ Computational modelsConnectionist networksNeural systemsLevels of descriptionSlide41

Emulates a child viewing a simple geometric scene and being told a word that describes something about that scene

Has universal structure – visual system

2 classes of visual featuresQuantitative geometric features (e.g. angles)Qualitative topological features (e.g. contact)ComponentsCenter-surround cells, edge-sensitive cells, etc.Trained with a series of word-image pairsStandard back-propagation learningLater extended with motion prepositions (into, through, around)

Reiger (1996)Slide42

ModelSlide43

Children perform and plan actions long before they learn to describe them

Idea of characterizing actions by

parametersMotor control has its hierarchyLower levelCoordination, inhibitionHigher levelDesired speedWe can create abstract neural models of motor control systemsexecuting schemas

Action wordsSlide44

“Push” and “walk” schemasSlide45

Child learning of action wordsPerforming an action and hearing her parent’s label

Restricted to actions that can be carried out by one hand on a table

Bailey (1997)Slide46

Intermediate set of feature structures

Parameterization of action

Chosen to fit the basic X-schemasBi-directional arrowsLabeling pathwayCommand pathwayModelSlide47

4 steps in learning “push”Slide48

Model how children learn their first rules of grammar and generalize them in more adult-like rules

Chang (2006)Slide49

Suppose the child knows lexical construction for words “throw” and “ball”

But does not know construction for the phrase “throw ball”

“You throw the ball”Slide50
Slide51

She learned that the second word determines which object fills the thrown role of a throw action

Only later learns generalization of this construction that works for any transitive verb

New grammar ruleSlide52

Key to understanding grammar acquisition is not the famous poverty of stimulus

but rather the

richness of the substrateChild already has rich base of conceptual and embodied experienceThe reason why understanding is ahead of productionChild can understand complex sentences by matching constructions to only parts of the utteranceConstructions are the same in bothGrammar learningSlide53

Decay of unused knowledgePeople always choose the set of constructions that best fits an input

If you keep track of best matches and

Increase the potential value of successful constructionsDecrease probability of trying not-useful constructionsThere would always be a better choiceBest-matchGiven a sentence S and a grammar G, the best analysis is the one that maximizes the probability of sentence S being generated by grammar GNo need for negative evidenceSlide54

Lifting (learning

superordinate

categories)Taking a collection of relations of similar form and replacing the common element with a parameterAfter learning that cows, dogs, horses and pigs all move and eat and make noises, a good learning system will postulate a category (animal) and just remember what goes in the category and what relations to apply to membersOccurs also in grammar learningVery early child generalizes e.g. throw-ball to other small objectsGeneralisation