An Incremental Multimodal BML Realizer for Behavior Co-Arti - PowerPoint Presentation

Slide1

An Incremental Multimodal BML Realizer for Behavior Co-Articulation and Coordination

Herwin van WelbergenDennis ReidsmaStefan KoppSlide2

Beyond turn taking interactionContinuous perception and behavior generation

Interpersonal coordinationTight coordination of the (ongoing) behavior of the agent with other (virtual) humansGoal: Fluent interaction with virtual humans (and robots)Slide3

Acknowledge reception, understanding, attitude while listeningRespond to turn taking signals

Graceful interruptionElicit backchannel feedback (and wait for it)Establish joint attentionExercise together with the interlocutorSynchronized movementConstruct utterances incrementally, on the flyNo long upfront planning

Fluent behavior: some examplesSlide4

Between the retraction of one gesture and the preparation of the nextRetraction of the first gesture is skipped

Co-articulation (or lack thereof) can have a communicative function (Kendon 1980)E.g. marking information boundariesHas to be established on the fly

Gesture co-articulationSlide5

Allow last minute changes to ongoing behaviorTop down (through BML)

Bottom up emergent behaviorBut don’t break BML constraintsReal-time, incremental, fluently connect BML incrementsUnder-specification handlingThe realizer should be smart enough to figure out unconstrained timing, generate co-articulation, ..

This should happen ‘on the fly’As late as possible

Behavior realizer requirementsSlide6

A BML block specifies some constraints in shape and timing on its behaviorsBML blocks are generally underspecified

The BML Realizer has certain realization freedomUsed to achieve natural looking motor behaviorBMLSlide7

BMLSlide8

ExampleSlide9

Offers specification for multimodal synchronization, but:No specification mechanisms for inter-personal synchronization

Limited specification capabilities for incrementalityNo specification for co-articulationBMLSlide10

SmartBody (Thiebeax

et al 2008)EMBR (Heloir and Kipp 2010)Greta (Mancini et al 2008)RealActor (Čerecović and

Pandžić 2010)Special focus on incrementality/interactional coordination

Elckerlyc (van Welbergen et al. 2010)ACE (Kopp and

Wachsmut 2004)BML

RealizersSlide11

Realizing BML the traditional waySlide12

Flexible behavior plan representation (

Reidsma et al. 2011)retains BML constraintsModified in top-down fashionSpecification mechanisms for interactional coordinationElckerlycSlide13

Incremental behavior planningSpecification mechanism for chunk composition

Emergent gesture co-articulation through bottom-up plan modificationsACESlide14
Slide15

Co-articulation can have a communicative functionThe

Behavior Planner should be able to specify whether or not co-articulation may occurResults: gesture co-articulationSlide16

<

bml id="bml2" composition="append-after(bml1)">

<speech id="speech1"> <text>

At <sync id="s1"/> 6 pm you have another appointment.

</text> </speech>

<gesture id="gesture1" lexeme="BEAT" stroke="speech1:s1"/>

</

bml

>

Specifying composition

<

bml

id="bml2" composition="

chunk-after(bml1)

">

<speech id="speech1">

<text>

At <sync id="s1"/>6 pm you have another appointment.

</text>

</speech>

<gesture id="gesture1" lexeme="BEAT" stroke="speech1:s1"/>

</

bml

>Slide17

Start a new BML block when a previous block is ‘retracting’All behaviors have a retraction phase

A BML block is ‘retracting’If all its behaviors are either finished or retractingIf needed, fluently overtake the retraction phase of a gesture in the previous block

Implementing gesture co-articulationSlide18

No co-articulationSlide19

Co-articulationSlide20

Initial planSlide21

Top down replanning stepSlide22

Bottom-up adaptation stepSlide23

Gracious interruption depends onBehavior state (e.g. not started, retracting)

Resting posture/stateCurrent position of e.g. the handThis information is available in the AnimationEngineSmart retraction using the AnimationEngineReplace

a interrupted gesture by a new oneCo-articulation

Results: gracious interruptionSlide24

InterruptionSlide25

Top down specification that a pointing gesture/gaze should remain on target until an interlocutor eventJoint attention

Feedback elicitationBottom up emergent insertion/deletion of ‘hold’ motionsUsage scenario: interactional coordinationSlide26

AsapRealizer combines state of the art features of ACE and

ElckerlycFusion provides new capabilities:Interactional coordination + automatic hold phase constructionTop down interrupt with bottom-up automatic graciousness + co-articulationTop down+bottom up adaptation capabilities are essential for fluent interaction

SummarySlide27

Questions?

http://asap-project.org

Thanks for your attentionSlide28

SAIBASlide29

Behavior/block state machineSlide30

Internal timing predictions of one or more modalities might be unpredictableE.g. for robot behavior

Bottom up adjustments can be used toAdjust the behavior itself (e.g. speed up to meet a time constraint)Move the time constraints that cannot be met, automatically adjusting the timing on other modalities linked to these constraintsUsage scenario: roboticsSlide31

Interruption in

ElckerlycResults: Interruption(a) Interrupting all behavior in bml1.

<bml id="yieldturn"><

bmlt:interrupt id="i1" target="bml1"/></bml>

(b) Interrupt all behavior in bml1 excluding gesture1.<bml id="yieldturn">

<

bmlt:interrupt

id="i1" target="bml1" exclude="gesture1"/>

</

bml

>

(c) Interrupt all behavior in bml1. Insert a behavior (

relaxArm

) that gracefully moves the gesturing arm back to its rest position.

<

bml

id="

yieldturn

">

<

bmlt:interrupt

id="i1" target="bml1"/>

<

bmlt:controller

id="

relaxArm

" class="

CompoundController

" name="

leftarmhang

"/>

</

bml

>Slide32

Gesture is specified in a TimedMotionUnit

TimedMotionUnit specification:StatePrioritySet of controlled jointsTimedMotionUnit execution:Set a joint rotation on the skeleton

Add a physical controllerSet a RestingTimedMotionUnit

AsapAnimationEngineSlide33

RestingTimedMotionUnitManages motion related to the resting state of the virtual human

Possible implementations:Static posture (as in ACE)Lower body physical balance controller…Is always executed with the lowest priorityCan create transition TMUs to the resting state

SlerpDrop the arm controller…

AsapAnimationEngineSlide34

The priority of a TMU drops in its SUBSIDING stateAsapAnimationEngine

executes TMUs in order of priorityIf a TMU needs to execute motion on joints that are already in use by a higher priority TMU, drop the TMUAutomatically create a cleanup TMU that moves other joints back to the resting stateGesture Coarticulation

in the AsapAnimationEngineSlide35

Gesture changes during execution:start

position of the handhand position at the start and end of the stroke posture state at the end of the gesture Adaptive timing and shape for gesture preparation and retraction phase

Using Fitts’ lawOnly update if no other constraints act on e.g. both start and stroke-start timing

TimedMotionUnit bottom up adaptationsSlide36

BML specification mechanisms for:

Interpersonal coordinationGracious interruption [but verbose and top-down only]Flexible plan representationAllows adaptation of the ongoing planWhile maintaining BML constraintsSo far such plan modifications were mainly managed top-down (e.g. by new BML blocks)

So far no combined shape and timing adaption mechanismsElckerlycSlide37

MURML specification for multimodal coordinationIncrementality

Gesture co-articulationBottom up adaptation of ongoing behaviorACESlide38

Top-down specification of a parameter changeIncrease the amplitude of an ongoing gesture

Bottom up adaptations:Update retraction and/or preparation timing and shape accordinglyMight adjust unconstrained shape and timing parameters to retain biological plausibilityUsage scenario: parameter changeSlide39

Gesture-speech alignmentSlide40

Real-time multimodal behavior generation for robots and virtual humansIncremental construction of behavior

Fluent connection of increments (e.g. behavior co-articulation)Allowing last minute changes of ongoing behaviorMultimodal synchronizationBuilds upon ACE, ElckerlycBML

BML 1.0 compliant (tested by RealizerTester)

AsapRealizer