Nobal B Niraula March 26 2014 Advisor Dr Vasile Rus Anaphora in Etymology Ancient Greek Anaphora anajora Anajora ana Ana back in an upward direction jora ID: 595201
Download Presentation The PPT/PDF document "A machine learning approach to Pronoun R..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
A machine learning approach to Pronoun Resolution in Dialogue based Intelligent Tutoring Systems
Nobal B. Niraula
March 26, 2014
Advisor: Dr.
Vasile
RusSlide2
Anaphora in EtymologyAncient Greek : Anaphora = anajora
(
Anajora)ana (Ana) back in an upward directionjora (Jora ) the act of carrying back upstreamExample:The queen hasn't arrived yet but she should be here any minute.she AnaphorThe queen(NP) AntecedentAnaphora Resolution the problem of resolving what a pronoun, or a noun phrase refers to.
Anaphora ResolutionSlide3
AnaphoraSusan dropped the plate.
It
shattered loudly.Fred was angry, and so was I. If Sam buys a new bike, I will do it as well.CataphoraBecause he was very cold, David put on his coat. His friends have been criticizing Jim for exaggerating. In their free time, the kids play video games.Pleonastic It is important to note that…..ExamplesSlide4
Susan dropped the plate. It shattered loudly.
Fred was
angry, and so was I. Coreference, Anaphora and Pronoun ResolutionsCoreference ResolutionAnaphora ResolutionPronoun ResolutionSlide5
Finding the referent of a pronoun
Plays a crucial role in written texts, dialogue, discourse
Needed to derive the “Correct Interpretation” of a textIs a complicated problems in NLP !Pronoun ResolutionTutor: What would be the path of the ball?Student: It would be a straight line.Slide6
Typical ApplicationsSlide7
Anaphora Resolution Techniques for English are adapted to specific domains and genres:
Multilingual: Spanish, French
Spoken Dialogs Spoken interactions differs from chatroom-like interactionsIn spoken dialog systems, the majority of pronouns are personal and demonstrative pronounsIn tutorial dialogues, the pronouns are mostly it, they, he and sheReferents in spoken dialogue systems can be VP-antecedents or NP-antecedents but almost all antecedents in ITSs are NP-antecedents.AdaptationSlide8
Current state-of-the-art ITSs are quite effectiveVanLehn (2011) showed that the effectiveness of computer tutors (d = 0.78) is as high as the effectiveness of human tutors.
Intelligent Tutoring SystemSlide9
InteractionsDialogue-based Learning
Learning Progression
DomainPhysics DomainScaleUsed by hundreds of students at parallelClient Desktop ComputerTablets, smart phonesTools and TechnologiesJ2EE, HTML5, Flex, NLPDeepTutorSlide10
DeepTutor
Dialogue History
Task/ Problem DescriptionCharacterInput BoxSlide11Slide12
Tutor-Student Interactions
Describe the motion of the ice cube right after the poke. Would the cube be moving in a straight line, or would it have a curved path?
TutorStudentI DONT WATN TOSlide13
Tutor-Student Interactions
Can you articulate the definition or principle that is most relevant to this problem?
TutorStudentNope
I don't quite get you. Maybe you should elaborate a bit more on your answer?
Tutor
Student
Did I stutter?Slide14
Tutor-Student Interactions
Your answer is a little too short. Can you please elaborate?
TutorStudentYou can go fall in a ditch and die and I wouldn't even careSlide15
Gaming the SystemImperfect ModelsDialogue ModelInteractivity
Assessment model
Failed to understanding students’ inputs…Possible ReasonsHow does the decomposition principle apply to this situation?TutorStudent
nope you're stupid, pick up my answers you stupid computer
Incorrect Assessment
Incorrect Feedback
FrustrationSlide16
Misspelling separately VS separatelythird
vs
thirrdon vs noNormalization0 vs zero1st vs first9.8m/s2 vs constant accelerationSemanticsvertical vs y-directionidentical vs constantis vs equalsEllipsisGroundingPronoun ResolutionSome issues on AssessmentTutor: What would be the path of
the ball
?
Student
:
It
would be a straight line.
?
?
Tutor
: What would be
the path
of
the ball
?
Student
:
It
would be a straight line.
Student
:
The path
would be a straight line.Slide17
Quite Frequent:
22% of the total students turns contain at least one pronoun
Pronoun Usage in ITSWhat can you say about the motion of the desk after the mover stops pushing ? Explain why.Tutor
Student
The
desk
will stop moving because it was only moving due to the applied force of the mover pushing on it. It does not have a constant velocity or acceleration to keep it going.Slide18
Intra-turn
Inter-turn immediate
Inter turn historyThree types of UsageWhat does Newton’s second law say ?Tutor
Student
For every
force
, there is another equal force to counteract
it
What can you say about the
acceleration
of the piano?
Tutor
Student
It
remains constant
Can you please elaborate ?
Tutor
Student
It
is increasing
Since the ball’s velocity is upward and its acceleration is downward, what is happening to the ball’s
velocity
?
Tutor
Student
increasingSlide19
A lot of methods are proposed in LiteratureFocus on Written TextsKnowledge-poor
Rule Based
Hand Crafted RulesClassificationMachine Learning (Supervised)Needs Training DataMethodsSlide20
1000 InstancesFreely Available: http://language.memphis.edu/nobal/AR
MUC-like Annotation
Pronoun<p id=“1627_1” refid=“16271_2”>it</p> has id and refid<np> is the referent of<p>refid is the id of the referent… <np id=“1627_2" min=“trajectory”> the trajectory of the puck</np>Data Set: The DARE CorpusSlide21
An InstanceSlide22
Example: I, me, us, my
etc.
Id: 0Annotation Example: Q: What can you say about<np id=“1627" min=“trajectory”> the trajectory of the puck</np> ?A: <id = “1627_2” refid= “0” > I </p> don't know. Cases(1/4): First person personal pronounsSlide23
Some pronouns do not have any referentsExample<p id=“501_1” refid
=“
-1”>It</p> is true that Newton's first law can be applied in such situations. Id: -1Cases(2/4): PleonosticSlide24
Soft
Hard
Cases (3/4): Communication BreakdownsSlide25
Q: What does Newton's second law tell you about <np id="84_1" min="acceleration">the acceleration of the ball</np>? A
: <p id="84_2"
refid="84_1">it</p> is equal to the force diveded <p id="84_3" refid="-3" comments="Typo" >my</p> massCases (4/4): OthersSlide26
5 pairs of AnnotatorsRemember, a Referent has: Location
,
Short referent (min attribute), Long referent Agreements on refid є {hasNPId,0,-1,-2,-3}Kappa: 0.83, 0.88, 0.72, 0.81, 0.82 Agreements on short referentKappa: 0.87, 0.83, 0.88, 0.83, 0.81Agreements on long referentKappa: 0.82, 0.65, 0.84, 0.65, 0.74Agreements on location + positionShort: 0.79, 0.66, 0.80, 0.60, and 0.70Long: 0.76, 0.58, 0.77, 0.58, 0.56AnnotationSlide27
Communication breakdownComplex nouns
DisagreementsSlide28
StatisticsSlide29
The Top PronounsSlide30
The Top LocationsSlide31
DARE: Deep Anaphora Resolution EngineBaseline Heuristics based
EDM 2013
DARE++: Deep Anaphora Resolution Engine++Based on Machine learning CICLing 2014Resolution of PronounsSlide32
Tutor: What does Newton’s second law say ?Student: for every
force
, there is another force to counteract itTutor: What can you say about the acceleration of the piano based on Newton’s second law and the fact that the force of gravity acts on the piano ?Student: It remains constant. DARE: The idea …Slide33
Uses heuristics + Rules (Stanford CoreNLP)
Algorithm
Input = “”If a pronoun “p” is at the beginning of student’s responseInput =corresponding tutor’s question(Q) +student response(A)Else Input = student response(A) Resolve Pronoun in input using Stanford CoreNLP LimitationsThe assumptions don’t cover the casesThe algorithm doesn’t look beyond Q DARE - AlgorithmSlide34
Machine learning basedUses classification techniqueNeeds +
ve
and –ve examplesTrain Machine learning algorithms DARE++Gender Agrees ?Number Agrees ?Person of P ......ClassYesYes3rd
...
...
true
No
Yes
3
rd
...
...
false
...
...
...
...
...
...
Features
Category
(Stuntman, he)
(Julia, he)
Mention-pairs
( …,…)Slide35
Positive Examples (+ve
mention pairs)
Easy to generate(motion , it) Negative Examples (-ve mention pairs)Any (noun, pronoun) pairs between the pronoun and its referent(stuntman , it)Pronouns without any referent (e.g. pleonastic) are considered –veDARE++ ApproachQ: What can you say about <p id=“3624_2" min=“motion”> the motion of the stuntman </np> after he jumps?A: <p id=“3624_2
"
refid
=“3624_1">
it
</p>
will be
parabolic Slide36
FeaturesSlide37
Total 3267 examplesPositive: 955
Negative: 2312
10 Fold-cross ValidationExperimentsGender Agrees ?Number Agrees ?Person of P ......ClassYesYes3rd
...
...
true
No
Yes
3
rd
...
...
false
...
...
...
...
...
...
Features
Category
(Stuntman, he)
(Julia, he)
Mention-pairs
( …,…)Slide38
ResultsSlide39
EllipsisCataphoraDemonstrative pronouns
Communication Breakdown
Soft HardPleonasticImprovements /ExtensionsSlide40
Not Useful: Unigrams, Bigrams and Trigrams features in A, Q and Qi and their part-of-speechesWhich features are informative in tutorial dialog ?
Feature Ranking is conducted
Information GainGain RatioProminent FeaturesGender, number, personLocation of the Referent !80% referents are located in Q and A aloneGovernor Relation counts for candidateprep_of, prep_about (dependency relations)What can you say about XX of the YY ?It equals ZZ.Feature AnalysisSlide41
Created Data Set for Pronoun Resolution in Tutorial DialoguesProposed algorithms for resolving pronouns in
Tutorial Dialogues
Guided by thousands of interactionsDARE: Baseline DARE++: Machine learningConclusionSlide42
Grant:This research was supported in part by Institute for Education Sciences under awards R305A100875. Any opinions, findings, and conclusions or recommendations expressed in this material are solely the authors
’.
Colleagues: Dr. Vasile Rus, Rajendra Banjade, Dr. Dan Ştefănescu, Dr. Bill Baggette, Brent Morgan, Vivek Datla, Borhan SameiWhole DeepTutor TeamAcknowledgementsSlide43
Niraula, N.B., Rus, V., Stefanescu
, D.: Dare: Deep anaphora resolution in dialogue based intelligent tutoring systems. In: Proceedings of the 6th International Conference on Educational Data Mining (EDM 2013). pp. 266-267 (2013)
Niraula, N.B., Rus, V., Banjade, R., Stefanescu, D., Baggett, W., Morgan, B.: The DARE corpus: A resource for anaphora resolution in Dialogue based intelligent tutoring systems. In: Proceedings of Language Resources and Evaluation (LREC) (2014)Niraula, N.B. and Rus, V., A Machine Learning Approach to Pronominal Anaphora Resolution in Dialogue based Intelligent Tutoring Systems, CICLing (2014)References