Sparse and Explicit Word Representations Omer Levy Yoav Goldberg Bar Ilan University Israel Papers in ACL 2014 Sampling error 100 Neural Embeddings Representing words as vectors is not new ID: 384780
Download Presentation The PPT/PDF document "Linguistic Regularities in" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Linguistic Regularities in Sparse and Explicit Word Representations
Omer Levy Yoav GoldbergBar-Ilan UniversityIsraelSlide2
Papers in ACL 2014*
* Sampling error: +/- 100%Slide3
Neural Embeddings
Slide4
Representing words as vectors is not new!Slide5
Explicit Representations (Distributional)
Slide6
Questions
Are analogies unique to neural embeddings?Compare neural embeddings with explicit representationsWhy does vector arithmetic reveal analogies?Unravel the mystery behind neural embeddings and their “magic”Slide7
BackgroundSlide8
Mikolov et al. (2013a,b,c)
Neural embeddings have interesting geometriesSlide9Slide10
Mikolov et al. (2013a,b,c)
Neural embeddings have interesting geometriesThese patterns capture “relational similarities”Can be used to solve analogies:
man
is to
woman
as
king
is to
queenSlide11
Mikolov et al. (2013a,b,c)
Slide12
Mikolov et al. (2013a,b,c)
Slide13
Mikolov et al. (2013a,b,c)Slide14
Mikolov et al. (2013a,b,c)Slide15
Mikolov et al. (2013a,b,c) Slide16
Mikolov et al. (2013a,b,c) Slide17
Mikolov et al. (2013a,b,c) Slide18
Mikolov et al. (2013a,b,c)
Slide19
Are analogies unique to neural embeddings?Slide20
Experiment: compare embeddings to explicit representations
Are analogies unique to neural embeddings?Slide21
Are analogies unique to neural embeddings?Experiment: compare embeddings to explicit representationsSlide22
Are analogies unique to neural embeddings?
Experiment: compare embeddings to explicit representationsLearn different representations from the same corpus:Slide23
Are analogies unique to neural embeddings?
Slide24
Analogy Datasets
Slide25
Embedding vs Explicit (Round 1)Slide26
Embedding vs Explicit (Round 1)
Many analogies recovered by explicit, but many more by embedding.Slide27
Why does vector arithmetic reveal analogies?Slide28
Why does vector arithmetic reveal analogies?
Slide29
Why does vector arithmetic reveal analogies?
Slide30
Why does vector arithmetic reveal analogies?
Slide31
Why does vector arithmetic reveal analogies?
Slide32
Why does vector arithmetic reveal analogies?
Slide33
Why does vector arithmetic reveal analogies?
Slide34
Why does vector arithmetic reveal analogies?
Slide35
Why does vector arithmetic reveal analogies?
royal?
female?Slide36
What does each similarity term mean?Observe the joint features with explicit representations!
uncrowned
Elizabeth
majesty
Katherine
second
impregnate
…
…Slide37
Can we do better?Slide38
Let’s look at some mistakes…Slide39
Let’s look at some mistakes…
Slide40
Let’s look at some mistakes…
Slide41
Let’s look at some mistakes…
Slide42
The Additive Objective
Slide43
The Additive Objective
Slide44
The Additive Objective
Slide45
The Additive Objective
Slide46
The Additive Objective
Slide47
The Additive Objective
Problem
:
one similarity might dominate the
rest
Much more prevalent in
explicit
representation
Might explain why explicit underperformedSlide48
How can we do better?Slide49
How can we do better?
Instead of adding similarities, multiply them!Slide50
How can we do better?
Slide51
How can we do better?
Slide52
Embedding vs Explicit (Round 2)Slide53
Multiplication > AdditionSlide54
Explicit is on-par with EmbeddingSlide55
Explicit is on-par with EmbeddingEmbeddings are not “magical”
Embedding-based similarities have a more uniform distributionThe additive objective performs better on smoother distributionsThe multiplicative objective overcomes this issueSlide56
Conclusion
Are analogies unique to neural embeddings?No! They occur in sparse and explicit representations as well.Why does vector arithmetic reveal analogies?
Because
vector arithmetic
is equivalent to
similarity arithmetic
.
Can we do better?
Yes!
The
multiplicative objective
is significantly better.Slide57
More Results and Analyses (in the paper)
Evaluation on closed-vocabulary analogy questions (SemEval 2012)Experiments with a third objective function (PairDirection)Do different representations reveal the same analogies?
Error analysis
A feature-level interpretation of how word similarity reveals analogiesSlide58
Slide59
Agreement
ObjectiveBothCorrectBoth
Wrong
Embedding
Correct
Explicit
Correct
MSR
43.97%
28.06%
15.12%
12.85%
Google
57.12%
22.17%
9.59%
11.12%Slide60Slide61
Error Analysis: Default BehaviorA certain word acts as a “prototype” answer for its semantic type
Examples:daughter for feminine answersFresno for US citiesIllinois for US statesTheir vectors are the centroid of that semantic typeSlide62
Error Analysis: Verb Inflections
In verb analogies:walked is to walking as danced is to… ?The correct lemma is often found ( dance )
But with the wrong inflection (
dances
)
Probably an artifact of the window contextSlide63
The Iraqi ExampleSlide64
The Iraqi ExampleSlide65
The Additive Objective
Slide66
The Iraqi Example (Revisited)