Maximum Likelihood-Maximum Entropy Duality PowerPoint Presentation, PPT

Maximum Likelihood-Maximum Entropy Duality PowerPoint Presentation, PPT - DocSlides

: Session 1. Pushpak Bhattacharyya. Scribed by . Aditya. Joshi. Presented in NLP-AI talk on 14. th. January, 2014. Phenomenon/Event could be a linguistic process such as POS tagging or sentiment prediction.. ID: 271485

By cheryl-pisano
Watch All docs

Likes : 8
Views : 83

Download this presentation

Click below link (As may be) to get this presentation.

Maximum Likelihood-Maximum Entropy Duality

Download Note - The PPT/PDF document "Maximum Likelihood-Maximum Entropy Duali..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Presentations text content in Maximum Likelihood-Maximum Entropy Duality

Maximum Likelihood-Maximum Entropy Duality: Session 1
Pushpak Bhattacharyya
Scribed by
Aditya
Joshi
Presented in NLP-AI talk on 14
th
January, 2014
Phenomenon/Event could be a linguistic process such as POS tagging or sentiment prediction.
Model uses data in order to “predict” future observations w.r.t. a phenomenon
Data/Observation
Phenomenon/Event
Model
Notations
X : x
1
, x
2
, x
3
....
x
m
(m observations)
A: Random variable with n possible outcomes such as a
1
, a
2
, a
3
... a
n
e.g. One coin throw :
a
1
= 0,
a
2
=1
One dice throw: a
1
=1, a
2
=2, a
3
=3, a
4
=4, a
5
=5, a
6
=6
Goal
Goal: Estimate P(ai) = Pi
Two paths
ML
ME
Are they equivalent?
Calculating probability from data
Suppose in X:
x
1
, x
2
, x
3
....
x
m
(m observations),
a
i
occurs
f(
a
i
) =
f
i
times
e.g. Dice: If outcomes are 1 1 2 3 1 5 3 4 2 1
F(1) = 4, f(2) = 2, f(3) = 2, f(4) = 1, f(5) = 1, f(6)=0 and m = 10
Hence, P
1
= 4/10, P
2
=2/10, P
3
=2/10, P
4
=1/10, P
5
=1/10, P
6
=1/10
In general, the task is...
Task:
Get
θ
: the probability vector <P(
θ
i
)> from X
MLE
MLE: θ* = argmax Pr (X; θ)With i.i.d. (identical independence) assumption, θ* = argmax π Pr (X; θ)Where, θ : <P1, P2, ... Pn>
θ
θ
i=1
m
P(a1)
P(a2)
P(an)
What is known about: θ : <P1, P2, ... Pn>
Σ Pi = 1Pi >= 0 for all iIntroducing Entropy: H(θ)= - ΣPi ln Pi
i=1
n
i=1
n
Entropy of distribution <P1, P2, ... Pn>
Some intuition
Example with diceOutcomes = 1,2,3,4,5,6P(1) + P(2)+P(3)+...P(6) = 1Entropy(Dice) = H(θ)= - ΣP(i) ln P(i)Now, there is a principle called Laplace’s Principle of Unbiased(?) reasoning
i=1
6
0
The best estimate for the dice
P(1) = P(2) = ... P(6) = 1/6
We will now prove it assuming:
NO KNOWLEDGE about the dice except that it has six outcomes, each with probability >= 0 and
Σ
P
i
= 1
1
What does “best” mean?
“BEST” means most consistent with the
situiation
.
“Best” means that these Pi values should be such that they maximize the entropy.
2
Optimization formulation
Max. - ΣP(i) log P(i)Subject to:
i=1
6
Σ Pi = 1Pi >= 0 for i = 1 to 6
i=1
6
3
Solving the optimization (1/2)
Using Lagrangian multipliers, the optimization can be written as:Q = - ΣP(i) log P(i) – λ ( ΣP(i) – 1) - Σ βi P(i)
i=1
6
i=1
6
i=1
6
For now, let us ignore the last term. We will come to it later.
4
Solving the optimization (2/2)
Differentiating Q w.r.t. P(i), we getδQ/δP(i) = - log (P(i) – 1 – λEquating to zero,log P(i) + 1 + λ = 0log P(i) = -(1+ λ)P(i) = e -(1+ λ)
This means that to maximize entropy, every P(
i
) must be equal.
5
This shows that P(1) = P(2) = ... P(6)
But,
P(1) + P(2) + .. + P(6) = 1
Therefore P(1) = P(2) = ... P(6) = 1/6
6
Introducing data in the notion of entropy
Now, we introduce data:X : x1, x2, x3.... xm (m observations)A: a1, a2, a3... an (n outcomes)
e.g. For a coin, In absence of data: P(H) = P(T) = 1/2 ... (As shown in the previous proof)However, if data X is observed as follows: Obs-1: H H T H T T H H H (m=10) (n=2) P(H) = 6/10, P(T) = 4/10Obs-2: T T H T H T H T T T (m=10) (n=2) P(H) = 3/10, P(T) = 7/10
Which of these is a valid estimate?
7
Change in entropy
Entropy reduces as data is observed!
Emax
(uniform distribution)
E2 : P(H) = 3/10
E1 : P(H) = 6/10
8
Start of Duality
Maximizing entropy in this situation is same as minimizing the `entropy reduction’ distance. i.e. Minimizing “relative entropy”
Emax
(Maximum entropy:uniform distribution)
Edata
(Entropy whenObservations are made)
Entropy reduction
9
Concluding remarks
Thus, in this discussion of ML-ME duality, we will show that:
MLE minimizes relative entropy distance from uniform distribution.
Question:
The entropy corresponds to probability vectors. The distance can be measured by squared distances.
WHY
relative entropy?
0
1
2
3