Maximum Likelihood-Maximum Entropy Duality:
MT
Published · 102 slides · 0 views
1 / 1
Description
Maximum Likelihood-Maximum Entropy Duality: Session 1 Pushpak Bhattacharyya Scribed by Aditya Joshi Presented in NLP-AI talk on 14th January, 2015 PhenomenonEvent could be a linguistic process such as POS tagging or sentiment prediction.
Related Topics
Share
Embed code
Download this presentation From Below
"Maximum Likelihood-Maximum Entropy Duality:" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
01
Maximum Likelihood-Maximum Entropy Duality: Session 1 Pushpak Bhattacharyya
Scribed by Aditya Joshi
Presented in NLP-AI talk on 14th January, 2015<br>
Scribed by Aditya Joshi
Presented in NLP-AI talk on 14th January, 2015<br>
02
Phenomenon/Event could be a linguistic process such as POS tagging or sentiment prediction.
Model uses data in order to “predict” future observations w.r.t. a phenomenon Data/Observation Phenomenon/Event Model<br>
Model uses data in order to “predict” future observations w.r.t. a phenomenon Data/Observation Phenomenon/Event Model<br>
03
Notations X : x1, x2, x3.... xm (m observations)
A: Random variable with n possible outcomes such as a1, a2, a3... an
e.g. One coin throw : a1= 0, a2=1
One dice throw: a1=1, a2=2, a3=3, a4=4, a5=5, a6=6<br>
A: Random variable with n possible outcomes such as a1, a2, a3... an
e.g. One coin throw : a1= 0, a2=1
One dice throw: a1=1, a2=2, a3=3, a4=4, a5=5, a6=6<br>
04
Goal Goal: Estimate P(ai) = Pi Two paths ML ME Are they equivalent?<br>
05
Calculating probability from data Suppose in X: x1, x2, x3.... xm (m observations),
ai occurs f(ai) = fi times
e.g. Dice: If outcomes are 1 1 2 3 1 5 3 4 2 1
F(1) = 4, f(2) = 2, f(3) = 2, f(4) = 1, f(5) = 1, f(6)=0 and m = 10
Hence, P1 = 4/10, P2=2/10, P3=2/10, P4=1/10, P5=1/10, P6=1/10<br>
ai occurs f(ai) = fi times
e.g. Dice: If outcomes are 1 1 2 3 1 5 3 4 2 1
F(1) = 4, f(2) = 2, f(3) = 2, f(4) = 1, f(5) = 1, f(6)=0 and m = 10
Hence, P1 = 4/10, P2=2/10, P3=2/10, P4=1/10, P5=1/10, P6=1/10<br>
06
In general, the task is... Task: Get θ : the probability vector <P(θi)> from X<br>
07
MLE MLE: θ* = argmax Pr (X; θ)
With i.i.d. (identical independence) assumption,
θ* = argmax π Pr (X; θ)
Where,
θ : <P1, P2, ... Pn> θ θ i=1 m P(a1) P(a2) P(an)<br>
With i.i.d. (identical independence) assumption,
θ* = argmax π Pr (X; θ)
Where,
θ : <P1, P2, ... Pn> θ θ i=1 m P(a1) P(a2) P(an)<br>
08
What is known about: θ : <P1, P2, ... Pn> Σ Pi = 1
Pi >= 0 for all i
Introducing Entropy:
H(θ)= - ΣPi ln Pi i=1 n i=1 n Entropy of distribution <P1, P2, ... Pn><br>
Pi >= 0 for all i
Introducing Entropy:
H(θ)= - ΣPi ln Pi i=1 n i=1 n Entropy of distribution <P1, P2, ... Pn><br>
09
Some intuition Example with dice
Outcomes = 1,2,3,4,5,6
P(1) + P(2)+P(3)+...P(6) = 1
Entropy(Dice) = H(θ)= - ΣP(i) ln P(i)
Now, there is a principle called Laplace’s Principle of Unbiased(?) reasoning i=1 6<br>
Outcomes = 1,2,3,4,5,6
P(1) + P(2)+P(3)+...P(6) = 1
Entropy(Dice) = H(θ)= - ΣP(i) ln P(i)
Now, there is a principle called Laplace’s Principle of Unbiased(?) reasoning i=1 6<br>
10
The best estimate for the dice P(1) = P(2) = ... P(6) = 1/6
We will now prove it assuming:
NO KNOWLEDGE about the dice except that it has six outcomes, each with probability >= 0 and Σ Pi = 1<br>
We will now prove it assuming:
NO KNOWLEDGE about the dice except that it has six outcomes, each with probability >= 0 and Σ Pi = 1<br>
11
What does “best” mean? “BEST” means most consistent with the situation.
“Best” means that these Pi values should be such that they maximize the entropy.<br>
“Best” means that these Pi values should be such that they maximize the entropy.<br>
12
Optimization formulation Max. - ΣP(i) log P(i)
Subject to: i=1 6 Σ Pi = 1
Pi >= 0 for i = 1 to 6 i=1 6<br>
Subject to: i=1 6 Σ Pi = 1
Pi >= 0 for i = 1 to 6 i=1 6<br>
13
Solving the optimization (1/2) Using Lagrangian multipliers, the optimization can be written as:
Q = - ΣP(i) log P(i) – λ ( ΣP(i) – 1) - Σ βi P(i) i=1 6 i=1 6 i=1 6 For now, let us ignore the last term. We will come to it later.<br>
Q = - ΣP(i) log P(i) – λ ( ΣP(i) – 1) - Σ βi P(i) i=1 6 i=1 6 i=1 6 For now, let us ignore the last term. We will come to it later.<br>
14
Solving the optimization (2/2) Differentiating Q w.r.t. P(i), we get
δQ/δP(i) = - log (P(i) – 1 – λ
Equating to zero,
log P(i) + 1 + λ = 0
log P(i) = -(1+ λ)
P(i) = e -(1+ λ) This means that to maximize entropy, every P(i) must be equal.<br>
δQ/δP(i) = - log (P(i) – 1 – λ
Equating to zero,
log P(i) + 1 + λ = 0
log P(i) = -(1+ λ)
P(i) = e -(1+ λ) This means that to maximize entropy, every P(i) must be equal.<br>
15
This shows that P(1) = P(2) = ... P(6)
But,
P(1) + P(2) + .. + P(6) = 1
Therefore P(1) = P(2) = ... P(6) = 1/6<br>
But,
P(1) + P(2) + .. + P(6) = 1
Therefore P(1) = P(2) = ... P(6) = 1/6<br>
16
Introducing data in the notion of entropy Now, we introduce data:
X : x1, x2, x3.... xm (m observations)
A: a1, a2, a3... an (n outcomes) e.g. For a coin, In absence of data: P(H) = P(T) = 1/2 ... (As shown in the previous proof)
However, if data X is observed as follows:
Obs-1: H H T H T T H H H (m=10) (n=2)
P(H) = 6/10, P(T) = 4/10
Obs-2: T T H T H T H T T T (m=10) (n=2)
P(H) = 3/10, P(T) = 7/10 Which of these is a valid estimate?<br>
X : x1, x2, x3.... xm (m observations)
A: a1, a2, a3... an (n outcomes) e.g. For a coin, In absence of data: P(H) = P(T) = 1/2 ... (As shown in the previous proof)
However, if data X is observed as follows:
Obs-1: H H T H T T H H H (m=10) (n=2)
P(H) = 6/10, P(T) = 4/10
Obs-2: T T H T H T H T T T (m=10) (n=2)
P(H) = 3/10, P(T) = 7/10 Which of these is a valid estimate?<br>
17
Change in entropy Entropy reduces as data is observed! Emax
(uniform distribution) E2 : P(H) = 3/10 E1 : P(H) = 6/10<br>
(uniform distribution) E2 : P(H) = 3/10 E1 : P(H) = 6/10<br>
18
Start of Duality Maximizing entropy in this situation is same as minimizing the `entropy reduction’ distance. i.e.
Minimizing “relative entropy” Emax
(Maximum entropy:
uniform distribution) Edata
(Entropy when
Observations are made) Entropy reduction<br>
Minimizing “relative entropy” Emax
(Maximum entropy:
uniform distribution) Edata
(Entropy when
Observations are made) Entropy reduction<br>
19
Concluding remarks Thus, in this discussion of ML-ME duality, we will show that:
MLE minimizes relative entropy distance from uniform distribution.
Question: The entropy corresponds to probability vectors. The distance can be measured by squared distances. WHY relative entropy?<br>
MLE minimizes relative entropy distance from uniform distribution.
Question: The entropy corresponds to probability vectors. The distance can be measured by squared distances. WHY relative entropy?<br>
20
Maximum Likelihood-Maximum Entropy Duality: Session 2 Pushpak Bhattacharyya
Scribed by Aditya Joshi
Presented in NLP-AI talk on 28th January, 2015<br>
Scribed by Aditya Joshi
Presented in NLP-AI talk on 28th January, 2015<br>
21
Recap (1/2) Whatever we can show with MLE, (in most cases), we can through ME as well
Laplace’s unbiased reasoning principle
Make only the most limited assumptions.
We assume data:
X: x1, x2.... Xm m: number of observations
Every xi is the outcome of the values of a random variable<br>
Laplace’s unbiased reasoning principle
Make only the most limited assumptions.
We assume data:
X: x1, x2.... Xm m: number of observations
Every xi is the outcome of the values of a random variable<br>
22
Recap (2/2) E.g. xi = {1, 2, ...6} for dice.
If no data then by Laplace’s principle, uniform distribution is the best estimate of the probability of each outcome.<br>
If no data then by Laplace’s principle, uniform distribution is the best estimate of the probability of each outcome.<br>
23
NLP Perspective Probability of mapping of words between parallel sentences.
E.g. The mapping between “blue” and “neela”, and “blue” and “haraa” (wrong translation)
So how can you maintain probabilities of correct mappings based on the corpus?<br>
E.g. The mapping between “blue” and “neela”, and “blue” and “haraa” (wrong translation)
So how can you maintain probabilities of correct mappings based on the corpus?<br>
24
Starting off This week, we assume that we have data
X: x1.... xm
Outcome of r.v. Is : a1, a2, a3... an in the data such that each aj occurs fi times.
In absence of any data, Pj = 1/n
When we have data?<br>
X: x1.... xm
Outcome of r.v. Is : a1, a2, a3... an in the data such that each aj occurs fi times.
In absence of any data, Pj = 1/n
When we have data?<br>
25
In presence of data When we have the data, Pj = ?
Answer: Pj = f j / m WHY? Can we arrive at this value through (a) MLE, or (b) ME ??<br>
Answer: Pj = f j / m WHY? Can we arrive at this value through (a) MLE, or (b) ME ??<br>
26
Taking the MLE route MLE: We maximize data likelihood
P(X; θ) where θ = <P1, P2, ... Pn>
Under i.i.d.,
P(X; θ) = π P(xi; θ)
= π Pjfj i=1 m i=1 m e.g. 1 1 2 3 4 5 4
P1 P1 P2 P3 P4 P5 P4
... P12 P21 P31 P42 P51 Σ fj = m j=1 n Where,<br>
P(X; θ) where θ = <P1, P2, ... Pn>
Under i.i.d.,
P(X; θ) = π P(xi; θ)
= π Pjfj i=1 m i=1 m e.g. 1 1 2 3 4 5 4
P1 P1 P2 P3 P4 P5 P4
... P12 P21 P31 P42 P51 Σ fj = m j=1 n Where,<br>
27
Maximization MLE demands maximize P(X; θ)
Subject to : Σ Pj = 1
Pj>= 0 for all i j=1 n Maximize ln (P(X; θ)) Σ Pj = 1
Pj >= 0 for all i i=1 n<br>
Subject to : Σ Pj = 1
Pj>= 0 for all i j=1 n Maximize ln (P(X; θ)) Σ Pj = 1
Pj >= 0 for all i i=1 n<br>
28
Evaluating the parameter Pj (1/2) Q =Σ fj ln Pj – λ( Σ Pj – 1) j=1 n j=1 n Taking derivative w.r.t. Pj δQ/δPj = fj / Pj - λ Equating to zero,
fj / Pj – λ = 0
Pj = fj / λ (1) Taking derivative w.r.t. λ δQ/δPj = Σ Pj – 1 Equating to zero,
Σ Pj = 1 (2)<br>
fj / Pj – λ = 0
Pj = fj / λ (1) Taking derivative w.r.t. λ δQ/δPj = Σ Pj – 1 Equating to zero,
Σ Pj = 1 (2)<br>
29
Evaluating the parameter Pj (2/2) From (1) and (2), Σ fj / λ = 1 j=1 n Σ fj = λ j=1 n λ = m Pj = fj / m Proved!<br>
30
Summary No observation: Pj = 1/n
Observation: Pj = fj/m ; Σ fj = m Entropy
(as seen last
time) MLE
(as shown
Today) ME??<br>
Observation: Pj = fj/m ; Σ fj = m Entropy
(as seen last
time) MLE
(as shown
Today) ME??<br>
31
Does entropy change Before we move on to discussion on ME in case of observed data, we first see how entropy gets affected in presence of data.
In context of cases given in the previous slide, we wish to see:
Is it true that Entropy (Case 2) <= Entropy (Case 1)??
Let’s verify.<br>
In context of cases given in the previous slide, we wish to see:
Is it true that Entropy (Case 2) <= Entropy (Case 1)??
Let’s verify.<br>
32
Situation 1: No observed data Entropy(Case1): - ΣP(j) log P(j)
= - Σ(1/n) log (1/n) j=1 n j=1 n = log (1/n) = log (n)<br>
= - Σ(1/n) log (1/n) j=1 n j=1 n = log (1/n) = log (n)<br>
33
Situation 2: Observed Data (1/2) Entropy(Case2): - ΣP(j) log P(j)
= - Σ(fj/m) log (fj/m) j=1 n j=1 n = -1/m Σfj log (fj/m) j=1 n = -1/m Σfj [ log (fj) – log(m) ] j=1 n<br>
= - Σ(fj/m) log (fj/m) j=1 n j=1 n = -1/m Σfj log (fj/m) j=1 n = -1/m Σfj [ log (fj) – log(m) ] j=1 n<br>
34
Situation 2: Observed Data (2/2) Entropy(Case2): - ΣP(j) log P(j) j=1 n = -1/m[ Σfj log (fj) –Σfj log(m) ] j=1 n j=1 n = -1/m[ Σfj log (fj) –m log(m) ] j=1 n = [ m log(m) - Σfj log (fj) ] / m j=1 n<br>
35
Comparing the entropies Now, we must show: m log(m) – m log(n) <= Σfj log (fj) j=1 n i.e. m log(m/n) <= Σfj log (fj) j=1 n .... We will show this in next class.<br>
36
Change in entropy Entropy reduces as data is observed! No data, 1/n, E1 Data, fj/m, E2<br>
37
Concluding remark If you can show that MLE is same as maximum entropy, then MLE-ME duality is shown.<br>
38
Maximum Likelihood-Maximum Entropy Duality: Session 3 Pushpak Bhattacharyya
Scribed by Aditya Joshi
Presented in NLP-AI talk on 11th February, 2015<br>
Scribed by Aditya Joshi
Presented in NLP-AI talk on 11th February, 2015<br>
39
-Σ fi log Pi
s.t. Σ Pi = 1 & Σ fi = N Recap X1, X2, X3.... XN : Data
A1, A2, A3... AM : Outcomes
Goal: Estimate P(Ai) = Pi In absence of any other information (not even data),
Pi = 1/M
This was obtained using ME
Max. -Σ Pi log Pi
s.t. Σ Pj = 1 i=1 m i=1 m When data is observed,
Pi = fi/N where fi = freq(Ai)
This was obtained using ME
Max. i=1 m i=1 m i=1 m<br>
s.t. Σ Pi = 1 & Σ fi = N Recap X1, X2, X3.... XN : Data
A1, A2, A3... AM : Outcomes
Goal: Estimate P(Ai) = Pi In absence of any other information (not even data),
Pi = 1/M
This was obtained using ME
Max. -Σ Pi log Pi
s.t. Σ Pj = 1 i=1 m i=1 m When data is observed,
Pi = fi/N where fi = freq(Ai)
This was obtained using ME
Max. i=1 m i=1 m i=1 m<br>
40
Change in entropy Entropy changes as data is observed.
Today, we show that the entropy is “reduced”.
i.e. Eu > Ed No data, 1/m, Eu Data, fj/n, Ed<br>
Today, we show that the entropy is “reduced”.
i.e. Eu > Ed No data, 1/m, Eu Data, fj/n, Ed<br>
41
Today’s goal Today, we will show that the entropy is “reduced”.
i.e. Eu > Ed
Two proofs<br>
i.e. Eu > Ed
Two proofs<br>
42
Proof 1 (1/..) Suppose without loss of generality,
P1 -> P + ε P2 -> P - ε Thus, in case of uniform distribution,
P1 = P2 =... P = 1/M, and ε = 0 -Σ Pi log Pi i=1 m Eu = Ed = -(P+ε) log(P+ε) -(P-ε) log(P-ε) -Σ Pi log Pi i=3 m<br>
P1 -> P + ε P2 -> P - ε Thus, in case of uniform distribution,
P1 = P2 =... P = 1/M, and ε = 0 -Σ Pi log Pi i=1 m Eu = Ed = -(P+ε) log(P+ε) -(P-ε) log(P-ε) -Σ Pi log Pi i=3 m<br>
43
Proof 1 (2/...) Eu – Ed = -P log P + (P+ε) log(P+ε)
-P log P + (P-ε) log(P-ε)
= P log ((P+ε)/P) + ε (log P + log (1+ε/P))
+ P log ((P-ε)/P) - ε (log P + log (1-ε/P))
= P [log (1+ε/P)+ log (1-ε/P)]
+ ε [log (1+ε/P) - log (1-ε/P)]<br>
-P log P + (P-ε) log(P-ε)
= P log ((P+ε)/P) + ε (log P + log (1+ε/P))
+ P log ((P-ε)/P) - ε (log P + log (1-ε/P))
= P [log (1+ε/P)+ log (1-ε/P)]
+ ε [log (1+ε/P) - log (1-ε/P)]<br>
44
Proof 1 (3/...) Eu – Ed = P [log (1+ε/P)+ log (1-ε/P)]
+ ε [log (1+ε/P) - log (1-ε/P)]
= P [log (1 – ε2/P2)]
+ ε [log ((1+ε/P)/(1-ε/P))]
= P [log (1 – y2)]
+ ε [log ((1+y)/(1-y))] Where, y= ε/P<br>
+ ε [log (1+ε/P) - log (1-ε/P)]
= P [log (1 – ε2/P2)]
+ ε [log ((1+ε/P)/(1-ε/P))]
= P [log (1 – y2)]
+ ε [log ((1+y)/(1-y))] Where, y= ε/P<br>
45
Proof 1 (4/...) Eu-Ed = P [log (1 – y2)] + ε [log ((1+y)/(1-y))]
= P [ -y2 –y4/2 –y6/3 – y8/4...] + ε[2y + 2y3/3+2y5/5+2y7/7...]
= -Py2 [1+ y2/2+ y4/3+ y6/4...]
+ Py2 [2 + 2y2/3+ 2y4/5+ 2y6/7...] log (1+x) = x – x2/2 + x3/3 – x4/4+ ..
log ((1+x)/(1-x)) = 2x +2x3/3+2x5/5+2x7/7+.. ...substitute ε = Py When we compare the above statement term by term, (1..2), (1/2.. 2/3), (1/3...2/5).. Etc., we see that the above value is > 0. i.e. Eu – Ed > 0. i.e. Eu > Ed<br>
= P [ -y2 –y4/2 –y6/3 – y8/4...] + ε[2y + 2y3/3+2y5/5+2y7/7...]
= -Py2 [1+ y2/2+ y4/3+ y6/4...]
+ Py2 [2 + 2y2/3+ 2y4/5+ 2y6/7...] log (1+x) = x – x2/2 + x3/3 – x4/4+ ..
log ((1+x)/(1-x)) = 2x +2x3/3+2x5/5+2x7/7+.. ...substitute ε = Py When we compare the above statement term by term, (1..2), (1/2.. 2/3), (1/3...2/5).. Etc., we see that the above value is > 0. i.e. Eu – Ed > 0. i.e. Eu > Ed<br>
46
Proof 2 (1/...) Identity: for x,y > 0
y – ylog y <= x – ylog x
Proof: log t <= log t -1
Put t = (x/y) . log (x/y) <= (x/y) – 1
i.e. log x - log y <= (x-y)/y
i.e. y log x – y log y <= x – y
i.e. y – y log y <= x – y log x<br>
y – ylog y <= x – ylog x
Proof: log t <= log t -1
Put t = (x/y) . log (x/y) <= (x/y) – 1
i.e. log x - log y <= (x-y)/y
i.e. y log x – y log y <= x – y
i.e. y – y log y <= x – y log x<br>
47
Proof 2 (2/...) p1, p2.... pm = 1/M .............. Uniform
q1, q2... qm ............... Perturbed distribution
qi – qi log qi <= pi - qi log pi
take sum over i = 1....m, We put x = p, y= q in y – ylog y <= x – ylog x Σ qi – Σqi log qi <= Σ pi - Σ qi log pi But, Σ qi = 1 & Σ pi = 1 1– Σqi log qi <= 1 + log M Σ qi 1 + Ed <= 1 + log M 1 + Ed <= 1 + Eu Eu >= Ed<br>
q1, q2... qm ............... Perturbed distribution
qi – qi log qi <= pi - qi log pi
take sum over i = 1....m, We put x = p, y= q in y – ylog y <= x – ylog x Σ qi – Σqi log qi <= Σ pi - Σ qi log pi But, Σ qi = 1 & Σ pi = 1 1– Σqi log qi <= 1 + log M Σ qi 1 + Ed <= 1 + log M 1 + Ed <= 1 + Eu Eu >= Ed<br>
48
Conclusion We showed today that there is a “reduction” in entropy when data is observed.
We showed two proofs: the first is more intuitive but long; the second is simpler.<br>
We showed two proofs: the first is more intuitive but long; the second is simpler.<br>
49
Maximum Likelihood-Maximum Entropy Duality: Session 4 Pushpak Bhattacharyya
Scribed by Aditya Joshi
Presented in NLP-AI talk on 25th March, 2015<br>
Scribed by Aditya Joshi
Presented in NLP-AI talk on 25th March, 2015<br>
50
A uniform distribution and any pd P Uniform, 1/n P Let us now compute relative entropy / KLD between
these two vectors.<br>
these two vectors.<br>
51
Relative entropy between two non-negative vectors P: <p1 p2 ... pn>, Q: <q1 q2 ... qn>
D(P||Q) = Σ pi log (pi /qi) - Σ pi + Σ qi i=1 n i=1 n i=1 n KL Divergence or relative entropy between the two vectors is defined as: & D(P||Q) >= 0
& D(P||Q) = 0 occurs, when pi = qi<br>
D(P||Q) = Σ pi log (pi /qi) - Σ pi + Σ qi i=1 n i=1 n i=1 n KL Divergence or relative entropy between the two vectors is defined as: & D(P||Q) >= 0
& D(P||Q) = 0 occurs, when pi = qi<br>
52
Relative entropy between uniform distribution and any p.d. p D(P||1/n) = Σ pi log (pi /(1/n)) - Σ pi + Σ (1/n)
= Σ pi log pi + Σ pi log n - 1 + 1
= - E(P) + log n i=1 n i=1 n i=1 n D(P || 1/n ) = - E(P) + log n<br>
= Σ pi log pi + Σ pi log n - 1 + 1
= - E(P) + log n i=1 n i=1 n i=1 n D(P || 1/n ) = - E(P) + log n<br>
53
Relation between entropy of a pd and its relative entropy with uniform This shows that, if we maximize entropy P, we minimize relative entropy with respect to uniform distribution.
Conclusion: The probability distribution found by maximizing entropy is the distribution with least KLD (relative entropy) from uniform distribution. D(P || 1/n ) = - E(P) + log n argmin D(P || 1/n ) = argmax E(P) P P<br>
Conclusion: The probability distribution found by maximizing entropy is the distribution with least KLD (relative entropy) from uniform distribution. D(P || 1/n ) = - E(P) + log n argmin D(P || 1/n ) = argmax E(P) P P<br>
54
Digression What constitutes a valid proof?
Statement S, Path P
The Euclidean distance path may not lead us to proving statement S.
How statement S can be proven may depend on path P and the choice of axioms.<br>
Statement S, Path P
The Euclidean distance path may not lead us to proving statement S.
How statement S can be proven may depend on path P and the choice of axioms.<br>
55
A uniform distribution and any pd with data Uniform, 1/n f1, f2, ...fn : F Now, when we have data (n-ary) given by frequencies (f1, f2... fn) as shown above.
Let us now compute relative entropy / KLD between
these two vectors.<br>
Let us now compute relative entropy / KLD between
these two vectors.<br>
56
Relative entropy bet. uniform and pd p U: <1/n 1/n .... 1/n> PD: <p1 p2 ... pn>
D(RF||1/n) = Σ(fi/m) log(fi/m)/(1/n))
= - (1/m) Σfi log fi
= - (1/m) [ log π Pjfj ] i=1 n KL Divergence or relative entropy between the two vectors is defined as:<br>
D(RF||1/n) = Σ(fi/m) log(fi/m)/(1/n))
= - (1/m) Σfi log fi
= - (1/m) [ log π Pjfj ] i=1 n KL Divergence or relative entropy between the two vectors is defined as:<br>
57
Relation between RF and uniform We find that relative entropy
D(RF||1/n) = log n + 1/m [log-likelihood]<br>
D(RF||1/n) = log n + 1/m [log-likelihood]<br>
58
Frequency distribution F and a pd P Suppose F is the frequency of outcomes:
F : f1, f2.... fn m : total no. of observations
Let P be “some” probability distribution
Let us now find the KLD between F and P<br>
F : f1, f2.... fn m : total no. of observations
Let P be “some” probability distribution
Let us now find the KLD between F and P<br>
59
Relative entropy between frequency distribution and any p.d. p D(F||P) = Σ fi log (fi / pi ) - Σ fi + Σ pi
= Σ fi log fi - Σ log pi fi - m + 1
= constant – log-likelihod - constant i=1 n i=1 n i=1 n D(F||P) = constant – log-likelihood<br>
= Σ fi log fi - Σ log pi fi - m + 1
= constant – log-likelihod - constant i=1 n i=1 n i=1 n D(F||P) = constant – log-likelihood<br>
60
Relation between entropy of P and its relative entropy with freq. dist. This shows that, if we maximize log-likelihood of P, we minimize relative entropy with respect to frequency distribution
Conclusion: The probability distribution found by maximizing log-likelihood is the distribution with least KLD (relative entropy) from frequency distribution. D(F||P) = constant – log-likelihood argmin D(F||P) = argmax LL(P) P P<br>
Conclusion: The probability distribution found by maximizing log-likelihood is the distribution with least KLD (relative entropy) from frequency distribution. D(F||P) = constant – log-likelihood argmin D(F||P) = argmax LL(P) P P<br>
61
Concluding remarks Observations today:
The P found by maximizing entropy is the one with least RE w.r.t. uniform distribution
The P found by maximizing likelihood is the one with least RE w.r.t. frequency distribution Work to be done:
Bring in fs into (1) above
Bring in 1/n into (2) above<br>
The P found by maximizing entropy is the one with least RE w.r.t. uniform distribution
The P found by maximizing likelihood is the one with least RE w.r.t. frequency distribution Work to be done:
Bring in fs into (1) above
Bring in 1/n into (2) above<br>
62
Maximum Likelihood-Maximum Entropy Duality: Session 5 Pushpak Bhattacharyya
Scribed by Aditya Joshi
Presented in NLP-AI talk on 17th June, 2015<br>
Scribed by Aditya Joshi
Presented in NLP-AI talk on 17th June, 2015<br>
63
Recap Some notations have been corrected in this presentation
Setting:
X: X1, X2, .... Xm (data)
A: A1, A2,.... An (outcomes)
P: P1, P2, .... Pn (Parameters to be estimated)
For example:
X: 2 1 3 5 6 3 4
A: 1 2 3 4 5 6
P: P1, P2, .... P6<br>
Setting:
X: X1, X2, .... Xm (data)
A: A1, A2,.... An (outcomes)
P: P1, P2, .... Pn (Parameters to be estimated)
For example:
X: 2 1 3 5 6 3 4
A: 1 2 3 4 5 6
P: P1, P2, .... P6<br>
64
Recap: Goal To estimate parameters P, there are two paths:
Maximum Likelihood: Pi = fi/m ... As shown
Maximum entropy: Pi = 1/n ; data not considered
We also showed that: Entropy is maximum for uniform distribution Pi = 1/n for all i Hu = Entropy for uniform
Distr. = log n H = - ΣPi log Pi
Hu= - Σ(1/n) log (1/n) = log (1/n) = log n i=1 n j=1 n = log (1/n) = log (n) i=1 n<br>
Maximum Likelihood: Pi = fi/m ... As shown
Maximum entropy: Pi = 1/n ; data not considered
We also showed that: Entropy is maximum for uniform distribution Pi = 1/n for all i Hu = Entropy for uniform
Distr. = log n H = - ΣPi log Pi
Hu= - Σ(1/n) log (1/n) = log (1/n) = log n i=1 n j=1 n = log (1/n) = log (n) i=1 n<br>
65
Recap: Reduction in entropy When we have data,
We have shown in two ways that: Hu > Hd. i.e. If we perturb a distribution 1/n, 1/n .... To 1/n+k, 1/n-k, 1/n ..... , the entropy decreases Hu = Entropy for uniform
Distr. Hd = Entropy in case of data<br>
We have shown in two ways that: Hu > Hd. i.e. If we perturb a distribution 1/n, 1/n .... To 1/n+k, 1/n-k, 1/n ..... , the entropy decreases Hu = Entropy for uniform
Distr. Hd = Entropy in case of data<br>
66
Recap: Relative Entropy P: p1, p2, p3, ... pn
Q: q1, q2, q3,... qn
D(p||q) = Σ Pi = 1 i=1 n Σ qi = 1 i=1 n Σ pi log (pi / qi ) i=1 n<br>
Q: q1, q2, q3,... qn
D(p||q) = Σ Pi = 1 i=1 n Σ qi = 1 i=1 n Σ pi log (pi / qi ) i=1 n<br>
67
Recap: Relative entropy For q: uniform distribution,
D(p||q) =
=
=
= - Hp + log n = Hu - Hp Σ pi log (pi / qi ) i=1 n Σ pi log (pi / (1/n) ) i=1 n Σ pi log pi - log (1/n) i=1 n PIVOT: Relative entropy and absolute entropy difference are same for uniform distribution.<br>
D(p||q) =
=
=
= - Hp + log n = Hu - Hp Σ pi log (pi / qi ) i=1 n Σ pi log (pi / (1/n) ) i=1 n Σ pi log pi - log (1/n) i=1 n PIVOT: Relative entropy and absolute entropy difference are same for uniform distribution.<br>
68
Approach to prove duality Theorem: MLE-ME converge when:
Distance measure = Relative entropy
Distributions belong to exponential family<br>
Distance measure = Relative entropy
Distributions belong to exponential family<br>
69
Data v/s feature matrix X1 X2 Xi Xm F1
F2
Fi
Fm F X promotion Word ends with “tion” Outcome: POS tags<br>
F2
Fi
Fm F X promotion Word ends with “tion” Outcome: POS tags<br>
70
A note on Outcomes A: A1, A2 ..... An
N is tied to the classification task.
For POS tagging: (Penn Tagset) A: A1, A2 ..... A39<br>
N is tied to the classification task.
For POS tagging: (Penn Tagset) A: A1, A2 ..... A39<br>
71
Exponential & Constrained distributions Pi = ci π λj fji = ci e λj ^(fji) j=1 k P = { P s.t. E P F = E P F } Expected value Distribution
Obtained from data Distribution
intended Expected value s.t. Feature
Value is F under distribution P<br>
Obtained from data Distribution
intended Expected value s.t. Feature
Value is F under distribution P<br>
72
Equivalence between constrained and exponential Now, let:
P: Constrained distribution
Q: exponential distribution
And P* = P intersection Q
By ME, we will show that: H(P*) >= H(q) for all q
By MLE, we will show that: LL(P*) >= LL(p) for all p This is done using concepts like:
(a) Pythagorean distance for relative entropy,
(b) LL in terms of P<br>
P: Constrained distribution
Q: exponential distribution
And P* = P intersection Q
By ME, we will show that: H(P*) >= H(q) for all q
By MLE, we will show that: LL(P*) >= LL(p) for all p This is done using concepts like:
(a) Pythagorean distance for relative entropy,
(b) LL in terms of P<br>
73
Maximum Likelihood-Maximum Entropy Duality: Session 6 Pushpak Bhattacharyya
Scribed by Aditya Joshi
Presented in NLP-AI talk on 2nd September, 2015<br>
Scribed by Aditya Joshi
Presented in NLP-AI talk on 2nd September, 2015<br>
74
Recap (1/3) O: o1, o2, o3 .... on observations
A: a1, a2, a3 .... am outcomes
P: p1, p2, p3 .... pm prob. distribution
F: f1, f2, f3 .... fm Frequency Pi = fi/n .... empirical distribution
acc to MLE ~ Pi = 1/m .... distribution acc to ME,
assuming uniform<br>
A: a1, a2, a3 .... am outcomes
P: p1, p2, p3 .... pm prob. distribution
F: f1, f2, f3 .... fm Frequency Pi = fi/n .... empirical distribution
acc to MLE ~ Pi = 1/m .... distribution acc to ME,
assuming uniform<br>
75
Recap (2/3) Entropy: Hu, entropy with uniform distrib = log m Hd ... when data Hd <= Hu<br>
76
Recap (3/3) P: <p1 p2 ... pn>, Q: <q1 q2 ... qn>
D(P||Q) = Σ pi log (pi /qi) - Σ pi + Σ qi i=1 n i=1 n i=1 n KL Divergence or relative entropy between the two vectors is defined as: D (P || Uniform) = Hu - H p<br>
D(P||Q) = Σ pi log (pi /qi) - Σ pi + Σ qi i=1 n i=1 n i=1 n KL Divergence or relative entropy between the two vectors is defined as: D (P || Uniform) = Hu - H p<br>
77
A small digression Why is entropy in log form?
e.g. Suppose two dice are thrown.
Sample space: 36
<x1, y1>, <x2, y2>, <x3, y3> .... <x36, y36>
If I say xi + yi = 12 v/s xi + yi = 7, which message has more information?<br>
e.g. Suppose two dice are thrown.
Sample space: 36
<x1, y1>, <x2, y2>, <x3, y3> .... <x36, y36>
If I say xi + yi = 12 v/s xi + yi = 7, which message has more information?<br>
78
Which message is more informative? 12 : <6, 6>
Probability of each outcome = 1
Uncertainty less
7: <1,6> <2,5>, <3,4>, <4,3>, <5,2>, <6,1>
Probability of each outcome = 1/6
Uncertainty more 0 1 P Uncertainty approx.<br>
Probability of each outcome = 1
Uncertainty less
7: <1,6> <2,5>, <3,4>, <4,3>, <5,2>, <6,1>
Probability of each outcome = 1/6
Uncertainty more 0 1 P Uncertainty approx.<br>
79
Additional certainty or information If there are two independent variables, then intuitively:
I(pq) = I(p) + I(q)
The log function seems to satisfy the above requirement
e.g. Uncertainty of POS for play >= played
Less certainty -- More entropy<br>
I(pq) = I(p) + I(q)
The log function seems to satisfy the above requirement
e.g. Uncertainty of POS for play >= played
Less certainty -- More entropy<br>
80
Familiar example: POS tagging Vocabulary, V : V1, V2, V3 ... V|V|
Tag set, T : T1, T2, T3 ..... T|T|
Call |V|X|T| = A
We wish to estimate
P: P1 P2 P3 ..... PA
where
Pi = P (Vv, Tt) e.g. P(“play”, “NN”)<br>
Tag set, T : T1, T2, T3 ..... T|T|
Call |V|X|T| = A
We wish to estimate
P: P1 P2 P3 ..... PA
where
Pi = P (Vv, Tt) e.g. P(“play”, “NN”)<br>
81
Dataset and features X: (w1,t1), (w2,t2), (w3,t3), .... (wn, tn)
We introduce features, F: F1, F2, F3 ... F|F|
example:
Ff = 1 if Vv ends with “-tion”
= 0 otherwise Binary<br>
We introduce features, F: F1, F2, F3 ... F|F|
example:
Ff = 1 if Vv ends with “-tion”
= 0 otherwise Binary<br>
82
Binding features with V X T example: <promotion, N>
Features = (1) ends with –tion, (2) has 3 syllables, etc.
F is a matrix with |F| rows and |V| X |T| columns
P : P1, P2, P3 .... P|V||T| ~<br>
Features = (1) ends with –tion, (2) has 3 syllables, etc.
F is a matrix with |F| rows and |V| X |T| columns
P : P1, P2, P3 .... P|V||T| ~<br>
83
Expected value of a feature F VXT <v1,t1> <v1,t2> ..... <vi, tj> ...... F1
F2
F3
.
.
.
.
F|F|<br>
F2
F3
.
.
.
.
F|F|<br>
84
Entropy For q: uniform distribution,
Ep (Ffij) =
For empirical distribution
Ep (Ffij) = similarly. Σ Pr (Ff) Value (Ff) i=1 |V| |T| i i<br>
Ep (Ffij) =
For empirical distribution
Ep (Ffij) = similarly. Σ Pr (Ff) Value (Ff) i=1 |V| |T| i i<br>
85
Introducing two distributions Constrained distribution: (w.r.t training data) Exponential distribution: R : { r(y) : Er(F) = EP (F) }
where y belongs to V X T ~ S: { s(y) : s(y) = k e
e.g. s(“promotion”, “NN”) Σ μf Ff f= 1 |F|<br>
where y belongs to V X T ~ S: { s(y) : s(y) = k e
e.g. s(“promotion”, “NN”) Σ μf Ff f= 1 |F|<br>
86
Pythagorean theorem D (r || s) = D ( r || p*) + D (p* || s)
p * is an intersection of r and s<br>
p * is an intersection of r and s<br>
87
Conclusion: Intuition The intuition is:
Constrained distribution brings in likelihood
Exponential distribution is due to entropy
Hence, the duality.
We will continue in the next session.<br>
Constrained distribution brings in likelihood
Exponential distribution is due to entropy
Hence, the duality.
We will continue in the next session.<br>
88
Maximum Likelihood-Maximum Entropy Duality: Session 7 Pushpak Bhattacharyya
Scribed by Aditya Joshi
Presented in NLP-AI talk on 20th September, 2015<br>
Scribed by Aditya Joshi
Presented in NLP-AI talk on 20th September, 2015<br>
89
Recap: POS Tagging (1/2) Vocabulary, V : V1, V2, V3 ... V|V|
Tag set, T : T1, T2, T3 ..... T|T|
Features, F: F1, F2, F3 ... F|F| “Promotion”, “NN”, has ‘tion’ Vi Tj Fk Goal: Phenomenon Algorithm Model<br>
Tag set, T : T1, T2, T3 ..... T|T|
Features, F: F1, F2, F3 ... F|F| “Promotion”, “NN”, has ‘tion’ Vi Tj Fk Goal: Phenomenon Algorithm Model<br>
90
Recap: POS Tagging (2/2) P: P1 P2 P3 ..... P|V| X |T|
P(T|W) = P(T,W) / P(W)
Hence, P(“Promotion”,”NN”) is the kind of probabilities we wish to estimate
Observations: O: O1, O2, O3... Om {o} ε {V} X {T} .... In a tagged corpus<br>
P(T|W) = P(T,W) / P(W)
Hence, P(“Promotion”,”NN”) is the kind of probabilities we wish to estimate
Observations: O: O1, O2, O3... Om {o} ε {V} X {T} .... In a tagged corpus<br>
91
Constrained Distribution R: {r | Er (Fk) = Ep (Fk) }
Where,
P = empirical distribution ~ K = 1 .... |F| ~ Er(Fk) = Σ V(Fk). Pr (Fk) All values Fk Fk are such that V(Fk) = 0 or 1 Er(Fk) = Pr(Fk) = Σ Pr (Fk, x)
= Σ Pr(x) Pr (Fk|x) X ε {V} X {T} x x<br>
Where,
P = empirical distribution ~ K = 1 .... |F| ~ Er(Fk) = Σ V(Fk). Pr (Fk) All values Fk Fk are such that V(Fk) = 0 or 1 Er(Fk) = Pr(Fk) = Σ Pr (Fk, x)
= Σ Pr(x) Pr (Fk|x) X ε {V} X {T} x x<br>
92
Exponential distribution Uniform distribution, U is a member of the exponential family where μf = 0 S: { s(y) : s(y) = k e
Weights of features Σ μf Ff f= 1 |F|<br>
Weights of features Σ μf Ff f= 1 |F|<br>
93
Pythagorian theorem Let there be a p* that is a member of S ∩ R
The theorem states that for such a p*
D(s||r) = D(s||p*) + D(p*||r)
where s and r come from R<br>
The theorem states that for such a p*
D(s||r) = D(s||p*) + D(p*||r)
where s and r come from R<br>
94
Proof (1/3) LHS = D(s||r)
= Σ s(x). log (s(x) - Σ s(x). log (r(x))
= - H(s) – Es log (r(x)) x x<br>
= Σ s(x). log (s(x) - Σ s(x). log (r(x))
= - H(s) – Es log (r(x)) x x<br>
95
Proof (2/3) RHS = D(s||p*) + D(p*||r)
= Σ s(x) log s(x) - Σ s(x) log (p*(x))
+Σ p*(x) log (p*(x)) - Σ p*(x) log (r(x)) x x x x<br>
= Σ s(x) log s(x) - Σ s(x) log (p*(x))
+Σ p*(x) log (p*(x)) - Σ p*(x) log (r(x)) x x x x<br>
96
Proof (3/3) We will show later that term 2 and 3 cancel and the last term in LHS will be equal to remaining in RHS
This proof is also based on the fact that relative entropy >= 0. This proof will be shown later.<br>
This proof is also based on the fact that relative entropy >= 0. This proof will be shown later.<br>
97
Taking the ME path (1/1) With Pythagorian theorem in view, we wish to show that: p* = argmax (H(r))
Consider D(r||u) where u is uniform distrib.
D(r||u) = D(r||p*) + D(p*||u)
Since D(.) >= 0,
D(r||u) >= D(p*||u) r<br>
Consider D(r||u) where u is uniform distrib.
D(r||u) = D(r||p*) + D(p*||u)
Since D(.) >= 0,
D(r||u) >= D(p*||u) r<br>
98
Taking the ME path (2/2) D(r||u) = -H(r) - Σ r(x). log (u(x))
= - H(r) + log (|V|X |T|) x D(p*||u) = -H(p*) - Σ p*(x). log (u(x))
= - H(p*) + log (|V|X |T|) x Therefore, H(r) + log (|V|X|T|) >= -H(p*) + log (|V|X|T|)
- H(r) >= -H(p*)
H(r)<= H(p*) ---------------- (A)<br>
= - H(r) + log (|V|X |T|) x D(p*||u) = -H(p*) - Σ p*(x). log (u(x))
= - H(p*) + log (|V|X |T|) x Therefore, H(r) + log (|V|X|T|) >= -H(p*) + log (|V|X|T|)
- H(r) >= -H(p*)
H(r)<= H(p*) ---------------- (A)<br>
99
Taking the MLE Path (1/3) Consider D(p || s) = D(p || p*) + D(p* || s)
Where p is a constrained distribution ~ ~ ~ D(p || s) >= D(p || p*) ~ ~ -H(p) - Σ p(x). log (s(x)) >= - H(p) - Σ p(x). log (p*(x)) ~ ~ Σ p(x). log (s(x)) <= Σ p(x). log (p*(x)) ~ ~ ~ ~<br>
Where p is a constrained distribution ~ ~ ~ D(p || s) >= D(p || p*) ~ ~ -H(p) - Σ p(x). log (s(x)) >= - H(p) - Σ p(x). log (p*(x)) ~ ~ Σ p(x). log (s(x)) <= Σ p(x). log (p*(x)) ~ ~ ~ ~<br>
100
Taking the MLE path (2/3) Suppose p(x) = fx / |o| ~ Σ f(x). log (s(x)) <= Σ f(x). log (p*(x)) Σ log (s(x)f(x)) <= Σ log (p*(x)f(x)) LL(s) <= LL (p*) ------ (B)<br>
101
The Duality! By (A) and (B),
p* maximizes entropy ( From (A))
maximizes log-likelihood (From B)<br>
p* maximizes entropy ( From (A))
maximizes log-likelihood (From B)<br>
102
Pending critical subproofs p*(x) is unique
Proof of Pythagorian theorem
D(p||q) >= 0
U(x) is a constrained distribution
Is p(x) = fx/|o| ~<br>
Proof of Pythagorian theorem
D(p||q) >= 0
U(x) is a constrained distribution
Is p(x) = fx/|o| ~<br>