Maximum Likelihood-Maximum Entropy Duality PowerPoint Presentation, PPT

Maximum Likelihood-Maximum Entropy Duality PowerPoint Presentation, PPT - DocSlides

: Session 1. Pushpak Bhattacharyya. Scribed by . Aditya. Joshi. Presented in NLP-AI talk on 14. th. January, 2015. Phenomenon/Event could be a linguistic process such as POS tagging or sentiment prediction.. ID: 396967

By tatiana-dople
Watch All docs

Likes : 64
Views : 103

Download this presentation

Click below link (As may be) to get this presentation.

Maximum Likelihood-Maximum Entropy Duality

Download Note - The PPT/PDF document "Maximum Likelihood-Maximum Entropy Duali..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Presentations text content in Maximum Likelihood-Maximum Entropy Duality

Maximum Likelihood-Maximum Entropy Duality: Session 1
Pushpak Bhattacharyya
Scribed by
Aditya
Joshi
Presented in NLP-AI talk on 14
th
January, 2015
Phenomenon/Event could be a linguistic process such as POS tagging or sentiment prediction.
Model uses data in order to “predict” future observations w.r.t. a phenomenon
Data/Observation
Phenomenon/Event
Model
Notations
X : x
1
, x
2
, x
3
....
x
m
(m observations)
A: Random variable with n possible outcomes such as a
1
, a
2
, a
3
... a
n
e.g. One coin throw : a
1
= 0, a
2
=1
One dice throw: a
1
=1, a
2
=2, a
3
=3, a
4
=4, a
5
=5, a
6
=6
Goal
Goal: Estimate P(ai) = Pi
Two paths
ML
ME
Are they equivalent?
Calculating probability from data
Suppose in X: x
1
, x
2
, x
3
....
x
m
(m observations),
a
i
occurs f(
a
i
) =
f
i
times
e.g. Dice: If outcomes are 1 1 2 3 1 5 3 4 2 1
F(1) = 4, f(2) = 2, f(3) = 2, f(4) = 1, f(5) = 1, f(6)=0 and m = 10
Hence, P
1
= 4/10, P
2
=2/10, P
3
=2/10, P
4
=1/10, P
5
=1/10, P
6
=1/10
In general, the task is...
Task:
Get
θ
: the probability vector <P(
θ
i
)> from X
MLE
MLE: θ* = argmax Pr (X; θ)With i.i.d. (identical independence) assumption, θ* = argmax π Pr (X; θ)Where, θ : <P1, P2, ... Pn>
θ
θ
i=1
m
P(a1)
P(a2)
P(an)
What is known about: θ : <P1, P2, ... Pn>
Σ Pi = 1Pi >= 0 for all iIntroducing Entropy: H(θ)= - ΣPi ln Pi
i=1
n
i=1
n
Entropy of distribution <P1, P2, ... Pn>
Some intuition
Example with diceOutcomes = 1,2,3,4,5,6P(1) + P(2)+P(3)+...P(6) = 1Entropy(Dice) = H(θ)= - ΣP(i) ln P(i)Now, there is a principle called Laplace’s Principle of Unbiased(?) reasoning
i=1
6
0
The best estimate for the dice
P(1) = P(2) = ... P(6) = 1/6
We will now prove it assuming:
NO KNOWLEDGE about the dice except that it has six outcomes, each with probability >= 0 and
Σ
P
i
= 1
1
What does “best” mean?
“BEST” means most consistent with the situation.
“Best” means that these P
i
values should be such that they maximize the entropy.
2
Optimization formulation
Max. - ΣP(i) log P(i)Subject to:
i=1
6
Σ Pi = 1Pi >= 0 for i = 1 to 6
i=1
6
3
Solving the optimization (1/2)
Using Lagrangian multipliers, the optimization can be written as:Q = - ΣP(i) log P(i) – λ ( ΣP(i) – 1) - Σ βi P(i)
i=1
6
i=1
6
i=1
6
For now, let us ignore the last term. We will come to it later.
4
Solving the optimization (2/2)
Differentiating Q w.r.t. P(i), we getδQ/δP(i) = - log (P(i) – 1 – λEquating to zero,log P(i) + 1 + λ = 0log P(i) = -(1+ λ)P(i) = e -(1+ λ)
This means that to maximize entropy, every P(
i
) must be equal.
5
This shows that P(1) = P(2) = ... P(6)
But,
P(1) + P(2) + .. + P(6) = 1
Therefore P(1) = P(2) = ... P(6) = 1/6
6
Introducing data in the notion of entropy
Now, we introduce data:X : x1, x2, x3.... xm (m observations)A: a1, a2, a3... an (n outcomes)
e.g. For a coin, In absence of data: P(H) = P(T) = 1/2 ... (As shown in the previous proof)However, if data X is observed as follows: Obs-1: H H T H T T H H H (m=10) (n=2) P(H) = 6/10, P(T) = 4/10Obs-2: T T H T H T H T T T (m=10) (n=2) P(H) = 3/10, P(T) = 7/10
Which of these is a valid estimate?
7
Change in entropy
Entropy reduces as data is observed!
Emax
(uniform distribution)
E2 : P(H) = 3/10
E1 : P(H) = 6/10
8
Start of Duality
Maximizing entropy in this situation is same as minimizing the `entropy reduction’ distance. i.e. Minimizing “relative entropy”
Emax
(Maximum entropy:uniform distribution)
Edata
(Entropy whenObservations are made)
Entropy reduction
9
Concluding remarks
Thus, in this discussion of ML-ME duality, we will show that:
MLE minimizes relative entropy distance from uniform distribution.
Question:
The entropy corresponds to probability vectors. The distance can be measured by squared distances.
WHY
relative entropy?
0
Maximum Likelihood-Maximum Entropy Duality: Session 2
Pushpak Bhattacharyya
Scribed by
Aditya
Joshi
Presented in NLP-AI talk on 28
th
January
, 2015
1
Recap (1/2)
Whatever we can show with MLE, (in most cases), we can through ME as well
Laplace’s unbiased reasoning principle
Make only the most limited assumptions.
We assume data:
X: x
1
, x
2
....
X
m
m: number of observations
Every xi is the outcome of the values of a random variable
2
Recap (2/2)
E.g. x
i
= {1, 2, ...6} for dice.
If no data then by Laplace’s principle, uniform distribution is the best estimate of the probability of each outcome.
3
NLP Perspective
Probability of mapping of words between parallel sentences.
E.g. The mapping between “blue” and “
neela
”, and “blue” and “
haraa
” (wrong translation)
So how can you maintain probabilities of correct mappings based on the corpus?
4
Starting off
This week, we assume that we have data
X: x
1
....
x
m
Outcome of
r.v
. Is : a
1
, a
2
, a
3
... a
n
in the data such that each
a
j
occurs
f
i
times.
In absence of any data,
Pj
= 1/n
When we have data?
5
In presence of data
When we have the data, Pj = ?Answer: Pj = f j / m
WHY?
Can we arrive at this value through (a) MLE, or (b) ME ??
6
Taking the MLE route
MLE: We maximize data likelihoodP(X; θ) where θ = <P1, P2, ... Pn>Under i.i.d., P(X; θ) = π P(xi; θ)= π Pjfj
i=1
m
i=1
m
e.g. 1 1 2 3 4 5 4 P1 P1 P2 P3 P4 P5 P4... P12 P21 P31 P42 P51
Σ fj = m
j=1
n
Where,
7
Maximization
MLE demands maximize P(X; θ)Subject to :
Σ Pj = 1Pj>= 0 for all i
j=1
n
Maximize
ln
(P(X;
θ))
Σ Pj = 1Pj >= 0 for all i
i=1
n
8
Evaluating the parameter Pj (1/2)
Q =Σ fj ln Pj – λ( Σ Pj – 1)
j=1
n
j=1
n
Taking derivative w.r.t. Pj
δQ/δPj = fj / Pj - λ
Equating to zero,fj / Pj – λ = 0 Pj = fj / λ
(1)
Taking derivative w.r.t. λ
δQ/δPj = Σ Pj – 1
Equating to zero,Σ Pj = 1
(2)
9
Evaluating the parameter Pj (2/2)
From (1) and (2),
Σ fj / λ = 1
j=1
n
Σ fj = λ
j=1
n
λ = m
Pj = fj / m
Proved!
0
Summary
No observation: Pj = 1/n Observation: Pj = fj/m ; Σ fj = m
Entropy
(as seen lasttime)
MLE
(as shownToday)
ME??
1
Does entropy change
Before we move on to discussion on ME in case of observed data, we first see how entropy gets affected in presence of data.
In context of cases given in the previous slide, we wish to see:
Is it true that Entropy (Case 2) <= Entropy (Case 1)??
Let’s verify.
2
Situation 1: No observed data
Entropy(Case1): - ΣP(j) log P(j) = - Σ(1/n) log (1/n)
j=1
n
j=1
n
= log (1/n) = log (n)
3
Situation 2: Observed Data (1/2)
Entropy(Case2): - ΣP(j) log P(j) = - Σ(fj/m) log (fj/m)
j=1
n
j=1
n
= -1/m Σfj log (fj/m)
j=1
n
= -1/m Σfj [ log (fj) – log(m) ]
j=1
n
4
Situation 2: Observed Data (2/2)
Entropy(Case2): - ΣP(j) log P(j)
j=1
n
= -1/m[ Σfj log (fj) –Σfj log(m) ]
j=1
n
j=1
n
= -1/m[ Σfj log (fj) –m log(m) ]
j=1
n
= [ m log(m) - Σfj log (fj) ] / m
j=1
n
5
Comparing the entropies
Now, we must show:
m log(m) – m log(n) <= Σfj log (fj)
j=1
n
i.e. m log(m/n) <= Σfj log (fj)
j=1
n
.... We will show this in next class.
6
Change in entropy
Entropy reduces as data is observed!
No data, 1/n, E1
Data,
fj
/m, E2
7
Concluding remark
If you can show that MLE is same as maximum entropy, then MLE-ME duality is shown.
8
Maximum Likelihood-Maximum Entropy Duality: Session 3
Pushpak Bhattacharyya
Scribed by
Aditya
Joshi
Presented in NLP-AI talk on 11
th
February, 2015
9
-Σ fi log Pi s.t. Σ Pi = 1 & Σ fi = N
Recap
X1, X2, X3.... XN : DataA1, A2, A3... AM : OutcomesGoal: Estimate P(Ai) = Pi
In absence of any other information (not even data), Pi = 1/MThis was obtained using MEMax.
-Σ Pi log Pis.t. Σ Pj = 1
i=1
m
i=1
m
When data is observed,
Pi = fi/N where fi = freq(Ai)This was obtained using MEMax.
i=1
m
i=1
m
i=1
m
0
Change in entropy
Entropy changes as data is observed.Today, we show that the entropy is “reduced”. i.e. Eu > Ed
No data, 1/m,
Eu
Data,
fj/n, Ed
1
Today’s goal
Today,
we will show that
the entropy is “reduced”.
i.e.
E
u
> E
d
Two proofs
2
Proof 1 (1/..)
Suppose without loss of generality,P1 -> P + ε P2 -> P - ε
Thus, in case of uniform distribution, P1 = P2 =... P = 1/M, and ε = 0
-Σ Pi log Pi
i=1
m
Eu =
Ed =
-(P+ε) log(P+ε) -(P-ε) log(P-ε) -Σ Pi log Pi
i=3
m
3
Proof 1 (2/...)
Eu
– Ed = -P log P + (P+
ε
) log(P+
ε
)
-P log P + (P-
ε
) log(P-
ε
)
= P log ((P+
ε
)/P) +
ε
(log P + log (1+
ε
/P))
+ P log ((P-
ε
)/P) -
ε
(log P + log (1-
ε
/P))
= P [log (1+
ε
/P)+ log (1-
ε
/P)]
+
ε
[log (1+
ε
/P) - log (1-
ε
/P)]
4
Proof 1 (3/...)
Eu – Ed = P [log (1+ε/P)+ log (1-ε/P)] + ε [log (1+ε/P) - log (1-ε/P)] = P [log (1 – ε2/P2)] + ε [log ((1+ε/P)/(1-ε/P))] = P [log (1 – y2)] + ε [log ((1+y)/(1-y))]
Where, y=
ε
/P
5
Proof 1 (4/...)
Eu-Ed = P [log (1 – y2)] + ε [log ((1+y)/(1-y))]= P [ -y2 –y4/2 –y6/3 – y8/4...] + ε[2y + 2y3/3+2y5/5+2y7/7...]= -Py2 [1+ y2/2+ y4/3+ y6/4...] + Py2 [2 + 2y2/3+ 2y4/5+ 2y6/7...]
log (1+x) = x – x2/2 + x3/3 – x4/4+ .. log ((1+x)/(1-x)) = 2x +2x3/3+2x5/5+2x7/7+..
...substitute ε = Py
When we compare the above statement term by term, (1..2), (1/2.. 2/3), (1/3...2/5).. Etc., we see that the above value is > 0. i.e.
Eu
– Ed > 0.
i.e.
Eu
> Ed
6
Proof 2 (1/...)
Identity
: for
x,y
> 0
y –
ylog
y <= x –
ylog
x
Proof
: log t <= log t -1
Put t = (x/y) . log (x/y) <= (x/y) – 1
i.e. log x - log y <= (x-y)/y
i.e. y log x – y log y <= x – y
i.e.
y – y log y <= x – y log x
7
Proof 2 (2/...)
p1, p2.... pm = 1/M .............. Uniformq1, q2... qm ............... Perturbed distributionqi – qi log qi <= pi - qi log pi take sum over i = 1....m,
We put x = p, y= q in y – ylog y <= x – ylog x
Σ qi – Σqi log qi <= Σ pi - Σ qi log pi
But, Σ qi = 1 & Σ pi = 1
1– Σqi log qi <= 1 + log M Σ qi
1 + Ed <= 1 + log M
1 + Ed <= 1 + Eu
Eu >= Ed
8
Conclusion
We showed today that there is a “reduction” in entropy when data is observed.
We showed two proofs: the first is more intuitive but long; the second is simpler.
9
Maximum Likelihood-Maximum Entropy Duality: Session 4
Pushpak Bhattacharyya
Scribed by
Aditya
Joshi
Presented in NLP-AI talk on 25
th
March, 2015
0
A uniform distribution and any pd P
Uniform, 1/n
P
Let us now compute relative entropy / KLD between
these two vectors.
1
Relative entropy between two non-negative vectors
P: <p1 p2 ... pn>, Q: <q1 q2 ... qn>D(P||Q) = Σ pi log (pi /qi) - Σ pi + Σ qi
i=1
n
i=1
n
i=1
n
KL Divergence or relative entropy between the two vectors is defined as:
& D(P||Q) >= 0
& D(P||Q) = 0 occurs, when p
i
=
q
i
2
Relative entropy between uniform distribution and any p.d. p
D(P||1/n) = Σ pi log (pi /(1/n)) - Σ pi + Σ (1/n)= Σ pi log pi + Σ pi log n - 1 + 1= - E(P) + log n
i=1
n
i=1
n
i=1
n
D(P || 1/n ) = - E(P) + log n
3
Relation between entropy of a pd and its relative entropy with uniform
This shows that, if we maximize entropy P, we minimize relative entropy with respect to uniform distribution.Conclusion: The probability distribution found by maximizing entropy is the distribution with least KLD (relative entropy) from uniform distribution.
D(P || 1/n ) = - E(P) + log n
argmin D(P || 1/n ) = argmax E(P)
P
P
4
Digression
What constitutes a valid proof?
Statement
S
, Path
P
The Euclidean distance path may not lead us to proving statement
S.
How statement S can be proven may depend on path P and the choice of axioms.
5
A uniform distribution and any pd with data
Uniform, 1/n
f
1
, f
2, ...fn : F
Now, when we have data (n-
ary
) given by frequencies (f
1
, f
2
... f
n
) as shown above.
Let us now compute relative entropy / KLD between
these two vectors.
6
Relative entropy bet. uniform and pd p
U: <1/n 1/n .... 1/n> PD: <p1 p2 ... pn>D(RF||1/n) = Σ(fi/m) log(fi/m)/(1/n)) = - (1/m) Σfi log fi = - (1/m) [ log π Pjfj ]
i=1
n
KL Divergence or relative entropy
between the two vectors is defined as:
7
Relation between RF and uniform
We find that relative entropy
D(RF||1/n) = log n + 1/m [log-likelihood]
8
Frequency distribution F and a pd P
Suppose F is the frequency of outcomes:
F : f
1
, f
2
.... f
n
m : total no. of observations
Let P be “some” probability distribution
Let us now find the KLD between F and P
9
Relative entropy between frequency distribution and any p.d. p
D(F||P) = Σ fi log (fi / pi ) - Σ fi + Σ pi = Σ fi log fi - Σ log pi fi - m + 1= constant – log-likelihod - constant
i=1
n
i=1
n
i=1
n
D(F||P) = constant – log-likelihood
0
Relation between entropy of P and its relative entropy with freq. dist.
This shows that, if we maximize log-likelihood of P, we minimize relative entropy with respect to frequency distributionConclusion: The probability distribution found by maximizing log-likelihood is the distribution with least KLD (relative entropy) from frequency distribution.
D(F||P) = constant – log-likelihood
argmin D(F||P) = argmax LL(P)
P
P
1
Concluding remarks
Observations today:The P found by maximizing entropy is the one with least RE w.r.t. uniform distributionThe P found by maximizing likelihood is the one with least RE w.r.t. frequency distribution
Work to be done:
Bring in
fs
into (1) above
Bring in 1/n into (2) above
2
Maximum Likelihood-Maximum Entropy Duality: Session 5
Pushpak Bhattacharyya
Scribed by
Aditya
Joshi
Presented in NLP-AI talk on 17
th
June, 2015
3
Recap
Some notations have been corrected in this presentation
Setting:
X: X
1
, X
2
, ....
X
m
(data)
A: A
1
, A
2
,.... A
n
(outcomes)
P: P
1
, P
2
, ....
P
n
(Parameters to be estimated)
For example:
X: 2 1 3 5 6 3 4
A: 1 2 3 4 5 6
P: P
1
, P
2
, .... P
6
4
Recap: Goal
To estimate parameters P, there are two paths:Maximum Likelihood: Pi = fi/m ... As shownMaximum entropy: Pi = 1/n ; data not consideredWe also showed that: Entropy is maximum for uniform distribution
P
i = 1/n for all i
H
u = Entropy for uniformDistr. = log n
H = - ΣPi log PiHu= - Σ(1/n) log (1/n) = log (1/n) = log n
i=1
n
j=1
n
= log (1/n) = log (n)
i=1
n
5
Recap: Reduction in entropy
When we have data,We have shown in two ways that: Hu > Hd. i.e. If we perturb a distribution 1/n, 1/n .... To 1/n+k, 1/n-k, 1/n ..... , the entropy decreases
H
u = Entropy for uniformDistr.
H
d
= Entropy in case of data
6
Recap: Relative Entropy
P: p1, p2, p3, ... pnQ: q1, q2, q3,... qnD(p||q) =
Σ Pi = 1
i=1
n
Σ qi = 1
i=1
n
Σ pi log (pi / qi )
i=1
n
7
Recap: Relative entropy
For q: uniform distribution,D(p||q) = = = = - Hp + log n = Hu - Hp
Σ pi log (pi / qi )
i=1
n
Σ pi log (pi / (1/n) )
i=1
n
Σ pi log pi - log (1/n)
i=1
n
PIVOT:
Relative entropy and absolute entropy difference are same for uniform distribution.
8
Approach to prove duality
Theorem: MLE-ME converge when:
Distance measure = Relative entropy
Distributions belong to exponential family
9
Data v/s feature matrix
X1 X2 Xi Xm
F1 F2 Fi Fm
F X
promotion
Word ends with “
tion
”
Outcome: POS tags
0
A note on Outcomes
A: A
1
, A
2
..... A
n
N is tied to the classification task.
For POS tagging: (Penn
Tagset
) A: A
1,
A
2
..... A
39
1
Exponential & Constrained distributions
Pi = ci π λj fji = ci e λj ^(fji)
j=1
k
P = { P s.t. E P F = E P F }
Expected value
Distribution
Obtained from data
Distribution
intended
Expected value
s.t
. Feature
Value is F under distribution P
2
Equivalence between constrained and exponential
Now, let:P: Constrained distribution Q: exponential distributionAnd P* = P intersection QBy ME, we will show that: H(P*) >= H(q) for all qBy MLE, we will show that: LL(P*) >= LL(p) for all p
This is done using concepts like:
(a) Pythagorean distance for relative entropy,
(b) LL in terms of P
3
Maximum Likelihood-Maximum Entropy Duality: Session 6
Pushpak Bhattacharyya
Scribed by
Aditya
Joshi
Presented in NLP-AI talk on 2
nd
September, 2015
4
Recap (1/3)
O: o1, o2, o3 .... on observationsA: a1, a2, a3 .... am outcomesP: p1, p2, p3 .... pm prob. distributionF: f1, f2, f3 .... fm Frequency
Pi = fi/n .... empirical distribution acc to MLE
~
P
i
= 1/m .... distribution acc to ME,
assuming uniform
5
Recap (2/3)
Entropy:
H
u, entropy with uniform distrib = log m
H
d ... when data
H
d
<=
H
u
6
Recap (3/3)
P: <p1 p2 ... pn>, Q: <q1 q2 ... qn>D(P||Q) = Σ pi log (pi /qi) - Σ pi + Σ qi
i=1
n
i=1
n
i=1
n
KL Divergence or relative entropy between the two vectors is defined as:
D (P || Uniform) =
H
u
- H
p
7
A small digression
Why is entropy in log form?
e.g. Suppose two dice are thrown.
Sample space: 36
<x
1
, y
1
>, <x
2
, y
2
>, <x
3
, y
3
> .... <x
36
, y
36
>
If I say x
i
+
y
i
= 12 v/s x
i
+
y
i
= 7, which message has more information?
8
Which message is more informative?
12 : <6, 6>Probability of each outcome = 1Uncertainty less7: <1,6> <2,5>, <3,4>, <4,3>, <5,2>, <6,1>Probability of each outcome = 1/6Uncertainty more
0
1
P
Uncertainty
approx.
9
Additional certainty or information
If there are two independent variables, then intuitively:
I(
pq
) = I(p) + I(q)
The log function seems to satisfy the above requirement
e.g. Uncertainty of POS for play >= played
Less certainty --
 More entropy
0
Familiar example: POS tagging
Vocabulary, V : V
1
, V
2
, V
3
... V
|V|
Tag set, T : T
1
, T
2
, T
3
..... T
|T|
Call |V|X|T| = A
We wish to estimate
P: P
1
P
2
P
3
..... P
A
where
Pi = P (V
v
,
T
t
) e.g. P(“play”, “NN”)
1
Dataset and features
X: (w1,t1), (w2,t2), (w3,t3), .... (wn, tn)We introduce features, F: F1, F2, F3 ... F|F|example: Ff = 1 if Vv ends with “-tion” = 0 otherwise
Binary
2
Binding features with V X T
example: <promotion, N>Features = (1) ends with –tion, (2) has 3 syllables, etc.F is a matrix with |F| rows and |V| X |T| columnsP : P1, P2, P3 .... P|V||T|
~
3
Expected value of a feature
F VXT
<v
1
,t
1
> <v
1
,t2> ..... <vi, tj> ......
F
1
F
2
F
3
.
.
.
.
F
|F|
4
Entropy
For q: uniform distribution,Ep (Ffij) = For empirical distributionEp (Ffij) = similarly.
Σ Pr (Ff) Value (Ff)
i=1
|V| |T|
i
i
5
Introducing two distributions
Constrained distribution: (w.r.t training data)
Exponential distribution:
R : { r(y) : Er(F) = EP (F) }where y belongs to V X T
~
S: { s(y) : s(y) = k ee.g. s(“promotion”, “NN”)
Σ μf Ff
f= 1
|F|
6
Pythagorean theorem
D (r || s) = D ( r || p*) + D (p* || s)
p * is an intersection of r and s
7
Conclusion: Intuition
The intuition is:
Constrained distribution brings in likelihood
Exponential distribution is due to entropy
Hence, the duality.
We will continue in the next session.
8
Maximum Likelihood-Maximum Entropy Duality: Session 7
Pushpak Bhattacharyya
Scribed by
Aditya
Joshi
Presented in NLP-AI talk on
20
th
September, 2015
9
Recap: POS Tagging (1/2)
Vocabulary, V : V1, V2, V3 ... V|V|Tag set, T : T1, T2, T3 ..... T|T|Features, F: F1, F2, F3 ... F|F|
“Promotion”, “NN”, has ‘tion’
Vi Tj Fk
Goal:
Phenomenon
Algorithm
Model
0
Recap: POS Tagging (2/2)
P: P1 P2 P3 ..... P|V| X |T|P(T|W) = P(T,W) / P(W)Hence, P(“Promotion”,”NN”) is the kind of probabilities we wish to estimateObservations: O: O1, O2, O3... Om
{o}
ε
{V} X {T} .... In a tagged corpus
1
Constrained Distribution
R: {r | Er (Fk) = Ep (Fk) }Where,P = empirical distribution
~
K = 1 .... |F|
~
Er(Fk) = Σ V(Fk). Pr (Fk)
All values Fk
Fk are such that V(Fk) = 0 or 1
Er(Fk) = Pr(Fk) = Σ Pr (Fk, x) = Σ Pr(x) Pr (Fk|x)
X ε {V} X {T}
x
x
2
Exponential distribution
Uniform distribution, U is a member of the exponential family where μf = 0
S: { s(y) : s(y) = k eWeights of features
Σ μf Ff
f= 1
|F|
3
Pythagorian theorem
Let there be a p* that is a member of S ∩ R
The theorem states that for such a p*
D(s||r) = D(s||p*) + D(p*||r)
where s and r come from R
4
Proof (1/3)
LHS = D(s||r) = Σ s(x). log (s(x) - Σ s(x). log (r(x)) = - H(s) – Es log (r(x))
x
x
5
Proof (2/3)
RHS = D(s||p*) + D(p*||r) = Σ s(x) log s(x) - Σ s(x) log (p*(x)) +Σ p*(x) log (p*(x)) - Σ p*(x) log (r(x))
x
x
x
x
6
Proof (3/3)
We will show later that term 2 and 3 cancel and the last term in LHS will be equal to remaining in RHS
This proof is also based on the fact that relative entropy >= 0. This proof will be shown later.
7
Taking the ME path (1/1)
With Pythagorian theorem in view, we wish to show that: p* = argmax (H(r))Consider D(r||u) where u is uniform distrib.D(r||u) = D(r||p*) + D(p*||u)Since D(.) >= 0,D(r||u) >= D(p*||u)
r
8
Taking the ME path (2/2)
D(r||u) = -H(r) - Σ r(x). log (u(x)) = - H(r) + log (|V|X |T|)
x
D(p*||u) = -H(p*) - Σ p*(x). log (u(x)) = - H(p*) + log (|V|X |T|)
x
Therefore,
H(r) + log (|V|X|T|) >= -H(p*) + log (|V|X|T|)
- H(r) >= -H(p*)
H(r)<= H(p*) ---------------- (A)
9
Taking the MLE Path (1/3)
Consider D(p || s) = D(p || p*) + D(p* || s)Where p is a constrained distribution
~
~
~
D(p || s) >= D(p || p*)
~
~
-H(p) - Σ p(x). log (s(x)) >= - H(p) - Σ p(x). log (p*(x))
~
~
Σ p(x). log (s(x)) <= Σ p(x). log (p*(x))
~
~
~
~
00
Taking the MLE path (2/3)
Suppose
p(x) = fx / |o|
~
Σ f(x). log (s(x)) <= Σ f(x). log (p*(x))
Σ log (s(x)f(x)) <= Σ log (p*(x)f(x))
LL(s) <= LL (p*) ------ (B)
01
The Duality!
By (A) and (B),
p* maximizes entropy ( From (A))
maximizes log-likelihood (From B)
02
Pending critical subproofs
p*(x) is uniqueProof of Pythagorian theoremD(p||q) >= 0U(x) is a constrained distributionIs p(x) = fx/|o|
~
03
04
05
06