Information Theory and coding Fourth stage By:
Description: Information Theory and coding Fourth stage By: MSC. Ridhab Sami Al-Mustaqbal University College Department of Computer Engineering Techniques Lecture 6 Mutual information for noisy channel Mutual information for noisy channel: Consider the
Related Topics
Download Presentation
"Information Theory and coding Fourth stage By:" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Presentation Transcript
slide1. Information Theory and coding
Β Fourth stage
By:
MSC. Ridhab Sami Al-Mustaqbal University College
Department of Computer Engineering Techniques<br>
slide2. Lecture 6 Mutual information for noisy channel<br>
slide3. Mutual information for noisy channel: Consider the set of symbols π₯1, π₯2,β¦.,π₯π, the transmitter ππ₯ my produce. The receiver π π₯ may receive π¦1, π¦2 β¦β¦β¦.π¦π. Theoretically, if the noise and jamming is neglected, then the set X=set Y.
The amount of information that π¦π provides about π₯π is called the mutual information between π₯π and π¦π.<br>
slide4. Properties of π°(ππ , ππ): It is symmetric, πΌ(π₯π , π¦π) = πΌ(π¦π , π₯π).
πΌ(π₯π , π¦π) > 0 if aposteriori probability > a priori probability, π¦π provides +ve information about π₯π .
πΌ(π₯π , π¦π) = 0 if aposteriori probability = a priori probability, which is the case of statistical independence when π¦π provides no information about π₯π .
πΌ(π₯π , π¦π) < 0 if aposteriori probability < a priori probability, π¦π provides -ve information about π₯π , or π¦π adds ambiguity.<br>
slide5. Joint entropy: In information theory, joint entropy is a measure of the uncertainty associated with a set of variables. P(x , y) joint probability Entropy Joint Entropy<br>
slide6. 2. Conditional entropy: In information theory, the conditional entropy quantifies the amount of information needed to describe the outcome of a random variable Y given that the value of another random variable X is known. P(x | y) conditional probability Conditional Entropy Entropy<br>
slide7. 3. Marginal Entropies: Marginal entropies is a term usually used to denote both source entropy H(X) defined as before and the receiver entropy H(Y) given by: Receiver Entropy Source Entropy<br>
slide8. 4. Relationship between joint, conditional and transinformation: Noise entropy: π»(π β£ π) = π»(π,π) β π»(π)
Loss entropy: π»(π β£ π) = π»(π,π) β π»(π) Also we have transinformation (average mutual information): πΌ(π,π) = π»(π) β π»(π β£ π)
πΌ(π,π) = π»(π) β π»(π β£ π)<br>
slide9. Example: The joint probability of a system is given by: Find:
Marginal entropies.
Joint entropy.
Conditional entropies.
The transinformation.<br>
slide10. 1- Marginal entropies: Solution: x1 x2 x3 y1 y2
P(x)=[0.75 0.125 0.125], P(y)= [0.5625 0.4375]<br>
slide11. 2- Joint entropy:<br>
slide12. 3. Conditional entropies : 4. The transinformation :<br>
Β Fourth stage
By:
MSC. Ridhab Sami Al-Mustaqbal University College
Department of Computer Engineering Techniques<br>
slide2. Lecture 6 Mutual information for noisy channel<br>
slide3. Mutual information for noisy channel: Consider the set of symbols π₯1, π₯2,β¦.,π₯π, the transmitter ππ₯ my produce. The receiver π π₯ may receive π¦1, π¦2 β¦β¦β¦.π¦π. Theoretically, if the noise and jamming is neglected, then the set X=set Y.
The amount of information that π¦π provides about π₯π is called the mutual information between π₯π and π¦π.<br>
slide4. Properties of π°(ππ , ππ): It is symmetric, πΌ(π₯π , π¦π) = πΌ(π¦π , π₯π).
πΌ(π₯π , π¦π) > 0 if aposteriori probability > a priori probability, π¦π provides +ve information about π₯π .
πΌ(π₯π , π¦π) = 0 if aposteriori probability = a priori probability, which is the case of statistical independence when π¦π provides no information about π₯π .
πΌ(π₯π , π¦π) < 0 if aposteriori probability < a priori probability, π¦π provides -ve information about π₯π , or π¦π adds ambiguity.<br>
slide5. Joint entropy: In information theory, joint entropy is a measure of the uncertainty associated with a set of variables. P(x , y) joint probability Entropy Joint Entropy<br>
slide6. 2. Conditional entropy: In information theory, the conditional entropy quantifies the amount of information needed to describe the outcome of a random variable Y given that the value of another random variable X is known. P(x | y) conditional probability Conditional Entropy Entropy<br>
slide7. 3. Marginal Entropies: Marginal entropies is a term usually used to denote both source entropy H(X) defined as before and the receiver entropy H(Y) given by: Receiver Entropy Source Entropy<br>
slide8. 4. Relationship between joint, conditional and transinformation: Noise entropy: π»(π β£ π) = π»(π,π) β π»(π)
Loss entropy: π»(π β£ π) = π»(π,π) β π»(π) Also we have transinformation (average mutual information): πΌ(π,π) = π»(π) β π»(π β£ π)
πΌ(π,π) = π»(π) β π»(π β£ π)<br>
slide9. Example: The joint probability of a system is given by: Find:
Marginal entropies.
Joint entropy.
Conditional entropies.
The transinformation.<br>
slide10. 1- Marginal entropies: Solution: x1 x2 x3 y1 y2
P(x)=[0.75 0.125 0.125], P(y)= [0.5625 0.4375]<br>
slide11. 2- Joint entropy:<br>
slide12. 3. Conditional entropies : 4. The transinformation :<br>