Proof XiiidlogpXiiidbytheweaklawoflargenumbers1 nlogpX1X2Xn1 n ID: 498118
Download Pdf The PPT/PDF document "AsymptoticequipartitionpropertytheoremII" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
AsymptoticequipartitionpropertytheoremII Proof. Xii.i.d=)logp(Xi)i.i.d,bytheweaklawoflargenumbers�1 nlogp(X1,X2,...,Xn)=�1 nåilogp(Xi)p�!�E(logp(X)=H(X). RaduTr^mbitas(UBB) AsymptoticEquipartitionProperty October20125/19 AsymptoticequipartitionpropertytheoremIII Denition3 ThetypicalsetA(n)#withrespecttop(x)isthesetofsequences(x1,x2,...,xn)2Xnwiththeproperty2�n(H(X)+#)p(x1,x2,...,xn)2�n(H(X)�#).(2) RaduTr^mbitas(UBB) AsymptoticEquipartitionProperty October20126/19 AsymptoticequipartitionpropertytheoremVI Proof-continuation. (3)1=åx2Xnp(x)åx2A(n)#p(x)åx2A(n)#2�n(H(X)+#)=2�n(H(X)+#)A(n)#HenceA(n)#2n(H(X)+#).(4)Forsucientlylargen,PA(n)#�1�#,sothat1�#PA(n)#åx2A(n)#2�n(H(X)�#)=2�n(H(X)�#)A(n)#whichyieldstoA(n)#(1�#)2n(H(X)�#). RaduTr^mbitas(UBB) AsymptoticEquipartitionProperty October20129/19 ConsequencesoftheAEP:DatacompressionI X1,X2,...,Xnp(x),i.i.d Wewishshortdescriptionsofsuchsequences WedividesequencesinXnintwosets:thetypicalsetA(n)#anditscomplement(Figure1) Allelementsineachsetareordered(e.g.lexicogracorder) WerepresenteachsequenceofA(n)#byitsindexofthesequenceintheset. Sincethereare2n(H+#)sequencesinA(n)#,theindexingrequiresn(H+#)+1bits Weprexeachsequenceby0:n(H+#)+2bitsrequired(Figure2) Similarly,forthecomplementofA(n)#weneednlogjXj+1;weprexeachsequenceby1 RaduTr^mbitas(UBB) AsymptoticEquipartitionProperty October201210/19 High-probabilitysetsandthetypicalsetsII Theorem7 LetX1,X2,...,Xnbei.i.d.p(x).Ford1 2andanyd0�0,ifPB(n)d�1�d,then1 nlogB(n)d�H�d0(5)fornsucientlylarge. Denition8 Thenotationan.=bnmeanslimn!¥1 nlogan bn=0. RaduTr^mbitas(UBB) AsymptoticEquipartitionProperty October201217/19 High-probabilitysetsandthetypicalsetsIIIThus,an.=bnimpliesanandbnareequaltotherstorderintheexponent.Wecanrestatetheabovetheorem:ifdn!0and#n!0,thenB(n)dn.=A(n)#n.=2nH. RaduTr^mbitas(UBB) AsymptoticEquipartitionProperty October201218/19 References ThomasM.Cover,JoyA.Thomas,ElementsofInformationTheory,2ndedition,Wiley,2006. DavidJ.C.MacKay,InformationTheory,Inference,andLearningAlgorithms,CambridgeUniversityPress,2003. RobertM.Gray,EntropyandInformationTheory,Springer,2009 RaduTr^mbitas(UBB) AsymptoticEquipartitionProperty October201219/19