IEEETRANSACTIONSONINFORMATIONTHEORY,VOL.XX,NO.Y,MONTH200X3II.Definitio - PDF document

IEEETRANSACTIONSONINFORMATIONTHEORY,VOL.XX,NO.Y,MONTH200X3II.Definitions,notationandmainresultsA.NotationThecardinalityofany“nitesetisdenotedby.Let,...,r(withthediscretetopology)and:=betheproductspace.indicatesthemapping,...,).Note[w]denotesthespaceofBorelprobabilitymeasureson.Weusetodenotetheprobabilitymeasureonwhichistheproductoftheequiprobabledistributiononeachfactor.Given,therestrictionofisdenoted,sothatfor loglog log(5)istheshiftoperator,(,foreach.Theactionofonthefunctionisgivenby):=denotesshift-invariantprobabilitymeasures.Forwede“netheshiftspacebyyn1)�0},:=nXŠ1n(n).(6)Thusifandonlyiff1,...,n]n1)�0forall:=supdenotesthe-algebrageneratedby.Wewritetomeanthefunctionmeasurable.localifthereexistssothat.Weusetheweaktopologyon(see[2]).Thereexistsasequenceoflocalfunctionswhichdeterminesthistopology.Infactonecouldtaketobethesetofindicatorfunctionsofeachwordinforeach.Theintegralofwithrespecttoisdenotedbyf,actsonmeasuresbyf,SSf,Weuse)todenotestheempiricalmeasure,):= (7)wheredenotesthemeasureconcentratedonthepoint).Thisisconvenientnotation,butitsconcealstherealsigni“cance.Measuresconcentratedatsinglepointsofthespaceareunimportant.However,thegiveseasyaccesstothestatisticsof).For):=1if0otherwise.givesthefractionoftimesthecharacterappearsinthe“rstentriesof.Byusingfunctionsofwhichdependonlyon,wecanusetoobtainpairprobabilitydistributions;functionswhichdependonyielddistributionson.Forthesetobeusefulweneedtobemuchlargerthan.Acomplicationisthefactthattheprobabilitydistributionsonresultingfrom)dependon,...,.Intheanalysisbelowwehavetoprovethat,inthelimitsweconsider,thedependenceof)onfor�knisnegligible.Shannonentropy):=lim )(8)):==n1)loggn1).(9)Wehave)=log,sincespeciÞcI-divergencewithrespecttoisgivenby):=lim n1)log n1),(10)whenthelimitexists.Forall)existsandequals)+logB.Mainresultsbeagivenstationaryprobabilitymeasure.istheshiftspacede“nedby(6).denotesthesetofBorelprobabilitymeasureson,andtheshift-invariantprobabilitymeasureson.Ourmainhypothesesareformulatedforconvenienceasfollows.H1Foranyneighbourhood,andforany0,thereexistsanergodicsuchthat IEEETRANSACTIONSONINFORMATIONTHEORY,VOL.XX,NO.Y,MONTH200X51and:=2.Letwbethewordconsistingof0sfollowedby1s.If)=w,thenandonlyif=1,sown,Bn,B.However,lemmaIV.1showsthat,if,whereisclosedandopen,thenforsu
cientlylargen,Fn,GTheoremII.2:Foreachclosedsubsetlimsup logn,F(18)Ifsatis“eshypothesisH1,thenforeachopensubsetliminf logn,G(19)TheoremII.1isformulatedforthefullshiftspace.FortheslightlystrongertheoremII.2,theupperboundwillbeshowntofollowfromthatfor;thelowerbounddoesnotobtainingeneral,butisvalidunderH1.ThenextresultsrelatetothelargedeviationprinciplefortheempiricalmeasurerelativetoPropositionII.1:satisfyH2.Thenforeach):=lim n1)log n1)(20)existsandequalsPropositionII.2:satisfyH2.Letbeaclosedsubsetof.Thenlimsup log(21)PropositionII.3:satisfyH1andH2.Letbeanopensubsetof.Thenliminf log(22)iscompact,theabovepropositionsshowthatsatis“esalargedeviationprinciplewithratefunction),whenH1andH2obtain.Thesametechniquesallowustocomputespeci“cR´enyientropiesandguesssworkmoments.Note,however,thatasetwhichsupportsthespeci“cguessworkmomentofcanbedistinctfromonewhichsupports.ForexampleintheBernoulli casewith,forlargethemeasureisconcentratedonwhichhave3,whilethedominantterminthe=1guessworkmomentcomesfromwhichhave 1/3/( 1/ TheoremII.3:satisfyH1andH2.If=1,then log (23)TheoremII.4:satisfyH1andH2.If1,then loggn1)rankkn1))=(1+)sup (24)III.ApplicationtodecodingProofsoftheaboveresultsaregiveninasectionbelow.Firstwediscusstheimplicationsandgiveanapplication.andletbeaclosedneighbourhoodofsuchthat,thenthesamplestatisticsofapproximatethedistributionof.Thenatureoftheapproximationdependsonthechoiceofneighbourhood.TheoremII.2meansthatthenumberofwordsoflengthwiththese IEEETRANSACTIONSONINFORMATIONTHEORY,VOL.XX,NO.Y,MONTH200X7ŠPG0156118612192244226732851036TABLEIIProbabilityandexpectedguesses,=2000Thedi
cultyisthatthisconditionaldistributionmaynothaveasimplemathematicalform.Acasewhichdoeshavearelativelysimpleformisasfollows.Thedistributionofisthedoubleevenshift:eachrunŽof1sor0smusthaveevenlength,andallallowedwordsoflengthhaveequalprobability.Thedistributionofisobtainedfrombyrequiringthattheeventsareindependentwithprobability.Wetaketobeeven.Foralongsequence,byshiftingtheindexby1ifnecessary,wecanarrangeforonlywheniseven.We,...,v2adjacento-pairs:()withodd.Wesaythatadefectoccursatisoddand,whichmeansthatexactlyoneofandobtains.Theprobabilitythatthereareexactlydefectsandexactlyo-pairsbothofwhichhavebeenchangedisGivendefects,thereare2choicesforwhichmemberofeachofthesepairswhichhaschanged;thisiscombinedwiththeprobabilitiescorrespondingtoo-pairsbothofwhichhavechanged.Onecanshowthatinthiscaseis2fortheprobabilitydistributionofthenon-defecto-pairs(thecorrespondingresultalsoholdsfor).Thuswecomputeforthedistribution+(1,...n,whichwedenoten,k.ItiseasytocomputethecorrespondingR´enyientropy:n,k+(1Theexpectednumberofguessesisn,k+(1n,kn,k,weobtain(1+4TableIIshowstheresidualprobabilityandexpectednumberofguessesfortruncatedguessingallowingforuptochangedo-pairsanddefects.IV.ProofsForaproofoftheoremII.1see[2],section6.2.WeprovetheoremII.2usingtheoremII.1,notingthatisaclosedsubsetofandisaclosedsubsetofLemmaIV.1:closedandopen.Thenthereexistssuchthatforalln,Fn,GProof:Iftherewerenosuch,thenwecould“ndasequence)suchthatF,T(28)M\(29) IEEETRANSACTIONSONINFORMATIONTHEORY,VOL.XX,NO.Y,MONTH200X9Theabovecorollaryandthewell-knownlimit(8)provepropositionII.1.LemmaIV.4:For0andasinlemmaIV.3,thereexistaninteger,numbers0,...,Kandsets,...,Ksothateachisopenandeachclosedand(41)| ,(42)| (43)Proof:and  (44)(45)(46) ProofofpropositionII.2:Usingthenotationof(15)from(18)wehavelimsup logn,F(47)Also log logn,F+maxn,F loggn1).(48)FromlemmaIV.3wededucethatforandwn,F loggn1)Šaj+2, e, aj+2.(49)FrompropositionII.1wededuce(50)Nowlimsup log)(51)equalsthemaximumoverthecorrespondinglimsupswithreplacedby.Wethenhavelimsup logmax)+4(52)0isarbitrary,thepropositionfollows. ProofofpropositionII.3:beopenandlet.Thenthereexistssothat.From(19)given0,forallsu
cientlylarge logn,G(53)ByanargumentsimilartothatintheproofofpropositionII.2,(54) IEEETRANSACTIONSONINFORMATIONTHEORY,VOL.XX,NO.Y,MONTH200X11maxn,FFn1).(65)Using(49)andthebound (66)wededucelimsup logmaxn,FFn1)maxlimsup (loglog((67):=limsup log(68)andthenchoosesothat(1+(1+,...,N(69),wehavemaxlimsup (loglog(max+max=(1+(70)becauseif,sothatif,wehave(1+(1+(71)Usingthesametechniquesasinthepreviousproof,wededucelimsup loggn1)rankkn1))supM
S(1+(72)Nextweusethesecondinequalityin(66)todeducen,FFn1)rank n,FFn1)(73)foreachn,Gn,F,wehaveliminf loggn1)rankkn1))
supM
S(1+(74)Thiscoversthe0case.The0issimilar,using (75) IEEETRANSACTIONSONINFORMATIONTHEORY,VOL.XX,NO.Y,MONTH200X13 CharlesEd.PÞsterreceivedthePh.D.degreefromtheSwissFederalInstituteofTechnologyinZ¬urich,Switzerland.HehasheldresearchpositionsattheZentrumf¬urinterdisziplin¬areForschunginBielefeld,Germany;atCNRSinMarseille,France;andatRutgersUniversityinNewBrunswick,N.J.USA.Since1979,hehasaresearchandteachingpositionattheSwissFederalInstituteofTechnologyinLausanne,Switzerland.Hehasworkedinstatisticalmechanics,quantumtheory,andappliedprobability,inparticularonGibbsrandomÞelds.Hiscurrentresearchinterestsareinapplicationoflargedeviationstheorytoinformationtheoryandergodictheory. WayneG.SullivanreceivedaD.Phil.inMathematicsfromOxfordUniversityin1968afterreceivinganundergrad-uatedegreeinChemistryfromGeorgiaInstituteofTechnology.OnthecompletionofhisD.Phil.,hereturnedtotheGeorgiaInstituteofTechnologyasanAssistantProfessorinMathematics.In1973hemovedtoDublin,initiallytotheDublinInstituteforAdvancedStudiesandthentotheDepartmentofMathematics,UniversityCollegeDublin.HeiscurrentlyonleaveofabsencefromUCD,workingasSeniorResearcherattheCommunicationsNetworkResearchInstituteoftheDublinInstituteofTechnology.Hiscurrentinterestistheapplicationoflargedeviationtechniquestoproblemsininformationtheory,symbolicdynamicsandnetworks.