NOTECommunicatedbyRichardZemelNarrowVersusWideTuningCurves:What - PDF document

NOTECommunicatedbyRichardZemelNarrowVersusWideTuningCurves:What’sBestforaPopulationCode?AlexandrePougetSophieDeneveJean-ChristopheDucomGeorgetownInstituteforComputationalandCognitiveSciences,GeorgetownUniver-sity,Washington,DC20007-2197,U.S.A.PeterE.LathamDepartmentofNeurobiology,UniversityofCaliforniaatLosAngeles,LosAngeles,CA90095-1763,U.S.A.Neurophysiologistsareoftenfacedwiththeproblemofevaluatingthequalityofacodeforasensoryormotorvariable,eithertorelateittotheperformanceoftheanimalinasimplediscriminationtaskortocomparethecodesatvariousstagesalongtheneuronalpathway.Onecommonbeliefthathasemergedfromsuchstudiesisthatsharpeningoftuningcurvesimprovesthequalityofthecode,althoughonlytoacertainpoint;sharpeningbeyondthatisbelievedtobeharmful.Weshowthatthisbe-liefreliesoneitherproblematictechnicalanalysisorimproperassump-tionsaboutthenoise.Weconcludethatonecannottell,inthegeneralcase,whethernarrowtuningcurvesarebetterthanwideones;theanswerdependscriticallyonthecovarianceofthenoise.ThesameconclusionappliestoothermanipulationsofthetuningcurveproŽlessuchasgainincrease.1IntroductionItiswidelyassumedthatsharpeningtuningcurves,uptoacertainpoint,canimprovethequalityofacoarsecode.Forinstance,attentionisbelievedtoimprovethecodefororientationbysharpeningthetuningcurvestoori-entationinthevisualareaV4(Spitzer,Desimone,&Moran,1988).ThisbeliefcomespartlyfromaseminalpaperbyHinton,McClelland,andRumelhart(1986),whichshowedthatthereexistsanoptimalwidthforwhichtheac-curacyofapopulationcodeismaximized,suggestingthatsharpeningisbeneŽcialwhenthetuningcurveshaveawidthlargerthantheoptimalone.Thisresult,however,wasderivedforbinaryunitsanddoesnotreadilygeneralizetocontinuousunits.Arecentattempttoshowexperimentallythat,forcontinuoustuningcurves,sharperisbetterreliedonthecenter-of-massestimatortoevaluateNeuralComputation11,85–90(1999)c°1999MassachusettsInstituteofTechnology 86A.Pouget,S.Deneve,J.-C.Ducom,&P.E.Lathamthequalityofthecode(Fitzpatrick,Batra,Stanford,&Kuwada,1997).Theseauthorsmeasuredthetuningcurvesofauditoryneuronstointerauraltimedifference(ITD),acueforlocalizingauditorystimuli.Theyarguedthatnarrowtuningcurvesarebetterthanwideones—intherangetheyobservedexperimentally—inthesensethattheminimumdetectablechange(MDC)inITDissmallerwithnarrowtuningcurveswhenusingacenter-of-massestimator.Theiranalysis,however,sufferedfromtwoproblems:(1)theydidnotconsiderabiologicallyplausiblemodelofthenoise,and(2)theMDCob-tainedwithacenterofmassisnot,inthegeneralcase,anobjectivemeasureoftheinformationcontentofarepresentation,becausecenterofmassisnotanoptimalreadoutmethod(Snippe,1996).AbetterwaytoproceedistouseFisherinformation,thesquarerootofwhichisinverselyproportionaltothesmallestachievableMDCindepen-dentofthereadoutmethod(Paradiso,1988;Seung&Sompolinsky,1993;Pouget,Zhang,Deneve,&Latham,1998).(Shannoninformationwouldbeanothernaturalchoice,butitissimply,andmonotonically,relatedtoFisherinformationinthecaseofpopulationcodingwithalargenumberofunits;seeBrunel&Nadal,1998.Itthusyieldsidenticalresultswhencomparingcodes.)Todeterminewhethersharptuningcurvesareindeedbetterthanwideones,onecansimplyplottheMDCobtainedfromFisherinformationasafunctionofthewidthofthetuningcurves.FisherinformationisdeŽnedasIDEµ¡@2@µ2logP.A|µ/¶;(1)whereP.A|µ/isthedistributionoftheactivityconditionedontheencodedvariableµandE[¢]istheexpectedvalueoverthedistributionP.A|µ/.Asweshownext,sharpeningincreasesFisherinformationwhenthenoisedistributionisŽxed,butsharpeningcanalsohavetheoppositeeffect:itcandecreaseinformationwhenthedistributionofthenoisechangeswiththewidth.Thelattercase,whichhappenswhensharpeningistheresultofcomputationinanetwork,isthemostrelevantforneurophysiologists.ConsiderŽrstthecaseinwhichthenoisedistributionisŽxed.Forin-stance,forapopulationofNneuronswithgaussiantuningcurvesandindependentgaussiannoisewithvariance¾2,FisherinformationreducestoIDNXiD1f0i.µ/2¾2;(2)wherefi.µ/isthemeanactivityofunitiinresponsetothepresentationangle,µ,andf0i.µ/isitsderivativewithrespecttoµ.Therefore,asthewidthofthetuningcurvedecreases,thederivativeincreases,resultinginanin- NarrowVersusWideTuningCurves87creaseofinformation.ThisimpliesthatthesmallestachievableMDCgoesupwiththewidthoftuning,asshowninFigure1A,becausetheMDCisinverselyproportionaltothesquarerootoftheFisherinformation.Thisisacasewherenarrowtuningcurvesarebetterthanwideones.Note,how-ever,thattheoptimaltuningcurveforFisherinformationhaszerowidth(or,moreprecisely,awidthontheorderof1=N,whereNisthenumberofneurons),unlikewhatHintonetal.foundforbinarytuningcurves.Notealsothatforthesamekindofnoise,theMDCmeasuredwithcenterofmassshowstheoppositetrend—wideisbetter—conŽrmingthattheMDCob-tainedwiththecenterofmassdoesnotreecttheinformationcontentofthe1ConsidernowacaseinwhichthenoisedistributionisnolongerŽxed,suchasinthetwo-layernetworkillustratedinFigure1B.Thenetworkhasthesamenumberofunitsinbothlayers,andtheoutputlayercontainslat-eralconnections,whichsharpenthetuningcurves.Thiscaseisparticularlyrelevantforneurophysiologistssincethistypeofcircuitisquitecommoninthecortex.Infact,someevidencesuggeststhatasimilarnetworkisinvolvedintuningcurvesharpeningintheprimaryvisualcortexfororientationse-lectivity(Ringach,Hawken,&Shapley,1997).Dotheoutputneuronscontainmoreinformationthantheinputneuronsjustbecausetheyhavenarrowertuningcurves?Theanswerisno,regardlessofthedetailsoftheimplementation,becauseprocessingandtransmissioncannotincreaseinformationinaclosedsystem(Shannon&Weaver,1963).Sharpeningisdoneatthecostofintroducingcorrelatednoiseamongneu-rons,andthelossofinformationintheoutputlayercanbetracedtothosecorrelations(Pouget&Zhang,1996;Pougetetal.,1998).Thisisacasewherewidetuningcurves(theonesintheinputlayer)arebetterthannarrowones(theonesintheoutputlayer).Thatwidetuningcurvescontainmoreinformationthannarrowonesinthisparticulararchitecturecanbeeasilymissedifoneassumesthewrongnoisedistribution.Unfortunately,itisdifŽculttomeasurepreciselythejointdistributionofthenoiseorevenitscovariancematrix.Itisthereforeoftenassumedthatthenoiseisindependentamongneuronswhendealingwithrealdata.Let’sexaminewhathappensifweassumeindependentnoisefortheoutputunitsofthenetworkdepictedinFigure1B.Weconsiderthecaseinwhichtheoutputunitsaredeterministic;theonlysourceofnoiseisintheinputactivities,andtheoutputtuningcurveshavethesamewidthastheinputtuningcurves.Wehaveshown(Pougetetal.,1998)thatinthiscase,thenetworkperformsacloseapproximationtomaximumlikelihoodandthenoiseintheoutputunitsisgaussianwithvariancef0i.µ/2=I1,whereI1is1Fitzpatricketal.(1997)reportedtheoppositeresult.Theyfoundsharptuningcurvestobebetterthanwideoneswhenusingacenter-of-massestimator.Thisisbecausethenoisemodeltheyusedisdifferentfromoursandbiologicallyimplausible. 88A.Pouget,S.Deneve,J.-C.Ducom,&P.E.Latham020406080051015202530 M i n i m u m D e t e c t a b l e C h a n g e Width (deg)AInputOutputBOrientationOrientation A c t i v i t y A c t i v i t y ( d e g ) Figure1:(A)ForaŽxednoisedistribution,theminimumdetectablechange(MDC)obtainedfromFisherinformation(solidline)increaseswiththewidth.Therefore,inthiscase,narrowtuningcurvesarebetter,inthesensethattheytransmitmoreinformationaboutthepresentationangle.Notethatusingacenter-of-massestimator(dashedline)tocomputetheMDCleadstotheoppositeconclusion:thatwidetuningcurvesarebetter.Thisisacompellingdemonstra-tionthatthecenterofmassisnotaproperwaytoevaluateinformationcontent.(B)Aneuralnetworkwith10inputunitsand10outputunits,fullyconnectedwithfeedforwardconnectionsbetweenlayersandlateralconnectionsintheout-putlayer.Weshowonlyonerepresentativesetofconnectionsforeachlayer.Thelateralweightscanbesetinsuchawaythatthetuningcurvesintheoutputlayerarenarrowerthanintheinputlayer(seePougetetal.,1998,fordetails).Becausetheinformationintheoutputlayercannotbegreaterthantheinformationintheinputlayer,sharpeningtuningcurvesintheoutputlayercanonlydecrease(oratbestpreserve)theinformation.Therefore,thewidetuningcurvesintheinputlayercontainmoreinformationaboutthestimulusthanthesharptuningcurvesintheoutputlayer.Inthiscase,widetuningcurvesarebetter.theFisherinformationintheinputlayer.Usingequation2forindependentgaussiannoiseweŽndthattheinformationintheoutputlayer,denotedI2,isgivenby:I2DNXiD1f0i.µ/2f0i.µ/2=I1DNXiD1I1DNI1:Theindependenceassumptionwouldthereforeleadustoconcludethattheinformationintheoutputlayerismuchlargerthanintheinputlayer,whichisclearlywrong. NarrowVersusWideTuningCurves89Thesesimpleexamplesdemonstratethatapropercharacterizationoftheinformationcontentofarepresentationmustrelyonanobjectivemeasureofinformation,suchasFisherinformation,anddetailedknowledgeofthenoisedistributionanditscovariancematrix.(Thenumberofvariablesbeingencodedisalsocritical,asshownbyZhangandSejnowski,1999)Usingestimatorssuchasthecenterofmass,orassumingindependentnoise,isnotguaranteedtoleadtotherightanswer.Therefore,attentionmaysharpen(Spitzeretal.,1988)tuningcurves(and/orincreasetheirgain;McAdams&Maunsell,1996),butwhetherthisresultsinabettercodeisimpossibletotellwithoutknowledgeofthecovarianceofthenoiseacrossconditions.Theemergenceofmultielectroderecordingsmaysoonmakeitpossibletomeasurethesecovariancematrices.AcknowledgmentsWethankRichZemel,PeterDayan,andKechenZhangfortheircommentsonanearlierversionofthisarticle.ReferencesBrunel,N.,&Nadal,J.P.(1998).Mutualinformation,Fisherinformationandpopulationcoding.NeuralComputation,Inpress.Fitzpatrick,D.C.,Batra,R.,Stanford,T.R.,&Kuwada,S.(1997).Aneuronalpopulationcodeforsoundlocalization.Nature,388,871–874.Hinton,G.E.,McClelland,J.L.,&Rumelhart,D.E.(1986).Distributedrepre-sentations.InD.E.Rumelhart,J.L.McClelland,&thePDPResearchGroup(Eds.),Paralleldistributedprocessing(Vol.1,pp.318–362).Cambridge,MA:MITPress.McAdams,C.J.,&Maunsell,J.R.H.(1996).Attentionenhancesneuronalre-sponseswithoutalteringorientationselectivityinmacaqueareaV4.SocietyforNeuroscienceAbstracts,22.Paradiso,M.A.(1988).Atheoryoftheuseofvisualorientationinformationwhichexploitsthecolumnarstructureofstriatecortex. BiologicalCybernetics, 58 ,35–49. Pouget,A.,&Zhang,K.(1996).Astatisticalperspectiveonorientationselectivityinprimaryvisualcortex.SocietyforNeuroscienceAbstracts,22,1704.Pouget,A.,Zhang,K.,Deneve,S.,&Latham,P.E.(1998).StatisticallyefŽcientestimationusingpopulationcoding. NeuralComputation,10 ,373–401. Ringach,D.L.,Hawken,M.J.,&Shapley,R.(1997).Dynamicsoforientationtuninginmacaqueprimaryvisualcortex.Nature,387,281–284.Seung,H.S.,&Sompolinsky,H.(1993).Simplemodelforreadingneuronalpopulationcodes. ProceedingsofNationalAcademyofSciences,USA,90 ,10749– 10753. Shannon,E.,&Weaver,W.(1963).Themathematicaltheoryofcommunication.Ur-bana:UniversityofIllinoisPress. 90A.Pouget,S.Deneve,J.-C.Ducom,&P.E.LathamSnippe,H.P.(1996).Parameterextractionfrompopulationcodes:Acritical NeuralComputation,8 ,511–530. Spitzer,H.,Desimone,R.,&Moran,J.1988.Increasedattentionenhancesbothbehavioralandneuronalperformance. Science,240 ,338–340. Zhang,K.,&Sejnowski,T.J.(1999).Neuronaltuning:Tosharpenorbroaden?. NeuralComputation,11 ,75–84. ReceivedMarch16,1998;acceptedJune25,1998.