PavlovianConditioningHowexcitationandinhibitionDetermineIdeomotionAdel - PDF document

PavlovianConditioningHowexcitationandinhibitionDetermineIdeomotionAdelaFlorentinaIliescu AthesispresentedforthedegreeofDoctorofPhilosophyCardiUniversitySchoolofPsychologyNovember,2019 SummaryAssociativetheoriesassumeasimpleordinalmappingbetweenthestrengthofanassociationbetweenaconditionedstimulus(CS)andanunconditionedstimulus(US)andconditionedbehaviourinanexperimentalpreparation.Recentstudiesthathavetakenmultiplemeasuresofconditionedbehaviourchallengethisassumption.Thepur-poseofthisthesisisabetterunderstandingofthenatureoftheseindividualdierencesinPavlovianconditioning,combiningempiricalevidencewiththeoreticaldevelopment.Ithasbeenobservedthatsimpleauto-shapingproceduresresultinmarkedindividualdierences:someratsshowlearningbyinteractingwiththesign(sign-trackers,STs),othersbyinteractingwiththefood-wellorthegoal(goal-trackers,GTs).InChapter2,Iexaminedthesensitivityofthesetwobehaviours(sign-trackingandgoal-tracking)tochangingcontingencies.InbothSTsandGTs,US-orientedbehaviourwasmoresensitivetocontingencychangesthanCS-orientedbehaviour.Mostattemptstoex-plainthisdissociationhaveappealedtoadual-mechanismsapproach.InChapter3,Ipresentanewtheoreticalmodel,HeiDI,whichintegrateslearningandperformancefromasingle-processperspective.InChapter4,Iexaminetwoofthesepredictions.TherstpredictionrelatestohowtheUS-valueaectsthedistributionofconditionedbehaviour.AccordingtoHeiDI,ahigherUS-valuewillresultinhigherlevelsofgoal-trackingincontrastwithlowerUSvalue.Experiment3suggestedthatthatahigherUS-valueresultsinmoreCS-orientedbehaviour,howeverthiswasnotreplicatedinExperiment4.Thesecondpredictionconcernsananalysisof

thefeaturepositiveeect,wherethediscriminationemergesmorereadilyforafeaturepositivedesign.HeiDIpredictsthatafeaturepositiveeectshouldbemoreevidentinCS-orientedbehaviour.Experiments4and5addressedthisprediction,howevertheanimalsdidnotshowlearninginthefeaturenegativedesign.TheimplicationsofthenewndingsandthenewmodelarediscussedinChapter5.1 AcknowledgementsIwouldliketorstexpressmygratitudetomysupervisors,ProfessorRobHoneyandProfessorDominicDwyer,fortheguidance,patience,encouragementandsupportduringthesewonderfulyears.IcannotexpresshowluckyIfeeltohaveyouasmymentors.Sergio,thankyouforsupportingmethroughthegoodandthebad.Youhavegivenmethepowertoovercomealltheobstacles.YouhavebeenmyinspirationandIwouldnothaveachievedthiswithoutyou.IwouldalsoliketothankallmycolleaguesfromtheBNL,Patricia,Sophie,andout-sidethe`bananaworld',Silvia,Oana,forbeingincrediblysupportiveandunderstanding.Thankyouforthe`shoulder'.Mami,multumescpentrusprijinulacordat^ntotiacestiani.Multumescpentruexemplulacordat,pentruc~ami-aifostmereual~aturi,pentrusfaturisiinspiratie.2 DeclarationThisworkhasnotbeensubmittedinsubstanceforanyotherdegreeorawardatthisoranyotheruniversityorplaceoflearning,norisbeingsubmittedconcurrentlyincandidatureforanydegreeorotheraward.Signed:AdelaF.IliescuDate18/09/2019STATEMENT1ThisthesisisbeingsubmittedinpartialfullmentoftherequirementsforthedegreeofPhD.Signed:AdelaF.IliescuDate18/09/2019STATEMENT2Thisthesisistheresultofmyownindependentwork/investigation,exceptwhereother-wisestated.Othersourcesareacknowledgedbyexplicitreferences.Theviewsexpressedaremyown.Signed:AdelaF.Ilies

cuDate18/09/2019STATEMENT3Iherebygiveconsentformythesis,ifaccepted,tobeavailableforphotocopyingandforinter-libraryloan,andforthetitleandsummarytobemadeavailabletooutsideorganisations.Signed:AdelaF.IliescuDate18/09/2019STATEMENT4:PREVIOUSLYAPPROVEDBARONACCESSIherebygiveconsentformythesis,ifaccepted,tobeavailableforphotocopyingandforinter-libraryloansafterexpiryofabaronaccesspreviouslyapprovedbytheAcademicStandards&QualityCommittee.Signed:AdelaF.Iliescu3 Date18/09/2019LargepartsofChapter2,3and5appearin:Iliescu,A.F.,Hall,J.,Wilkinson,L.S.,Dwyer,D.M.,&Honey,R.C.(2018).ThenatureofphenotypicvariationinPavlovianconditioning.JournalofExperimentalPsychology:AnimalLearningandCognition,44(4),358{369.Honey,R.C.,Dwyer,D.M.,&Iliescu,A.F.(2019).HeiDI:AmodelforPavlovianlearningandperformancewithreciprocalassociations.PsychologicalReview(inrevi-sion).IcontributedtotheconceptualdevelopmentofHeiDI,andtoitsformalpresentationforpublication.IalsobuilttheHeiDIapp,whichgeneratesthesimulatedresults,andderivedpredictionsfromit.TheaccuracyofthesepredictionswastestedinChapter4.4 Contents1GeneralIntroduction101.1TheoreticalanalysesofPavlovianlearningandbehaviour.........131.2Individualdierencesinconditionedbehaviour...............141.3Conditionsthataectsignandgoaltracking................161.4Theoreticalbackground............................191.4.1S-SvsS-Rlearning..........................201.4.2Behavioursystemstheory......................221.4.3Reinforcementlearning........................241.5ThesisOutline.................................292Investigatingthenatureofphenotypicvariation312.1Introduction...............................

...312.2Experiment1.................................352.2.1Method................................352.2.2Results.................................392.2.3DiscussionExperiment1.......................452.3Experiment2.................................452.3.1Method................................462.3.2Results.................................492.3.3DiscussionExperiment2.......................552.4Chapter2Discussion.............................553HeiDI:AmodelforPavlovianlearningandperformancewithrecipro-calassociations593.1TheRescorla-WagnerModel.........................603.2Individualdierences.............................623.3HeiDI:Rationale,architectureandoverarchingassumptions........633.4Learningrules.................................675 3.4.1Excitatorylearninganderrorcorrection..............683.4.2Extinction...............................723.4.3Inhibitorylearning..........................743.5Performancerules...............................743.5.1IndividualdierencesinUS.....................763.6Simulationsoflearningandperformance..................793.6.1Excitatoryconditioningandextinction...............793.6.2ChangeinUSvalue..........................823.6.3Compoundconditioningandthepoolederrorterm........833.6.4Combiningstimuliwithdierentassociativehistories.......853.6.5Blocking:Learningandperformance................913.6.6Latentinhibition:Analternativeassociativeanalysis.......963.7Listofequations...............................994TestingpredictionsofHeiDI1004.1IndividualdierencesinUS.........................1004.1.1Introduction..............................1004.1.2Experiment3............

.................1014.1.3Experiment4.............................1094.1.4DiscussionExperiments3and4...................1164.2Featurepositiveeect............................1174.2.1Introduction..............................1174.2.2Experiment5:Levercondition....................1204.2.3Experiment6:Lightcondition....................1324.2.4DiscussionExperiments4and5...................1375GeneralDiscussion1395.1Empiricalndings...............................1405.2Theoreticalimplications:HeiDI.......................1446 5.2.1Conditions,contentandperformance................1455.2.2Associationsbetweenthecomponentsofacompound.......1465.2.3Elementalandconguralprocesses.................1485.2.4Limitationsandfurtherdevelopment................1495.2.5Concludingcomments........................1506Appendix1526.1ReproducibilityandOpenScienceFramework(OSF)...........1526.2HeiDIapp...................................1526.3Additionalnotes...............................1547 ListofFiguresFigure1:Distributionofsign-trackingandgoal-trackingbehaviourforExper-iment1....................................39Figure2:ResultsfromExperiment1:theeectsofareversalprocedureonsign-trackingandgoal-tracking.......................40Figure3:CorrelationresultsfromExperiment1:theeectsofareversalprocedureonsign-trackingandgoal-tracking................41Figure4:Distributionofsign-trackingandgoal-trackingbehaviourforExper-iment2....................................49Figure5:ResultsfromExperiment2:theeectsofaswitchinUSvalueonsign-trackingandgoal-tracking.......................50Figure6:CorrelationresultsfromExperiment2:theeectsof

aswitchinUSvalueonsign-trackingandgoal-tracking..................51Figure7:Aschematicforassociativestructuresthatunderpinthetranslationofexcitatorylearningintoperformance...................64Figure8:HeiDI:Simulationsoftheeectsofanextinctionprocedureonsign-trackingandgoal-tracking..........................81Figure9:SimulationsofHeiDIfortheeectsofachangeinUSvalueonsign-trackingandgoal-tracking.......................83Figure10:SimulationsofHeiDIofthechangesinassociativestrengthwhencombiningstimuliwithdierentassociativehistories............86Figure11:SimulationsofHeiDIofthechangesinassociativestrengthduringstandardblockinganddownshiftunblockingprocedures..........94Figure12:Distributionofsign-trackingandgoal-trackingbehaviourforEx-periment3...................................103Figure13:SimulationofHeiDIforindividualdierencesinUSonsignandgoal-tracking.................................1048 Figure14:ResultsfromExperiment3:theeectofdierentvaluesofUSonsign-trackingandgoal-tracking.......................105Figure15:ResultsfromExperiment4:theeectofdierentvaluesofUSonsign-trackingandgoal-tracking(replicationofExperiment3).......111Figure16:Distributionofsign-trackingandgoal-trackingbehaviourforEx-periment4...................................112Figure17:SimulationsofHeiDIofafeaturepositiveeectwhenparametersarearrangedforabiastowardsRCS.....................122Figure18:SimulationsofHeiDIofafeaturepositiveeectwhenparametersarearrangedforabiastowardsRUS.....................123Figure19:Distributionofsign-trackingandgoal-trackingbehaviourforEx-periment5.................

..................124Figure20:ResultsfromExperiment5:theeectsofafeaturepositivedesignonsign-trackingandgoal-tracking......................125Figure21:ResultsfromExperiment5splitbytypeofstimuli:AvsBvsB(AB)126Figure22:Distributionofsign-trackingandgoal-trackingbehaviourforEx-periment6...................................134Figure23:ResultsfromExperiment6:theeectsofafeaturepositivedesignonsign-trackingandgoal-tracking......................1359 Chapter1GeneralIntroduction\Givemeadozenhealthyinfants,well-formed,andmyownspeciedworldtobringthemupinandI'llguaranteetotakeanyoneatrandomandtrainhimtobecomeanytypeofspecialistImightselect|doctor,lawyer,artist,merchant-chiefand,yes,evenbeggar-manandthief,regardlessofhistalents,penchants,tendencies,abilities,vocations,andraceofhisancestors"(Watson,1924,p.104).Watsonbelievedthatexperienceortrainingwastheprincipalfactorthatcontributestowhatapersonbecomes.ItisdiculttobelievethatanyonecouldnothaveacceptedatleastsomeversionofWatson'sclaim,thatdierenttrainingallowspeopletobecomedoctors,lawyers,artists,merchant-chiefs.Accordingtothisview,allindividualdier-encesinbehaviourre ectdierentlearningexperiences.However,thefocusofthisthesisisonwhatisunacknowledgedbyWatson'sstatement,namelythatmaterialindividualdierencesstillemerge,inspiteofsharedexperienceandtraining(e.g.,Flagel,Akil,&Robinson,2009;Patitucci,Nelson,Dwyer,&Honey,2016).Akeyquestionthatthisthesiswilladdress,iswhethertheseindividualdierencesre ectdierencesinthecontentoflearning(whatislearntinagivensituation)asop-posedtopossibledierencesinthemechanismsoflearningbetweensubjects.In

ordertoaddressthiskeyquestion,oneneedstounderstandwhatislearnt(thecontentoflearning)inagivensituation,whatisthemechanismunderlyingthatlearning,andhowwhathasbeenlearntistranslatedintobehaviour(Rescorla,1988).Beforeaddressingtheseissues,Iwillrstbrie yreviewtheconceptoflearningfromanassociativeperspec-tive,beforeturningtoindividualdierencesinaubiquitousformoflearning(Pavlovianconditioning)andoutlinecontemporaryaccountsofsuchdierences.Theformalstudyofanimallearninghasalonghistoryanditispossibletoiden-tifytwomainapproachesthathaveledtowhatwenowknow.Therstapproachwasconcernedwiththestudyofadaptivebehaviourfromanevolutionarypointofview,10 contrastingadaptivespecialisationofabilitiestotheenvironmentofdierentspecies(Mackintosh,1974,p.2).Thiscomparativeapproachfocussedonnaturallyoccurringacquiredbehaviours(e.g.,thedevelopmentofbirdsongandlialimprinting),andcanbecontrastedwiththestudyoftheprocessesunderlyingmodicationsofbehaviourinprocedureschosentoinvolvemorearbitrarystimuli,responsesandtherelationshipsbetweenthem.Thelatterapproachisexempliedbytheexperimentalinvestigationofchangesinbehaviourandtheconditionswhenthesechangesoccurinlaboratorytasks,whichhaveattemptedtorevealgeneralprinciplesoflearning.Thisformofinvestigationinvolvedattemptstounderstandtheprocessesinvolvedinlearning,orthegeneralprin-ciplesinoperation,throughananalysisofwhensuchchangesinbehaviouroccur.Inthiscase,changesinbehaviourcanbeseenasboththethingtobeunderstoodandthemeansofdoingso,whichhasattendantproblems.Forexample,ifoneassumesthatthechangeinbehaviourbroughtaboutbyaPavlovianconditioningprocedurere ectsanassociation

betweentheeventsthathavebeenpresented(e.g.,achimingbellandfood),thenoneneedstospecifytheperformancerule(s)bywhichthatassociationisrevealedinbehaviour.Infact,asIwillshow,theoriesofassociativelearninghavetypicallyoeredonlythemostimpoverishedofperformancerules.Intheeldofanimallearning,Mackintosh(1974)notedtwohistoricalgures,PavlovandThorndike,whoprofoundlyshapeddecadesofresearch,basedontheirinitialproce-duresandtheoreticalinsights.Fromtheiroriginalexperimentalwork,twoexperimentalparadigmshavedeterminedthecourseofmostsubsequentresearchonlearning:clas-sicalconditioning(Pavlov)andinstrumentalconditioning(Thorndike).Notethatthedistinctionbetweenclassicalandinstrumentalconditioningreferstotheexperimentalarrangementstostudylearning(operationallytherulesthatdeterminedthedeliveryofareinforcerorreward),butdoesnotnecessarilyimplydierentlearningprocesses.Experimentsonclassicalconditioningarethoseinwhichthecontingenciesarearrangedbetweenstimulipresentedindependentlyoftheactionsofananimal(e.g.,astimulusprecedinganoutcome)andthoseoninstrumentalconditioning,thecontingenciesare11 arrangedbetweentheactionsofananimalandotherevents(e.g.,aresponsehasthescheduledconsequenceofanoutcome).Thorndike'slawofeectstatedthatanybehavioursthatarefollowedbypositiveconsequencesarelikelytoberepeatedandtheonesfollowedbynegativeconsequencesarelikelytobesuppressed(Thorndike,1927).Ininstrumentalconditioning,learningisgenerallyassessedbymeasuringthebehaviourorsetofbehavioursthatarefollowedbyreinforcement,andanincrease(ordecrease)inrespondingistakenasevidenceoflearning.Inclassicalconditioningthesubjectispresentedwitharelationshipbe

tweenastimulusandanoutcome(e.g.,atonefollowedbyfood)andchangesinbehaviouroccurswithoutanyobviouscontingenciesbetweenthebehaviouritselfandtheoutcomethatfollowsit.InatypicalclassicalorPavlovianconditioningprocedure,aneutralcondi-tionedstimulus(CS)ispairedwithamotivationallysignicantunconditionedstimulus(USe.g.,food),whichproducesunconditionedresponses(UR;salivation).AsaresultofpairingtheCSandUStheCScomestoevokeaconditionedresponse(CR).ThisfundamentallearningprocesswasrstformallydocumentedbyPavlov(1927),whono-ticedthatdogswouldsalivateinthepresenceofcuesthatpredictedbeingfed(e.g.,theexperimentercomingintotheroom).ToexplainwhyaCScomestoevokeaCR,Pavlov(1930)suggestedthattheanimaltreatstheCSasitwouldbetheUS(stimulussubstitutiontheory).Thisanalysispre-dictsthattheCSwillcometoevokearesponsegreatlyresemblingthatevokedbytheUS,asitappearedtodoinPavlov'soriginalstudiesandmanyofthosethatfollowed.Forexample,inpigeonautoshaping,whentheCSisanilluminatedkeyandtheUSisthedeliveryofgraintoafood-well,theCRtothekeylightresemblestheURtothedeliveryoffood(i.e.,pecking).Moreover,anexperimentbyJenkinsandMoore(1973)manipulatedthenatureoftheUS(grainorwater)inapigeonautoshapingprocedureandshowedthatwhentheUSwasgrainthepresentationofthekeylightelicited\foodpecks"whereaswhenitwaswaterthekeylightelicited\waterpecks":theCRsresem-bledthedistinctconsummatoryresponsestofoodandwater.However,theprinciple12 ofstimulussubstitutionischallengedbyconditioningprocedureswhichresultinaCRthatisdierentfromtheUR.Forexample,whenratsaregivenpairingsofaCSwithelectricshock(theUS),whiletheCRisoftenacessationofallactivity(i.e.,freezing),theUSitselfiniti

allyprovokesaburstinactivity(e.g.,Blanchard,Fukunaga,&Blan-chard,1976).Whiletheideaofstimulussubstitutionhasbeenchallenged,contemporarytheoriesofPavlovianconditioning,thefocusofthisthesis,havedonelittlebywayofaddressinghow(associative)learningbecomesevidentinacquiredbehaviour.1.1TheoreticalanalysesofPavlovianlearningandbehaviourMorerecenttheoreticaltreatmentsofPavlovianconditioninghavefollowedPavlov'sleadinassumingthatpairingsofaCSwithaUSresultintheformationofanassociationbetweenthecentralrepresentationsoftheevents(e.g.,Mackintosh,1975;Pearce&Hall,1980;Rescorla&Wagner,1972;Wagner,1981).Thisproposition{thatthecriticalassociationisstimulus-stimulus(S-S)ratherthanbetweenastimulusandresponse(S-R){carrieswithitaneedtospecifyhowlearningistranslatedintoperformance.TheRescorla-Wagnermodelremainsoneofthemostin uentialintheeld,anditsapplicationtotheissuesthatarecentraltothisthesiswillbeoutlinednext,includingitsapproachtoobservedperformance.TheRescorla-WagnerModel(Rescorla&Wag-ner,1972)assumesthatconditionedbehaviourre ectstheformationofanassociationbetweentherepresentationsactivatedbytheCSandUS.ThepresentationoftheCScomestoassociativelyactivatetherepresentationoftheUSandtherebybehaviour.Thechangeintheassociativestrength(
VCS�US)ofaCSonagiventrialisdeterminedbythedierencebetweenthemaximumassociativestrengthsupportablebyaUS()andthepooledassociativestrengthofallstimulipresentedonthattrial(PVTotal�US).ThecurrentassociativestrengthofthatstimulusVCS�USisthesumofVCS�USontheprevioustrialandthechangeinassociativestrengthonthecurrenttrial(
VCS�US).13 Therateoflearningismodulatedbytwolearningparame

tersCSandUSandthemodelassumesthatCSandUSarerelatedtothephysicalproperties(i.e.,salienceorintensity)oftheCSandUS.IwilldiscusstheRescorla-ModelinmoredetailinChapter3.However,theresultingequationforthechangeinlearningonagiventrialproposedbythismodeltakestheform:
VCS�US=CSUS(-PVTotal�US).AconsequenceofadoptingtheviewthatPavlovianconditioninginvolvesS-Sassoci-ations(asopposedtoS-Rassociations)isthatmodelsliketheonedevelopedbyRescorlaandWagner(1972)neededtoprovidesomeanalysisofhowsuchassociationsaretrans-latedintoobservablebehaviour.Thisneedissharedbyotherformalassociativelearningmodels(e.g.,Mackintosh,1975;Pearce&Hall,1980;Wagner,1981).Ofcourse,asso-ciativetheoristsarenotblindtothisissue,andsohavetypicallyoeredthesimplifyingassumptionthatrespondingelicitedbytheassociativeactivationofarepresentationofaUSismonotonicallyrelatedtothestrengthoftheunderlyingassociation.Whilethisassumptionhasservedforsometime,Iwillnextdiscusstheresultsofrecentstudies,whichposeachallengetoanylearningtheorythatassumesamonotonicrelationshipbetweenlearningandperformance.1.2IndividualdierencesinconditionedbehaviourIndividualdierenceswereacentralfocusofpsychologistsintheearly20thcentury,butthestudyofindividualdierencesinanimallearninghasplayedlittleroleinthedevelopmentofcontemporarytheory(Matzel&Sauce,2017).Perhapsthisfactre ectedtheviewthatthecontrolledandsharedenvironmentsandrestrictedgeneticbackgroundofmanylaboratoryanimalsshouldlimittheextentofindividualdierencesinlearntbehaviourbetweenthem.However,recentevidencesuggeststhatPavlovianconditioningprocedurescanproducemarkedindiv

idualdierencesinconditionedbehaviour.Forexample,Patitucci,Nelson,Dwyer,andHoney(2016,Experiment2)presentedratswithtwolevers.Onelever(e.g.,leftlever),thereinforcedlever,wasinsertedintheexperimentalchamberfor10sandwasimmediatelyfollowedbyaccesstosucroseas14 reward.Thecontrollever(e.g.,rightlever)ontheotherhand,wasinsertedinthechamber,howeveritwasnotreinforced.Theratsreceivedtwentytrialsofeachleverfor12days.Someratsshowedthattheyhadlearnttherelationshipbetweenthereinforcedleverandfoodbypredominantlyinteractingwiththelever(orthesign)andothersbyinvestigatingthefood-well(orthegoal),wherethereinforcerwasabouttobedelivered.Likeothers,Irefertothebehaviourofengagingwiththelever(typicallyautomaticallyrecordedasleverpresses)assign-tracking,andtotheanimalsthatshowatendencytowardsthesignortheleverassign-trackers(STs).Similarly,Irefertothebehaviourofengagingwiththefood-well(typicallyautomaticallyrecordedasfood-wellentries)asgoal-tracking,andtotheanimalsthatshowatendencytoengagewiththegoalasgoal-trackers(GTs)(Brown&Jenkins,1968;Hearst&Jenkins,1974).InthestudyconductedbyPatituccietal.(2016,Experiment2),inwhichtheinser-tionofoneleverwasfollowedbysucroseandtheinsertionofanother(controllever)wasnot,amediansplitwasusedtoseparateratsintotwogroups(STsandGTs)onthebasisofwhethertheiractivityduringthenalblockoftrainingwaspredominantlydirectedto-wardstheleverorfood-well.Thisanalysisallowedthedevelopmentofthesign-trackingandgoal-trackingbehaviourstobetracedacrosstraining.However,analysisatthelevelofindividualratsrevealsthatthebiastowardssign-trackingorgoal-trackingwasrelativelycontinuousinnature.Whenleveractivityorsign-trackingwas

usedtoasseslearning,theSTsshowedbetterlearningthantheGTs;butwhenfood-wellactivityorgoal-trackingisusedthenthereverseistrue.Focussingononemeasure(e.g.,sign-tracking)leadstotheconclusionthatlearninghadproceededmorereadilyinonesetofratsthantheother,whilefocussingonthesecondmeasure(e.g.,goal-tracking)leadstotheoppositeconclusion.Asitwillbediscussedinlengthinfuturesections,learningtheoriesthatassumeamonotonicrelationshipbetweenlearningandacquiredbehaviour(e.g.,Mackintosh,1975;Pearce&Hall,1980;Rescorla&Wagner,1972;Wagner,1981)areunabletoexplainthisdissociationintheexpressionoflearning:thereisnosimplymonotonicmappingbetweenthestrengthoftheCS-USassociationandthedierent15 levelsofleverpressandfood-wellentriesseenacrosssign-andgoal-trackinganimals.Theseexperimentshighlighttheimportanceofhowlearningisassessed,howittrans-latesinperformance,andthefactthatonemeasurementalonemightnotbeenoughtodrawgeneralconclusionsaboutwhattheanimalhaslearntinanygivensetofcircum-stances.Inthefollowingsection,IdiscussstudiesonPavlovianconditioningwheremultiplemeasurementsofconditionedbehaviourhavebeencontrasted,whichsuggeststhatsign-trackingandgoal-trackingbehavioursareaecteddierentlybydierenttypesofmanipulations.1.3ConditionsthataectsignandgoaltrackingThisthesisisabouthowlearningistranslatedintobehaviour.Therehasbeenagreatdealofresearchconcerningsign-andgoal-tracking,althoughnotnecessarilyinthecontextofindividualdierences.Iwillfocusnextontheexperimentalmanipulationsthataectsign-andgoaltracking.Considerationoftheeectsofthesemanipulationsarekeytothedevelopmentofanygeneraltheorythatintegrateslearningandperfo

rmance;suchtheorywillbepresentedinChapter3.StudiesbyWasserman,Franklin,andHearst(1974)showthatpositivecontingencybetweentheCSandUSisneededforthetwocriticalbehaviourstoemerge.Wassermanetal.(1974)exposedpigeonstoeitherapositivecontingency(alightedkeyCSwhichwasalwaysfollowedbygraindelivery,US),negativecontingency(theCSandUSwereexplicitlyunpaired),backwardspairing(theUSwaspresentedbeforetheCS)orCSonly(theCSwasneverfollowedbyanygraindelivery)procedure.Theonlybirdsthatshowedsign-tracking,bypeckingatthelightedkey,weretheonesexposedtoapositivecontingencybetweentheCSandUS.Onthebasisofkey-peckingmeasurementalonethebehaviourofbirdsintheothergroupscouldnotbedierentiatedfromeachother.However,onthebasisofapproach-withdrawalmeasurements,thegroupsofbirdscouldbeclearlydistinguished.BirdsintheCSonlyandbackwardpairingsgroupdidnotdisplayanyapproach-withdrawalbehaviour,whilebirdsinthenegativecontingency16 groupspenttheirtimeontheothersideofthechamber,awayfromtheCS.EvenifWassermanetal.(1974)didnotmeasuregoal-trackingdirectly,theirmeasurementsofapproachandwithdrawalseemtosuggestthatthesameistrueforgoal-tracking,aspigeonsspenttheirtimeawayfromthesiteofUSdelivery,makingithardforthemtoshowanygoal-trackingbehavioureither.IntheCSpositivecontingencygroup,grainalwaysfollowedtheCS,howeverithasbeenseenthatpartialreinforcementofaslittleasof25percentoftheCStrials,producessign-trackingbehaviour(Wasserman,1974).Partialreinforcementincreasedthenumberoftrialstotherstpeck,howeveritproducedhigherasymptoticsign-trackinglevels(Davey&Cleland,1982;Gottlieb,2004,2006).Incontrast,goal-trackingwassuppressedwithpartialreinforcementscheduleswhencompar

edwithacontinuouslyreinforcedgroup(Davey&Cleland,1982).PriorexperiencewiththeCSortheUSaectsdierentlysign-andgoal-trackingbehaviour.Interestingly,studiesinpigeonsshowedthatpreexposuretoCSalonetrialsbeforepairingwithaUSinterferedwiththedevelopmentofsign-tracking,butnotwithgoal-trackingresponsesdirectedatthesiteoffooddelivery(Boughner&Papini,2003).PreexposuretotheUSbeforepairingitwithaCSdelayedthedevelopmentofsign-tracking(Engberg,Hansen,Welker,&Thomas,1972),buthadnoeectongoal-trackingbehaviour(Costa&Boakes,2009).InSection1.4,theoreticalbackground,Iwilldiscusstheproposedmechanismsunderlyingsign-andgoal-tracking,whichprovidesomeinsightintowhysomemanipulationsaectonebehaviourmorethantheother.Morrison,Bamkole,andNicola(2015)changedthevalueoftheUSaftertheini-tialtraining(leverCS{sucroseUSpairings)bysubsequentlypairingtheUSwithillnessintheabsenceoftheCS(i.e.,conditionedtasteaversion).Thisprocedurere-sultedinenhanced,ratherthandiminished,sign-trackingbehaviour,butincontrast,areductioningoal-trackingbehaviour.Thisdissociationhascommonlybeenusedasevidencefordierentialmechanismsunderlyingthesebehaviours,onerelyingonastimulus-stimulusassociation(goal-tracking)andtheotheronastimulus-responseasso-ciation(sign-tracking).IntheTheoreticalBackground,section1.4,Iwillreturntothis17 issueatmorelength.TemporalparametersbetweentheCSonsetandosetandUSdeliveryhaveimpor-tanteectsonthestrengthofCRs.Ingeneral,theCRisstrongerthemorewidelyspacedthetrialsareandshorterthegapbetweenCSosetandUSdelivery(Hearst&Jenkins,1974).Withlongergapsbetweentrials(intertrialinterval,ITI)sign-trackingbehaviourisenhanced(e.g

.,Terrace,Gibbon,Farrell,&Baldock,1975)andwithshorterITIs,goal-trackingisenhanced(e.g.,Cinotti,Marchand,Roesch,Girard,&Khamassi,2019;Leeetal.,2018).IthasbeensuggestedthatvariationsinITIcouldproducethisdierencethroughaectingthe\predictiveness"or\informativeness"oftheCS(Hearst&Jenkins,1974).ShortITIscouldleadtotheCSnotbeingveryinformativeofwhenthereinforcerwillbedelivered.Incontrast,withlargeITIstheoccurrenceoftheCSisinformativefortheUSdelivery.CSdurationseemsnottoaectthegeneraltopographyoftheresponse,durationsof3and8secondswereequallyeective(Brown&Jenkins,1968).ThephysicalpropertiesoftheCS,aswellasthenatureoftheUSaecttheto-pographyoftheCRsevokedbytheCS.Furthermore,dierentspeciesdisplaydierentpatternofCRs(Shettleworth,1972).AlocalisedCSgenerallyevokesmoresign-trackingthatadiuseone,andtheresponsedependsonthetypeofCS,andhowitsupportssign-trackingbehaviour(alevercanbetouched,bitten,licked,howeveralightcanonlybepeckedororientedtowards,Leslie&Millenson,1996).ThelocationoftheCSinrelationwiththeUSdeliverysiteisalsoimportant.Generally,closespatialproximitybetweentheCSandtheUSdeliverysitesustainsmoresign-trackingbehaviour(Leslie&Millenson,1996).CRsevokedbyCSsdependalsoonthenatureoftheUS.JenkinsandMoore(1973)comparedfoodandwaterreinforcers.Mostpecksatwaterpredictivesignalswerechar-acterizedbyaveryslightopeningofthebeak,sometimeslickingandswallowingmove-ments,whichwereirregularlyspaced,sustainedandrelativelyweak.Incontrast,pecksatthefood-predictivesignalwerewiththebeakwide-open,evenlyspaced,strongand18 brief.RackhamHearstandJenkins(1974)observedthatiftheCSpredictstheoppor-tunitytomatein

malepigeons,sign-trackingbehaviourwascharacterisedbyapproach,nodding,cooing,circlingandbowing,typicalcourtshipbehaviour.Thereisstillmuchtolearnabouttheconditionsthatpromoteorreducesign-andgoal-trackingbehaviour.MostoftheexperimentsdescribedaboveassumethatanymeasurementofanyCRwouldleadtothesameconclusionaboutwhathasbeenlearnt(orthecurrentvalueofassociativestrength)inanygivenexperimentalprocedure.ThedierentpatternsobservedwhenmultiplemeasurementsofdierentCRshavebeentakenchallengethisassumption(e.g.,Patituccietal.,2016).InthefollowingsectionIfocusondierentapproachesthathaveattemptedtoprovideanexplanationofthedierentwaysinwhichlearningistranslatedintoconditionedbehaviour.1.4TheoreticalbackgroundGeneralprocessmodelsoflearning(e.g.,Mackintosh,1975;Pearce&Hall,1980;Rescorla&Wagner,1972;Wagner,1981)focusonlearningandhowassociativestrengthchangesundervariousexperimentalconditionsandnotonhowlearningisexpressed.Accordingtosuchmodels,thepredictiverelationshipbetweentheCS(e.g.,lever)andUS(e.g.,food)shouldresultintheformationofanassociationbetweentherepresen-tationsofthetwoallowingfuturepresentationsofthelevertoactivateamemoryofthefood.Thisanalysisallowstheexactformofconditionedresponsetodierfromthespecicnatureoftheresponsetofooditself(Wagner&Brandon,1989),butdoesnotprovideacoherentaccountforindividualdierencesinconditionedbehaviour.Thisisperhapsunsurprisinggiventhefactthatthesegeneral-processaccountswerenotin-tendedtoprovideacharacterizationofindividualdierences.However,wehaveseenthatdierentbehaviouralassessmentsoflearningleadtodierentconclusionsaboutlearning(e.g.,Patituccieta

l.,2016).Mosttheoreticalapproacheshaveignoredtheissueofhowlearningistranslatedintoperformance.However,therearesomegeneralprinciplesthatoeranexplanationforsomeofthechallengingresults.19 OneoftheproposedmechanismstoexplainindividualdierencesintheformPavlo-vianconditionedbehaviouriscentredonwhetheritisbasedonadirectassociationbetweentheCSandtheUS(stimulus-stimulusorS-Slearning)ortheCSisassociatedwiththeresponseelicitedbytheUS(stimulusresponseorS-Rlearning).Idiscusstheimplicationsofthisviewinthefollowingsection.1.4.1S-SvsS-RlearningFormaltheoriesofassociativelearningassumethatorganismsformassociationsbetweentherepresentationsofconditionedandunconditionedstimuli.Thesestim-ulus{stimulus(S-S)associationshavebeencontrastedwiththeformationofstimu-lus{response(S-R)associationsbetweentheprocessesactivatedbythestimulusandthemotorprogramforgeneratingaresponse(Hull,1943).Skinner(1948)suggestedthatsuperstitiousconditioningofresponsesdirectedto-wardstheCSmightexplainthedevelopmentandmaintenanceofsign-tracking.Onecouldarguethatitispossibleforpigeonsbeingexposedtoalightedkey,toperformasetofbehaviours(orienting,approaching,contacting)whichaccidentallyareassociatedwiththereinforcer,inanoperantconditioningfashion(S-R).Consistentwithview,Patituccietal.(2016)suggestedthatgoal-trackingcouldbebasedonanS-Slearningmechanismandsign-trackingonanS-Rmechanismsthatoperateinparallel,eachofthemcontrollingpartoftheCRrepertoire.Goal-trackingactivityseemstore ectthecurrentstatusoftherelationshipbetweentheleverandthereinforcer(i.e.,anS-Sassociation)asdevaluatingtheUSaectsrats'tendencytoenterthefood-wellmorethanlever-pressingbehaviour(M

orrisonetal.,2015).Moreover,Patituccietal.(2016)observedthatthebiastowardapproachingthefood-wellrelativetoleverpressingwaspositivelycorrelatedwiththepalatabilityofthereinforcer,andthatsatingrodentswiththereinforceraectsconditionedfood-wellactivitybutnotlever-orientedactivity.Evidencemotivatingtheviewofsign-trackingbehaviourre ectsaS-Rassociationcomesfromstudieswherecontingencychangesaectlesssign-trackingthan20 goal-tracking(e.g.,extinction,devaluation).Ratskeeppressingtheleverregardlessofthecurrentstatusoftherelationshipbetweentheleverandthereinforcer,viewedashabitualbehaviour(e.g.,Ahrens,Singer,Fitzpatrick,Morrow,&Robinson,2016;Morrisonetal.,2015).However,thestudyconductedbyAhrensetal.(2016)needstobetreatedwithsomecautionastheydidnotmeasurebothlever-orientedandfood-wellorientedbehavioursinratsclassiedasSTsandGTs.ThusitisunclearwhetherthedissociationineectsbetweenSTandGTanimalsistrulyadierencebetweenindividualsthemselvesorisinsteadadierenceinthespecicbehavioursmeasuredineachsetofanimals.IwillcomebacktothisissueinmoredetailintheReinforcementLearningsectionandinChapter2,asthisistheaimoftheexperimentspresentedinthischapter.Theviewofsign-trackingasastimulus-responseassociationischallengedhoweverbyaninterestingexperimentconductedbyBrowne(1976),whichsuggeststhatsign-trackingemergesevenwhenresponsestowardstheCSarenotreinforced.Inthisex-periment,pigeonscouldonlyobservetherelationshipbetweentheCS(keylight)andUS(grain)butcouldnotphysicallyinteractwiththeCSoraccessthegrainsastheywereseparatedfromtheCSandUSbyatransparentPlexiglasscreen.Pigeonswereexposedtoeitherpositive,negativeornocorrelat

ionbetweenthepresentationsoftheCSandUS.Whensubsequentlygivenaccesstoboththekeylightandgrainhopper,theanimalsinthepositivecorrelationgroupinteractedwiththeCSfarmorethanthenocorrelationgroup.WhilethebirdsinthenegativecorrelationgroupbarelyinteractedwiththeCS.Visualinspectionofthebirds'behaviourduringtheobservationphasedidnotsuggestevidenceofsign-trackingresponses.ResponsestowardstheCSwereneverreinforcedduringthelearning(observation)phaseandyettheywererapidlyexpressedwhengiventheopportunity.Browne'sinterpretationofhispatternofresultswasthatitseemedimplausibletosupposetheanimalshavelearnttorespondtothekeylightthroughaprocessofstimulus-responselearning.EvidencefromomissionproceduresfromWilliamsandWilliams(1969),whereany21 peckatthelightedkeywouldterminatethetrialandpreventscheduledgrains,alsosuggeststhatresponse{reinforcerrelationsdonotexplainoratleastarenotsucienttofullyexplainsign-trackingbehaviourtowardstheCS.Pigeonsstillpeckedatthelightedkey,albeittoalesserdegree,evenifthisbehaviourwasneverreinforced.Theseexperimentssuggestthatsign-trackingemergesastheanimalslearnthere-lationshipbetweenthecuesandthereinforcer,andsecondly,sign-trackingisdirectedtowardsthemostpredictivecues.Sign-trackingemergesevenifcontactresponsesareneverpairedwithareinforcer,suggestingthatstimulus-responselearningisnotsolelyresponsibleforthisbehaviour,oratleastnotfortheacquisitionstage.However,thefactthatoutcomedevaluationorextinction(e.g.,Ahrensetal.,2016;Morrisonetal.,2015)aectssign-trackinglessthangoal-trackingmightsuggestapossibleroleforstimulus-responseassociationinthemaintenanceofthebehaviour.Iwillpresentanalternativetheoreticalan

alysisfortheseandotherresultsinChapter3.Thistheoreticalframework(S-SvsS-R)addresseswhatitislearntduringPavlovianconditioning.However,itisnotabletoexplainwhysomeanimalsengagewiththesignandotherswithgoal.TheevidencesuggeststhatS-Slearningisnecessaryforacquisitionofsign-andgoal-tracking,buttheroleofS-Rlearningseemstobelessclearandcouldpotentiallycontributeatmaintainingsign-trackingbehaviour.However,itisuncleariftheseprocessesoperateinparallelor,ifthereisashift,whenitwouldoccur.Ifweassumethatgoal-trackingisbasedonS-Slearning,andsign-trackingonS-Rlearning(atleastduringthelaterstages),itwouldprovideageneralaccountforwhysign-trackingislesssensitivetochangethangoal-tracking,butstillleavesunexplainedtheoriginoftheseindividualdierences:whysomeanimalsengageinS-SandsomeinS-Rlearning.TheexperimentsthatIpresentinChapter2aimtodevelopabetterunderstandingofthedierentsensitivitytochangeofthesebehaviours,inSTsandGTs.1.4.2BehavioursystemstheoryTimberlake(1994)proposedthatananimalcomestoanylearningsituationwith22 anevolutionaryhistorythathasshapedpre-existinghierarchicallyorganizednetworkofbehaviour.Agivenmotivationalsystem(likethefeedingsystem)isformedofdif-ferentsub-systems(e.g.,predationorsocialforaging),modes(GeneralSearch,FocalSearch,andHandling/Consuming),andmorespecicstimulus-response(S-R)modules.Afeedingsystemwouldbeanexhaustivesetofbehavioursavailabletotheanimalthatgenerallyhavethefunctionofndingandconsumingfood.DuringPavloviancondi-tioning,theCRvariesaccordingofwhichmodegetsconditionedtotheCS.IftheCScomestoassociatewithageneralsearchmode,thenbehavioursthataredistanttothegoalobjectshouldbeconditioned.Howe

ver,iftheCSbecomesassociatedwithamorespecicfocalsearchmode,thenbehavioursthataremoreproximaltothegoalobjectshoulddominate.Inoneexperiment,Timberlake,Wahl,andKing(1982)pairedaballbearingCSwithfoodwitha6s(long)CS-USintervalandtheynoticedthatratslearnedtoapproach,contact,andcarrytheballbearing.However,whentheyuseda2sCS-USinterval(short),theratsavoidedcontactwiththeballbearinganddirectlyapproachedthefoodmagazinewhentheballbearingCSappeared.Inotherwords,itlookedasthoughamoregeneralsearchmodewasconditionedwiththelongCS-USintervalandafocalsearchmodewasconditionedwithashortCS-USinterval.ThereisthepossibilitythatduringagivenconditioningproceduredierentanimalsassociatetheCSwithdierentmodes(e.g.,aSTassociatestheCSwithageneralsearchmodeandaGTwithamorefocalsearchmode)whichthenleadtotheobserveddierencesinengagingwitheithertheCSorthegoallocation.Whythiswouldbethecaseremainsunclear.However,onepossibleexplanationcouldbetherelatedtodierenceinanimals'perceptionofthetimingbetweentheCSandUS.Butwhilethebehavioursystemsapproachhasaclearvalueintermsofananalysisofperformance,itleaveslearninglargelyaside.Itdoesnotprovidetestablepredictionsinrelationtowhattheanimalhaslearntandhowitistranslatedintotheactivationofaspecicsystem.Furthermore,italsofailstopredictwhatmodeananimalwouldbeingivenanexperimentalprocedureorwhatleadstotheactivationofthatspecicsystem.23 1.4.3ReinforcementlearningComputationalanalysesoflearninghavepaidsubstantialattentiontothedistinc-tionbetweenmodel-freeandmodel-basedformsoflearning.Model-basedstrategiesareheldtogenerategoal-directedchoicesemployingamodelorcognitive-stylerepresenta-tion

,whichisaninternalmapofeventsandstimulifromtheexternalworld(Dickinson&Balleine,2002a).Theinternalmodelsupportsprospectiveassessmentoftheconse-quencesoftakingparticularactions.Bycontrast,model-freestrategieshavenomodelofoutsideevents,butinsteadmerelylearnbycachinginformationabouttheutilitiesofoutcomesencounteredonpastinteractionswiththeenvironment.LearninginModel-Freesystemsreliesonacomputedreinforcementsignal,therewardpredictionerror(Lesaint,Sigaud,Flagel,Robinson,&Khamassi,2014).Thisleadstodirectrulesforhowtobehave,orpropensitiesforperformingparticularactions(Dayan&Berridge,2014).Lesaintetal.(2014)modelledindividualdierencesinPavlovianconditioningusingthisdual-processapproach.Theassumptionisthatratsusethetwolearningsys-tems,employingdistinctmechanismstolearnthesametask,withbothsystemspresentinanygivenanimalandthebalancebetweenthemisgivenbyaparameter(!).ForaST,thisbalancefavoursthemodel-freesystem,andforaGTitfavoursamodel-basedsystem.Thisapproachimpliesthatobserveddierencesbetweensign-trackingandgoal-trackingbehaviourareduetoanaturaltendencyoftheanimaltoengagewitheithertheCSorthelocationoftheUSorthegoal.STsarethoughttoattributetheCSwithincentivesalience,makingtheCSmoredesirable.Apotentialissuearisesfromthefactthatwhatdetermines!isunclear.Therewasnosuggestion!couldbepredictedbeforehand,resultinginasomethingofcircularargument:aSTisaSTbecauseitdisplayssign-trackingbehaviour(albeit,therearesuggestionsofotherindependently-assessabledierencesbetweensign-andgoal-trackinganimals,suchastheirimpulsivityorsensitivitytoconditionedreinforcement,e.g.,Lovic,Saunders,Yager,&Robinson,2011).Dopamineisthoughttobecent

ralforreward-relatedprocesses,buttheexactna-24 tureofitsroleremainscontroversial(Schultz,2006).Phasicneurotransmissioninthemesolimbicdopaminesystemisinitiallytriggeredbythereceiptofreward(US),butshiftstoacuethatpredictsareward(CS)afterassociativelearning(whentherewardpredictionerrorisreduced;Flageletal.,2011;Schultz,Dayan,&Montague,1997).Lesaintetal.(2014)examinedphasicdopaminereleaseinthenucleusaccumbens,andobserveddopaminewasreleasedduringtheacquisitionofasign-trackingbutnotgoal-trackingresponse.ApotentialissuewiththisinterpretationarisesfromthefactthatdopaminereleaseismeasuredinSTswhiletheyarelever-pressingbutnotwhentheyareenteringthefood-well,andviceversafortheGTs.Fromthisstudyisunclearwhetherthedopaminereleaseisrelatedtothepropensityoftheanimaltoengagewiththeleverorthefood-well,orofthebehaviouritself(lever-pressingorfood-wellentries).Inanycase,thelinkbetweendopaminereleaseandsign-tracking,togetherwiththeapparently`non-adaptive'natureofthisbehaviour(e.g.,resistancetoextinctionorengagingwithaCSevenifthispreventsthedeliveryofarelevantfoodrewardforahungryanimal,Ahrensetal.,2016;Williams&Williams,1969)hasledtotheproposalthatSTsrepresentapotentialmodelforaddictionorimpulsivity(Bissonetteetal.,2015;Lovic,Saunders,Yager,&Robinson,2011;Ostlund&Balleine,2008).ThisviewassumesthatcuesassociatedwithrewardsnotonlyevokeCRsbuttheycanalsobecomeattractiveanddesirableintheirownright,actingasincentivestimuli.Alongthesamelines,Lesaintetal.(2014)assumethatdierencesinsign-trackingandgoal-trackingbehaviourarebecauseofanaturaltendencyoftheanimaltoattributesomeCSswithincentivesalienceandmotivatebehaviour.Lovicetal.(2011)compared

STsontwotestsofso-calledimpulsiveaction(atwo-choiceserialreactiontimetaskandadierentialreinforcementoflowratesofrespondingtask)andonetestofimpulsivechoice(adelaydiscountingchoiceprocedure).Forthersttest,theyinitiallytrainedratswithalever,thatwasalsoilluminated,followedbythedeliveryofonepellet.Subsequently,theydividedtheratsinSTsandGTsusing25 anindexwhichindicatedthepropensitytoapproacheithertheleverorthefood-well(seeMeyeretal.,2012).Forthesubsequentphasesoftheexperiment,theyonlyusedanimalsthatwouldpredominantlysign-orgoal-track.Duringthetwo-choiceserialre-actiontesttheypresentedratswithtwoilluminatednose-pokeports,ofwhichonlyonewasfollowedbyareinforcer.Responsesintotheilluminatednose-pokeportduringthetrialterminatedthetrialandresultedinthedeliveryofafoodpellet(forthecorrectport).Iftheanimalmadenoresponses,thenose-pokeportstayedilluminatedfor5sandwasfollowedbythereward.TheyfoundthatSTsmademoreprematureresponsesthanGTs.AproblemwiththisexperimentarisesfromtheshiftfromPavloviancon-ditioningtooperantconditioning.Intherststage,theleverwasreinforcedregardlessoftheanimal'sresponse,butinthesecond,anose-pokeintheport(aresponse)isnowreinforced.GTsalreadyhadapredominantbehaviourofengagingwiththefood-well,whiletheCSfortheSTshaschangedfromalevertonose-pokeport.Itisunclearhowthesechangesmighthaveaectedthegeneraltendencytointeractwiththefood-wellandtheresultsshouldbetobetreatedwithcaution.Forthesecondtestofimpulsiveaction,anothergroupofratswentthroughthesametrainingprocedure,afterwhichtheywereplacedonanFR1(xedratioof1response)operanttask,untiltheyreached100responses.Subsequently,theyweretestedonDRL-10s(d

ierentialreinforcementoflowratesofrespondingtask)schedulefor5daysandDRL-20sschedulefor15days.OntheDRL-10sandDRL-20sschedules,ratswerereinforcedonlyifatleast10sor20s,respectively,elapsedbetweenresponses.TheyfoundthatSTswerelessecientinthetaskthanGTs.Again,previoussign-trackingbehaviourintermsofleverinteractionmighthaveaectedtheresults,giventhetendencyinthetrainingphasetoengagewiththelever.Fortheirdelaydiscountingchoiceprocedure,theyfoundnodierencesbetweensign-andgoal-trakinggroups.Onthebasisofthispatternofresults,theauthorsconcludethatSTsaremore`impulsive'thanGTs,therearehowever,alternativeinterpretations.Ahrens,Singer,Fitzpatrick,Morrow,andRobinson(2016)trainedratswithalever26 CSfollowedbyareinforcer.Thentheyintroducedperiodsofnon-reinforcement(ex-tinction)oftheCSandnoticedthatSTsextinguishedleverpressesmoreslowlythanGTsextinguishedfood-wellentries.TheyconcludedthatSTsareless exibleintheirbehaviour.ThisconclusionwasbasedonleverpressesdataforSTsandfood-wellentriesdatafortheGTs.Asnotedbefore,thefactthatSTandGTgroupswereonlyassessedintermsoftheirpredominantbehaviourmeansthatthedierencein exibilitycouldbeattributedeithertothebehaviour(leverpressorfood-wellentry)ortheanimalsthemselves(STsorGTs).Thustofullysupporttheclaimthatthisre ectdierencesintheanimals,itwouldbenecessaryforbothbehaviours(leverpressesandfood-wellentries)tobeassessedinanimalsclassiedasSTsandGTs.Itisunclearwhetherslowerextinctionisapropertyoftheanimalorthebehaviouritself,whichwillbediscussedatlengthinChapter2.Inanotherexperimentwhichinvestigatedthelinkbetweensign-trackingandaddic-tion,Pena-Oliveretal.(2015)examinedwh

etherselectivelybredalcohol-preferringandalcohol-non-preferringratsshowdierentiallevelsofimpulsivityandconditionedbe-haviouralresponsestofoodincentives.Interestingly,theyfoundthatalcohol-preferringratsshowedmoregoal-orientedbehaviourthancontrolsandnoevidenceofimpulsivebehaviourwasfound.Partoftheargumentlinkingsign-trackingandaddictionisbasedonthefactthatFlageletal.(2011)observedthatratsclassiedaseitherSTsorGTsdierindopaminereleaseduringlearning.InSTstheyobservedanincreaseinCS-evokeddopamineandadecreaseinUS-evokeddopaminerelease.ThispatternwasnotobservedforGTswhenassessingtheirfood-wellentriesbehaviour.However,theyonlyexaminedSTswhiletheyleverpressedandGTswhiletheywereinteractingwiththefood-well.Itwouldbeinterestingtoassesswhetheranincreaseisobservedinsign-trackinganimalswhentheygoal-track,asthiswouldpointtoadenitiveoriginofthesedierencesintheanimal,notthebehaviourperse.But,aswillbeshowninChapter2,thisissuecanalsobeaddressedatabehaviourallevel.27 Thelinkbetweendopamine,addictionandsign-trackingisgenerallyintermsofsimilaritiesofcertaintypesofbehavioursortraitsassumedtobecharacteristicofad-dictedindividuals(e.g.,impulsivity,in exibility).Itisgenerallybasedontheideathatdopamineisinvolvedinreward-processesandthefactthatindividualsaddictedtoharmfulsubstanceshaveincreasedconcentrationofdopaminereleaseinlimbicre-gions(Volkow,Fowler,Wang,&Swanson,2004).However,thelinkwithsign-trackingbehaviourissomewhatunclearandbasedonexperimentsthatdonotassessbothbe-haviours(sign-andgoal-tracking)inanimalsclassiedasSTsorGTs.Whetherthereisalinkornotbetweenaddictionandsign-trackingbehaviourisnotthecentralissu

ehere,andIpresentedthisapproachbecauseitisrelevantforunder-standingindividualdierencesinconditionedresponses.Theassumptionforthislineofargumentisthatgeneticindividualdierencesareresponsiblefortherat'stendencytosignorgoal-track.AstudybyPatituccietal.(2016)challengesthisview.Theypre-sentedtwolevers,onereinforcedwithfoodandanotheronesucrose.Theyfoundthattherat'sbiastowardssign-orgoal-trackingbehaviourdidnotcorrelateacrossthesetwolevers.Inotherwords,itwaspossibleforarattobeaSTononeleverbutnottheother.Dierencesinthebalancebetweensign-andgoal-trackingbehaviouracrosssituations,appearstobeinconsistentwiththerebeingageneraltendencytoonetypeofbehaviourorother,andhencetheideaof`anaddictivepersonality'asrevealedinageneralpropensitytosign-trackingacrosssituationsappearsdiculttosustain.Therearesimilaritiesbetweentheanalysesbasedonthedivisionbetweenmodel-basedandmodel-freeprocessesononehand,andS-SandS-Rprocessesontheother.Model-basedisbasedonthecurrentvalueoftheCS-USrelationshipanditisagoal-directedaction(Beierholm,Anen,Quartz,&Bossaerts,2011).Modelfree,incontrast,isheldtobeafastsystembasedongeneralresponsesorhabitsperformedinanygivensituation.InthesamewayastheS-SandS-Rframework,wecouldassumebehaviourdirectedtowardthefood-wellandleveraregeneratedbyindependentsystems(model-basedormodel-free)thatoperateinparallel.Thisanalysispredictsthatagivenformof28 responsewillexhibitthesamecharacteristicsinaSTsandGTs(e.g.,dierentsensitivitytochangebeingre ectedinthebehaviour).However,insteadofbothsystemsoperatinginparallel,themodelproposedbyLesaintetal.(2014)assumesthatthebehaviourofagivenrodentisgovernedpredominantly

bytheoperationofasinglesystem(eithermodel-basedormodel-free),whichcontrolsbothtypesofbehaviours.Thisaccountpredictsthatsign-andgoal-trackingbehaviourswillexhibitdierentpropertiesinSTsandGTs.InChapter2IinvestigatethesepredictionsbyexaminingbothbehavioursinSTsandGTsbychangingthecontingencybetweentheCSandUSorthevalueoftheUS.Insummary,thefewattemptstoexplainindividualdierencesinconditionedre-sponses,ormorespecicallysign-andgoaltrackingbehaviour,haveappealedtoadualmechanismwhichunderpineachofthetwobehaviours.Thereisevidenceconsistentwithadual-systemaccountwhereactionsarecontrolledbyeitherafasthabitualsys-temoraslowermoregoal-directedsystem,basedoncurrentassociationsbetweenstimuli(Beierholmetal.,2011).However,neitherapproachprovidesclearpredictionsofhoweachtypeofbehaviour(sign-orgoal-tracking)willbeaectedbyacertainexperimen-talmanipulation.Moreover,giventhecontinuousnatureofthebalancebetweenthetendencytosign-orgoaltrack(e.g.,Patituccietal.,2016),adual-mechanismaccountwouldneedtoaddresstheinteractionbetweenthetwoprocessesinagivenratorproce-dure.Thisissuehasnotbeendealtwithbyanyofthedual-processesframeworks.Thatsaid,generallearningtheories,whichrelyonasingle-process(e.g.,Mackintosh,1975;Pearce&Hall,1980;Rescorla&Wagner,1972;Wagner,1981),areunabletopredictdierencesinresponding,asperformanceisviewedmerelyasatranslationofassociativestrengthbasedonlearningthecontingencybetweentheCSandUS.1.5ThesisOutlineThisthesisisconcernedwithbetterunderstandingofthenatureofindividualdif-ferencesinPavlovianconditioningandbehaviour.Ipresentnewempiricalevidencein29 Chapter2andChapter4.InChapter2,Iexaminethedieren

tialsensitivitytocontin-gencychangesinbothSTsandGTs,measuringbothsign-andgoal-trackingbehaviourinbothgroups.Thisevidencecontributedtothedevelopmentofatheoreticalmodel,HeiDI,presentedinChapter3.Themodeldistancesitselffromadualprocessapproachinexplainingperformanceandfromasingle-processperspectivedierencesand,atthesametime,leadstonovelpredictions.InChapter4,Iexaminetwopredictions.OnepredictionrelatestohowchangesinthevalueoftheUSaectthedistributionofsign-andgoaltracking.Thesecondpredictionconcernsananalysisofthefeaturepositiveeect,wherethediscriminationemergesmorereadilyforafeaturepositivedesign,wherethecompoundisreinforced(AB+)andtheelementisnon-reinforced(B-),incontrastwithafeaturenegativedesign,wherethecompoundisnon-reinforced(AB-)andtheelementisreinforced(B+).30 Chapter2Investigatingthenatureofphenotypicvari-ation2.1IntroductionPavlovianconditioningisperhapsthemostwell-knownpsychologicalphenomenon,anditstheoreticalimportancewasevidentfromthepointofitsinitialdescription.Theuseofthisparadigmiswidespread,particularlyacrosstheeldsofbehaviouralandcognitiveneuroscience(forarecentreview,seeMurphy&Honey,2016)andbehaviouralgenetics(e.g.,Duvarci,Nader,&LeDoux,2008;Lonsdorfetal.,2009).Intheeldofbehaviouralneuroscience,oneofitsprincipaluseshasbeeninprovidingatestbedforformaltheoriesofassociativelearning,whichassumethatorganismsformassociationsbetweentherepresentationsofconditionedandunconditionedstimuli(e.g.,Mackintosh,1975;Pearce,1994;Pearce&Hall,1980;Rescorla&Wagner,1972;Wagner,1981).Thesestimulus{stimulus(S-S)associationshavebeencontrastedwiththeformationofstimulus{response(S-R)associationsbetwee

ntheprocessesactivatedbythestimulusandthemotorprogramforgeneratingaresponse(Hull,1943;Spence,1936,1937).Theideathattwo(associative)systemsmightunderpinconditionedbehaviourhasclearcounterpartsincognitiveneuroscienceasdiscussedinChapter1(Reinforcementlearningsection)(e.g.,Daw,Gershman,Seymour,Dayan,&Dolan,2011;Dayan&Berridge,2014).LikePavlov,theformaltheoriesofassociativelearningidentiedaboveappealtotheideathatthememoryorrepresentationofonestimuluscancometoexcite(ortoinhibit)therepresentationofanotherstimulusthroughanexcitatory(orinhibitory)associationformedbetweenthem.UnlikePavlov,however,suchtheorieshaveeschewedconsiderationofindividualdierencesinPavlovianconditioning,apartfrominsofarastheyrepresentonesourceofvarianceinbehaviouralmeasuresoflearning(seeMatzelet31 al.,2003).Theviewthattherearemarkedindividualdierencesinsimpleconditioninghasbeenamplyconrmedinmorerecentexperimentswithrodents,wherethedierencesareperhapsmorestrikingandwellcharacterized.Thesedierencestoohavepotentialtranslationalsignicance(seeFlageletal.,2009;Lovicetal.,2011).AsdiscussedinChapter1(Individualdierencessection),criticalbehaviouralob-servationscomefromsimpleautoshapingprocedures.Inaprocedurewhereratsreceivebriefpresentationsofaleverthatarepairedwiththedeliveryofareinforcerintoafood-well(e.g.,Patituccietal.,2016),markedindividualdierencesareobservedinbehaviour:someratspredominantlyinteractwiththeleverwhileothersapproachthefood-wellduringtheleverpresentations.Thebasisforthesedierentphenotypesisthecentralissuethatisaddressedhere.Oneanalysisofindividualdierencesinsign-andgoal-trackingbehavi

ourcanbede-rivedfromtheassumptionthatthetypesofassociativestructuresdescribedabove(i.e.,S-SorS-R)mightbedierentlyrepresentedacrossindividuals(seeLesaintetal.,2014;Patituccietal.,2016).Thegeneralideathatacquiredbehavioursmightbetheproductofdierentsystemswithdistinctcharacteristicshasaclearprecedentinthecontextofstudiesofinstrumentalconditioning(e.g.,Dickinson&Balleine,2002b),andtherearetwosourcesofevidencethatareconsistentwithitfromstudiesofphenotypicdierencesinPavlovianconditionedresponding.First,food-wellactivityinratsclassiedasgoal-trackers(GTs)declinesmorerapidlyduringanextinctionprocedurethandoesleverpressinginratsclassiedassign-trackers(STs;Ahrensetal.,2016).Theseobservationssuggestthatfood-wellactivityre ectsthecurrentstatusoftherelationshipbetweentheleverandthereinforcer(i.e.,anS-Sassociation),whereaslever-orientedbehaviourwasbasedonaS-Rhabitthatwasmoreresistanttochangesincontingencies.Second,thebiastowardapproachingthefood-wellrelativetoleverpressingispositivelycorre-latedwiththepalatabilityofthereinforcer(Patituccietal.,2016),andsatingrodentswiththereinforceraectsconditionedfood-wellactivitybutnotlever-orientedactivity(Morrisonetal.,2015;Patituccietal.,2016).32 Theresultsdescribedintheprecedingparagraphareconsistentwiththegeneralideathattherearetwolearningsystemsthatoperatedierentlyacrossrats;butthereareatleasttwoformsthatthisanalysiscouldtake.Forexample,whilefood-wellbehaviourmightbethedominantresponsegeneratedbytheS-Ssystemandlever-orientedbe-haviourthedominantresponsegeneratedbytheS-Rsystem,bothsystemsmighthavethecapacitytogeneratebothresponses(seeLesaintetal.,2014).

Ifasinglesystemgov-ernedallbehaviourinagivenratthenbothfood-wellandlever-pressresponsesshouldexhibitthecharacteristicpropertyofthatsystem:WhengovernedbyanS-Ssystem,activitydirectedtowardboththeleverandthefood-wellwillchangerapidlyinthefaceofachangeincontingencies;whereaswhengovernedbyanS-Rsystembothwillchangerelativelyslowly.Inprinciple,theaccuracyofthispredictioncouldhavebeenassessedbyAhrensetal.(2016),but,theyonlypresentedactivitydirectedtowardthefood-wellforGTsandtowardtheleverforSTs.Itisnot,therefore,possibletoassesswhetherthetwoformsofresponsewereaecteddierentlyinratsclassiedasGTsorSTs.Thesingle-systemanalysisjustoutlinedis,however,challengedbythefollowingobservation:AgivenrodentcanbeclassiedasaGT(orST)withrespecttotheirbehaviouronaleverthatpredictsonereinforcer(e.g.,foodpellets),butnotclassiedinthesamewayonanotherleverthatpredictsadierentreinforcer(e.g.,sucrose;Patituccietal.,2016,Experiment1).Ifasingle-system(S-SorS-R)governedbehaviourinagivenan-imalthenthepatternsofbehaviourshouldbeconsistentacrossdierentmanipulanda(i.e.,theleftandrightlevers).Asimplealternativetotheanalysisdescribedinthepreviousparagraphassumesthatbehavioursdirectedtowardthefood-wellandleveraregeneratedbyindependentsystems(S-SandS-R,respectively)thatoperateinparallel.ThisanalysispredictsthatagivenformofresponsewillexhibitthesamecharacteristicsindependentlyofwhethertheanimalinwhichitisobservedisclassiedasaSToraGT;withfood-wellactivitybeingderivedfromtheoperationofaS-Ssystemandlever-orientedbehaviourbeingderivedfromaS-Rsystemthatoperatetodierentdegreesinallrodents.Thedominantresponsemightbetowar

dthefood-wellinonerodentand33 leverinanother,butinbothratsfood-wellactivityshouldmorerapidlytrackchangesinreinforcementcontingenciesthanshouldlever-orientedactivity.Asalreadynoted,thispredictionwasnotassessedbyAhrensetal.(2016),butPatituccietal.(2016,Experi-ment2)reportedthatsatiationhadamarkedeectonfood-wellactivitywhentheeectofthismanipulationwasconsideredacrossratsthathadbeenclassiedasGTsorSTs.Thisobservationisconsistentwiththeideathatactivitydirectedtowardthefood-wellandleverhavethesamepropertiesirrespectiveofwhethertheywereexhibitedinSTsorGTs.Tosummarize,tothebestofmyknowledgenobodyhasdirectlyinvestigatedthefollowingsimplequestion:Doesagiventypeofbehaviour(e.g.,leveroriented)havethesameordierentcharacteristicswhenassessedinSTsandGTs?Here,Iaddressedthisquestionintwoexperiments.Inbothexperiments,ratsreceivedtrainingproce-duresthatshouldallowthetwophenotypestodevelop,andthenthecontingencieswerechanged(e.g.,thereinforcedleverbecamenon-reinforcedandviceversa,orreinforcerischangedfromamoredesirableonetoalessdesirableoneandviceversa).Thechangesinbehavioursdirectedtowardtheleverandfood-wellwerethenassessedasafunctionofwhethertherodentshadbeenclassiedasSTsorGTsattheendoftherststageoftraining.EvidencefavouringtheclaimthattheS-Ssystemgeneratesfood-wellac-tivityandtheS-Rsystemgenerateslever-orientedbehaviourwouldtaketheformofacompellingdissociation:Morerapidchangesinfood-wellactivitythaninlever-orientedbehaviourinbothST>groups,despitethequitedierentlevelsofperformanceanticipatedinratsclassiedaspredominantlygoal-trackingorsign-tracking.34 2.2Experiment12.2.1MethodAnimalsandappara

tusSixteenfemaleSpragueDawleyratswereused(suppliedbyCharlesRiver,UK).Theyhadbeensubjectsinabehaviouraltaskinvolvingdrinkingdierentconcentrationsofsucrose,butwerenavewithrespecttotheapparatusandproceduresusedinExperiment11.Theirmeanadlibitumweightbeforethestartoftheexperimentwas321g(range:280-366g)andtheyweremaintainedatbetween85and95%oftheseweightsbygivingthemrestrictedaccesstofoodattheendofeachday.Theanimalswerehousedingroupsrangingfromtwotofourinstandardcageswithenvironmentalenrichment(smallwood,paperandcardboardtubes)andmaintainedon12-hr/12-hrlight/darkcycle(lightsonat7a.m.).TheresearchwasconductedinaccordancewithHomeOceregulationsundertheAnimal(ScienticProcedures)Act1986(PPL303243,PIDominicM.Dwyer).Theapparatusconsistedofeightidenticalconditioningboxesmeasuring30Ö24Ö21cm(HÖWÖD;MedAssociates,Georgia,VT).Eachboxwasplacedinasound-attenuatingshellthatincorporatedaventilationfan,whichmaintainedthebackgroundnoiseat68dB(A).Theboxeshadaluminumsidewallsandclearacrylicfrontbackandtop.The oorwasconstructedfrom19steelrods(4.8mmdiameter,16mmapart)andwassituatedaboveastainlesssteeltray.Foodpellets(45mg:suppliedbyMLab:Richmond,IN)weredeliveredtoa oor-levelrecessedfood-well(aperture:5.3Ö5.3cm)inthecentreoftheleftwall.Thefood-wellwasequippedwithinfrareddetectorsthatallowedthepresenceoftheratinthewelltobeautomaticallyrecorded.Asingleresponsewasregisteredwhenthedetectorwasinterrupted(e.g.,whenarat'ssnoutenteredthefood-well).Tworetractablelevers(4.5Ö1.8Ö0.2cm)werelocated3cmtotheleftandright 1Experiment1wasreplicatedaspartofalargerstudyinwhichtheratswerefromthesamesourceasExperiment1,butweremalerathert

hanfemale.TheresultsfromthisreplicationmatchedthoseofExperiment1.Experiment2alsousedmaleratstoconrmthegeneralityoftheresultsfromExperiment1.35 ofthefood-wellandataheightof4.6cmand1.5cmfromtheedgeofthewall.Aleverpresswasrecordedeachoccasionthattheleverwasdepressedby4mmfromitsusualhorizontalrestingposition.MED-PCsoftwarewasusedtoinsertlevers,deliverfoodpellets,andtorecordfood-wellentriesandleverpresses.ProcedureTheratshadtwo24-minpre-trainingsessionswherefoodpelletsweredeliveredonavariable-time(VT)60-sschedule(range:40{80s).Ratsthenreceivedasinglesessionoftrainingoneachofthenext12daysoftraining,whichoccurredatthesametimeofdayforagivenrat(seedesigninTable1).Thesesessionsconsistedof20trialsonwhichtheleftleverwasinsertedfor10sandthenretractedand20trialsonwhichtherightleverwasinsertedfor10sandwasthenwithdrawn.Forhalfoftherats,thereinforcedlever(L1)wastheleftleverandthenon-reinforcedlever(L2)wastherightlever;andfortheotherhalfL1wastherightleverandL2theleftlever.Theorderinwhichtheleftandrightleverswerepresentedwasrandomwiththeconstraintthattherecouldnotbemorethanthreesametypetrialsinsuccession.Ratswereassignedtothegroupsrandomly.Thetrialsweredeliveredonavariable-time(VT)60-sschedule(range:40{80s).Allratsthenreceivedreversaltrainingfor12daysinwhichL1(e.g.,leftlever)wasnon-reinforcedandL2(e.g.,rightlever)wasreinforced.L1denotestheleverthatwasreinforcedduringtrainingandnon-reinforcedduringreversalandL2theleverthatwasnon-reinforcedduringtrainingandreinforcedduringreversal.Theprocedureusedforthereversalstagewasinotherrespectsidenticaltothetrainingstage.DataAnalysisFormanipulatingthedata,datacleaning,statisticalanalysisan

ddatavisualization,IusedopensourcesoftwareR(RStudio,2015)andJASP(JASPTeam,2018).Fordatacleaningandmanipulation,Iused\tidyr"(Wickham&Henry,2018)and\dplyr"36 Table1:DesignoftheExperiment1 ClassicationTrainingReversalNotation STL1+L1-L1(+k-)L2-L2+L2(-k+) GTL1+L1-L1(+k-)L2-L2+L2(-k+) Note:STreferstoasign-trackerratandGTreferstoagoal-trackerrat.L1andL2refertotwolevers(leftandright,counterbalanced).Duringtraining,L1wasreinforced(\+";foodpellet)andL2wasnon-reinforced(\-").DuringthereversalstageL1wasnotreinforcedandL2wasreinforced(foodpellet).RatswereclassiedasSTorGTonthebasisoftheirbiastowardsleverpressingorenteringthefood-wellduringthenalblockoftrainingonthereinforcedlever.Rpackages(Wickham,Henry,&Muller,2019).Forstandardhypothesistesting,Iused\ez"Rpackage(Lawrence,2016)andforBayesianstatisticsIusedJASP(JASPTeam,2018).Fordatavisualizationandrepresentation,Iused\ggplot2"(Wickhametal.,2018)Rpackage,and\gridExtra"Rpackages(Auguie&Anotonov,2017).TheanalysisareautomaticallyreportedintextinAPAformatstyleusing\apa"Rpackage(Gromer,2019).Foranalysisofvariance(ANOVA)whenthesphericityassumptionwasviolated,theanalysiswasreportedwithGreenhouse-Geissercorrection.Forpost-hocunplannedcomparisonsBonferronicorrectionwasapplied(thethresholdfordeclaringap-valuesignicantwasequalto0.05dividedbythenumberofcomparisons).Standardhypothesistestingdoesnotdirectlyassesswhethertheabsenceofasigni-canteectissucientevidencetoconcludethatthereisnoeect.Incontrast,Bayesianstatisticsprovidesaratiooftheprobabilityfortheobserveddataunderdierentmod-els,suchasamodelbasedonthenullhypothesisrelativetoamodelbasedonsom

especiedalternative.TheresultingBayesfactorscanthenbeinterpretedaccordingtotheconventionsuggestedbyRouder,Speckman,Sun,Morey,andIverson(2009),whereaBayesfactorbetween1and3providesanecdotalsupport,afactorbetween3and10suggestssomesupportingevidence,whileafactorbeyond10indicatesstrongevidence.Ihave,therefore,supplementedstandardnull-hypothesisstatisticaltestingwiththepre-37 sentationofequivalentBayesfactors,whennullresultsareoftheoreticalsignicance.BayesiananalysiswasconductedwithBayesfactorsformaineectsandinteractionsforfactorialANOVAinthewaydescribedbyRouder,Morey,Speckman,andProvince(2012)andRouder,Morey,Verhagen,Swagman,andWagenmakers(2017).Successivesessionsduringthetrainingandreversalstageswerecombinedinto12Ö2-dayblocks(6Ötraining:T1-T6;and6Öreversal:R1-R6).Attheendofthetrainingphase,theratsweresplitintotwogroups,sign-trackers(ST)andgoal-trackers(GT),basedontheirtendencytoengagewiththeleverandthefood-well.Abiasscorewascalculatedusingthenumberofleverpressesandfood-wellentriesforthereinforcedlever,L1:(Goal-tracking-Sign-tracking)/(Goal-tracking+Sign-tracking).Individualscoresforeachratwascalculatedbasedontherawleverpressesandfood-wellentriesaveragedacrosslasttwodaysoftraining(T6).Atthisstagebothbehavioursareremarkablystableandhavereachedasymptote.Amediansplitwasusedtodivideratsintothosewithhigherscores(groupGTforgoal-trackers)andthosewithlowerscores(groupSTforsign-trackers).Ascorecloseto1indicatesatendencytoengagesolelywiththefood-wellandascorecloseto-1atendencytoengagesolelywiththelever.Withafairlyevendistributionthemediansplitcriterionisaround0(scoresabove0areclassiedasGTandscoresbelow0a

reclassiedasST).ThedistributionofbehaviourisshowninFigure1.Biasscoresabove-.16wereclassiedasGTsandbelow-.09asSTs.Subsequentanalyseswereconductedseparatelyforleverpressesandfood-wellen-tries,withthemainfocusbeingonthetransitionbetweenthenalblockoftraining(T6)andtherstblockofreversal(R1).Ialsoconductedcomplementaryanalysesinwhichthenumberofleverpressesandfood-wellentriesweretreatedinacontinuousfashion.MixedANOVAswereconductedseparatelyforleverpressingandfood-wellentrieswithwithin-subjectsfactorsblock(training6levels:T1-T6;transitionblocks2levels:T6-R1;reversal6levels:R1-R6;)andlever(2levels:reinforced\+"vsnon-reinforced\-")andbetween-subjectsfactorclassication(2levels:STvsGT).38 Figure1:Distributionofsign-tracking(leverpresses)andgoal-tracking(food-wellentries)behaviourforExperiment1per(10-s)trialduringlastblockoftraining(T6)forL1(+jj-).Theblacksymbolscorrespondtosign-trackers(ST)andtheclearsymbolstogoal-trackers(GT).2.2.2ResultsThemainresultsfromExperiment1areshowninFigure2andinFigure3.Theanalysiswillbeginwithresultsfromthetrainingstage(left-handpanelsofFigure2),beforemovingtothecriticaltransitionbetweentrainingandreversal(identiedbythegreysection),andnallythereversalstageasawhole(right-handpanels).TrainingInspectionoftheresultsfromtherststageoftraining(left-handsideoftheupperandlowerpanelsofFigure2)suggeststhatastrainingprogressedratsinbothgroups(STandGT)showedmoreleverpressesandfood-wellentriesduringthereinforced39 Figure2:ResultsfromExperiment1:theeectsofareversalprocedureonsign-trackingandgoal-tracking.Mean(+SEM)leverpresses(upperpanel)andfood-wellentries(lowerpanel

)per(10-s)trialacrossthetwostages:training(T1{T6)andreversal(R1{R6).Duringtraining,ratsreceivedpresentationsofoneleverpairedwithfoodpellets(L1(+jj-))andnonreinforcedpresentationsofasecondlever(L2(-jj+));ratswereclassiedassign-trackers(STs)andgoal-trackers(GTs)onthebasisoftheirbehaviourduringthenalblockoftraining(T6).Theythenreceivedareversal:L1non-reinforcedandL2reinforced.Thegreysectionindicatestransitionbetweeninitialtrainingandthereversalofthecontingencies.L1thanthenonreinforcedL2.Thefactthatduringtheinitialtrainingsessionstherewasahigherleveloffood-wellentriesthanleverpressesprobablyre ectstheimpactofthepretrainingsessionsinwhichfoodpelletsweredeliveredintothefood-well.Inanyevent,thediscriminationinvolvingleverpresseswasmoreevidentinGroupSTthanGroupGT,whilethediscriminationinvolvingfood-wellentrieswasmoreevidentinGroupGTthanGroupST,withthesebetween-groupsdierencesbeingmostapparentonreinforcedL1trials.Thedescriptionofthetrainingresultsissupportedbyseparateanalysesofleverpressesandfood-wellentries.40 Figure3:CorrelationresultsfromExperiment1:theeectsofareversalprocedureonsign-trackingandgoal-tracking.Theupperpanelsshowtherelationshipbetweenthemeannumberofresponsesper(10-s)trialbetweenblocksT5andT6forleverpresses(A)andforfood-wellentries(B)onreinforcedtrials.ThelowerpanelsshowtherelationshipT6andR1forleverpresses(C)andfood-wellentries(D)onreinforcedtrials.Theblacksymbolscorrespondtosign-trackers(ST)andtheclearsymbolstogoal-trackers(GT).LeverpressesAnANOVAconductedforleverpressesrevealedmaineectsofclassication,F(1,14)=6.36,p=.024,2p=.31,block,F(2.96,41.41)=9.74,p.001,2p=.41

andlever,F(1,14)=40.06,p.001,2p=.74,aswellasinteractionsbetweenclas-sicationandblock,F(2.96,41.41)=2.15,p=.109,2p=.13,classicationandleverF(1,14)=6.88,p=.020,2p=.33,andblockandleverF(2.60,36.35)=18.75,p.001,2p=.57.Therewasatripleinteractionbetweenclassication,blockandlever,F(2.60,36.35)=3.42,p=.033,2p=.20.41 Food-wellentriesAparallelanalysiswasconductedforfood-well-entriesandrevealedmaineectsofclassicationF(1,14)=12.19,p=.004,2p=.47,andlever,F(1,14)=37.82,p.001,2p=.73,butnotablockeect,F(2.68,37.49)=11.98,p.001,2p=.46.Therewereinteractionsbetweenclassicationandlever,F(1,14)=12.90,p=.003,2p=.48,classicationandblock,F(2.68,37.49)=7.30,p.001,2p=.34andbetweenblockandleverF(2.83,39.69)=7.40,p.001,2p=.35.Therewasatripleinteractionbetweenclassication,blockandlever,F(2.83,39.69)=10.33,p.001,2p=.42.TransitionblocksTheresultsfromthetransitionbetweenthenalblockoftrainingandrstblockofreversal(graysectionofFigure2)areofcentralinterest.Inspectionofthistransi-tionhighlightsthefactthatleverpressesremainedstableinspiteofthereversedrein-forcementcontingencies(upperpanel),whereasfood-wellentrieschangedrapidly(lowerpanel).Moreover,thesedierencesbetweentheeectsofthereversalonleverpressesandfood-wellentrieswereevidentinbothGroupsSTandGT:Thelevelsofleverpress-ingremainedlargelyunchangedinbothgroups;andwhiletherewasamarkeddecreaseinfood-wellentriestothepreviouslyreinforcedleverinGroupGTthereweremarkedincreasesinfood-wellentriestothepreviouslynonreinforcedleverinbothGroupGTandST.Also,inGroupSTtherewasamoremarkedincr

easeinrespondingbetweentheT6andR1forL2thanL1.LeverpressesAnANOVAconductedforleverpressesonthecriticalreversalblocksrevealedmaineectsofclassication,F(1,14)=7.71,p=.015,2p=.36andlever,F(1,14)=55.27,p.001,2p=.80,butnoeectofblock,F(1,14)=0.82,p=.380,2p=.06.Therewasaninteractionbetweenclassicationandlever,F(1,14)=6.90,p=.020,2p=.33,butnointeractionbetweenclassicationandblockF(1,14)=0.07,p=.800,2p.01,andcriticallynointeractionbetweenblockandlever,F(1,14)=1.68,p=.216,2p=.11.42 Therewasnotripleinteractionbetweenclassication,blockandleverF(1,14)=0.13,p=.728,2p.01.TheBayesfactorforthebestmodelwithouttheinteractionbetweenblockandleverrelativetothemodelwiththeinteractionwas9.52,whichindicatesevidenceagainstthepresenceoftheinteraction.TheBayesfactorforthebestmodelwithouttheinteractionbetweenblock,leverandgrouprelativetothemodelwiththeinteractionwas62.50,whichrepresentsstrongevidenceagainstthepresenceoftheinteraction.ThisanalysisindicatesthattherewerenoimmediateeectsofthereversalonleverpressesineitherofgroupsSTandGT.Food-wellentriesAparallelanalysisoffood-wellentriesforthecriticaltransitionblocksrevealedaneectofclassication,F(1,14)=18.90,p.001,2p=.57,blockF(1,14)=12.04,p=.004,2p=.46andlever,F(1,14)=11.88,p=.004,2p=.46.Therewereinteractionsbetweenclassicationandblock,F(1,14)=9.68,p=.008,2p=.41,classicationandlever,F(1,14)=10.75,p=.005,2p=.43andmoreimportantlyaninteractionbetweenblockandlever,F(1,14)=43.68,p.001,2p=.76.Therewasatripleinteractionbetweenclassication,blockandlever,F(1,14)=11.87,p=.004,2p

=.46.Theanalysisjustpresentedinvolveddividingratsintotwogroups(STandGT)usingtheirbiasesduringthenalblockoftraining.However,thesameconclusionsaresupportedbyananalysisinwhichtheirleverpressesandfood-wellentriesaretreatedasacontinuum.TheupperpanelsofFigure3depicttherelationshipbetweenleverpresses(panelA)onreinforcedL1trialsforthenalblocksoftraining(i.e.,T5andT6;left-handpanel),andbetweenfood-wellentriesonL1trialsforthesameblocks(panelB).Thelowerpanelsdepicttherelationshipsbetweenleverpressesonthenalblockoftrainingandtherstblockofreversal(i.e.,T6andR1,panelC)onL1trials,andbetweenfood-wellentriesforthesametwoblocks(panelD)onL1trials.Thegroupmembershipofeachratisidentied.Forbothtypesofresponse,therewasasignicantcorrelationbetweenT5andT6,leverpresses,(r(14)=.94,p.001;food-wellentries,r(14)=.93,p.001).However,whiletherewasacorrelationbetweenT6andR1for43 leverpresses(r(14)=.83,p.001,therewasnotforfood-wellentries(r(14)=-.25,p=.355).Food-wellentrieschangedbetweenT6andR1,butleverpressesdidnot.ReversalAcrosstheblocksofreversaltraining(right-handsideoftheupperandlowerpanelsofFigure2),thenumbersofleverpressesincreasedduringL2anddecreasedduringL1inGroupST,andthischangewasnumericallysmallerinGroupGT.Incontrast,thenumberoffood-wellentriesincreasedduringL2anddecreasedinL1inGroupGT,andthischangewaslessapparentinGroupST.LeverpressesAnANOVAforleverpressesfortheblocksfollowingthereversalrevealedmaineectsofblock,F(2.95,41.24)=17.07,p.001,2p=.55,butnoeectsofclassication,F(1,14)=3.60,p=.079,2p=.20orlever,F(1,14)=1.03,p=.327,2p=.07.Therewasablockbyleverinteraction,F(1.68,23.58)=2

1.96,p.001,2p=.61,butnointeractionbetweenclassicationandblock,F(2.95,41.24)=2.56,p=.069,2p=.15,orclassicationandlever,F(1,14)=0.01,p=.919,2p.01.Therewasnosignicantinteractionbetweenclassication,blockandlever,F(1.68,23.58)=3.48,p=.054,2p=.20.Food-wellentriesAparallelanalysisforfood-wellentriesrevealedamaineectoflever,F(1,14)=22.76,p.001,2p=.62,butnoeectofclassication,F(1,14)=0.13,p=.721,2p.01,orblock,F(1.70,23.75)=1.42,p=.259,2p=.09.TherewasablockbyleverinteractionF(1.85,25.90)=13.04,p.001,2p=.48,butnointeractionbetweenclassicationandblock,F(1.70,23.75)=0.46,p=.604,2p=.03,orclassi-cationandlever,F(1,14)=2.36,p=.147,2p=.14.Therewasatripleinteractionbetweenclassication,blockandlever,F(1.85,25.90)=3.67,p=.042,2p=.21.44 2.2.3DiscussionExperiment1Discriminationtrainingwherethepresentationofonelever(L1)waspairedwithfoodpelletsandanother(L2)wasnot,resultedinmarkedindividualdierencesinconditionedresponding;withsomeratsinteractingwithL1(butnotL2)andothersapproachingthesiteoffooddeliveryduringL1(butnotL2).Whenthecontingencieswerereversed,withL1nownon-reinforcedandL2reinforced,thedierentlevelsofleverpressingtoL1(andL2)inGroupsSTandGTremainedremarkablystableduringtherstblockofreversal.Incontrast,thelevelsoffood-wellentrieschangedmorerapidlyinbothGroupsSTandGT(Figure2).Thisdierentialsensitivityofthetworesponseformstochangingcontingencieswasalsoevidentwhentheywereconsideredascontinuousvariables(Figure3).Theseresultsdemonstratethatthedissociationbetweenleverpresses(inratsdesignatedasSTs)andfood-wellentries

(inratsdesignatedasGTs)doesnotre ectadierenceinthesensitivityofthetwogroupstochangedreinforcementcontingenciesperse(cf.Ahrensetal.,2016).Instead,theseresultsshowthattheleverpressandfood-wellentryresponsesaredierentlysensitivetosuchchangesirrespectiveofthephenotypeoftherat.Theseobservationssuggestthatthedistinctbehavioursre ecttheparalleloperationofS-SandS-Rsystemswithinanindividual,ratherthantheoperationofasinglesystem(eitherS-SorS-R)thatgivesrisetobothbehaviours(cf.Lesaintetal.,2014).Experiment2attemptedtoextendtheseobservationsbyexaminingwhetherchangesinthenatureofthereinforcer(betweenalternativesthatproducedierentlevelsofresponding)producemorerapidchangesinfood-wellactivitythaninleverpressinginSTsandGTs.2.3Experiment2InExperiment1,IexaminedhowtheextinctionofanassociationbetweenaleverandanappetitiveUSaectedbehaviouronleverpressesandfood-wellentriesinbothSTsandGTs.InExperiment2,ratsreceivedseparatepresentationsoftwolevers(L145 andL2)thatwerebothpairedwiththesamereinforcerduringtraining(eitherfoodpelletsorsucrose).Pilotresearchhadestablishedthatfoodpelletsmaintainhigherlevelsofbothleverpressingandfood-wellentriesthandoessucrose(seealsoPatituccietal.,2016)whichshouldbeevidentintherststageoftraininginExperiment2.Theratsthatwerereinforcedwithpelletsandsucrosewerefurtherdividedintotwogroups(GroupSTadGT)onthebasisoftheirbiasesattheendoftraining.Duringthesecondstage,thereinforcersassociatedwiththetwoleverswereswitched:theratsgivenpelletsduringtrainingreceivedsucroseduringtheswitchandthosegivensucroseduringtrainingreceivedfoodpelletsduringtheswitch(seedesigninTable2).Theissueofcentralinterestwas

theextenttowhichthetwotargetbehavioursthathaddevelopedduringL1andL2(leverpressesandfood-wellentries)wouldchangetore ectthefactthattheleverswerenowpairedwithreinforcersthatmaintaineddierentlevelsofperformance(i.e.,foodpelletsandsucrose).IfthebehaviourofratsinGroupSTisgeneratedbyaS-Rsystem,thenbothleverpressesandfood-wellentriesshouldbelesssensitivetothechangeinreinforcertypethanthoseinGroupGT,whosebehaviourisgeneratedbyaS-Ssystem.However,ifleverpressingisbasedonaS-Rsystemwhereasfood-wellentriesre ectaS-Ssystem,thenleverpressingshouldbelesssensitivetothechangeincontingenciesbetweenthetrainingandswitchstagesthanshouldfood-wellentries,irrespectiveofwhetherthosebehavioursareexpressedinGroupSTorGroupGT.2.3.1MethodAnimalsandapparatusThirty-twonavemaleListerHoodedrats(suppliedbyEnvigo,Bicester,U.K.)werehousedinthesamewayasdescribedinExperiment1.Theirmeanadlibitumweightwas295g(range284{320g).Ratshadfreeaccesstowater,andtheyweremaintainedbetween85and95%oftheiradlibweightsbygivingthemrestrictedaccesstofoodattheendofeachday.TheexperimentalchamberswerethoseusedinExperiment46 1.ThedesignoftheexperimentissimilartoExperiment1,withtheadditionthatinExperiment2,thesucrosedipperdelivered0.05mlofsucrosesolution(8%weight/weightwithwater)forhalftherats(seeTable2).Thesucrosedipperwassituatedattheright-handofthefood-wellincloseproximityofthepelletdispenser.Thedefaultpositionofthedipperislevelledupandaccessibletoratsandatpointofrewarddeliverythedipperlowersandgetsrelled.Thatis,sucroseremainsavailableindenitely,likepellets,butonlytheamountdeliveredbyoneloweringofthedipper.Respondingwasrecordedinthesamewayasdescribedfor

foodpelletsinExperiment1.ProcedureTheratshadtwo24-minpretrainingsessionsbeforethetrainingandswitchstages.Duringthesesessions,theratsreceivedthereinforcer(foodpelletsorsucrose)thatwastobedeliveredintheimmediatelysucceedingstage.Thereinforcersweredeliveredonavariable60-sschedule(range40{80s).Ratsreceived12daysoftrainingthatwerear-rangedinthesamewayasExperiment1withtheexceptionthatthepresentationofbothlevers(L1andL2-leftandrightlevers,counterbalanced)werefollowedbyareinforcer(foodpelletsforhalfoftheratsandsucrosefortheremainder).Theswitchstagealsoconsistedof12days.Thisstagewasidenticaltothetrainingstagewiththeexceptionthattheratsthathadreceivedfoodpelletsduringthetrainingstagereceivedsucroseduringtheswitchstage,andthosethathadreceivedsucroseduringtrainingreceivedfoodpelletsduringtheswitch.Ratswererandomlyassignedtotheexperimentalgroups.DataAnalysisTherewerestrongpositivecorrelationsbetweenleverpressbehaviourduringthepre-sentationsofL1andL2thatwerebothpairedwiththesameoutcome(eitherfoodpelletsorsucrose)andbetweenfood-wellbehaviouronthetwolevers.TheseobservationshavesometheoreticalsignicancewhencontrastedwiththeresultsofPatituccietal.(2016),47 Table2:DesignofExperiment2 ClassicationTrainingSwitchNotation STL1&L2PelL1&L2SucSTPelkSucororL1&L2SucL1&L2PelSTSuckPel GTL1&L2PelL1&L2SucGTPelkSucororL1&L2SucL1&L2PelGTSuckPel Note:STreferstoasign-trackerratandGTreferstoagoal-trackerrat.L1andL2refertotwolevers(leftandright,counterbalanced).Duringtraining,bothleverswerepairedwithonereinforcer(foodpelletsorsucrose),andduringtheswitch,bothleverswerethenpairedwiththeotherreinforcer(sucroseorfoodpellets,respectively).Ratswe

reclassiedasSTorGTonthebasisoftheirbiastowardsleverpressingorenteringthefood-wellduringthenalblockoftraining.whoreportednocorrelationbetweenthesign-andgoal-trackingbiasesontwoleversthatsignalleddierentoutcomes.IshallconsidertheimplicationsofthisevidenceintheChapter2Discussion.However,tosimplifytheresultssection,theprincipalanalysisoftheresultsofExperiment2willbeconductedwiththefrequencyofresponsescombinedacrosstheleftandrightlevers.AsinExperiment1,thetrainingandswitchsessionswerecombinedinto2-dayblocksforthepurposeofanalysis.Theratsweresplitintotwogroups,STsandGTs,usingthebiasscoredescribedinExperiment1.Thesplitwasconductedseparatelyforthesubgroupsofratsthatreceivedfoodpelletsandsucroseduringthetrainingstage.Thisresultedinfourgroups(8ineachgroup):STPelkSuc(pelletsduringtraining,sucroseduringtheswitchstage),GTPelkSuc(GTs;pelletsduringtraining,sucroseduringtheswitch),STSuckPel(STs;sucroseduringtraining,pelletsduringtheswitch)andGTSuckPel(GTs;sucroseduringtraining,pelletsdur-ingtheswitch).ForthePelkSucgroupbiasscoresabove.33wereclassiedasGTsandscoresbelow.31asSTs.FortheSuckPelgroupscoresabove.58wereclassiedasGTsandbelow.54asSTs.Thedistributionofsign-trackingandgoal-trackingbehaviourisshowninFigure4.48 MixedANOVAswereconductedseparatelyforleverpressingandfood-wellentrieswithwithin-subjectsfactorblock(forthetrainingandtheswitchstage6levels:T1-T6andS1-S6;transitionstage2levels:T6-S1),andbetween-subjectsfactorsreinforcer(2levels:pelvssuc)andclassication(2levels:STvsGT). Figure4:Distributionofsign-tracking(leverpresses)andgoal-tracking(food-wellentries)behaviourforExperiment2per(10-s)trialdu

ringlastblockoftraining(T6)averagedacrossL1andL2.Theblacksymbolscorrespondtosign-trackers(ST)andtheclearsymbolstogoal-trackers(GT).2.3.2ResultsThemainresultsfromExperiment2areshowninFigure5andinFigure6.AsinExperiment1,theanalysisoftheresultsofExperiment2willbeginwithresultsfromthetrainingstage,beforemovingtoacomparisonofthenalblockoftrainingwiththerstblockofreversal(identiedbythegreysection),andnallytheswitchstageasawhole.49 Figure5:ResultsfromExperiment2:theeectsofaswitchinUSvalueonsign-trackingandgoal-tracking.Mean(SEM)leverpresses(upperpanel)andfood-wellentries(lowerpanel)per(10-s)trialacrossthetwostages:training(T1{T6)andswitch(S1{S6).Duringtraining,ratsreceivedpresentationsoftwoleverspairedwitheitherpelletsorsucrose.Ratswereclassiedassign-trackers(ST)andgoal-trackers(GT)onthebasisoftheirbehaviourduringthenalblockoftraining(T6).Thereinforcersthatfollowedtheleverswereswappedduringthesecondswitchstage.Thegreysectionindicatestransitionbetweeninitialtrainingandtheswapfromsucrosetofoodpelletrewards(orfrompelletstosucrose).Theblacksymbolscorrespondtosign-trackers(ST)andtheclearsymbolstogoal-trackers(GT).Solidlinesindicategroupsreceivingpelletsinphase1andsucroseaftertheswitchtophase2(PelkSuc)anddashedlinesthoseswitchedfromsucrosetopellets(SuckPel).Forexample,themeansfortheSTgroupthatreceivedfoodpelletsduringtrainingandsucroseduringtheswitcharegivenbyblacksymbols(ST)andcontinousline(PelkSuc).TrainingInspectionoftheleft-handsideoftheupperandlowerpanelsinFigure5suggeststhattheSTgroupsaremorelikelytoengageinleverpressingthanaretheGTgroups,andthattheGTgroupsaremorelikelytoenterthefood-w

ellthantheSTgroups.Thesegroupdierences,especiallyinthecaseoffood-wellactivity,weremostmarkedwhenfoodpelletswerethereinforcer.Thisdescriptionofthetrainingresultspresented50 Figure6:CorrelationresultsfromExperiment2:theeectsofaswitchinUSvalueonsign-trackingandgoal-tracking.Theupperpanelsshowtherelationshipbetweenthemeannumberofresponsesper(10-s)trialbetweenblocksT5andT6forleverpresses(A)andforfood-wellentries(B)onreinforcedtrials.ThelowerpanelsshowtherelationshipT6andR1forleverpresses(C)andfood-wellentries(D)onreinforcedtrials.Theresponse(leverpressingandfood-wellentries)hasbeenpooledacrossL1andL2trials.Theblacksymbolscorrespondtosign-trackers(ST)andtheclearsymbolstogoal-trackers(GT).inFigure5issupportedbyseparateanalysesofleverpressesandfood-wellentries.LeverpressesANOVAconductedonleverpresses,pooledacrossthetwolevers,conrmedthereweremaineectsofreinforcer,F(1,28)=31.02,p.001,2p=.53,classi-cation(STvsGT),F(1,28)=9.66,p=.004,2p=.26,andblock(T1-T6),F(3.15,88.33)=24.96,p.001,2p=.47.Therewereinteractionsbetweenrein-forcerandblock,F(3.15,88.33)=7.76,p.001,2p=.22,andbetweenclassication51 andblock,F(3.15,88.33)=7.11,p.001,2p=.20,butnotbetweenreinforcerandclassication,F(1,28)=2.76,p=.108,2p=.09.Thethree-wayinteractionbetweenreinforcer,classicationandblockwasnotsignicant,F(3.15,88.33)=1.06,p=.373,2p=.04.Food-wellentriesAparallelanalysisoffood-wellentriesrevealedmaineectsofreinforcer,F(1,28)=9.86,p=.004,2p=.26,classication,F(1,28)=10.47,p=.003,2p=.27andblock,F(2.81,78.57)=19.48,p.001,2p=.41.Therewerealsointeractionsbetwee

nreinforcerandclassication,F(1,28)=5.42,p=.027,2p=.16,andbetweenclassicationandblock,F(2.81,78.57)=8.65,p.001,2p=.24.Therewasnointer-actionbetweenclassicationandblock,F(2.81,78.57)=2.14,p=.105,2p=.07,andnotripleinteractionbetweenreinforcer,classicationandblock,F(2.81,78.57)=2.64,p=.059,2p=.09.TransitionblocksInspectionofthegreypanelinFigure5showsthattherewererapidchangesinfood-wellentries(lowerpanel)butnotinleverpressing(upperpanel).Tobemorespecic:Thehighleveloffood-wellactivity-previouslymaintainedbypellets-declinedfromT6toS1,andthelowleveloffood-wellactivity-previouslymaintainedbysucrose-increasedfromT6toS1.Incontrast,leverpressingwaslargelyunchangedacrossT6andS1.Thisdescriptionwassupportedbyseparateanalysisofleverpressesandfood-wellentries.LeverpressesANOVAconductedforleverpressesrevealedaneectofreinforcer,F(1,28)=33.57,p.001,2p=.55andclassication,F(1,28)=14.72,p.001,2p=.34,butnoeectofblock,F(1,28)=1.86,p=.183,2p=.06.Therewasaninteractionbetweenreinforcerandclassication,F(1,28)=4.74,p=.038,2p=.14,andaninteractionbetweenclassicationandblock,F(1,28)=4.70,p=.039,2p=.14.Critically,therewasnointeractionbetweenreinforcerandblock,F(1,28)=0.00,p=.952,2p.01,and52 nothree-wayinteractionbetweenreinforcer,classicationandblock,F(1,28)=0.23,p=.637,2p.01.TheBayesfactorforthebestmodelwithouttheblockbyleverinteractionrelativetothebestmodelwiththeinteractionis5.88,indicatingevidenceagainstthepresenceoftheinteraction.TheBayesfactorforthebestmodelwithouttheblockbyleverbygroupinteractionrelativetothemodelwiththeinteraction

is100,indicatingstrongevidenceagainstthepresenceoftheinteraction.TheswitchinreinforcerhadlittleimpactonleverpressbehaviourineithertheSTorGTgroups.Food-wellentriesAparallelanalysisoffood-wellentriesrevealedamaineectofclassication,F(1,28)=15.59,p.001,2p=.36andblock,F(1,28)=4.18,p=.050,2p=.13,butnoeectofreinforcer,F(1,28)=1.22,p=.278,2p=.04.Critically,therewasaninteractionbetweenreinforcerandblock,F(1,28)=95.42,p.001,2p=.77,aswellasbetweenclasicationandblock,F(1,28)=5.76,p=.023,2p=.17,butnotbetweenreinforcerandclassication,F(1,28)=1.94,p=.174,2p=.06,andnotripleinterac-tionbetweenreinforcer,classicationandblock,F(1,28)=2.14,p=.155,2p=.07.TheBayesfactorforthebestmodelwithouttheinteractionbetweenblock,leverandgrouprelativetothemodelwiththeinteractionis25,indicatingstrongevidenceagainsttheinteraction.Thatistheswitchinreinforcershadanimmediateimpactonbehaviourdirectedtothefood-well,andcriticallythiswasequivalentinboththeSTandGTgroups.Thesameconclusionsaresupportedbyananalysisinwhichleverpressesandfood-wellentriesweretreatedasacontinuum.TheupperpanelsofFigure6depicttherelationshipbetweenleverpressesonthenalblocksoftraining(i.e.,T5andT6;left-handpanel)andbetweenfood-wellentriesonthesameblocks(right-handpanel)pooledacrossleftandrightlevertrials.Thelowerpanelsdepicttherelationshipsbetweenleverpressesonthenalblockoftrainingandtherstblockofswitch(i.e.,T6andS1left-handpanel),andbetweenfood-wellentriesonthesametwoblocks(right-handpanel).ForbothtypesofresponsetherewasasignicantcorrelationbetweenT5andT6(lever,53 r(30)=.92,p.001;food-wellentries,r(

30)=.81,p.001).BetweenT6andS1thereisasignicantcorrelationforleverpresses(r(30)=.83,p.001),butnotforfood-wellentries(r(30)=.06,p=.734).SwitchThepatternofresultsevidentontherstblockoftheswitch(i.e.,S1)was,forthemostpart,evidentacrossthelaterblocksoftheswitchstage.Morespecically,themarkedchangesinfood-wellentriesweresustainedacrosstheswitchstageandwereaccompaniedbylittlechangeinleverpressing:whilethelowlevelofleverpressingincreasedwhensucrosewasreplacedwithfoodpelletsduringtheswitch,thehighlevelofleverpressingwasmaintainedwhenfoodpelletswerereplacedwithsucrose.LeverpressesANOVAconductedonleverpresses,pooledacrossthetwolevers,conrmedtherewasamaineectofclassication,F(1,28)=13.69,p.001,2p=.33,howevertherewerenoeectsofreinforcer,F(1,28)=3.35,p=.078,2p=.11orblock,F(3.51,98.17)=2.20,p=.083,2p=.07.Therewereinteractionsbetweenrein-forcerandblock,F(3.51,98.17)=5.63,p.001,2p=.17,andbetweenclassicationandblock,F(3.51,98.17)=4.52,p=.003,2p=.14,butnotbetweenreinforcerandclassication,F(1,28)=0.68,p=.417,2p=.02.Thethree-wayinteractionbetweenreinforcer,classicationandblockwasnotsignicant,F(3.51,98.17)=1.98,p=.112,2p=.07.Food-wellentriesAparallelanalysisoffood-wellentriesrevealedmaineectsofreinforcer,F(1,28)=20.46,p.001,2p=.42,classication,F(1,28)=6.90,p=.014,2p=.20,andblockF(2.87,80.25)=3.59,p=.019,2p=.11.Therewasaninteractionbetweenreinforcerandblock,F(2.87,80.25)=5.35,p=.002,2p=.16,butnointeractionsbetweenreinforcerandclassication,F(1,28)=0.03,p=.862,2p.01,orbetweenclassicationandblock,F(2.87,80.2

5)=0.82,p=.481,2p=.03,andnotriplein-54 teractionbetweenreinforcer,classicationandblock,F(2.87,80.25)=1.41,p=.247,2p=.05.2.3.3DiscussionExperiment2TheresultsofExperiment2conrmtheprincipalconclusionsderivedfromtheresultsofExperiment1.First,lever-pressbehaviourwaslesssensitivetochangesinreinforce-mentcontingenciesthanwasfood-wellbehaviour.Second,thisdierenceinsensitivitywasequallyapparentinratsthatwereclassiedasSTsandGTs.InExperiment1,theseconclusionsweresupportedbytheeectsofareversalbetweentherelationshipsbetweentwolevers(L1andL2)andthepresenceandabsenceoffoodpellets,whereasinExperiment2theyweresupportedbythesubstitutionofreinforcersthatmaintainedmore(pellets)orless(sucrose)behaviour.ThefactsthatExperiment2usedmaleratswhileExperiment1usedfemalerats(anditsresultshavebeenreplicatedinmalerats)andthetwoexperimentsuseddierentstrains(Sprague-DawleyandListerHooded,respectively),suggeststhatthedierenceinsensitivityofleverandfood-welldirectedbehaviourtochangesinreinforcementcontingenciesispreservedacrossratstrainsandmale/femaleanimals.2.4Chapter2DiscussionDuringappetitivePavlovianconditioning,rodentswillreliablydisplaybehaviourdirectedbothtowardthestimulus(sign-tracking)andtowardthesiteoffoodpelletdelivery(goal-tracking).Althoughindividualdierencesinconditionedrespondinghavetypicallyreceivedscantconsiderationintheoriesofassociativelearning,itisclearthatthedistributionofthesebehavioursdiersacrossindividuals(e.g.,Fitzpatricketal.,2013).Forexample,whenaleveristemporarilyinsertedintoaconditioningchamberandpairedwithfoodpelletssomeratsdevelopaconsistenttendencytointeractwith

theleverwhereasothersdevelopatendencytoapproachthefood-well.Thesebehavioursaredierentlysensitivetothecurrentvalueofthereinforcerandindeeditspresence.55 Patituccietal.(2016)demonstratedthatthebiastowardengaginginfood-wellactivityratherthanlever-pressactivitywaspositivelycorrelatedwiththepalatabilityofthereinforcer;andsatingratsonthereinforcerreducedfood-wellbutnotlever-orientedactivity;andAhrensetal.(2016)showedthatleverpressing,inratsthatpredominantlyengagedinsign-tracking,waslesssensitivetoextinctionthanfood-wellactivity,inratsthatpredominantlyengagedingoal-tracking.Myresultsconrmthatlever-pressbehaviourisindeedlesssensitivetochangesinreinforcementcontingenciesthanisfood-wellbehaviour.InExperiment1,thiswasevidentintheeectsofareversalintherelationshipsbetweentwoleversandthepres-enceandabsenceoffoodpellets,whereasinExperiment2itwasevidentintheeectsofthesubstitutionofreinforcersthatmaintainedmore(pellets)orless(sucrose)be-haviour.Moreover,inbothexperiments,theseconclusionsreceivedadditionalsupportfromtreatinglever-pressandfood-wellactivityinacontinuousway:lever-pressactivitywascorrelatedbetweenthenalblockoftraining(T6)andtherstblockofthechangedcontingencies(R1inExperiment1andS1inExperiment2),butfood-wellactivitywasnot.Here,Icontrastedtwopossibleaccountsofthebehaviouralphenotypes,basedontheassumptionthatsign-andgoal-trackingaremodulatedbytwosystems(e.g.,S-S/S-R,ormodel-based/model-free).First,thatthebehaviourofagivenrodentisgovernedbytheoperationofasinglesystemandthatthecontrolofbothtypesofbehavioursimplyre ectsthenatureofthegoverningsystem(i.e.,inagoal-trackingratthepredominantsystemth

atdrivesbehaviourisS-Sormodel-based,andinasign-trackingratisS-Rormodel-free).Thisaccountpredictsthatfood-wellandlever-orientedbehaviourswillexhibitdierentpropertiesinSTsandGTs.Second,thatbehaviourdirectedtowardthefood-wellandleveraregeneratedbyindependentsystems(S-S/model-basedandS-R/model-free,respectively)thatoperateinparallel.ThisanalysispredictsthatagivenformofresponsewillexhibitthesamecharacteristicsinSTsandGTs.InExperiments1and2,lever-pressandfood-wellbehaviourinbothgoal-trackingandsign-trackingratsshowedthesamepatternofsensitivitytochangesinreinforcercontingencies.Thispat-56 ternofresultsprovidessupportforthesecondoftheseaccounts:dierentialsensitivitytocontingencychangesisapropertyofthebehaviour,nottheanimal.WhiletheresultsthatIhavepresentedsofarhaveclearimplicationsregardingthecontrolofbehavioursinthetwobehaviouralphenotypestheydonotcontributetoourunderstandingoftheoriginofthetwophenotypes.Patituccietal.(2016)arguedthatfood-wellactivitywasmorelikelytodominatein(goal-tracking)ratsthat,forwhateverreason,valuedthereinforcermore,basedonthefactthatgoal-trackingcorrelatedwiththepalatability(preference)ofthereinforcer.But,theyalsoobservedthattheclassi-cationofarataseitheraGT(orST)onaleverthatwaspairedwithonereinforcer(e.g.,foodpellets)wasunrelatedtotheclassicationofthesameratonasecondleverthatwaspairedwithadierentreinforcer(e.g.,sucrose).Theyarguedthatifagivenratvaluedonereinforcer(e.g.,foodpellets)morethantheother(e.g.,sucrose)thenthiswouldresultinmoregoal-trackingononeleverthananother.Whilethisanalysisiscertainlyconsistentwithotherfeaturesoftheirresults,amoreprosaicaccountcanbedevelope

dforthelackofcorrelationsbetweenthebehavioursdirectedtotwolevers:Itmighthavere ectedsuperstitiousreinforcementofdierentbehaviours(e.g.,leverorfood-welloriented)thathappenedtooccurduringthetwotypesoftrials.However,inthecurrentExperiment2,thetwoleverswerebothpairedwiththesamereinforcer(foodpelletsorsucrose),andwhilethisnecessarilymeansthatthereisnodierenceinthevalueofthereinforcerthatispairedwiththelevers,itremainspossiblethatratswillbeengagingindierentbehavioursduringthetwoleversthatwouldbesubjecttosuperstitiousreinforcement.TheresultsofExperiment2providesupportfortheexplanationpreferredbyPatituccietal.(2016):Whentheleverswerepairedwiththesamereinforcerthereweresignicantcorrelationsbetweenfood-wellactivityontheleftandrightleversonBlock6(r(30)=.94,p.001),andbetweenlever-pressactivityonthetwoleversduringthesameblock(r(30)=.74,p.001).Thefactthattheextenttowhichphenotypicvariationinsign-andgoal-trackingbehavioursisconsistentacrossleversdependsonwhethertheyarepairedwiththesameordierentoutcomessuggests57 thatoutcomevaluecontributestoresponseselection.Tosummarize,theresultspresentedinthischapterindicatethatindividualdier-encesinthetopographyofconditionedbehaviourre ectthebehaviourthatisbeingmeasuredanditsproperties.Theobserveddierencesre ecttheoperationofdistinctassociativeprocessesthatdierintheirsensitivitytorewardvalueandchangesincon-tingencies,thisbeingapropertyofthebehaviourandnottheanimal,perse.AsIalreadymentionedintheintroduction,adual-processaccountisabletoexplainsomeresultswhereindividualdierencesareobserved(e.g.,dierentialsensitivitytochange)butdoesnotprovideasat

isfactoryexplanationoftheoriginofthesedierences.TheevidencesuggeststhatS-Slearningisnecessaryforacquisitionofsign-andgoal-tracking,buttheroleofS-Rlearningislessclear.Sign-trackingdisplaysthepropertiesofanhabitualsystemgiventheresistancetochange,howeverwehaveseenthatanassociationbetweentheCSandaresponseisnotnecessaryforthebehaviourtoemerge(e.g.,Browne,1976;Williams&Williams,1969).Theseresultsmotivatedthedevelopmentofanewformalmodeloflearningandperformance,HeiDI(describedinChapter3),inwhichtheassociativestructuresthatareacquiredduringPavlovianconditioningareintegratedwithananalysisofhowtheknowledgeembodiedinthesestructuresdeterminesthenatureoftheresponseselicitedbyaCS.Neitherofthedual-processaccountsisabletoexplaintheoriginoftheobservedindividualdierencesortopredictwhenorhowthebalancefavoursonesystemortheother.HeiDI,echoestheconclusionspresentedinChapter2andatthesametimedistancesitselffromthedual-processaccountinexplainingconditionedbehaviour.ThemodelisexplainedindetailthefollowingChapter.58 Chapter3HeiDI:AmodelforPavlovianlearningandperformancewithreciprocalassociationsHeidi,oneoftheworld'smostpopularchildren'sstories,wasoriginallywrittenbyJohannaSpyriastwocompanionpieces:Heidi:Heryearsofwanderingandlearning,andHeidi:Howsheusedwhatshelearned.TheydescribehowHeidi'spredispositiontowanderandlearnwaslaterevidentinherbehaviour.ThecentralconcernofthemodeldevelopedhereisthenatureoftheassociativestructuresthatareacquiredduringPavlovianconditioningandhowthesestructuresresultintheirbehaviouralsequelae.Pavlovianconditioningisprobablythebest-knownphenomenoninthehistoryofthescienticstudyofpsychology.Thebasicproc

edureandobservationscanberecountedbypeoplewithlittleornootherknowledgeoftheeld:dogsgivenpairingsofaringingbellwithfoodcometosalivatewhenthebellrings.HeiDIisasignicantrevisionofthemodelofPavlovianconditioningdevelopedbyRescorlaandWagner(Rescorla&Wagner,1972;Wagner&Rescorla,1972),andre ectsPavlov'svisionthatthestudyofconditioningprovidesassociativepsychologywithascienticbasis(Pavlov,1941,171).Theirmodelhashadaprofoundandenduringin uenceontheeldofanimallearning(e.g.,Mackintosh,1975;McLaren,Kaye,&Mackintosh,1989;Pearce&Mackintosh,2010;Wagner,1981),butalsoonpsychologymorebroadly(e.g.,Gluck&Bower,1988;Kruschke,1992;Rumelhart,Hinton,&Williams,1986),andonneuroscience(e.g,Leeetal.,2018;Schultz,Dayan,&Montague,1997);with8667citationsatthetimeofwritingthisthesis.However,theRescorla-Wagnermodeloersonlythemostrudimentaryanalysisoftheassociativestructuresthatareacquiredduringconditioningandhowthesemapontochangesinbehaviour.Moreover,themodelprovidesnoexplanationforrecentevidence,wheredierentbehaviouralindicesoflearningcanbetakentosupportdierentconclusionsaboutthestrengthofanassociation(e.g.,Flagel,Akil,&Robinson,59 2009;Flageletal.,2011;Patitucci,Nelson,Dwyer,&Honey,2016).Thisfundamentalproblem,togetherwithothersthatIshallcometo(e.g.,Dickinson,Hall,&Mackintosh,1976;Lubow,1989;Miller,Barnet,&Grahame,1995;Rescorla,2000,2001),providedtheimpetusforthedevelopmentofHeiDI.Thenameofthemodel,HeiDI,re ectstheliteraryreferenceandlinkstomysurnameandthoseofmysupervisorstooneoftheprincipalissuesthatthemodelseekstoaddress:Howexcitationandinhibitiondetermineideo-motion.3.1TheRescorla-WagnerModelTheRescorla-Wa

gnermodelproposesthatPavlovianconditionedbehaviourre ectstheformationofanassociationbetweenthenodesactivatedbytheconditionedstimulus(CS)andunconditionedstimulus(US).ThepresentationoftheCScomestoassociativelyactivatetherepresentationorideaoftheUSandtherebybehaviour,whichcanbethusconsideredideo-motive:Aseeminglyre exivemovementeectedinresponsetoanidea,inthiscasetheevokedmemoryoftheUS.Themodelhasbeenfundamentaltothedevelopmentoftheoreticaltreatmentsofassociativelearningforalmost50yearsandhasin uencedneurobiologicalanalysesoflearningandmemory.Ibrie yreviewthemodelherebecauseitprovidestheprincipalsourceofinspirationforthenewmodelthatisdevelopedintheremainderofChapter3.AccordingtotheRescorla-Wagnermodel,thechangeintheassociativestrength(VCS�US)ofaCSonagiventrialisdeterminedbythedierencebetweenthemaximumassociativestrengthsupportablebyaUS()andthepooledassociativestrengthofallstimulipresentedonthattrial(PVTotal�US).Theglobalorpoolederrorterm(-PVTotal�US)allowsthemodeltoaccommodateawiderangeofphenomena:blocking(e.g.,Kamin,1969),conditionedinhibition,(e.g.,Rescorla,1969),contingencyeects(e.g.,Rescorla,1968),overshadowing(e.g.,Mackintosh,1976),relativevalidity(e.g.,Wagner,Logan,Haberlandt,&Price,1968),superconditioning(e.g.,Rescorla,1971).Thesephenomenawerebeyondthescopeofmodelswithseparateerrortermsforeach60 componentofapatternofstimulation(e.g.,Bush&Mosteller,1951;Hull,1943).Italsoprovidesanelegantintegrationofexcitatoryconditioning,wherethememoryofaCSprovokesthememoryoftheUS,andinhibitorylearning,whereaCScanreducethelikelihoodoftheUSmemoryfrombecomingactivewhenitotherwisewould.
VCS�

;US=CS�US(�XVTotal�US)(0)Brie y,thepoolederrortermmeansthat
VCS�USisaectednotonlybythecurrentassociativestrengthofthatstimulus(i.e.,VCS�US),butalsobythepresenceofotherstimulithathaveassociativestrength(i.e.,byPVTotal�US).AccordingtotheRescorla-Wagnermodel,thechangeinassociativestrengthdrivenbythediscrepancywithinthepoolederrorterm(-PVTotal�US)ismodulatedbytheproductoftwolearningrateparameters,CSandUS.RescorlaandWagner(1972)notethat\thevalueofCSroughlyrepresentsstimulussalience"andthat\theassignmentofdierentUSvaluestodierentUSsindicatesthattherateoflearningmaydependontheparticularUSemployed".Thetwolearningrateparameterswereconnedtotheunitinterval:0CS,US1,andenabledthemodeltocapturethefactthatthesalienceoftheCS(CS)andnatureoftheUS(US)aecttherateofexcitatorylearning1.Ofparticularnote,however,isthefactthatthismodelofPavlovianconditioningdidnotaddress{inanysystematicfashion{thein uenceofassociativestrength(i.e.,V)onconditionedresponding.Indevelopingtheirmodelanditsapplicationtoexperimentalndings,RescorlaandWagner(1972,p.77)notedthatitwas\sucientsimplytoassumethatthemappingofVsintomagnitudeorprobabilityofconditionedrespondingpreservestheirordering",andthatanysuchmappingwouldinevitablydependonthedetailsofeachexperimen- 1ToenableinhibitoryconditioningtooccurontrialswhentheUSisabsent,RescorlaandWagner(1972)assumedthattakesapositivevaluewhentheUSisabsentbuttheCSispresent;withthisvalueassumedtobelowerthanontrialswhenboththeCSandUSarepresent.ThiscomplexityisavoidedinHeiDI.61 talsituationandon\perfo

rmance"factors.Inacompanionpaper,whencomparingconditioninginvolvingasingleCSwithconditioninginvolvingacompoundoftwoCSs,theyalsonoted\thatthegreaterthenumberofcueswhichismadeavailable,themorelikelyitisthatthesubjectwillbeprovided(andperhapsidiosyncraticallyso)withasinglesalientcuetowhichconditioningcanrapidlyoccur"(Wagner&Rescorla,1972,pp.303-304).Thisstatementacknowledges(parenthetically)thefactthatindividualdierencesmightaectconditioning(seealso,Pavlov,1941,pp.378-378)buttherehasbeenlittleappetitetoaddresssuchdierences(empiricallyortheoretically)andtomovebeyondsimple(grouplevel)assumptionsaboutthetranslationoflearningintoperformance(seealso,Mackintosh,1975;Miller&Matzel,1988;Pearce,1994;Pearce&Hall,1980).However,thereisnowevidencedemonstratingthattherelianceonsuchassumptionscannolongerbesustained;andnorcantheideathatPavloviancondition-ingresultsinunconditionedresponsessnippedfromtheUSbeinggraftedontotheCSthroughaprocessofstimulussubstitution(seeDwyer,Burgess,&Honey,2012;Pavlov,1927;Wagner&Brandon,1989).3.2IndividualdierencesIthasbeendiscussedinChapter1,thatevenasimpleauto-shapingprocedurepro-ducesmarkedindividualdierencesinbehaviour:someratspredominantlyinteractwiththeCS(e.g.,lever),othersinvestigatethelocationwherethereinforcerisabouttobedelivered,andtheremaindershowpatternsofbehaviourinbetweenthesetwoextremes(e.g.,Patituccietal.,2016).Whenleveractivityisusedastheassayofdiscriminationlearning,thesign-trackinggroupshowbetterlearningthanthegoal-trackinggroup;butwhenfood-wellactivityisusedthenthereverseisthecase.Thatis,itisnotpossibletoprovideamappingofVsontoconditionedbehaviourthatprovide

sacoherentinter-pretation:Focussingononemeasure(e.g.,sign-tracking)leadstotheconclusionthatassociativelearninghadproceededmorereadilyinonesetofratsthantheother,whilefocussingonthesecondmeasure(e.g.,goal-tracking)leadstotheoppositeconclusion.62 Asitstands,theRescorla-Wagnermodelisunabletoexplainwhy,foranygivenrat,oneresponsewasstrongerthantheother,andwhyinsomeratsgoal-trackingwasstrongerthansign-trackingwhereasinotherratsthisrelationshipwasreversed.Infact,theseresultsposeaproblemforanytheoryoflearningthatassumesamonotonicrelationshipbetweenasingleconstructthatrepresentslearningandacquiredbehaviour(e.g.,Gallistel&Gibbon,2000;Stout&Miller,2007).3.3HeiDI:Rationale,architectureandoverarchingassumptionsThepurposeofHeiDIistooeranaccountinwhichtheassociativestructuresthatareacquiredduringPavlovianconditioningareintegratedwithananalysisofhowtheknowledgeembodiedinthesestructuresdeterminesthenatureoftheresponseselicitedbyaCS,andtheirrelativestrengths.Indoingso,themodelseekstoaddresschallengestotheRescorla-Wagnermodel,andothermodelsofPavlovianlearning(e.g.,Mackintosh,1975;Pearce&Hall,1980;Wagner,1981).Figure7providesaschematicfortheassociativestructures,towhichIwillaligntheanalysisofthelearningandperformanceequationsthatfollow.Theleft-handpanelshowsthestructureofthemodelbeforeconditioninghastakenplaceandtheright-handpanelshowsthestructureofthemodelafterconditioning.Beforeconditioning,theCSisprincipallylinkedtoasetofunconditionedresponses(r1-r3;e.g.,orienting,leverapproach,rearing),whereastheUSisweaklylinkedtoasetofunconditionedresponses(r4-r6;e.g.,food-wellapproach,chewing,swallowing).UnconditionedlinksfromtheCStor4-r6a

ndtheUStor1-r3areassumedtobeveryweak;andtheweightsofthelinesbetweentheCSandr1-r6andbetweenUSandr1-r6denotetherelativestrengthsoftheseuntrainedorunconditionedlinks.Inthisway,ageneraldistinctionisadoptedbetweenCS-orientedresponses(r1-r3)andUS-orientedresponses(r4-6;seeHolland,1977,1984).Importantly,asaconsequenceofconditioningitisassumedthatreciprocalCS-USand63 ]Figure7:Aschematicforassociativestructuresthatunderpinthetranslationofex-citatorylearningintoperformance.Theleft-handsidedepictsthemodelbeforecon-ditioning(i.e.,theunconditionedstructure),withthedarknessofthelinksindicatingtheirstrength,andtheright-handsidedepictsthemodelafterconditioning(i.e.,theconditionedstructure).ConditioningresultsinchangesinthestrengthofthereciprocalCS-USandUS-CSassociationsbetweennodesactivatedbytheCSandUS(denotedbythehashedlines).US-CSassociationsareacquired,whicharedepictedasthepresenceofhashedlinesintheconditionedstructure.Thegeneralrationaleforthisassumption,whichdoesnotfeatureinotherformalmodelsofPavlovianconditioning(e.g.,Mackintosh,1975;Pearce&Hall,1980;Rescorla&Wagner,1972),isoutlinednext.Amorespecicjusticationisreserveduntilthelearningrulesforthesereciprocalassociationsarepresented.IwillpresentinlatersectionshowtheinclusionofUS-CSassociations,aswellasCS-USassociations,providesthebasisforHeiDItoexplainawiderangeofphenomena:Inparticular,thosethathaveprovendiculttoreconcilewiththeRescorla-Wagnermodel,unequalchangeintheassociativestrengthsofthecomponentsofacompound(e.g.,Rescorla,2000),downshiftunblocking,(e.g.,Dickinson,Hall,&Mackintosh,1976)orthathavebeentakentoprovidesupportformodelsthathaveemphasized\predictiv

eness"(e.g.,Mackintosh,1975;Pearce&Hall,1980).64 TheformationofreciprocalassociationsbetweentheCSandUSnodescreatesafunctionalcellassemblyandenables\resonance"betweenthem:WhentheCSispre-sentedactivationpropagatestotheUSnode,whichispropagatedbacktotheCS(e.g.,Grossberg,1980).Arcediano,Escobar,andMiller(2005),inaseriesofexperimentswithbothhumanandnon-humansubjects,providedevidenceoftemporalintegrationbetweenthepre-sentationoftwodierenttrainingexperiences(Experiment1:outcome!S1andS2!S1;Experiment2(A-C):S2!S1andUS!S1)basedontheelementcommontobothexperiences(S1).Thistemporalintegrationsuggeststheformationnotonlyofspecictemporalforwardassociations(i.e.,S2!S1)butalsoofspecictemporalbackwardassociations(i.e.,US S1).Thereisfurtherevidencethatsuchreciprocalassociationsareacquiredduringfor-wardconditioninginavarietyofpreparations(e.g.,Asch&Ebenholtz,1962;Cohen-Hatton,Haddon,George,&Honey,2013;Honey&Bolhuis,1997;Honey&Ward-Robinson,2002);andacomplementaryliteratureontheconditionsunderwhichUS-CSpairingsresultinconditionedrespondingtotheCS(e.g.,Barnet&Miller,1996;Cole&Miller,1999).Atatheoreticallevel,intypicalPavlovianconditioningprocedures{wheretheCSprecedesbutdoesnotco-existwiththeUS{thememorytraceoftheCSmustbesucienttosupportthedevelopmentofexcitatoryassociations(cf.Barnet&Miller,1996;Gallistel,1990;Miller,Barnet,&Grahame,1995;Wagner,1981).Im-portantly,whilethedevelopmentoftheCS-USassociationincreasesthelikelihoodthatthepresentationoftheCSwillactivatetheUSandtherebyprovoker4-r6,withoutthebackwardassociationstherewouldbelittlechangeinthelikelihoodthattheCSwouldprovoker1-r3.TheCS-USassociationallowsthepresen

tationoftheCStoactivatetheUSnodeandUS-CSassociationallowsactivationoftheUSnodetoincreaseactivationintheCSnode,whichincreasesthetendencyforr1-r3tobecomeactiveasaconsequenceofconditioning.WhenaCSispresented,therearetwosourcesofinformationthatareimmediately65 availabletoananimaluponwhichperformancecouldbebased:TheperceivedsalienceoftheCS,(whichisrelatedtoCS)andtheperceivedsalienceoftheUSthatisactivatedbytheCS(whichisrelatedtoVCS�US).AfullyeectiveCSisheldtoactivatetheUSrepresentationtothevalueoftheperceivedsalienceofthepresentedUS(whichrelatestoUS).HeiDIassumesthatbothofthesesourcescontributetothenatureofperformance(cf.Hull,1949).Inparticular,themodelproposesthattherelativevaluesofCSandVCS�USdeterminehowlearningistranslatedintoperformancethroughtwovalues,RCSandRUS.InadvanceofdescribingindetailhowRCSandRUSarecalculated,forthetimebeingsimplyassumethatifCS�VCS�US,thenRCS�RUS,whereasifVCS�US�CSthenRUS�RCS.ReturningtoFigure7,RCSaectsbehaviourviaconnectionsfromtheCStor1-r6inFigure7,andRUSaectsbehaviourviaconnectionsfromtheUStor1-r6.Themodelassumesthattheprecisenatureofthe(alternative)responsesgeneratedinagivenconditioningpreparationwillbeafunctionoftheinteractionbetweenthenatureoftheCSandUS(Holland,1977,1984).Inthenextsections,IrstpresentthelearningrulesusedbyHeiDItodeterminethedevelopmentofthereciprocalCS-USandUS-CSassociations(Equations1and2);andprovideasimpleruleforcombiningthesevaluesuponpresentationoftheCS(Equation3).Itisworthbrie ynotingthatEquations1and2embodytheideathattheperceivedsalienceoftheCSandUS,andtheirassociativelygenerated

counterparts,thatdetermineslearning.ThissuggestionisconsistentwiththeideathatindividualdierencesintheperceivedsalienceoftheCSandUSplayacentralroleindeterminingindividualdierencesintheexpressionoflearning.Ithenprovideadetailedanalysisofhowthiscombinedvalueisdistributedinperformance(Equations4-6).Thecorrespondingsimulationsoflearningandperformancearethenpresentedineachexperimentalchapter.Finally,IillustratehowHeiDIprovidesanaturalaccountforphenomenathatchallengetheRescorla-Wagnermodel,andhowitprovidesalternativeanalysesforresultsthathaveprovidedthebasisformodelsofPavlovianlearningthatincludelearntchangesinattentionorassociability(e.g.,Mackintosh,1975;Pearce&Hall,1980).66 3.4LearningrulesTheuseofapoolederrortermwasthecentralcontributionoftheRescorla-Wagnermodel,allowingittoprovideareadyaccountoftheconditionsunderwhichexcitatoryandinhibitorylearningoccur.
VCS�US=CS(cUS�XVTotal�US)(1)
VUS�CS=US(cCS�XVTotal�CS)(2)HeiDIadoptsversionsofthepoolederrorterminEquation1andEquation2,fortheformationofCS-USandUS-CSassociations,respectively(alistoftheHeiDIequa-tionsisavailableattheendofthechapter).ThereisrecentevidencethatprovidesdirectsupportforthisfeatureofHeiDIinthecontextofCS-orientedbehaviourandUS-orientedbehaviour:AleverCSthatprovokessign-trackingcanblocktheacquisitionofgoal-trackingtoanauditoryCS,andanauditorystimulusthatprovokesgoal-trackingcanblockacquisitionofsign-trackingtoaleverCS(e.g.,Derman,Schneider,Juarez,&Delamater,2018).However,asIshallshow,whileEquations1and2haveformallyequivalentpoolederrorterms,theirfunctionalpropertiesdierwhenastimuluscom-pou

nd(AB)ispairedwithaUS:Equation1functionsasapoolederrortermfortheA-USandB-US,whereasEquation2functionsasaseparateerrortermfortheUS-AandUS-Bassociations.Brie y,Iwilllatershowhowthisobservationenablestheuseofapoolederrortermtobereconciledwithresultsshowingthatcompound(AB)condi-tioningcanresultinunequalchangesinconditionedrespondingtoAandBdependingontheirpriortraininghistories(e.g.,Allman,Ward-Robinson,&Honey,2004;Rescorla,2000,2001).AnimportantfeatureofEquation1isthattheperceivedsalienceoftheUS(US)sets67 themaximumperceivedvalueoftheUSretrievedbytheCS(i.e.,VCS�US).Similarly,theperceivedsalienceoftheCSinEquation2(CS)setsthemaximumperceivedvalueoftheCSretrievedbytheUS(i.e.,VUS�CS).Theideathattheperceivedsalienceofthedirectlyactivatedandassociativelyactivatedrepresentationsofthestimuli(CSandUS)determineassociativechangereceivesdirectsupportfromresultsreportedbyDwyer,Figueroa,Gasalla,andLopez(2018).Theyexaminedthedevelopmentofa avourpreferencethroughpairinga avourCSwithan8%sucroseUS.Theyobservedthatprecedingthisconcentrationofsucrosebyeither2%sucrose(generatingpositivecontrast)or32%sucrose(generatingnegativecontrast)aectedtheacquisitionofthe avourpreference:The avourpreferencesupportedby8%sucrosewaslargerwhenitwasprecededby2%sucrosethanwhenitwasprecededby32%sucrose.Moreover,whenthechangesintheperceivedsalienceoftheUS(8%sucrose)producedbycontrastweredirectlyassessed,throughtheanalysisoflickingmicrostructure,theydirectlycorrelatedwiththesizeoftheresultingpreferencefortheCS avours.3.4.1ExcitatorylearninganderrorcorrectionEquations1and2aresymmetricalrulesgoverningtheformationofCS-USandUS-CSassociat

ions,respectively.Equation1representsasimplicationtotheRescorla-Wagnerlearningrule(Equation0)anddeterminestheformationofCS-USassociations;andEquation2providestheformallyequivalentruleforUS-CSassociations.WhileEquations1and2areformallyequivalentpoolederrorcorrectingrules,theyhavequitedierentfunctionalpropertiesinconventionalconditioningproceduresinwhichacom-poundoftwoCSs(AB)precedesaUS.Inshort,Equation1functionsasapoolederrorterminconventionalcompoundconditioningprocedures(Rescorla&Wagner,1972)whereasEquation2functionsasaseparateerrorterminsuchprocedures(Bush&Mosteller,1951;Hull,1943).However,itisalsoworthnotingthatthemodelpredictsthatifasingleCSweretobefollowedbyacompoundoftwoUSs(US1andUS2),thentheassociationofUS1withtheCSwouldbeweakerthanifUS1hadbeenpairedwith68 theCSinisolation.ThepredictionthattherewillbecompetitionorovershadowingbetweenthecapacityoftwoUSstobecomeassociatedwithasingleCShasreceivedempiricalsupport(seeMiller&Matute,1998).InEquation1,CSisalearningrateparameterconnedtotheunitinterval0CS1,andc*USdeterminestheasymptotefortheCS-USassociation;whereasinEquation2,USisalearningrateparameterconnedtotheunitinterval0US1,andc*CSdeterminestheasymptotefortheUS-CSassociation.NotethatCSandUSaredimensionlessscalars,butwhentheyserveastheasymptotesforassociativestrengththeyaremultipliedbyaconstantof1inunitsofV(c).TherequirementforcisthathasunitsofV,butthenumericvalueisnotxedbythisrequirement.Theconstantcisassumedheretobe1forsimplicity.However,theasymptoticlimitsoflearningneednotbeUSinEquation1orCSinEquation2,butsomemultip

leofthesevalues.WhentheCSisabsent,CSandc*CSaresetto0andwhentheUSisabsentUSandc*USaresetto0.InkeepingwiththeRescorla-Wagnermodel,CSandUSareassumedtore ecttheperceivedsalienceoftheCSandUS,respectively.AccordingtoEquation1,thestrengthoftheassociationfromtheCStotheUS(i.e.,VCS�US)convergesasymptoticallyonc*US.ThechangeinthestrengthoftheassociationbetweenCSandtheUSonagiventrial(
VCS�US)isdeterminedbytheerrorordierencewithinthepoolederrorterm(c*US{PVTotal�US);andPVTotal�USdenotesthenetassociativestrengthofallofthestimulipresentedonthattrial.DuringsimpleCS-USpairings,excitatorylearningceaseswhenPVTotal�US=c*US;andthelearningrateparameterCSaectstherateatwhichVCSapproachesc*US.Inthiscase,thepoolederrortermmeansthattheacquisitionofassociativestrengthbyagivenstimuluswillbein uencedbytheassociativestrengthofotherstimulithataccompanyit;forexample,whenacompoundoftwostimuli(AandB)ispairedwithaUS.Equation2isthecomplementarylearningrulegoverningtheformationoftheUS-CSassociation.Thechangeinthestrengthofthisassociation
VUS�CSonagiventrialisalsodeterminedbythediscrepancywithinthepoolederrorterm(c*CS{PVTotal�CS).69 CSre ectsthesalienceoftheCS,andPVTotal�CSdenotestheassociativestrengthoftheUS(intypicalconditioningprocedures).LearningceaseswhenPVTotal�CS=c*CS,andthelearningrateparameterUSaectstherateatwhichVUS�CSapproachesc*CS.BecauseinatypicalPavlovianconditioningprocedurethereisonlyoneUS(cf.Miller&Matute,1998),thec*CSvalueofeachCSinacompound(e.g.,AandB)setstheasymptotefortheassociationfromtheUStothatC

S.ThismeansthattheUS-CSassociationswillproceedindependentlyforeachofthecomponentsofacompoundthatispairedwithaUS.Thatis,whileEquation1hasboththeformalandfunctionalpropertiesofEquation(0)andpredictsthesamephenomenaasthatmodel,Equation2hasequivalentformalproperties,butfunctionsinthesamewayasaseparateerrortermduringcompoundconditioning(e.g.,Bush&Mosteller,1951;Hull,1943).HowcouldonetestwhethertheanalysisprovidedbyEquations1and2isaccurate?ConsiderrstthesimplecaseinwhichtwoCSs(AandB)arepresentedtogetherandpairedwithaUS.Undertheseconditions,theassociativestrengthaccruedbyA(VA�US)andB(VB�US)willbelessthanifthesestimulihadbeenseparatelypairedwiththeUS.Aneectknownasovershadowing(e.g.,Mackintosh,1976).However,thestateofaairswillbedierentforthereciprocalassociations(i.e.,VUS�AandVUS�B).Theywillundergothesamechangeinassociativestrengthastheywouldhavedonehadconditioningwitheachoccurredinisolation;becausethec*Aandc*BforstimulusAandBsetseparateasymptotesfortheassociationbetweentheUS-AandUS-B.Ofcourse,thendingthatovershadowingisobservedundersuchconditionsisuninformative;becauseEquations0and1predictionthatVA�USwillbelowerwhenithasbeenconditionedincompoundwithBthanwhenithasbeenconditionedalone.But,nowimaginethesamecompoundconditioningscenario,butthatonthisoccasionapreviousstageoftraininghadestablishedAasconditionedexciter(bypairingitwithaUS)andBhadbeenestablishedasaconditionedinhibitor(bypairingitpairedwiththeabsenceofanotherwisepredictedUS).AccordingtoEquations0and1,providedAandBareequallyintense(i.e.,A=B)thentheyshouldgainequivalentassociative70 strengthasaconsequenceoft

heABcompoundbeingpairedwiththeUS.However,accordingtoEquation2,whiletheassociationbetweentheUSandAwillnotincrease(havingreachedasymptoteduringtherststage)theassociationbetweentheUSandBwillincrease,becausetheUShadnotpreviouslybeenpairedwithB.Ifthechangesinthereciprocalassociationsweretobecombined,thenBshouldhavegainedgreatercombinedassociativestrengththanA.Rescorla(2000,2001)haspublishedaseriesofingeniousexperimentsthathasconrmedthispredictionunderavarietyofcircumstances(seealsoAllman&Honey,2005;Allman,Ward-Robinson,&Honey,2004;Holmes,Chan,&Westbrook,2019).Iwillprovideaformalsimulationoftheanalysisoftheseresults,whichhavebeentakentoimplicateseparateerrortermsinPavlovianconditioning,oncetherulesforcombiningthereciprocalassociationshasbeendescribed,andthewayinwhichassociativestrengthaectsperformancepresented.Asjustnoted,CS-USpairingscreateafunctionalcellassemblythroughreciprocalassociationsbetweentheCSandUS.Tocapturethisinteractionandtosimplifytheperformancerules,itisdesirabletocombinethenetassociativestrengthsoftheCS-USassociationreturnedbyEquation1(forVCS�US)andtheUS-CSassociationreturnedbyEquation2(forVUS�CS).Thecombinedassociativestrengthwithinthisassembly(VCOMB)isgivenbyEquation3a2.Thecombinedassociativestrengthofacompoundstimulus(VCOMB�AB)composedoftwoCSs(AandB)isgivenbyEquation3b;inwhichPVAB�USisthesumofVA�USandVB�US,andVUS�AandVUS�BarethestrengthsoftheassociationsbetweentheUSandA,andtheUSandB.VCOMB=VCS�US+(1 cVCS�USVUS�CS)(3a)VCOMB�AB=XVCS�US+[1 cXVAB�US(VUS�A+VUS�B)](3b) 2ThesymmetricalcombinationrulescanbeusediftheUS(ratherthanth

eCS)wastestedalone,e.g.,VUS�CS+(1/c*VUS�CS*VCS�US)71 ThischoiceofcombinationrulerecognizesthefactthatwhiletheCSdirectlyacti-vatestheCS-USassociation,theUS-CSassociationisonlyindirectlyactivatedbythepresentationoftheCS.Therulehasthegeneralpropertythatthedirectlyactivatedlinkinachainofassociationswillconstraintheimpactoftheindirectlyactivatedlinkonperformance.ThisistrueregardlesswhetherthestimulusisaCSoraUS.IftheCSactivatestherepresentationoftheUS,VCS�USwillconstraintheimpactonperfor-manceofVUS�CS,howeveriftheUSactivatestherepresentationoftheCS,VUS�CSwillconstraintheimpactonperformanceofVCS�US.Forsimplicity,whenpresentingaCS,ifVCS�USwas0andVUS�CSwaspositive,thenVCOMB(andhencebothRCSandRUS)0inspiteofthefactthattherelationshipbetweentheCSandUShadbeenencoded(i.e.,asVCS�US).ThesignicanceofthispropertyinthecontextofHeiDIwillbecomeapparentwhentheblockingphenomenonisconsideredingreaterdetail.(e.g.,Kamin,1969).3.4.2ExtinctionWhenconditioningtrialswithaCSarefollowedbyextinctiontrialswheretheCSispresented,butnoUSoccurs,c*USissetto0andPVTotal�USwillbepositive.Undertheseconditions,Equation1returnsanegativevaluefor
VCS�US,butEquation2returns0for
VUS�CS(becauseUS=0).Itisworthhighlightingthisasymmetrybetweenwhatislearnedduringconditioningandextinction:excitatorylearninginvolveschangestoVCS�USandVUS�CS,butconventionalextinctionproceduresinvolveonlychangestoVCS�US.ThenegativevaluesreturnedbyEquation1duringextinctioncanbeinterpretedintwoways:First,theycoulddenotethegrowthofnegativeassociativestrength(Konorski,1948;Rescorla&Wagner,1972;Wagner&Rescorla,1972

).Second,theycoulddenotetheformationofanexcitatoryassociationbetweentheCSanda`NoUS'node,whichinturninhibitstheUSnodeandtherebyreducesconditionedbehaviour(seeKonorski,1967;Pearce&Hall,1980;Zimmer-Hart&Rescorla,1974).In72 therstcase,thenegativevaluesaredirectlyre ectedintheunderpinningassociativestructure,andinthesecondcasetheyre ecttheproductofanexcitatoryCS-NoUSassociationmultipliedbyaninhibitoryNoUS-USassociation.However,accordingtobothinterpretations,thenetassociativestrengthoftheforwardassociationinvolvingtheCS(VCS�US)isthesumofthepositiveandnegativeassociativevaluesreturnedbyEquation1;andthenetassociativestrengthofVCS�USisthesumofthepositiveandnegativevaluesreturnedbyEquation2.NegativevaluesofVUS�CSwillbereturnedbyEquation2whenPVTotal�CS�c*CS.ThissituationwouldarisewhentheUSispresentedaloneafterconditioninghastakenplaceorifadditionalUSswerepresentedintheinter-trialintervalsbetweenCS-USpairings.BothofthesemanipulationsresultinareductioninconditionedrespondingtotheCS(seeRescorla,1968,1973)3.Infact,accordingtoHeiDIwhilebothofthesemanipulationswillresultinextinctionoftheUS-CSassociation,addingUSpresentationsduringtheintervalsbetweenCS-UStrialsallowtheformationofacontext-USassociation,whichshouldblockthedevelopmentoftheCS-USassociation.Inkeepingwiththisanalysis,ithasbeenarguedthattheeectsofmanipulatingCS-UScontingency,byaddingUSalonepresentationsduringconditioning,mightbemultiplydetermined(Baker&Mercier,1982).Latersimulationswillconrmthedescriptionoftheconsequencesofextinctionpre-sentedinthepreviousparagraph.Forthetimebeing,itisimportanttonotethataccordingtoEquations1and2,ext

inctionleavesasignicantcontributiontoperfor-mancecompletelyunchanged(i.e.,theUS-CSassociation,VUS�CS),ratherthansimplybeingcounteractedbyadditionalinhibitorylearning,asisthecasewiththenetCS-USassociation(VCS�US).ThisfeatureofHeiDIisconsistentwiththegeneralobservationthatpost-extinctionmanipulationscanrevealthepresenceofresidualexcitationinper-formance,whichhasrepresentedanongoingchallengetotheRescorla-Wagnermodel(e.g.,Bouton,2004).ManipulationsthatenabletheUStobeactivated(orthatdisrupt 3ThereisalsoevidencethatwhenCS-USpairingsarefollowedbyseparatepresentationsofthesameUSbutatahighersalience(calledUSin ation)theCRtotheCSisamplied(Bouton,1984;Rescorla,1973).Undertheseconditions,inadditiontoanyreductioninnetVUS�CS,presentationsofahighersalienceUSmightchangetheresponseunitsactivatedbytheUS,whichcouldaectlaterperformancetotheCS.73 theCS-NoUSassociation)willresultinareturninperformancetotheCS.3.4.3InhibitorylearningIfconditioningtrialsinwhichstimulusAispairedwithaUSareintermixedwithtrialsonwhichAispresentedwithstimulusBandtheUSisnotdelivered,thennon-reinforcedABtrialswillresultinareductioninthenetassociativestrengthofA,andBwillbecomeanetinhibitor.ThenetassociativestrengthofABisgivenbyaddingthepositiveandnegativevaluesreturnedbyEquation1forstimulusAandB.ThenetassociativestrengthoftheUS,VUS�CS,isthesumofthepositiveandnegativeassociativevaluesreturnedbyEquation2.AccordingtoEquation2,onnon-reinforcedABtrialstherewillbenochangeintheUS-AorUS-Bassociations;againbecauseUS=0.However,inhibitorylearningcanalsobeproducedifABispairedwithaUSthatissmallerinmagnitudethantheUSthatispairedwithA(e.g.,Cotto

n,Goodall,&Mackintosh,1982).Undertheseconditions,US�0andHeiDIpredictsthattherewouldbeanincreaseintheexcitatorystrengthoftheUS-AandUS-Bassociations,whichwouldcontributetothevaluesofVCOMBforA,BandAB.Thepredictionthatconventionalconditionedinhibitiontrainingandconditionedinhibitionproducedbyareductioninreinforcermagnituderesultindierentassociationstructureshasnotbeenevaluated.3.5PerformancerulesWhenaCSispresentedtherearetwosourcesofinformationthatareavailabletoananimal,theperceivedsalienceoftheCS(relatedtoCS)andtheassociativestrengthofthatCS(VCS�US),whichcanbeconsideredanestimateofUSgivenitsrelationshipwithrelationshipwithc*US.Thesetwosourcesofinformationareheldtodeterminethenatureofconditionedbehaviour.74 RCS=cCS cCS+jVCS�USjVCOMB(4)RUS=jVCS�USj cCS+jVCS�USjVCOMB(5)Equation4andEquation5generatetwovalues,RCS,andRUS,whichre ecttheproportionsofCSandVCS�US.Asbefore,intheseequations,CSismultipliedbycinordertotranslateitintounitsofV.TodeterminetheextenttowhichtheseproportionsaecttheabsolutelevelsofperformancetheyaremultipliedbytheoverallassociativepropertiesoftheCS-USensemble(i.e.,VCOMB).RCSaectsbehaviourviaconnectionsfromtheCStor1-r6,andRUSaectsbehaviourviaconnectionsfromtheUStor1-r6(seeFigure7).BecauseinthesimulationspresentedherenetVCS�US�0,therealvaluesofVCS�UScanbeusedtodetermineRCSandRUSinEquations4and5.However,toaddressthefactthatEquation1(andEquation2)canreturnnegativevalues,theuseofabsolutevaluesensuresthattheproportionsinEquations4and5are1.Thischoicealsoleavesopenthepossibilitythatanetinhibitorco

uldprovokerespondingwhenpresentedalone(cf.Konorski,1967;Pearce&Hall,1980),ratherthanhavingnoeectonperformanceunlessitispresentedwithanexcitor(Konorski,1948;Wagner&Rescorla,1972).Fornow,itissucienttonotethatEquation4returnsahighervalueforRCSasthevalueofc*CSincreasesrelativetothevalueofVCS�US,andEquation5returnsahighervalueforRUSasVCS�USincreasesrelativetoc*CS.Thesetwoequationsarereadilyextendedtoaccommodatestimuluscompounds(AB).Todoso,thec*valuesforAandBaresimplycombined(e.g.,added)toformc*AB,andthenetVsofAandBarecombined(e.g.,added)toformVAB�US.Similarly,agivenstimulus(CSorUS)canbeconceivedofasasetofelementswiththeirownc*valuesandnetVs,whichcouldbeenteredintoEquations4and5usingthesameapproach(cf.Atkinson&Estes,1963;Delamater,2012;Wagner&Brandon,1989).WhileEquations4and5provideasimplebasisforthedistributionoftheassociative75 propertiesoftheCS-USensemble(i.e.,VCOMB)totheresponse-generatingunits(r1-r6)thoughRCSandRUS,theydonotspecifyhowtheseresponseunitsonceactivatedaectbehaviour.OnesimplepossibilityisthatagivenvalueofRCS,forexample,resultsinthesameamountofCS-orientedresponding(r1-r3)irrespectiveofthevalueofRUS.Thispossibilityequatestotherebeingparallelactivationoftheresponse-generatingunits(r1-r6),andisformallyexpressedinEquation6,whereRCSandRUSaretranslatedintodimensionlessvaluesbybeingmultipliedbythereciprocaloftheconstant,\c".AccordingtoEquation6,theactivationofagivenresponseunit(e.g.,r1)issimplydeterminedbyaddingtheproductsof(i)multiplyingRCSbytheconnectionbetweentheCSandr1(CS-r1),and(ii)multiplyingRUSbythestrengthofconnectionbetweentheUSandthesameresponseunit(e.

g.,US-r1).Therearemorecomplexpossibilitiesinvolvingtheinteractionbetweentheactivationvaluesoftheresponse-generatingunits(e.g.,McClelland&Rumelhart,1981),butfornowthisservesassimpleplaceholderforfuturetheoreticalelaboration.r1=(1 cRCSCSr1)+(1 cRUSUSr1)(6)ThesimulationspresentedinlatersectionsarederivedfromEquations1-5.Equation6simplyinvolvesmultiplyingtheresultingRCSandRUSvaluesbythexedstrengthlinks(withvaluesbetween0and1)betweentheCSandUSnodesandtheresponseunits(e.g.,theCS-r1andUS-r1links).Ifthetwosetsoflinksareequivalent(seeFigure7),thendierencesinactivationoftheresponseunitswilldependsolelyonRCSandRUS.3.5.1IndividualdierencesinUSHeiDIassumesthatCSandUSarexedforagivenCSandUSinagivenanimal,butproposethattheperceivedsalienceoftheCS(relatingtoCS)andUS(relating76 toUS),andhencec*CSandVCS�USinEquations4and5,canvarybetweenani-mals.ThisassumptionprovidesthebasisforindividualdierencesinRCSandRUS,becausec*CSandVCS�USaectperformanceaccordingtoEquations4and5(remem-berVCS�USconvergesonc*USatasymptote)4.Thisanalysisreceivessupportfromtheobservationthatrodentswhoshowedastronglikingforsucrose(asmeasuredbylickingmicrostructure;seeDwyer,2012)aremorelikelytobegoal-trackers(whensucrosewastheUS)thanthosewhoexhibitedaweakerlikingforsucrose(Patituccietal.,2016).IndividualvariationinthepalatabilityofsucrosecanbereadilyalignedtodierencesinUSthatwillaectbothlearning(i.e.,theasymptoticvalueofVCS�US)andtherateatwhichVUS�CSreachesasymptote(throughEquations1and2)andthedistribu-tionofVCOMBinperformance(throughVCS�USinEquatio

ns3-6).Moreover,Dwyeretal.(2018)showedthatindividualdierencesinthepalatabilityofsucrose(duringtheirexperimentsinvolvingcontrasteects)werepositivelycorrelatedwiththe avourpreferencelearning.ThereisadditionalevidencethatisconsistentwiththepropositionthatUSfordierentUSsvariesbetweenanimals,andindeedwithinagivenanimal:WhenseparatepresentationsoftwoleversarepairedwiththesameUS(e.g.,foodorsucrose)thenthebiastowardssign-trackingorgoal-trackingononelevercorrelateswiththebiasontheother,resultspresentedinExperiment2.However,whenthepresentationofoneleverispairedwithsucroseandtheotherleverispairedwithfood,thenthereisnocorrelationbetweenthebiasonthetwolevers(Patituccietal.,2016).ThispatternofresultsisconsistentwiththeviewthattheUSvaluesfortwoUSs(i.e.,foodandsucrose)canvarybetweenanimalsandwithinagivenanimal.Furtherevidence.AcentralpropositionofHeiDIisthatvariationinVCS�USinteractswithc*CStodetermineperformance.ThispropositionreceivessupportfromtheresultspresentedinExperiment1,inwhichaCSisrstpairedwithaUSisthenpresentedaloneacrossaseriesoftrials.ExtinctiontrialsshouldaectnetVCS�US,conditionalon 4Equations4and5canbetransformedforthecaseinwhichtheUSispresentedalone:Undertheseconditions,c*USreplacesc*AandVUS�CSreplacesVCS�US77 thereductionofc*USfromapositivevalueto0inEquation1,butnotCS.TheclearpredictionisthatwhilebothRCSandRUSshoulddecreaseduringextinction(VCOMBwillre ectthereductioninVCS�US;seeEquation3),Equations4and5predictthatthisdecreasewillbelessmarkedforRCSthanforRUS:CSwillremainthesameandVCS�USwillbelower.Thispredictionwasconrmedinrat

sthatweredesignatedaseithersign-trackersorgoal-trackersinExperiment1.Inbothgroups,thetendencyforratstointeractwiththelever(sign-tracking)extinguishedlessrapidlythaninteractingwiththefood-well(goal-tracking).Theresultsfromarelatedconditioningpreparationprovideconvergingevidencefromtheproposedinteractionbetweenc*CSandVCS�USindeterminingRCSandRUS.KayeandPearce(1984)gaveratspresentationsofalocalizedlightthatwaseitherpairedwiththedeliveryofafoodpelletoneverytrial(ingroupcontinuous)oronarandomlyscheduled50%ofoccasionsonwhichitispresented(ingrouppartial).Theyobservedthatwhenthelightwascontinuouslyreinforceditmaintainedahigherlevelofgoal-tracking(food-wellentries)andalowerlevelofsign-tracking(orientingandapproachtothelight)thanwhenthelightwaspartiallyreinforced(seealso,Anselme,Robinson,&Berridge,2013).AccordingtoEquation1and2,netVCS�USwillbehigherduringacontinuousthanapartialreinforcementschedule,andacontinuousreinforcementshouldresultinagreaterbiastowardsgoal-tracking(RUS)andasmallerbiastowardssign-tracking(RCS)thanpartialreinforcement,whichcouldresultintheoppositebias(seeEquations4and5).However,KayeandPearce(1984)alsoobservedthatsign-trackingwashigherinabsolutetermsduringpartialthancontinuousreinforcement.Thisndingmightre ectthefactthathighlevelsofgoal-tracking,duringcontinuousreinforcement,weremorelikelytointerfere(atthelevelofresponseoutput)withsign-trackingthanthelowerlevelsofgoal-trackingengenderedbypartialreinforcement(seediscussionofEquation6).Inanycase,thefactthatCS-orientedbehaviourismaintainedbypartialreinforcementshouldalsoimproveananimal'slaterabilitytodetectnewrelationshipsinvolvingthatCS(cf

.Pearce&Hall,1980;Wilson,Boumphrey,&Pearce,1992).78 3.6SimulationsoflearningandperformanceSimulationsweredoneinaShinydashboardapp(HeiDI)usingRprogram-minglanguage.ThecodeisavailableontheOpenScienceFramework(OSF;https://osf.io/j8tps/?view only=c7e9286d80464cf1a244c9cface69b4e).Thesimulationgurespresentedinthethesisarebasedonthemodelfunctionscreatedintheapp(`model.R'),howeverthegraphicalrepresentationhasbeenadaptedtoincludeseveralsimulationsinthesamegure.ThecodeforthesimulationspresentedinthethesiscanbefoundontheOSFwebsite(https://osf.io/h4fyg/),morespecicallyat`The-sis/Simulations'.ThesoftwareusedfortheappwasR(version3.3.3)andRStudio(Version1.1.463.).ToreproducethesimulationsdownloadandinstallRsoftwarefortheappropriateoper-atingsystem.AftersuccessfullyinstallingR,downloadandinstallRStudio.Next,open`install packages.R'leandrunthelewhichinstallsalltheneededpackages.Theonlineliveversionoftheappincludesa`Readme'(READMEfolder)lewithdetailedinstructionsforreproducingthecode.Thisdocumenthasatimestamp(everytimeitiscompiledthedateisupdated)andanychangesfromapreviousversionaredetailedin`log'folder.Eachupdateoftheapphasandwillhaveanumberedlogle.IntheAppendix(HeiDIapp)Iincludeamoredetailedexplanationoftheappstructureandles.3.6.1ExcitatoryconditioningandextinctionFigure8showssimulationsofacquisitionofconditioning(leftsideofeachpanel,Trials1-6)andextinction(right-sideofeachpanel,Trials7-12).PanelsAandCde-pictsimulationsofthedevelopmentoftheCS-USassociationderivedfromEquation1(VCS-US),theUS-CSassociationderivedfromEquation2(VUS-CS)andtheircom-binedvalues(VCOMB).MaximumVCS-USisdeterminedbyc

*USandmaximumVUS-CSisdeterminedbyc*CS.ThelearningrateatwhichVCS-USreachestheasymptoteis79 determinedbyc*CS(Equation1).ThelearningrateatwhichVUS-CSreachestheasymptoteisdeterminedbyc*US(Equation2).PanelsBandDshowhowEquations4and5translatelearningintotheperformancevaluesofRCS(CS-orientedresponses)andRUS(US-orientedresponses)acrossaseriesofCS-USpairings.SimulationswereconductedinwhicheitherRCSdominatedperformanceduringcon-ditioning(PanelAandB;CS=.50andUS=.30;thisbiaswouldbeconsistentwithasign-tracker)orRUSdominatedperformance(panelsCandD;CS=.30andUS=.50;thisbiaswouldbeconsistentwithagoal-tracker).WhenCS�US,RCSdominatedRUS,butwhenUS�CSthenthereverseisthecase.Thegeneralconclusionisthatifc*CS=VCS-US,thenEquations4and5returnthesameorsimilarcontributionsforRCSandRUS;butifc*CS6=VCS-USthenthecomponentwiththelargestvalue(c*CSorVCS-US)contributesproportionatelymoretoperformance.Inallofthesimulationsthatfollow,itisassumedthattheconstant(c)is1inunitsofV.Therefore,thevaluesofCSandUSarethesameasthoseofc*CSandc*US,respectively.Foreaseofpresentation,IwillonlyrefertoCSandUSwithoutexplicitreferencetothetransformationsthatappearinEquations1-6.StartingwithpanelsAandB,whenCSandUShavebeenarrangedforRCStodominateperformance,itisclearfrompanelAthatduringconditioningVUS�CS�VCS�US(whenCS=.50andUS=.30),andthatVCOMBissimilartoVUS�CS.Duringextinction,inspectionofpanelAshowsthatVCS�USandVCOMBdecline,butVUS�CSdoesnot(becauseUS=0).PanelBshowsthatthereductioninRCSis(numerically)lessmar

kedthanRUS.MovingtopanelsCandD,whenCSandUShavebeenarrangedforRUStodominateperformance,duringconditioningVCS�US�VUS�CSandVCOMB�VCS�US.Duringextinction,VCS�USandVCOMBdecline,butVUS�CSdoesnot.PanelDshowsthatthereductioninRCSoccursmuchlessrapidlythanthereductioninRUS.Forbothtypesofsimulatedrats,STs(PanelAandB)andGTs(PanelCandD),HeiDIpredictsthatUS-orientedbehaviour(goal-tracking)declinemorerapidlythanCSorientedbehaviour(sign-tracking).80 InExperiment1,ratsreceivedtraininginwhichonelever,(L1)wasfollowedbyareinforcerandanotherlever(L2)wasnon-reinforced.Duringthereversalstagethecontingenciesontheleverwerereversed:L1wasnon-reinforcedandL2wasnowrein-forced.TheresultsofExperiment1conrmHeiDI'sprediction,goal-trackingchangedmorerapidlythansign-trackingwhenthecontingencieswerereversed.Notethatinthesimulations,theeectsofreversingcontingenciesatthebeginningoftrial6,inthesim-ulationsareevidentintrial7.Thisisbecausetrial7showsthechangesinassociativestrengthderivedfromeventsduringtrial6. Figure8:Simulationsoftheeectsofanextinctionprocedureonsign-trackingandgoal-tracking:Simulationforconditioning(trials1-6)andextinction(trials7-12)forthereinforced(+)lever.PanelsAandCdepicttheoutputvaluesforVCS-US,VUS-CS,VCOMB,andpanelsBandDshowthecorrespondingoutputvaluesforRCSandRUS.TheparametersduringconditioningwerechosentoresultinabiastowardsRCS(i.e.,CS=.50andUS=.30;panelsAandB)orabiastowardsRUS(i.e.,CS=.30andUS=.50;panelsCandD).Duringextinction,USwassetto0.81 3.6.2ChangeinUSvalueInExperiment2ratsreceivedduringtrainingseparatepresentationsoftwolevers(L1andL2),thatwer

ebothpairedwiththesamereinforcerduringtraining(eitherfoodpelletsorsucrose).Workinourlaboratoryindicatedthatfoodpelletsmaintainhigherlevelsofbothleverpressingandfood-wellentriesthandoessucrose;whichshouldbeevidentintherststageoftraining.Duringthesecondstage,thereinforcersassociatedwiththetwoleverswereswitched:theratsgivenpelletsduringtrainingreceivedsucroseduringtheswitchandthosegivensucroseduringtrainingreceivedfoodpelletsduringtheswitch.Becausethetransitionbetweenreinforcersdoesnotaectthestimulus(CS)inanyobviousway,changesinperformancemustre ecttheincreaseordecreaseinUSaectsperformance.Thechangefromsucrose(lowUS)tofoodpellets(highUS)increaseUS,whereasthechangefromfoodpellets(highUS)tosucrose(lowUS)decreaseUS.ResultsfromExperiment2conrmthispredictionasithasbeenseenthatgoal-trackingwasmoreaectedbyachangeinUSvaluethansign-tracking.Figure9showssimulationsofacquisitionofconditioning(leftsideofeachpanel,Trials1-6)andswitchinUSvalue(right-sideofeachpanel,Trials7-12).PanelsAandCdepicttheoutputvaluesforVCS-US,VUS-CS,VCOMB,andpanelsBandDshowthecorrespondingoutputvaluesforRCSandRUS.TheparameterswerechosentoresultineitheranincreaseinUSvalueaftertheswitch(CS=.50,US(before)=.30andUS(after)=.50;PanelAandB)oradecreaseinUSvalue(CS=.50,US(before)=.50andUS(after)=.30;PanelCandD).ForHeiDImanipulationsthatchangeUSshouldaectgoal-trackingand,asasecondaryconsequence,sign-tracking.InspectionofpanelsBandDshowsthatRUSismoreaectedbythechangeinUSvaluethanRCS.ThesesimulationsareimplementedinanopensourceappHeiDIapp.Moredetailsabouthow

IconductedthesimulationscanbefoundintheAppendix,sectionSimulations.InChapter4IpresentanddiscusssimulationsderivedfromHeiDIforinhibitoryconditioningandindividualdierencesinUSvalue.Fornow,IdiscusshowusingapoolederrorterminEquations1and2enablesHeiDItoexplaineectsthathaveposed82 achallengeforRescorla-Wagnermodel. Figure9:SimulationsoftheeectsofachangeinUSvalueonsign-trackingandgoal-tracking:Simulationforconditioning(trials1-6)andaftertheswitchinUSvalue(trials7-12).PanelsAandCdepicttheoutputvaluesforVCS-US,VUS-CS,VCOMB,andpanelsBandDshowthecorrespondingoutputvaluesforRCSandRUS.TheparameterswerechosentoresultineitheranincreaseinUSvalueaftertheswitch(CS=.50,US(before)=.30andUS(after)=.50;PanelAandB)oradecreaseinUSvalue(CS=.50,US(before)=.50andUS(after)=.30;PanelCandD).3.6.3CompoundconditioningandthepoolederrortermHeiDIprovidesapotentialreconciliationoftheuseofapoolederrortermswiththeobservationthatstimuliwithdierentassociativehistoriesappeartoundergounequalchangewhentheyareconditionedincompound;anobservationthatwastakentobeinconsistentwiththeRescorla-Wagnermodelanditssuccessors(seeHolmesetal.,2019).Torecap:Inonesetofexperiments,Rescorla(2000)initiallytrainedratswithtwoexcitatorystimuli(AandC),eachfollowedbyreinforcement(A+,C+)aswellastwoinhibitorystimuli(BandD)eachnon-reinforcedinthepresenceofX(X+;XB-;XD-).Thenacompoundconsistingofoneexcitorandoneinhibitorisreinforced(AB+).83 Testingisconductedwithtwocompoundseachcontainingoneexcitorandoneinhibitor.Thetestcompoundsdierinwhethertheycontaintheexcitor(AD)orinhibitor(BC)fromtheABreinforcementphase.LetassumethatAandCboth

hadexcitatoryassociativestrengthof.50,andBandDbothhadinhibitoryassociativestrengthof-.50beforethecompound,AB,waspairedwiththeUS(i.e.,AB-�US).AccordingtoEquations0and1,theassociativestrengthofbothshouldincreaseanequivalentamount:Afrom.50to.75andBfrom-.50to-.25.ThiswouldmeanthattheADcompoundshouldhaveanassociativestrengthof.25(.75+-.50)andtheBCcompoundshouldalsohaveanassociativestrengthof.25(.50+-.25).However,accordingtoHeiDIonealsoneedstoconsiderthefateofthebackwardassociationsduringcompoundconditioning:betweentheUSandA,andbetweentheUSandB.Ifitisassumedthatforallstimuliis.30,thenVUS�Awillbe.30bytheendoftherststageoftraining,butVUS�Bwillbe0,becauseBhasnotbeenpairedwiththeUS.ThiswillmeanthatwhileVUS�AwillnotchangeduringpairingsofABwiththeUS(theasymptoteforVUS�Adeterminedby=.30willhavebeenreachedasaresultoftherststageoftraining),VUS�Bcanincrease(e.g.,from0to.30).Thiswillmeanthatduringthetest,VCOMB�BCwillbehigherthanVCOMB�AD.Thisanalysisretainsapoolederrortermforallassociations,butrecognizesthefact{hithertounacknowledged{thatassociationsfromtheUStoAandBwillproceedindependentlyofoneanotherinconventionalconditioningprocedures(i.e.,whenthereisonlyasingleUS).Simulationsconrmtheaccuracyofthisanalysisacrossabroadrangeofparameters5,butintheinterestsofconsistencytheparametersweresetinthewaydescribedinthepreviousparagraph:ThevaluesofA,B,CandDweresetat.30;bytheendofstage1,VA�USandVC�USwere.50(i.e.,US=.50)whileVB�USandVD�USwere{.50;andVUS�AandVUS�Cwere.30,whereasVUS�BandVUS�Dwere0.Havingsettheseparameters,IthensimulatedhowtheCS-USassociationsi

nvolvingAandBchangedduringconditioningwiththeABcompound(seeFigure10).InspectionofPanelAinFigure10conrmsthatVA�USandVB�USincreasedbyequivalentamounts,andthat 5ThesesimulationsareimplementedintheHeiDIappas\Poolederrorterm".84 whileVUS�Aremainedthesame,VUS�Bincreasedto.30.PanelBinFigure10showshowtheassociativestrengthsofADandBCchangewhenthechangesinvolvingAandBwereaddedtotheexistingstrengthsofDandB,respectively.InspectionofPanelBinFigure10conrmsthatthenetVAD�USandVBC�USincreaseequivalentlyasaconsequenceofABconditioningtrials(theblacksymbolsoverlapwithoneanother).However,whileVUS�BCincreases,VUS�ADdoesnot.PanelCinFigure10showsthattheVCOMB�BCisgreaterthanVCOMB�AD,re ectingthegreatercontributionofVUS�BCtoBCthanVUS�ADtoAD.Finally,PanelDinFigure10revealsthatthedierencebetweenBCandADisevidentinbothRUSandRCS;butinabsolutetermsismostevidentforRCS.Thisdierencere ectsthefactthatwiththeparametersemployedintheillustrativesimulation,thecombinedalphascores(ADandBC=.60)aregreaterthantheVAD�USandVBC�US(both=.25).Whenotheraspectsofthesimulationareheldconstant,butforallstimuliwassetat.10(i.e.,ADandBC=.20),theabsolutedierencebetweenBCandADis(approximately)equallyevidentforRUSandRCS.RescorlaandWagner(1972)isunabletoaccountforthefactthatreinforcementofacompound(AB+)aectsdierentlyapreviousexcitor(A)oraninhibitor(B)withoutweakeningassumptionsgoverningthelearningrulesoraddingmorecomplexitytothemodel.HeiDIcombinesapoolederrortermfortheforwardassociationsandseparateerrortermsforbackwardsassociations.Thisway,HeiDIisabletoaccommodatediere

ntialincreasesordecreasesinassociativestrengthforexcitorsandinhibitors,whichmodelsbasedonpurelycombinederrortermareunabletoaccountfor.3.6.4CombiningstimuliwithdierentassociativehistoriesRescorlaandWagner(1972)madethesimplifyingassumptionthattheassociativestrengthofacompoundstimulus(VAB�US)issimplythesumoftheindividualassocia-tivestrengthsofAandB(i.e.,VA�US+VB�US).TogetherwiththeassumptionthatVbearsanordinalrelationshiptoperformance,themodelisconstrainedtopredictedthattherewillbeanordinalrelationshipbetweenperformancetoA,BandAB.Forexam-85 Figure10:SimulationsforHeiDIofthechangesinassociativestrengthwhencombiningstimuliwithdierentassociativehistories.Aconditionedexcitor(A)andaninhibitor(B)areconditionedincompound(AB),andtestedwithaninhibitor(D)andexcitor(C)incompoundsADandBC.PanelAshowstheoutputvaluesforchangesinassociativestrengthofthecomponents(AandB)ofastimuluscompound(AB)thatispairedwithaUS.StimulusA(andC)begincompoundconditioningwithaVCS�USof.50,andVUS�CSof.30;whereasB(andD)beginwithaVCS�USof-.50andVUS�CSof0.PanelBdepictstheoutputvaluesforthetestcompounds:VAD�US,VUS�AD,VBC�USandVUS�BC.PanelCshowstheoutputvaluesforthecombinationoftheforwardandbackwardassociationsforAD(VCOMB�AD)andBC(VCOMB�BC),whilepanelDillustrateshowthedierencesinVCOMB�ADandVCOMB�BCarere ectedintheoutputvaluesforRCS(CS-orientedbehaviour)andRUS(US-orientedbehaviour)duringthetestcompoundsADandBC.ple,iftwostimuliwithexcitatoryassociativestrengtharecombinedthenperformancetothecompoundABshouldexceedbothAandB;whereasifonestimulusisexcitatory86 (A)andtheother(B)isuntrained(andwithoutassociat

ivestrength)thenperformancetoABshouldmatchA,andbothshouldexceedB.Finally,ifAisexcitatoryandBinhibitorythenperformancetoABshouldbelessthanAandgreaterthanB,unlesstheexcitatoryvalueofAwaslessthanorequaltotheinhibitoryvalueofB.WhilethepredictionsofHeiDIandtheRescorla-Wagnermodelmirroroneanotherinsomeofthesecases,theydivergeinothers.SummationTheanalysisbeginswiththerstexample,wheretwoCSs(AandB)thathavebeenseparatelypairedwithUSarepredictedtosummatewhentheyarecombinedattest.Equations1and2wereusedtogeneratetherequisiteindividualVsforstimulusAandB,andEquations4and5todetermineperformance.First,itwasconrmedthatsummationwasevidentinbothRCSandRUSirrespectiveofwhethertheparameterswerechosentoresultinabiastowardstheRCS(e.g.,AandB=.50,andUS=.30),orRUS(e.g.,AandB=.50,andUS=.70).However,atanempiricallevel,summationisnotaninevitableconsequenceofpresentingtwoexcitatorystimuliincom-pound.Thecircumstancesunderwhichsummationdoesanddoesnotoccurhaveyettobefullydetermined(Pearce,Aydin,&Redhead,1997;Pearce,Redhead,&George,2002),withtheoreticalanalysestendingtofocusonhowthecombinationorcongura-tionofstimulichangesthewayinwhichtheyareprocessed(e.g.,Brandon,Vogel,&Wagner,2000;Pearce,1994).Iwillnotcommentfurtherhereonthenatureofsuch\congural"processes,buttheaforementionedtheoreticalanalysesmakeanimportantassumption:SeparateconditioningtrialswithAandBresultsinthemacquiringassocia-tivestrength(relatively)independentlyofoneanother(seeBrandonetal.,2000;Pearce,1994).HeiDIdoesnotmakethisassumption,andthisfacthasimportantimplicationsfortheconditionsunderwhichsummationwillbeobserved.HeiDIassumesthatas

sociationsformfromtheUStotheCS.UnlikethedevelopmentofA-USandB-USassociations,whichproceedsindependently,thenetUS-Aassociation87 willbeweakenedonatrialonwhichBispairedwiththeUSandthenetUS-BassociationwillbeweakenedonatrialwhenAispairedwiththeUS.Thisfactdoesnotinitselfaectthepredictionthatsummationwillbeobserved(simulationsincludedthesereciprocalassociations),howeveritdoesraisethepossibilitythatanotherformoflearningwilloccurthatcouldconstrainsummation.TotheextentthattheA-USandUS-BassociationsenabletheBnodetobecomeactiveonatrialwithA,andtheB-USandUS-AenabletheAnodetobecomeactiveonatrialwithB,thereisthepotentialforinhibitiontodevelopbetweenAandB(seeMcLaren,Kaye,&Mackintosh,1989;McLaren&Mackintosh,2000).
VA�B=A(cB�XVTotal�B)(7)
VB�A=B(cA�XVTotal�A)(8)FirstitisassumedthatthechangeinthestrengthoftheassociationbetweennodesAandBisgovernedbyEquation7,andthereciprocalB-AassociationisgovernedbyEquation8.TheseequationsareformallyequivalenttoEquations1and2.Theyprovideabasisfortheformationofassociationsbetweentheelementsofacompound(AB),allowingbehaviourestablishedtoonestimulus(e.g.,A)totransfertotheother(e.g.,B).IwillreturntotheseCS-CSassociationsinthecontextofapotentialanalysisoffeaturesofblocking.TheequationsalsoprovidethebasisforthedevelopmentofinhibitionbetweenAandBwhenbothhavebeenpairedwiththesameUS.Accord-ingtoEquations7and8,netinhibitionwilldevelopbetweentheAandBnodes,totheextentthatthecombinedeectoftheforward(e.g.,A-US)andbackwardassocia-tions(e.g.,US-B)provideanindirectbasisforVA�BtobepositivewhenBisabsent.Thus,onatrialwhenAispresented,B=0andtheabili

tyofAtoactivateB(i.e.,VA�B)willdependonmultiplyingthestrengthsoftheA-USandUS-Bassociations:88 1/c*VA�US*VUS�B;andonatrialwhenBispresented,A=0andVB�Awilldependon:1/c*VB�US*VUS�A.ThedevelopmentofthisinhibitionwillmeanthatwhenAandBarepresentedtogether(e.g.,forasummationtest)theircorrespondingnodeswillbelesslikelytobecomeactivethanifeitherhadbeenpresentedalone:PerformancetoanABcompoundwillbeconstrainedbythedevelopmentofinhibitionbetweenthenodesactivatedbyAandBwhenbotharefollowedbythesameUS.ItisworthnotingthatsuchaconstraintonsummationwouldbelesslikelyifAandBweretobefol-lowedbydierentreinforcersduringconditioning;reinforcerswiththesametendencytoprovokeconditionedrespondingbutwithdistinctsensoryproperties(e.g.,A-foodandB-sucrose).Inkeepingwiththeanalysisoutlinedinthepreviousparagraph(i.e.,thatHeiDIpredictssummationbetweentwostimuliseparatelypairedwiththesameUSwillbeconstrainedbyinhibitionbetweenthem),WattandHoney(1997)observedthatacom-pound(AB)wasmorelikelytoprovokeconditionedrespondingattestifitscomponentshadbeenseparatelypairedwithdierentappetitivereinforcers(foodandsucrose)thatsupportthesameconditionedresponse,thaniftheyhadbeenpairedwiththesamereinforcer(foodorsucrose;orbothfoodandsucrose,ondierenttrials).ItpredictsthisbecausethereislessinhibitionbetweenAandBwhentheyarefollowedbydierentappetitiveUSsthanwhentheyarefollowedbythesameappetitiveUS:ThesharedsensorypropertiesofthesharedUSsprovideabasisforinhibitionbetweenAandB,andthismeansthattheycansuppressoneanother'sactivation.Ingeneralterms,dier-encesinthedevelopmentofinhibitionbetweenAandBengenderedbydierenttraining

proceduresshouldaectthelikelihoodofsummationbeingobserved.ThedevelopmentofinhibitionbetweenAandB,whenbotharepairedwiththesameoutcome,hasnotbeendirectlyassessedinstudiesofsummationorconsideredatatheoreticallevel(cf.Brandon,Vogel,&Wagner,2000;Pearce,1994).However,thereisevidencethatiscon-sistentwiththissuggestionfromstudiesofcategorization(Aitken,Bennett,McLaren,&Mackintosh,1996)andperceptuallearning(e.g.,Dwyer&Mackintosh,2002;Mundy,89 Dwyer,&Honey,2006).ExternalinhibitionWhenanassociativelyneutralstimulus(B)ispresentedwithastimuluswithassocia-tivestrength(A)theconditionedresponsetothatstimulusisoftendisrupted;aneectknownasexternalinhibition.Forexample,Pavlov(1927,p.44)originallyobservedthattheamountofconditionedrespondingtoaCS(inhiscasetheamountofsaliva-tionindogs)wasreducedwhenastimuluswithnoassociativepropertieswaspresentedwiththeCS.ThiseectisnotpredictedbytheRescorla-Wagnermodel,andhasbeeninterpretedintermsofadecreaseinattentiontotheCS(Mackintosh,1974,p.16).InasetofsimulationsinwhichtheassociativestrengthofVB�USwassettozeroanditwaspresentedwithastimuluswithexcitatoryassociativestrength(A)thatpossessedexcitatoryassociativestrength(VA�US�0),thepresenceofBincreasedRCSandre-ducedRUS,forABrelativetoA;whichwouldincreaseCS-orientedbehaviourrelativetoUS-orientedbehaviour.Thereisalsosomeevidencethatisconsistentwiththisanalysisfromstudiesofarelatedeect,knownasdisinhibition,whereconditionedresponding(e.g.,instrumentalleverpressingforfood)canbeaugmentedbythepresentationofastimulus(e.g.,alightorwhitenoiseBrimer,1970;Brimer&Kamin,1963).Infact,thiseectappearstobemostapparentwhenthelevelof

leverpressingislow(e.g.,attheonsetofaxedinterval,Flanagan&Webb,1964;Hinrichs,1968).Unfortunately,noneofthesestudiesmeasuredongoinggoal-tracking,whichshouldbethemirrorimageofbehaviourdirectedtowardsthelever.SummationtestsforconditionedinhibitionFinally,combiningastimuluswithstrongexcitatoryproperties(A)andastimuluswithmodestnetinhibitoryproperties(B)willmeanthatVAB�USwilltakealowervaluethanVA�US.Equations1and2wereusedtogeneratetheindividualVsforareinforcedstimulus(A)andastimulus(B)thatwasnon-reinforcedinthepresenceofA.Equations90 3-5wereusedtodetermineperformance.WhethertheparameterswerechosentoresultinabiastowardsRCS(e.g.,AandB=.50,andUS=.30),orRUS(e.g.,AandB=.50,andUS=.70),combiningAwithBresultedinlowerlevelsofboth.ThevaluesforRCSandRUSfortheABcompoundwouldremainpositive(albeitlowerthanthoseforAalone)becauseVCOMBwillstillbepositive.However,ifAhadmodestexcitatorypropertiesandBhadstronginhibitoryproperties,thenVCOMBwouldbenegative-andasaresultRCSandRUSwouldalsobenegative.AdoptingEquation6wouldmeanthatr1wouldbenegative(unlesseitherCS-r1orUS-r1werealsonegative).Inthiscase,anexampleofapositiver1mightbetoapproachtheleverandanegativer1towithdrawfromthelever.IfthenegativevaluesreturnedbyEquations1and2wereconstruedasinvolvingtheactivationofaNoUSnode(Konorski,1967;Pearce&Hall,1980),thentheexcitatoryCS-NoUSassociationwouldresultinRCSandRNO�USbeingpositive,andRNO�UScouldthendirectlygeneratedierentformsofrespondingnotsupportedbyeithertheCSorUS.3.6.5Blocking:LearningandperformanceItwasnotedintheintroductionthatoneofthekeyfeaturesoftheRescorla-Wagnermodelwasitsabilitytoexplai

nhowtheassociativestrengthofonestimuluswithinacompoundaectstheassociativestrengthgainedbyanotherstimuluswithinthecom-pound(e.g.,blocking,Kamin,1969).TheformalsimilaritybetweenEquation1andtheRescorla-Wagnermodelisclear,andlikethismodel,Equation1generatestheseimpor-tanteectsonthedevelopmentoftheCS-USassociation.However,otherfeaturesofHeiDImeanthatblockingisnot{astheRescorla-Wagnermodelpredicts{inevitable.Inextremis,Equations1-3inconcertwithEquations4and5provideanaccountofblockingthatisclearlyrelatedtotheRescorla-Wagnermodel:IfVA�USc*USattheendofaperiodoftrainingwhereAhasbeenpairedwithaUS,thenconditioningwithacompound(AB)willresultinlittleornoincreaseintheB-USassociation(i.e.,VB�US0).However,accordingtoHeiDI,thereciprocalUS-Bassociation(VUS�B)willbe91 unaectedbythefactthatAhasareciprocalassociationwiththeUS(VUS�A),becausethec*Aandc*BvaluesofAandBprovideaseparatebasisfortheformationoftheseassociations.ThepredictionthattheUS-BassociationisnotblockedwillordinarilybewithoutconsequencebecauseEquation3willreturnaVCOMBforB0(i.e.,ifVB�US0thenVB�US+(1/c*VB�US*VUS�B)0).AccordingtoEquations4and5,RCSandRUS0becauseVCOMB0.However,oneclearimplicationofthisanalysisisthattreatmentsthatenabletheUS-Bassociationtoin uenceperformanceshouldreducetheblockingeect;andthereisevidencethattheperformancetoablockedstimuluscanbeaugmentedundersomeconditions(forareview,seeMilleretal.,1995).BothHeiDIandtheRescorla-WagnermodelpredictthatVB�US(andVA�US)willincreaseduringthecompoundconditioningphaseofablockingprocedureifVA�USc*US.However,unliketh

eRescorla-Wagnermodel,HeiDIpredictsthatthepatternofperformancewhenBistestedwillre ectthevaluesofc*BandVB�US.Undertheseconditions,AmightgenerateUS-orientedbehaviour(i.e.,VA�US&#x]TJ/;༕ ;.9;Ւ ;&#xTf -;е.;व ;&#x-23.;ई ;&#xTd [;c*A),buttheassociativestrengthgainedbyBmightbeevidentasCS-orientedbehaviour(i.e.,c*B&#x]TJ/;༕ ;.9;Ւ ;&#xTf -;е.;व ;&#x-23.;ई ;&#xTd [;VB�US).Thissimpleobservationhasanimportantimplication:AblockingeectmightnotbeevidentiftheexperimentalassaywasmoresensitivetoCS-orientedbehaviourthantoUS-orientedbehaviour.ThefactthatVB�USislowwillreduceVCOMBinEquation3,butitwillsimultaneouslyincreasethecontributiontoperformanceoftheCS-orientedcomponent(i.e.,RCS)andreducetheUS-orientedcomponent(i.e.,RUS).Whileitwouldbetendentioustoarguethatfailurestoobserveblocking(e.g.,Maesetal.,2016)providesupportfortheanalysispresentedabove{groundsforsuchfailuresabound{therecanlittledoubtthatblockingeectscanbelesscompletethanasimplerenderingoftheRescorla-Wagnermodelwouldpredict(forarecentreviewandanalysis,seeUrcelay,2017).However,perhapsthemostseriouschallengetotheaccountofblockingoeredbytheRescorla-Wagnermodelinvolvestheconditionsunderwhich\unblocking"occurs.Conventionalproceduresforblockinginvolvetwostagesinwhichthereinforceristhe92 same:A-�USandthenAB-�US.ThefactthatincreasingthenumberofUSsbetweenstage1(e.g.,A-�US1)andstage2(AB-�US1-US2)resultsinunblocking(i.e.,learningaboutB)isperfectlyconsistentwiththemodel,becausethischangeintroducesapositivediscrepancyinthepoolederrorterm(seeEquations0and1).T

heproblematicresultisthefactthatreducingthereinforcer(i.e.,A-�US1-US2andthenAB-�US1)canalsoresultinrespondingtoB(i.e.,unblocking,Dickinson,Hall,&Mackintosh,1976).Takeninisolation,Equations0and1predictthatthereductioninthenumberofreinforcersshouldhaveresultedinBacquiringinhibitoryproperties(e.g.,Cottonetal.,1982).`Downshiftunblocking',asitisknown,hasbeentakenasevidencethatthereductionintheUSpreventsthereductioninattentiontoBthatwouldordinarilyresultfromthefactthattheUSwaspredictedbyA;andallowsBtobelearntabout(e.g.,Mackintosh,1975;Pearce&Hall,1980).Whiletherehasbeensomeprogressinunderstandingtheconditionsunderwhichdownshiftunblockingoccurs(Holland,1988)thereisnoconsensusaboutitsexplanation.Manyhavesimplyadoptedtheviewthatdownshiftunblockingisprimafacieevidencethatattentioncanchangeasaresultofexperience(Pearce&Mackintosh,2010).However,aspeculativeexplanationforthiseectcanbederivedfromapplicationofHeiDI,withoutappealingtochangesinattention(seeFigure11).Theessenceoftheanalysisisthattheremovalofthesecondshockallowsawithin-compoundB-Aassociationtoformmoreeectivelyduringdownshiftunblockingthanduringstandardblocking;andthisassociationallowsBto\borrow"theassociativepropertiesofA.ConsiderablockingprocedureinwhichAisrstfollowedbysuccessivepresentationsofthesamenominalUS.EachUScanbetreatedashavingpartiallyseparaterepresentations(US1andUS2).Undertheseconditions,AwillbecomelinkedtobothUS1andUS2untileachlinkreachestheasymptotedeterminedbyc*US1andc*US2;andcriticallylinkswillbestrengthenedbetweenUS1andA,andUS2andA,untiltheircombinedassociativestrength=c*A.WhenABispairedwithUS1andUS2,thea

ssociationsbetweenBandbothUS1andUS2willbeblocked;andthe93 Figure11:SimulationsforHeiDIofthechangesinassociativestrengthduringstandardblocking(panelsA-C)anddownshiftunblockingprocedures(panelsD-E).Theparam-etersusedwere:A=B=.30,andUS1=US2=.30.Attheoutsetofcompoundconditioning,A-US1andA-US2weresetto.30,andUS1-AandUS2-Awerebothsetto.15.PanelsAandDshowtheoutputvaluesforthestrengthsoftheAandBwithUS1associations(A-US1,US1-A,B-US1,US1-B)andPanelsBandEshowtheoutputvaluesforthestrengthsoftheAandBwithUS2associations(A-US2,US2-A,B-US2,US2-B),returnedbyEquations1and2combinedwithEquations7and8.NotethatUS2-AandUS2-Bissetto0inpanelEtore ectthefactthattheUS2isabsent;buttheseassociationswillnotchangeduringunblocking.PanelsCandFshowthestrengthoftheA-BandB-Aassociations.AkeyobservationisthattheB-Aassociationgainsstrengthduringdownshiftunblocking(panelF),butnotstandardblocking(panelC).combinedeectoftheUS1-AandUS2-AassociationswillmeanthatBwillnotbeabletoenterassociationwithA.However,thiswillnotbethecasewhenUS2isomitted.IfthechangeintheB-AassociationisassumedtobedeterminedbyB*(c*A{PVTOTAL�A),withPVTOTAL�A=VUS1�A+VUS2�A+VB�A,thentheremovalofUS2willenablethestrengtheningoftheB-Aassociation(andfurtherincreasesintheUS1-Aassociation).Undertheseconditions,downshiftunblockingwilloccurtotheextentthatthein uenceoftheB-AassociationinretrievingtheassociativepropertiesofA(strongerfollowingdownshiftunblockingthanstandardblocking)outweighsthefact94 thatA-US2isweakerandB-US2isnegativeafterdownshiftunblocking.Thisaccountisnecessarilyspeculative,mirroringthefactthattheunderstandingoftheconditionsunder

whichdownshiftunblockingoccursremainsincomplete(seeHolland,1988).However,itreceivessupportfromstudiesconductedbyRescorlaandColwill(1983),whichshowedthatmanipulationsthatshoulddisruptB-AassociationsalsoreducethedierenceinperformancetoBbetweenstandardblockinganddownshiftunblocking6.Thesimulations7presentedinFigure11forthecompoundconditioningstagearebased{intheinterestsofsimplicity{onthefollowingparameters:A=B=c*A=c*B=.30,andUS1=US2=c*US1=c*US2=.308.However,thecriticaldierenceintheB-Aassociationduringstandardblockinganddownshiftunblockingisageneralone.Attheoutsetofsimulatedcompoundconditioning,forbothstandardblocking(panelsA-C)anddownshiftunblocking(panelsD-F),VA�US1wassetto.30andVA�US2wassetto.30tore ecttheassumptionthatUS1=US2=c*US1=c*US2=.30.Critically,VUS1�AandVUS2�Aweresetat.15forstandardblocking,whereasfordownshiftunblockingVUS1�Awassetat.15andVUS2�Awassetto0(tore ectthefactthatUS2isabsent).Forthesamereason,VUS2�Bwasalsosetto0.PanelsA-C(standardblocking)andpanelsD-F(downshiftunblocking)depictthevaluesreturnedbythecombinationofEquations1and2withEquations7and8.PanelsAandDshowassociationsbetweenAandBwithUS1(A-US1,US1-A,B-US1,US1-B)andPanelsBandEshowassociationsbetweenAandBwithUS2(A-US2,US2-A,B-US2,US2-B)andPanelsCandFshowassociationsbetweenAandB(VA�Band 6Itisworthnotingthatwithin-compound(A-B)associationscouldalsoformduringtheexperimentsdemonstratingunequalchangeintheassociativestrengthoftheelementsofacompound(AB).However,inthiscase,therewasevidencethattheseassociationswerenotresponsiblefortheeectsthatwereobserved(s

eeAllmanetal.,2004;Rescorla,2000)7ThesesimulationsareimplementedintheHeiDIappas\Downshiftunblocking".8ThesimulationsthatIreportdonotincludeassociationsbetweenUS1andUS2,becausetheywouldnotin uencetheformationtheexcitatoryB-Aassociationuponwhichtheanalysisrests.Moreover,whiletheformationofUS2-US1andUS1-US2associationswouldtendtoreducerespectivelytheA-US1andA-US2associationsduringconditioningwithA,theabsenceofUS2duringdownshiftunblockingwouldallowincreasesinboththeA-US1andB-US1associations.Furthermore,thereductionsinthenetassociativestrengthoftheA-US2andB-US2associationsproducedbytheabsenceofUS2wouldbelessmarkedthanthosedepictedinFigure8Dand8E,becauseUS1wouldgainaproportiontheoverallnetreduction.Insummary,theinclusionofUS1-US2associationsincreasesthelikelihoodthatdownshiftunblockingwouldbeobserved.95 VB�A).InspectionofpanelsA-CconrmsthatduringstandardblockingassociationsinvolvingAremainedthesame,andthatassociationsfromUS1,US2andAtoBallincreasedbyequivalentamounts.Critically,theB-Aassociationdidnotdevelop,andthisassociationcanprovidenobasisuponwhichBcouldprovokeconditionedresponding;andthereciprocalUS1-BandUS2-Bassociationscannot{inisolation{contributetoperformance.Incontrast,duringdownshiftunblockingbecauseUS2isabsenttheUS1-AandB-Aassociationscanstrengthen.ThiswillmeanboththatVCOMB�AwillbehigherfollowingdownshiftunblockingthanstandardblockingandthatBwillbeabletoaccessVCOMB�AthroughtheB-Aassociation.InorderforthisstateofaairstogeneratemoreperformancetoBitwouldneedtooutweighthefactthattheA-US2andB-US2areweakerorinhibitoryafterdownshiftunblockingthanstandardblocking.InChapter5,Iwillconsiderhowtheasso

ciativestrength(VCOMB�A)borrowedbyonestimulus(B)fromanotherstimulus(A),withwhichithasanassociation(VB�A),ismanifestinperformance.Fornow,itissucienttonotethatHeiDIprovidesoneformalanalysisofhowwithin-compoundassociationsmightaecttheoutcomeofblockingandunblockingprocedures(cf.Urcelay,2017).3.6.6Latentinhibition:AnalternativeassociativeanalysisRescorlaandWagner(1972)recognizedthefactthatwhiletheirmodelprovidedareadyaccountforblocking,itdidnotaddressthefactthatsimplepreexposuretoaCSretardslaterexcitatoryandinhibitoryconditioning(forareview,seeHall,1991;Lubow,1989).Thatis,theoriginalmodeldidnotprovideanaccountoflatentinhibition(Lubow&Moore,1959).But,whyshouldrepeatedpresentationofato-be-conditionedstimulusaecttherateatwhich(excitatoryandinhibitory)conditionedperformanceemergestothatstimulus?Thisobservationinparticular,aswellasdownshiftunblocking,haspromptedtheoriststoconcludethatmodelsofPavlovianconditioningneedtoincludeanotherprocessthatchangesasafunctionofexperience:attention,associabilityofCSprocessing(e.g.,Mackintosh,1975;Pearce&Hall,1980;Wagner,1981).96 However,acriticalfeatureoflatentinhibition,whichprovidesapotentialtheoreticallinkwithanassociativeanalysisofblocking,isthatlatentinhibitioniscontextspecic.IfpreexposuretotheCSoccursinonecontext(denedbythecuespresentinoneexperimentalchamber)andconditioningtakesplaceinanothercontext,thenlatentinhibitionismuchreduced(e.g.,Hall&Honey,1989;Honey&Good,1993).Thegeneralsignicanceofthisobservationisthatitsuggeststhat{duringthepreexposurestage{animalsencodewherethestimulushasbeenpresented;forexample,byformingacontext-CSassociation(cf.Wagn

er,1981).ThisobservationenablesHeiDItoprovideasimpleanalysisoflatentinhibition:theblockingoftheUS-CSassociationbythecontext-CSassociation9Itwasarguedthatduringexcitatoryconditioning,performanceisdeterminedbybothaCS-USassociationandaUS-CSassociation,andthatduringinhibitoryconditioning,performancecouldre ectthestatusofbothaCS-NoUSandaNoUS-CSassociation(Konorski,1967).Whileacontext-CSassociationwillnotblocktheCS-USandCS-NoUSassociations,itwillblockthedevelopmentoftheUS-CSandNoUS-CSassociations.Thus,thesimpleinclusionofaUS-CSassociation(andNoUS-CSassociation)enablesanaccountoflatentinhibitionthatdoesnotrequireaseparateattentionalorassociabilityprocess(e.g.,Mackintosh,1975;Pearce&Hall,1980)orchangesinCSprocessingoftheformenvisagedbyWagner(1981).Inadditiontothisnovelanalysisoflatentinhibition,thepresenceofaUS-CSassoci-ationmeansthattheeectivesalienceofCSsthataregoodpredictorscanbeaugmented(cf.Mackintosh,1975).IthasbeenshownthatthevalueofastimulusaectstherateatwhichCS-orientedandUS-orientedcomponentsofperformancedevelop(seeFig-ure8).TheUS-CSassociationprovidesanaturalwayinwhichactivationoftheUSmightbere ectedbacktotheCSandmaintainitsactivation.Moreover,Ihavealready 9Itshouldbeacknowledgedthatwhilethecontextspecicityoflatentinhibitionisconsistentwiththeviewthatcontext-CSassociationsprovideapotentialexplanationforlatentinhibition(andhabit-uation),thefactthatattemptstoextinguishthecontext-CSassociationhaveoftenhadnoeectonlatentinhibitionisinconsistentwiththisaccount(seeBaker&Mercier,1982;Hall&Minor,1984).However,theinterpretationoffailuresofthiskindisnotstraightforward(seeHoney,Iordanova,&Good,2010

).97 notedthatwhenaCSisfollowedbyareductioninUSmagnitude(e.g.,duringextinc-tionorpartialreinforcement),CS-orientedrespondingincreasesrelativetoUS-orientedresponding,whichcouldalsoaectthesubsequentlearninginvolvingthatCS.HeiDItherebyprovidesasimpleanalysisofphenomenathatareroutinelytakentoindicatethattheassociabilityofstimuli(theirvalue)ortheirprocessingchangesasaresultofexperience(e.g.,Mackintosh,1975;Pearce&Hall,1980;Pearce&Mackintosh,2010;Wagner,1981).Accordingtothisanalysis,thesephenomenaareanotherproductofthereciprocalassociationsthatformbetweentheCSandUS,andbetweenthecomponentsofstimuluscompounds.InthischapterIexaminedhowHeiDIisabletoaccommodatesomeofphenomenathatarebeyondofthescopeofRescorla-WagnerModel,andhowlearningwouldtrans-lateintoconditionedbehaviourdirectedtowardstheCSortheUS.Inthefollowingchapter,Chapter4,Idiscussandtestnewpredictionsderivedfromthemodel.Chapter5,theGeneralDiscussion,willthenreturntothewidertheoreticalimplicationsoftheHeiDImodel.98 3.7Listofequations
VCS�US=CS(cUS�XVTotal�US)(1)
VUS�CS=US(cCS�XVTotal�CS)(2)VCOMB=VCS�US+(1 cVCS�USVUS�CS)(3a)VCOMB�AB=XVCS�US+[1 cXVAB�US(VUS�A+VUS�B)](3b)RCS=cCS cCS+jVCS�USjVCOMB(4)RUS=jVCS�USj cCS+jVCS�USjVCOMB(5)r1=(1 cRCSCSr1)+(1 cRUSUSr1)(6)
VA�B=A(cB�XVTotal�B)(7)
VB�A=B(cA�XVTotal�A)(8)99 Chapter4TestingpredictionsofHeiDI4.1IndividualdierencesinUS4.1.1IntroductionAsdiscussedinChapter3(Section3.2),recentevidencesuggeststhatperformanceis

aectedbyhowtheoutcomeisvaluedorperceived.Dwyeretal.(2018)showedthatindividualdierencesinthepalatabilityofsucrose(duringtheirexperimentsin-volvingcontrasteects)werepositivelycorrelatedwiththe avourpreferencelearning.Moreover,rodentswhoshowedastronglikingforsucrose(asmeasuredbylickingmi-crostructure;seeDwyer,2012)aremorelikelytobegoal-trackers(whensucrosewastheUS)thanthosewhoexhibitedaweakerlikingforsucrose(Patituccietal.,2016).HeiDIassumesthattheperceivedsalienceoftheCS(relatingtoCS)andUS(relat-ingtoUS;Equations4and5)variesbetweenanimals,howeverthevaluesofCSandUSarexedforagivenanimal.ThisvariationinhowtheUSisvaluedaectsbothlearning(i.e.,theasymptoticvalueofVCS�USandtherateatwhichVUS�CSreachesasymptote,throughEquations1and2)andthedistributionofVCOMBinperformance(throughVCS�USinEquations3-6).SimulationsofHeiDIarepresentedinFigure13.TheparametersduringconditioningwerechosentoresulteitherinahigherUSvalue(CS=.30andUS=.70;panelsAandB)oralowerUSvalue(CS=.30andUS=.50;panelsCandD).Simulationforconditioning(lefthandsideofeachpanel;trials1-6)andextinction(right-handsideofeachpanel;trials7-12)forareinforced(+)lever.PanelsAandCdepicttheoutputvaluesforVCS-US,VUS-CS,VCOMB,andpanelsBandDshowthecorrespondingoutputvaluesforRCSandRUS.ItcanbeobservedfromFigure13thatahigherUSvaluewillresultinhigherlevelsofgoal-trackingorRUS(panelB),incontrastwithlowerUSvalue100 (PanelD).Notethat,incontrastwiththeanalysispresentedfordownshiftunblockinginChapter3(section3.6.5),atwopelletrewardispresentedasahigherUSvalueinsteadoftwoseparateUSvaluesforUS1andUS2.Toexploreth

isprediction,ratsweregiventrainingwithonereinforcedlever(e.g.,leftlever)andonenon-reinforcedlever(e.g.,rightlever).Halftheanimalswerereinforcedwithonepelletrewardandtheotherhalfwithtwopelletsreward,aftereachpresentationofthereinforcedlever.HeiDIpredictsthatthismanipulationwillbeshownintherat'stendencytointeractwiththefood-well,inotherwords,animalsinthetwopelletgroupwillhavehigheroverallgoal-trackinglevelsthananimalsintheonepelletgroup.4.1.2Experiment3MethodAnimalsandapparatusTwenty-threenaivemaleSpragueDawleyratswereused(suppliedbyCharlesRiver,UK).Theanimalsweresplitintotwogroups:oneortwopelletsreward(11ratsinonepelletgroup,12ratsinthetwopelletgroup).TheratswerehousedasdescribedinExperiment1.Theirmeanadlibitumweightbeforethestartoftheexperimentwas384g(range:391-577g)andtheyweremaintainedatbetween85and95%oftheseweightsbygivingthemrestrictedaccesstofoodattheendofeachday.Theratshadcontinuousaccesstowaterwhentheywereintheircages.TheexperimentalchamberswerethoseusedinExperiment1.ThedesignoftheexperimentissimilartoExperiment1,withtheadditionthatinExperiment3,halfoftheratsreceivedtwopelletsasareward,insteadofone(seeTable3).ProcedureTheprocedureusedforExperiment3wasthesameasforExperiment1withtheadditionofanotherexperimentalgroupwerethereinforcerwasatwopelletrewardinsteadofone.101 Table3:DesignOftheExperiment3and4 ClassicationTrainingReversalNotation 1pelL1+L1-L1(+k-)1pelL2-L2+L2(-k+)1pel 2pelL1+L1-L1(+k-)2pelL2-L2+L2(-k+)2pel Note:1peland2pelrefertotherewardmagnitude(onegroupofratswasreinforcedwithonepelletandtheothergroupwithtwopellets);L1andL2refertotwolevers(leftandright,counterbalanced).Duringtraining,L1wasreinf

orced(\+";onefoodpelletinthe1pelgrouportwofoodpelletsinthe2pelgroup)andL2wasnon-reinforced(\-").DuringthereversalstageL1wasnon-reinforcedandL2wasreinforced(oneortwopellets).DataAnalysisDataanalysiswasconductedasdescribedinExperiment1.RatsweresplitintoSTsandGTstakingamediansplit(asdescribedinExperiment1)foreachrewardgroup(oneandtwopellets).Forthe1pellet,groupbiasscoresabove.67wereclassiedasGTsandbelow.58asSTs.Forthetwopelletgroup,biasscoresabove.23wereclassiedasGTsandbiasscoresbelow.19asSTs.Thedistributionofsign-trackingandgoal-trackingbehaviourisshowninFigure12.MixedANOVAswereconductedseparatelyforleverpressingandfood-wellentrieswithwithin-subjectsfactorsblock(training6levels:T1-T6;transitionblocks2levels:T6-R1;reversal6levels:R1-R6;)andlever(2levels:reinforced\+"vsnon-reinforced\-")andbetween-subjectsfactorclassication(2levels:STvsGT)asinExperiment1,withtheadditionofrewardasbetween-subjectsfactor(2levels:onevstwopellets).ResultsSimulationsofHeiDIarepresentedinFigure13.ThemainresultsfromExperiment3areshowninFigure14.Theanalysiswillbeginwithresultsfromthetrainingstage(lefthandpanelsofFigure14),beforemovingtothetransitionbetweentrainingandreversal(identiedbythegreysection),andnallythereversalstageasawhole(right-hand102 Figure12:Distributionofsign-tracking(leverpresses)andgoal-tracking(food-wellentries)behaviourforExperiment3per(10-s)trialduringlastblockoftraining(T6)forL1(+jj-).Theblacksymbolscorrespondtosign-trackers(ST)andtheclearsymbolstogoal-trackers(GT).pannels).TrainingInspectionoftheresultsfromtherststageoftraining(left-handsideoftheupperandlowerpanelsofFigure14)suggestthatastra

iningprogressedratsinbothgroups(STandGT)showedmoreleverpressesandfood-wellentriesduringthereinforcedL1thanthenonreinforcedL2.Forleverpresses,bothSTsandGTsshowhigherlevelsofresponsefortwopelletsreward,howeverthedierencewasnumericallylargerinSTsthanGTs.Food-wellentriesdidnotshowadierencebetweenoneandtwopelletsrewardforeitherSTsorGTs.Thedescriptionofthetrainingresultsissupportedbyseparateanalysesofleverpressesandfood-wellentries.103 Figure13:SimulationofHeiDIforindividualdierencesinUSonsignandgoal-tracking:Simulationforconditioning(left-handsideofeachpanel;trials1-6)andex-tinction(right-handsideofeachpanel;trials7-12)forthereinforced(+)lever.PanelsAandCdepicttheoutputvaluesforVCS-US,VUS-CS,VCOMB,andpanelsBandDshowthecorrespondingoutputvaluesforRCSandRUS.TheparametersduringconditioningwerechosentoresulteitherinahigherUSvalue(i.e.,CS=.30andUS=.70;panelsAandB)oralowerUSvalue(i.e.,CS=.30andUS=.50;panelsCandD).Duringextinctionofreinforcement,USwassetto0.LeverpressesAnANOVAconductedforleverpressesrevealedmaineectsofreward,F(1,19)=6.45,p=.020,2p=.25,classication,F(1,19)=40.41,p.001,2p=.68,block,F(2.43,46.20)=8.96,p.001,2p=.32,andlever,F(1,19)=65.83,p.001,2p=.78.Therewasaninteractionbetweenclassicationandblock,F(2.43,46.20)=4.73,p=.009,2p=.20,andcriticallybetweenrewardandlever,F(1,19)=6.75,p=.018,2p=.26,aswellasclassicationandlever,F(1,19)=45.42,p.001,2p=.71andbetweenblockandlever,F(2.36,44.78)=14.22,p.001,2p=.43,butnotbetweenrewardandclassication,F(1,19)=2.28,104 Figure14:ResultsfromExperiment3:theeectofdi

;erentvaluesofUSonsign-trackingandgoal-tracking.Mean(+SEM)leverpresses(upperpanel)andfood-wellentries(lowerpanel)per(10-s)trialacrossthetwostages:training(T1{T6)andreversal(R1{R6).Duringtraining,ratsreceivedpresentationsofoneleverpairedwithfoodpellets(L1(+jj-))andnonreinforcedpresentationsofasecondlever(L2(-jj+));ratswereclassiedassign-trackers(STs)andgoal-trackers(GTs)onthebasisoftheirbehaviourduringthenalblockoftraining(T6).Theythenreceivedareversal:L1non-reinforcedandL2reinforced.Thegreysectionindicatestransitionbetweeninitialtrainingandthereversalofthecontingencies.p=.147,2p=.11orbetweenrewardandblock,F(2.43,46.20)=0.52,p=.631,2p=.03.Therewasatripleinteractionbetweenclassication,blockandlever,F(2.36,44.78)=6.83,p=.002,2p=.26,butnointeractionbetweenreward,classi-cationandblock,F(2.43,46.20)=0.67,p=.544,2p=.03,reward,classicationandlever,F(1,19)=2.18,p=.156,2p=.10,reward,blockandlever,F(2.36,44.78)=0.71,p=.520,2p=.04.Therewasnofour-wayinteraction,F(2.36,44.78)=0.70,p=.522,2p=.04.Food-wellentries105 AnANOVAconductedforfood-wellentriesrevealedmaineectsofclassication,F(1,19)=7.16,p=.015,2p=.27,block,F(2.31,43.84)=16.86,p.001,2p=.47,lever,F(1,19)=43.07,p.001,2p=.69,butnotofreward,F(1,19)=0.26,p=.616,2p=.01.Therewasaninteractionbetweenclassica-tionandblock,F(2.31,43.84)=4.28,p=.016,2p=.18,andbetweenblockandlever,F(2.62,49.70)=14.28,p.001,2p=.43,butnotbetweenrewardandclassication,F(1,19)=0.03,p=.855,2p.01,rewardandblock,F(2.31,43.84)=0.81,p=.468,2p=.04,betweenrewardandlever,F(1,19)=0.00,p=.950,

2p.01orbetweenclas-sicationandlever,F(1,19)=0.68,p=.419,2p=.03.Therewasatripleinteractionbetweenclassication,blockandlever,F(2.62,49.70)=5.24,p=.005,2p=.22,butnotbetweenreward,classicationandblock,F(2.31,43.84)=0.73,p=.505,2p=.04,reward,classicationandlever,F(1,19)=0.25,p=.625,2p=.01,orbetweenreward,blockandlever,F(2.62,49.70)=0.43,p=.708,2p=.02.Therewasnofour-wayinter-actionbetweenreward,classication,blockandlever,F(2.62,49.70)=0.60,p=.598,2p=.03.TransitionblocksInspectionoftheresultsforthetransitionblocks(greysectioninFigure14)forleverpressessuggestsnodierencebetweenonepelletandtwopelletsrewardforeitherSTsorGTs(upperpanels).Thesameistrueforfood-wellentries,nodierenceisobservedbetweentheoneandtwopelletsineitherSTsorGTs.Thereversalincontingenciesaectedmorethelevelsoffood-wellentries(lowerpanels)thanlever-presses(upperpan-els)inbothgroups(1peland2pel).Thisdescriptionofthetrainingresultsissupportedbyseparateanalysesofleverpressesandfood-wellentries.UnlikeinExperiment1,thelargerandfastereectofreversingcontingenciesforfood-wellentriesthanlever-presseswasadierenceofdegree,asreversaldidimpactonleverpresstosomedegreeduringthetransitionblocks.LeverpressesAnANOVAconductedforleverpressesforthetransitionblocksrevealedmainef-106 fectsofclassication,F(1,19)=19.38,p.001,2p=.50,block,F(1,19)=4.57,p=.046,2p=.19,lever,F(1,19)=7.73,p=.012,2p=.29,butnotofreward,F(1,19)=1.15,p=.297,2p=.06.Therewasaninteractionbetweenclassica-tionandlever,F(1,19)=1.73,p=.204,2p=.08,andcriticallybetweenblockandlever,F(1,19)=43.37,p.0

01,2p=.70,butnotbetweenrewardandclassication,F(1,19)=0.06,p=.813,2p.01,criticallyrewardandblock,F(1,19)=0.15,p=.702,2p.01,classicationandblock,F(1,19)=3.79,p=.066,2p=.17,orrewardandlever,F(1,19)=0.80,p=.383,2p=.04.Therewerenotripleinter-action,reward,classicationandblock,F(1,19)=0.00,p=.996,2p.01,reward,classicationandlever,F(1,19)=0.16,p=.697,2p.01,reward,blockandlever,F(1,19)=1.18,p=.292,2p=.06,classication,blockandlever,F(1,19)=23.69,p.001,2p=.55.Therewasnofour-wayinteraction,reward,classication,blockandlever,F(1,19)=0.16,p=.695,2p.01.Food-wellentriesAnANOVAconductedforfood-wellentriesrevealedmaineectsofclassication,F(1,19)=5.67,p=.028,2p=.23,andlever,F(1,19)=45.03,p.001,2p=.70,butnotofreward,F(1,19)=3.92,p=.062,2p=.17orblock,F(1,19)=0.49,p=.494,2p=.02.Critically,therewereinteractionsbetweenrewardandblock,F(1,19)=5.82,p=.026,2p=.23,classicationandblock,F(1,19)=7.05,p=.016,2p=.27,classi-cationandlever,F(1,19)=1.81,p=.195,2p=.09,andcriticallybetweenblockandlever,F(1,19)=59.50,p.001,2p=.76,butnotbetweenrewardandclassication,F(1,19)=0.00,p=.974,2p.01,orbetweenrewardandlever,F(1,19)=0.37,p=.552,2p=.02.Therewasnotripleinteraction,reward,classicationandblock,F(1,19)=0.55,p=.467,2p=.03,reward,classicationandlever,F(1,19)=0.14,p=.713,2p.01,reward,blockandlever,F(1,19)=0.20,p=.660,2p=.01,classi-cation,blockandlever,F(1,19)=6.91,p=.017,2p=.27orafour-wayinteractionbetweenreward,classication,blockandlever,F(1,19)=0.24,p=.631,2

p=.01.IncontrasttoExperiment1andExperiment2therewasaninteractionbetweenblockand107 leverforleverpresses(F(1,19)=43.37,p.001,2p=.70),howevertheeectsizeforfood-wellentriesisconsiderablylarger(F(1,19)=59.50,p.001,2p=.76).ReversalAcrosstheblocksofreversaltraining(right-handsideoftheupperandlowerpanelsofFigure14),nodierenceswereobservedbetweentheoneandtwopelletgroupsinneitherleverpressesnorfood-wellentries.BothSTsandGTsincreasedrespondingforL2anddecreasedforL1forbothleverpressesandfood-wellentries.LeverpressesAnANOVAconductedforleverpressesforthereversalstagere-vealednomaineectsofreward,F(1,19)=0.21,p=.653,2p=.01,classication,F(1,19)=0.62,p=.440,2p=.03,block,F(1.52,28.84)=1.58,p=.225,2p=.08,orlever,F(1,19)=0.68,p=.420,2p=.03.Therewasaninteractionbetweenclassica-tionandblock,F(1.52,28.84)=3.45,p=.057,2p=.15andbetweenblockandlever,F(1.29,24.60)=16.44,p.001,2p=.46,butnotbetweenrewardandclassication,F(1,19)=0.00,p=.997,2p.01,rewardandblock,F(1.52,28.84)=1.30,p=.281,2p=.06,betweenrewardandlever,F(1,19)=0.69,p=.415,2p=.04orbetweenclas-sicationandlever,F(1,19)=1.25,p=.278,2p=.06.Therewasatripleinteractionbetweenclassication,blockandlever,F(1.29,24.60)=3.10,p=.082,2p=.14,butnotbetweenreward,classicationandblock,F(1.52,28.84)=0.09,p=.867,2p.01,reward,classicationandlever,F(1,19)=0.05,p=.822,2p.01orbetweenreward,blockandlever,F(1.29,24.60)=1.01,p=.347,2p=.05.Therewasnofour-wayinter-actionbetweenreward,classication,blockandlever,F(1.29,24.60)=0.38,p=.595,2p=.02.Food-wellentriesAnANOVAconduct

edforfood-wellentriesforthereversalstagerevealedamainef-fectoflever,F(1,19)=12.41,p=.002,2p=.40,butnoeectofreward,F(1,19)=1.75,p=.202,2p=.08,classication,F(1,19)=0.18,p=.674,2p.01orblock,F(2.51,47.72)=1.20,p=.318,2p=.06.Critically,therewasaninteractionbe-tweenblockandlever,F(2.08,39.60)=26.60,p.001,2p=.58,butnointeractions108 betweenrewardandclassication,F(1,19)=0.16,p=.694,2p.01,rewardandblock,F(2.51,47.72)=2.27,p=.102,2p=.11,classicationandblock,F(2.51,47.72)=1.48,p=.237,2p=.07,betweenrewardandlever,F(1,19)=0.23,p=.635,2p=.01,orbe-tweenclassicationandlever,F(1,19)=0.08,p=.775,2p.01.Therewerenotripleinteractionsbetween,reward,classicationandblock,F(2.51,47.72)=0.64,p=.568,2p=.03,reward,classicationandlever,F(1,19)=1.49,p=.237,2p=.07,reward,blockandlever,F(2.08,39.60)=0.09,p=.919,2p.01,classication,blockandlever,F(2.08,39.60)=2.31,p=.111,2p=.11orfour-wayinteractionbetweenreward,classication,blockandlever,F(2.08,39.60)=0.87,p=.432,2p=.04.DiscussionThepurposeofthisexperimentwastoexplorehowreinforcersofdierentvaluesaectconditionedbehaviour.HeiDIpredictshigherlevelsofUS-orientedbehaviourwithUSsofhighervalue.Incontrasttothisprediction,theresultsshowedthataUSwithahigherUSvalue(twopellets)producedmoresign-trackingcomparedtoaUSwithalowerUSvalue(onepellet),andnodierenceswereobservedingoal-tracking.Thismaybetheresultofmanipulatingrewardvaluethroughincreasingthenumberofrewardsratherthantheirsizeorvaluemoredirectly.Regardless,theresultsfromExperiment3,providefurtherevidencefortheres

ultsobtainedinExperiment1andExperiment2,replicatingthendingthatleverpressingbehaviourismoreresistanttocontingencychangesthanfood-wellbehaviour.4.1.3Experiment4WhileExperiment3providedresultsthatwerenotconsistentwiththepredictionsoftheHeiDImodelregardingtheeectsofincreasingthevalueoftheUS.Experiment4repeatedthesameexperimentaldesign.109 MethodAnimalsandapparatusTwenty-fourSpragueDawlleyfemaleratswereused(suppliedbyCharlesRiver,UK).TheratswerehousedasdescribedinExperiment1.Theirmeanadlibitumweightbeforethestartoftheexperimentwas238.17(range:205-309g)andtheyweremaintainedatbetween85and95%oftheseweightsbygivingthemrestrictedaccesstofoodattheendofeachday.Theratshadcontinuousaccesstowaterwhentheywereintheircages.TheapparatusisthesameasdescribedinExperiment1.ProcedureThesameprocedurewasusedasinExperiment3.DataanalysisThedatawasanalysedinthesamewayasinExperiment3.RatsweresplitintoSTsandGTstakingamediansplit(asdescribedinExperiment1)foreachrewardgroup(oneandtwopellets).Thedistributionoflever-pressesandfood-wellentriesbehaviourisshowninFigure16whereitisclearthatveryfewratspressedthelever.Fortheonepelletgroupbiasscoresbelow.99wereclassiedasSTandallscoresof1wereclassiedasGT.ThemediansplitinthisexperimentmeantthatsomeanimalswereclassiedasSTswhentheywerebarelypressingthelever.Forthetwopelletgroupbiasscoresbelow.66wereclassiedasSTsandscoresabove.86asGTs.Thisimpliesadistributionskewedtowardsgoal-trackingbutlesssothanfortheonepelletgroup.ResultsTrainingInspectionoftheresultsfromtherststageoftraining(left-handsideoftheupperandlowerpanelsofFigure15)suggeststhatonlytheanimalsinSTgrouparepressing110

Figure15:ResultsfromExperiment4(replicationofExperiment3infemalerats):theeectofdierentvaluesofUSonsign-trackingandgoal-tracking.Mean(+SEM)leverpresses(upperpanel)andfood-wellentries(lowerpanel)per(10-s)trialacrossthetwostages:training(T1{T6)andreversal(R1{R6).Duringtraining,ratsreceivedpresenta-tionsofoneleverpairedwithfoodpellets(L1(+jj-))andnonreinforcedpresentationsofasecondlever(L2(-jj+));ratswereclassiedassign-trackers(STs)andgoal-trackers(GTs)onthebasisoftheirbehaviourduringthenalblockoftraining(T6).Theythenreceivedareversal:L1non-reinforcedandL2reinforced.Thegreysectionindicatestransitionbetweeninitialtrainingandthereversalofthecontingencies.thelever,withnodierencesbetweenoneortwopelletsasareinforcer.Incontrastwithpreviousexperiments,theGTanimalsshownolever-orientedbehaviour.Thedistributionoflever-pressesandfood-wellentriesbehaviourshowninFigure16suggeststhatcomparedtoothercohorts,relativelyfewanimalsshowleverpressbehaviourandtheGTgroupdisplayespeciallylittleleverpressbehaviour.Theimplicationsofthisissueareaddressedinthediscussionsection.Dierencesinthemagnitudeofrewardwerenotobservedineitherleverpressesorfood-wellentries.Thisdescriptionofthetrainingresultsissupportedbyseparateanalysesofleverpressesandfood-wellentries.111 Figure16:Distributionofsign-tracking(leverpresses)andgoal-tracking(food-wellentries)behaviourforExperiment4per(10-s)trialduringlastblockoftraining(T6)forL1(+jj-).Theblacksymbolscorrespondtosign-trackers(ST)andtheclearsymbolstogoal-trackers(GT).LeverpressesAnANOVAconductedforleverpressesforthetrainingstagerevealedmaineectsofclassication,F(1,44)=

0.02,p=.888,2p.01,block,F(2.46,108.12)=0.55,p=.616,2p=.01,lever,F(1,44)=0.14,p=.711,2p.01,butcriticallynoef-fectofreward,F(1,44)=2.16,p=.148,2p=.05.Therewasatwo-wayinteractionbetweenclassicationandblock,F(2.46,108.12)=0.31,p=.777,2p.01,classi-cationandlever,F(1,44)=0.16,p=.691,2p.01,aswellasbetweenblockandlever,F(2.58,113.36)=0.46,p=.683,2p=.01,butnotbetweenrewardandclassi-cation,F(1,44)=2.46,p=.124,2p=.05,andcriticallynotbetweenrewardandblock,F(2.46,108.12)=3.26,p=.033,2p=.07orrewardandlever,F(1,44)=2.86,p=.098,2p=.06.Therewasathree-wayinteractionbetweenclassication,blockandlever,F(2.58,113.36)=0.38,p=.740,2p.01,butnotbetweenreward,clas-sicationandblock,F(2.46,108.12)=2.57,p=.070,2p=.06,reward,classication112 andlever,F(1,44)=3.08,p=.086,2p=.07orbetweenreward,blockandlever,F(2.58,113.36)=3.46,p=.024,2p=.07.Therewasnofour-wayinteractionbetweenreward,classication,blockandlever,F(2.58,113.36)=3.09,p=.037,2p=.07.Food-wellentriesParallelanalysisconductedforfood-wellentriesforthetrainingstagerevealedmaineectsofclassication,F(1,44)=0.01,p=.913,2p.01andlever,F(1,44)=4.69,p=.036,2p=.10,butnoeectofreward,F(1,44)=0.40,p=.528,2p.01orblock,F(2.47,108.72)=1.64,p=.192,2p=.04.Critically,therewasaninteractionbetweenblockandlever,F(2.58,113.43)=2.07,p=.117,2p=.04,butnoothertwo-wayinteractionsbetweenrewardandclassication,F(1,44)=1.41,p=.241,2p=.03,criticallyrewardandblock,F(2.47,108.72)=0.55,p=.615,2p=.01,classicationandblock,F(2.47,108.72)=1.33,p=.269,2p=.03

,rewardandlever,F(1,44)=2.78,p=.102,2p=.06,orclassicationandlever,F(1,44)=0.82,p=.370,2p=.02.Therewerenothree-wayinteractionsbetweenreward,classicationandblock,F(2.47,108.72)=0.93,p=.414,2p=.02,reward,classicationandlever,F(1,44)=3.36,p=.073,2p=.07,reward,blockandlever,F(2.58,113.43)=1.83,p=.154,2p=.04,classication,blockandlever,F(2.58,113.43)=0.60,p=.593,2p=.01,orafour-wayinteractionbetweenreward,classication,blockandlever,F(2.58,113.43)=1.78,p=.162,2p=.04.TransitionblocksInspectionoftheresultsforthetransitionblocks(greysectionofFigure15)showthatleverpresses(whentheyareatmateriallevels)arelessaectedbythechangeincontingenciesthanfood-wellentries.LeverpressesANOVAconductedforthetransitionblocksforleverpressesrevealedmaineectsofclassication,F(1,44)=0.07,p=.792,2p.01,andlever,F(1,44)=0.01,p=.906,2p.01,butnotofreward,F(1,44)=3.51,p=.068,2p=.07,orblock,F(1,44)=0.20,p=.654,2p.01.Therewasatwo-wayinteractionbetweenclassi-113 cationandlever,F(1,44)=0.01,p=.910,2p.01,butnotbetweenrewardandclassication,F(1,44)=1.75,p=.193,2p=.04,rewardandblock,F(1,44)=0.71,p=.405,2p=.02,classicationandblock,F(1,44)=0.04,p=.852,2p.01,rewardandlever,F(1,44)=2.63,p=.112,2p=.06orcriticallybetweenblockandlever,F(1,44)=0.10,p=.752,2p.01.Therewerenothree-waybetweeninteractionsre-ward,classicationandblock,F(1,44)=0.27,p=.604,2p.01,reward,classicationandlever,F(1,44)=3.20,p=.080,2p=.07,reward,blockandlever,F(1,44)=0.81,p=.374,2p=.02,classication,blockandlever,F(1,44)=0.00,p=.955,

7;2p.01orafour-wayinteractionreward,classication,blockandlever,F(1,44)=0.04,p=.842,2p.01.Food-wellentriesANOVAconductedforthetransitionblocksforfood-wellentriesrevealedamaineectoflever,F(1,44)=5.35,p=.025,2p=.11,butnomaineectsofreward,F(1,44)=0.13,p=.720,2p.01,classication,F(1,44)=0.00,p=.968,2p.01orblock,F(1,44)=0.35,p=.555,2p.01.Critically,therewasaninteractionbetweenblockandlever,F(1,44)=2.42,p=.127,2p=.05,butnotbetweenrewardandclassication,F(1,44)=1.33,p=.255,2p=.03,betweenrewardandblock,F(1,44)=0.24,p=.629,2p.01,classicationandblock,F(1,44)=0.79,p=.380,2p=.02,rewardandlever,F(1,44)=1.35,p=.252,2p=.03,orclassicationandlever,F(1,44)=0.01,p=.938,2p.01.Therewerenothree-wayinteractionsbetweenreward,classicationandblock,F(1,44)=0.00,p=.997,2p.01,reward,classicationandlever,F(1,44)=1.07,p=.307,2p=.02,reward,blockandlever,F(1,44)=1.50,p=.228,2p=.03,orclassication,blockandlever,F(1,44)=5.13,p=.028,2p=.10.Therewasnofour-wayinteractionbetweenreward,classication,blockandlever,F(1,44)=6.71,p=.013,2p=.13.ReversalAcrosstheblocksofreversaltraining(right-handsideoftheupperpanelsofFigure14),leverpressesincreasedforL2anddecreasedduringL1inSTs,howevertheincrease114 waslargerforthe1pelgroup.Lever-pressbehaviourremainednegligibleintheGTgroups.Food-wellbehaviour(right-handsideofthelowerpanelsofFigure14)increasedforL2anddecreasedduringL1.Thedescriptionoftheseresultsissupportedbythefollowinganalysis.LeverpressesANOVAconductedforleverpressesforthereversalstagerevealedmaineectsofclassication,

F(1,44)=1.23,p=.273,2p=.03andlever,F(1,44)=2.23,p=.143,2p=.05,butnoeectofreward,F(1,44)=0.07,p=.795,2p.01,orblock,F(3.48,153.07)=3.29,p=.017,2p=.07.Therewasatwo-wayin-teractionbetweenrewardandlever,F(1,44)=0.43,p=.515,2p.01andbe-tweenblockandlever,F(1.66,72.91)=0.81,p=.427,2p=.02,butnotbe-tweenrewardandclassication,F(1,44)=0.08,p=.779,2p.01,notbetweenrewardandblock,F(3.48,153.07)=4.53,p=.003,2p=.09,classicationandblock,F(3.48,153.07)=1.28,p=.282,2p=.03,orbetweenclassicationandlever,F(1,44)=1.73,p=.195,2p=.04.Therewasatripleinteractionbetweenreward,classicationandlever,F(1,44)=1.49,p=.228,2p=.03andbetweenclassication,blockandlever,F(1.66,72.91)=0.49,p=.582,2p=.01,butnotbetweenreward,classicationandblock,F(3.48,153.07)=2.95,p=.028,2p=.06,orreward,blockandlever,F(1.66,72.91)=0.32,p=.688,2p.01.Therewasnotafour-wayinter-actionbetweenreward,classication,blockandlever,F(1.66,72.91)=0.30,p=.697,2p.01.Food-wellentriesANOVAconductedforfood-wellentriesforthereversalstagerevealedmaineectsofblock,F(3.06,134.81)=0.62,p=.605,2p=.01andlever,F(1,44)=2.21,p=.144,2p=.05,butnotofreward,F(1,44)=2.87,p=.097,2p=.06orclas-sication,F(1,44)=0.39,p=.536,2p.01.Therewasatwo-wayinteractionbetweenblockandlever,F(2.27,99.98)=5.05,p=.006,2p=.10,butnotbe-tweenrewardandclassication,F(1,44)=2.29,p=.137,2p=.05,notbetween115 rewardandblock,F(3.06,134.81)=1.37,p=.253,2p=.03,classicationandblock,F(3.06,134.81)=0.32,p=.813,2p.01,rewardandlever,F(1,44)=0.11,p=.746,2p.01orbetwee

nclassicationandlever,F(1,44)=4.27,p=.045,2p=.09.Therewasatripleinteractionbetweenreward,classicationandlever,F(1,44)=6.09,p=.018,2p=.12,butbetweenreward,classicationandblock,F(3.06,134.81)=0.36,p=.786,2p.01,reward,blockandlever,F(2.27,99.98)=0.83,p=.454,2p=.02,orclassication,blockandlever,F(2.27,99.98)=0.56,p=.594,2p=.01.Therewasnotafour-wayinteractionbetweenreward,classication,blockandlever,F(2.27,99.98)=1.40,p=.250,2p=.03.DiscussionThepurposeofthisexperimentwastoreplicatethendingsinExperiment3.How-ever,theresultsfromthisexperimentarenotabletoconrmordisprovetheseresults,asveryfewanimalsshowedsign-trackingbehaviour.ThemediansplitusedtoclassifytheratsinSTandGT,labelsanimalsasGTeveniftheyhaveveryfewlever-presses.Thishasnotbeenanissueinthepast,asthedistributionofSTandGTwasequal,withanimalsthatexclusivelylever-press,othersthatonlyengagewiththefood-well,andamajoritythatdisplaybothtypesofbehaviourdespiteabiastooneorother.Dif-ferencesincohortsinthedistributionofbehaviouraredocumented(e.g.,Fitzpatricketal.,2013),howeverforthisexperimentthedistributionmateriallylimitstheconclusionsthatcanbedrawn.4.1.4DiscussionExperiments3and4ThepurposeofExperiment3andExperiment4wastoinvestigatethepredictionfromHeiDIthatincreasingrewardvalue,andthuspresumablyincreasingUS,wouldse-lectivelyincreasetheratesofgoal-trackingbehaviourrelativetosign-trackingbehaviour.Experiment3suggestedthatincreasingUSvaluebyprovidingtworewardpelletsratherthanoneresultsinhigherlevelsofsign-tracking,contrarytoHeiDIpredictions.How-116 ever,Experiment4didnotreplicatethesendings.Instead,Expe

riment4showednodierencesinconditionedbehaviourforthemanipulationofrewardmagnitude.TheresultsfromExperiment4,however,needtobetreatedwithcautionbecausethedis-tributionofbehaviour,heavilyskewedtowardsgoal-trackingbehaviour,isprobablynotsensitiveenoughtoprovideanyinsightintothepatternsofleverpressandfood-wellentrybehavioursacrossbothSTandGTanimals.NotwithstandingthepotentialissueswithExperiment4,itisrelativelyclearacrosstheseexperimentsthatincreasingthenumberofrewardpelletsfromonetotwodidnot-asmayhavebeenpredictedbytheHeiDImodel-produceanincreaseingoal-trackingbehaviourrelativetosign-trackingbehaviour.Thatsaid,thesimulationsprovidedinChapter3relatingtodownshiftunblockingdosuggestthatchangingthenumberofrewardsmayhaveeectsbeyondsimplyincreasingtheoverallvalueofreward.ThusamorediagnostictestofthepredictionsdrawnfromHeiDIregardingchangesinrewardmagnitude(andpresumablyofBus)mayinvolvemanipulationsthatdonotconfoundrewardnumberwithmagnitude(e.g.usingdierenttypesofpelletwhereoneisprefer-abletotheother,ordierentvolumesofsucrosereward).GiventheimportanceofUSvalueinthecontrolofbehaviour-especiallypotentiallymaladaptivebehaviour(e.g.,Flageletal.,2009;Meyeretal.,2012)-suchfuturestudieswouldbeimportant.Inaddition,whileExperiments3and4didnotconclusivelyaddressoneofthenovelpredictionsfromHeiDI,theydidreinforcethendingsfromExperiment1thatreversaloftherewardcontingenciesacrosslevershadafaster/largerimpactongoal-trackingbehaviourthanonsign-trackingbehaviourinbothpredominantlySTorGTanimals.TherelativedierenceinsensitivitytocontingencyreversalisaccountedforbyHeiDI.4.2Featurepositiveeect4.2.1Introduction

JenkinsandSainsbury(1970)investigateddiscriminationlearninginpigeonsus-117 ingparadigmsinwhichexposedpigeonstoeitherafeaturepositivedesign(FP;e.g.,AB+/B-)orfeaturenegativedesign(FN;e.g.,B+/AB-).Forthefeaturepositivede-sign,acompoundofvisualstimuli(e.g.,AB)wouldbefollowedbyareward,butthepresentationofonlyoneoftheelements(e.g.,B)wouldnotbefollowedbyareward.Forthefeaturenegativedesign(B+,AB-)asingleelementisfollowedbyareward(e.g.,B),butthepresentationofacompoundwhichcontainstheuniqueelementwouldnotbefollowedbyareward(e.g.,AB).TheynoticedthatthebirdsintheFPgroupacquiredthediscriminationrapidlyandceasedrespondingtoBwhenpresentedalone,howeverthebirdsintheFNgroupexhibitedlittleevidenceofsuchlearning.AnotherinterestingndingwasthatthebirdsstarteddirectingmostofthepecksintheFPgrouptowardsA,theuniquefeature.Thesuperiorityinlearningafeaturepositivediscriminationcom-paredtoafeaturenegativediscriminationisarobusteectandhasbeenseeninrats(e.g.,Crowell&Bernhardt,1979),honeybees(e.g.,Abramsonetal.,2013),aswellashumans(e.g.,Lotz,Uengoer,Koenig,Pearce,&Lachnit,2012).InthestudyofJenkinsandSainsbury(1970),pigeonsshowednosignoflearninginthefeaturenegativegroup,howeverthereareinstanceswherethisdiscriminationcanbeacquired.Theevidencesuggeststhatlearninginafeaturenegativediscriminationisslowerthaninamatchedfeaturepositivediscrimination,andmayhappenonlyunderparticularconditions(e.g.,morelikelytoobservelearninginasequentialbutnotsimultaneouspresentationofthestimuli,seeGokey&Collins,1980).SimulationsofHeiDI'spredictionsareshowninFigure17andFigure18.Figure17showssimulationsofconditioningforaFPdesign(PanelAandB)andFNdesign(PanelCandD

)forwhenparametersarearrangedforabiastowardsRCS(AandB=.50,US=.30.).ExaminationsofFigure17(PanelsAandC)showsthatthedierenceinassociativestrengthbetweenABandBislargerfortheFPthanFNdesign-aclearfeaturepositiveeect.However,whentheseassociativestrengthsaredecomposedintoRCSandRUS(PanelsBandD)itisapparentthatthedierenceinresponsetendencybetweenABandBbetweenFPandFNdesignsisparticularlyapparentforRCSand118 negligibleforRUS.Thus,HeiDIpredictsthatinanimalsbiasedtosign-tracking,thefeaturepositiveeectshouldbemoreapparentintheirsign-trackingbehaviourthantheirgoal-trackingbehaviour.Figure18showssimulationsofconditioningforaFPdesign(PanelAandB)andFNdesign(PanelCandD)forwhenparametersarearrangedforabiastowardsRUS(AandB=.30,US=.50).ExaminationofPanelsAandCofFigure18alsoshowsacleardierenceinassociativestrengthbetweenABandAconsistentwithafeaturepositiveeectwhentheparametersre ectabiastowardsgoal-trackingbehaviour.However,whentheseassociativestrengthsaredecomposedintoRCSandRUS(PanelsBandD)itisapparentthatthedierenceinresponsetendencybetweenABandBbetweenFPandFNdesignsislargerforRCSthanRUS.Thus,HeiDIpredictsthateveninanimalsbiasedtogoal-tracking,thefeaturepositiveeectshouldbemoreapparentintheirsign-trackingbehaviourthantheirgoal-trackingbehaviour.Totestthesepredictionsratswereexposedtoeitherafeaturepositive(FP)orafea-turenegative(FN)discriminationlearning.Thepurposeofthisexperimentwastoseehowafeaturepositiveandnegativediscriminationlearningtranslatedintoperformance.Moreover,itisknownthatdierenttypesofstimulileadtodierentsign-trackingbe-haviour(e.g.,Hearst&F

ranklin,1977).Thesimulationspredictdierentlevelsofob-serveddierenceswhenthebiasisarrangedtowardstheCSorUSandforthisreason,forhalfoftheanimals,Aisalever(designTable4)andfortheotherhalfisalight(designTable5),whichpotentiallyimplydierentlevelsofCS.TheanimalsintheFPgroupwereexposedtotwotypesoftrials:acompoundAB,whichwasfollowedbyareinforcer(AB+)andsinglepresentationsofB,whichwasnon-reinforced(B-).TheanimalsintheFNgroupreceivedpresentationsofthecompoundABwhichwerenotre-inforced(AB-)andpresentationsofB,whichwerefollowedbyareinforcer(B+).BwasalwaysaleverandAwasforhalftheanimalsalever(Experiment5:Levercondition;e.g.,foraparticularanimalAisleftleverandBisrightlever)andfortheotherhalfwasalight(Experiment6:Lightcondition;e.g.,foraparticularanimalAisleftlight119 andBisleftlever).WhenAisalever,Experiment5,IrecordleverpressesandIamabletolookathowtheanimalsinteractbothwithAandB,butforwhenAisalight,theanimalscanapproachandorienttowardsthelight,howeverthisbehaviourwasnotrecorded.4.2.2Experiment5:LeverconditionMethodAnimalsandapparatusThirty-twoListerHoodednavemaleratswereused(suppliedbyCharlesRiver,UK).Theirmeanadlibitumweightbeforethestartoftheexperimentwas286g(range:244-326g)andtheyweremaintainedatbetween85and95%oftheseweightsbygivingthemrestrictedaccesstofoodattheendofeachday.Theratshadcontinuousaccesstowaterwhentheywereintheircages.TheconditioningboxesusedinthisexperimentaredescribedinExperiment1.ProcedureTheratshadtwo24-minpre-trainingsessionswherefoodpelletsweredeliveredonavariable-time(VT)60-sschedule(range:40{80s).Ratsthenreceivedasinglesessionoftrainingoneachofthenext12daysoftraining,whichoccurr

edatthesametimeofdayforagivenrat.Theexperimentconsistedintwoconditions:featurepositive(FP)andfeaturenegative(FN).TheanimalsintheFPgroupreceivedtwotypesoftrials:AB,whichwasfollowedbyareinforcer(AB+)andB,whichwasnon-reinforced(B-).TheanimalsintheFNgroupreceivedpresentationsofABwhichwerenotreinforced(AB-)andpresentationsofB,whichwasfollowedbyareinforcer(B+).AandBweretwolevers(counterbalanced,forhalftheanimalsAwastheleftleverandBtherightlever,andfortheremaininganimalsAwastherightleverandBwastheleftlever).OnABtrials,thetwoleverswerepresentedsimultaneously.Alltheanimalshad20trialsofthecompoundABand20presentationofBpersession,onesessionperday.Theorderinwhichtypeoftrialswerepresentedwasrandomwiththeconstraintthatthere120 couldnotbemorethanthreesametypetrialsinsuccession.Thetrialsweredeliveredonavariable-time(VT)60-sschedule(range:40{80s).Ratswererandomlyassignedtotheexperimentalgroups.Table4:DesignOftheExperiment5:LeverCondition ClassicationGroupTraining STFeaturepositive(FP)AB+/B-Featurenegative(FN)AB-/B+ GTFeaturepositive(FP)AB+/B-Featurenegative(FN)AB-/B+ Note:STreferstoasign-trackerandGTreferstogoal-tracker.RatswereclassiedasSTorGTonthebasisoftheirbiastowardsleverpressingorenteringthefood-wellduringthenalblockoftrainingonthereinforcedlever(orcompound).Both\A"and\B"werelevers.\+"denotesthestimulusorcompoundwasreinforced,and\-"denotesthatitwasnon-reinforced.DataAnalysisSuccessivesessionsduringthetrainingwerecombinedinto6Ö2-dayblocks(6Ötraining:T1-T6).Attheendofthetrainingphase,theratsweresplitintotwogroups,sign-trackers(ST)andgoal-trackers(GT),basedontheirtendencytoengagewiththeleverandthefood-well.The

ratsweresplitintotwogroups,STsandGTs,usingthebiasscoredescribedinExperiment1.Biasscoresabove-.12wereclassiedasGTsandbiasscoresbelow-.18asSTs.Thedistributionofsign-trackingandgoal-trackingbehaviourisshowninFigure19.MixedANOVAswereconductedseparatelyforleverpressingandfood-wellentrieswithwithin-subjectsfactorsofblock(6levels:T1-T6)andlever(2levels:reinforcedvsnon-reinforced),andbetween-subjectsfactorsfeature(2levels:FPvsFN)andclassi-cation(2levels:STvsGT).TolookathowratsresponddierentlytoAandBwhenlearningoccurrsaseparateanalysisisalsopresented,wherecomparisonsaremadeforrespondingtoA,BandB(AB)(whenBispartofthecompoundAB).121 Figure17:Simulationsofconditioningforafeaturepositivedesign(FP;PanelAandB)andafeaturenegativedesign(FN;PanelCandD).PanelsAandCdepicttheoutputvaluesforVCOMB-BandVCOMB-ABandpanelsBandDshowthecorrespondingoutputvaluesforRCSandRUSforbothBandAB.TheparameterswerechosentoresultinabiastowardstheCSwithAandB=.50,US=.30.ResultsThemainresultsfromExperiment5areshowninFigure20andinFigure21.Myanalysiswillbeginwiththelevercondition,whereIseparatebyresponse(leverpressingandfood-wellentries)andcondition(featurepositiveandfeaturenegative).Thisanalysisalsoincludesseparatingbetween\A",\B"and\B(AB)"forleverpressing.Forthelightcondition,Iseparatebyresponse(leverpressingandfood-wellentries)andcondition(featurepositiveandfeaturenegative).122 Figure18:Simulationsofconditioningforafeaturepositivedesign(FP;PanelAandB)andafeaturenegativedesign(FN;PanelCandD).PanelsAandCdepicttheoutputvaluesforVCOMB-BandVCOMB-ABandpanelsBandDshowthecorrespondingoutputvaluesforRCSandRUSforbothBandAB.Theparameters

werechosentoresultinabiastowardstheUSwithAandB=.30,US=.50.LeverpressesInspectionoftheleftandright-handsideoftheupperpanelFigure20,whichdepictsleverpressesforthelevercondition,suggeststhattheanimalsintheFPgrouplearnthediscriminationbetweenreinforcedandnon-reinforcedstimulibetterthantheFNgroup.FortheFPconditionbothSTsandGTsshowadiscriminationbetweenreinforcedandnon-reinforcedlever,whichisnotobservedintheFNgroup.Thisdescriptionissupportedbythefollowinganalysis.AnANOVAconductedforleverpressesshowedmaineectsoffeature(feature123 Figure19:Distributionofsign-tracking(leverpresses)andgoal-tracking(food-wellentries)behaviourforExperiment5per(10-s)trialduringlastblockoftraining(T6)forL1(+jj-).Theblacksymbolscorrespondtosign-trackers(ST)andtheclearsymbolstogoal-trackers(GT).positivevsfeaturenegative),F(1,28)=11.56,p=.002,2p=.29,classication,F(1,28)=26.81,p.001,2p=.49,block,F(3.55,99.43)=41.71,p.001,2p=.60,andlever,F(1,28)=68.61,p.001,2p=.71.Therewasalsoaninteractionbe-tweenfeatureandclassication,F(1,28)=8.14,p=.008,2p=.23,featureandblock,F(3.55,99.43)=7.05,p.001,2p=.20,classicationandblock,F(3.55,99.43)=13.83,p.001,2p=.33,criticallybetweenfeatureandlever,F(1,28)=95.95,p.001,2p=.77(re ectingthebetterdiscriminationinthefeaturepositivegroups),classi-cationandlever,F(1,28)=12.62,p=.001,2p=.31andbetweenblockandlever,F(2.15,60.27)=11.87,p.001,2p=.30.Thereweretripleinteractionsbetweenfeature,classicationandblock,F(3.55,99.43)=5.36,p.001,2p=.16,betweenfeature,classicationandlever,F(1,28)=10.79,p=.003,2p=.28,andbetween124 Fig

ure20:ResultsfromExperiment5:theeectsofafeaturepositivedesignonsign-trackingandgoal-tracking.Mean(+SEM)leverpresses(upperpanels)andfood-wellentries(lowerpanels)per(10-s)trialacrosstraining(T1{T6)separatedbycondition(leftpanels:featurepositive;rightpanels:featurenegative).RatsingroupFPreceivedreinforcedtrialsofacompoundAB\Reinforced(+)"andnon-reinforcedtrialsofB\Non-reinforced(-)"andratsingroupFNreceivednon-reinforcedtrialsofacompoundAB\Non-reinforced(-)"andreinforcedtrialsofB\Reinforced(-)".Theblacksymbolscorrespondtosign-trackers(ST)andthegreysymbolstogoal-trackers(GT).feature,blockandlever,F(2.15,60.27)=12.48,p.001,2p=.31,butnotbetweenclassication,blockandlever,F(2.15,60.27)=2.19,p=.117,2p=.07.Thefour-wayinteractionwasnotsignicant,F(2.15,60.27)=1.65,p=.198,2p=.06.Thcriticalinteractionbetweenfeatureandlever,(F(1,28)=95.95,p.001,2p=.77)isfurtheranalysedtoaddresswhetherthediscriminationisacquiredatallintheFNgroup.Featurepositive:AnANOVAconductedforleverpressesforfeaturepositivede-signshowedmaineectsofclassication,F(1,14)=6.58,p=.022,2p=.32,block,F(1.97,27.61)=11.82,p.001,2p=.46,andlever,F(1,14)=89.35,p.001,2p=.86.Therewereinteractionsbetweenclassicationandlever,F(1,14)=12.78,125 Figure21:ResultsfromExperiment5splitbytypeofstimuli:AvsBvsB(AB).Mean(+SEM)leverpressestrialacrosstraining(T1{T6)separatedbyclassication(leftpanels:ST,sign-trackers;rightpanels:GTs,goal-trackers)anddesign(upperpanels:FP;lowerpanels:FN).RatsingroupFPreceivedreinforcedtrialsofacompoundAB,andnon-reinforcedtrialsofB,andratsingroupFNreceivednon-reinforcedtrialsofacompoundAB,an

dreinforcedtrialsofBp=.003,2p=.48andbetweenblockandlever,F(1.78,24.93)=14.10,p.001,2p=.50,butnotbetweenclassicationandblock,F(1.97,27.61)=1.62,p=.216,2p=.10,andtherewasnotripleinteractionbetweenclassication,blockandlever,F(1.78,24.93)=2.11,p=.147,2p=.13.Next,Ianalysetheinteractionbetweenclas-sicationandleverbyconductingseparatedANOVAsfortheSTandtheGTgroups.AnANOVAconductedfortheSTgroupforleverpresses(FP)showsmaineectsofblock,F(1.56,9.38)=6.87,p=.018,2p=.53lever,F(1,6)=156.78,p.001,2p=.96,andaninteractionbetweenblockandlever,F(5,30)=8.26,p.001,2p=.58.AparallelanalysisfortheGTgroupshowsmaineectsofblock,F(5,40)=4.17,p=.004,2p=.34,lever,F(1,8)=14.21,p=.005,2p=.64andaninteractionbetweenblockandlever,F(1.95,15.58)=4.72,p=.026,2p=.37.BothSTandGTlearnthedis-criminationbetweenthereinforcedandthenon-reinforcedlever,howevertheeectsize126 isbiggerforST.Featurenegative:Aparallelanalysiswasconductedforthefeaturenegativedesignwhichshowedmaineectsofclassication,F(1,14)=20.29,p.001,2p=.59,block,F(5,70)=30.19,p.001,2p=.68,andlever,F(1,14)=6.69,p=.022,2p=.32.Therewasaninteractionbetweenclassicationandblock,F(5,70)=13.28,p.001,2p=.49,butnotbetweenclassicationandlever,F(1,14)=0.21,p=.654,2p=.01,orcriticallybetweenblockandlever,F(5,70)=0.41,p=.838,2p=.03.Therewasnotatripleinteractionbetweenclassication,blockandlever,F(5,70)=0.79,p=.562,2p=.05.Itisworthnotingthatthereweremoreresponsesonthenon-reinforcedleverthanthereinforcedlever,theoppositeofanaccuratediscrimination.Food-wellentriesInspectionofthe

leftandright-handsideofthelowerpanelinFigure20,whichdepictsfood-wellentriesforthelevercondition,suggeststhattheGTanimalsintheFPgrouplearnthediscriminationbetweenreinforcedandnon-reinforcedstimulibetterthanST.TheanimalsintheFNgroupdonotlearnthediscrimination.Thisdescriptionissupportedbythefollowinganalysis.AnANOVAconductedforfood-wellentriesrevealedmaineectsofclassication,F(1,28)=31.07,p.001,2p=.53,block,F(2.58,72.32)=3.50,p=.025,2p=.11,andlever,F(1,28)=28.88,p.001,2p=.51,butnoeectoffeature,F(1,28)=1.62,p=.214,2p=.05.Therewereinteractionsbetweenclassicationandblock,F(2.58,72.32)=2.98,p=.044,2p=.10,criticallybetweenfeatureandlever,F(1,28)=18.05,p.001,2p=.39,andbetweenclassicationandlever,F(1,28)=12.70,p=.001,2p=.31,butnotbetweenfeatureandclassication,F(1,28)=1.17,p=.290,2p=.04,orbetweenfeatureandblock,F(2.58,72.32)=1.93,p=.140,2p=.06.Thereweretripleinteractionsbetweenfeature,classicationandlever,F(1,28)=8.37,p=.007,2p=.23andbetweenclassication,blockandlever,F(2.77,77.65)=7.90,p.001,2p=.22,butnotbetweenfeature,classicationandblock,F(2.77,77.65)=4.55,p=.007,2p=.14F(2.58,72.32)=0.87,p=.449,2p=.03,127 orbetweenfeature,blockandlever,F(2.77,77.65)=2.50,p=.070,2p=.08.Therewasafour-wayinteraction,F(2.77,77.65)=2.98,p=.040,2p=.10.Featurepositive:AnANOVAconductedforfood-wellentriesinthefeaturepositivedesignshowedmaineectsofclassication,F(1,14)=15.25,p=.002,2p=.52,block,F(2.89,40.52)=4.40,p=.010,2p=.24andlever,F(1,14)=24.15,p.001,2p=.63.Therewereinteractionsbetweenclassicationan

dblock,F(2.89,40.52)=4.19,p=.012,2p=.23,classicationandlever,F(1,14)=10.87,p=.005,2p=.44andcriticallyblockandlever,F(2.29,32.04)=4.26,p=.019,2p=.23,andatripleinteractionbetweenclassication,blockandlever,F(2.29,32.04)=6.42,p=.003,2p=.31.TheresultssuggestthatanimalslearntheFPdiscrimination(blockandleverinteraction,F(2.29,32.04)=4.26,p=.019,2p=.23)howevertheinteractionbetweenclassicationandlever(F(1,14)=10.87,p=.005,2p=.44)suggestslearningisbetterforGTsanimals.Next,IanalysetheinteractionbetweenclassicationandleverbyconductingseparatedANOVAsfortheSTandtheGTgroups.AnANOVAconductedfortheSTgroupforfood-wellentries(FP)showsamaineectofblock,F(2.38,14.26)=6.50,p=.008,2p=.52andinteractionbetweenblockandlever,F(5,30)=8.31,p.001,2p=.58,howeverthereisnomaineectoflever,F(1,6)=1.27,p=.303,2p=.17.AparallelanalysisfortheGTgroup,showsmaineectsofblock,F(5,40)=3.49,p=.010,2p=.30,lever,F(1,8)=36.23,p.001,2p=.82andaninteractionbetweenblockandlever,F(2.04,16.31)=6.09,p=.010,2p=.43.TheseresultssuggestthattheGTgroupshowslearningbetweenthereinforcedandthenon-reinforcedlever.STsshowadierencebetweenthereinforcedandnon-reinforcedleversuggestedbytheblockandleverinteraction,howeverthisdierenceislessclearwithnooveralleectoflever.Featurenegative:Aparallelanalysisconductedforthefeaturenegativedesignre-vealedmaineectsofclassication,F(1,14)=16.55,p=.001,2p=.54andlever,F(1,14)=7.63,p=.015,2p=.35,butnotaneectofblock,F(2.12,29.74)=2.26,p=.119,2p=.14.Therewasnotaninteractionbetweenclassicationandblock,F(2.12,29

.74)=1.31,p=.286,2p=.09,classicationandlever,F(1,14)=2.72,128 p=.121,2p=.16,criticallyblockandlever,F(5,70)=0.75,p=.591,2p=.05,andnotripleinteractionbetweenclassication,blockandlever,F(5,70)=1.77,p=.130,2p=.11.Theseresultssuggesttheanimalsmightdiscriminatebetweenthereinforcedandnon-reinforcedstimuli,buttheeectisnotlargeenoughtoshowablockandleverinteraction.NumericallythisseemstobetrueonlyforGTs,possiblyduetoa ooreectinSTs,howevertherewasnointeractionbetweenleverandclassication.Analysisbystimulus:AvsBvsB(AB)Aspreviouslymentionedinthedataanalysissection,forExperiment5,bothre-spondingtoAandBwasmeasured,eitheralone,orwithinABcompound.AsthestimuliA,BandB(AB)havedierentmeaningdependingoftheconditionthiswillbeanalysedseparatedforfeaturepositiveandnegative,respectively.InspectionoftheFPgroup(upperpanelofFigure21),wherethecompoundABwasreinforcedandthepre-sentationofBwasnot,suggeststhatrespondingtothereinforcedcompoundismainlydrivenbyrespondingtoA.Neither,STsorGTslearnthediscriminationbetweenthepresentationofBaspartofthecompound(reinforced)andnon-reinforcedpresentationsofB,interactingmorewithBwhenpresentedalone.InspectionoftheFNgroup(lowerpanelofFigure21),wherethecompoundABwasnon-reinforcedandthepresentationofBwasreinforced,suggeststhattherespondingismainlydrivenbyB,howeverthean-imalsdonotlearnwhenBpredictsornotareinforcer,showingnoevidenceoflearning.Thisdescriptionissupportedbythefollowinganalysis.Featurepositive:AnANOVAconductedforthefeaturepositivedesignonleverpressesintheleverconditionseparatedbystimulus,AvsBvsB(AB),showedmaineectsofclassication,F(1,14)=6.58,p

=.022,2p=.32,block,F(1.97,27.61)=11.82,p.001,2p=.46andstimulus,F(1.07,15.04)=91.27,p.001,2p=.87.Therewasaninteractionbetweenclassicationandstimulus,F(1.07,15.04)=11.11,p=.004,2p=.44,andbetweenblockandstimulus,F(2.17,30.31)=16.35,p.001,2p=.54,butnotbetweenclassicationandblock,F(1.97,27.61)=1.62,p=.216,2p=.10.129 Therewasnotathreewayinteraction,F(2.17,30.31)=1.63,p=.211,2p=.10.Next,IanalyseseparatelyAvsBandBvsB(AB),toseewhichstimulitheanimalsdiscriminatebetween.AnANOVAconductedtheforfeaturepositivedesign,withlevelsforstimulusAvsB,revealedmaineectsofclassication,F(1,14)=7.18,p=.018,2p=.34,block,F(1.97,27.61)=14.43,p.001,2p=.51andstimulus,F(1,14)=77.04,p.001,2p=.85.Therewasaninteractionbetweenclassicationandstimulus,F(1,14)=12.60,p=.003,2p=.47,andbetweenblockandstimulus,F(2.00,28.06)=15.81,p.001,2p=.53,butnotbetweenclassicationandblock,F(1.97,27.61)=1.54,p=.233,2p=.10.Therewasnotathree-wayinteraction.F(2.00,28.06)=1.81,p=.182,2p=.11.AparallelcomparisonbetweenBandB(AB),revealedmaineectsofblock,F(2.73,38.16)=3.64,p=.024,2p=.21,andstimulus(BvsB(AB)),F(1,14)=79.69,p.001,2p=.85,butnotclassication,F(1,14)=0.00,p=.991,2p.01.Therewasaninteractionbetweenblockandstimulus,F(5,70)=2.97,p=.017,2p=.18,butnotbetweenclassicationandblock,F(2.73,38.16)=0.20,p=.881,2p=.01,orbetweenclassicationandstimulus,F(1,14)=0.19,p=.673,2p=.01,orathree-wayinteraction,F(5,70)=0.38,p=.858,2p=.03.TheinteractionbetweenblockandstimulusfortheBvsB(AB)analysis(F(5,70)=2.97,p=.017,2p=.18),showsre-spo

ndingtoBbeinghigherwhenpresentedalonethaninthecompound,B(AB).ThisisinconsistentlearningaboutBpartofpartofthecompoundbeingreinforced,suggestingthatalltherespondingfortheABcompoundisdrivenbyrespondingtoA.Featurenegative:Aparallelanalysisforthefeaturenegativedesignonleverpressesintheleverconditionseparatedbystimulus,showedmaineectsofclassication,F(1,14)=20.29,p.001,2p=.59,block,F(5,70)=30.19,p.001,2p=.68andstimulus,F(1.05,14.69)=77.79,p.001,2p=.85.Therewasaninteractionbetweenclassicationandblock,F(5,70)=13.28,p.001,2p=.49,classicationandstimulus,F(1.05,14.69)=22.26,p.001,2p=.61,blockandstimulus,F(4.02,56.25)=31.61,130 p.001,2p=.69andathree-wayinteractionbetweenclassication,blockandstimu-lus,F(4.02,56.25)=11.75,p.001,2p=.46.AsforFP,next,IanalyseseparatelyAvsBandBvsB(AB),toseebetweenwhichstimulitheanimalsdiscriminate.AnANOVAconductedforfeaturepositivenegative,withlevelsforstimulusAvsB,revealedmaineectsclassication,F(1,14)=17.42,p.001,2p=.55,block,F(2.87,40.14)=23.27,p.001,2p=.62andstimulus(AvsB),F(1,14)=71.23,p.001,2p=.84.Therewasaninteractionbetweenclassicationandblock,F(2.87,40.14)=12.16,p.001,2p=.46,classicationandstimulus,F(1,14)=20.51,p.001,2p=.59,andcriticallybetweenblockandstimulus,F(5,70)=33.41,p.001,2p=.70.Therewasatripleinteractionbetweenclassication,blockandstimulus,F(5,70)=13.57,p.001,2p=.49.AparallelcomparisonbetweenBandB(AB),revealedmaineectsofclassication,F(1,14)=21.27,p.001,2p=.60,block,F(5,70)=33.02,p.001,2p=.70,butnotstimulus,F(1,14)=0.36,p=.560,2p=.02.There

wasaninteractionbetweenclassi-cationandblock,F(5,70)=13.67,p.001,2p=.49,butnotbetweenclassicationandstimulus,F(1,14)=0.20,p=.660,2p=.01,blockandstimulus,F(5,70)=1.91,p=.104,2p=.12orathree-wayinteraction,F(5,70)=0.95,p=.454,2p=.06.TheseresultsshowthatevenifthereisnorespondingforAasthenon-reinforcedstimulus(AvsBcomparison,blockandstimulusinteraction,F(5,70)=33.41,p.001,2p=.70),theanimalsdonotlearnaboutthefactthatBisreinforcedwhenpresentedalone(BvsB(AB)analysis,blockandstimulusinteraction,F(5,70)=1.91,p=.104,2p=.12).DiscussionOverall,theseresultsshowafeaturepositiveeect,animalslearnFPdiscrimination,howeverthereislittleevidenceoflearningtheFNdiscrimination.TheanalysissplitbystimulusshowsthatlearningintheFPdesignisdrivenbylearningaboutthefeaturepositive,A.RespondingtoB,doesnotshowlearningaboutB,whenitisornon-reinforced.ItisworthnotingthatbothSTandGTshowlearninginFPaslever131 presses,howeverforfood-wellentriesitisonlyevidentforGTanimals(upperleftpanelofFigure20).ThissupportsHeiDI'spredictionthatafeaturepositiveeectwouldbemoreevidentinCS-orientedresponses.4.2.3Experiment6:LightconditionMethodAnimalsandapparatusThirty-twomaleratswereused(suppliedbyCharlesRiver,UK).Theirmeanadlibitumweightbeforethestartoftheexperimentwas270g(range:220-312g)andtheyweremaintainedatbetween85and95%oftheseweightsbygivingthemrestrictedaccesstofoodattheendofeachday.Theratshadcontinuousaccesstowaterwhentheywereintheircages.TheconditioningboxesusedinthisexperimentaredescribedinExperiment1.ProcedureTheprocedurewasidenticaltoExperiment5withtheexceptionthatBwasalwaysaleverandAwasalightabovethestimulus

B(counterbalanced,forhalftheanimalsBwastheleftleverandAwastheleftlightandfortheotherhalf,BwastherightleverandAwastherightlight).DataanalysisTheanalysishasbeenconductedinasimilarwaytoExperiment5,howeverIdidnotrecordrespondingtothelightinanyway.Thismeansthatthecomparisonbetweenreinforcedandnon-reinforcedleverforthelightconditionisonlybasedonrespondingto\B",theleverwhichwaseitherpresentedaloneorincompoundwiththelight.FortheFPdesign,Bwasreinforcedwhenpresentedwiththelightandnon-reinforcedwhenpresentedaloneandforFNdesign,Bwasreinforcedwhenpresentedaloneandnon-reinforcedwhenpresentedwiththelight.Successivesessionsduringthetrainingwerecombinedinto6Ö2-dayblocks(6Ö132 Table5:DesignOftheExperiment6:LightCondition ClassicationGroupTraining STFeaturepositive(FP)AB+/B-Featurenegative(FN)AB-/B+ GTFeaturepositive(FP)AB+/B-Featurenegative(FN)AB-/B+ Note:STdenotesasign-tracker;GTdenotesagoal-tracker.RatswereclassiedasSTorGTonthebasisoftheirbiastowardsleverpressingorenteringthefood-wellduringthenalblockoftrainingonthereinforcedlever(orcompound).\A"wasalightand\B"wasalever.\+"denotesthestimulusorcompoundwerereinforced,and\-"denotestheywerenon-reinforced.training:T1-T6).Attheendofthetrainingphase,theratsweresplitintotwogroups,sign-trackers(ST)andgoal-trackers(GT),basedontheirtendencytoengagewiththeleverandthefood-well,asdescribedinExperiment1.Biasscoresabove.08wereclassiedasGTsandbiasscoresbelow-.08asSTs.Thedistributionofsign-trackingandgoal-trackingbehaviourisshowninFigure22.ResultsLeverpressesInspectionoftheleftandright-handsideoftheupperpanelinFigure23,whichdepictsleverpressesforthelightcondition,suggeststha

tneithertheanimalsintheFPgrouporintheFNgrouplearnthediscriminationbetweenreinforcedandnon-reinforcedstimuli.InbothFPandFNdesigns,STsshowhighernumberofleverpressesthanGTs.Thisdescriptionissupportedbythefollowinganalysis.AnANOVAconductedforthelightconditiononleverpressesrevealedmaineectsofclassication,F(1,28)=11.24,p=.002,2p=.29,andblock,F(2.14,60.02)=8.55,p.001,2p=.23,butnoeectoffeature,F(1,28)=0.04,p=.838,2p.01orlever,F(1,28)=0.27,p=.607,2p.01.Therewerenointeractionsbetweenfeatureandclassication,F(1,28)=0.18,p=.678,2p.01,orbetweenfeatureandblock,133 Figure22:Distributionofsign-tracking(leverpresses)andgoal-tracking(food-wellentries)behaviourforExperiment6per(10-s)trialduringlastblockoftraining(T6)forL1(+jj-).Theblacksymbolscorrespondtosign-trackers(ST)andtheclearsymbolstogoal-trackers(GT).F(2.14,60.02)=0.20,p=.833,2p.01,classicationandblock,F(2.14,60.02)=3.03,p=.052,2p=.10,criticallyfeatureandlever,F(1,28)=1.27,p=.269,2p=.04,classicationandlever,F(1,28)=0.59,p=.449,2p=.02orbetweenblockandlever,F(5,140)=1.39,p=.231,2p=.05.Therewerenotripleinteractionsbe-tweenfeature,classicationandblock,F(2.14,60.02)=0.17,p=.855,2p.01,betweenfeature,classicationandlever,F(1,28)=0.52,p=.477,2p=.02,fea-ture,blockandlever,F(5,140)=2.02,p=.080,2p=.07orbetweenclassication,block,lever,F(5,140)=1.03,p=.403,2p=.04.Therewasnofour-wayinteraction,F(5,140)=0.51,p=.772,2p=.02.Astherewasnointeractionbetweenfeatureandlever(F(1,28)=1.27,p=.269,2p=.04)nofurtheranalysisarepresentedforleverpresses.Theseresultssuggestthattheani

malsdonotshowlearningineithertheFP134 Figure23:ResultsfromExperiment6:theeectsofafeaturepositivedesignonsign-trackingandgoal-tracking.Mean(+SEM)leverpresses(upperpanels)andfood-wellentries(lowerpanels)per(10-s)trialacrosstraining(T1{T6)separatedbycondition(leftpanels:featurepositive;rightpanels:featurenegative).RatsingroupFPreceivedreinforcedtrialsofacompoundAB\Reinforced(+)"andnon-reinforcedtrialsofB\Non-reinforced(-)"andratsingroupFNreceivednon-reinforcedtrialsofacompoundAB\Non-reinforced(-)"andreinforcedtrialsofB\Reinforced(-)".Theblacksymbolscorrespondtosign-trackers(ST)andthegreysymbolstogoal-trackers(GT).orFNdesign.Food-wellentriesInspectionoftheleftandright-handsideofthelowerpanelinFigure23,whichdepictsfood-wellentriesforthelightcondition,suggeststhattheGTsanimalsintheFPgrouplearnthediscriminationbetweenreinforcedandnon-reinforcedstimuli.TheanimalsintheFNgroupshownoevidenceoflearning.Thisdescriptionissupportedbythefollowinganalysis.ANOVAconductedforthelightconditiononfood-wellentriesrevealedmaineectsofclassication,F(1,28)=18.88,p.001,2p=.40,andlever,F(1,28)=17.86,p.001,2p=.39,butnomaineectsoffeature,F(1,28)=0.02,p=.881,2p.01135 orblock,F(3.09,86.63)=0.65,p=.593,2p=.02.Therewereinteractionsbetweenfeatureandlever,F(1,28)=7.99,p=.009,2p=.22,betweenclassicationandlever,F(1,28)=7.70,p=.010,2p=.22andbetweenblockandlever,F(3.47,97.22)=5.30,p=.001,2p=.16,butnotbetweenfeatureandclassication,F(1,28)=0.02,p=.881,2p.01,featureandblock,F(3.09,86.63)=0.55,p=.653,2p=.02,orbetweenclassi-cationandblock,F(3.09,86.63)=2.14,p=.099,2p=.07.Ther

eweretripleinteractionsbetweenfeature,blockandlever,F(3.47,97.22)=3.95,p=.008,2p=.12andbetweenclassication,blockandlever,F(3.47,97.22)=2.28,p=.075,2p=.08,notbetweenfeature,classicationandblock,F(3.09,86.63)=0.49,p=.698,2p=.02,orbetweenfeature,classicationandlever,F(1,28)=5.32,p=.029,2p=.16.Therewasafour-wayinteractionbetweenfeature,classication,blockandlever,F(3.47,97.22)=3.05,p=.026,2p=.10.Featurepositive:AnANOVAconductedforfood-wellentriesforthefeaturepos-itivedesignrevealedmaineectsoflever,F(1,14)=13.02,p=.003,2p=.48,butnoeectofclassication,F(1,14)=7.81,p=.014,2p=.36orblock,F(2.32,32.48)=0.17,p=.871,2p=.01.Therewasaninteractionbetweenclassi-cationandlever,F(1,14)=6.76,p=.021,2p=.33,andcriticallybetweenblockandlever,F(3.07,43.04)=5.46,p=.003,2p=.28,butnotbetweenclassicationandblock,F(2.32,32.48)=1.01,p=.386,2p=.07.Therewasathree-wayinteractionbetweenclassication,blockandlever,F(3.07,43.04)=3.07,p=.036,2p=.18.Next,IconductseparateanalysisfortheSTandGTgroups,toseewhichanimalshavelearntthediscriminationbetweenthereinforcedandnon-reinforcedstimulus.AnANOVAconductedfortheSTgroupshowsamaineectoflever,F(1,4)=4.02,p=.115,2p=.50,butnoeectofblock,F(5,20)=1.30,p=.303,2p=.25andnointeractionbetweenblockandlever,F(2.46,9.85)=1.14,p=.370,2p=.22.AparallelanalysisfortheGTgroup,showsamaineectoflever,F(1,10)=22.61,p.001,2p=.69andaninteractionbetweenblockandlever,F(5,50)=9.89,p.001,2p=.50,butnoblockeect,F(2.27,22.70)=0.96,p=.408,2p=.09.TheseresultssuggestthatGTsshow136 bett

erdiscriminationbetweenthereinforcedandnon-reinforcedstimulusthanSTs.Featurenegative:Aparallelanalysisconductedforthefeaturenegativeconditionre-vealedamaineectofclassication,F(1,14)=12.45,p=.003,2p=.47,butnotlever,F(1,14)=11.00,p=.005,2p=.44,oraneectofblock,F(2.12,29.74)=1.40,p=.262,2p=.09.Therewerenointeractionsbetweenclassicationandblock,F(2.12,29.74)=1.89,p=.166,2p=.12,classicationandlever,F(1,14)=1.23,p=.286,2p=.08,critciallybetweenblockandlever,F(3.68,51.50)=0.30,p=.864,2p=.02,orathree-wayinteraction,F(3.68,51.50)=0.58,p=.663,2p=.04.TheseresultssuggeststhattheanimalsdonotlearntheFNdiscrimination.DiscussionTheseresultsshowsomeevidenceofFPdiscriminationlearning,mainlydrivenbytheGTsfood-wellentries,butagainnoevidenceoflearningtheFNdiscrimination.Experiment5showedthatthefeaturepositiveeectismainlyexplainedbytheanimals'respondingtotheuniquefeature,whichinthelightconditionisnotmeasured.4.2.4DiscussionExperiments4and5TheresultsfromExperiment5and6showoverallthattheanimalslearnafea-turepositivebetterthanafeaturenegativediscrimination,wheretheyshowedlittleevidenceoflearning.Therewasadierencebetweentheleverandlightconditions,astheanimalsshowedlearningthediscriminationbymainlyinteractingwithA(Figure21),andinthelightcondition(Experiment6)behaviourtowardsAwasnotmeasured.DirectingbehaviourtowardstheuniquefeatureisconsistentwiththeresultsfoundbyJenkinsandSainsbury(1970).Sign-trackersexpressedlearningmoreinleverpressesandgoal-trackersinfood-wellentries,whichwasmoreobviousintheExperiment5(Leverexperiment,(Figure23).InExperiment6,leverpressesshowedlittlelear

ningintheFPdesign,howeverthisisprobablyduetothefactthatbehaviourtowardsAwasnotmeasured(Figure23).Goal-trackingratsshowlearningofafeaturepositive137 discriminationinfood-wellentries.Thishighlightstheimportanceofhowlearningisassessed,asdierentmeasurementscouldleadtodierentconclusions.Forexample,inExperiment6(Lightcondition),onlymeasuringbehaviourtowardsBwouldleadustoconcludeeitherthatSTshavenotlearntafeaturepositivediscriminationorthatlearningismainlyshowedbyinteractingwiththefood-well.Experiment5(Levercondition)revealsthefactthatlearningisobservedinbothleverpressesandfood-wellentriesinbothSTsandGTs,howeveritismainlydrivenbytheuniquefeature.HeiDIpredictedafeaturepositiveeect,howeveritpredictedthatwhileaFNdis-criminationishardertolearnbutitstillshouldbelearnt.OthernovelpredictionsderivedfromHeiDIarediculttoassessastheyrelyontheanimalslearningafeaturenegativediscrimination,howeverthegeneralpredictionthatitshouldbemoreobviousinleverpressesthanfood-wellentrieswasnotpresentintheoverallresultsbecausesuchaneectwasapparentinExperiment5butnotExperiment6(albeit,sign-trackingororientationtothepositivefeaturelightwasnotassessed).Mostassociativelearningmodelspredictafasteracquisitionofafeaturepositivediscrimination,howevertheyalsofailtopredictnolearninginafeaturenegativediscrimination(e.g.,Mackintosh,1975;McLaren,Kaye,&Mackintosh,1989;Pearce&Hall,1980;Rescorla&Wagner,1972;Wagner,1981).TheconditionsunderwhichtheanimalsareabletolearnaFNdiscriminationarestilltobeelucidated.138 Chapter5GeneralDiscussionTheoriesofassociativelearningassumethatpairingsofaCSwithaUSresultintheformationofanassociationbetweenthece

ntralrepresentationsoftheevents(e.g.,Mackintosh,1975;Pearce&Hall,1980;Rescorla&Wagner,1972;Wagner,1981).ThisassociationallowstheCStoactivatetherepresentationorideaoftheUSandtherebybehaviour.Thesetheoriesassumeamonotonicrelationshipbetweenasingleconstructthatrepresentslearningandacquiredbehaviour.Thisviewischallengedbythefactthatasimpleauto-shapingprocedureproducesmarkedindividualdierencesinbehaviour:someratspredominantlyinteractwiththeCS(e.g.,lever),othersinvestigatethelocationwheretheUSisabouttobedelivered,andtheremaindershowpatternsofbehaviourinbetweenthesetwoextremes(e.g.,Patituccietal.,2016).Focusingononemeasure(e.g.,leverpresses)leadstotheconclusionthatonegroup(sign-trackers)betteracquiretheCS-USassociation,whilefocusingonanothermeasure(e.g.,food-wellentries)leadstotheoppositeconclusion.Thepurposeofthisthesiswastoincreaseourunderstandingofhowlearningistranslatedintoperformance,andtheoriginofindividualdierencesinhowlearningisexpressed.TheempiricalresultsfromChapter2contributedtothedevelopmentofanewmodelwhichintegrateslearningandperformance,HeiDI(Chapter3).Thismodelprovidesnewapproachtoknownphenomena,whichinthepasthaveposedachallengeformanylearningtheories,butitalsoprovidesnewtestablepredictionsabouthowlearningistranslatedinconditionedbehaviour.InChapter4Itestedsomeofthesenovelpredictions.Iwillrstsummarisethenovelempiricalndings,fromChapter2and4,beforemovingontodiscusstheimplicationsofthetheoreticaladvancesprovidedbyHeiDI.139 5.1EmpiricalndingsThefewattemptsatexplainingindividualdierencesinconditionedbehaviourhaveappealedtotheassumptionoftwolearningsystems(e.g.,S-S/S-R,model-based

/model-free),eachofthemgeneratingonetypeofbehaviour(e.g.,Lesaintetal.,2014).Thereareatleasttwoformsthatthisanalysiscouldtake.Onepossibilityisthatthetwolearningsystemsoperatedierentlyacrossrats,thebalancefavouringonesystemoranother.Ifasinglesystemgovernedallbehaviourinagivenratthenbothfood-wellandlever-pressresponsesshouldexhibitthecharacteristicpropertyofthatsystem:WhengovernedbyanS-Ssystem(ormodelbased),activitydirectedtowardboththeleverandthefood-wellwillchangerapidlyinthefaceofachangeincontingencies;whereaswhengovernedbyanS-R(model-free)systembothwillchangerelativelyslowly.Anotherpossibilityisthatbothsystemsoperateinparallelinagivenrat,howevereachsystemgeneratesonetypeofbehaviour.ThisanalysispredictsthatagivenformofresponsewillexhibitthesamecharacteristicsindependentlyofwhethertheanimalinwhichitisobservedisclassiedasaSToraGT;withfood-wellactivitybeingderivedfromtheoperationofaS-Ssystem(ormodelbased)andlever-orientedbehaviourbeingderivedfromaS-Rsystem(ormodel-free)thatoperatetodierentdegreesinallrodents.Thedominantresponsemightbetowardthefood-wellinonerodentandleverinanother,butinbothratsfood-wellactivityshouldmorerapidlytrackchangesinreinforcementcontingenciesthanshouldlever-orientedactivity.ThesepredictionswereassessedinExperiment1andExperiment2.InExperiment1ratsreceiveddiscriminationtrainingwherethepresentationofonelever(L1)waspairedwithfoodpelletsandanotherlever(L2)wasnon-reinforced.Whenratslearntthediscrimination,thecontingenciesontheleverswerereversedwithL1nownon-reinforcedandL2reinforced.Thisprocedureresultedinmarkedindividualdierencesinconditionedresponding;withsomeratsinteractingwit

hL1(butnotL2)andothersapproachingthesiteoffooddeliveryduringL1(butnotL2).RatswereclassiedasST140 andGTonthebasisoftheirrespondinginthelastblockoftraining.Resultsshowedthatwhenthecontingencieswerereversed,thedierentlevelsofleverpressingtoL1(andL2)inGroupsSTandGTremainedremarkablystableduringtherstblockofreversal.Incontrast,thelevelsoffood-wellentrieschangedmorerapidlyinbothGroupsSTandGT(Figure2).InExperiment2ratswerepresentedwithtwolevers(L1andL2),bothreinforcedwiththesametypeofreinforcer.AsnotedinChapter2,thereweretwogroupsofanimalswhichhadtwodierentreinforcers:onegrouphadpelletsasareinforcerandanothergrouphadsucrose.Previousworkinmylabsuggestedpelletssustainmorerespondingthansucrose.Afterinitialtraining,thecontingenciesontheleverswerethenswitched.Thegroupofanimalsthathadpelletsasareinforcerreceivedsucroseand,thegroupthathadsucrosereceivedpellets.Whenthecontingencieswereswitched,bothSTandGT,changedtheirrespondingforfood-wellentries,increasingtheirrespondingfromsucrosetopelletsanddecreasingtheirrespondingfrompelletstosucrose.Lever-pressrespondingremainedstablefollowingtheswitchincontingencies(Figure5).InbothExperiment1andExperiment2,lever-pressbehaviourwaslesssensitivetochangesinreinforcementcontingenciesthanwasfood-wellbehaviour.ThisdierenceinsensitivitywasequallyapparentinratsthatwereclassiedasSTsandGTs.Tosummarize,theseresultsindicatethatdierentialsensitivitytochangeisapropertyofthebehaviourthatisbeingmeasurednotanoverallpropertyoftheanimalbeinggovernedbyonesystemoranother.AsdiscussedinChapter3(section3.6.1.Excitatoryconditioningandextinction),HeiDIpredictsthatVCS�USandVCOM

Bdecline,butVUS�CSdoesnot,whichleadstodierentialsensitivitytoUSchanges:CS-orientatedbehaviourisextinguishedslowerthanUS-orientedbehaviour.TheresultsobtainedinExperiment1andExperiment2arepredictedbyHeiDI.Experiments3,4,5and6aimedtotestnewpredictionderivedfromHeiDI.HeiDIassumesthatCSandUSarexedforagivenCSandUSinagivenanimal,141 butproposethattheperceivedsalienceoftheCS(relatingtoCS)andUS(relatingtoUS),andhencec*CSandVCS�US(Equations4and5),canvarybetweenanimals.Experiment3examinedhowperformanceisaectedbyasimplemanipulation,thenumberoffoodpellets(oneortwo),whichconstitutedtheUSonagiventrial.HeiDIpredictshigherlevelsofUS-orientedbehaviour(orgoal-tracking)forUSsofhighervalue.However,theresultsfromthisexperiment,showedthataUSassumedtohaveahighervalue(2pellets)producedmoresign-trackingcomparedtoaUSwithalowervalue(1pellet),andnodierenceswereobservedingoal-tracking.ThesampleofratsinExperiment3wasrelativelylow,andExperiment4aimedtoreplicatethendings.However,thedistributionofbehaviourinthiscohort,withveryfewanimalsthatshowedanysign-trackingbehaviour,madetheinterpretationoftheresultsinExperiment4dicult.ThisexperimentcouldnotbeexpectedtoreplicatethendingsofExperiment3,astheanimalsbehavedatypically.Discriminationlearningparadigmsshowthatpigeonsacquiremorerapidlyafea-turepositivediscrimination(FP;e.g.,AB+/B-)thanafeaturenegativediscrimination(FN;e.g.,B+/AB-)(Jenkins&Sainsbury,1970).Thefeaturepositiveeectisawell-establishedcharacteristicofPavlovianphenomenagenerally,butrelativelylittleresearchhasinvestigatedhowittranslatesintoconditionedbehaviour.HeiDIpredicts

afeaturepositiveeect:learningshouldbebetterinanFPdesignthananFNdesign.WhenthebiasisarrangedtowardstheCS(whenCS�US,Figure17),thisismoreevidentinRCSthanRUS.WhenthebiasisarrangedtowardtheUS(whenCSUS,Figure18),foraFPdesignthisdiscriminationshouldbemoreevidentinRCSthanRUS,howeverforanFNdesignthediscriminationisequalorbetterforRUScomparedtoRCS(again,suggestingthefeaturepositiveeectshouldbemoreapparentinRCSthanRUS).SimulationspredicteddierentlevelsofobserveddierenceswhenthebiasisarrangedtowardstheCSorUSandforthisreasonforhalfoftheanimalsAisalever(Experiment5)andfortheotherhalfisalight(Experiment6),whichpotentiallyimplydierentlevelsof.142 TheresultsfromExperiment5andExperiment6,showoverallthattheanimalslearntthefeaturepositivediscrimination,howeverforthefeaturenegativediscriminationtheyshowedlittleevidenceoflearning.Interestingly,thefeaturepositiveeectwasmoreobviousinExperiment5,whererespondingtoA(theuniquefeature)wasmeasured.DirectingbehaviourtowardstheuniquefeatureisconsistentwiththeresultsfoundbyJenkinsandSainsbury(1970).Itisunclearwhytheanimalsdidnotlearnthefeaturenegativediscriminationbutitispossible,eventhoughunlikely,theamountoftrainingwasnotenough.AnotherpossibilityliesintheuseofsimultaneousrathersuccessivepresentationoftheABcompound.OthernovelpredictionsderivedfromHeiDI,suchasthefeaturepositiveeectbeingmoreevidentinsign-trackingthangoaltracking,arediculttoassess.Thesepredictionsrelyonthefactthatanimalsacquireafeaturenegativediscrimination.Theseexperimentshighlight,yetagain,theimportanceofhowlearningisassessed,asdierentmeasurementscouldl

eadtodierentconclusions.IftheconclusionweremadeonthebasisofExperiment6(Lightcondition),measuringbehaviourtowardsBwouldleadtotwowrongconclusions.First,IwouldbeinclinedtoconcludethatSTsdonotlearnafeaturepositivediscriminationorsecond,thatlearningismainlyexpressedbyinteractingwiththefood-well.Experiment5(Levercondition)revealsthefactthatlearningisobservedinbothleverpressesandfood-wellentriesinbothSTsandGTs,howeveritismainlydrivenbyrespondingtotheuniquefeature,whichinExperiment6isnotmeasured.Overalltheempiricalworkpresentedinthisthesiselucidatesoneimportantquestion:isittheanimalorthebehaviour?Someofthepreviousresultshavebeenattributedtotheanimals'\personality"(e.g.,Ahrensetal.,2016;Bissonetteetal.,2015;Flageletal.,2009;Lesaintetal.,2014;Lovicetal.,2011).However,theresultspresentedhereindicatethattheapparent\in exibility"or\resistancetochange"isapropertyofthebehaviourbeingmeasured.Anopenremainingquestionconcernswhetherauniedsinglemechanismtheory(e.g.,HeiDI)isabletoexplainhowlearningtranslatesinbehaviour,oradual-mechanismisnecessarytoexplainalltheobserveddierencesin143 conditionedbehaviour.Theempiricalworkpresentedherehasnotfullyaddressedthiscrucialquestion.TheresultsobtainedinExperiment1andExperiment2areconsistentwithbothoftheseaccounts.Experiment3andExperiment4couldhaveprovidedevidencefortheoriginofin-dividualdierencesandiftheyarerelatedtotheUSvalue,howevertheresultswereinconclusive.OthernovelpredictionsaboutthedistributionofconditionedbehaviourbetweentheCSandtheUS,wereaddressedinExperiment5andExperiment6,howevertheyreliedontheanimalslearningafeaturenegativediscrimination.InthefollowingsectionIwil

ldiscussbothhowHeiDIisabletoaccommodateknownphenomena,whichhaveposedachallengeinthepasttootherlearningtheories(in-cludingthosethatrelyondual-processmechanismstoexplainthepattersofsign-andgoal-trackingbehaviour),anddescribeothernovelpredictionsofhowlearningistrans-latedintoperformance.5.2Theoreticalimplications:HeiDIIndispellingout-dated(academictextbook)descriptionsofPavlovianconditioning,Rescorla(1988,p.151)referredtothreeprimaryissuestobeaddressedinthestudyofanylearningprocess:\Whatarethecircumstancesthatproducelearning?Whatisthecontentofthelearning?Howdoesthatlearningaecttheorganism'sbehaviour?".ItisperhapsespeciallysurprisingthatinthecontextofPavlovianlearningthenalissue{concerningconditionedbehaviouritself-hasbecomesecondarytotheorizingdirectedtowardaddressingthersttwoquestions.IndeedformaltheoriesofPavlovianlearninghaveoftenfollowedthesimplifyingstanceexpressedbyRescorlaandWagner(1972)thatitis\sucientsimplytoassumethatthemappingofVsintomagnitudeorprobabilityofconditionedrespondingpreservestheirordering.".ThefactthattheformofconditionedbehaviourdependsonthenatureofboththeCSandUS(e.g.,Holland,1977,1984)andthattherearemarkedindividualdierencesinhowlearningisexhibited144 (e.g.,Patituccietal.,2016)representasignicantimpetusfordevelopingtheoriesthatrecognizethisvariety.HeiDIdoesthis.5.2.1Conditions,contentandperformanceIstartedbysimplifyingtheRescorla-Wagnerlearningruleforforward,CS-USas-sociations,andsupplementingitwithaformallyequivalentruleforreciprocal,US-CSassociations(seeEquation1and2).Thevaluesreturnedbytheseequationswerethencombined(toformVCOMB)usingarulethatweightstheassocia

tivevalueofthestim-ulusthatispresent(e.g.,VCS�US)morethananassociationinvolvingassociativelyactivatednodes(e.g.,VUS�CS;seeEquation3).Finally,whentheCSispresented,VCOMBisdistributedintwoformsofbehaviour(CS-oriented,RCS,andUS-oriented,RUS)accordingtotheratioofCSandVCS�US(seeEquation4and6).Theresultingmodel,HeiDI,providesthefollowinganswerstothethreequestionsposedbyRescorla(1984):(1)Onagiventrial,learningoccurstotheextentthatthereisadierencebe-tweentheperceivedsalienceofanevent(re ectedinUS)andtheperceivedsalienceoftheretrievedrepresentationofthateventbasedonthecombinedassociativestrengthsofthestimulipresentedonthattrial(PVTOTAL�US;oradierencebetweenc*CSandPVTOTAL�CS).(2)Learningisrepresentedinthereciprocalassociationsbetweenthenodesactivatedbydierentstimuli(e.g.,CSandUS).(3)Performancere ectstherelativeintensityoftheCS(asre ectedinCS)andtheassociativestrengthoftheCS(VCS�US;whichre ectsUSthroughc*US)multipliedbythecombinedassocia-tivestrengthsinvolvingtheCSandUS(i.e.,VCOMB).Inthisway,HeiDIprovidesawaytocapturetwoclassesofconditionedbehaviour,andindividualdierencestherein,togetherwiththeeectofgroup-levelmanipulations.IhavehighlightedtheapplicationofHeiDItosign-trackingandgoal-tracking,whichareexamplesofthegeneraldistinctionbetweenCS-orientedandUS-orientedbehaviours.Thespatialseparationofthetwoclassesofresponseandtheeasewithwhichtheyareautomaticallyrecordedcertainlymeansthattheyhavesomemethodologicaladvan-145 tagesoverotherresponses(e.g.,thoseelicitedbyaversiveUSs).Nevertheless,IassumethatmanyPavlovianconditioningproceduresresultingreatervarietyi

nconditionedresponsesthanisroutinelymeasuredandusedtoguidetheorizing.Ihavealreadyil-lustratedhowthispracticemightcomplicateinterpretationofpatternsofresultsinthecaseofblocking.However,thetwoclassesofresponsesthatIhaveconsideredmightthemselvesbefurtherdivided,withtheindividualelementsoftheCSandUSgivingrisetothedierentresponsesdened(r1-6;seeJenkins&Moore,1973).ExpandingHeiDItoaccommodatethiscomplexitywouldnotpresentspecictheoreticalchallenge:witheachindividualelementhavingitsownorvaluesandaliated(unconditioned)re-sponses.However,thereareissuesthatdorequirefurtherdiscussion.Theseinvolvehowassociationsbetweenthecomponentsofacompoundstimulusmightaectperformance,andthenatureoftherepresentationsoftheCSandUS.5.2.2AssociationsbetweenthecomponentsofacompoundConditionedrespondingtoaCSisnotonlydeterminedbywhetherithasadirectassociationwithaUS.Forexample,afterexposuretoastimuluscompound(AB),condi-tionedrespondingthatisestablishedtoBwillalsobeevidentwhenAispresented(e.g.,Rescorla&Cunningham,1978).ThiseectisknownassensorypreconditioninganditisoftenattributedtotheformationofanassociativechainthatallowsAtoactivatetheUSthroughA-BandB-USassociations(butsee,Lin&Honey,2016).IhavealreadyprovidedananalysisofhowA-Blinksmightform(seeEquation7and8),andhaveappealedtosuchlinksinprovidingananalysisofdownshiftunblocking(cf.Rescorla&Colwill,1983).ThewayinwhichthelinksinthechaincanbecombinedtodeterminethelevelofperformancegeneratedbyAcanbederivedfromanextensionofEquation3:VCHAIN=1/c*VA�BVCOMB�B,whereVCOMB�B=VB�US+(1/c*VB�US*VUS�B).ThisformulationmeansthatVCHAINVCOMB�BifVA�B1.Theway

inwhichVCHAINisdistributedintoRCSandRUScanbedeterminedusingEquations4and5:c*Aissubstitutedforc*CS,j1/c*VA�B*VB�USjissubstitutedforj1/c*VCS�USj,146 andVCHAINreplacesVCOMB.IntermsofthenatureofthebehaviourelicitedbyA,themostobviouspredictionisthatitwillmirrorthatevokedbyBthroughdirectcondition-ing(Holland,1984).However,accordingtoHeiDIthedistributionofCS-orientedandUS-orientedbehaviourwilldierbetweenAandB:withCS-orientedrespondingbeingmoreevident(andUS-orientedbehaviourlessevident)duringAthanduringB:Totheextentthatwhilec*Aandc*Bwillbethesame,j1/c*VA�B*VB�USjjVB�USj(seeDwyer,2012).Thisanalysisofsensorypreconditioning,andofthepotentialimpactofwithin-compoundassociationsinconditioningproceduresmorebroadly,isrelativelystraightforward.However,thereisanotherapproachtoconditionedperformancethathasalsobeenappliedtosensorypreconditioningandcuecompetitioneects(e.g.,over-shadowingandblocking).ItdeservesconsiderationbecauseitaddressessomeofthesameissuesandphenomenaasHeiDI.ThecomparatormodelproposedbyStoutandMiller(2007)focusesonhowperfor-mancetoateststimulus,A,isaectedbythestimuliwithwhichitwastrained(e.g.,BafterconditioningwithanABcompound).Thismodelbuildsontheideathatper-formancetoAattestisdeterminedbyacomparisonbetween(i)therepresentationoftheUSdirectlyretrievedbyA,and(ii)therepresentationofthesameUSindirectlyretrievedbytheassociativechain:A-BandB-US(seeMiller&Matute,1998).Inthiscase,BiscalledthecomparatorstimulusforA,andfollowingpairingsofABwithaUS,thetendencyforAtogenerateperformanceattestisheldtoberestrictedbythefactthatitscomparator,B,hasretrievedamemoryoftheUS.Theanal

ysistherebyexplainsovershadowingandblocking,butalsootherndingsthatareproblematicforanunreconstructedRescorla-Wagnermodel.However,inthecaseofsensoryprecondi-tioning,whereABisrstnon-reinforced,themodelisforcedtoassumethatthefactthatBhasacquiredexcitatoryassociativepropertiesduringasecondstageincreasesthepotentialforAtogenerateperformance.Thesedieringeectsofthecomparatorterm(B;termedsubtractiveandadditive)areheldtobedeterminedbyexperiencewithcomparingtheUSrepresentationretrievedbyAwiththeUSrepresentationindirectly147 retrievedbyB.Theadditiveeectoccurswhentherehasbeenlittleornoopportunitytoexperiencethetwotypesofretrievedrepresentations(e.g.,duringsimpleexposuretoABinsensorypreconditioning),andthesubtractiveeectincreaseswithexperiencethataordssuchacomparison(e.g.,duringmulti-trialcompoundconditioning;Stout&Miller,2007,p.765).Inanycase,liketheRescorla-Wagnermodel,themoreso-phisticatedanalysisofperformancedevelopedbyStoutandMiller(2007)providesnoreadyexplanationforthefactthatdierentbehaviouralmeasurescanprovidesupportforopposingconclusionsabouthowassociativestrengthistranslatedintoperformance,whichisthefocusofinteresthere.Thatbeingsaid,thefactthatwithinHeiDIthedistri-butionofCS-orientedandUS-orientedcomponentsofperformancere ectstherelativevaluesofc*andVCS�USinvolvesacomparisonprocessofsorts.Certainly,changingtheassociativestrengthofstimulibeforetestingwillnotonlyaectVCOMB,butwillalsoaectRCSandRUSthroughchangingVCS�US.AsIhavealreadynoted,inthecontextofmypreviousdiscussionofblocking,asecureinterpretationoftheimpactofsuchchangesonperformancerequiresbehaviourala

ssaysthataresensitivetobothRCSandRUS.5.2.3ElementalandconguralprocessesAnalissue,whichImentionedinthesectiononsummation(section3.6.4),con-cernshowmodelsthatdonothaveconguralprocessesaddressthefactthatanimalscanlearndiscriminationsthatarenotlinearlyseparable.Forexample,animalscanlearnthatatonesignalsfoodandaclickersignalsnofoodinoneexperimentalcontextandthetonesignalsnofoodandaclickersignalsfoodinasecondcontext(seeAllmanetal.,2004).Thistypeofdiscriminationisinterestingbecausean`elemental'animal{oneonlycapableofrepresentingindividualevents{shouldbeincapableoflearningthem:Thetoneandclickerhavethesamereinforcementhistory,asdothespottedandcheckedchambers,andthereforeeachofthefourcombinationsorcompoundsshouldbeequallycapableofgeneratingperformance.Thereisanongoingdebateabouthow148 dierentcombinationsofthesamestimulimightberepresentedinwaysthatwouldpermitthesediscriminationstobeacquired(e.g.,Brandonetal.,2000;Honeyetal.,2010;Pearce,1994).Forexample,dierentstimuluselementsofagivenauditorystim-ulusmightbecomeactivedependingonthecontextinwhichtheyareencountered(e.g.,Brandonetal.,2000),ortheelementsactivatedbyagivenpatternofstimulationmightcometoactivateasharedconguralrepresentation(e.g.,Honeyetal.,2010;Pearce,1994).Ineithercase,theelementsorcongurationsthereof(orboth;seeHoney,Ior-danova,&Good,2014)couldbesubjecttothesamelearningandperformancerulesdescribedinEquations1to6(seealsoDelamater,2012).However,Ishouldalsonotethattheresponseunits(r1-r6)withintheproposedassociativearchitectureforHeiDI(seeFigure7provideanotherlocusinwhichcombinationsofCSsandindeedUSsmightberepresented:Thestrengthofth

econnectionsfromcombinationsofCSsandUSstotheseresponseunitscouldbemodiedduringconditioning(forarelateddiscussion,seeHoneyetal.,2010).5.2.4LimitationsandfurtherdevelopmentIhavealreadynotedthatEquation6providesasimplisticanalysisofhowchangesinRCSandRUSmightaectactivityinasetofresponse-generatingunits(r1-r6).However,takingastepback,whatisneededinordertoprovideadetailedassessmentoftheaccuracyofthepredictionsthatIhavederivedfromHeiDI,isestimatesoftheperceivedsalienceofboththeCSandUSonanindividual-by-individualbasis.Armedwiththeseestimates,Icouldthenprovideaquantitativeanalysisofthetbetweenpredictionsofthemodelandthebehaviourofanimalsonanindividualbasis.IthasbeenarguedthatpalatabilitymightprovideanestimateofperceivedUSsalience(cf.Patituccietal.,2016),andonepotentialestimateoftheperceivedsalienceofaCSistheunconditionedorientingbehaviourthatitspresentationprovokesbeforeconditioninghastakenplace(cf.Kaye&Pearce,1984).149 5.2.5ConcludingcommentsPavlovianconditioninghasprovidedafertiletestbedinwhichtoinvestigateissuesconcerningwhenassociativelearningoccurs,itscontent,andhowitistranslatedintoperformance.Ofthesethreeissues,formalmodelshavepaidleastattentiontohowlearn-ingistranslatedintoperformance:considerationofperformancehasbeensecondarytoanalysesoftheconditionsandcontentoflearning.HeiDIbeginstoredressthisim-balancebyprovidinganintegratedanalysisofallthreeissues.Thisanalysiscouldbedevelopedinordertoprovideamorequantitativeanalysis,modellingperformanceatanindividual-by-individuallevel,withthecharacteristicsoftheschematicnetworkfullyspecied.Asalreadynoted,itcouldalsobeextendedtoexplicitlydistinguishbetweend

ierentfeaturesofboththeCSandtheUS,whichcouldbetiedtodierenttypesofresponse(seealsoDelamater,2012).Intheprocessofdevelopingthisrelativelysimplemodel,ithasbecomeclearthatitisdiculttoaddressoneofRescorla'sthreeissueswithoutadetailedconsiderationoftheothers:developingamorecompleteunderstand-ingofassociativelearningthroughthestudyofPavlovianconditioninginvolvesmultipleconstraintsatisfaction(Marr,1982).HeiDIprovidesgeneralinsightsintolearning,itscontentandperformancethatare{atleastinpart{bornoutofamoredetailedanalysisofthevarietyandindividualdierencesinconditionedbehaviour.Thisevidencehasbeentoooftenneglected,givenitstheoreticalimportanceandpotentialtranslationalsignicance.ItistruethattheempiricalresultspresentedherearenotfullyconsistentwiththenovelpredictionsofHeiDI:increasingthenumberofreinforcersdidnotselectivelyin u-encegoal-trackingoversigntrackinginExperiment3andExperiment4;andthefeaturepositiveeectwasnotgenerallylargerforsign-thangoal-trackingresponses(compareExperiment5withExperiment6).However,therearereasonstothinkthatthesemayhavebeenduetoissuesotherratherthan awedpredictions:changingreinforcernumbermaynotbeadirectin uenceonUS;andtheabsenceoffeaturenegativediscriminationmakesassessingthefeaturepositiveeectambiguous.So,bothdualprocess(e.g.,S-Svs150 S-R;ormodelfreevsmodelbased)accountsofthedierencesinsign-andgoal-trackingremainasviableasthesingle-processaccountoeredbyHeiDI.Thatsaid,thesedualprocessaccountsdonotoerthesamerangeofnovelexplanationsforpriorresultsthathaveprovedproblematicforexistingassociativetheory.Thus,thereisaclearargumenttobemadeforthesuperio

rityofthetypeofsingle-processaccountoeredbyHeiDI.151 Chapter6Appendix6.1ReproducibilityandOpenScienceFramework(OSF)ThisdocumentisSweaveRproject.ItcombinesSweaveles( .Rnw extension)whenRcodeneedstobeintegratedwithinthetextorsimpleLATEXles( .tex extension)whenonlytextisneeded.ThedataandthecodearefreelyavailableonOSF(https://osf.io/h4fyg)website.TheREADMEle(OSF/Thesis/README/README.pdf)detailsthestructureoftheSweaveRproject.ForreproducibilitypurposesthecodefordatacleaningandmanipulationofthedatafromalltheexperimentsreportedherehavebeenuploadedontheOpenScienceFramework/Experiments(osf.io/h4fyg)website.Thecodesfortheexperimentsarethefollowing:ˆExperiment1:DDR066(Experiments/2017)ˆExperiment2:RH001(Experiments/2017)ˆExperiment3:DDR074(Experiments/2017)ˆExperiment4:DDR084(Experiments/2018)ˆExperiment5and6:DDR087(Experiments/2018)6.2HeiDIappForconductingthesimulations,IbuiltaShinydashboardapp(HeiDI)usingRprogramminglanguage.ThecodeforthesimulationstogetherwithaREADMEle152 whichdetailsthenecessarystepstoreproducethesimulationsisavailableontheOSFwebsite(https://osf.io/h4fyg/).AppstructureandlesThemainleis`app.R'.Thisleisresponsibleforthemainbodyoftheappandsourcesthelesneededtoconductthesimulations:`formulas.html',`load.packages',`model.R',`my theme.R',`style.css'aswellasallthelesinthe`phenomenon'folder.app.R:Wheneverpossible,tocreateacleanerappearancetheappsourcesotherles.The ui functionbuildsauserinterfaceandthe server denestheserverlogic.Morein-formationaboutshinyappscanbefoundathttps://shiny.rstudio.com/tutorial/written-tutorial/lesson1

/.formulas.html:writestheformulasdisplayedinthe'Formulas'tabinhtmllanguage.install packages.R:thisleinstallsallthepackagesusedforthisapp.load packages.R:sourcesfromtheapptoloadallthepackages.model.R:containsfunctionsformodelling,convertingfromwidetolongformatandlabellingdataframes.Each'model'functionhasacomplementary`to.long'(transformsgenerateddatatolongformat)and`label'(re-labelsvariablestouseforplots)function.Every`model'functionhascommentedthephenomenonforwhichitisbeingused.Therearethreefunctionsandeachofthemisusedforasetnumberofvariables(e.g.twovaluesandonevalue).my theme.R:containsfunctionsforcontrollingplotformatting(e.g.textsize,spac-ing,axisticks)in`ggplot'plots.phenomenonfoldercontainslessourcedfromtheapptocreatesimulationsforeachofthephenomenon.READMEfolder:containsthe/README.texlethatwasusedtogeneratethisdocument.rsconnectfolder:generatedbytheapp.wwwfolder:/style.css:controlsappearance(e.g.textformat,boxsizes,colour)oftheapp.153 logfolder:containsloglestotrackchangesintheapp.Eachnumbered log letrackschangesusingatimestamp.Theappisnotnalandwillincludesimulationsofseveralmorephenomena.6.3AdditionalnotesForreferencegenerationthe\natbib"package(Daly&Ogawa,2013)hasbeenused.AspecialthankyoufortheStackover owcommunityforallthesupportinallR,SweaveandLATEXrelatedquestions.Chapter4,Experiment5andExperiment6:Intheappconditionedinhibitionsimu-latesonlyFNdesignAB-/B+(FPnotsimulatedyetintheapp).Thishasnoimplicationfortheconclusionsdrawn,howeveritisworthmentioningforreproducibilitypurposes.154 ReferencesAbramson,C.I.,Cakmak,I.,Duell,M.E.,Bates-Albers,L.M.,Zuniga,E.M.,Pende-gra

ft,L.,...Wells,H.(2013).Feature-positiveandfeature-negativelearninginhoneybees.TheJournalofexperimentalbiology,216(Pt2),224{9.Ahrens,A.M.,Singer,B.F.,Fitzpatrick,C.J.,Morrow,J.D.,&Robinson,T.E.(2016).Ratsthatsign-trackareresistanttoPavlovianbutnotinstrumentalextinction.Behaviouralbrainresearch,296,418{430.Aitken,M.R.F.,Bennett,C.H.,McLaren,I.P.L.,&Mackintosh,N.J.(1996).Perceptualdierentiationduringcategorizationlearningbypigeons.JournalofExperimentalPsychology:AnimalBehaviorProcesses,22(1),43{50.Allman,M.J.,&Honey,R.C.(2005).Associativechangeinconnectionistnetworks:anaddendum.JournalofExperimentalPsychology:AnimalBehaviorProcesses,31(3),363{7.Allman,M.J.,Ward-Robinson,J.,&Honey,R.C.(2004).Associativechangeintherepresentationsacquiredduringconditionaldiscriminations:furtheranalysisofthenatureofconditionallearning.JournalofExperimentalPsychology:AnimalBehaviorProcesses,30(2),118{28.Anselme,P.,Robinson,M.J.F.,&Berridge,K.C.(2013).Rewarduncertaintyenhancesincentivesalienceattributionassign-tracking.BehaviouralBrainResearch,238,53{61.Arcediano,F.,Escobar,M.,&Miller,R.R.(2005).BidirectionalAssociationsinHu-mansandRats.JournalofExperimentalPsychology:AnimalBehaviorProcesses,31(3),301{318.Asch,S.E.,&Ebenholtz,S.M.(1962).Theprincipleofassociativesymmetry.Pro-ceedingsoftheAmericanPhilosophicalSociety,106,135{163.Atkinson,R.C.,&Estes,W.K.(1963).Stimulussamplingtheory.InR.D.Luce,R.R.Bush,&E.Galanter(Eds.),Handbookofmathematicalpsychology(Vol2ed.).NewYork:Wiley.155 Auguie,B.,&Anotonov,A.(2017).package`gridExtra':MiscellaneousFunctionsfor"Grid"Graphics.Retrievedfromhttps://cran.r-project.org/web/packages/gridExtra/index.h

tmlBaker,A.G.,&Mercier,P.(1982).Priorexperiencewiththeconditioningevents:Evidenceforarichcognitiverepresentation.InM.L.Commons,R.J.Hernstein,&A.R.Wagner(Eds.),Quantitativeanalysesofbehavior(pp.117{143).Cambridge,MA:Ballinger.Barnet,R.C.,&Miller,R.R.(1996).Second-orderexcitationmediatedbyaback-wardconditionedinhibitor.Journalofexperimentalpsychology.Animalbehaviorprocesses,22(3),279{96.Beierholm,U.R.,Anen,C.,Quartz,S.,&Bossaerts,P.(2011).Separateencodingofmodel-basedandmodel-freevaluationsinthehumanbrain.NeuroImage,58(3),955{962.Bissonette,G.B.,Orsini,C.A.,Simon,N.W.,Calu,D.J.,Nasser,H.M.,Chen,Y.,&Fiscella,K.(2015).Individualvariabilityinbehavioral exibilitypredictssign-trackingtendency.FrontBehavNeurosci,9,Article289.Blanchard,R.J.,Fukunaga,K.K.,&Blanchard,C.(1976).Environmentalcontrolofdefensivereactionstofootshock(Vol.8;Tech.Rep.No.2).Boughner,R.L.,&Papini,M.R.(2003).Appetitivelatentinhibitioninrats:Nowyouseeit(signtracking),nowyoudon't(goaltracking).AnimalLearning&Behavior,31(4),387{392.Bouton,M.E.(1984).Dierentialcontrolbycontextinthein ationandreinstate-mentparadigms.JournalofExperimentalPsychology:AnimalBehaviorProcesses,10(1),56{74.Bouton,M.E.(2004).ContextandBehavioralProcessesinExtinction.Learning&Memory,11(5),485{494.Brandon,S.E.,Vogel,E.H.,&Wagner,A.R.(2000).AcomponentialviewofconguralcuesingeneralizationanddiscriminationinPavlovianconditioning.Behavioural156 BrainResearch,110(1-2),67{72.Brimer,C.J.(1970).Inhibitionanddisinhibitionofanoperantresponseasafunctionoftheamountandtypeofpriortraining.PsychonomicScience,21(3),191{192.Brimer,C.J.,&Kamin,L.J.(1963).Disinhibiton,habituation,sensitizatio

n,andtheconditionedemotionalresponse.JournalofComparativeandPhysiologicalPsychology,56(3),508{516.Brown,P.L.,&Jenkins,H.M.(1968).Auto-shapingofthepigeon'skey-peck.JournaloftheExperimentalAnalysisofBehavior,11(1),1{8.Browne,M.P.(1976).Theroleofprimaryreinforcementandovertmovementsinautoshapinginthepigeon(Vol.4;Tech.Rep.No.3).Bush,R.R.,&Mosteller,F.(1951).Amodelforstimulusgeneralizationanddiscrimi-nation.PsychologicalReview,58(6),413{423.Cinotti,F.,Marchand,A.R.,Roesch,M.R.,Girard,B.,&Khamassi,M.(2019).Impactsofinter-trialintervaldurationonacomputationalmodelofsign-trackingvs.goal-trackingbehaviour.Psychopharmacology,2373{2388.Cohen-Hatton,S.R.,Haddon,J.E.,George,D.N.,&Honey,R.C.(2013).Pavlovian-to-instrumentaltransfer:ParadoxicaleectsofthePavlovianrelationshipexplained.JournalofExperimentalPsychology:AnimalBehaviorProcesses,39(1),14{23.Cole,R.P.,&Miller,R.R.(1999).ConditionedExcitationandConditionedInhibitionAcquiredthroughBackwardConditioning.LearningandMotivation,30(2),129{156.Costa,D.S.,&Boakes,R.A.(2009).Contextblockinginratautoshaping:Sign-trackingversusgoal-tracking.LearningandMotivation,40(2),178{185.Cotton,M.M.,Goodall,G.,&Mackintosh,N.J.(1982).Inhibitoryconditioningresultingfromareductioninthemagnitudeofreinforcement.TheQuarterlyjournalofexperimentalpsychology.B,Comparativeandphysiologicalpsychology,34(Pt3),163{80.Crowell,C.R.,&Bernhardt,T.P.(1979).Thefeature-positiveeectandsign-tracking157 behaviorduringdiscriminationlearningintherat.AnimalLearning&Behavior,7(3),313{317.Daly,P.W.,&Ogawa,A.(2013).package`natbib':Flexiblebibliographysupport.Retrievedfromhttps://ctan.org/pkg/natbib?lang=enDavey,G.

C.L.,&Cleland,G.G.(1982).Theeectofpartialreinforcementontheacquisitionandextinctionofsign-trackingandgoal-trackingintherat(Vol.19;Tech.Rep.No.2).Daw,N.D.,Gershman,S.J.,Seymour,B.,Dayan,P.,&Dolan,R.J.(2011).Model-basedin uencesonhumans'choicesandstriatalpredictionerrors.Neuron,69(6),1204{15.Dayan,P.,&Berridge,K.C.(2014).Model-basedandmodel-freePavlovianrewardlearning:revaluation,revision,andrevelation.Cognitive,aective&behavioralneuroscience,14(2),473{92.Delamater,A.R.(2012).OnthenatureofCSandUSrepresentationsinPavlovianlearning.Learning&Behavior,40(1),1{23.Derman,R.C.,Schneider,K.,Juarez,S.,&Delamater,A.R.(2018).Sign-trackingisanexpectancy-mediatedbehaviorthatreliesonpredictionerrormechanisms.Learning&Memory,25(10),550{563.Dickinson,A.,&Balleine,B.W.(2002a).Theroleoflearninginmotivation.InC.Gallistel(Ed.),Stevens'handbookofexperimentalpsychology.vol.3.(pp.497{533).Wiley.Dickinson,A.,&Balleine,B.W.(2002b).TheRoleofLearningintheOperationofMotivationalSystems.InStevens'handbookofexperimentalpsychology.Hoboken,NJ,USA:JohnWiley&Sons,Inc.Dickinson,A.,Hall,G.,&Mackintosh,N.J.(1976).Surpriseandtheattenuationofblocking.JournalofExperimentalPsychology:AnimalBehaviorProcesses,2(4),313{322.Duvarci,S.,Nader,K.,&LeDoux,J.E.(2008).DenovomRNAsynthesisisrequiredfor158 bothconsolidationandreconsolidationoffearmemoriesintheamygdala.Learning&Memory,15(10),747{755.Dwyer,D.M.(2012).LickingandLiking:TheAssessmentofHedonicResponsesinRodents.QuarterlyJournalofExperimentalPsychology,65(3),371{394.Dwyer,D.M.,Burgess,K.V.,&Honey,R.C.(2012).Avoidancebutnotaversionfol-lowingsensorypreconditioningwith avors:Achallengetostimulussubsti

tution.JournalofExperimentalPsychology:AnimalBehaviorProcesses,38(4),359{368.Dwyer,D.M.,Figueroa,J.,Gasalla,P.,&Lopez,M.(2018).RewardAdaptationandtheMechanismsofLearning:ContrastChangesRewardValueinRatsandDrivesLearning.PsychologicalScience,29(2),219{227.Dwyer,D.M.,&Mackintosh,N.J.(2002).Alternatingexposuretotwocompound a-vorscreatesinhibitoryassociationsbetweentheiruniquefeatures.Animallearning&behavior,30(3),201{7.Engberg,L.A.,Hansen,G.,Welker,R.L.,&Thomas,D.R.(1972).AcquisitionofKey-PeckingviaAutoshapingasaFunctionofPriorExperience:"LearnedLaziness"?Science,178(4064),1002{1004.Fitzpatrick,C.J.,Gopalakrishnan,S.,Cogan,E.S.,Yager,L.M.,Meyer,P.J.,Lovic,V.,...Morrow,J.D.(2013).VariationintheFormofPavlovianConditionedAp-proachBehavioramongOutbredMaleSprague-DawleyRatsfromDierentVen-dorsandColonies:Sign-Trackingvs.Goal-Tracking.PLoSONE,8(10),e75042.Flagel,S.B.,Akil,H.,&Robinson,T.E.(2009).Individualdierencesintheat-tributionofincentivesaliencetoreward-relatedcues:Implicationsforaddiction.Neuropharmacology,56(SUPPL.1),139{148.Flagel,S.B.,Clark,J.J.,Robinson,T.E.,Mayo,L.,Czuj,A.,Willuhn,I.,...Akil,H.(2011).Aselectiverolefordopamineinstimulus{rewardlearning.Nature,469(7328),53{57.Flanagan,B.,&Webb,W.B.(1964).Disinhibitionandexternalinhibitioninxedintervaloperantconditioning.PsychonomicScience,1(1-12),123{124.159 Gallistel,C.R.(1990).Theorganizationoflearning.Cambridge,MA:MITPress.Gallistel,C.R.,&Gibbon,J.(2000).Time,RateandConditioning.PsychologicalReview,107(2),289{344.Gluck,M.A.,&Bower,G.H.(1988).Fromconditioningtocategorylearning:Anadaptivenetworkmodel.JournalofExperimentalPsychology:General,117(

3),227{247.Gokey,D.S.,&Collins,R.L.(1980).Conditionedinhibitioninfeaturenegativediscriminationlearningwithpigeons.AnimalLearning&Behavior,8(2),231{236.Gottlieb,D.A.(2004).Acquisitionwithpartialandcontinuousreinforcementinpigeonautoshaping.Learning&behavior,32(3),321{34.Gottlieb,D.A.(2006).Eectsofpartialreinforcementandtimebetweenreinforcedtrialsonterminalresponserateinpigeonautoshaping.BehaviouralProcesses,72(1),6{13.Gromer,D.(2019).package`apa':FormatOutputsofStatisticalTestsAccordingtoAPAGuidelines.Retrievedfromhttps://cran.r-project.org/web/packages/apa/index.htmlGrossberg,S.(1980).Howdoesabrainbuildacognitivecode?PsychologicalReview,87,1{51.Hall,G.(1991).PerceptualandAssociativeLearning.Oxford,UK:OxfordUniversityPress.Hall,G.,&Honey,R.C.(1989).Contextualeectsinconditioning,latentinhibition,andhabituation:Associativeandretrievalfunctionsofcontextualcues.JournalofExperimentalPsychology:AnimalBehaviorProcesses,15(3),232{241.Hall,G.,&Minor,H.(1984).ASearchforContext-StimulusAssociationsinLatentInhibition.TheQuarterlyJournalofExperimentalPsychologySectionB,36(2b),145{169.Hearst,E.,&Franklin,S.R.(1977).Positiveandnegativerelationsbetweenasignal160 andfood:Approach-withdrawalbehaviortothesignal.JournalofExperimentalPsychology:AnimalBehaviorProcesses,3(1),37{52.Hearst,E.,&Jenkins,H.M.(1974).Signtracking:Thestimulus-reinforcerrelationanddirectedaction.Austin:TX:PsychonomicSociety.Hinrichs,J.V.(1968).Disinhibitionofdelayinxed-intervalinstrumentalconditioning.PsychonomicScience,12(7),313{314.Holland,P.C.(1977).ConditionedstimulusasadeterminantoftheformofthePavlovianconditionedresponse.JournalofExperimenta

lPsychology:AnimalBehaviorProcesses,3(1),77{104.Holland,P.C.(1984).OriginsofBehaviorinPavlovianConditioning.PsychologyofLearningandMotivation,18,129{174.Holland,P.C.(1988).Excitationandinhibitioninunblocking.Journalofexperimentalpsychology.Animalbehaviorprocesses,14(3),261{79.Holmes,N.M.,Chan,Y.Y.,&Westbrook,R.F.(2019).Acombinationofcommonandindividualerrortermsisnotneededtoexplainassociativechangeswhencueswithdierenttraininghistoriesareconditionedincompound:AreviewofRescorla'scompoundtestprocedure.JournalofExperimentalPsychology:AnimalLearningandCognition,45(2),242{256.Honey,R.C.,&Bolhuis,J.J.(1997).Imprinting,Conditioning,andWithin-eventLearning:.Quantitativeanalysesofbehavior,50(2),97{110.Honey,R.C.,&Good,M.(1993).Selectivehippocampallesionsabolishthecontextualspecicityoflatentinhibitionandconditioning.Behavioralneuroscience,107(1),23{33.Honey,R.C.,Iordanova,M.D.,&Good,M.(2010).Latentinhibitionandhabituation:evaluationofanassociativeanalysis.InR.E.Lubow&I.Weiner(Eds.),Latentinhibition(pp.163{182).Cambridge:CambridgeUniversityPress.Honey,R.C.,Iordanova,M.D.,&Good,M.(2014).Associativestructuresinan-imallearning:Dissociatingelementalandconguralprocesses.Neurobiologyof161 LearningandMemory,108,96{103.Honey,R.C.,&Ward-Robinson,J.(2002).Acquiredequivalenceanddistinctivenesofcues:I.Exploringaneuralnetworkapproach.JournalofExperimentalPsychology:AnimalBehaviorProcesses,28(4),378{387.Hull,C.L.(1943).Principlesofbehavior.NewYork:Appleton-Century-Crofts.Hull,C.L.(1949).Stimulusintensitydynamism(V)andstimulusgeneralization.PsychologicalReview,56(2),67{76.JASPTeam.(2018).JASP[Computersoftware].Jenkins,H.M.,&

Moore,B.R.(1973).Theformoftheauto-shapedresponsewithfoodorwaterreinforcers.Journaloftheexperimentalanalysisofbehavior,20(2),163{81.Jenkins,H.M.,&Sainsbury,R.S.(1970).Discriminationlearningwiththedistinctivefeatureonpositiveandnegativetrials.InD.Mostofsky(Ed.),Attention:Contem-porarytheoryandanalysis(pp.239{273).NewYork:Appleton-Century-Crofts.Kamin,L.J.(1969).Selectiveassociationandconditioning.InN.J.Mackintosh&W.K.Honig(Eds.),Fundamentalissuesinassociativelearning(pp.42{89).Halifax:DalhousieUniversityPress.Kaye,H.,&Pearce,J.M.(1984).ThestrengthoftheorientingresponseduringPavlo-vianconditioning.Journalofexperimentalpsychology.Animalbehaviorprocesses,10(1),90{109.Konorski,J.(1948).Conditionedre exesandneuronorganization.Cambridge:Cam-bridgeUniversityPress.Konorski,J.(1967).Integrativeactivityofthebrain.Chicago:UniversityofChicagoPress.Kruschke,J.K.(1992).ALCOVE:anexemplar-basedconnectionistmodelofcategorylearning.Psychologicalreview,99(1),22{44.Lawrence,M.A.(2016).package`ez':EasyAnalysisandVisualizationofFactorialExperiments.Retrievedfromhttps://cran.r-project.org/web/packages/ez/162 index.htmlLee,B.,Gentry,R.N.,Bissonette,G.B.,Herman,R.J.,Mallon,J.J.,Bryden,D.W.,...Roesch,M.R.(2018).Manipulatingtherevisionofrewardvalueduringtheintertrialintervalincreasessigntrackinganddopaminerelease.PLOSBiology,16(9),e2004015.Lesaint,F.,Sigaud,O.,Flagel,S.B.,Robinson,T.E.,&Khamassi,M.(2014).ModellingIndividualDierencesintheFormofPavlovianConditionedApproachResponses:ADualLearningSystemsApproachwithFactoredRepresentations.PLoSComputBiol,10(2).Leslie,J.C.,&Millenson,J.R.(1996).Principlesofbehavioralanalysis.Amsterdam,TheNeth

erlands:HarwoodAcademicPublishers.Lin,T.E.,&Honey,R.C.(2016).LearningAboutStimuliThatArePresentandThoseThatAreNot.InThewileyhandbookonthecognitiveneuroscienceoflearning(pp.69{85).Chichester,UK:JohnWiley&Sons,Ltd.Lonsdorf,T.B.,Weike,A.I.,Nikamo,P.,Schalling,M.,Hamm,A.O.,&Ohman,A.(2009).GeneticGatingofHumanFearLearningandExtinction.PsychologicalScience,20(2),198{206.Lotz,A.,Uengoer,M.,Koenig,S.,Pearce,J.M.,&Lachnit,H.(2012).Anexplorationofthefeature-positiveeectinadulthumans.Learning&Behavior,40(2),222{230.Lovic,V.,Saunders,B.T.,Yager,L.M.,&Robinson,T.E.(2011).Ratspronetoattributeincentivesaliencetorewardcuesarealsopronetoimpulsiveaction.BehaviouralBrainResearch,223(2),255{261.Lubow,R.E.(1989).Latentinhibitionandconditionedattentiontheory.Cambridge:CambridgeUniversityPress.Lubow,R.E.,&Moore,A.U.(1959).Latentinhibition:Theeectofnonreinforcedpre-exposuretotheconditionalstimulus.JournalofComparativeandPhysiologicalPsychology,52(4),415{419.Mackintosh,N.J.(1974).Thepsychologyofanimallearning.London:AcademicPress.163 Mackintosh,N.J.(1975).Atheoryofattention:Variationsintheassociabilityofstimuliwithreinforcement.PsychologicalReview,82(4),276{298.Mackintosh,N.J.(1976).Overshadowingandstimulusintensity.AnimalLearning&Behavior,4(2),186{192.Maes,E.,Boddez,Y.,Alfei,J.M.,Krypotos,A.M.,D'Hooge,R.,DeHouwer,J.,&Beckers,T.(2016).Theelusivenatureoftheblockingeect:15failurestoreplicate.JournalofExperimentalPsychology:General,145(9),e49{e71.Marr,D.(1982).Vision:Acomputationalinvestigationintothehumanrepresentationandprocessingofvisualinformation.Cambridge,MA:MITPress.Matzel,L.D.,Han,Y.R.,Grossman,H.,Karnik,M.S.,Patel,D

.,Scott,N.,...Gandhi,C.C.(2003).Individualdierencesintheexpressionofa\general"learningabilityinmice.TheJournalofneuroscience:theocialjournaloftheSocietyforNeuroscience,23(16),6423{33.Matzel,L.D.,&Sauce,B.(2017).Individualdierences:Casestudiesofrodentandprimateintelligence.JournalofExperimentalPsychology:AnimalLearningandCognition,43(4),325{340.McClelland,J.L.,&Rumelhart,D.E.(1981).Aninteractiveactivationmodelofcontexteectsinletterperception:I.Anaccountofbasicndings.PsychologicalReview,88(5),375{407.McLaren,I.P.L.,Kaye,H.,&Mackintosh,N.J.(1989).Anassociativetheoryoftherepresentationofstimuli:applicationstoperceptuallearningandlatentinhibition.InR.Morris(Ed.),Paralleldistributedprocessing:Implicationsforpsychologyandneurobiology(pp.102{130).Oxford:ClarendonPress.McLaren,I.P.L.,&Mackintosh,N.J.(2000).Anelementalmodelofassociativelearn-ing:I.Latentinhibitionandperceptuallearning.AnimalLearning&Behavior,28(3),211{246.Meyer,P.J.,Lovic,V.,Saunders,B.T.,Yager,L.M.,Flagel,S.B.,Morrow,J.D.,&Robinson,T.E.(2012).Quantifyingindividualvariationinthepropensityto164 attributeincentivesaliencetorewardcues.PLoSONE,7(6),e38987.Miller,R.R.,Barnet,R.C.,&Grahame,N.J.(1995).AssessmentoftheRescorla-Wagnermodel.Psychologicalbulletin,117(3),363{86.Miller,R.R.,&Matute,H.(1998).CompetitionBetweenOutcomes.PsychologicalScience,9(2),146{149.Miller,R.R.,&Matzel,L.D.(1988).TheComparatorHypothesis:AResponseRuleforTheExpressionofAssociations.PsychologyofLearningandMotivation,22,51{92.Morrison,S.E.,Bamkole,M.A.,&Nicola,S.M.(2015).Signtracking,butnotgoaltracking,isresistanttooutcomedevaluation.FrontiersinNeuroscience,9(DEC),

1{12.Mundy,M.E.,Dwyer,D.M.,&Honey,R.C.(2006).Inhibitoryassociationscontributetoperceptuallearninginhumans.JournalofExperimentalPsychology:AnimalBehaviorProcesses,32(2),178{184.Murphy,R.A.,&Honey,R.C.(2016).TheWileyHandbookontheCognitiveNeuro-scienceofLearning(R.A.Murphy&R.C.Honey,Eds.).Chichester,UK:JohnWiley&Sons,Ltd.Ostlund,S.B.,&Balleine,B.W.(2008).Onhabitsandaddiction:Anassociativeanalysisofcompulsivedrugseeking.Drugdiscoverytoday.Diseasemodels,5(4),235{245.Patitucci,E.,Nelson,A.J.D.,Dwyer,D.M.,&Honey,R.C.(2016).TheOriginsofIndividualDierencesinHowLearningIsExpressedinRats:AGeneral-ProcessPerspective.JExpPsycholAnimLearnCogn,42(4),313{324.Pavlov,I.P.(1927).Conditionedre exes.London:OxfordUniversityPress.Pavlov,I.P.(1930).Abriefoutlineofthehighernervousactivity.InC.Murchison(Ed.),Psychologiesofthe1930(pp.207{220).Worcester,MA:ClarkUniversitiesPress.Pavlov,I.P.(1941).Conditionedre exesandpsychiatry.NewYork,NY:International165 Publishers.Pearce,J.M.(1994).Similarityanddiscrimination:aselectivereviewandaconnec-tionistmodel.Psychologicalreview,101(4),587{607.Pearce,J.M.,Aydin,A.,&Redhead,E.S.(1997).Conguralanalysisofsummationinautoshaping.JournalofExperimentalPsychology:AnimalBehaviorProcesses,23(1),84{94.Pearce,J.M.,&Hall,G.(1980).AmodelforPavlovianlearning:Variationsintheeectivenessofconditionedbutnotofunconditionedstimuli.PsychologicalReview,87(6),532{552.Pearce,J.M.,&Mackintosh,N.J.(2010).Twotheoriesofattention:Areviewandpossibleintegration.InC.Mitchel&M.E.LePelley(Eds.),Attentionandasso-ciativelearning:Frombraintobehaviour(pp.11{39).Oxford:OxfordUniversityPress.Pearce,J.M.,Redhead,E.S.,&George,

D.N.(2002).Summationinautoshapingisaectedbythesimilarityofthevisualstimulitothestimulationtheyreplace.Journalofexperimentalpsychology.Animalbehaviorprocesses,28(2),175{89.Pena-Oliver,Y.,Giuliano,C.,Economidou,D.,Goodlett,C.R.,Robbins,T.W.,Dalley,J.W.,&Everitt,B.J.(2015).Alcohol-PreferringRatsShowGoalOrientedBehaviourtoFoodIncentivesbutAreNeitherSign-TrackersNorImpulsive.PLoSONE,10(5),e0131016.Rescorla,R.A.(1968).ProbabilityofshockinthepresenceandabsenceofCSinfearconditioning.Journalofcomparativeandphysiologicalpsychology,66(1),1{5.Rescorla,R.A.(1969).Pavlovianconditionedinhibition.PsychologicalBulletin,72(2),77{94.Rescorla,R.A.(1971).Variationintheeectivenessofreinforcementandnonrein-forcementfollowingpriorinhibitoryconditioning.LearningandMotivation,2(2),113{123.Rescorla,R.A.(1973).EectofUShabituationfollowingconditioning.Journalof166 comparativeandphysiologicalpsychology,82(1),137{43.Rescorla,R.A.(1988).Pavlovianconditioning.It'snotwhatyouthinkitis.TheAmericanpsychologist,43(3),151{60.Rescorla,R.A.(2000).Associativechangesinexcitorsandinhibitorsdierwhentheyareconditionedincompound.Journalofexperimentalpsychology.Animalbehaviorprocesses,26(4),428{38.Rescorla,R.A.(2001).Unequalassociativechangeswhenexcitorsandneutralstimuliareconditionedincompound.TheQuarterlyjournalofexperimentalpsychology.B,Comparativeandphysiologicalpsychology,54(1),53{68.Rescorla,R.A.,&Colwill,R.M.(1983).Within-compoundassociationsinunblocking.JournalofExperimentalPsychology:AnimalBehaviorProcesses,9(4),390{400.Rescorla,R.A.,&Cunningham,C.L.(1978).Within-compound avorassociations.JournalofExperimentalPsychology:AnimalBehav

iorProcesses,4(3),267{275.Rescorla,R.A.,&Wagner,A.R.(1972).Atheoryofpavlovianconditioning:vari-ationsintheeectivenessofreinforcementandnonreinforcement.InB.A.H.&W.Prokasy(Eds.),Clasicalconditioningii:currentresearchandtheory(pp.64{69).NewYork:Appleton-Century-Crofts.Rouder,J.N.,Morey,R.D.,Speckman,P.L.,&Province,J.M.(2012).DefaultBayesfactorsforANOVAdesigns.JournalofMathematicalPsychology,56(5),356{374.Rouder,J.N.,Morey,R.D.,Verhagen,J.,Swagman,A.R.,&Wagenmakers,E.(2017).Bayesiananalysisoffactorialdesigns.PsychologicalMethods,22(2),304{321.Rouder,J.N.,Speckman,P.L.,Sun,D.,Morey,R.D.,&Iverson,G.(2009).Bayesianttestsforacceptingandrejectingthenullhypothesis.PsychonomicBulletin&Review,16(2),225{237.RStudio,T.(2015).RStudio:IntegratedDevelopmentEnvironmentforR.Boston,MA:RStudio,Inc.Retrievedfromhttps://rstudio.comRumelhart,D.E.,Hinton,G.E.,&Williams,R.J.(1986).Learningrepresentationsbyback-propagatingerrors.Nature,323(6088),533{536.167 Schultz,W.(2006).BehavioralTheoriesandtheNeurophysiologyofReward.AnnualReviewofPsychology,57(1),87{115.Schultz,W.,Dayan,P.,&Montague,P.R.(1997).Aneuralsubstrateofpredictionandreward.Science(NewYork,N.Y.),275(5306),1593{9.Shettleworth,S.J.(1972).ConstraintsonLearning.InD.S.Lehrman,R.A.Hinde,&E.Shaw(Eds.),Advancesinthestudyofbehavior(pp.1{68).AcademicPress.Skinner,B.F.(1948).\Superstition"inthepigeon.JournalofExperimentalPsychology,38(2),168{172.Spence,K.W.(1936).Thenatureofdiscriminationlearninginanimals.PsychologicalReview,43(5),427{449.Spence,K.W.(1937).Thedierentialresponseinanimalstostimulivaryingwithinasingledimension.PsychologicalReview,44(5),430{444.Stout,S.C.,&Miller,R

.R.(2007).Sometimes-competingretrieval(SOCR):Aformal-izationofthecomparatorhypothesis.PsychologicalReview,114(3),759{783.Terrace,H.S.,Gibbon,J.,Farrell,L.,&Baldock,M.D.(1975).Temporalfactorsin uencingtheacquisitionandmaintenanceofanautoshapedkeypeck.AnimalLearning&Behavior,3(1),53{62.Thorndike,E.L.(1927).TheLawofEect.TheAmericanJournalofPsychology,39(1/4),212.Timberlake,W.(1994).Behaviorsystems,associationism,andPavlovianconditioning.PsychonomicBulletin&Review,1(4),405{420.Timberlake,W.,Wahl,G.,&King,D.A.(1982).Stimulusandresponsecontingenciesinthemisbehaviorofrats.JournalofExperimentalPsychology:AnimalBehaviorProcesses,8(1),62{85.Urcelay,G.P.(2017).Competitionandfacilitationincompoundconditioning.JournalofExperimentalPsychology:AnimalLearningandCognition,43(4),303{314.Volkow,N.D.,Fowler,J.S.,Wang,G.,&Swanson,J.M.(2004).Featurereview:Dopamineindrugabuseandaddiction:resultsfromimagingstudiesandtreatment168 implications.MolecularPsychiatry,9,557{569.Wagner,A.R.(1981).SOP:Amodelofautomaticmemoryprocessinginanimalbehavior.InN.E.Spear&R.R.Miller(Eds.),Informationprocessinginanimals:Memorymechanisms(pp.5{148).Hillsdale,N.J.:Erlbaum.Wagner,A.R.,&Brandon,S.E.(1989).EvolutionofastructuredconnectionistmodelofPavlovianconditioning(SOP).InS.B.Klein&R.R.Mowrer(Eds.),Con-temporarylearningtheories:Pavlovianconditioningandthestatusoftraditionallearningtheories(pp.149{189).Hillsdale,N.J.:Erlbaum.Wagner,A.R.,Logan,F.A.,Haberlandt,K.,&Price,T.(1968).Stimulusselectioninanimaldiscriminationlearning.JournalofExperimentalPsychology,64,191{199.Wagner,A.R.,&Rescorla,R.A.(1972).InhibitioninPavlovianconditioning:Ap-plicationsofat

heory.InR.A.Boakes&M.S.Haliday(Eds.),Inhibitionandlearning(pp.301{336).NewYork:AcademicPress.Wasserman,E.A.(1974).Stimulus-reinforcerpredictivenessandselectivediscriminationlearninginpigeons.JournalofExperimentalPsychology,103(2),284{297.Wasserman,E.A.,Franklin,S.R.,&Hearst,E.(1974).Pavlovianappetitivecontin-genciesandapproachversuswithdrawaltoconditionedstimuliinpigeons.Journalofcomparativeandphysiologicalpsychology,86(4),616{27.Watson,J.B.(1924).Behaviorism.NewYoek:People'sInstitutePublishingCompany.Watt,A.,&Honey,R.C.(1997).CombiningCSsassociatedwiththesameordier-entUSs.TheQuarterlyjournalofexperimentalpsychology.B,Comparativeandphysiologicalpsychology,50(4),350{67.Wickham,H.,Chang,W.,Henry,L.,LinPederse,T.,Takashashi,K.,Wike,C.,&Woo,K.(2018).package`ggplot2':CreateElegantDataVisualisationsUsingtheGrammarofGraphics.Retrievedfromhttps://ggplot2.tidyverse.orgWickham,H.,&Henry,L.(2018).package`tidyr':EasilyTidyDatawith`spread()'and`gather()'Functions.Retrievedfromhttps://cran.r-project.org/web/packages/tidyr/index.html169 Wickham,H.,Henry,L.,&Muller,K.(2019).package`dplyr':Agrammarofdatamanipulation.Retrievedfromhttps://cran.r-project.org/web/packages/dplyr/index.htmlWilliams,D.R.,&Williams,H.(1969).Auto-maintenanceinthepigeon:sustainedpeckingdespitecontingentnon-reinforcement.Journaloftheexperimentalanalysisofbehavior,12(4),511{20.Wilson,P.N.,Boumphrey,P.,&Pearce,J.M.(1992).RestorationoftheOrientingResponsetoaLightbyaChangeinitsPredictiveAccuracy:.QuarterlyJournalofExperimentalPsychology,44(B),17{36.Zimmer-Hart,C.L.,&Rescorla,R.A.(1974).ExtinctionofPavlovianconditionedinhibition.Journalofcomparativeandphysi