DaiyiPengXuanyiDongEstebanRealMingxingTanYifengLuHanxiaoLiuGabrielBenderAdamKraftChenLiangQuocVLeGoogleResearchBrainTeamdaiyiperealtanmingxingyifengluhanxiaolgbenderadamkraftcrazydonkeyqvlgooglecomxua ID: 883281
Download Pdf The PPT/PDF document "PyGloveSymbolicProgrammingforAutomatedMa..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1 PyGlove:SymbolicProgrammingforAutomatedM
PyGlove:SymbolicProgrammingforAutomatedMachineLearning DaiyiPeng,XuanyiDong,EstebanReal,MingxingTan,YifengLuHanxiaoLiu,GabrielBender,AdamKraft,ChenLiang,QuocV.LeGoogleResearch,BrainTeam{daiyip,ereal,tanmingxing,yifenglu,hanxiaol,gbender,adamkraft,crazydonkey,qvl}@google.comxuanyi.dxy@gmail.comAbstractNeuralnetworksaresensitivetohyper-parameterandarchitecturechoices.Auto-matedMachineLearning(AutoML)isapromisingparadigmforautomatingthesechoices.CurrentMLsoftwarelibraries,however,arequitelimitedinhandlingthedynamicinteractionsamongthecomponentsofAutoML.Forexample,efcientNASalgorithms,suchasENAS[1]andDARTS[2],typicallyrequireanimple-mentationcouplingbetweenthesearchspaceandsearchalgorithm,thetwokeycomponentsinAutoML.Furthermore,implementingacomplexsearchow,suchassearchingarchitectureswithinaloopofsearchinghardwarecongurations,isdifcult.Tosummarize,changingthesearchspace,searchalgorithm,orsearchowincurrentMLlibrariesusuallyrequiresasignicantchangeintheprogramlogic.Inthispaper,weintroduceanewwayofprogrammingAutoMLbasedonsymbolicprogramming.Underthisparadigm,MLprogramsaremutable,thuscanbemanipulatedeasilybyanotherprogram.Asaresult,AutoMLcanbereformulatedasanautomatedprocessofsymbolicmanipulation.Withthisformulation,wedecouplethetriangleofthesearchalgorithm,thesearchspaceandthechildprogram.Thisdecouplingmakesiteasytochangethesearchspaceandsearchalgorithm(withoutandwithweightsharing),aswellastoaddsearchcapabilitiestoexistingcodeandimplementcomplexsearchows.WethenintroducePyGlove,anewPythonlibrarythatimplementsthisparadigm.ThroughcasestudiesonImageNetandNAS-Bench-101,weshowthatwithPyGloveuserscaneasilyconvertastaticprogramintoasearchspace,quicklyiterateonthesearchspacesandsearchalgorithms,andcraftcomplexsearchowstoachievebetterresults.1IntroductionNeuralnetworksaresensitivetoarchitectureandhyper-parameterchoices[3,4].Forexample,ontheImageNetdataset[5],wehaveobservedalargeincreaseinaccuracythankstochangesinarchitectures,hyper-parameters,andtrainingalgorithms,fromtheseminalworkofAlexNet[5]torecentstate-of-the-artmodelssuchasEfcientNet[6].However,asneuralnetworksbecomeincreasinglycomplex,thepotentialnumberofarchitectureandhyper-parameterchoicesbecomesnumerous.Hand-craftingneuralnetworkarchitecturesandselectingtherighthyper-parametersis,therefore,increasinglydifcultandoftentakemonthsofexperimentation.AutomatedMachineLearning(AutoML)isapromisingparadigmfortacklingthisdifculty.InAutoML,selectingarchitecturesandhyper-parametersisformulatedasasearchproblem,whereasearchspaceisdenedtorepresentallpossiblechoicesandasearchalgorithmisusedtondthe WorkdoneasaresearchinternatGoogle.34thConferenceonNeuralInformationProcessingSystems(NeurIPS2020),Vancouver,Canada. bestchoices.Forhyper-parametersearch,thesearchspacewouldspecifytherangeofvaluestotry.Forarchitecturesearch,thesearchspacewouldspecifythearchitecturalcongurationstotry.Thesearchspaceplaysacriticalroleinthesuccessofneuralarchitecturesearch(NAS)[7,8],andcanbesignicantlydifferentfromoneapplicationtoanother[811].Inaddition,therearealsomanydifferentsearchalgorithms,suchasrandomsearch[12],Bayesianoptimization[13],RL-basedmethods[1,9,14,15],evolutionarymethods[16],gradient-basedmethods[2,10,17]andneuralpredictors[18].ThisproliferationofsearchspacesandsearchalgorithmsinAutoMLmakesitdifculttoprogramwithexistingsoftwarelibraries.Inparticular,acommonproblemofcurrentlibrariesisthatsearchspacesandsearchalgorithmsaretightlycoupled,makingithardtomodifysearchspaceorsearchalgorithmalone.Apracticalscenariothatarisesistheneedtoupgradeasearchalgorithmwhilekeepingtherestoftheinfrastructurethesame.Forexample,recentyearshaveseenatransitionfromAutoMLalgo-rithmsthattraineachmodelfromscratch[8,9]tothosethatemployweight-sharingtoattainmassiveefciencygains,suchasENASandDARTS[1,2,14,15,19].Yet,upgradinganexistingsearchspacebyintroducingweight-sharingrequiressignicantchangestoboththesearchalgorithmandthemodelbuildinglogic,aswewillseeinSection2.2.Suchcouplingbetweensearchspacesandsearchalgo-rithms,andtheresultinginexibility,imposeaheavyburdenonAutoMLresearchersandpractitioners.WebelievethatthemainchallengeliesintheprogrammingparadigmmismatchbetweenexistingsoftwarelibrariesandAutoML.Mostexistinglibrariesarebuiltonthepremiseofimmutableprograms,whereaxedprogramisusedtoprocessdifferentdata.Onthecontrary,AutoMLrequiresprograms(i.e.modelarchitectures)tobemutable,astheymustbedynamicallymodiedbyanotherprogram(i.e.thesearchalgorithm)whosejobistoexplorethesearchspace.Duetothismismatch,predenedinterfacesforsearchspacesandsearchalgorithmsstruggletoaccommodateunanticipatedinteractions,makingitdifculttotrynewAutoMLapproaches.Symbolicprogramming,whichoriginatedfromLISP[20],providesapotentialsolutiontothisproblem,byallowingaprogramtomanipulateitsowncomponentsasiftheywereplaindata[21].However,despiteitslonghistory,symbolicprogramminghasnotyetbeenwidelyexploredintheMLcommunity.Inthispaper,wereformulateAutoMLasanautomatedprocessofmanipulatingMLprogramssymbolically.Underthisformulation,programsaremutableobjectswhichcanbeclonedandmodiedaftertheircreation.Thesemutableobjectscanexpressstandardmachinelearningconcepts,fromaconvolutionalunittoacomplexuser-denedtrainingprocedure.Asaresult,allpartsofaMLprogramaremutable.Moreover,throughsymbolicprogramming,pr
2 ogramscanmodifyprograms.Thereforetheinte
ogramscanmodifyprograms.Thereforetheinteractionsbetweenthechildprogram,searchspace,andsearchalgorithmarenolongerstatic.Wecanmediatethemorchangethemviameta-programs.Forexample,wecanmapthesearchspaceintoanabstractviewwhichisunderstoodbythesearchalgorithm,translatinganarchitecturalsearchspaceintoasuper-networkthatcanbeoptimizedbyefcientNASalgorithms.Further,weproposePyGlove,alibrarythatenablesgeneralsymbolicprogramminginPython,asanimplementationofourmethodtestedonreal-worldAutoMLscenarios.WithPyGlove,PythonclassesandfunctionscanbemademutablethroughbriefPythonannotations,whichmakesitmucheasiertowriteAutoMLprograms.PyGloveallowsAutoMLtechniquestobeeasilydroppedintopreexistingMLpipelines,whilealsobenetingopen-endedresearchwhichrequiresextremeexibility.Tosummarize,ourcontributionsarethefollowing:WereformulateAutoMLunderthesymbolicprogrammingparadigm,greatlysimplifyingtheprogramminginterfaceforAutoMLbyaccommodatingunanticipatedinteractionsamongthechildprograms,searchspacesandsearchalgorithmsviaamutableobjectmodel.WeintroducePyGlove,ageneralsymbolicprogramminglibraryforPythonwhichim-plementsoursymbolicformulationofAutoML.WithPyGlove,AutoMLcanbeeasilydroppedintopreexistingMLprograms,withallprogrampartssearchable,permittingrapidexplorationondifferentdimensionsofAutoML.Throughcasestudies,wedemonstratetheexpressivenessofPyGloveinreal-worldsearchspaces.WedemonstratehowPyGloveallowsAutoMLresearchersandpractitionerstochangesearchspaces,searchalgorithmsandsearchowswithonlyafewlinesofcode.2SymbolicProgrammingforAutoMLManyAutoMLapproaches(e.g.,[2,9,22])canbeformulatedasthreeinteractingcomponents:thechildprogram,thesearchspace,andthesearchalgorithm.AutoML'sgoalistodiscoveraperformantchildprogram(e.g.,aneuralnetworkarchitectureoradataaugmentationpolicy)outofalargeset2 ofpossibilitiesdenedbythesearchspace.Thesearchalgorithmaccomplishesthesaidgoalbyiterativelysamplingchildprogramsfromthesearchspace.Eachsampledchildprogramisthenevaluated,resultinginanumericmeasureofitsquality.Thismeasureiscalledthereward2.Therewardisthenfedbacktothesearchalgorithmtoimprovefuturesamplingofchildprograms.IntypicalAutoMLlibraries[2331],thesethreecomponentsareusuallytightlycoupled.Thecou-plingbetweenthesecomponentsmeansthatwecannotchangetheinteractionsbetweenthemunlessnon-trivialmodicationsaremade.Thislimitstheexibilityofthelibraries.Somesuccessfulat-temptshavebeenmadetobreakthesecouplings.Forexample,Vizier[26]decouplesthesearchspaceandthesearchalgorithmbyusingadictionaryasthesearchspacecontractbetweenthechildprogramandthesearchalgorithm,resultinginmodularblack-boxsearchalgorithms.AnotherexampleistheNNIlibrary[27],whichtriestounifysearchalgorithmswithandwithoutweightsharingbycarefullydesignedAPIs.Thispaper,however,solvesthecouplingprobleminadifferentandmoregeneralway:withsymbolicprogramming,programsareallowedtobemodiedbyotherprograms.Therefore,in-steadofsolvingxedcouplings,weallowdynamiccouplingsthroughamutableobjectmodel.Inthissection,wewillexplainourmethodandshowhowthismakesAutoMLprogrammingmoreexible.2.1AutoMLasanAutomatedSymbolicManipulationProcessAutoMLcanbeinterpretedasanautomatedprocessofsearchingforachildprogramfromasearchspacetomaximizeareward.Wedecomposethisprocessintoasequenceofsymbolicoperations.A(regular)childprogram(Figure1-a)issymbolizedintoasymbolicchildprogram(Figure1-b),whichcanbethenclonedandmodied.Thesymbolicprogramisfurtherhyperiedintoasearchspace(Figure1-c)byreplacingsomeofthexedpartswithto-be-determinedspecications.Duringthesearch,thesearchspaceismaterializedintodifferentchildprograms(Figure1-d)basedonsearchalgorithmdecisions,orcanberewrittenintoasuper-program(Figure1-e)toapplycomplexsearchalgorithmssuchasefcientNAS. Figure1:AutoMLasanautomatedsymbolicmanipulationprocess.AnanalogytothisprocessistohavearobotbuildahousewithLEGO[32]brickstomeetahumanbeing'staste:symbolizingaregularprogramislikeconvertingmoldedplasticpartsintoLEGObricks;hyperifyingasymbolicprogramintoasearchspaceislikeprovidingablueprintofthehousewithvariations.Withthehelpofthesearchalgorithm,thesearchspaceismaterializedintodifferentchildprogramswhoserewardsarefedbacktothesearchalgorithmtoimprovefuturesampling,likearobottryingdifferentwaystobuildthehouseandgraduallylearningwhathumansprefer. Figure2:Symbolizingclassesintomutablesym-bolictrees.Theirhyper-parametersarelikethestudsofLEGObricks,whiletheirimplementationsarelessinterestingwhilewemanipulatethetrees.Symbolization.A(regular)childprogramcanbedescribedasacomplexobject,whichisacomposi-tionofitssub-objects.Asymbolicchildprogramissuchacompositionwhosesub-objectsarenolongertiedtogetherforever,butaredetachablefromeachotherhencecanbereplacedbyothersub-objects.Thesymbolicobjectcanbehierarchical,formingasymbolictreewhichcanbemanipulatedorexecuted.Asymbolicobjectismanipulatedthroughitshyper-parameters,whicharelikethestudsofaLEGObrick,interfacingconnectionswithotherbricks.However,symbolicobjects,unlikeLEGObricks,canhaveinternalstateswhichareautomaticallyrecomputeduponmodications.For 2WhileweuseRLconceptstoillustratethecoreideaofourmethod,aswillbeshownlater,theproposedparadigmisapplicabletoothertypesofAutoMLmethodsaswell.3 V \ P E R O L ] H K \ S H U L I \ S \ T ] P D W H U L D O L ] H U H Z U L W H T S ] [ \ D E F G