Download presentation
1 -

StatTheISIsJournalfortheRapidDisseminationofStatisticsResearchwileyonl


StatTYangandZTaninhigh-dimensionalsettingswherepisclosetoorgreaterthannbutthenumberofnonzerofunctionsf3j1uptosomecenteringisstillsmallerthannSeeSection2forareviewofrelatedworkInthisarticleweproposeaba

sadie's Recent Documents

TOWNSHIP O
TOWNSHIP O

F HEMPFIELD2020 BUSINESS PRIVILEGE MERCANTILETAXQUARTERLY RETURNName Address ATTACH SUPPORTING DOCUMENTATION OF GROSS INCOME GROSS SALES OR GROSS RECEIPTS FOR EACH QUARTER FINAL QUARTER SUBMISSIONS A

published 0K
RESOLUTION
RESOLUTION

18-17page 1RESOLUTION18-17 A RESOLUTION OF THE CITY COUNCIL OF THE CITY OF MOUNTAIN HOME IDAHO AUTHORIZING THE EXECUTION AND DELIVERY OF A MUNICIPAL EQUIPMENT LEASE-PURCHASE AGREEMENT WITH MOUNTAIN

published 0K
Roll Call Number Agenda Item NumberIDDATE April 19 2021ACCEPTING COMPL
Roll Call Number Agenda Item NumberIDDATE April 19 2021ACCEPTING COMPL

Roll Call NumberAgenda Item NumbernDATE April 5 2021COUNCIL ACTIONCOWNIEWESTERGAARDGRAYBOESENvossMANDELBAUMGATTOTOTALMOTION CARRIEDYEASNAYSPASSABSENTAPPROVEDMayoiI P Kay Cmelik City Clerk of said City

published 0K
is clearly stated near the beginning of the manuscript The purpose of
is clearly stated near the beginning of the manuscript The purpose of

manuscript is to communicate how you conducted the work and the results obtained so that your audience can assess the veracity of the results and the conclusionAudienceYour audience consists of your p

published 0K
2020Media ContactRiverside County Joint Information Center5087Public
2020Media ContactRiverside County Joint Information Center5087Public

ublic Health is not creating list of participants and will not collect the individual information from the study147The data gleaned from the study provides important information that will help guide o

published 0K
GraecoArabicTranslationMovementinBaghdadandEarlyAbbsidSociety2nd4th8
GraecoArabicTranslationMovementinBaghdadandEarlyAbbsidSociety2nd4th8

theTwelfthCenturyScienceinContext142001 24988CharlesBurnettDanielle

published 0K
How many hours per week do you plan on devoting to this business  Will
How many hours per week do you plan on devoting to this business Will

Page of 4 CG-100-P 3/06Yes indicate the total weekly hours that will be devoted to other businessHave you ever owned or controlled directly or indirectly more than 10 of the voting stock of a busines

published 0K
COPAA 2021 All Rights Reserved
COPAA 2021 All Rights Reserved

Ensuring Free Appropriate Public Education FAPE During Covid-19Denise Marshall COPAA CEOAmy Bonn COPAA Consulting AttorneyThe Council of Parent Attorneys and AdvocatesChangingLivesCreating Opportuniti

published 0K
Download Section

Download - The PPT/PDF document "" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.






Document on Subject : "StatTheISIsJournalfortheRapidDisseminationofStatisticsResearchwileyonl"— Transcript:

1 StatTheISI'sJournalfortheRapidDisseminat
StatTheISI'sJournalfortheRapidDisseminationofStatisticsResearch(wileyonlinelibrary.com)DOI:10.100X/sta.0000 StatTYangandZTaninhigh-dimensionalsettings,wherepisclosetoorgreaterthann,butthenumberofnonzerofunctionsfj(),uptosomecentering,isstillsmallerthann.SeeSection2forareviewofrelatedwork.Inthisarticle,weproposeabackttingalgorithm,calledblockdescentandthresholding(BDT),toimplementdoublypenalizedestimationstudiedinTan&Zhang(2017),usingtotal-variationandempirical-normpenalties.Forafunctiongj()ofjthcovariateinaninterval[aj;bj],thetotalvariationisdenedasTV(gj)=sup(kXi=1jgj(zi)�gj(zi�1)j:z0z1:::zkisapartitionof[aj;bj]foranyk1):Ifgjisdierentiablewithderivativeg(1)j,thenTV(gj)=Rjg(1)j(z)jdz.Theempiricalnorm(precisely,empiricalL2norm)ofgjisdenedaskgjkn=fn�1Pni=1g2j(xij)g1=2.Forsomeintegerm1andtuningparameters(;),thedoublypenalizedestimator(^;^f1;:::;^fp)isdenedasaminimizerofthefollowingpenalizedloss1 2nnXi=1(yi��f(xi))2+pXj=1nTV(f(m�1)j)+kfjkno;(1)over(;f1;:::;fp),wheref=Ppj=1fj,andf(m�1)jisthe(m�1)thderivativeoffjwithf(0)jfj.Iffjismthdierentiable,thenTV(f(m�1)j)=Rjf(m)(z)jdz.ByextendingMammen&vandeGeer(1997)forunivariatesmoothing,Tan&Zhang(2017)showedthateach^fjcanbechosentobeasplineoforderm�1,thatis,apiecewisepolynomialofdegreem�1and,ifm2,an(m�2)thcontinuouslydierentiablefunction.Moreover,theknotsofeach^fjcanbeobtainedfromthedatapointsfxij:i=1;:::;ngifm=1or2,butnotnecessarilysoifm3.Forsimplicity,wealwaysrestricteachfj()tobeasplineoforderm�1withknotsfromthedatapoints.Thepenaltytermin(1)involvestwopenalties,playingcomplementaryroles,foreachcomponentfunctionfj.Thetotalvariation,TV(f(m�1)j),isusedtoinducesmoothnessoffj.Infact,asshallbeseeninSection3.1,thetotal-variationpenaltyleadstoanautomaticselectionofknotsfromdatapointsforeachfj,similarlyasLassoselectionofnonzerocoecientsinlinearregression(Tibshirani,1996).Theempiricalnormkfjknisusedtoinducesparsityforfj(i.e.,settingfjtozero).SeeLemma1forhowazerosolutionforfjcanbecausedbythepresenceoftheempirical-normpenalty.Inthespecialcasewherefj(z)= jzislinearforallj,theoverallpenaltyin(1)reducestotheLassopenaltyPpj=1j jjuptosomescalingforlinearregression.Inotherwords,thepenaltyin(1)canberegardedasafunctionalexten

2 sionoftheLassopenalty,withj jjreplac
sionoftheLassopenalty,withj jjreplacedbyacombinationofTV(f(m�1)j)andkfjkn.ThetheoreticalanalysisinTan&Zhang(2017)showsundersometechnicalconditionsthatifandareproperlyspecied,thenthepredictionerrorof^f=Ppj=1^fjachievesaminimaxratek^f�fk2n=Op(1)(MF+M0)n�2m 2m+1+log(p) n;(2)foranydecompositionf=Ppj=1fjsuchthat#f1jp:fj60gM0andPpj=1TV(fj(m�1))MF.Asimilarresultholdsforout-of-samplepredictionerrors.Theerrorbound(2)consistsoftwoterms,reectingtheerrorsfromrespectivelynonparametricsmoothingandvariableselection.Therstterm,ofordern�2m=(2m+1),istheminimaxrateofestimationinunivariatesmoothing(p=1)overthefunctionclassff1:TV(f(m�1)1)CgforaconstantC(Mammen&vandeGeer,1997).Thesecondterm,oforderM0log(p)=n,isknownastheLassofastrateofpredictionerrorsinlinearregressionwithatmostM0nonzeroregressioncoecients(Bickeletal.,2009).OurbackttingalgorithmcanbemodiedtoimplementdoublypenalizedestimationusingL2Sobolevseminormandempirical-normpenalties,wherethepenaltytermin(1)isreplacedbyPpj=1fkf(m)jkL2+kfjkng,withkf(m)jkL2=..................................................................................................Copyrightc 2012JohnWiley&Sons,Ltd.2Stat2012,00130Preparedusingstaauth.cls BackttingfordoublypenalizedadditivemodelingStatfR(f(m)j(z))2dzg1=2.Denoteby(~f1;:::;~fp)theresultingestimators.Thereare,however,twonotabledierencesfromwhenthetotal-variationpenaltyisused.First,each~fjisasmoothingsplinewithallthedatapointsfxij:i=1;:::;ngasknots(Meieretal.,2009).Noselectionofknotsisachieved.Second,theerrorbound(2)isalsovalidwith^freplacedby~f=Ppj=1~fjandMFreplacedbyPpj=1kfj(m)kL2(Koltchinskii&Yuan,2010;Raskuttietal.,2012).Butaboundedfunctionclassffj:kf(m)jkL2Cgisstrictlysmallerthanffj:TV(f(m�1)j)CgforanyconstantC.Forunivariatesmoothing(p=1),smoothingsplinescannotachievetheraten�2m=(2m+1)asdototal-variationsplinesinthelargerfunctionclassff1:TV(f(m�1)1)Cg(Donoho&Johnstone,1994;Mammen&vandeGeer,1997).Weusethefollowingnotation.Forafunctionh(y;x),theempiricalnormofhisdenedaskhkn=fn�1Pni=1h2(yi;xi)g1=2.Forexample,ky��fk2n=n�1Pni=1(yi��f(xi))2.Thesampleaverageofhish=n�1Pni=1h(yi;xi).Forexample,y=n�1Pni=1yi.Moreover,foravector

3 u=(u1;:::;un)T,denotekukn=(n�1Pni=1u2
u=(u1;:::;un)T,denotekukn=(n�1Pni=1u2i)1=2.Foravectorv=(v1;:::;vk)T,denotekvk1=Pkj=1jvjjandkvk1=maxj=1;:::;kjvjj.Therestofthepaperisorganizedasfollows.Section2reviewsrelatedwork.InSection3,wedevelopabackttingalgorithmforminimizingthepenalizedloss(1)inlinearadditivemodeling.InSection4,weextendthealgorithmtologisticadditivemodeling.Section5presentsnumericalexperiments.Section6concludesthepaper.TheSupplementaryMaterialcontainsproofsandadditionaldiscussionandresults.2.RelatedworkRecently,therehasbeenconsiderableresearchonpenalizedestimationforadditivemodeling,beyondearlierworkasinHastie&Tibshirani(1990).Forexample,Lin&Zhang(2006)usedapenaltyPpj=1kf(m)jkL2.Huangetal.(2010)studiedasimilarmethodusingadaptivegroupLasso.Ravikumaretal.(2009)usedapenaltyPpj=1kfjkn,andrestrictedfj=j jforanndbasismatrixj,whichispre-specied.Meieretal.(2009)usedapenaltyintheform1Ppj=1fkfjk2n+2kf(m)jk2L2g1=2,andparameterizedfjusingB-splinebasisfunctionswithpre-speciedKknotsinnumericalimplementation.Koltchinskii&Yuan(2010)andRaskuttietal.(2012)usedapenaltytermPpj=1fkf(m)jkL2+kfjkng,butdidnotpresentnumericalalgorithmsforminimizingthepenalizedloss.Asmentionedabove,suchdoublypenalizedestimationcanbehandledbyextendingourbackttingalgorithm.ThedoublypenalizedmethodinTan&Zhang(2017)diersfromKoltchinskii&Yuan(2010)andRaskuttietal.(2012)intheuseofthetotal-variationpenalty,whichleadstoautomaticknotselectionforeachcomponentfjandallowsthesamerateofconvergenceachievedinmuchlargerbounded-variationfunctionclasses.Useoftotal-variationpenaltiesseemstoberststudiedbyMammen&vandeGeer(1997)forunivariatesmoothing,wherethepenalizedlossisky�f1k2n=2+TV(f(m�1)1).Recently,Kimetal.(2009)proposedarelatedmethodforunivariatesmoothing,calledtrendltering,byminimizingthepenalizedlossover1,ky�1k22=2+kD(m)P11k1,wherekk2denotestheL2norm,y=(y1;:::;yn)T,1=(f1(x11);:::;f1(xn1))T,P1isthepermutationmatrixthatsorts(x11;:::;xn1)intheascendingorder,andD(m)isthemth-orderdierencematrix.Tibshirani(2014)showedthattrendlteringisequivalenttototal-variationsplinesinMammen&vandeGeer(1997)form=1and2,butnotingeneralform3.Inaddition,trendlteringisshowntoachievethesame(minimax)rateofconvergenceastotal-variationsplinesinbounded-variationclasseswhenthed

4 atapointsareevenlyspacedform1.Forad
atapointsareevenlyspacedform1.Foradditivemodeling,Petersenetal.(2016)proposedafusedlassoadditivemodel(FLAM),byminimizingthepenalizedlossover(;1;:::;p):1 2ky��pXj=1jk22+pXj=1n kD(1)Pjjk1+(1� )kjk2o;..................................................................................................Stat2012,001303Copyrightc 2012JohnWiley&Sons,Ltd.Preparedusingstaauth.cls StatTYangandZTanwhereisatuningparameter, 2[0;1],j=(fj(x1j);:::;fj(xnj))T,andPjisthepermutationmatrixthatsortsthedatapoints(x1j;:::;xnj).Thisttingprocedureiseasilyshowntobestatisticallyequivalenttothatofthedoublypenalizedmethodwithm=1inTan&Zhang(2017),uptoatransformationbetweenthetuningparameters(; )aboveand(;)in(1).Thettedvalues^+Ppj=1^jfromFLAMand(^f(x1);:::;^f(xn))Tfromourmethodcanbematched.However,thereisasubtledierenceinchoosinghowout-of-samplepredictionsaremade.Thettedfunction^fjofjthcovariateisdenedbylinearinterpolationofthettedvalues^jinFLAM,butdirectlybypiecewise-constantinterpolationinourmethod,whichismorealignedwiththerst-ordertotal-variationpenaltyused.Toobtainttedfunctions^fjthatarepiecewiselinear,ourmethodusesthesecond-ordertotal-variationpenalty(i.e.,m=2).Ingeneral,ourmethodalsoaccommodatesuseofhigher-ordertotal-variationpenalties.Sadhanala&Tibshirani(2017)alsoproposedadditivemodelingwithtrendltering,byminimizingthepenalizedloss1 2ky��pXj=1jk22+pXj=1kD(m)Pjjk1:Thismethodallowshigher-ordertrendlteringoncomponentfunctions,butdoesnotincorporateempirical-normpenalties,whicharecrucialtoachievesparsityinhigh-dimensionalsettings.Thetheoryandnumericalevaluationsareprovidedonlyinlow-dimensionalsettingsinSadhanala&Tibshirani(2017).Ourbackttingalgorithmcanbemodiedtohandlebothtrendlteringandempirical-normpenalties,i.e.,apenaltytermPpj=1fkD(m)Pjjk1+kjkng,byrecastingtrendlteringusingthefallingfactorialbasisfunctions(Wangetal.,2014).3.LinearadditivemodelingWedevelopabackttingalgorithmforminimizingtheobjectivefunction(1).AsdiscussedinSection1,werestricteachfjtobeasplineoforderm�1withknotsfromthedatapointsfxij:i=1;:::;ng.For1jp,denetheknotsupersetforfjas(Mammen&vandeGeer,1997)ft(1)j;:::;t(n�

5 ;m)jg=(fx((m�1)=2+2)j;:::;x(n�(m&#
;m)jg=(fx((m�1)=2+2)j;:::;x(n�(m�1)=2)jgifmisodd;fx(m=2+1)j;:::;x(n�m=2)jgifmiseven:wherex(1)j:::x(n)jaretheorderedvaluesofthejthcovariate,andt(1)j:::t(n�m)j.Thedatapointsneartheleftandrightboundariesareremovedtoavoidover-parameterization.AsshowninTan&Zhang(2017),asolution(^f1;:::;^fp)obtainedwiththisrestrictionisalsoanunrestrictedminimizerof(1)ifm=1or2,butnotnecessarilysoifm3.Forunivariatesmoothing,Mammen&vandeGeer(1997)alsoshowedthattherestrictedandunrestrictedsolutionsachievethesamerateofconvergenceunderamildconditiononthemaximumspacingbetweenthedatapoints.Asimilarresultcanbeexpectedtoholdforadditivemodels.Torepresentsplinesoforderm�1,weusethetruncatedpowerbasis,denedas k;j(z)=zk;k=1;:::;m�1; m�1+k;j(z)=(z�t(t)j)m�1+;k=1;2;:::;n�m:where(c)+=max(0;c)and(c)0+=0ifc0or1ifc0.Denotefj= 0j+ j j,where j=( 1;j;:::; n�1;j)and j=( 1;j;:::; n�1;j)T.Aftersimplealgebra,theobjectivefunction(1)canbewrittenas1 2ky��fk2n+pXj=1fkD jk1+k 0j+ j jkng;(3)whereDisadiagonalmatrixwithonlynonzeroelementsdm==dn�1=1.UsingthetruncatedpowerbasistransformsthetotalvariationTV(f(m�1)j)intoaLassopenaltykD jk1=Pn�mk=1j m�1+k;jj...................................................................................................Copyrightc 2012JohnWiley&Sons,Ltd.4Stat2012,00130Preparedusingstaauth.cls BackttingfordoublypenalizedadditivemodelingStatBecauseisnon-penalized,itfollowsthatfor(3)tobeminimized,^=y,andeach^fjisempiricallycentered,thatis,^ 0j=� j^ j,where j=n�1Pni=1 j(xij).Thenminimizationof(3)reducestothatof1 2ky�y�fk2n+pXj=1kD jk1+k~ j jkn ;(4)over( 1;:::; p),wheref=Ppj=1fj,fj=~ j j,and~ j= j� j,theempiricallycenteredversionof j.3.1.BackttingTominimize(3)orequivalently(4),abackttingalgorithminvolvessolvingpsub-problemsupdatingthecomponents(f1;:::;fp)sequentially.Forany1jp,thejthsub-problemismin j1 2krj�~ j jk2n+kD jk1+k~ j jkn;(5)whererj=y�y�Pk6=j^fkandf^fk:k6=jgarethecurrentestimates.Byabuseofnotation,~ jcanalsobetreatedasthen(n�1)matrixwithithrow~ j(xij),andrjbetreatedasthen1vectorwithithelementyi�y�Pk6=j^

6 fk(xik).Proposition1providesthemainideao
fk(xik).Proposition1providesthemainideaofouralgorithmforsolvingproblem(5).ThisresultservesasageneralizationofCorollary3.1inPetersenetal.(2016).Proposition1Supposethat~ jisasolutiontomin j1 2krj�~ j jk2n+kD jk1:(6)Thenasolutiontoproblem(5)is^ j=1� k~ j~ jkn+~ j.Proposition1saysthatasolutiontoproblem(5)isdeterminedbydirectlythresholdingasolutionoftheLassoproblem(6).Fromourproof,amoregeneralresultholdswhererjisa"response"vector,~ jisadatamatrix,andkD jk1isreplacedbyasemi-normpenaltyR( j),includingR( j)=kD jk2inthecaseofL2semi-normkf(m)jkL2usedinsteadofTV(f(m�1)j)in(1).Proposition1shedslightontheconsequencesofusingthetwopenaltiesin(5).Useofthetotal-variationorLassopenaltycaninduceasparsesolution~ jwithonlyafewnonzerocomponents,correspondingtoanautomaticselectionofknotsforfj,asshowninOsborneetal.(1998)forunivariatesmoothing.Useoftheempirical-normpenaltycanresultinazerosolutionforfjviathresholding~ jandhenceachievesparsityinhigh-dimensionaladditivemodelling.FromProposition1,weproposethefollowingbackttingalgorithmtominimize(4).TosolveLassoproblem(6),itispossibletouseavarietyofnumericalmethodsincludingcoordinatedescent(Friedmanetal.,2010;Wu&Lange,2008),gradientdescentrelatedmethods(Beck&Teboulle,2009;Kimetal.,2007),andactive-setdescent(Osborneetal.,2000).Inparticular,Petersenetal.(2016)usedafastfused-Lassoalgorithm(Hoeing,2010)forsolving(6)withm=1,whichseemsnotapplicableform2.Weemployavariantofactive-setdescent,whichisattractiveforthefollowingreasons.First,theperformanceofbackttingdependsonhowaccuratelythesub-problem(6)issolvedwithineachblock.Moreaccuratewithin-blocksolutionsmayresultinfewerbackttingcyclestoachieveconvergencebyacertaincriterion.Theactive-setmethodndsanexactsolutionafteranitenumberofiterations,andthecomputationalcostisoftenreasonableinsparsesettings.Whentheestimate~ jfromthepreviousbackttingcycleisusedasaninitialvalue,themethodalsoallowsproblem(6)tobesolvedwithoneorafewiterationsifthepreviousestimate~ jisclosetothedesiredsolution.SeetheSupplementaryMaterialfordetails..................................................................................................Stat2012,001305Copyrightc 2012JohnWiley&Sons,Ltd.Preparedusingstaauth.cls StatTYangandZTan Algor

7 ithm1BlockDescentandThresholding(BDT)alg
ithm1BlockDescentandThresholding(BDT)algorithm 1:Initialize:Set^ j=0forj=1;:::;p.2:forj=1;2;:::;pdo3:ifanyscreeningcondition(Section3.2)issatisedthen4:Return^ j.5:else6:Updatetheresidual:rj=y�y�Pk6=j~ k^ k.7:Computeasolution~ jtoproblem(6).8:Thresholdthesolution:^ j=1� k~ j~ jkn+~ j.9:endif10:endfor11:Repeatline2-10untilconvergenceoftheobjective(4). oftheactive-setmethod.Manyothermethods,notablycoordinatedescent,onlyprovideanapproximatesolutionwithapre-speciedprecision.Toobtainmorepreciseasolution,moreiterationsareneeded.Inaddition,theactive-setmethodistuningfree,whereasgradientdescentrelatedmethodsinvolvetuningofstepsizestoachievesatisfactoryconvergence.SelectionofsuchstepsizescanbecumbersomeforallpLassoproblems(6)inbacktting.3.2.ScreeningrulesInprinciple,tosolvesub-problem(5),werstcompute~ jandthenthresholditto^ j.Forrelativelylarge,evenif~ jisnonzero,thesolution^ jafterthresholdingmaybecome0.Forrelativelylarge,itmayoccurthatonlytherstm�1componentsof^ j(notpenalizedbykD jk1)arenonzero.Tospeedupcomputation,wederivescreeningrulestodirectlydetectthesescenariosanddetermine^ j,withoutsolvingtheLassoproblem(5).Bythematrixinterpretationfor~ jandrj,rewritethesub-problem(5)asmin( (1)j; (2)j)1 2krj�~ (1)j (1)j�~ (2)j (2)jk2n+k (2)jk1+k~ (1)j (1)j+~ (2)j (2)jkn;(7)where~ jispartitionedinto(~ (1)j;~ (2)j)with~ (1)jgivingtherstm�1columnsof~ j,andaccordingly jispartitionedinto( (1)Tj; (2)Tj)T,with (1)jgivingtherstm�1componentsof j(notpenalizedbyLasso).Ifm=1,then~ (1)jand (1)jbecomedegenerate.Proposition2providesthemainresultforourconstructionofscreeningrules.Proposition2Form2,thefollowingresultsholdforasolution^ j=(^ (1)Tj;^ (2)Tj)Ttoproblem(7).1.^ (1)j6=0and^ (2)j=0ifkjrjkn�andk~ (2)Tj(rj�jrj)k1=n;(8)wherejistheprojection("hat")matrixontothecolumnspaceof~ (1)j.2.^ j=0ifthereexistsavectoru2Rnsatisfyingkuknand~ (1)Tj(rj�u)=0andk~ (2)Tj(rj�u)k1=n:(9)Form=1,result2holdswiththesecondequalityin(9)removed...................................................................................................Copyrightc 2012John

8 Wiley&Sons,Ltd.6Stat2012,00130Prepa
Wiley&Sons,Ltd.6Stat2012,00130Preparedusingstaauth.cls BackttingfordoublypenalizedadditivemodelingStatForeaseofapplication,Corollary1giveareformulationofcondition(9),intermsoftheexistenceofasuitablescalar foranarbitrarilyxedvectorh2Rn.Corollary1Form2,^ j=0isasolutiontoproblem(7)ifforanyh2Rn,thereexists 2Rsatisfyingkrj�(1� )(h�jh)knandk(1� )~ (2)Tj(h�jh)k1=n:(10)Form=1,theresultholdswithh�jhreplacedbyhor,equivalently,jsetto0.Fromtheseresults,weobtainthefollowingscreeningrules.Form=1,weskipstep1andsetj=0. Algorithm2Screeningrules 1:Return^ (1)j=1� kjrjkn(~ (1)Tj~ (1)j)�1~ (1)Tjrjand^ (2)j=0if(a)kjrjkn�andk~ (2)Tj(rj�jrj)k1=n.2:Return^ (1)j=0and^ (2)j=0ifoneofthefollowingconditionsissatised:(a)krjkn.(b)krjkn�andkjrjknandk(1� )~ (2)Tj(rj�jrj)k1=n;where =f(n2�rTjjrj)=(rTjrj�rTjjrj)g1=2(1).(c)krjkn&#x]TJ/;༡ ; .96;& T; 10;&#x.516;&#x 0 T; [0;andk(1� )~ (2)Tj(h�jh)k1=n,whereh=rj�~ j~ jwith~ jobtainedfromthepreviousbackttingcycleand isdeterminedsuchthatkrj�(1� )(h�jh)kn=. Condition1(a)isderivedfrom(8),andcondition2(a)isfrom(9)withu=rj.Condition2(b)isfrom(10)withh=rj,wheretherstinequalityin(10)reducesto 2(rTjrj�rTjjrj)n2�rTjjrj.Condition2(c)isfrom(10)withh=rj�~ j~ j,where~ jisasolutionofLassoproblem(6),andhence~ j~ jisthevectorofttedvaluesbeforethresholding,fromthepreviousbackttingcycle.Themotivationforthischoiceisthatifthe"response"vectorrjissimilartothepreviousone,thenk~ (2)Tj(h�jh)k1=nwouldremainsimilarto.ThescreeningrulesinAlgorithm2aremoreeectivethanwouldbederivedbyrstdetecting~ j=0forproblem(6)andthendeciding^ j=0.Form=1,itholdsthat^ j=0ifeitherkrjknorkrjkn�and(1�=krjkn)k~ Tjrjk1=n,whereasanecessaryandsucientconditionfor~ j=0isk~ Tjrjk1=n.Asanotheruseofthescreeningrules,wealsoobtainthefollowingconditionsonthetuningparameters(;)toimplyacompletelyzerosolutiontoproblem(4),^ 1=:::=^ p=0.Corollary2Itholds

9 that^ 1=:::=^ p=0isasolutiontopr
that^ 1=:::=^ p=0isasolutiontoproblem(4)ifeither(a)ky�yknor(b)ky�ykn�,kj(y�y)kn,andk~ (2)Tj(y�y�jy)k1=nforj=1;:::;p.Innumericalimplementation,werestrictthesearchof(;)toagridsuchthatmaxandmax,wheremax=ky�yknandmax=k~ (2)Tj(y�y�jy)k1=n.Thisregionincludesall(;)yieldinganonzerosolutionto(4)form=1withj=0,andmaybepracticallysucientform2.Inaddition,becausethetheoreticalanalysisofTan&Zhang(2017)suggestschoosing=O(1)2undersparsity,wealsorestrictinthesearch...................................................................................................Stat2012,001307Copyrightc 2012JohnWiley&Sons,Ltd.Preparedusingstaauth.cls StatTYangandZTan4.LogisticadditivemodelingAsanextension,weprovideabackttingalgorithmforlogisticadditivemodelingwithabinaryresponse:P(yi=1jxi)=expitn+pXj=1fj(xij)o;whereexpit(c)=f1+exp(�c)g�1.Doublypenalizedestimationinvolvesminimizingtheobjectivefunction1 nnXi=1`(yi;+f(xi))+pXj=1fkD jk1+k~ j jkng;(11)where`(yi;+f(xi))=logf1+exp(+f(xi))g�yi(+f(xi))withf=Ppj=1fjandfj=~ j jasin(4),butcannotbedirectlysolved.Foreachcycleofbacktting,thejthsub-problemismin(; j)1 nnXi=1`(yi;+^f(�j)(xi)+~ j(xij) j)+kD jk1+k~ j jkn;(12)where^f(�j)=Pk6=j^fkandf^fk:k6=jgarethecurrentestimates.SimilarlyasinFriedmanetal.(2010),weformaquadraticapproximationtothenegativelog-likelihoodterm(viaaTaylorexpansionaboutthepreviousestimates^and^ j)andsolvethefollowingproblem,similarto(5)butwithweightedleastsquares:min(; j)1 2nnXi=1wi(ij��~ j j)2+kD jk1+k~ j jkn;(13)whereij=^+~ j(xij)^ j+^w�1i(yi�^pi),^pi=expit(^+^f(xi)),andwi=^pi(1�^pi).Proposition1canbeeasilyextendedforsolving(13).However,weemployasimplemodicationtotheLassoproblemassociatedwith(13)whenusingtheactive-setalgorithm.Iftheweights(w1;:::;wn)areupdatedduringbacktting,therecanbesubstantialcostinaccordinglyupdatingtheCholeskydecompositionfortheactive-setalgorithm.Therefore,wereplaceeachwibytheconstant1=4andthensolveproblem(13)inthesamewayas(5).Becau

10 se^pi(1�^pi)isupperboundedby1=4,itcan
se^pi(1�^pi)isupperboundedby1=4,itcanshownthattheresultingupdateof(^;^ j)remainsadescentupdate,whichdeceasestheobjectivevalue(12),bythequadraticlowerboundprinciple(Böhning&Lindsay,1988;Wu&Lange,2010).WesummarizethebackttingalgorithmasAlgorithm3. Algorithm3BlockDescentandThresholdingalgorithmforlogisticmodeling(BDT-Logit) 1:Initialize:Set^=0and^ j=0forj=1;:::;p.Setw0=1=4.2:forj=1;2;:::;pdo3:Compute^p=expit(^+Ppj=1~ j^ j)andj=^+~ j^ j+w�10(y�^p).4:Update^=j,thesampleaverageofj.5:Update^ jasasolutionto(usingAlgorithm1,line3-9)min jnw0 2kj�j�~ j jk2n+kD jk1+k~ j jkno:6:endfor7:Repeatline2-7untilconvergenceoftheobjective(11). ..................................................................................................Copyrightc 2012JohnWiley&Sons,Ltd.8Stat2012,00130Preparedusingstaauth.cls BackttingfordoublypenalizedadditivemodelingStat5.NumericalexperimentsWeevaluateBDTalgorithmsanddoublypenalizedadditivemodeling(dPAM)intwoaspects.Oneiscomputationalperformance:wecompareactive-setandcoordinatedescentmethodsforsolvingtheLassosub-problem(6)andinvestigateeectivenessofthescreeningrules.Theotherisstatisticalperformanceintermsofmeansquarederrorsorlogisticlosses:wecomparetheestimatorsobtainedfromdPAMandrelatedmethodsSpAM(Ravikumaretal.,2009)andhGAM(Meieretal.,2009),implementedinRpackagesSAM(Zhaoetal.,2014)andhGAM(Fricketal.,2013).5.1.LinearadditivemodelingWegeneratedataaccordingtoyi=Ppi=1fj(xij)+iwithxijUniform[�2:5;2:5]andiN(0;1)fori=1;:::;n(=100).Considerfourscenarios(piecewiseconstant,piecewiselinear,smooth,andmixed),wherefournonzerofunctions,f1;:::;f4,arespecied(Figure1).Therstscenario(piecewise-constant)isthesameasinPetersenetal.(2016).Theremainingfunctions,f5;:::;fp,arezero,withp=100.5.1.1.ComputationalspeedWecomparetheactive-set(AS)andcoordinate-descent(CD)methodsforsolvingsub-problem(6)inBDT.Tofocusonthiscomparison,thetwoversionsofBDTwithoutscreeningrulesaredenotedasAS-BDTandCD-BST.Thetotalnumberofbasisfunctionsfrompcovariatesislarge,(n�1)p.Insteadofstoringthebasismatrices,(~ 1;:::;~ p),weonlystoretheinner-productmatrices,(~ T1~ S1;:::;~ Tp~ Sp),where~ Sjisasubmatrixof~ jwithcolumnindicesinSj,andSjindicatesthes

11 ubsetofbasisfunctionsofjthcovariatethata
ubsetofbasisfunctionsofjthcovariatethatareeverincludedintheactivesetbeforetheterminationoftraining.Forjthcovariate(orblock),theactivesetisdenedasthesubsetofbasisfunctionswhosecoecientsarecurrentlyestimatedasnonzerowhenusingeitherASorCDmethod.Thetotalcolumnsetofstoredinner-productmatricesisS=S1[[Sp,anditssizeisdenotedasjSj.Form=2;3,weapplyAS-BDTandCD-BDTwitharangeoftuningparameters,=max=4kand=max=4lfor2kl4.Withineachblock,thecoordinate-descentalgorithmisterminatedwiththetolerance10�5;10�6,or10�7inthedecreaseofobjectivevalueswhensolvingLassosub-problem(6).ThebackttingcyclesusingAS-BDTorCD-BDTareterminatedwhenthedecreaseoftheobjective(4)issmallerthan10�4.Figure2and3showtraceplotsoftheobjectivevalues,withsixchoicesof(;)inscenario2(piecewiselinear)form=2;3.SimilarplotsintheothercasesarepresentedintheSupplementaryMaterial.ItisevidentthatnotonlyAS-BDTis10-100timesfasterthanCD-BDTtoreachthesamestoppingcriterion,butalsoAS-BDTachievessmallerobjectivevaluesthanCD-BDTwhenthestoppingcriterionismet.Thesmallerthetuningparameters(;)are,themoresubstantialthespeedgainofAS-BDTisoverCD-BDT.Inaddition,thesmallerthewithin-blocktolerance,from10�5to10�7,thelongerittakesCD-BDTtoreachthestoppingcriterion.Table1summarizesseveralperformancemeasuresforscenario2andm=2;3.SimilarresultsintheothercasesarepresentedintheSupplementaryMaterial.Inadditiontotheobjectivevaluesachieved("obj"),wealsostudythenumberofbackttingcycles("cycle"),theaveragenumberofiterationspercycle(iter"),andthecolumnsizeofstoredinner-productsmatrices("jSj").Foreachcycleofbacktting,thenumberofiterationsisdenedbysummingthenumbersofiterationsoverallpblocks.Thenumberofiterationswithinjthblockishowmanytimesthedescentdirectionisadjustedwhenusingtheactive-setalgorithm,orhowmanyscansareperformedoverthebasisfunctionsofjthcovariatewhenusingthecoordinate-descentalgorithm,tosolvethejthLassosub-problem.TheresultsfromTable1areconsistentwithFigure2and3.ComparedwithCD-BDT,AS-BDTachievessmallerobjectivevalues,withsmallernumbersofbackttingcyclesoriterationspercycle,especiallywithsmall(;).ThenumberofbackttingcyclesfromCD-BDTislargewhenthewithin-blocktoleranceisrelativelylarge,whereasthe.................................................................

12 .................................Stat201
.................................Stat2012,001309Copyrightc 2012JohnWiley&Sons,Ltd.Preparedusingstaauth.cls StatTYangandZTanaveragenumberofiterationspercycleincreasessubstantiallywhenthewithin-blocktoleranceisreduced.Inaddition,thecolumnsizeofstoredinner-productmatricesismuchsmallerinAS-BDTthanCD-BDT.Theactive-setalgorithmismorecarefulthancoordinatedescentinselectingbasisfunctionsintoandremovingthemfromtheactiveset.WealsoapplyAS-BDTwithscreeningrules,denotedasAS-BDT-S.Withineachblock,ifscreeningissuccessful,thenthenumberofiterationsis0.FromTable1,theaveragenumberofiterationspercyclefromAS-BDTisconsiderablysmallerthanfromAS-BDT-S,especiallywhen(;)becomelarge.AdditionalresultsareprovidedintheSupplementaryMaterialonrelativefrequenciesofwhenthescreeningrulesinAlgorithm2aresuccessful.5.1.2.StatisticalperformanceWegeneratetraining,validation,andtestsets,eachwithp=100covariatesandn=100observations.Thetuningparameters(;)fordPAMareselectedtominimizethemeansquarederror(MSE)onthevalidationset,calculatedfromthemodelttedonthetrainingset,overagridmax,max,and(Section3.2).Themodelisthenttedwiththeselected(;)onthetrainingset,andthetestMSEiscalculatedonthetestset.Thecalculationisrepeatedover100datasets,andtheaveragetestMSEisreportedinTable3.Similarly,averagetestMSEsarecalculatedforSpAMwithd=3;6;10andhGAMwithK=15;20;30.FromTable3,thesmallestMSEisachievedbydPAMwithm=1inscenario1(piecewiseconstant),andbydPAMwithm=2intheotherthreescenarios.BothSpAMandhGAMappeartoyieldMSEsconsiderablylargerthantheminimumMSEsobtainedbydPAM,eveninscenario3(smooth).5.2.LogisticadditivemodelingWegeneratedataaccordingtoyiBernoulli(expit(Ppi=jfj(xij))),wherexijUniform[�2:5;2:5],thefunctionsf1;:::;f4arethesameasinSection5.1,andtheremainingf5;:::;fparezerowithp=100.5.2.1.ComputationalspeedWeapplythreeversionsofBDT-logit:AS-BDT-logitandAS-BDT-logit-S,usingtheactive-setmethodwithoutorwiththescreeningrules,andCD-BDT-logit,usingthecoordinate-descentmethod.Table2summarizesseveralperformancemeasuressimilarlyasinTable1.ItisevidentthatAS-BDT-logitoutperformsCD-BDT-logit,inachievingsmallerlogisticlosses,smallernumbersofbackttingcyclesoriterationspercycle,andsmallersizesofstoredinner-productmatrices.Inaddition,AS-BDT-logit-SoutperformsAS-

13 BDT-logit,withsmalleraveragenumbersofite
BDT-logit,withsmalleraveragenumbersofiterationspercycle,duetothescreeningrules.5.2.2.StatisticalperformanceSimilarlyasinSection5.1,weconduct100repeatedsimulations,eachwithtrainingandvalidationsets,tocalculatetestlogisticlossesonatestsetfordPAMandSpAM(Table2).ThereiscurrentlynologisticmodelingallowedintheRpackageforhGAM.Toobtainmeaningfulcomparison,weincreasethesamplesizeton=500,becausethesamplesizen=100appearstobeinsucienttoachievereasonableestimationforlogisticmodeling.AsshowninTable2,dPAMwithm=1yieldsthesmallestlogisticlossesinscenario1(piecewiseconstant),whereasdPAMwithm=2givesthesmallestlossesintheotherthreescenarios.6.ConclusionWedevelopbackttingalgorithmsfordoublypenalizedadditivemodelingusingtotal-variationandempirical-normpenalties,anddemonstratecomputationalandstatisticaleectivenessoftheproposedmethod.ForsolvingtheLassosub-problems(6),weadvocatetheuseoftheactive-setmethodwhencomparedwiththecoordinate-descentmethod.Itcanbeofinteresttoconductmoresimulationsandinvestigatepossibleimprovementandextensions...................................................................................................Copyrightc 2012JohnWiley&Sons,Ltd.10Stat2012,00130Preparedusingstaauth.cls BackttingfordoublypenalizedadditivemodelingStatReferencesBeck,A&Teboulle,M(2009),`Afastiterativeshrinkage-thresholdingalgorithmforlinearinverseproblems,'SIAMJournalonImagingSciences,2(1),pp.183202.Bickel,PJ,Ritov,Y,Tsybakov,ABetal.(2009),`SimultaneousanalysisofLassoandDantzigselector,'AnnalsofStatistics,37(4),pp.17051732.Böhning,D&Lindsay,BG(1988),`Monotonicityofquadraticapproximationalgorithms,'AnnalsoftheInstituteofStatisticalMathematics,40(4),pp.641663.Donoho,DL&Johnstone,JM(1994),`Idealspatialadaptationbywaveletshrinkage,'Biometrika,81(3),pp.425455.Frick,H,Kondofersky,I,Kuehnle,OS,Lindenlaub,C,Pfundstein,G,Speidel,M,Spindler,M,Straub,A,Wickler,F,Zink,K,Eugster,M&Hothorn,T(2013),hGAM:High-dimensionalAdditiveModelling,RPackageVersion0.1-2.Friedman,J,Hastie,T&Tibshirani,R(2010),`Regularizationpathsforgeneralizedlinearmodelsviacoordinatedescent,'JournalofStatisticalSoftware,33(1),pp.122.Hastie,T&Tibshirani,R(1990),GeneralizedAdditiveModels,Wiley.Hoeing,H(2010),`ApathalgorithmforthefusedLassosignalapproximator,'JournalofComputationalandGraphicalStatistics,19(4),pp.9841006.

14 Huang,J,Horowitz,JL&Wei,F(2010),`Variabl
Huang,J,Horowitz,JL&Wei,F(2010),`Variableselectioninnonparametricadditivemodels,'AnnalsofStatistics,38(4),pp.22822313.Kim,SJ,Koh,K,Boyd,S&Gorinevsky,D(2009),``1trendltering,'SIAMReview,51(2),pp.339360.Kim,SJ,Koh,K,Lustig,M,Boyd,S&Gorinevsky,D(2007),`Aninterior-pointmethodforlarge-scale`1-regularizedleastsquares,'IEEEJournalofSelectedTopicsinSignalProcessing,1(4),pp.606617.Koltchinskii,V&Yuan,M(2010),`Sparsityinmultiplekernellearning,'AnnalsofStatistics,pp.36603695.Lin,Y&Zhang,HH(2006),`Componentselectionandsmoothinginmultivariatenonparametricregression,'AnnalsofStatistics,34(5),pp.22722297.Mammen,E&vandeGeer,S(1997),`Locallyadaptiveregressionsplines,'AnnalsofStatistics,25(1),pp.387413.Meier,L,VandeGeer,S&Bühlmann,P(2009),`High-dimensionaladditivemodeling,'AnnalsofStatistics,37(6B),pp.37793821.Osborne,MR,Presnell,B&Turlach,BA(1998),`Knotselectionforregressionsplinesviathelasso,'ComputingScienceandStatistics,30,pp.4449.Osborne,MR,Presnell,B&Turlach,BA(2000),`Anewapproachtovariableselectioninleastsquaresproblems,'IMAJournalofNumericalAnalysis,20(3),pp.389403.Petersen,A,Witten,D&Simon,N(2016),`FusedLassoadditivemodel,'JournalofComputationalandGraphicalStatistics,25(4),pp.10051025.Raskutti,G,Wainwright,MJ&Yu,B(2012),`Minimax-optimalratesforsparseadditivemodelsoverkernelclassesviaconvexprogramming,'JournalofMachineLearningResearch,13,pp.389427.Ravikumar,P,Liu,H,Laerty,J&Wasserman,L(2009),`SpAM:Sparseadditivemodels,'JournaloftheRoyalStatisticalSociety,SeriesB,71(5),pp.10091030...................................................................................................Stat2012,0013011Copyrightc 2012JohnWiley&Sons,Ltd.Preparedusingstaauth.cls StatTYangandZTanSadhanala,V&Tibshirani,RJ(2017),`Additivemodelswithtrendltering,'arXivpreprintarXiv:1702.05037.Stone,CJ(1986),`Thedimensionalityreductionprincipleforgeneralizedadditivemodels,'AnnalsofStatistics,14(2),pp.590606.Tan,Z&Zhang,CH(2017),`Penalizedestimationinadditiveregressionwithhigh-dimensionaldata,'arXivpreprintarXiv:1704.07229.Tibshirani,R(1996),`RegressionshrinkageandselectionviatheLasso,'JournaloftheRoyalStatisticalSociety,SeriesB,pp.267288.Tibshirani,RJ(2014),`Adaptivepiecewisepolynomialestimationviatrendltering,'AnnalsofStatistics,42(1),pp.285323.Wang,Y

15 X,Smola,A&Tibshirani,R(2014),`Thefalling
X,Smola,A&Tibshirani,R(2014),`Thefallingfactorialbasisanditsstatisticalapplications,'inInternationalConferenceonMachineLearning(ICML),pp.730738.Wood,SN(2017),GeneralizedAdditiveModels:anIntroductionwithR,CRCpress.Wu,TT&Lange,K(2008),`CoordinatedescentalgorithmsforLassopenalizedregression,'AnnalsofAppliedStatistics,pp.224244.Wu,TT&Lange,K(2010),`TheMMalternativetoEM,'StatisticalScience,25(4),pp.492505.Zhao,T,Li,X,Liu,H&Roeder,K(2014),SAM:SparseAdditiveModelling,RPackageVersion1.0.5...................................................................................................Copyrightc 2012JohnWiley&Sons,Ltd.12Stat2012,00130Preparedusingstaauth.cls BackttingfordoublypenalizedadditivemodelingStat Figure1.Nonzerofunctionsf1;:::;f4usedtogeneratedata:(1)scenario1(piecewise-constant);(2)scenario2(piecewise-linear);(3)scenario3(smooth);(4)scenario4(combination)...................................................................................................Stat2012,0013013Copyrightc 2012JohnWiley&Sons,Ltd.Preparedusingstaauth.cls -2-1012 -2-1012 (1)xf(x) -2-1012 -2-1012 (2)xf(x) -2-1012 -2-1012 (3)xf(x) -2-1012 -2-1012 (4)xf(x) StatTYangandZTan Figure2.ObjectivevaluesoverrunningtimefromASD-BDT( ),CD-BDTwithtolerance10�5( ),CD-BDTwithtolerance10�6( )andCD-BDTwithtolerance10�7( )forscenario2andm=2inregressionsetting. Figure3.ObjectivevaluesoverrunningtimefromASD-BDT( ),CD-BDTwithtolerance10�5( ),CD-BDTwithtolerance10�6( )andCD-BDTwithtolerance10�7( )forscenario2andm=3inregressionsetting...................................................................................................Copyrightc 2012JohnWiley&Sons,Ltd.14Stat2012,00130Preparedusingstaauth.cls 0123456 1.0801.0851.0901.0951.100 max max runtime (seconds)objective function 051015202530 0.9150.9200.9250.9300.9350.940 max max runtime (seconds)objective function 020406080100 0.780.800.820.84 max max runtime (seconds)objective function 051015202530 0.500.520.540.56 max max runtime (seconds)objective function 020406080 0.340.360.380.400.420.440.46 max max runtime (seconds)objective function 020406080 0.150.200.250.30 max max runtime (seconds)objective function 0246810 0.7850.7900.7950.8000.8050.8100.815 max max runtime (seconds)objective function 0102030405060 0.770.780.790.800.810.820.83 max max runtime

16 (seconds)objective function 05010015020
(seconds)objective function 050100150200250 0.720.740.760.780.800.820.84 max max runtime (seconds)objective function 010203040 0.280.300.320.340.360.380.40 max max runtime (seconds)objective function 050100150200 0.300.350.400.45 max max runtime (seconds)objective function 050100150200 0.100.150.200.250.30 max max runtime (seconds)objective function BackttingfordoublypenalizedadditivemodelingStat Figure4.ObjectivevaluesoverrunningtimefromASD-BDT-Logit( ),CD-BDT-Logitwithtolerance10�5( ),CD-BDT-Logitwithtolerance10�6( )andCD-BDT-Logitwithtolerance10�7( )forscenario2andm=2inclassicationsetting. Figure5.ObjectivevaluesoverrunningtimefromASD-BDT-Logit( ),CD-BDT-Logitwithtolerance10�5( ),CD-BDT-Logitwithtolerance10�6( )andCD-BDT-Logitwithtolerance10�7( )forscenario2andm=3inclassicationsetting...................................................................................................Stat2012,0013015Copyrightc 2012JohnWiley&Sons,Ltd.Preparedusingstaauth.cls 02468 0.5450.5500.5550.560 max max runtime (seconds)objective function 01020304050 0.4800.4850.4900.4950.5000.505 max max runtime (seconds)objective function 050100150 0.410.420.430.440.450.46 max max runtime (seconds)objective function 0102030405060 0.280.300.320.340.36 max max runtime (seconds)objective function 050100150 0.220.240.260.280.30 max max runtime (seconds)objective function 050100150 0.100.150.200.25 max max runtime (seconds)objective function 051015 0.4400.4450.4500.4550.4600.465 max max runtime (seconds)objective function 020406080100 0.430.440.450.46 max max runtime (seconds)objective function 0100200300400 0.390.400.410.420.430.440.45 max max runtime (seconds)objective function 020406080 0.180.200.220.240.260.28 max max runtime (seconds)objective function 050100200300 0.180.200.220.240.260.28 max max runtime (seconds)objective function 050100200300 0.100.150.20 max max runtime (seconds)objective function StatTYangandZTanTable1.Computationcomparisonforscenario2inregressionsetting MetricMethod=max=16=max=64=max=256 ===4==16===4= m=2objAS-BDT1.08000.91430.76540.49280.33330.1563AS-BDT-S1.08000.91430.76540.49280.33330.1563CD-BDT10�51.08010.91820.77540.49560.34050.1637CD-BDT10�61.08010.91610.76770.49380.33610

17 .1588CD-BDT10�71.08010.91540.76590.49
.1588CD-BDT10�71.08010.91540.76590.49310.33420.1568 cycleAS-BDT646121430AS-BDT-S646121430CD-BDT10�56512172848CD-BDT10�6646121432CD-BDT10�7646121331 iterAS-BDT118180262127182143AS-BDT-S3812013813483CD-BDT10�534412411406450612430CD-BDT10�6378268062328862064995CD-BDT10�7655543516291185353672368 jSjAS-BDT204328688303677694AS-BDT-S204311663301656693CD-BDT10�5203862368551599284898572CD-BDT10�6203861938584601184618588CD-BDT10�7204661798585602784558600 m=3objAS-BDT0.78270.77220.72480.27600.25890.0788AS-BDT-S0.78270.77220.72480.27600.25890.0788CD-BDT10�50.78270.77480.78140.27600.31400.1319CD-BDT10�60.78270.77300.73110.27600.26270.0791CD-BDT10�70.78270.77240.72590.27600.25950.0788 cycleAS-BDT657161536AS-BDT-S657161536CD-BDT10�5661317189262CD-BDT10�6657162037CD-BDT10�7657161636 iterAS-BDT159224296133197138AS-BDT-S0712171511744CD-BDT10�51023427416021448273241CD-BDT10�61036653912686173244762556CD-BDT10�710589870362722298129095355 jSjAS-BDT378500848454825779AS-BDT-S300412785381779741CD-BDT10�5432670218338678682428309CD-BDT10�6432770018325679282198239CD-BDT10�7432770108339680582148255 ..................................................................................................Copyrightc 2012JohnWiley&Sons,Ltd.16Stat2012,00130Preparedusingstaauth.cls BackttingfordoublypenalizedadditivemodelingStatTable2.Computationcomparisonforscenario2inclassicationsetting MetricMethod=max=16=max=64=max=256 ===4==16===4= m=2objAS-BDT-Logit0.54110.47790.40390.26660.20720.1035AS-BDT-Logit-S0.54110.47790.40390.26660.20720.1035CD-BDT-Logit10�50.54230.48040.40790.26750.21190.1059CD-BDT-Logit10�60.54230.47910.40490.26680.20930.1048CD-BDT-Logit10�70.54220.47870.40420.26680.20760.1039 cycleAS-BDT-Logit6813323060AS-BDT-Logit-S6813323060CD-BDT-Logit10�571116353779CD-BDT-Logit10�66813333061CD-BDT-Logit10�77813323060 iterAS-BDT-Logit122159205114145125AS-BDT-Logit-S773144269964CD-BDT-Logit10�54789961475427650399CD-BDT-Logit10�6571263544467221723995CD-BDT-Logit10�7710608411297156446542488 jSjAS-BDT-Logit216396828356764789AS-BDT-Logit-S213365797356747789CD-BDT-Logit10�5349369508653726684718887CD-BDT-Logit10�6348869408627725685128867CD-BDT-Logit10&

18 #0;7349669528640726484878867 m=3objAS-BD
#0;7349669528640726484878867 m=3objAS-BDT-Logit0.43700.42480.38650.18260.17260.0665AS-BDT-Logit-S0.43700.42480.38650.18260.17260.0665CD-BDT-Logit10�50.43700.42690.40070.18250.19380.0835CD-BDT-Logit10�60.43700.42540.38940.18260.17520.0665CD-BDT-Logit10�70.43700.42490.38690.18260.17320.0665 cycleAS-BDT-Logit10913283151AS-BDT-Logit-S10913283151CD-BDT-Logit10�51011172971164CD-BDT-Logit10�6101014283457CD-BDT-Logit10�710913283153 iterAS-BDT-Logit138188231124156139AS-BDT-Logit-S179159199255CD-BDT-Logit10�5889298916151116460295CD-BDT-Logit10�691047898339152231652121CD-BDT-Logit10�79701017427507245894965538 jSjAS-BDT-Logit392576966525953990AS-BDT-Logit-S302506901470929975CD-BDT-Logit10�5507173428481725884368648CD-BDT-Logit10�6507173538457725783948619CD-BDT-Logit10�7507373518447725683888609 ..................................................................................................Stat2012,0013017Copyrightc 2012JohnWiley&Sons,Ltd.Preparedusingstaauth.cls StatTYangandZTanTable3.TestMSEsfromlinearadditivemodeling Scenario1Scenario2Scenario3Scenario4 SpAM(Ravikumaretal.,2009)d=32.88(0.05)1.75(0.03)1.62(0.02)1.78(0.03)d=63.07(0.05)2.41(0.04)2.03(0.03)2.13(0.04)d=103.37(0.06)3.18(0.05)2.69(0.05)2.61(0.05) hGAM(Meieretal.,2009)K=153.32(0.04)1.59(0.03)1.97(0.03)1.84(0.02)K=203.29(0.04)1.57(0.03)1.95(0.03)1.82(0.02)K=303.25(0.04)1.53(0.03)1.89(0.03)1.78(0.02) dPAM(thispaper)m=12.03(0.04)1.88(0.03)1.76(0.03)1.74(0.02)m=22.28(0.04)1.40(0.02)1.40(0.02)1.60(0.02)m=32.40(0.04)1.51(0.02)1.46(0.02)1.64(0.02) Note:FLAM(Petersenetal.,2016)correspondstodPAMwithm=1,exceptlinearinterpolationisusedwhenevaluatingthettedfunctionsonthevalidationandtestsets.Table4.Testlogisticlosses(10)fromlogisticadditivemodeling Scenario1Scenario2Scenario3Scenario4 SpAM(Ravikumaretal.,2009)d=35.38(0.01)5.38(0.01)5.33(0.01)5.08(0.01)d=65.18(0.01)5.49(0.01)5.42(0.01)5.18(0.01)d=105.13(0.02)5.55(0.01)5.46(0.01)5.25(0.01) dPAM(thispaper)m=14.97(0.02)5.08(0.01)5.18(0.01)5.04(0.02)m=25.11(0.02)4.88(0.01)5.01(0.01)5.01(0.01)m=35.14(0.02)4.88(0.01)5.04(0.01)5.01(0.01) Note:LogisticmodelingiscurrentlynotimplementedintheRpackageforhGAM(Meieretal.,2009)...................................................................................................Copyrightc 2012JohnWiley&Sons,Ltd.18Stat2012,00130Prepare