Submodular Optimization with Submodular Cover and Submodular Knapsack Constraints Rishabh - PDF document

SubmodularOptimizationwithSubmodularCoverandSubmodularKnapsackConstraints RishabhIyerDepartmentofElectricalEngineeringUniversityofWashingtonrkiyer@u.washington.eduJeffBilmesDepartmentofElectricalEngineeringUniversityofWashingtonbilmes@u.washington.eduAbstractWeinvestigatetwonewoptimizationproblems—minimizingasubmodularfunctionsubjecttoasubmodularlowerboundconstraint(submodularcover)andmaximizingasubmodularfunctionsubjecttoasubmodularupperboundconstraint(submodularknapsack).Wearemotivatedbyanumberofreal-worldapplicationsinmachinelearningincludingsensorplacementanddatasubsetselection,whichrequiremaximizingacertainsubmodularfunction(likecoverageordiversity)whilesimultaneouslyminimizinganother(likecooperativecost).Theseproblemsareoftenposedasminimizingthedifferencebetweensubmodularfunctions[9,25]whichisintheworstcaseinapproximable.Weshow,however,thatbyphrasingtheseproblemsasconstrainedoptimization,whichismorenaturalformanyapplications,weachieveanumberofboundedapproximationguarantees.Wealsoshowthatboththeseproblemsarecloselyrelatedandanapproximationalgorithmsolvingonecanbeusedtoobtainanapproximationguaranteefortheother.Weprovidehardnessresultsforbothproblemsthusshowingthatourapproximationfactorsaretightuptolog-factors.Finally,weempiricallydemonstratetheperformanceandgoodscalabilitypropertiesofouralgorithms.1IntroductionAsetfunctionf:2V!Rissaidtobesubmodular[4]ifforallsubsetsS;TV,itholdsthatf(S)+f(T)f(S[T)+f(S\T).Deningf(jjS),f(S[j)�f(S)asthegainofj2VinthecontextofSV,thenfissubmodularifandonlyiff(jjS)f(jjT)forallSTandj=2T.Thefunctionfismonotoneifff(jjS)0;8j=2S;SV.Forconvenience,weassumethegroundsetisV=f1;2;


;ng.Whilegeneralsetfunctionoptimizationisoftenintractable,manyformsofsubmodularfunctionoptimizationcanbesolvednearoptimallyorevenoptimallyincertaincases.Submodularity,moreover,isinherentinalargeclassofreal-worldapplications,particularlyinmachinelearning,thereforemakingthemextremelyusefulinpractice.Inthispaper,westudyanewclassofdiscreteoptimizationproblemsthathavethefollowingform:Problem1(SCSC):minff(X)jg(X)cg;andProblem2(SCSK):maxfg(X)jf(X)bg;wherefandgaremonotonenon-decreasingsubmodularfunctionsthatalso,w.l.o.g.,arenormalized(f(;)=g(;)=0)1,andwherebandcrefertobudgetandcoverparametersrespectively.Thecorrespondingconstraintsarecalledthesubmodularcover[29]andsubmodularknapsack[1]respectivelyandhencewerefertoProblem1asSubmodularCostSubmodularCover(henceforthSCSC)andProblem2asSubmodularCostSubmodularKnapsack(henceforthSCSK).Ourmotivationstemsfromaninterestingclassofproblemsthatrequireminimizingacertainsubmodularfunctionfwhilesimultaneouslymaximizinganothersubmodularfunctiong.Weshallseethatthesenaturally 1Amonotonenon-decreasingnormalized(f(;)=0)submodularfunctioniscalledapolymatroidfunction.1 occurinapplicationslikesensorplacement,datasubsetselection,andmanyothermachinelearningapplications.Astandardapproachusedinliterature[9,25,15]hasbeentotransformtheseproblemsintominimizingthedifferencebetweensubmodularfunctions(alsocalledDSoptimization):Problem0:minXV�f(X)�g(X)
:(1)WhileanumberofheuristicsareavailableforsolvingProblem0,intheworst-caseitisNP-hardandinapproximable[9],evenwhenfandgaremonotone.Althoughanexactbranchandboundalgorithmhasbeenprovidedforthisproblem[15],itscomplexitycanbeexponentialintheworstcase.Ontheotherhand,inmanyapplications,oneofthesubmodularfunctionsnaturallyservesaspartofaconstraint.Forexample,wemighthaveabudgetonacooperativecost,inwhichcaseProblems1and2becomeapplicable.TheutilityofProblems1and2becomeapparentwhenweconsiderhowtheyoccurinreal-worldapplicationsandhowtheysubsumeanumberofimportantoptimizationproblems.SensorPlacementandFeatureSelection:Often,theproblemofchoosingsensorlocationscanbemodeled[19,9]bymaximizingthemutualinformationbetweenthechosenvariablesAandtheunchosensetVnA(i.e.,f(A)=I(XA;XVnA)).Alternatively,wemaywishtomaximizethemutualinformationbetweenasetofchosensensorsXAandaquantityofinterestC(i.e.,f(A)=I(XA;C))assumingthatthesetoffeaturesXAareconditionallyindependentgivenC[19,9].Boththesefunctionsaresubmodular.Sincetherearecostsinvolved,wewanttosimultaneouslyminimizethecostg(A).Oftenthiscostissubmodular[19,9].Forexample,thereistypicallyadiscountwhenpurchasingsensorsinbulk(economiesofscale).ThisthenbecomesaformofeitherProblem1or2.Datasubsetselection:AdatasubsetselectionprobleminspeechandNLPinvolvesndingalimitedvocabularywhichsimultaneouslyhasalargecoverage.Thisisparticularlyuseful,forexampleinspeechrecognitionandmachinetranslation,wherethecomplexityofthealgorithmisdeterminedbythevocabularysize.Themotivationforthisproblemistondthesubsetoftrainingexampleswhichwillfacilitateevaluationofprototypesystems[23].Oftentheobjectivefunctionsencouragingsmallvocabularysubsetsandlargeacousticspansaresubmodular[23,20]andhencethisproblemcannaturallybecastasaninstanceofProblems1and2.PrivacyPreservingCommunication:GivenasetofrandomvariablesX1;


;Xn,denoteIasaninformationsource,andPasprivateinformationthatshouldbelteredout.ThenonewayofformulatingtheproblemofchoosingainformationcontainingbutprivacypreservingsetofrandomvariablescanbeposedasinstancesofProblems1and2,withf(A)=H(XAjI)andg(A)=H(XAjP),whereH(
j
)istheconditionalentropy.MachineTranslation:Anotherapplicationinmachinetranslationistochooseasubsetoftrainingdatathatisoptimizedforgiventestdataset,aproblempreviouslyaddressedwithmodularfunctions[24].DeningasubmodularfunctionwithgroundsetovertheunionoftrainingandtestsampleinputsV=Vtr[Vte,wecansetf:2Vtr!R+tof(X)=f(XjVte),andtakeg(X)=jXj,andb0inProblem2toaddressthisproblem.WecallthistheSubmodularSpanproblem.Apartfromthereal-worldapplicationsabove,bothProblems1and2generalizeanumberofwell-studieddiscreteoptimizationproblems.ForexampletheSubmodularSetCoverproblem(henceforthSSC)[29]occursasaspecialcaseofProblem1,withfbeingmodularandgissubmodular.SimilarlytheSubmodularCostKnapsackproblem(henceforthSK)[28]isaspecialcaseofproblem2againwhenfismodularandgsubmodular.BoththeseproblemssubsumetheSetCoverandMaxk-Coverproblems[3].Whenbothfandgaremodular,Problems1and2arecalledknapsackproblems[16].Thefollowingaresomeofourcontributions.WeshowthatProblems1and2areintimatelyconnected,inthatanyapproximationalgorithmforeitherproblemcanbeusedtoprovideguaranteesfortheotherproblemaswell.Wethenprovideaframeworkofcombinatorialalgorithmsbasedonoptimizing,sometimesiteratively,subproblemsthatareeasytosolve.Thesesubproblemsareobtainedbycomputingeitherupperorlowerboundapproximationsofthecostfunctionsorconstrainingfunctions.WealsoshowthatmanycombinatorialalgorithmslikethegreedyalgorithmforSK[28]andSSC[29]alsobelongtothisframeworkandprovidetherstconstant-factorbi-criterionapproximationalgorithmforSSC[29]andhencethegeneralsetcoverproblem[3].Wethenshowhowwithsuitablechoicesofapproximatefunctions,wecanobtainanumberofboundedapproximationguaranteesandshowthehardnessforProblems1and2,whichinfactmatchsomeofourapproximationguarantees.Ourguaranteesandhardnessresultsdependonthecurvatureofthesubmodularfunctions[2].Weobserveastrongasymmetryintheresultsthatthefactorschange2 polynomiallybasedonthecurvatureoffbutonlybyaconstant-factorwiththecurvatureofg,hencemakingtheSKandSSCmucheasiercomparedtoSCSKandSCSC.2BackgroundandMainIdeasWerstintroduceseveralkeyconceptsusedthroughoutthepaper.Thispaperincludesonlythemainresultsandwedeferalltheproofsandadditionaldiscussionstotheextendedversion[11].Givenasubmodularfunctionf,wedenethetotalcurvature,fas2:f=1�minj2Vf(jjVnj) f(j)[2].Intuitively,thecurvature0f1measuresthedistanceofffrommodularityandf=0ifandonlyiffismodular(oradditive,i.e.,f(X)=Pj2Xf(j)).Anumberofapprox-imationguaranteesinthecontextofsubmodularoptimizationhavebeenrenedviathecur-vatureofthesubmodularfunction[2,13,12].Inthispaper,weshallwitnesstheroleofcurvaturealsoindeterminingtheapproximationsandthehardnessofproblems1and2. Algorithm1:GeneralalgorithmicframeworktoaddressbothProblems1and2 1:fort=1;2;


;Tdo2:Choosesurrogatefunctions^ftand^gtforfandgrespectively,tightatXt�1.3:ObtainXtastheoptimizerofProblem1or2with^ftand^gtinsteadoffandg.4:endfor Themainideaofthispaperisaframeworkofalgorithmsbasedonchoosingappropriatesur-rogatefunctionsforfandgtooptimizeover.ThisframeworkisrepresentedinAlgorithm1.Wewouldliketochoosesurrogatefunctions^ftand^gtsuchthatusingthem,Problems1and2becomeeasier.Ifthealgorithmisjustsinglestage(notiterative),werepresentthesurrogatesas^fand^g.Thesurrogatefunctionsweconsiderinthispaperareintheformsofbounds(upperorlower)andapproximations.Modularlowerbounds:Akintoconvexfunctions,submodularfunctionshavetightmodularlowerbounds.Theseboundsarerelatedtothesubdifferential@f(Y)ofthesubmodularsetfunctionfatasetYV[4].DenoteasubgradientatYbyhY2@f(Y).Theextremepointsof@f(Y)maybecomputedviaagreedyalgorithm:LetbeapermutationofVthatassignstheelementsinYtotherstjYjpositions((i)2YifandonlyifijYj).EachsuchpermutationdenesachainwithelementsS0=;,Si=f(1);(2);:::;(i)gandSjYj=Y.ThischaindenesanextremepointhYof@f(Y)withentrieshY((i))=f(Si)�f(Si�1).Denedasabove,hYformsalowerboundoff,tightatY—i.e.,hY(X)=Pj2XhY(j)f(X);8XVandhY(Y)=f(Y).Modularupperbounds:Wecanalsodenesuperdifferentials@f(Y)ofasubmodularfunction[14,10]atY.Itispossible,moreover,toprovidespecicsupergradients[10,13]thatdenethefollowingtwomodularupperbounds(whenreferringeitherone,weusemfX):mfX;1(Y),f(X)�Xj2XnYf(jjXnj)+Xj2YnXf(jj;) , mfX;2(Y),f(X)�Xj2XnYf(jjVnj)+Xj2YnXf(jjX):ThenmfX;1(Y)f(Y)andmfX;2(Y)f(Y);8YVandmfX;1(X)=mfX;2(X)=f(X).MMalgorithmsusingupper/lowerbounds:UsingthemodularupperandlowerboundsaboveinAlgorithm1,provideaclassofMajorization-Minimization(MM)algorithms,akintothealgorithmsproposedin[13]forsubmodularoptimizationandin[25,9]forDSoptimization(Problem0above).AnappropriatechoiceoftheboundsensuresthatthealgorithmalwaysimprovestheobjectivevaluesforProblems1and2.Inparticular,choosing^ftasamodularupperboundofftightatXt,or^gtasamodularlowerboundofgtightatXt,orboth,ensuresthattheobjectivevalueofProblems1and2alwaysimprovesateveryiterationaslongasthecorrespondingsurrogateproblemcanbesolvedexactly.Unfortunately,Problems1and2areNP-hardevenifforg(orboth)aremodular[3],andthereforethesurrogateproblemsthemselvescannotbesolvedexactly.Fortunately,thesurrogateproblemsareoftenmucheasierthantheoriginalonesandcanadmitlogorconstant-factorguarantees.Inpractice,moreover,thesefactorsarealmost1.Furthermore,withasimplemodicationoftheiterativeprocedureofAlgorithm1,wecanguaranteeimprovementateveryiteration[11].Whatisalsofortunateandperhapssurprising,asweshowinthispaperbelow,isthatunlikethecaseofDSoptimization(wheretheproblemisinapproximableingeneral[9]),theconstrainedformsofoptimization(Problems1and2)dohaveapproximationguarantees. 2Wecanassume,w.l.o.gthatf(j)�0;g(j)�0;8j2V3 EllipsoidalApproximation:Wealsoconsiderellipsoidalapproximations(EA)off.ThemainresultofGoemanset.al[6]istoprovideanalgorithmbasedonapproximatingthesubmodularpolyhedronbyanellipsoid.Theyshowthatforanypolymatroidfunctionf,onecancomputeanapproximationoftheformp wf(X)foracertainmodularweightvectorwf2RV,suchthatp wf(X)f(X)O(p nlogn)p wf(X);8XV.Asimpletrickthenprovidesacurvature-dependentapproximation[12]—wedenethef-curve-normalizedversionoffasfollows:f(X),f(X)�(1�f)Pj2Xf(j)=f.Then,thesubmodularfunctionfea(X)=fp wf(X)+(1�f)Pj2Xf(j)satises[12]:fea(X)f(X)Op nlogn 1+(p nlogn�1)(1�f)fea(X);8XV(2)feaismultiplicativelyboundedbyfbyafactordependingonp nandthecurvature.WeshallusetheresultaboveinprovidingapproximationboundsforProblems1and2.Inparticular,thesurrogatefunctions^for^ginAlgorithm1canbetheellipsoidalapproximationsabove,andthemultiplicativeboundstransformintoapproximationguaranteesfortheseproblems.3RelationbetweenSCSCandSCSKInthissection,weshowapreciserelationshipbetweenProblems1and2.Fromtheformu-lationofProblems1and2,itisclearthattheseproblemsaredualsofeachother.Indeed,inthissectionweshowthattheproblemsarepolynomiallytransformableintoeachother. Algorithm2:Approx.algorithmforSCSKus-inganapproximationalgorithmforSCSC. 1:Input:AnSCSKinstancewithbudgetb,an[;]approx.algo.forSCSC,&2[0;1).2:Output:[(1�);]approx.forSCSK.3:c g(V);^Xc V.4:whilef(^Xc)�bdo5:c (1�)c6:^Xc [;]approx.forSCSCusingc.7:endwhile Algorithm3:Approx.algorithmforSCSCus-inganapproximationalgorithmforSCSK. 1:Input:AnSCSCinstancewithcoverc,an[;]approx.algo.forSCSK,&�0.2:Output:[(1+);]approx.forSCSC.3:b argminjf(j);^Xb ;.4:whileg(^Xb)cdo5:b (1+)b6:^Xb [;]approx.forSCSKusingb.7:endwhile Werstintroducethenotionofbicriteriaalgorithms.Analgorithmisa[;]bi-criterionalgorithmforProblem1ifitisguaranteedtoobtainasetXsuchthatf(X)f(X)(approximateoptimality)andg(X)c0=c(approximatefeasibility),whereXisanoptimizerofProblem1.Similarly,analgorithmisa[;]bi-criterionalgorithmforProblem2ifitisguaranteedtoobtainasetXsuchthatg(X)g(X)andf(X)b0=b,whereXistheoptimizerofProblem2.Inabi-criterionalgo-rithmforProblems1and2,typically1and1.Anon-bicriterionalgorithmforProblem1iswhen=1andanon-bicriterionalgorithmforProblem2iswhen=1.Algorithms2and3providetheschematicsforusinganapproximationalgorithmforoneoftheproblemsforsolvingtheother.Theorem3.1.Algorithm2isguaranteedtondaset^Xcwhichisa[(1�);]approximationofSCSKinatmostlog1=(1�)[g(V)=minjg(j)]callstothe[;]approximatealgorithmforSCSC.Similarly,Algorithm3isguaranteedtondaset^Xbwhichisa[(1+);]approximationofSCSCinlog1+[f(V)=minjf(j)]callstoa[;]approximatealgorithmforSCSK.Theorem3.1impliesthatthecomplexityofProblems1and2areidentical,andasolutiontooneofthemprovidesasolutiontotheother.Furthermore,asexpected,thehardnessofProblems1and2arealsoalmostidentical.Whenfandgarepolymatroidfunctions,moreover,wecanprovideboundedap-proximationguaranteesforbothproblems,asshowninthenextsection.AlternativelywecanalsodoabinarysearchinsteadofalinearsearchtotransformProblems1and2.ThisessentiallyturnsthefactorofO(1=)intoO(log1=).Duetolackofspace,wedeferthisdiscussiontotheextendedversion[11].4 4ApproximationAlgorithmsWeconsiderseveralalgorithmsforProblems1and2,whichcanallbecharacterizedbytheframeworkofAlgorithm1,usingthesurrogatefunctionsoftheformofupper/lowerboundsorapproximations.4.1ApproximationAlgorithmsforSCSCWerstdescribeourapproximationalgorithmsdesignedspecicallyforSCSC,leavingtox4.2thepresentationofouralgorithmsslatedforSCSK.Werstinvestigateaspecialcase,thesubmodularsetcover(SSC),andthenprovidetwoalgorithms,oneofthem(ISSC)isverypracticalwithaweakertheoreticalguarantee,andanotherone(EASSC)whichisslowbuthasthetightestguarantee.SubmodularSetCover(SSC):WestartbyconsideringaclassicalspecialcaseofSCSC(Problem1)wherefisalreadyamodularfunctionandgisasubmodularfunction.Thisproblemoccursnaturallyinanumberofproblemsrelatedtoactive/onlinelearning[7]andsummarization[21,22].ThisproblemwasrstinvestigatedbyWolsey[29],whereinheshowedthatasimplegreedyalgorithmachievesbounded(infact,log-factor)approximationguarantees.WeshowthatthisgreedyalgorithmcannaturallybeviewedintheframeworkofourAlgorithm1bychoosingappropriatesurrogatefunctions^ftand^gt.Theideaistousethemodularfunctionfasitsownsurrogate^ftandchoosethefunction^gtasamodularlowerboundofg.Akintotheframeworkofalgorithmsin[13],thecrucialfactoristhechoiceofthelowerbound(orsubgradient).Denethegreedysubgradientas:(i)2argminf(j) g(jjSi�1)j=2Si�1;g(Si�1[j)c:(3)Oncewereachaniwheretheconstraintg(Si�1[j)ccannolongerbesatisedbyanyj=2Si�1,wechoosetheremainingelementsforarbitrarily.Letthecorrespondingsubgradientbereferredtoash.Thenwehavethefollowinglemma,whichisanextensionof[29],andwhichisasimplerdescriptionoftheresultstatedformallyin[11].Lemma4.1.ThegreedyalgorithmforSSC[29]canbeseenasaninstanceofAlgorithm1bychoosingthesurrogatefunction^fasfand^gash(withdenedinEqn.(3)).Whengisintegral,theguaranteeofthegreedyalgorithmisHg,H(maxjg(j)),whereH(d)=Pdi=11 i[29](henceforthwewilluseHgforthisquantity).Thisfactoristightuptolower-orderterms[3].Furthermore,sincethisalgorithmdirectlysolvesSSC,wecallittheprimalgreedy.WecouldalsosolveSSCbylookingatitsdual,whichisSK[28].AlthoughSSCdoesnotadmitanyconstant-factorapproximationalgorithms[3],wecanobtainaconstant-factorbi-criterionguarantee:Lemma4.2.UsingthegreedyalgorithmforSK[28]astheapproximationoracleinAlgorithm3providesa[1+;1�e�1]bi-criterionapproximationalgorithmforSSC,forany�0.Wecallthisthedualgreedy.Thisresultfollowsimmediatelyfromtheguaranteeofthesubmodularcostknapsackproblem[28]andTheorem3.1.Weremarkthatwecanalsouseasimplerversionofthegreedyiterationateveryiteration[21,17]andweobtainaguaranteeof(1+;1=2(1�e�1)).Inpractice,however,boththesefactorsarealmost1andhencethesimplevariantofthegreedyalgorithmsufces.IteratedSubmodularSetCover(ISSC):WenextinvestigateanalgorithmforthegeneralSCSCproblemwhenbothfandgaresubmodular.Theideahereistoiterativelysolvethesubmodularsetcoverproblemwhichcanbedonebyreplacingfbyamodularupperboundateveryiteration.Inparticular,thiscanbeseenasavariantofAlgorithm1,wherewestartwithX0=;andchoose^ft(X)=mfXt(X)ateveryiteration.ThesurrogateproblemateachiterationbecomesminfmfXt(X)jg(X)cg.Hence,eachiterationisaninstanceofSSCandcanbesolvednearlyoptimallyusingthegreedyalgorithm.WecancontinuethisalgorithmforTiterationsoruntilconvergence.Ananalysisverysimilartotheonesin[9,13]willrevealpolynomialtimeconvergence.Sinceeachiterationisonlythegreedyalgorithm,thisapproachisalsohighlypracticalandscalable.Theorem4.3.ISSCobtainsanapproximationfactorofKgHg 1+(Kg�1)(1�f)n 1+(n�1)(1�f)HgwhereKg=1+maxfjXj:g(X)cgandHgistheapproximationfactorofthesubmodularsetcoverusingg.5 Fromtheabove,itisclearthatKgn.NoticealsothatHgisessentiallyalog-factor.Wealsoseeaninterestingeffectofthecurvaturefoff.Whenfismodular(f=0),werecovertheapproximationguaranteeofthesubmodularsetcoverproblem.Similarly,whenfhasrestrictedcurvature,theguaranteescanbemuchbetter.Moreover,theapproximationguaranteealreadyholdsaftertherstiteration,soadditionaliterationscanonlyfurtherimprovetheobjective.EllipsoidalApproximationbasedSubmodularSetCover(EASSC):Inthissetting,weusetheellipsoidalapproximationdiscussedinx2.Wecancomputethef-curve-normalizedversionoff(f,seex2),andthencomputeitsellipsoidalapproximationp wf.Wethendenethefunction^f(X)=fea(X)=fp wf(X)+(1�f)Pj2Xf(j)andusethisasthesurrogatefunction^fforf.Wechoose^gasgitself.Thesurrogateproblembecomes:min8:fq wf(X)+(1�f)Xj2Xf(j)g(X)c9=;:(4)Whilefunction^f(X)=fea(X)isnotmodular,itisaweightedsumofaconcaveovermodularfunctionandamodularfunction.Fortunately,wecanusetheresultfrom[26],wheretheyshowthatanyfunctionoftheformofp w1(X)+w2(X)canbeoptimizedoveranypolytopePwithanapproximationfactorof(1+)forany�0,whereistheapproximationfactorofoptimizingamodularfunctionoverP.Thecomplexityofthisalgorithmispolynomialinnand1 .Weusetheiralgorithmtominimizefea(X)overthesubmodularsetcoverconstraintandhencewecallthisalgorithmEASSC.Theorem4.4.EASSCobtainsaguaranteeofO(p nlognHg 1+(p nlogn�1)(1�f)),whereHgistheapproxima-tionguaranteeofthesetcoverproblem.Ifthefunctionfhasf=1,wecanuseamuchsimpleralgorithm.Inparticular,wecanminimize(fea(X))2=wf(X)ateveryiteration,givingasurrogateproblemoftheformminfwf(X)jg(X)cg.ThisisdirectlyaninstanceofSSC,andincontrasttoEASSC,wejustneedtosolveSSConce.WecallthisalgorithmEASSCc.Corollary4.5.EASSCcobtainsanapproximationguaranteeofO(p nlognp Hg).4.2ApproximationAlgorithmsforSCSKInthissection,wedescribeourapproximationalgorithmsforSCSK.Wenotethedualnatureofthealgorithmsinthiscurrentsectiontothosegiveninx4.1.Werstinvestigateaspecialcase,thesubmodularknapsack(SK),andthenprovidethreealgorithms,twoofthem(GrandISK)beingpracticalwithslightlyweakertheoreticalguarantee,andanotherone(EASK)whichisnotscalablebuthasthetightestguarantee.SubmodularCostKnapsack(SK):WestartwithaspecialcaseofSCSK(Problem2),wherefisamodularfunctionandgisasubmodularfunction.Inthiscase,SCSKturnsintotheSKproblemforwhichthegreedyalgorithmwithpartialenumerationprovidesa1�e�1approximation[28].ThegreedyalgorithmcanbeseenasaninstanceofAlgorithm1with^gbeingthemodularlowerboundofgand^fbeingf,whichisalreadymodular.Inparticular,dene:(i)2argmaxg(jjSi�1) f(j)j=2Si�1;f(Si�1[fjg)b;(5)wheretheremainingelementsarechosenarbitrarily.Thefollowingisaninformaldescriptionoftheresultdescribedformallyin[11].Lemma4.6.Choosingthesurrogatefunction^fasfand^gash(withdenedineqn(5))inAlgorithm1withappropriateinitializationobtainsaguaranteeof1�1=eforSK.Greedy(Gr):AsimilargreedyalgorithmcanprovideapproximationguaranteesforthegeneralSCSKproblem,withsubmodularfandg.Unliketheknapsackcasein(5),however,atiterationiwechooseanelementj=2Si�1:f(Si�1[fjg)bwhichmaximizesg(jjSi�1).IntermsofAlgorithm1,thisisanalogoustochoosingapermutation,suchthat:(i)2argmaxfg(jjSi�1)jj=2Si�1;f(Si�1[fjg)bg:(6)6 Theorem4.7.ThegreedyalgorithmforSCSKobtainsanapprox.factorof1 g(1�(Kf�g Kf)kf)1 Kf,whereKf=maxfjXj:f(X)bgandkf=minfjXj:f(X)b&8j2X;f(X[j)�bg.Intheworstcase,kf=1andKf=n,inwhichcasetheguaranteeis1=n.Theboundabovefollowsfromasimpleobservationthattheconstraintff(X)bgisdown-monotoneforamonotonefunctionf.However,inthisvariant,wedonotuseanyspecicinformationaboutf.Inparticularitholdsformaximizingasubmodularfunctiongoveranydownmonotoneconstraint[2].Henceitisconceivablethatanalgorithmthatusesbothfandgtochoosethenextelementcouldprovidebetterbounds.Wedonot,however,currentlyhavetheanalysisforthis.IteratedSubmodularCostKnapsack(ISK):Here,wechoose^ft(X)asamodularupperboundoff,tightatXt.Let^gt=g.Thenateveryiteration,wesolvemaxfg(X)jmfXt(X)bg,whichisasubmodularmaximizationproblemsubjecttoaknapsackconstraint(SK).Asmentionedabove,greedycansolvethisnearlyoptimally.WestartwithX0=;,choose^f0(X)=Pj2Xf(j)andtheniterativelycontinuethisprocessuntilconvergence(notethatthisisanascentalgorithm).Wehavethefollowingtheoreticalguarantee:Theorem4.8.AlgorithmISKobtainsasetXtsuchthatg(Xt)(1�e�1)g(~X),where~Xistheopti-malsolutionofmaxng(X)jf(X)b(1+(Kf�1)(1�f) KfoandwhereKf=maxfjXj:f(X)bg.Itisworthpointingoutthattheaboveboundholdsevenaftertherstiterationofthealgorithm.ItisinterestingtonotethesimilaritybetweenthisapproachandISSC.Noticethattheguaranteeaboveisnotastandardbi-criterionapproximation.Weshowintheextendedversion[11]thatwithasimpletransformation,wecanobtainabicriterionguarantee.EllipsoidalApproximationbasedSubmodularCostKnapsack(EASK):ChoosingtheEllip-soidalApproximationfeaoffasasurrogatefunction,weobtainasimplerproblem:max8:g(X)fq wf(X)+(1�f)Xj2Xf(j)b9=;:(7)Inordertosolvethisproblem,welookatitsdualproblem(i.e.,Eqn.(4))anduseAlgorithm2toconverttheguarantees.WecallthisprocedureEASK.WethenobtainguaranteesverysimilartoTheorem4.4.Lemma4.9.EASKobtainsaguaranteeofh1+;O(p nlognHg 1+(p nlogn�1)(1�f))i.Inthecasewhenthesubmodularfunctionhasacurvaturef=1,wecanactuallyprovideasimpleralgorithmwithoutneedingtousetheconversionalgorithm(Algorithm2).Inthiscase,wecandirectlychoosetheellipsoidalapproximationoffasp wf(X)andsolvethesurrogateproblem:maxfg(X):wf(X)b2g:Thissurrogateproblemisasubmodularcostknapsackproblem,whichwecansolveusingthegreedyalgorithm.WecallthisalgorithmEASKc.Thisguaranteeistightuptologfactorsiff=1.Corollary4.10.AlgorithmEASKcobtainsabi-criterionguaranteeof[1�e�1;O(p nlogn)].4.3ExtensionsbeyondSCSCandSCSKSCSCandSCSKcaninfactbeextendedtomoreexibleandcomplicatedconstraintswhichcanarisenaturallyinmanyapplications[18,8].Theseincludemultiplecoveringandknapsackconstraints–i.e.,minff(X)jgi(X)ci;i=1;2;


kgandmaxfg(X)jfi(X)bi;i=1;2;


kg,androbustoptimizationproblemslikemaxfminigi(X)jf(X)bg,wherethefunctionsf;g;fi'sandgi'saresubmodular.WealsoconsiderSCSCandSCSKwithnon-monotonesubmodularfunctions.Duetolackofspace,wedeferthesediscussionstotheextendedversionofthispaper[11].4.4HardnessInthissection,weprovidethehardnessforProblems1and2.Thelowerboundsservetoshowthattheapproximationfactorsabovearealmosttight.7 Theorem4.11.Forany�0,thereexistssubmodularfunctionswithcurvaturesuchthatnopolynomialtimealgorithmforProblems1and2achievesabi-criterionfactorbetterthan =n1=2� 1+(n1=2��1)(1�)forany�0.TheaboveresultshowsthatEASSCandEASKmeettheboundsabovetologfactors.Weseeaninterestingcurvature-dependentinuenceonthehardness.Wealsoseethisphenomenonintheapproximationguaranteesofouralgorithms.Inparticular,assoonasfbecomesmodular,theproblembecomeseasy,evenwhengissubmodular.Thisisnotsurprisingsincethesubmodularsetcoverproblemandthesubmodularcostknapsackproblembothhaveconstantfactorguarantees.5ExperimentsInthissection,weempiricallycomparetheperformanceofthevariousalgorithmsdiscussedinthispaper.Wearemotivatedbythespeechdatasubsetselectionapplication[20,23]withthesubmodularfunctionfencouraginglimitedvocabularywhilegtriestoachieveacousticvariability.Anaturalchoiceofthefunctionfisafunctionoftheformj�(X)j,where�(X)istheneighborhoodfunctiononabipartitegraphconstructedbetweentheutterancesandthewords[23].Forthecoveragefunctiong,weusetwotypesofcoverage:oneisafacilitylocationfunctiong1(X)=Pi2Vmaxj2Xsijwhiletheotherisasaturatedsumfunctiong2(X)=Pi2VminfPj2Xsij;Pj2Vsijg.BoththesefunctionsaredenedintermsofasimilaritymatrixS=fsijgi;j2V,whichwedeneontheTIMITcorpus[5],usingthestringkernelmetric[27]forsimilarity.Sincesomeofouralgorithms,liketheEllipsoidalApproximations,arecomputationallyintensive,werestrictourselvesto50utterances. Figure1:Comparisonofthealgorithmsinthetext.WecompareourdifferentalgorithmsonProblems1and2withfbeingthebipartiteneighborhoodandgbeingthefacilitylocationandsaturatedsumrespectively.Furthermore,inourexperiments,weobservethattheneigh-borhoodfunctionfhasacurvaturef=1.Thus,itsufcestousethesimplerversionsofalgorithmEA(i.e.,algorithmEASSCcandEASKc).TheresultsareshowninFigure1.Weobservethatonthereal-worldinstances,allouralgorithmsperformalmostcomparably.Thisimplies,moreover,thattheiterativevariants,viz.Gr,ISSCandISK,performcomparablytothemorecomplicatedEA-basedones,althoughEASSCandEASKhavebettertheoreticalguarantees.Wealsocompareagainstabaselineofselectingrandomsets(ofvaryingcardinality),andweseethatouralgorithmsallperformmuchbetter.Intermsoftherunningtime,computingtheEllipsoidalApproximationforj�(X)jwithjVj=50takesabout5hourswhilealltheiterativevariants(i.e.,Gr,ISSCandISK)takelessthanasecond.Thisdifferenceismuchmoreprominentonlargerinstances(forexamplejVj=500).6DiscussionsInthispaper,weproposeaunifyingframeworkforproblems1and2basedonsuitablesurrogatefunctions.Weprovideanumberofiterativealgorithmswhichareverypracticalandscalable(likeGr,ISKandISSC),andalsoalgorithmslikeEASSCandEASK,whichthoughmoreintensive,obtaintightapproximationbounds.Finally,weempiricallycompareouralgorithms,andshowthattheiterativealgorithmscompeteempiricallywiththemorecomplicatedandtheoreticallybetterapproximationalgorithms.Forfuturework,wewouldliketoempiricallyevaluateouralgorithmsonmanyoftherealworldproblemsdescribedabove,particularlythelimitedvocabularydatasubsetselectionapplicationforspeechcorpora,andthemachinetranslationapplication.Acknowledgments:SpecialthankstoKaiWeiandStefanieJegelkafordiscussions,toBethanyHerwaldtforgoingthroughanearlydraftofthismanuscriptandtotheanonymousreviewersforusefulreviews.ThismaterialisbaseduponworksupportedbytheNationalScienceFoundationunderGrantNo.(IIS-1162606),aGoogleandaMicrosoftaward,andbytheIntelScienceandTechnologyCenterforPervasiveComputing.8 20 40 60 80 100 0 100 200 300 f(X)g(X)Saturated Sum/ Bipartite Neighbor ISSC EASSCc ISK Gr EASKc Random 0 100 200 250 0 10 20 30 40 50 f(X)g(X)Fac. Location/ Bipartite Neighbor. ISSC EASSCc ISK Gr EASKc Random References[1]A.Atamt¨urkandV.Narayanan.Thesubmodularknapsackpolytope.DiscreteOptimization,2009.[2]M.ConfortiandG.Cornuejols.Submodularsetfunctions,matroidsandthegreedyalgorithm:tightworst-caseboundsandsomegeneralizationsoftheRado-Edmondstheorem.DiscreteAppliedMathematics,7(3):251–274,1984.[3]U.Feige.Athresholdoflnnforapproximatingsetcover.JournaloftheACM(JACM),1998.[4]S.Fujishige.Submodularfunctionsandoptimization,volume58.ElsevierScience,2005.[5]J.Garofolo,F.Lamel,L.,J.W.,Fiscus,D.Pallet,andN.Dahlgren.Timit,acoustic-phoneticcontinuousspeechcorpus.InDARPA,1993.[6]M.Goemans,N.Harvey,S.Iwata,andV.Mirrokni.Approximatingsubmodularfunctionseverywhere.InSODA,pages535–544,2009.[7]A.GuilloryandJ.Bilmes.Interactivesubmodularsetcover.InICML,2010.[8]A.GuilloryandJ.Bilmes.Simultaneouslearningandcoveringwithadversarialnoise.InICML,2011.[9]R.IyerandJ.Bilmes.Algorithmsforapproximateminimizationofthedifferencebetweensubmodularfunctions,withapplications.InUAI,2012.[10]R.IyerandJ.Bilmes.ThesubmodularBregmanandLov´asz-Bregmandivergenceswithapplications.InNIPS,2012.[11]R.IyerandJ.Bilmes.SubmodularOptimizationwithSubmodularCoverandSubmodularKnapsackConstraints:Extendedarxivversion,2013.[12]R.Iyer,S.Jegelka,andJ.Bilmes.CurvatureandOptimalAlgorithmsforLearningandMinimizingSubmodularFunctions.InNIPS,2013.[13]R.Iyer,S.Jegelka,andJ.Bilmes.Fastsemidifferentialbasedsubmodularfunctionoptimization.InICML,2013.[14]S.JegelkaandJ.A.Bilmes.Submodularitybeyondsubmodularenergies:couplingedgesingraphcuts.InCVPR,2011.[15]Y.KawaharaandT.Washio.Prismaticalgorithmfordiscretedcprogrammingproblems.InNIPS,2011.[16]H.Kellerer,U.Pferschy,andD.Pisinger.Knapsackproblems.SpringerVerlag,2004.[17]A.KrauseandC.Guestrin.Anoteonthebudgetedmaximizationonsubmodularfunctions.TechnicalReportCMU-CALD-05-103,CarnegieMellonUniversity,2005.[18]A.Krause,B.McMahan,C.Guestrin,andA.Gupta.Robustsubmodularobservationselection.JournalofMachineLearningResearch(JMLR),9:2761–2801,2008.[19]A.Krause,A.Singh,andC.Guestrin.Near-optimalsensorplacementsinGaussianprocesses:Theory,efcientalgorithmsandempiricalstudies.JMLR,9:235–284,2008.[20]H.LinandJ.Bilmes.Howtoselectagoodtraining-datasubsetfortranscription:Submodularactiveselectionforsequences.InInterspeech,2009.[21]H.LinandJ.Bilmes.Multi-documentsummarizationviabudgetedmaximizationofsubmodularfunctions.InNAACL,2010.[22]H.LinandJ.Bilmes.Aclassofsubmodularfunctionsfordocumentsummarization.InThe49thAnnualMeetingoftheAssociationforComputationalLinguistics:HumanLanguageTechnologies(ACL/HLT-2011),Portland,OR,June2011.[23]H.LinandJ.Bilmes.Optimalselectionoflimitedvocabularyspeechcorpora.InInterspeech,2011.[24]R.C.MooreandW.Lewis.Intelligentselectionoflanguagemodeltrainingdata.InProceedingsoftheACL2010ConferenceShortPapers,pages220–224.AssociationforComputationalLinguistics,2010.[25]M.NarasimhanandJ.Bilmes.Asubmodular-supermodularprocedurewithapplicationstodiscriminativestructurelearning.InUAI,2005.[26]E.Nikolova.Approximationalgorithmsforofinerisk-aversecombinatorialoptimization,2010.[27]J.RousuandJ.Shawe-Taylor.Efcientcomputationofgappedsubstringkernelsonlargealphabets.JournalofMachineLearningResearch,6(2):1323,2006.[28]M.Sviridenko.Anoteonmaximizingasubmodularsetfunctionsubjecttoaknapsackconstraint.OperationsResearchLetters,32(1):41–43,2004.[29]L.A.Wolsey.Ananalysisofthegreedyalgorithmforthesubmodularsetcoveringproblem.Combinatorica,2(4):385–393,1982.9