huangintelcom Anthony D Joseph UC Berkeley adjcsberkeleyedu Blaine Nelson University of T57596bingen blainenelsonwsiiuni tuebingende Benjamin I P Rubinstein Microsoft Research benrubinsteinmicrosoftcom J D Tygar UC Berkeley tygarcsberkeleyedu ABSTRAC ID: 54368
Download Pdf The PPT/PDF document "Adversarial Machine Learning Ling Huang ..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
43 AdversarialMachineLearningingHuangIntelLabsBerkeleyAnthonyD.JosephUCBerkeleyadj@cs.berkeley.eduBlaineNelsonUniversityofTübingenblaine.nelson@wsii.uni-BenjaminI.P.RubinsteinMicrosoftResearchben.rubinstein@microsoft.comJ.D.TygarUCBerkeleytygar@cs.berkeley.eduABSTRACTInthispaper(expandedfromaninvitedtalkatAISEC2010),wediscussanemerging\feldofstudy:adversarialma-chinelearning|thestudyofe ectivemachinelearningtech-niquesagainstanadversarialopponent.Inthispaper,we:giveataxonomyforclassifyingattacksagainstonlinema- hispaperexpandsuponJ.D.Tygar'sinvitedtalkatAISec2010onAdversarialMachineLearningdescribingtheSecMLprojectatUCBerkeley,andincludesmaterialfrommanyofourcollaborators.Wekindlythankthisyear's 44 attackers.Inthispaper,we:giveataxonomyforclassifyingttacksagainstonlinemachinelearningalgorithms;discussapplication-speci\fcfactorsthatlimitanadversary'scapa-bilities;introducetwomodelsformodelinganadversary'scapabilities;explorethelimitsofanadversary'sknowledgeaboutthealgorithm,featurespace,training,andinputdata;explorevulnerabilitiesinmachinelearningalgorithms;dis-cusscountermeasuresagainstattacks;introducetheeva-sionchallenge;anddiscussprivacy-preservinglearningtech-niques.2.TAXONOMYInpriorwork[2],weintroducedaqualitativetaxonomyofattacksagainstamachinelearningsystem,whichhassincebeenextendedtoproposeaframeworkforquantitativelyevaluatingsecuritythreats[37].Ourtaxonomycategorizesanattackbasedonthefollowingthreeproperties:In\ruenceCausative-Causativeattacksalterthetrainingpro-cessthroughin\ruenceoverthetrainingdata.Exploratory-Exploratoryattacksdonotalterthetrainingprocessbutuseothertechniques,suchasprobingthedetector,todiscoverinformationaboutitoritstrainingdata.SecurityviolationIntegrity-Integrityattacksresultinintrusionpointsbeingclassi\fedasnormal(falsenegatives).Availability-Availabilityattackscausesomanyclas-si\fcationerrors,bothfalsenegativesandfalsepositives,thatthesystembecomese ectivelyun-usable.Privacy-Inaprivacyviolation,theadversaryob-tainsinformationfromthelearner,compromisingthesecrecyorprivacyofthesystem'susers.Speci\fcity(acontinuousspectrum)Targeted-Inatargetedattack,thefocusisonasin-gleorsmallsetoftargetpoints.Indiscriminate-Anindiscriminateadversaryhasamore\rexiblegoalthatinvolvesaverygeneralclassofpoints,suchas\anyfalsenegative."The\frstaxisdescribesthecapabilityoftheattacker:whether(a)theattackerhastheabilitytoin\ruencethetrainingdatathatisusedtoconstructtheclassi\fer(acaus-ativeattack)or(b)theattackerdoesnotin\ruencethelearnedclassi\fer,butcansendnewinstancestotheclassi\ferandpossiblyobserveitsdecisionsonthesecarefullycraftedin-stances(anexploratoryattack).Thesecondaxisindicatesthetypeofsecurityviolationtheattackercauses:either(a)allowingharmfulinstancestoslipthroughthe\flterasfalsenegatives(anintegrityolation);(b)creatingadenialofserviceeventinwhichbe-nigninstancesareincorrectly\flteredasfalsepositives(anavailabilityviolation);or(c)usingthe\flter'sresponsestoinfercon\fdentialinformationusedinthelearningprocessprivacyviolation).Privacyviolationswerenotoriginallycapturedin[2];andsincetheyarequalitativelydi erenttointegrityandavailabilityviolationswediscussthemsepa-ratelyinSection5.Thethirdaxisreferstohowspeci\fctheattacker'sin-tentionis:whether(a)theattackishighlytargetedtode-gradetheclassi\fer'sperformanceononeparticularinstanceor(b)theattackaimstocausetheclassi\fertofailinanindiscriminatefashiononabroadclassofinstances.Eachaxis,especiallythisone,canpotentiallybeaspectrumofchoices,butforsimplicity,wewillcategorizeattacksanddefensesintothesegroupings.2.1GameSpecicationandInterpretationWemodelsecurelearningsystemsasagamebetweenanattackerandadefender|theattackermanipulatesdatatomis-trainorevadealearningalgorithmchosenbythede-fendertothwarttheattacker'sobjective.Thisgamecanbeformalizedintermsofalearningalgorithm;andtheat-tacker'sdatacorruptionstrategies(train)and(eval).Theresultinggamecanbedescribedasfollows:1.DefenderChooselearningalgorithmforselectinghypothesesbasedonobserveddata2.AttackerChooseattackprocedures(train)and(eval)(potentiallywithknowledgeof3.Learning:Obtaindataset(train)withcontaminationfrom(train)Learnhypothesis:(train)4.Evaluation:Obtaindataset(eval)withcontaminationfrom(eval)Comparepredictions)toforeachdatapointx;y(eval)Thisgamestructuredescribesthefundamentalinterac-tionsbetweenthedefenderandattackerinchoosing(train)and(eval);thesestepsaredepictedinFigure1.Thede-fenderchoosestoselecthypothesesthatpredictwellre-gardlessof(train)and(eval),whiletheattackerchooses(train)and(eval)toproducepoorpredictions.Thechar-acteristicsspeci\fedbythetaxonomy'saxesfurtherspec-ifysomeaspectsofthisgame.Theinfluenceaxisdeter-minesthestructureofthegameandthelegalmovesthateachplayercanmake.Inexploratoryattacks,theprocedure(train)isnotusedinthegame,andtherebytheattackeronlyin\ruences(eval).Meanwhile,inthecausativegametheattackeralsohasindirectin\ruenceonthroughhischoiceof(train).Thespecificityandsecurityviola-axesofthetaxonomydeterminewhichinstancestheadversarywouldliketohavemisclassi\fedduringtheevalua-tionphase.Inanintegrityattack,theattackerdesiresfalsenegativesandthereforewilluse(train)and/or(eval)createordiscoverfalsenegatives,whereasinanavailabilitytheattackerwillalsotrytocreateorexploitfalsepositives.Finally,inatargetedattacktheattackeronlycaresaboutthepredictionsforasmallnumberofinstances,whileanindiscriminateattackercaresaboutpredictionforabroadrangeofinstances.3.CAUSATIVEATTACKSWe\frstdiscusscausativeattacks,inwhichtheadver-saryin\ruencesthetrainingdata.Mostimportantly,theadversaryinacausativeattackaltersthetrainingdatawithatransformation(train).Theattackermayhavevarious otethat,sincethegamedescribedhereisbatchtrain-ing,anadaptiveprocedure(train)isunnecessaryunlessthedistribution(train)isnon-stationaryinwhichcaseperiodicretrainingmaybedesirable.WereturntothisissueinSec-tion3.4.2. 45 PZ train) eval) val val H f Evaluator train) eval) Figure1:Diagramofanattackagainstalearningsystemwheresthedata'struedistribution,(train)(eval)areadversary'sattackprocedures,(train)(eval)arethetrainingandtestdatasets,isthelearningalgorithm,andisthehypothesisitlearnsfromthetrainingdata.Thehypothesisisevaluatedonthetestdatabycomparingitspredictiontothetruelabelforeachx;y(eval)typesofin\ruenceoverthisdata,rangingfromarbitrarycontroloversomefractionoftraininginstancestoabias-ingin\ruenceoversomeaspectofdataproduction;thesedetailsdependlargelyontheapplicationaswediscussbe-low.Regardless,theattackeruseshisin\ruencetomisleadthelearnercausingittoproduceabadclassi\fer,whichtheadversarysubsequentlyexploitsduringevaluation.Asinexploratoryattacks(seeSection4),acausativeadversaryalsocanuse(eval)toaltertheevaluationdata.Naturally,causativeadversarycancoordinate(train)and(eval)bestachievehisobjective,althoughinsomecausativeat-tacks,theadversarymayonlybeabletoexertcontroloverthetrainingdata(e.g.,theattackerinthecasestudybelowcannotcontroltheevaluationnon-spammessages).Severalresearchershavestudiedcausativeattacks.New-someetal.[52]constructedcausativeattacksagainstthePolygraphvirusdetector,apolymorphic-virusdetectorthatlearnsvirussignaturesusingbothaconjunctionlearnerandanaive-Bayes-likelearner.Thecorrelatedoutlierattack,causativeavailabilityattack,targetsthenaive-Bayes-likecomponentofthisdetectorbyaddingspuriousfeaturestopositivetraininginstances,causingthe\fltertoblockbenigntracwiththosefeatures.Newsomeetal.alsodevelopedcausativeintegrityattackagainsttheconjunctionlearnercomponentofPolygraph.Thisredherringattacksintro-ducesspuriousfeaturesalongwiththeirpayload;oncethelearnerhasconstructeditssignaturefromthis\raweddata,thespuriousfeaturesarediscardedtoavoidsubsequentde-tection.Venkataramanetal.alsopresentlowerboundsforlearningwormsignaturesbasedonredherringattacks[63].ChungandMokalsodevelopedallergyattacksagainsttheAutographwormsignaturegenerationsystem[12,13].Au-tographoperatesintwophases.First,itidenti\fesinfectednodesbasedonbehavioralpatterns,inparticularscanningbehavior.Second,itobservestracfromthesuspectnodesandinfersblockingrulesbasedonobservedpatterns.ChungandMokdescribeanattackthattargetstractoapartic-ularresource.Inthe\frstphase,anattacknodeconvincesAutographthatitisinfectedbyscanningthenetwork.Inthesecondphase,theattacknodesendscraftedpacketsmimick-ingtargetedtrac,causingAutographtolearnrulesthatblocklegitimateaccess;thus,thisisacausativeavailabil-ityattack.Wenowdescribetwoattacksthatwestudiedinpriorwork[47,48,57],andsubsequentlywediscussgen-eralperspectivesandguidelineswedevelopedforanalyzinglearningsystemsthataugmentourtaxonomy.CaseStudy:SpamBayesSpamBayesisacontent-basedstatisticalspam\flterthatclassi\fesemailusingtokencounts[55].SpamBayescom-putesaspamscoreforeachtokeninthetrainingcorpusbasedonitsoccurrenceinspamandnon-spamemails;thisscoreismotivatedasasmoothedestimateoftheposteriorprobabilitythatanemailcontainingthattokenisspam.The\fltercomputesamessage'soverallspamscorebasedontheassumptionthatthetokenscoresareindependentandthenitappliesFisher'smethod[26]forcombiningsig-ni\fcanceteststodeterminewhethertheemail'stokensaresucientlyindicativeofoneclassortheother.Themessagescoreiscomparedagainsttwothresholdstoselectthelabelspamhami.e.,non-spam),orunsureInanalyzingthevulnerabilitiesofSpamBayes,weweremotivatedbythetaxonomyofattacks.Knownreal-worldattacksthatspammersuseagainstdeployedspam\flterstendtobeexploratoryintegrityattacks:eitherthespam-merobfuscatestheespeciallyspam-likecontentofaspamemailorheincludescontentnotindicativeofspam.Bothtacticsaimtogetthemodi\fedmessageintothevictim'sinbox.Thiscategoryofattackhasbeenstudiedindetailintheliterature[16,39,40,65].However,wefoundthestudyofcausativeattacksmorecompelling.Inparticular,wedemonstratedacausativeavailabilityattackthatcreatedapowerfuldenialofservice[47];i.e.,ifmanylegitimatemes-sagesare\flteredbytheuser'sspam\flter,theuserislikelytodisablethe\flterandthereforeseethespammer'sadver-tisements.Alternatively,anunscrupulousbusinessownermaywishtousespam\flterdenialofservicetopreventacompetitorfromreceivingemailordersfrompotentialcus-tomers.Wedesignedtwotypesofcausativeavailabilityattacks,oneindiscriminateandtheothertargeted,againstSpam-Bayes.The\frstisanindiscriminatedictionaryattack,inwhichtheattackersendsattackmessagesthatcontainaverylargesetoftokens|theattack'sdictionary.Aftertrainingontheseattackmessages,thevictim'sspam\flterwillhaveahigherspamscoreforeverytokeninthedictionary.Asa 46 PercentcontroloftrainingsetPercenttesthammisclassi\fedOptimalUsenet(90k)Usenet(25k)Aspell PercentcontroloftrainingsetPercenttargethammisclassi\fed012345678910Figure2:E ectofthedictionaryandfocusedattacks.Weplotpercentofhamclassi\fedasspam(dashedlines)andasunsurespam(solidlines)againstpercentofthetrainingsetcontaminated.Left:Threedictionaryattacksonaninitialtrainingsetof10,000messages(50%spam).Weshowtheoptimalattack(black),theUsenetdictionaryattackwith90,000words(magenta),theUsenetdictionaryattackwith25,000words(blue),andtheAspelldictionaryattack(greenRight:Theaveragee ectof200focusedattacksontheirtargetswhentheattackerguesseseachtargettokenwith50%probability.Theinitialinboxcontains5000emails(50%spam).result,futurelegitimateemailismorelikelytobemarkedasspamsinceitwillcontainmanytokensfromthatlexicon(seeSection3.1).Forinstance,ifonlythevictim'slanguageisknownbytheattacker,theattackdictionarycanbethatlanguage'sentirelexicon.Are\fnementofthisattackinsteadusesatokensourcewithadistributionclosertothevictim'strueemaildistribution(seeSection3.3formoreonattackerknowledge).Usingthemostcommontokensmayallowtheattackertosendsmalleremailswithoutlosingmuche ec-tiveness.However,thereisaninherenttrade-o inchoosingtokens:raretokensarethemostvulnerablesincetheirscoreshavelesssupportandwillchangequicklywithfewattackemailsbuttheyarealsolesslikelytoappearinfuturemes-sages,dilutingtheirusefulness.Wediscussthistrade-o inSection3.1.3.Oursecondattackisatargetedattack|theattackerhassomeknowledgeofaspeci\fclegitimateemailhetargetstobeincorrectly\fltered.Iftheattackerhasexactknowledgeofthetargetemail,placingallofitstokensinattackemailspro-ducesanoptimaltargetedattack.Realistically,though,theattackeronlyhaspartialknowledgeaboutthetargetemailandcanguessonlysomeofitstokenstoincludeinattackemails(seeSection3.3).Wemodeledthisknowledgebylet-tingtheattackerknowacertainfractionoftokensfromthetargetemail,whichareincludedintheattackmessage.Theattackerconstructsattackemailthatcontainwordslikelytooccurinthetargetemail;i.e.,thetokensknownbytheat-tacker.Theattackemailmayalsoincludeadditionaltokensaddedbytheattackertoobfuscatetheattackmessage'sin-tent.WhenSpamBayestrainsontheresultingattackemail,thespamscoresofthetargetedtokensgenerallyincreaseandthetargetmessageismorelikelytobe\flteredasspam.Thisisthefocusedattack.Inourpriorwork[47],wepresentedresultsdemonstratingthee ectivenessofbothdictionaryandfocusedattacks,re-producedinFigure2.Thesegraphsdepictthee ectivenessofdictionaryattacks(leftmost\fgure)andfocusedattacks(rightmost\fgure)incausingmisclassi\fcationsintermsofthepercentofattackmessagesinthetrainingset(seeSec-tion3.2fordiscussionoftheattacker'scapabilities).NoticethatsinceSpamBayeshasthreepredictions(hamunsureandspam),weplotthepercentageofmessagesmisclassi\fedasspam(dashedlines)andeitherasunsureorspam.Asthe\fguredemonstrates,theseattacksarehighlye ectivewithasmallpercentageofcontamination.CaseStudy:AnomalousTrafcDetectionAdversariescanusecausativeattackstonotonlydisruptnormaluseractivitybutalsotoachieveevasionbycausingthedetectortohavemanyfalsenegativesthroughanin-tegrityattack.Indoingso,suchadversariescanreducetheriskthattheirmaliciousactivitiesaredetected.Herewere\rectonourstudyofthesubspaceanomalyde-tectionmethodsfordetectingnetwork-wideanomaliessuchasdenial-of-service(DoS)attacks.Inthisstudy,weshowedthatbyinjectingcraftycha intothenetworkduringtrain-ing,thedetectorcanbepoisonedsothatitisunabletoe ectivelydetectasubsequentDoSattack.Thedetectorweanalyzewas\frstproposedasamethodforidentifyingvolumeanomaliesinabackbonenetworkbasedonthePrin-cipalComponentAnalysis(PCA)dimensionalityreductiontechnique[36].Whiletheirsubspace-basedmethodisabletosuccessfullydetectDoSattacksinnetworktrac,itas-sumesthedetectoristrainedonnon-maliciousdata(inanunsupervisedfashionunderthesettingofanomalydetec-tion).Instead,weconsideredanadversarywhoknowsthatanISPisusingthesubspace-basedanomalydetectorandattemptstoevadeitbyproactivelypoisoningitstrainingdata.Thegoaloftheadversaryweconsideredwastocircumventdetectionbypoisoningthetrainingdata;i.e.,anintegritygoaltoincreasethedetector'sfalsenegativerate,whichcor- 47 0.00.20.40.60.81.0 Single Poisoning Period: Evading PCAMean chaff volumeEvasion success (FNR) 20%30%40%50% UninformedLocally-informed 10 05101520 0.00.20.40.60.81.0 Boiling Frog Poisoning: Evading PCAAttack duration (weeks)Evasion success (average test FNR) Growth rates1.011.02 Figure3:E ectofpoisoningattacksonthePCA-baseddetectortorLeft:EvasionsuccessofPCAversusrelativecha volumeunderSingle-TrainingPeriodpoisoningattacksusingthreecha methods:uninformed(dottedblackline)locally-informed(dashedblueline)andglobally-informed(solidredline).Right:EvasionsuccessofPCAunderBoilingFrogpoisoningattacksintermsoftheaverageFNRaftereachsuccessiveweekoflocally-informedpoisoningforfourdi erentpoisoningschedules(,aweeklygeometricincreaseinthesizeofthepoisoningbyfactors010205,and15respectively).Moreaggressiveschedules(,growthratesof0515)signi\fcantlyincreasetheFNRwithinafewweekswhilelessaggressiveschedulestakemanyweekstoachievethesameresultbutaremorestealthyindoingso.respondedtotheevasionsuccessrateoftheattacker'ssub-sequentDoSattack.Whentrainedonthispoisoneddata,thedetectorlearnedadistortedsetofprincipalcomponentsthatareunabletoe ectivelydiscerntheseDoSattacks|atargetedattack.BecausePCAestimatesthedata'sprinci-palsubspacesolelyonthecovarianceofthelinktrac,weexploredpoisoningschemesthataddcha (additionaltraf-\fc)intothenetworkalongthe\rowtargetedbytheattackertosystematicallyincreasethetargeted\row'svariance.Insodoing,theattackercausedtheestimatedsubspacetoun-dulyshifttowardthetarget\rowmakinglarge-volumeeventsalongthat\rowlessdetectable.Weconsideredthreegen-eralcategoriesofattacksbasedontheattacker'scapabilities:uninformedattacks,locally-informedattacks,andglobally-informedattacks.Eachofthesere\rectdi erentlevelsofknowledgeandresourcesavailabletotheattacker;seeSec-tion3.3and3.1formoredetaileddiscussionofthesemodels.Intheaboveattacks,cha wasdesignedtoimpactasin-gleperiod(oneweek)inthetrainingcycleofthedetector,butwealsoconsideredthepossibilityofepisodicpoisoningwhicharecarriedoutovermultipleweeksofretrainingthesubspacedetector;seeSection3.4.2forfurtherdiscussionofiterativeretraining.Multi-weekpoisoningstrategiesvarytheattackaccordingtothetimehorizonoverwhichtheyarecarriedout.Aswithsingle-weekattacks,duringeachweektheadversaryinsertscha alongthetarget\rowthrough-outthetrainingperiodaccordingtohispoisoningstrategy.However,inthemulti-weekattacktheadversaryincreasesthetotalamountofcha usedduringeachsubsequentweekaccordingtoapoisoningschedule.Thispoisonsthemodeloverseveralweeksbyinitiallyaddingsmallamountsofcha andincreasingthecha quantitieseachweeksothatthedetectorisgraduallyacclimatedtocha andfailstoade-quatelyidentifytheeventuallylargeamountofpoisoning.WecallthistypeofepisodicpoisoningtheBoilingFrogpoi-soningmethodafterthefolktalethatonecanboilafrogbyslowlyincreasingthewatertemperatureovertime.ThegoalofBoilingFrogpoisoningistograduallyrotatethenormalsubspace,injectinglowlevelsofcha relativetotheprevi-ousweek'straclevelssothatPCA'srejectionratesstaylowandalargeportionofthepresentweek'spoisonedtracmatrixistrainedon.AlthoughPCAisretrainedeachweek,thetrainingdatawillincludesomeeventsnotcaughtbythepreviousweek'sdetector.Thus,moremalicioustrainingdatawillaccumulateeachsuccessiveweekasthePCAsub-spaceisgraduallyshifted.ThisprocesscontinuesuntiltheweekoftheDoSattack,whentheadversarystopsinjectingcha andexecutestheirdesiredDoSInourpriorwork[57],weempiricallydemonstratedourattacksagainstPCAandinFigure3wereproducethere-sultsofourexperiments.Thesegraphsdepictthee ec-tivenessoftheSingle-TrainingPeriod(leftmost\fgure)andBoilingFrogattacks(rightmost\fgure)incausingfalsenega-tivesintermsofthepercentofaverageincreaseinthemeanlinkratesduetocha (seeSection3.2fordiscussionoftheattacker'scapabilities)andthelengthoftheattackdura-tion,respectively.FortheBoilingFrogattacks,weassumedthatthePCA-subspacemethodisretrainedonaweeklyba-sisusingthetracobservedinthepreviousweektoretrainthedetectoratthebeginningofthenewweek.Further,wesanitizethedatafromthepriorweekbeforeretrainingsothatalldetectedanomaliesareremovedfromthedata.AstheFigure3demonstrates,theseattackscausehighratesofmisdetectionwithrelativelysmallincreasesinthevolumeoftrac:e.g.,alocally-informedattackercanincreasehis 48 evasionsuccessto28%fromthebaselineof37%viaa10%averageincreaseinthemeanlinkratesduetocha .3.1IncorporatingApplication-SpecicFactorsOnetypeofadversariallimitationweconsiderarelimitsonhowanadversarycanalterdatapointsintermsofeachfeature.Featuresrepresentdi erentaspectsofthestateoftheworldandhavevariousdegreesofvulnerabilitytoattack.Somefeaturescanbearbitrarilychangedbytheadversary,butothersmayhavestochasticaspectsthattheadversarycannotcompletelycontrol,andsomefeaturesmaynotbealterableatall.Forinstance,insendinganemail,thead-versarycancompletelycontrolthecontentofthemessagebutcannotcompletelydeterminetheroutingofthemessageoritsarrivaltime.Further,thisadversaryhasnocontrolovermeta-informationthatisaddedtothemessagebymailrelayswhilethemessageisenroute.Providinganaccuratedescriptionoftheadversary'scontroloverthefeaturesisessentialandwediscussitfurtherinthefollowingsection.3.1.1DomainLimitationsThe\frstconsiderationarelimitationsontheadversarythatarisefromtheapplicationdomainitself.Theseincludelimitsonhowtheadversarycaninteractwiththeapplica-tionandwhatkindsofdataarerealisticfortheadversarytomanipulate.ForSpamBayes,theusualusagescenarioisthatthisdetectorisintendedtobeusedforindividualsorsmallorganizationsto\fltertheiremail.Insuchascenario,itisdiculttobelievethat\flter'suserswouldintentionallymislabelmessagestocorruptthe\flter.Thus,toobtainfaultylabels,theadversarywouldhavetorelyonbeingabletofooluserstomislabeltheirmessages.However,itisnoteasytotrickuserstomislabelspammessagesasnon-spam.Itismorerealisticthatausercouldbetrickedintomisla-belingnon-spammessagesasspam(e.g.,byscramblingthelegitimatemessageandaddingspam-likeelements).Fromthisobservation,weconcludedthatintegrityattacksrequir-ingmislabelednon-spamwereunrealisticandinsteadde-velopedouravailabilityattackusingtrickymessagesthatauserwouldprobablymislabelasspameveniftheywerenotatraditionaladvertisingpitch.Theintendeduse-caseforthePCA-basedanomalydetec-torwascompletelydi erent.Itwasintendedtooperatealongtheroutinginfrastructureofabackbonenetwork.Forthisscope,thedetectorisnotgivenanylabelinformationatall;alltrainingdataisunlabeledandanomalousdataisidenti\fedbythedetectoronlyafteritistrained.Thus,forthisapplication,theattackerhasabroadabilitytomanipu-latedatasothatbothintegrityandavailabilityattackswerepotentiallyfeasible.3.1.2ContrastingFeatureSpacesAsecondconsiderationarelimitationsimposedonthead-versarybythespaceoffeaturesusedbythelearningalgo-rithm.Inmanylearningalgorithms,dataisrepresentedinafeaturespace,inwhicheachfeaturecapturesarelevantaspectofdatapointsforthelearningtaskathand.Forspamdetection,itiscommonforeachfeaturetorepresent orweb-basedmailers,thesituationchangesdramaticallysincemanyusersoftheseservicesmaybemaliciousandmay,infact,intentionallymislabelmessages.Thispotentialforattacksbytheuserscanbeviewedasatypeofinsiderthreatwhichisatopicforfuturework.whetherornotaparticularwordortokenappearsinthemessage;thisisthefeaturespaceusedbySpamBayes.Inthenetworkvolumeanomalydetectionapplication,thefeaturespaceusedrepresentedthevolumeoftracalongeachlinkwithinthenetworkduringaspeci\fcperiodoftime.Thesedi erencesinfeaturespacerepresentationsprofoundlyim-pactthesetofactionsavailabletotheadversaryandthussigni\fcantlyshapehowattackscanbecarriedout.SpamBayeshasalarge(onecouldsayin\fnite)featurespaceofpossibletokensthatcanoccurinanemailmessage.However,whiletherearemanyfeatures,thein\ruencetheadversaryhasoneachisseverelylimited.Infact,allthead-versarycandecideiswhetherornottoincludethattokeninhisattackmessages;thisboundedformofin\ruenceensuresthattheadversarycannotsimplymanipulateasinglefea-turetohavealargeimpactonthedetector'sperformance.Thatis,itwasnotpossibleforasmallnumberofemailscontrolledbytheadversarytoonlyuseasmallnumberoffeaturesinordertodamagethe\flter.However,thefactthatasingleemailcouldin\ruencemanyfeaturessimultaneously(andindependently)meantthatattacksmessagescouldbeconstructedwithunduein\ruenceonthedetectorasawhole.Thismotivatedthedictionaryattacksdiscussedabove.ThedomainofthePCA-basedanomalydetectorhadverydi erentcharacteristics.Itonlyusedasmallnumberoffea-turestorepresenteachtime-baseddatapoint;asinglefea-tureforeachlinkintheAbilenebackbonenetworkforatotalof54features.However,unlikethebinaryfeaturesinthespamapplication,eachofthesefeatureswasreal-valued.Thismeantthattheadversarydidnotneedtoin\ruencealargenumberoffeatures,butinsteadhecoulddramaticallyalterasmallnumberoffeatures.Bydoingso,theadver-sarycouldhavenearlyunlimitedcontrolofasingledatapointbutafewrestrictionswerenecessary.First,inthisdomain,therewerephysicalcapacitiesinthenetworklinksthatcouldnotbeexceeded.Wealsoassumedtheadversarydoesnothavecontroloverexistingtrac(i.e.,hecannotdelayordiscardtrac).Similarly,theadversarycannotfal-sifySNMPreportstoPCAbecausestealthisamajorgoalforthisattacker|hedoesnotwanthisDoSattackorhispoisoningtobedetecteduntiltheDoSattackhassuccess-fullybeenexecuted.Assuch,welimitedtheattackstoonlyaddspurioustractothenetworktherebyfurtherlimitingtheadversary'scapabilities.3.1.3ContrastingDataDistributionsAnotherapplication-speci\fcaspectofathreattoalearn-ingalgorithmistheunderlyingpropertiesofthedata.Thedata'sdistributioncanprofoundlyimpactnotonlytheper-formanceofthelearningalgorithmbutalsoitsvulnerability.Thedata'sdistributionmayhavepropertiesthatcon\rictwiththelearner'sassumptions.WefoundthatSpamBayeswasvulnerableinpartduetotheinherentsparsityofemails;thatis,thefactthatmostmessagesonlycontainasmallsubsetofthepossiblesetofwordsandtokensthatcanappearinamessage.Thisisawell-knownpropertyofmostdocumentsand,moreover,itiswellknownthatthewordsinemailstendtofollowaZipfdistribution;thatis,thefrequencythatawordoccursinadocumentisapproximatelyinverselyproportionaltotheword'spopularityrank.ThesepropertiescouplewithSpamBayes'independenceassumption(seenextsection)tohavethefollowingconsequences:(a)populartokensoccur 49 ofteninthecorpusandthushavestableprobabilityesti-atesthatcannoteasilybechanged(b)mosttokens,how-ever,occurrarelyandthushaveverylowsupportinthenon-attackdatasotheywereeasilyin\ruencedbythepoisonmessages.SpamBayesbene\ftsfromthe\frstpropertybe-causethemoreoftenawordappearsingoodtrainingdata,thelessvulnerablethatwordistopoisoning.Further,rarewordsarevulnerablebutwillnotbelikelytoappearinfu-turemessages.However,whileitisunlikelythataparticularrarewordwouldappearinanewmessage,itisstillquitelikelythatseveralrarewordswillappearinthemessage|theZipfdistributionislong-tailedmeaningthatthetailofithassigni\fcantmass.Thus,bypoisoningmany(orall)rarewords,theattackagainstSpamBayeswasspectacularlysuc-cessful.Thedatainthenetworkvolumeanomalydetectionalgo-rithmalsohadparticularpropertiesthatanadversarywouldbeabletoexploit.Unlikespam,thedatawasnotsparseinlinkspace|theAbilenebackbonenetworkcarriedlargevol-umesoftracacrossalmostallofit'slinkinevenunitoftime.However,itiswell-establishedthattherearesigni\f-cantsizediscrepanciesinthe\rowspaceofthenetwork;i.e.thereareasmallnumberofend-to-end\rowsinthenet-workthataccountformostofitstrac.These\rows(nick-named`elephant'\rows)dwarfedthenumeroussmall\rows(nicknamed`mouse'\rows).Moreover,whileelephant\rowsalwayscarriedtrac,mouse\rowstendedtobespikywithalmostnotracmostofthetimeandoccasionalspikesoflargeamountsofdata.AswithSpamBayes,attacksagainstthePCA-baseddetectorexploitthisdistributionalproperty.3.2ModelingAttackerCapabilitiesTwoelementsarecriticaltode\fneamodelforthead-versary:hismotivation/objectiveandhiscapabilities.Thetaxonomypartiallydescribesboth,butherewedelvefurtherintohowonecandescribethecapabilitiesofanattackerincausativeattack.Itiscriticaltode\fnehowtheadversarycanalterdatatomisleadorevadetheclassi\fer.Forthispurpose,weneedtomodeltherestrictionsontheadversaryandjustifytheserestrictionsforaparticulardomain.3.2.1CorruptionModelsHere,weoutlinetwocommonmodelsforadversarialcor-ruption,andwedescribehowtheadversaryislimitedwithineach.The\frstmodelassumestheadversaryhasunlimitedcontrolofasmallfractionofthedata;i.e.,theadversaryisrestrictedtoonlymodifyalimitedamountofdatabutcanalterthosedatapointsarbitrarily.Wecallthisaninsertionmodelbecause,inthisscenario,theadversarycraftsasmallnumberofattackinstancesandinsertsthemintothedatasetfortrainingorevaluation(orperhapsreplacesexistingdatapoints).Forexample,intheexampleofaspam\flter,theadversary(spammer)cancreateanyarbitrarymessagefortheirattackbutheislimitedinthenumberofattackmes-sageshecaninject;thus,thespammer'sattackonthespam\fltercanbeanalyzedintermsofhowmanymessagesarerequiredfortheattacktobee ective.Forthisreason,theinsertionmodelwasappropriateinanalyzingattacksontheSpamBayes.Thesecondcorruptionmodelinsteadassumesthattheadversarycanalterany(orall)ofthedatapointsinthedatasetbutislimitedinthedegreeofalteration;i.e.,analterationmodel.Forexample,toattackadetectorthatismonitoringnetworktracvolumesoverwindowsoftime,theadversarycanaddorremovetracwithinthenetworkbutonlycanmakealimiteddegreeofalteration.Suchanadversarycannotinsertnewdatasinceeachdatapointcor-respondstoatimesliceandtheadversarycannotarbitrarilycontrolanysingledatapointsinceotheractorsarealsocre-atingtracinthenetwork;thus,thisisthemodelweusedtoanalyzeattacksonthePCA-subspacedetector.Here,theadversaryisrestrictedbythetotalamountofalterationtheymake,andsothee ectivenessofhisattackcanbean-alyzedintermsofthesizeofalterationrequiredtoachievetheattacker'sobjective.3.2.2ClassLimitationsAsecondlimitationonattackersinvolveswhichpartsofthedatatheadversaryisallowedtoalter|thepositive(ma-licious)class,thenegative(benign)class,orboth.Usually,attackersexternaltothesystemareonlyabletocreatema-liciousdataandsotheyarelimitedtoonlymanipulatingpositiveinstances.Thisisthemodelweusethroughoutthistext.However,thereisalsoanalternativethreatthatin-siderscouldattackalearningsystembyalteringnegativeinstances;alucrativedirectionforfuturework.3.3AttackerKnowledgeThe\fnalaspectoftheattackerthatwemodelistheamountofinformationtheattackerhasaboutthelearningsystem.Generally,theadversaryhasdegreesofinformationaboutthreecomponentsoflearner:itslearningalgorithm,itsfeaturespace,anditsdata.Aswiththeattacker'scapa-bilities,itiscriticaltomakeareasonablemodeloftheat-tacker'sinformation.TherelevantguidelineforassumptionsabouttheadversaryisKerckho s'Principle[34];i.e.,thesecurityofasystemshouldnotrelyonunrealisticexpecta-tionsofsecrecy.Anover-dependenceonsecretstoprovidesecurityisdangerousbecauseifthesesecretsareexposedthesecurityofthesystemisimmediatelycompromised.Ide-ally,securesystemsshouldmakeminimalassumptionsaboutwhatcanrealisticallybekeptsecretfromapotentialat-tacker.Ontheotherhand,ifthemodelgivestheadver-saryanunrealisticdegreeofinformation,ourmodelmaybeoverlypessimistic;e.g.,anomnipotentadversarywhocom-pletelyknowsthealgorithm,featurespace,anddatacanexactlyconstructthelearnedclassi\feranddesignoptimalattacksaccordingly.Thus,itisnecessarytocarefullycon-siderwhatarealisticadversarycanknowaboutalearningalgorithmandtoquantifythee ectsofthatinformation.Withtheseconstraintsinmind,wegenerallyassumethattheattackerhasknowledgeofthetrainingalgorithm,andinmanycasespartialorcompleteinformationaboutthetrain-ingset,suchasitsdistribution.Forexample,theattackermayhavetheabilitytoeavesdroponallnetworktracovertheperiodoftimeinwhichthelearnergatherstrainingdata.Weexaminedi erentdegreesoftheattacker'sknowledgeandassesshowmuchhegainsfromdi erentsourcesofpo-tentialinformation.3.3.1KnowledgeabouttheLearningAlgorithmInouranalysesofreal-worldlearningalgorithms,wehavegenerallyassumedthattheadversarywillknowtheexactlearningalgorithmthattheytarget.Forourcasestudies,thealgorithmswerepublicand,forSpamBayes,thesourcecodewasfreelyavailable.However,wegenerallybelievethatsys- 50 temdesignersshouldassumethattheirlearningalgorithmisownbytheadversary;justastheencryptionalgorithmsincryptography,aregenerallyassumedtobeknown.Onepotentialsourceofsecretinformationabouttheal-gorithmisasecretrandomcomponent(e.g.,arandomizedinitialstate).Suchacomponentcouldbeanimportantse-cretifittrulymakesarandomcontribution(e.g.,manyrandominitialstatesmayyieldthesamedecisionfunction).Theauthorsarenotawareofanystrongguaranteespro-videdbythissortofrandomizationforthesecuritydomainalthoughrandomizationisdiscussedbelowinthecontextofprivacy-preservinglearning.3.3.2KnowledgeabouttheFeatureSpaceThefeaturespaceispotentiallyacomponentofthelearn-ingalgorithmthatmaybekeptsecret(althoughinbothSpamBayesandthePCA-basednetworkanomalydetector,thefeaturespacewaspublished).Specializedfeaturescon-structedforaspeci\fclearningproblemorrelevantdiscrim-inantfeaturesfoundbyafeatureselectionalgorithmmaynotbeknownorinferablebytheadversary.However,thesystemdesignershouldcarefullyassesstheviabilityofthisassumptionssincemanyfeaturesarewidelyusedforsomelearningtasks(e.g.,unigramfeaturesindocumentclassi\f-cation)andspeciallyselectedfeaturesmaybeapproximatedwithsimplerfeatures.3.3.3KnowledgeofDataThe\fnalcomponentoftheattacker'sknowledgeishisknowledgeaboutthetrainingandevaluationdatausedbythealgorithm.Strongerrestrictionsonthiscomponentaremorereasonablebecause,inmanyscenarios,strongprotec-tionsonuserdataareaseparatecomponentofthesystem.However,asystemdesignershouldconsiderwaysinwhichtheadversarycanlearnaboutthesystemsdataandhowspeci\fcthatknowledgeis.Forinstance,partofthedatamaybeavailabletotheadversarybecauseofactionsthead-versarymakesoutsidethesystem(e.g.,aspammercansendspamtoaparticularspam\flter)orbecausetheadversaryisaninsider.Adversariesalsocanhaveadegreeofglobalinformationaboutthedataalthoughthistendstobelessspeci\fc(e.g.,anadversarymayhavedistributionalinforma-tionaboutthewordscommonlyusedinemailsorthelengthofmessages).InourSpamBayescasestudy,weconsideredanadver-sarywithseveraldi erentdegreesofinformationaboutthedata.Whentheattackeronlyhasvagueinformationaboutemailwordcharacteristics(e.g.,thelanguageofthevictim),weshowedthatabroaddictionaryattackcanrenderthespam\flterunusable,causingthevictimtodisablethe\fl-ter.Withmoredetailedinformationabouttheemailworddistribution,theattackercanselectasmallerdictionaryofhigh-valuewordsthatareatleastase ective.Finally,whentheattackerwantstopreventavictimfromseeingparticu-laremailsandhassomeinformationaboutthoseemails,theattackercantargetthemwithafocusedattackthatspecif-icallytargetsthewordsthatarelikelytobeusedinthetargetedmessage.SimilarlyinourstudyofPCA-basedanomalydetectors,weconsideredpoisoningstrategiesinwhichtheattackerhasvariouspotentiallevelsofinformationathisdisposal.Theweakestattackerisonethatknowsnothingaboutthetraf-\fc\rows,andaddscha randomly(calledanuninformedattack).Alternatively,apartially-informedattackerknowsthecurrentvolumeoftracatacompromisedingresslinkthatheintendstoinjectcha on.Wecallthistypeofpoi-soningalocally-informedattackbecausethisadversaryonlyobservesthelocalstateoftracattheingressPoPoftheattack.Inathirdscenario,theattackerisglobally-informedbecausehisglobalviewoverthenetworkenableshimtoknowthetraclevelsonallnetworklinksandthisattackerhasknowledgeofallfuturelinktrac.Althoughglobalin-formationisimpossibletoachieve,westudiedthisscenariotobetterunderstandthelimitsofvarianceinjectionpoison-ingschemes.3.4IdentifyingLearningVulnerabilitiesHereweconsiderthemechanismbywhichtheadversaryattackslearners|thevulnerabilitiesoflearningalgorithms.The\frstpartofanalgorithm'svulnerabilityliesintheas-sumptionsitmakesaboutthetrainingdata.Thesecondpartarisesfromretrainingprocedures,whichcanbeusedbytheadversarytoamplifyaweakattackintoamuchstrongeronebycoordinatingovermanyretrainingiterations.3.4.1LearningAssumptionsEverylearningalgorithmmustmakesomeassumptionsaboutthetrainingdataandthespaceofpossiblehypothesestomakelearningtractable[46].However,thesemodelingassumptionscanalsoleadtovulnerabilitiestoadversarialcorruption.Therearetwotypesoflearningassumptions:assumptionsmadebythelearningmodelandassumptionsmadebythetrainingalgorithm.ModellingAssumptionsAssumptionsmadebythelearningmodelareintrinsictothemodelitself;i.e.,theyareassumptionsinvolvingtheprop-ertiesoftheclassi\feranditsrepresentationofthedata.Commonmodellingassumptionsincludedatalinearity(thelearningtaskcanberepresentedbyalinearfunction),sepa-rability(thereexistssomefunctionthatseparatesthedataintodistinctclasses),andfeatureindependence(eachfeatureofanobjectisanindependentindicatorofalatentvariablethatrepresentsthetrueclassoftheoverallobject).The\frstmodelingvulnerabilityofSpamBayescomesfromitsassumptionthatthedataandtokensareindependent,forwhicheachtokenscoreisestimatedbasedsolelyonthepresenceofthattokeninspamandnon-spammessages.Thesecondvulnerabilitycomesfromitsassumptionthatonlyto-kensthatoccurinamessagecontributetoitslabel.Whilethereissomeintuitionbehindthelatterassumption,inthismodel,itcausesraretokenstohavelittlesupportsothattheirscorescanbeeasilychanged(asdiscussedabove).Ul-timately,thesetwovulnerabilitiesleadtoafamilyofdictio-naryattacksdiscussedabove.InthePCA-baseddetector,themodelingassumptionisalinearityassumptionthatnormaldataiswell-representedbylow-dimensionalsubspaceinthelinkspaceofthealgo-rithm(withasmallresidualcomponent).Thisassumptionisempiricallyvalidated[36],butcanbeviolatedbyacleveradversaryasdiscussedbelow.Inthiscase,theattackarisesmorefromanassumptionmadebythetrainingalgorithmsincemostofthedatadoesobeythelow-dimensionalityas-sumptionbutthelearningalgorithmover-leveragesit. 51 DataAssumptionusedinTraininganyvulnerabilitiesarisefromthedistributionalassump-tionsmadeinthelearningprocedureaboutthedatausedduringthelearningandevaluationphases.Oftentheseas-sumptionsareusedtoconstructmoreecientlearningpro-ceduresthatrequirelessdatabutcanresultinvulnerabili-tieswhenviolated.Todefendagainstorpatchthesevulner-abilities,analternativelearningalgorithmcanbeselectedthatmakesweakerassumptionsalthoughitmaybelessef-\fcient;i.e.,thereisafundamentaltrade-o betweentheeciencyofaprocedureanditsrobustnesstoviolationsinitsassumptions.Manylearningmethodsmakeastationarityassumption;i.e.,thetrainingdataandevaluationdataaredrawnfromthesamedistribution.However,real-worldsourcesofdataoftenarenotstationaryand,evenworse,attackerscaneas-ilybreakthestationarityassumptionwithsomecontrolofeithertrainingorevaluationinstances.Violationsofthesta-tionarityassumptioncomprisethefundamentalvulnerabil-ityoflearningalgorithms,andthus,analyzingandstrength-eninglearningmethodstowithstandormitigateviolationsofthestationarityassumptionisthecruxofthesecurelearn-ingproblem.Anothercommonassumptionisthateachdatapointisindependentandidenticallydistributed(i.i.d.);clearly,thisassumptionisviolatediftheadversarycancoordinatetocreatecorrelationbetweendatapointsoriftheadversar-ialdatacomesfromanalternativedistribution.BoththeSpamBayesandPCA-basedanomalydetectorassumetheirdataisi.i.d.andbotharevulnerablebecauseofit.InSpam-Bayes,thebackgrounddataisnotaltered,buttheattackdatathatisintroducedisdesignedtocomefromadi erentdistribution(recallthedatasparsityinSpamBayesdiscussedabove).Similarly,ourattacksonthePCA-basedetectorvi-olatethei.i.d.assumptionbyintroducingaperturbationtoasubsetofthedatathatsystematicallyshiftsthesepointsontoasecondsubspace;thusthedataisnolongeridenti-callydistributed.Moreover,thisperturbationcanoccur(inlocally-informedandgloballyinformedattacks)inadata-dependentfashionalsoviolatingindependence.3.4.2IterativeRetrainingIterativeretrainingisperhapsthemostdesirablebutalsoleastwell-understoodaspectoflearninginadversarialenvi-ronments.Manydetectorsinsuchenvironmentsrequirepe-riodicretrainingbecauseofnon-stationarityinregulardatacausingitsdistributiontograduallyshiftovertime.Retrain-ingcanbealsobeane ectivedefenseagainstadversariestocountertheirabilitytolearnaboutandadapttoadetector.Finally,iterativelearninghasanextensivetheoryforcom-biningclassi\fersevenunderstrongadversarialsettings[9].Simultaneously,retraining,ifappliedimproperly,canbeex-ploitedbytheadversarypotentiallytoamplifytheimpactofweakattacksovermanyiterations.Thereislittledoubtthatpastexperiencecouldbetremen-douslyvaluableforimprovingclassi\fers,buthowcanthepastbeusede ectivelyandsafely?Ifthepastisnotusedatall,everyiterationofretrainingisseeminglyindependent,buteveninthiscase,pastmodelsmaysubtletyin\ruencefuturemodels.Mostusually,someofthedatausedforthetrainingthenextiterationofmodelsisselectedorlabeledbythelastlearnedmodel|thissmallin\ruencecanbeusedinattackssuchastheBoilingFrogattackagainstthePCA-baseddetectortomakeasmallattackfarmoree ectiveasdiscussedabove.Moresubtlety,thebehavioroftheclassi\fermaycauseausertopaymoreattentiontosomedatapointsthanothersandthusimpactthesubsequentretrainingpro-cess.Forinstance,inthecaseofSpamBayes,themistakesSpamBayesmakesmaycausetheusertospeci\fcallyidentifyandretrainonattackmessageswhenthesemessagesappearwithhisnormalmail.However,mostusersarelesslikelytonoticemessagesthatinadvertentlymislabeledasspam.Ifthepastiswhollyincorporatedintothenextgenera-tionofmodel,theadvantageseemstoliewiththelearningmodelasweexploredinpriorworkwithretraininghyper-sphereanomalydetectors[49].Inthatwork,weshowedthatwhenallpastdataisusedtoupdatethemeanofthehyper-sphere,thenumberofdatapointstheadversarymustcon-troltoshiftthemodelisexponentialinthedesireddistance.However,thiscomesataprice;becausepastdataisneverdiscarded,themodelbecomesincreasinglylessadaptivetonormalchangesinthedata.KloftandLaskovextendedthismodelbyconsideringmorerealisticpoliciesfordataagingandamorerealisticsettingfortheattack[35].However,thegeneraltaskofretrainingmodelsinasecurefashionremainsanopenissue.3.5DefensesandCountermeasuresThe\fnalcomponentinourdiscussionofcausativeat-tacksismethodsfordefendingagainstthem.Twogeneralstrategiesfordefensearetoremovemaliciousdatafromthetrainingsetandtohardenthelearningalgorithmagainstmalicioustrainingdata.BelowwediscussbothofthesetechniquesinthecontextoftheSpamBayesandPCA-baseddetectorsdiscussedabove.3.5.1DataSanitizationInsidiouscausativeattacksmakelearninginherentlymoredicult.Inmanycircumstances,datasanitizationmaybetheonlyrealisticmechanismtoachieveacceptableperfor-mance.Forexample,forSpamBayesweexploredsuchasanitizationtechniquecalledtheRejectOnNegativeImpact(RONI)defense[48],atechniquethatmeasurestheem-piricale ectofaddingeachtraininginstanceanddiscardsinstancesthathaveasubstantialnegativeimpactonclassi\f-cationaccuracy.Todeterminewhetheracandidatetraininginstanceismaliciousornot,thedefendertrainsaclassi\feronabasetrainingset,thenaddsthecandidateinstancetothetrainingsetandtrainsasecondclassi\fer.Thedefenderappliesbothclassi\ferstoaquizsetofinstanceswithknownlabelsandmeasuresthedi erenceinaccuracybetweenthetwoclassi\fers.Ifaddingthecandidateinstancetothetrain-ingsetcausestheresultingclassi\fertoproducesubstantiallymoreclassi\fcationerrors,thedefenderpermanentlyremovestheinstanceasdetrimentalinitse ect.TheRONIdefenserejectseverysingledictionaryattackfromanyofthedictionaries(optimal,Aspell,andUsenet).Infact,thedegreeofchangeinmisclassi\fcationratesforeachdictionarymessageisgreaterthan\fvestandarddevia-tionsfromthemedian,suggestingthattheseattacksareeas-ilyeliminatedwithonlyminorimpactontheperformanceofthe\flter.Whentrainedon1000uncensoredmessages,theresulting\fltercorrectlyclassi\fes98%ofhamand80%ofthespam.AfterremovingthemessagesrejectedbytheRONIdefense,theresulting\flterstillcorrectlyclassi\fes95%ofhamand87%ofthespam. 52 3.5.2RobustLearninghe\feldofrobuststatisticsexploresproceduresthatlimittheimpactofasmallfractionofdeviant(adversarial)train-ingdata.Inthesettingofrobuststatistics[30],itisassumedthatthebulkofthedataisgeneratedfromaknownwell-behavedmodel,butafractionofthedatacomesfromanunknownmodel(outliers)|toboundthee ectofthisun-knownsourceitisassumedtobeadversarial.Becausethenetworkdatahasmanyoutlierevents(bothmaliciousandbenign)wechosetoreplacethePCA-baseddetectorwithamorerobustvariant.ItisknownthatPCAcanbestronglya ectedbyout-liers[54].Insteadof\fndingtheprincipalcomponentsalongdirectionsthatmaximizevariance,alternativePCA-liketech-niques\fndmorerobustcomponentsbymaximizingalterna-tivedispersionmeasureswithdesirablerobustnessproper-ties.Inparticular,thePCA-GridalgorithmwasproposedbyCrouxetal.asanecientmethodforestimatingdirectionsthatmaximizethemedianabsolutedeviation(MAD)with-outunder-estimatingvariance[15].WeadaptPCA-GridforanomalydetectionbycombiningthemethodwitharobustLaplacecuto threshold.Together,werefertothemethodasAntidote.Becauseitbuildsonrobustsubspaceestimates,thismethodsubstantiallyreducesthee ectofoutliersandisabletorejectpoisonoustrainingdata.4.EXPLORATORYATTACKSThemostfrequentlystudiedattacksareexploratoryin-tegrityattacksinwhichtheadversaryattemptstopassivelycircumventthelearningmechanismtoexploitblindspotsinthelearnerthatallowmiscreantactivitiestogoundetected.Inanexploratoryintegrityattack,theattackercraftsin-trusionssoastoevadetheclassi\ferwithoutdirectin\ruenceovertheclassi\feritself.Instead,attacksofthissortoftenat-tempttosystematicallymakethemiscreantactivityappeartobenormalactivitytothedetectororobscurethemiscre-antactivity'sidentifyingcharacteristics.Someexploratoryintegrityattacksmimicstatisticalpropertiesofthenormaltractocamou\rageintrusions;e.g.,theattackerexaminestrainingdataandtheclassi\fer,thencraftsintrusiondata.Intheexploratorygame,theattacker'smoveproducesma-liciousinstancesin(eval)thatstatisticallyresemblenormaltracinthetrainingdata(train)Exploratoryintegrityattackshavebeenextensivelystud-iedinattacksagainstintrusiondetectorsystems(IDS)andspam\flters.FoglaandLeeintroducedpolymorphicblendingattacksthatevadeintrusiondetectorsusingencryptiontech-niquestomakeattacksstatisticallyindistinguishablefromnormaltrac[27].Featuredeletionattacksinsteadspecif-icallyexcludehigh-valueidentifyingfeaturesusedbythedetector[28];thisformofattackstressestheimportanceofproperfeatureselectionaswasalsodemonstratedempiri-callyby[42]intheirstudyofthebehaviorofintrusiondetec-tionsystemsontheDARPA/LincolnLabdataset.Tanetal.describeamimicryattackagainstthesequence-basedIDS[62].Theymodifyexploitsoftheandprogramstoaccomplishthesameendsusingdi erentsequencesofsystemcalls:theshortestsubsequenceinat-tacktracthatdoesnotappearinnormaltracislongerthantheIDSwindowsize.Byexploitingthe\fnitewin-dowsizeofthedetector,thistechniquemakesattacktracindistinguishablefromnormaltracforthedetector.Inde-pendently,WagnerandSotoalsodevelopedmimicryattacksagainstpH,asequence-basedIDS[64].Usingthemachineryof\fniteautomata,theyconstructedaframeworkfortestingwhetheranIDSissusceptibletomimicryforaparticularexploit.Addingorchangingwordsinaspammessagecanallowittobypassthe\flter.LiketheattacksagainstanIDSabove,theseattacksallusebothtrainingdataandinfor-mationabouttheclassi\fertogenerateinstancesintendedtobypassthe\flter.Studyingthesetechniqueswas\frstsug-gestedbyJohnGraham-CumminginhispresentationHowtoBeatanAdaptiveSpamFilteratthe2004MITSpamConference,inwhichhepresentedaBayesvs.Bayesattackthatusesasecondstatisticalspam\flterto\fndgoodwordsbasedonfeedbackfromthetarget\flter.Severalauthorshavefurtherexploredevasiontechniquesusedbyspammersanddemonstratedattacksagainstspam\fltersusingsimilarprinciplesasthoseagainstIDSsasdiscussedabove.LowdandMeekandWittelandWudevelopedattacksagainststa-tisticalspam\fltersthataddgoodwords,orwordsthe\flterconsidersindicativeofnon-spam,tospamemails[40,65].Thisgoodwordattackmakesspamemailsappearinnocuoustothe\flter,especiallyifthewordsarechosentobeonesthatappearofteninnon-spamemailandrarelyinspamemail.Finally,obfuscationofspamwords(i.e.,changingcharactersinthewordorthespellingofthewordsoitnolongerrecognizedbythe\flter)isanotherpopulartechniqueforevadingspam\flters[38,58].4.1Cost-basedEvasionAnotherveinofresearchintoexploratoryintegrityattacksfocusesonthecostsincurredduetotheadversary'sevasiveactions;i.e.,instancesthatevadedetectionmaybelessde-sirabletotheadversary.Bydirectlymodellingadversarialcost,thisworkexplicitlycastsevasionasaproblem,inwhichtheadversarywantstoevadedetectionbutwantstodosousinghigh-valueinstances(anassumptionthatwasimplicitintheotherworkdiscussedinthissection).Dalvietal.ploitthesecoststodevelopacost-sensitivegame-theoreticclassi\fcationdefensethatisabletosuccessfullydetectopti-malevasionoftheoriginalclassi\fer[16].Usingthisgame-theoreticapproach,thistechniquepreemptivelypatchesthenaiveclassi\fer'sblindspotsbyconstructingamodi\fedclas-si\ferdesignedtodetectoptimally-modi\fedinstances.Sub-sequentgametheoreticapproachestolearninghaveextendedthissettingandsolvedforequilibriaofthegame[8,32].Costmodelsoftheadversaryalsoledtoatheoryforquery-basednear-optimalevasionofclassi\fers\frstpresentedbyLowdandMeek,inwhichtheycastthedicultyofevad-ingaclassi\ferintoacomplexityproblem[39].Theypre-sentedalgorithmsforanattackertoreverseengineeraclas-si\fer.Theattackerseeksthelowestcostinstance(fortheattacker)thattheclassi\fermislabelsasanegativeinstance.Inthenextsectionwediscussourextensiontothisframe-work[50].Wegeneralizedthetheoryofnear-optimalevasiontoabroaderclassofclassi\fersanddemonstratedthattheproblemiseasierthanreverse-engineeringapproaches.4.1.1Near-OptimalEvasionThenear-optimalevasionproblemformalizesthenaturalsecuritysettingforevasion.Theproblemabstractsthesce-narioofanadversarywhowishestolaunchaspeci\fcattackthatisblockedbyaclassi\fer-baseddefense.Theattacker 53 hasalimitednumberofprobingopportunitiesafterwhichemustsendanattackascloseaspossibletohisoriginallyintendedattack|anear-optimalattack.Inthecaseofemailspam,thespammermayoriginallyhaveamessagethatwillbedetectedasspam.Heprobes,\fndsanear-optimalmes-sagethatevadesthe\flter,andsendsthismessageinstead.Inthecaseofanintruder,hehasapreferredsequenceofsystemcallsthatwillbedetectedasintrusions.Again,heprobes,then\fndsandexecutesanear-optimalsequencethatevadesthedetector.Withthisframeworkinmind,wenowclearlyseetheroleofadefender:toprovideaclassi\ferthatlimitsorresistsnear-optimalevasion.Practicalimplementa-tionrequirescarefulselectionofcostsandrealisticboundsonthenumberofprobesanadversarycanperform.Re-sultinglower-boundsonthenumberofprobesrequiredfornear-optimalevasionprovidesigni\fcantevidenceofe ectivesecurity.Theproblemofnear-optimalevasioni.e.,\fndingalowcostnegativeinstancewithfewqueries)wasintroducedbyLowdandMeek[39].Wecontinuedstudyingthisproblembygeneralizingittothefamilyofconvex-inducingclassi\fer|classi\fersthatpartitiontheirinstancespaceintotwosetsoneofwhichisconvex.Thefamilyofconvex-inducingclas-si\ferisanimportantandnaturalsetofclassi\ferstoexam-inewhichincludesthefamilyoflinearclassi\fersstudiedbyLowdandMeekaswellasanomalydetectionclassi\fersusingboundedPCA[36],anomalydetectionalgorithmsthatusehyper-sphereboundaries[5],one-classclassi\fersthatpredictanomaliesbythresholdingthelog-likelihoodofalog-concave(oruni-modal)densityfunction,andsomequadraticclassi-\fers.Thefamilyofconvex-inducingclassi\feralsoincludesmorecomplicatedbodiessuchasthecountableintersectionofhalfspaces,cones,orballs.Inourwork,wedemonstratedthatnear-optimalevasiondoesnotrequirecompletereverseengineeringoftheclassi-\fer'sinternalstateordecisionboundary,butinstead,onlypartialknowledgeaboutitsgeneralstructure[50].Thealgo-rithmpresentedbyLowdandMeekforevadinglinearclas-si\fersinacontinuousdomainreverseengineersthedeci-sionboundarybyestimatingtheparametersoftheirsepa-ratinghyperplane.Thealgorithmswepresentedforevad-ingconvex-inducingclassi\ferdonotrequirefullyestimatingtheclassi\fer'sboundary(whichishardinthecaseofgen-eralconvexbodies[53])ortheclassi\fer'sparameters(in-ternalstate).Instead,thesealgorithmsdirectlysearchforaminimalcost-evadinginstance.Thesesearchalgorithmsrequireonlypolynomial-manyqueries,withonealgorithmsolvingthelinearcasewithbetterquerycomplexitythanthepreviously-publishedreverse-engineeringtechnique.Fi-nally,wealsoextendednear-optimalevasiontogeneralcosts;i.e.,costsbasedondistances.Weshowedthatthealgorithmsforcostscanalsobeextendedtonear-optimalevasiononcosts,butaregenerallynotecient.However,inthecaseswhenthesealgorithmsarenotecient,weshowthatthereisnoecientquery-basedalgorithm.Thereareavarietyofwaystodesigncountermeasuresagainstexploratoryattacks.Forexample,Biggioetal.pro-moterandomizedclassi\fersasadefenseagainstexploratoryevasion[4].Theyproposetheuseofmultipleclassi\fersys-tems(MCSs),whicharemainlyusedtoimproveclassi\fca-tionaccuracyinmachinelearningcommunity,toimprovetherobustnessofaclassi\ferinadversarialenvironments.Theyconstructamultipleclassi\fersystemusingthebag-gingandrandomsubspacemethods,andconductaseriesofexperimentstoevaluatetheirsysteminarealspam\flteringtask.Theirresultsshowedthat,althoughtheirmethoddidnotimprovetheperformanceofasingleclassi\ferwhentheyarenotunderattack,itwassigni\fcantlymorerobustun-derattack.TheseresultsprovideasoundmotivationtotheapplicationofMCSsinadversarialclassi\fcationtasks.How-ever,itisnotknownifrandomizedclassi\fershaveprovablyworsequerycomplexities.4.1.2Real-WorldEvasionWhilethecost-centricevasionframeworkpresentedbyLowdandMeekprovidesaformalizationthenear-optimalevasionproblem,itfailstocaptureseveralimportantaspectsofevasioninreal-worldsettings.Fromthetheoryofnear-optimalevasion,certainclassesoflearnerscanbeevadedef-\fcientlywhereasothersrequireapracticallyinfeasiblenum-berofqueriestoachievenear-optimalevasion.However,real-worldadversariesoftendonotrequirenear-optimalcostevasiveinstancestobesuccessful;itsucesforthemto\fndanysucientlylow-costinstancethatevadesthedetector.Understandingquery-strategiesforareal-worldadversaryrequiresincorporatingreal-worldconstraintsthatwerere-laxedorignoredinthetheoreticalversionofthisproblem.Here,wesummarizethechallengesforreal-worldevasion.Real-worldnear-optimalevasionismoredicult(i.e.,re-quiresmorequeries)thanissuggestedbythetheorybecausethetheorysimpli\festheproblemfacedbytheadversary.Evenassumingthatareal-worldadversarycanobtainqueryresponsesfromtheclassi\fer,hecannotdirectlyqueryitinthefeaturespaceoftheclassi\fer.Real-worldadversariesmustmaketheirqueriesintheformofreal-worldobjectslikeemailthataresubsequentlymappedintothefeaturespaceoftheclassi\fer.Evenifthismappingisknownbytheadversary,designinganobjectthatmapstoaspeci\fcdesiredqueryisitselfdicult|theremaybemanyobjectsthatmaptoasinglequeryandpartsofthefeaturespacemaynotcorrespondtoanyreal-worldobject.Thus,futureresearchonreal-worldevasionmustaddressthequestion:Howcanthefeaturemappingbeinvertedtodesignreal-worldinstancestomaptodesiredqueries?Real-worldevasionalsodi ersdramaticallyfromthenear-optimalevasionsettinginde\fninganecientclassi\fer.Forareal-worldadversary,evenpolynomially-manyqueriesinthedimensionalityofthefeaturespacemaynotreasonable.Forinstance,ifthedimensionalityofthefeaturespaceislarge(e.g.,hundredsofthousandsofwordsinunigrammod-els)theadversarymayrequirethenumberofqueriestobesub-linear,butfornear-optimalevasion,thisisnotpossibleevenforlinearclassi\fers.However,real-worldadversariesalsoneednotbeprovablynear-optimal.Near-optimalityisasurrogateforadversary'strueevasionobjective:touseasmallnumberofqueriesto\fndanegativeinstancewithac-ceptablylow-cost.Clearly,near-optimalevasionissucienttoachievethisgoal,butinreal-worldevasion,oncealow-costnegativeinstanceislocated,thesearchcanterminate.Thus,insteadofquantifyingthequerycomplexityrequiredforafamilyofclassi\fers,itismorerelevanttoquantifythequeryperformanceofanevasionalgorithmfora\fxednumberofqueriesbasedonatargetcost.Thisraisessev-eralquestions:Whatistheworst-caseorexpectedreductionincostforaqueryalgorithmaftermakinga\fxednumberofqueriestoaclassi\fer,whatistheexpectedvalueofeach 54 querytotheadversaryandwhatisthebestquerystrategyora\fxednumberofqueries?The\fnalchallengeforreal-worldevasionistodesignal-gorithmsthatcanthwartattemptstoevadetheclassi\fer.Promisingpotentialdefensivetechniquesincluderandomiz-ingtheclassi\ferandidentifyingqueriesandsendingmislead-ingresponsestotheadversary.However,noprovendefenseagainstevasionhasthusfarbeenproposed.5.PRIVACYVIOLATIONSPrivacy-preservinglearninghasbeenstudiedbyresearchcommunitiesinSecurity,Databases,Theory,MachineLearn-ingandStatistics.Thebroadgoalofthisresearchistore-leaseaggregatestatisticsonadatasetwithoutdisclosinglo-calinformationaboutindividualdataelements.Inthelan-guageofourtaxonomy,privacy-preservinglearningshouldberobusttoExploratoryorCausativeattackswhichaimtoviolatePrivacy.Anattackerwithaccesstoareleasedstatis-tic,modelorclassi\fermayprobeitinanattempttorevealinformationaboutthetrainingdata;moreoveranattackerwithin\ruenceoversomeproportionofthetrainingexamplesmayattempttomanipulatethemechanismintorevealinginformationaboutunknowntrainingexamples.InthiswaythePrivacyviolationrepresentsanimportantextensionofthesecurityviolationsofmachinelearningalgorithmscon-sideredbyouroriginaltaxonomy.5.1DifferentialPrivacyHistorically,formalmeasuresforquantifyingthelevelofprivacypreservedbyadataanalysisordatareleasehavebeenelusive.Numerousde\fnitionshavebeenproposedandputasideduetothepropositionsbeingofasyntacticratherthansemanticnature,mostnotably-anonymityanditsvariants[61,41].Howeverrecentlytheconceptofdi erentialprivacyduetoDwork[19]hasemergedasastrongguaranteeofprivacy,withformalrootsin\ruencedbycryptography.Thisde\fnitionhasenjoyedasigni\fcantamountofinterestintheTheorycommunity[17,6,22,19,1,7,24,23,44,33,25,21,3,31,59]wherethegeneralconsensusisthattheformalde\fnitionismeaningfulandappropriatelystrong,whileallowingforstatisticallearningmethodsthatpreservethenotionofprivacytobeofpracticaluse[56,43,1,33,17,25,3,31,11].Wenowproceedtorecallthede\fnitionofdi erentialprivacyandthentodiscussitsprevailingfeaturesinthecurrentcontextofadversarialmachinelearning.databaseisasequenceofrows;:::;xthataretypicallybinaryorrealvectorsbutcouldbelongtoanydo-main.Givenaccessto,amechanismistaskedwithreleasingaggregateinformationaboutwhilemaintainingprivacyofindividualrows.Inparticularweassumethattheresponse2Tistheonlyinformationreleasedbythemechanism.Thisresponsecouldbeascalarstatis-ticonsuchasamean,medianorvariance;oramodelsuchastheparameterstoaestimatedjointdensityortheweightvectorstoalearnedclassi\fer.Wesaythatapairofdatabases;Dareneighborsiftheydi erononerow. northogonalbodyofworkexists,inwhichsecuremulti-partycomputationisappliedtomachinelearning[18,29].Thereonehastoshareindividualdatapointswithuntrustedrdpartiesinordertotrainalearner;e.g.,whenemploy-ingcloud-basedcomputation,orpoolingwithothers'datatotrainamorepredictivemodel.Theresultingmodelorclassi\fer,however,doesnotpreservetrainingdataprivacy.Withthesede\fnitionsin-handwecandescribethefollowingformalmeasureofprivacy.Definition1([19]).Forany�,arandomizedmech-anismachieves-di erentialprivacyif,forallpairsofneighboringdatabases;Dandallresponses2Tthemechanismsatis\feslogPr()= Pr(:Tounderstandthisde\fnitionatahighlevel,consideradi erentiallyprivatemechanismthatpreservesdataprivacybyaddingnoisetotheresponseofsomedesirednon-privatedeterministicstatistic),saytheaverageofasequenceofscalars;:::;x.Thedef-inition'slikelihoodratiocomparesthedistributionsofnoisymeanresponses,whenonescalar(adatabaserow)ischanged.Ifthelikelihoodratioissmall(whenprivacylevel1),thenthelikelihoodofrespondingwithnoisymeanondatabaseisexceedinglyclosetothelikelihoodofrespondingwiththesameondatabasewithper-turbed:themechanism'sresponsedistributionsonthetwoneighboringdatabasesarepoint-wiseclose.5.1.1Example:PrivateSVMLearningAsamorepracticalexamplewehavepreviouslystudieddi erentiallyprivatemechanismsforsupportvectormachine(SVM)learning[56].Therethesettingisagainaprivatedatabaseonwhichwewishtoperforminference.Howeverthedatabaseisnowcomposedofrowsoffeaturevectorsandbinarylabelsmakingupatrainingsetofsupervisedbinaryclassi\fcation.Thedesiredinferenceisnowthemoresophis-ticatedtaskofSVMlearning[14]:inthelinearcase\fndahyperplanenormalvectorthatmaximizesmarginonthetrainingset,andinthenon-linearcaseperformthismargin-maximizationinahigh-dimensionalfeaturespaceinducedbyauser-de\fnedkernelfunction.Themechanismhererespondswiththeweightvectorrep-resentingthelearnedclassi\feritself;theresponseistheparameterizationofafunction.OurmechanismforlinearSVMsimplyaddsLaplacenoisetotheweightvector,whichweproveusingthealgorithmicstabilityofSVMlearningtoachievedi erentialprivacy.Forthenon-linearcasewe\frstsolvelinearSVMinarandomfeaturespacewithinner-productapproximatingthedesiredkernelbeforeaddingnoisetothecorrespondingsolution;this\frststepallowsustoachievedi erentialprivacyevenforkernelssuchastheRa-dialBasisFunctionthatcorrespondstolearninginanin\fnite-dimensionalfeaturespace.Anotherapproachtodi erentiallyprivateSVMlearningisduetoChaudhurietal.[11]whoinsteadofaddingnoisetothesolutionofSVMlearning,randomlyperturbtheop-timizationusedforSVMlearningitself.Numerousotherpracticallearningalgorithmshavebeenmadedi erentiallyprivateincludingregularizedlogisticre-gression[10],severalcollaborative\flteringalgorithms[43],pointestimation[59],nearestneighbor,histograms,percep-tron[6],andmore. heprobabilitiesinthede\fnitionareovertherandom-izationofmechanismnotoverthedatabaseswhichare\fxed. 55 5.2Exploratory&CausativePrivacyAttacksnimportantobservationondi erentialprivacyisthatthede\fnitionprovidesforverystrong,semanticguaranteesofprivacy.Evenwithknowledgeofuptorandomnessandwithknowledgeof(say)the\frst1rows,anadversarycannotlearnanyadditionalinformationonthethrowfromasublinear(in)sampleof).Theadversarymayevenattemptabrute-forceexploratoryattackwithsuchauxiliaryinformationandunboundedcomputationalresources:1.Foreachpossibleconsider;:::;x;xneighboringdatabase(a)Oine:Calculatetheresponsedistributionof)bysimulation.2.Estimatethedistributionof)as^byqueryingthemechanism(asublinearnumberoftimes).3.IdentifybythemostcloselyresemblingHoweverforhighlevelsofprivacy(sucientlysmall),thesamplingerrorin^willbegreaterthanthedi erencesbetweenalternate,andsoeventhispowerfulbrute-forceexploratoryattackwillfailwithhighprobability.Thesamerobustnessholdseveninthesettingoftheanalogouscausativeattack,wheretheadversarycanarbitrarymanip-ulatethe\frst1rows.5.3OntheRoleofRandomizationAssuggestedabove,thepredominantavenuefordesign-ingadi erentiallyprivateversionofastatistic,modelorlearningalgorithmistoproducearandomizedmechanismthatcorrespondstothedesirednon-privatealgorithm,withnoiseaddedatsomestageoftheinference.Themostcommonapproachtoachievingdi erentialpri-vacyistoaddnoisetothetargetnon-privatemechanism'sresponse),typicallyLaplacenoise[22,19,6,20,56]butmoregenerallyanexponentialmechanismisusedtoran-domlyselectaresponsebasedondistanceto)[44,7,33].Intheformercase,thescaleofLaplacenoiseisdirectlyproportionaltothesensitivityofthetargetmechanismresponsetochangingbetweenneighboringdatabases,andinverselyproportionalto.Often-timesthesensitivityitselfdecreaseswithincreasingdatabasesize,andsoforlargerdatabaseslessnoiseisaddedtoguaranteethesamelevelofdi erentialprivacy.Di erentialprivacyhasalsobeenachievedbyrandomizingtheobjectivefunctionofanoptimizationperformedtoexe-cutelearning.ForexampleChaudhurietal.havedevelopeddi erentiallyprivateversionsofempiricalriskminimizersin-cludingtheSVM[11]andregularizedlogisticregression[10].Intheirwork,theoriginallearningalgorithmisformulatedasanoptimizationtypicallyminimizingtheweightedsumofempiricalriskandaregularizationterm.Byaddinganad-ditionaltermwhichistheinner-productbetweentheweightvectorandarandomdirection,learningtendstoslightlypre-fertherandomdirectionandinsodoingcanbeproventoyielddi erentialprivacyundercertaintechnicalconditions.Theroleofrandomizationofthedesiredstatisticorlearn-ingalgorithm,eitherthroughaddingoutputnoise,random-izinganobjectivefunction,orsimilar,iscrucialinprovidingadi erentiallyprivatemechanismthatcanreleaseaggre-gateinformationonadatabasewhilepreservingtheprivacyofindividualdata.Whilewehavesuspectedrandomiza-tioncouldprovebene\fcialto\fghtingattacksthatviolateintegrityoravailability[51],fewpositiveresultsareknown.5.4UtilityintheFaceofRandomnessThemoreatargetnon-privateestimatorisrandomizedthemoreprivacyispreserved,butatacosttoutility.Severalresearchershaveconsideredthisinherenttrade-o betweenprivacyandutilityInourworkondi erentiallyprivateSVMlearning[56],wede\fnetheutilityofourprivatemechanismtobethepoint-wisedi erencebetweenreleasedprivacy-preservingclassi-\fersandnon-privateSVMclassi\fers.Aprivateclassi\fer(trainedon)thatwithhighprobabilityyieldsverysimilarclassi\fcationstoanSVM(trainedon),foralltestpoints,isjudgedtobeofhighutilitysinceitwell-approximatesthedesirednon-privateSVMclassi\fer.SimilarnotionsofutilityareconsideredbyBaraketal.[1]whenreleasingcon-tingencytableswhosemarginalsareclosetotruemarginals;Blumetal.[7]whosemechanismreleasesanonymizeddataonwhichaclassofanalysesyieldsimilarresultsasontheoriginaldata;andKasiviswanathanetal.[33]andBeimelal.[3]whoconsiderutilityascorrespondingtoPAClearn-ingwhereresponseandtargetconceptslearnedonsensi-tivedataareaveragedovertheunderlyingmeasure.OtherssuchasChaudhurietal.[10,11]measuretheutilityofadi erentialprivatemechanismnotbyitsapproximationofatargetnon-privatealgorithm,butratherbytheabsoluteerroritachieves.Inalloftheseworks,thedi erentiallypri-vatemechanismisanalyzedwiththechosenutilitytoupperboundtheutilityachievedbythatparticularmechanism.Fundamentallimitsonthetrade-o betweendi erentialprivacyandutilityhavealsobeenofgreatinterestinpastwork,throughnegativeresults(lowerbounds)thatessen-tiallystatethatmechanismscannotachievebothhighlevelsofprivacypreservationandutilitysimultaneously.Inourworkondi erentiallyprivateSVMlearning[56]weestablishlowerboundsforapproximatingbothlinearandRBFSVMlearningwithanydi erentiallyprivatemechanism,quanti-fyinglevelsofdi erentialprivacyandutilitythatcannotbeachievedtogether.Dinur&Nissim[17]showthatifnoiseofrateonly isaddedtosubsetsumqueriesonadatabaseofbitsthenanadversarycanreconstructa1(1)frac-tionof:ifaccuracyistoogreatthenprivacycannotbeguaranteedatall.Hardt&Talwar[31]andBeimeletal.[3]alsorecentlyestablishedupperandlowerboundsforthetrade-o betweenutilityandprivacyinrespectivesettingswherethemechanismrespondswithlineartransformationsofdata,andthesettingofprivatePAClearning.Whilesigni\fcantprogresshasbeenmadeinachievingdif-ferentialprivacyandutility,understandingconnectionsbe-tweendi erentialprivacyandlearnability[3],algorithmicstability[56],robuststatistics[21],andevenmechanismde-sign[44],manyopenproblemsremainin\fndingmorecom-pleteunderstandingsoftheseconnections,makingpracticallearningalgorithmsdi erentiallyprivate,andunderstandingthetrade-o betweenprivacyandutility.6.CONCLUSIONThe\feldofadversarialmachinelearningcanbethoughtofasstudyingthee ectsofbringingthe\Byzantine"tomachinelearning.Inthispaper,weshowedhowtwoma-chinelearningmethods,SpamBayesandPCA-basednet-workanomalydetection,arevulnerabletocausativeattacks 56 anddiscussedhowapplicationdomain,featuresanddataistributionrestrictanadversary'sactions.Wealsooutlinedmodelsfortheadversary'scapabilities,intermsofinputcorruptionandclasslimitations,andtheirknowledgeofthelearningsystemsalgorithm,featurespace,andinputdata.Wealsoconsideredhowanadversaryat-tacksalearningsystem,anddiscusseddefensesandcounter-measures.Weexaminedexploratoryattacksagainstlearn-ingsystemsandpresentedanimportanttheoreticalresultinthe\feldofnearoptimalevasion,showingthatitwaseasyforadversariestosearchconvexclassi\fersto\fndinputthatcanavoidbeingclassi\fedasnegative.Finally,weexploredapproachesandchallengesforprivacy-preservinglearning,includingdi erentialprivacy,exploratoryandcausativeprivacyattacks,andrandomization.7.ACKNOWLEDGMENTSWewouldliketothankMarcoBarreno,PeterBartlett,BattistaBiggio,ChrisCai,FuchingJackChi,MichaelJor-dan,MariusKloft,PavelLaskov,Shing-honLau,StevenLee,SatishRao,UdamSaini,RussellSears,CharlesSut-ton,NinaTaft,AnthonyTran,andKaiXiaformanyfruit-fuldiscussionsandcollaborationsthathavein\ruencedourthinkingaboutadversarialmachinelearning.Wegratefullyacknowledgethesupportofoursponsors.ThisworkwassupportedinpartbyTRUST(TeamforResearchinUbiq-uitousSecureTechnology),whichreceivessupportfromtheNationalScienceFoundation;DETERlab(cyber-DEfenseTechnologyExperimentalResearchlaboratory),whichre-ceivessupportfromDHSHSARPA;andtheBerkeleyCounter-CensorshipIncubator,whichreceivessupportfromtheUSDepartmentofState;andbytheAlexandervonHumboldtFoundation.Wealsoreceivedsupportfrom:Amazon,Cisco,Cloudera,eBay,facebook,Fujitsu,Google,HP,Intel,Mi-crosoft,NetApp,Oracle,SAP,VMware,andYahoo!Re-search.Theopinionsexpressedinthispaperaresolelythoseoftheauthorsanddonotnecessarilyre\recttheopinionsofanysponsor.8.REFERENCES[1]B.Barak,K.Chaudhuri,C.Dwork,S.Kale,F.McSherry,andK.Talwar.Privacy,accuracy,andconsistencytoo:aholisticsolutiontocontingencytablerelease.InPODS'07,pages273{282,2007.[2]M.Barreno,B.Nelson,R.Sears,A.D.Joseph,andJ.D.Tygar.Canmachinelearningbesecure?InASIACCS'06,pages16{25,2006.[3]A.Beimel,S.Kasiviswanathan,andK.Nissim.Boundsonthesamplecomplexityforprivatelearningandprivatedatarelease.InTheoryofCrypto.,volume5978ofLNCS,pages437{454.2010.[4]B.Biggio,G.Fumera,andF.Roli.Multipleclassi\fersystemsunderattack.InProc.Int.WorkshopMultipleClassi\ferSystems,volume5997,pages74{83,2010.[5]C.M.Bishop.PatternRecognitionandMachineLearning.Springer,2006.[6]A.Blum,C.Dwork,F.McSherry,andK.Nissim.Practicalprivacy:theSuLQframework.InPODS'05pages128{138,2005.[7]A.Blum,K.Ligett,andA.Roth.Alearningtheoryapproachtonon-interactivedatabaseprivacy.InSTOC'08,pages609{618,2008.[8]M.BrucknerandT.Sche er.Nashequilibriaofstaticpredictiongames.InNIPS,pages171{179.2009.[9]N.Cesa-BianchiandG.Lugosi.Prediction,Learning,andGames.CambridgeUniversityPress,2006.[10]K.ChaudhuriandC.Monteleoni.Privacy-preservinglogisticregression.InNIPS,pages289{296,2009.[11]K.Chaudhuri,C.Monteleoni,andA.D.Sarwate.Di erentiallyprivateempiricalriskminimization.JMLR,12:1069{1109,2011.[12]S.P.ChungandA.K.Mok.Allergyattackagainstautomaticsignaturegeneration.InRAID'09,volume4219ofLNCS,pages61{80,2006.[13]S.P.ChungandA.K.Mok.Advancedallergyattacks:Doesacorpusreallyhelp?InRAID'07,volume4637ofLNCS,pages236{255,2007.[14]N.CristianiniandJ.Shawe-Taylor.AnIntroductiontoSupportVectorMachines.CambridgeUniversityPress,2000.[15]C.Croux,P.Filzmoser,andM.R.Oliveira.Algorithmsforprojection-pursuitrobustprincipalcomponentanalysis.ChemometricsandIntelligentLaboratorySystems,87(2):218{225,2007.[16]N.Dalvi,P.Domingos,Mausam,S.Sanghai,andD.Verma.Adversarialclassi\fcation.InKDD'04pages99{108,2004.[17]I.DinurandK.Nissim.Revealinginformationwhilepreservingprivacy.InPODS'03,pages202{210,2003.[18]Y.Duan,J.Canny,andJ.Zhan.P4P:Practicallarge-scaleprivacy-preservingdistributedcomputationrobustagainstmalicioususers.InUSENIXSecuritypages207{222,2010.[19]C.Dwork.Di erentialprivacy.InICALP'06,pages1{12,2006.[20]C.Dwork.A\frmfoundationforprivatedataanalysis.Comms.ACM,54(1):86{95,2011.[21]C.DworkandJ.Lei.Di erentialprivacyandrobuststatistics.InSTOC'09,pages371{380,2009.[22]C.Dwork,F.McSherry,K.Nissim,andA.Smith.Calibratingnoisetosensitivityinprivatedataanalysis.InTCC'06,pages265{284,2006.[23]C.Dwork,F.McSherry,andK.Talwar.ThepriceofprivacyandthelimitsofLPdecoding.InSTOC'07pages85{94,2007.[24]C.Dwork,M.Naor,O.Reingold,G.N.Rothblum,andS.Vadhan.Onthecomplexityofdi erentiallyprivatedatarelease:ecientalgorithmsandhardnessresults.InSTOC'09,pages381{390,2009.[25]C.DworkandS.Yekhanin.Newecientattacksonstatisticaldisclosurecontrolmechanisms.InCRYPTO'08,pages469{480,2008.[26]R.A.Fisher.Question14:Combiningindependenttestsofsigni\fcance.AmericanStatistician,2(5):30{31,1948.[27]P.FoglaandW.Lee.Evadingnetworkanomalydetectionsystems:Formalreasoningandpracticaltechniques.InCCS'06,pages59{68,2006.[28]A.GlobersonandS.Roweis.Nightmareattesttime:Robustlearningbyfeaturedeletion.InICML'06pages353{360,2006. 57 [29]R.Hall,S.Fienberg,andY.Nardi.Securemultipartyinearregressionbasedonhomomorphicencryption.J.OcialStatistics,2011.Toappear.[30]F.R.Hampel,E.M.Ronchetti,P.J.Rousseeuw,andW.A.Stahel.RobustStatistics:TheApproachBasedonIn\ruenceFunctions.ProbabilityandMathematicalStatistics.JohnWileyandSons,1986.[31]M.HardtandK.Talwar.Onthegeometryofdi erentialprivacy.InSTOC'10,pages705{714,2010.[32]M.Kantarcioglu,B.Xi,andC.Clifton.Classi\ferevaluationandattributeselectionagainstactiveadversaries.TechnicalReport09-01,PurdueUniversity,February2009.[33]S.P.Kasiviswanathan,H.K.Lee,K.Nissim,S.Raskhodnikova,andA.Smith.Whatcanwelearnprivately?InFOCS'08,pages531{540,2008.[34]A.Kerckho s.Lacryptographiemilitaire.JournaldesSciencesMilitaires,9:5{83,January1883.[35]M.KloftandP.Laskov.Onlineanomalydetectionunderadversarialimpact.InAISTATS'10,2010.[36]A.Lakhina,M.Crovella,andC.Diot.Diagnosingnetwork-widetracanomalies.InSIGCOMM'04pages219{230,2004.[37]P.LaskovandM.Kloft.Aframeworkforquantitativesecurityanalysisofmachinelearning.InAISec'09pages1{4,2009.[38]C.LiuandS.Stamm.Fightingunicode-obfuscatedspam.InProceedingsoftheAnti-PhishingWorkingGroupsndAnnualeCrimeResearchersSummitpages45{59,2007.[39]D.LowdandC.Meek.Adversariallearning.InKDD'05,pages641{647,2005.[40]D.LowdandC.Meek.Goodwordattacksonstatisticalspam\flters.InCEAS'05,2005.[41]A.Machanavajjhala,D.Kifer,J.Gehrke,andM.Venkitasubramaniam.-diversity:Privacybeyond-anonymity.ACMTrans.KDD,1(1),2007.[42]M.V.MahoneyandP.K.Chan.Ananalysisofthe1999DARPA/LincolnLaboratoryevaluationdatafornetworkanomalydetection.InRAID'03,volume2820ofLNCS,pages220{237,2003.[43]F.McSherryandI.Mironov.Di erentiallyprivaterecommendersystems:buildingprivacyintothenet.InKDD'09,pages627{636,2009.[44]F.McSherryandK.Talwar.Mechanismdesignviadi erentialprivacy.InFOCS'07,pages94{103,2007.[45]T.A.MeyerandB.Whateley.SpamBayes:E ectiveopen-source,Bayesianbased,emailclassi\fcationsystem.InCEAS'04,2004.[46]T.Mitchell.MachineLearning.McGrawHill,1997.[47]B.Nelson,M.Barreno,F.J.Chi,A.D.Joseph,B.I.P.Rubinstein,U.Saini,C.Sutton,J.D.Tygar,andK.Xia.Exploitingmachinelearningtosubvertyourspam\flter.InLEET'08,pages1{9,2008.[48]B.Nelson,M.Barreno,F.J.Chi,A.D.Joseph,B.I.P.Rubinstein,U.Saini,C.Sutton,J.D.Tygar,andK.Xia.Misleadinglearners:Co-optingyourspam\flter.InJ.J.P.TsaiandP.S.Yu,editors,MachineLearninginCyberTrust:Security,Privacy,Reliability,pages17{51.Springer,2009.[49]B.NelsonandA.D.Joseph.Boundinganattack'scomplexityforasimplelearningmodel.InProc.WorkshoponTacklingComputerSystemsProblemswithMachineLearningTechniques,2006.[50]B.Nelson,B.I.P.Rubinstein,L.Huang,A.D.Joseph,S.honLau,S.Lee,S.Rao,A.Tran,andJ.D.Tygar.Near-optimalevasionofconvex-inducingclassi\fers.InAISTATS,2010.[51]B.Nelson,B.I.P.Rubinstein,L.Huang,A.D.Joseph,andJ.D.Tygar.Classi\ferevasion:Modelsandopenproblems(positionpaper).InProc.WorkshoponPrivacy&SecurityissuesinDataMiningandMachineLearning,2010.[52]J.Newsome,B.Karp,andD.Song.Paragraph:Thwartingsignaturelearningbytrainingmaliciously.InRAID,volume4219ofLNCS,pages81{105,2006.[53]L.RademacherandN.Goyal.Learningconvexbodiesishard.InCOLT,pages303{308,2009.[54]H.Ringberg,A.Soule,J.Rexford,andC.Diot.SensitivityofPCAfortracanomalydetection.InSIGMETRICS,pages109{120,2007.[55]G.Robinson.Astatisticalapproachtothespamproblem.LinuxJournal,Mar.2003.[56]B.I.P.Rubinstein,P.L.Bartlett,L.Huang,andN.Taft.Learninginalargefunctionspace:Privacy-preservingmechanismsforSVMlearning,2009.Insubmission;http://arxiv.org/abs/0911.5708v1[57]B.I.P.Rubinstein,B.Nelson,L.Huang,A.D.Joseph,S.honLau,S.Rao,N.Taft,andJ.D.Tygar.ANTIDOTE:Understandinganddefendingagainstpoisoningofanomalydetectors.InA.FeldmannandL.Mathy,editors,IMC'09,pages1{14,NewYork,NY,USA,November2009.ACM.[58]D.Sculley,G.M.Wachman,andC.E.Brodley.Spam\flteringusinginexactstringmatchinginexplicitfeaturespacewithon-linelinearclassi\fers.InTREC'06,2006.[59]A.Smith.Privacy-preservingstatisticalestimationwithoptimalconvergencerates.InSTOC'2011,pages813{822,2011.[60]S.J.Stolfo,W.jenLi,S.Hershkop,K.Wang,C.weiHu,andO.Nimeskern.Detectingviralpropagationsusingemailbehaviorpro\fles.InACMTrans.InternetTechnology,May2004.[61]L.Sweeney.-anonymity:amodelforprotectingprivacy.Int.J.Uncertainty,FuzzinessandKnowledge-basedSystems,10(5):557{570,2002.[62]K.M.C.Tan,K.S.Killourhy,andR.A.Maxion.Underminingananomaly-basedintrusiondetectionsystemusingcommonexploits.InRAID'02,volume2516ofLNCS,pages54{73,2002.[63]S.Venkataraman,A.Blum,andD.Song.Limitsoflearning-basedsignaturegenerationwithadversaries.InNDSS'08,2008.[64]D.WagnerandP.Soto.Mimicryattacksonhost-basedintrusiondetectionsystems.InCCS'02pages255{264,2002.[65]G.L.WittelandS.F.Wu.Onattackingstatisticalspam\flters.InCEAS'04,2004. 43 AdversarialMachineLearningIntelLabsBerkeleyAnthonyD.JosephUCBerkeleyadj@cs.berkeley.eduBlaineNelsonUniversityofTübingenblaine.nelson@wsii.uni-BenjaminI.P.RubinsteinMicrosoftResearchben.rubinstein@microsoft.comJ.D.TygarUCBerkeleytygar@cs.berkeley.eduABSTRACTInthispaper(expandedfromaninvitedtalkatAISEC2010),wediscussanemerging\feldofstudy:adversarialma-chinelearning|thestudyofe ectivemachinelearningtech-niquesagainstanadversarialopponent.Inthispaper,we:giveataxonomyforclassifyingattacksagainstonlinema-chinelearningalgorithms;discussapplication-speci\fcfac-torsthatlimitanadversary'scapabilities;introducetwomodelsformodelinganadversary'scapabilities;explorethelimitsofanadversary'sknowledgeaboutthealgorithm,fea-turespace,training,andinputdata;explorevulnerabili-tiesinmachinelearningalgorithms;discusscountermeasuresagainstattacks;introducetheevasionchallenge;anddiscussprivacy-preservinglearningtechniques.CategoriesandSubjectDescriptorsD.4.6[SecurityandProtection]:Invasivesoftware(e.g.,viruses,worms,Trojanhorses);I.5.1[Models]:Statistical;I.5.2[DesignMethodologyGeneralTermsAlgorithms,Design,Security,TheoryKeywordsAdversarialLearning,ComputerSecurity,GameTheory,IntrusionDetection,MachineLearning,SecurityMetrics,SpamFilters,StatisticalLearning1.INTRODUCTIONInthispaper,wediscussanemerging\feldofstudy:ad-versarialmachinelearning|thestudyofe ectivemachine ThispaperexpandsuponJ.D.Tygar'sinvitedtalkatAISec2010onAdversarialMachineLearningdescribingtheSecMLprojectatUCBerkeley,andincludesmaterialfrommanyofourcollaborators.Wekindlythankthisyear'sAISecorganizersforallowingustopresentthispaper.Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforprotorcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationontherstpage.Tocopyotherwise,torepublish,topostonserversortoredistributetolists,requirespriorspecicpermissionand/orafee.October21,2011,Chicago,Illinois,USA.Copyright2011ACM978-1-4503-1003-1/11/10...$10.00.learningtechniquesagainstanadversarialopponent.Toseewhythis\feldisneeded,islearning|thestudyofe ectivemachinelearningtechniquesagainstanadversarialoppo-nent.Toseewhythis\feldisneeded,itishelpfultorecallacommonmetaphor:securityissometimesthoughtofasachessgamebetweentwoplayers.Foraplayertowin,itisnotonlynecessarytohaveane ectivestrategy,onemustalsoanticipatetheopponent'sresponsetothatstrategy.Statisticalmachinelearninghasalreadybecomeanim-portanttoolinasecurityengineer'srepertoire.However,machinelearninginanadversarialenvironmentrequiresustoanticipatethatouropponentwilltrytocausemachinelearningtofailinmanyways.Inthispaper,wediscussbothatheoreticalframeworkforunderstandingadversar-ialmachinelearning,andthendiscussanumberofspeci\fcexamplesillustratinghowthesetechniquessucceedorfail.Advancesincomputingcapabilitieshavemadeonlinesta-tisticalmachinelearningapracticalandusefultoolforsolv-inglarge-scaledecision-makingproblemsinmanysystemsandnetworkingdomains,includingspam\fltering,networkintrusiondetection,andvirusdetection[36,45,60].Inthesedomains,amachinelearningalgorithm,suchasaBayesianlearneroraSupportVectorMachine(SVM)[14],istypicallyperiodicallyretrainedonnewinputdata.Unfortunately,sophisticatedadversariesarewellawarethatonlinemachinelearningisbeingappliedandwehavesubstantialevidencethattheyfrequentlyattempttobreakmanyoftheassumptionsthatpractitionersmake(e.g.,datahasvariousweakstochasticproperties;independence;asta-tionarydatadistribution).Thelackofstationarityprovidesampleopportunityformischiefduringtraining(includingperiodicre-training)andclassi\fcationstages.Inmanycases,theadversaryisabletopoisonthelearner'sclassi\fcations,ofteninahighlytargetedmanner.Forinstance,anadversarycancraftinputdatathathassimilarfeaturepropertiestonormaldata(e.g.,cre-atingaspammessagethatappearstobenon-spamtothelearner),ortheyexhibitByzantinebehaviorsbycraftingin-putdatathat,whenretrainedon,causesthelearnertolearnanincorrectdecision-makingfunction.Thesesophisticatedadversariesarepatientandadapttheirbehaviorstoachievevariousgoals,suchasavoidingdetectionofattacks,caus-ingbenigninputtobeclassi\fedasattackinput,launchingfocusedortargetedattacks,orsearchingaclassi\ferto\fndblind-spotsinthealgorithm.Adversarialmachinelearningisthedesignofmachinelearningalgorithmsthatcanresistthesesophisticatedat-tacks,andthestudyofthecapabilitiesandlimitationsof