jectpredictions.Thismeans,forexample,thatduringoneactivelearningloopah - PDF document

jectpredictions.Thismeans,forexample,thatduringoneactivelearningloopahumanmaybeaskedtonameanob-ject,whereasinthenexts/hemaybeaskedtostatewhetheraparticularattributeispresent.Thegoalistoselectthosepairsofimagesandlabelingquestionsthatwillbemostuse-fulgiventhecurrentmodels.Bysimultaneouslyweighingrequestsinbothlabelspaces,weexpectthelearnertomoreefcientlyreneitsobjectmodels.Why?Knowledgeofanattribute'spresenceinanimagecanimmediatelyinuencemanyobjectmodels,sinceattributesarebydenitionsharedacrosssubsetsoftheobjectcategories.Atthesametime,attributes'presenceorabsenceinanimageisoftencorrelated(e.g.,ifsomething“hasskin”itisunlikelytobe“metallic”aswell),suggest-ingthatmanyimagesdonotrequireafullannotationofallattributes.1SeeFig.1.Toimplementtheproposedidea,weadoptadiscrimi-nativelatentmodel[29]thatcapturesobject-attributeandattribute-attributerelationships,andthendeneasuitableentropyreductionselectioncriteriontopredicttheinu-enceanewlabelofeithertypewillhavethroughoutthoseconnections.Thiscriterionestimatestheexpectedentropychangeonalllabeledandunlabeledexamples,shouldthelabelunderconsiderationbeobtained.Weadapttheexist-ingclassiertoextractthenecessaryposteriorprobabilityestimates,andshowhowtohandlepartiallylabeledexam-ples(i.e.,thosewithonlysomeattributesknown)suchthattheycanhaveimmediateinuenceontheactiveselection.Anovelaspectofourapproachisthatitbothweighsdifferentannotationrequestsandalsomodelsdependencieswithinmulti-labelinstances.Onlylimitedpriorworkex-ploreseitheroneortheotheraspect[25,23,17],andinadifferentcontextthanoursettinghere.Furthermore,incontrasttoanyexistingactivelearningwork,ourapproachexploitsdependenciesbetweenthetargetlabelspaceandalatentbuthuman-describablelabelspace,andisthersttolearnobjectsandattributesactivelyinconcert.Thiscanalsobeviewedasanewwaytoefcientlysupervisejointmulti-classtraining,inthattheactivelyselectedattributela-belingquestionsaredirectlytiedtopropertiessharedacrossclasses.2.RelatedWorkRecentworkexploresseveralwaystousevisualat-tributesinobjectrecognition.Sinceattributesaresharedacrosscategories,theyenableknowledgetransfertorecog-nizeunseenobjects[15]anddescribenovelinstances[8].Byintegratingthelearningprocessforbothobjectsandat-tributes,onecanuseweaksupervisionmoreeffectively[27]andevenimproveobjectrecognitionaccuracy[14,29].In 1Infact,blindlyrequestingallattributesmaynotonlybewasteful,butitmayalsobeimpracticalattheinterfacelevelonceweconsiderlargeattributevocabulariesofhundredsormoreproperties.addition,capturingtherelationshipsbetweenattributescanstrengthenobjectcategorymodels,asrstshownbyWangandMori[29].Weemploytheirlatentdiscriminativemodelforclassication,asitsuitablyrepresentsalltheobjectandattributeinteractionsofinteresttoouractivelearningap-proach.Mostworkusingattributesassumesthatimagesarefullylabeledwithalltheirattributes,eitherthroughatop-downlabelingoftheobjectclasses(e.g.,allbearsare`furry'[15])orbyindividuallyprovidingattributesforeachimage[8,6].Toalleviatethisburden,researchersstudywaystolearnat-tributeclassiersfromnoisykeywordsearchdata[10],ortoautomaticallydiscovertheattributesandobjects'semanticrelatednessfromWebimagesandtextsources[1,19].Incontrasttotheseunsupervisedmethods,ourworkexplicitlyengagesahumanannotatortorespondtoobjectorattributequerieswheremostneeded.Activelearningforobjectrecognitiontypicallyreduceshumanlabelingeffortbyselectingthemostuncertainex-emplartogetlabeledwithitsobjectcategoryname(s)[17,31,13,12].Morecloselyrelatedtoourapproach,someworkfurthershowshowtoactivelyintegrateannotationsofdifferentlevels,i.e.,byalternatelyrequestingsegmentedre-gionsoraskingaboutthecontextualrelationshipsbetweenobjectsinanimage[25,26,23].Intherealmofnaturallanguageprocessing,researchersdevelopwaystoactivelyaskhumanswhichwordsmayberelevantforadocumentclassicationtask[18,5];wordscouldbeseenasalooseanalogueforattributes,thoughwedonotconsiderrequestsaboutvisualattributerelevance.Activevisuallearningmethodsgenerallydonotaccountforthedependenciesbetweenlabelsonthesameimage.Anexceptionisthesceneclassicationmethodof[17],whichlearnswithmulti-labelimagesandrequeststhemostin-formativeimage-labelpair.However,itsselectionstrategyconsidersonlythelocaleffectsofacandidatelabelrequest,bymeasuringtheuncertaintyandlabelcorrelationsforeachindividualimageinisolation.Incontrast,theproposedse-lectionmethodevaluatestheinuenceofthecandidatela-belifpropagatedtoallcurrentmodels,whichiscriticaltoachieveourgoalofexploitingsharedlatentattributestore-duceannotationeffort.Whiletheaboveworktacklesactivelearning,asystemforactiveclassicationisdevelopedin[2],wherethesys-teminteractivelydeducestheobjectlabelforasinglenovelimagebyaskingahumantolabelasequenceofitsat-tributes.Incontrast,ourmethodusesthehumanannotatorsduringtheiterativetrainingprocessforallobjectcategories,andthenmakespredictionsonnovelimageswithouthumanintervention.Furthermore,ourmethodrequestsinforma-tionfromtheannotatorsontwolevels(objectandattributelabels),whereasthemethodin[2]onlyrequestsattributelabels. Figure2.Overviewofourapproach.Left:thecurrentmodelisdeterminedbywhateverlabeledorpartiallylabeledobjectandattributedataisavailable.Usingthatclassier,wescoreallhimage,labelrequestipairsintheunlabeledpoolaccordingtotheirexpectedentropyreduction.Center:TheNtopscoringpairsarepresentedtoanannotatorwiththetargetedobjectorattributequestion.Right:Dependingontheanswersandlabeltypes,theannotatorresponsesinuencedifferentcomponentsofthefullmodel,assigniedbythetwosetsofdottedarrows.NotethatthefourrightmostboxesparallelthefourtermsofthemainmodelinEqn.3.Loop:Finally,weloopback,andrepeattheselectionprocessusingthenewlystrengthenedmodel.Bestviewedincolor.3.2.1AnnotationSetDenitionsLetLdenoteanylabeledorpartiallylabeledtrainingdata.Duetothedifferenttypesofannotationsandclassiersincorporatedbythefullmodeloutlinedabove,wemustmaintainseveralseparatetrainingsets.Assuch,wethinkofLascontainingseveral(potentiallyoverlapping)sets:L=fT;TO;TA1;:::;TAK;TAg,whereTcontainsfullylabeledimagesusedtotrainthefullmodelw,TOcontainsobject-labeledimagesusedtotraintheobjectclassierthatyieldsfeature,eachTAmcontainsattribute-labeledim-agesusedtotraintheattributeclassierthatyieldsfeature',andTAcontainsattribute-labeledimagesusedtocom-putetheattributerelationshipgraph.Notethatanannota-tioninLnTstillaffectsthefullmodel,becauseitalterstheinnercomponentsontopofwhichwislearned.LetUdenoteallunlabeled(orpartiallyunlabeled)data.Similartoabove,wemaintainseparatesetsaccordingtothelabel“state”ofagivenexample:U=fUO;UAg,whereexamplesinUOhavenoobjectlabel,andexamplesinUAlackoneormoreattributelabels.3.2.2Entropy-BasedSelectionFunctionAttheonset,wearegivensomeinitialpooloflabeleddatainL.Ateachiterationofactivelearning,weneedtode-cidewhichimagetohaveannotatedandwhichannotationtorequestforit.Thus,wemustrankthepoolofcandidatehimage,labelrequestipairsinU.Akeypointinourap-proachisthatforagivenimage,thereareK+1optionsforthelabelquery;itiseithertheobjectclass,oroneoftheKattributes.Tothisend,wedeneaselectionfunctionthatscorestheexpectedentropyreductionforacandidaterequest.Let(y(i);h(i)1;:::;h(i)K)denotethefulllabelsforthei-thimagex(i).ThetotalentropyoveralllabeledandunlabeleddatagiventhelabeleddataLisdenedas:H(L)=�jL[UjXu=1jYjXl=1PLy(u)=ljx(u)logPLy(u)=ljx(u);(4)wherePL(yjx)denotestheposteriorestimatesobtainedwhenthemodelistrainedwithlabeleddataL.NotethatentropyismeasuredoverobjectpredictionsY,asthatistheultimatetargetlabelspace.Ingeneral,theunlabeledinstancethatmaximizestheex-pectedentropyreduction[22,20]is:x=argmaxx2U�H(L)�jYjXl=1PL(y=ljx)H(L[hx;li)
;(5)orequivalently,ifwedroptheconstantcurrententropyvalue:x=argminx2U�jYjXl=1PL(y=ljx)H(L[hx;li)
:(6)Inourcase,wemustconsiderexpandingLbyeithertheobjectlabelyoranattributelabelhm.Thuswedenetwointermediateentropyscoringterms:SY(x(i))=jYjXl=1PL(y(i)=ljx(i))H(L[hx(i);y(i)=li);(7)whereposteriorsareobtainedwiththefullobjectmodel,andSHm(x(i))=jAjXa=1PL(h(i)m=ajx(i))H(L[hx(i);h(i)m=ai);(8)whereposteriorsareobtainedfromthemodel'sinnerat-tributeclassier'.Inboth,Lreferstothecurrentlabeleddata.NotethatSYandSHmarecomparableinthattheybothreecttheentropyoftheobjectlabelprediction.Finally,thebestimageandlabelrequestisgivenby:(x;q)=argminx2U;q2fY;H1;:::;HKgSq(x):(9)ThelowerthescoreinEqn.9,themoreinuenceweexpectthelabelrequesttohaveonthecompletemodel.Becauseweconsidertheimpactofacandidatelabelingoverallthe Algorithm1Theproposedactivelearningapproach. 1:Given:labeleddataL=fT;TO;TA1;:::;TAK;TAg,andpoolofunlabeleddataU=fUO;UAg.2:Computeinitialattributerelationshipgraph(V;E).3:ComputefeaturesandtraininitialmodelusingL.4:whileLabelingeffortstillavailabledo5:ComputeSY(x)forallimagesinUO.6:ComputeSHm(x)forallimagesinUA,forallyet-unlabeledattributesamongm=1;:::;K.7:SelecttheNmostinformativeimage-labelpairs(Eqn.9),andaskhumanannotator.8:Removeobject-annotatedimagesfromUO.9:Removefullyattribute-annotatedimagesfromUA.10:Addnewobject-annotatedimagestoTO.11:AddimageswithnewlabelsforattributemtoTAm.12:Infervaluesforanymissingattributelabelsforpar-tiallylabeledimagesinUA\TO.13:Addthoseandfullyattribute-labeledimagestoTA.14:AddimagesinTO\TAtoT;removethemfromU.15:Recomputeinnerclassiers'“features”usingL,up-dateattributegraph.16:RetrainthefullmodelwusingT.17:endwhile dataandmodelcomponents,thisselectionfunctionrevealswhichattributeorobject-basedquestionismostvaluable,achievingtheintuitiongiveninFig.1.Inordertocomputetheobjectclassposteriorprobabil-itiesrequiredforentropy,wedesignamappingfromtherawfwfunctionoutputstomulti-classprobabilities.FirstweestimatethepairwiseprobabilitiesforanytwoobjectclasseslA,lB2Y,byttingasigmoidtooutputvaluesforfw(x;y=lA)�fw(x;y=lB)onthetrainingdatainT.ThedifferencebetweenoutputvaluesmimicstheformofthelatentSVMlabelconstraints.Thenweusethepair-wisecouplingapproach[30]toobtainmulti-classproba-bilitiesfromthesepairwiseprobabilities.Inthisway,weessentiallyadaptPlatt'smethod[16]toaccommodatelatentmulti-classSVMoutputs.Fortheattributeposteriors,wesimplyusePlatt'smethodonthebinarySVMscores.Procedurally,computingthebestrequestrequirescy-clingovertheunlabeledorpartiallylabeledimages.Then,foreachlabelrequestwecouldmakeforthecurrentim-age,wecycleovereachpossiblelabelresponse,and(1)addittothelabeledsettemporarily,(2)retrainthemodel,(3)evaluateentropyunderthenewmodel,and(4)weighttheresultingentropybytheprobabilityofthehypothesizedlabelundertheoldmodel.Torequestmorethanonelabelperiteration,wesimplytaketheNquerieswiththelowestSqscores.Thus,onebatchadditionmayincludenewlabelsforbothobjectsandanyoftheattributes,andagivenimagemayreceivemultiplelabels.3.2.3UpdatestoLabeledandPartiallyLabeledSetsFinally,wedetailtheimplicationsthattheabovestrategyhasontheretrainingstep,whetheraddingtrueorhypothe-sizedlabels.Recallthatthemodelweareactivelylearninghastwostagesoftraining:therstupdatestheinnercomponents(e.g.,independentobjectorattributeclassiers),whilethesecondupdatesthe“outer”mainparametersofw(seeEqn.3).Updatestoeitherofthetwoannotationtypesdonotaffectallinnercomponentsofthemodelatthesametime,buttheydoalwaysaffectthefullobjectpredictionmodelparameters.Inparticular,newobjectlabelsareinsertedintoTO,anddirectlyaffectboththeobjectclassierandlearnedobject-attributeinteractionterms.Newattributelabelsforthem-thattributeareinsertedintoTAm,anddirectlyaffectthem-thattributeclassier,theattribute-attributerelation-shipgraphG=(V;E),andtheobject-attributeinteractionterms.Thesedependenciesarereectedbythedottedar-rowsintheexampleshowninFig.2.Therefore,whenwereceiveanewobjectlabel,weaddittoTOandremoveitfromUO.Whenwereceiveanewlabelforattributem,weaddittoTAm;however,itisnotremovedfromUAuntilallotherattributesforthatimageareobtained.Ifanewlabelhappenstocompletealllabelsforagivenimage(i.e.,x(i)2TO\TA),weremoveitfromUandinsertitintoT.Intermsofupdatingtheattributerelationshipgraph,ifanobject-labeledimageinTOhasonlypartialattributelabels,therearetwooptions:(1)aconservativeapproach,wherewesimplywaituntilallattributelabelsarepresentbeforeaddingittoTA,or(2)apartialapproach,whereweaddtheimagetoTAwithitsmissingattributelabelsinferred.Toinferthemissinglabels,weaddconstraintsinthelin-earprogrammingproblemthatsolvesforhtoreectthatanyknownattributesshouldbeassignedtheircorrectlabels.Afterinferringtheselabels,wetreatthemasobserveddur-ingtraining.Forthepartialapproach,wekeeptheimageinUA,sothatitsmissinglabelsmaystillbeaddedbyahumanannotator(ifselectedwithactivelearning).Wepursuethispartialformulationinourexperiments,asweexpectmoreimmediateimpactofnewlabelstohelptheactivelearner.Notethatthepartialupdatepolicyisapplicablewhetherweareintroducinganewlylabeledinstancereceivedfromanannotator(i.e.,attheendofanactivelearningloop),ortemporarilyupdatingthemodelduringtheselectionpro-cess.Intheformertheyarepermanentupdates,whileinthelattertheyareremovedappropriatelyafterthenecessaryposteriorsarecomputedforEqn.9.Oncewehaveupdatedthetrainingandunlabeledsetsaccordingly,weretraintheinnerclassiers,computetheirfeatures,andretrainthefullmodel.SeeAlg.1forarecapofthemethod. Figure3.Representativelearningcurvesfromallthreedatasetsshowingthepredictedprobabilityofthecorrectlabelonaheld-outtestsetwithincreasingnumberoflabelsobtained:rstthreearebest,andfourthrepresentsafailurecase.Ourapproachisconstantlymoreaccuratethanthebaselines,indicatingthatjointlyselectingfromobjectandattributelabelsisworthwhile.(Highercurvesarebetter.) Figure4.Entropyofalltrainingandunlabeledexampleswithincreasingnumberofobtainedlabelsforourapproachincomparisontooptimalselectionandthebaselines(rstthreearegood,andlastisafailurecase).Asexpected,ourapproachreducestheoverallclassicationuncertaintyfasterthanthebaselinesandsimilarlytooptimalselection.(Lowercurvesarebetter.)themeanentropyonL[Uasmorelabelsareaddedforthethreeapproachesandtheoptimalselection.Wewanten-tropytodecreaseasmoretrainingdataisadded,solowercurvesarebetter.Optimalselectionshowstheresultofus-inggroundtruthinformationinordertocomputethebestpossibleselectionbasedonentropy.6Theoverallentropydecreasessteadilywithmorelabelsforbothourapproachandtheoptimalselection,showingthattheclassierisabletobetterseparatetheexamplesintothedifferentclassesbyjointlylearningfromobjectandattributelabels.Notethatourapproachperformsquitesimilarlytooptimalselection.Thesameisnottrueforthebaselines,wherethereductionintheoveralluncertaintyisslower.ThelastgureinFig.4showsacasewhereentropyispoorlyestimated—compareourresultversustheoptimalselectioninthefourthplot.ThisexplainsthefailurecaseforthetestsetlearningcurvesinthefourthplotinFig.3.QualitativeResultsToexamineinmoredepththese-lectionsmadebyouractivelearningapproach,wepresentsomequalitativeresults.InFig.5,weshowthedistributionoflabelrequestsfortheobjectandeachoftheattributela-bels.Weseethatthemajority(75%)ofrequestsareforattributelabels.Thereisaslighttendency(notshown)thatmoreobjectlabelsarerequestedearlierintheactivelearn-ingloop.Weseethatthedistributionfora-Yahooistheleastbalancedone,whichmightindicateaparticularrela-tionshipbetweenobjectsandattributesthatwouldexplaintheweakerperformanceofourmethodonthisdataset.InFig.6,weshowsomesamplerequeststhatweremadebyouralgorithmforAwA-1. 6Thisresultwouldstrictlybeanupperboundtoourapproach,butsincemultiplelabelsareaddedatatime,theentropyreductionpredictedforindividuallabelsandtheactualentropyreductioncandiffer. Figure5.Distributionofrequestsperlabeltype(object/attributes).5.ConclusionsWeproposeamethodforactivelyselectingthebestob-jectorattributelabelsonimagesinawaythatcansimul-taneouslyaffectmultipleobjectcategories.Ourresultsonthreechallengingdatasetsindicatethatourmethodisin-deedabletolearnmorequicklythaneitherpassivelearningorastrongbaselineapproachthatcanonlyrequestobjectlabels.Theproposedstrategycanbeseenasameanstoenhancemulti-classobjectcategorylearning,byefcientlystrengtheningmodelsthroughsharedattributes.Asfuturework,wewouldliketoaddathirdrequesttypewhichexplicitlyasksabouttherelationshipbetweenobjectsandattributes.Wealsoplantoinvestigatealternativemea-suresofuncertaintyreductionandstrategiesformakingourselectionapproachcomputationallyscalable.AcknowledgementsWethankYangWangforhelpfuldiscussions.Thisre-searchwasfundedinpartbygrantONRATLN00014-11-1-0105,theHenryLuceFoundation,andgrantNSFEIA-0303609. Figure6.Samplehimage,labelirequeststhatourmethodgeneratesforAwA-1.The1strequestmaybeexplainedbythelackofdarkbrownhamstersinthetrainingset.The2ndand3rdrequestsareduetothesimilaritytoclassesthathavetheattributesinquestion.The4thand5threquestsshowanimagewhichconfusesthesystemandappearsinmultiplelabelingrequests.The6thimageislikelyambiguousbecauseitviolatestheassumptionofoneobjectperimage.References[1]T.L.Berg,A.C.Berg,andJ.Shih.Automaticattributedis-coveryandcharacterizationfromnoisywebdata.InECCV,2010.[2]S.Branson,C.Wah,F.Schroff,B.Babenko,P.Welinder,P.Perona,andS.Belongie.Visualrecognitionwithhumansintheloop.InECCV,2010.[3]J.Deng,W.Dong,R.Socher,L.-J.Li,K.Li,andL.Fei-Fei.ImageNet:Alarge-scalehierarchicalimagedatabase.InCVPR,2009.[4]T.-M.-T.DoandT.Artieres.Largemargintrainingforhid-denMarkovmodelsandpartiallyobservedstates.InICML,2009.[5]G.Druck,B.Settles,andA.McCallum.Activelearningbylabelingfeatures.InEMNLP,2009.[6]I.Endres,A.Farhadi,D.Hoiem,,andD.Forsyth.Thebene-tsandchallengesofcollectingricherobjectannotations.InACVHL(inconjunctionwithCVPR),2010.[7]M.Everingham,L.VanGool,C.K.I.Williams,J.Winn,andA.Zisserman.ThePascalvisualobjectclasses(VOC)challenge.IJCV,88(2):303–338,June2010.[8]A.Farhadi,I.Endres,D.Hoiem,andD.Forsyth.Describingobjectsbytheirattributes.InCVPR,2009.[9]P.Felzenszwalb,R.Girshick,D.McAllester,andD.Ra-manan.Objectdetectionwithdiscriminativelytrainedpartbasedmodels.PAMI,32(9),September2010.[10]V.FerrariandA.Zisserman.Learningvisualattributes.InNIPS,2007.[11]A.GuptaandL.Davis.Beyondnouns:Exploitingprepo-sitionsandcomparativeadjectivesforlearningvisualclassi-ers.InECCV,2008.[12]P.JainandA.Kapoor.Activelearningforlargemulti-classproblems.InCVPR,2009.[13]A.Joshi,F.Porikli,andN.Papanikolopoulos.Multi-classactivelearningforimageclassication.InCVPR,2009.[14]N.Kumar,A.C.Berg,P.N.Belhumeur,andS.K.Nayar.Attributeandsimileclassiersforfaceverication.InICCV,2009.[15]C.H.Lampert,H.Nickisch,andS.Harmeling.Learningtodetectunseenobjectclassesbybetween-classattributetrans-fer.InCVPR,2009.[16]J.C.Platt.Probabilisticoutputforsupportvectormachinesandcomparisonstoregularizedlikelihoodmethods.InAd-vancesinLargeMarginClassiers,1999.[17]G.-J.Qi,X.-S.Hua,Y.Rui,J.Tang,andH.-J.Zhang.Two-dimensionalactivelearningforimageclassication.InCVPR,2008.[18]H.Raghavan,O.Madani,andR.Jones.InterActivefeatureselection.InIJCAI,2005.[19]M.Rohrbach,M.Stark,G.Szarvas,I.Gurevych,andB.Schiele.Whathelpswhere-andwhy?semanticrelat-ednessforknowledgetransfer.InCVPR,2010.[20]N.RoyandA.McCallum.Towardoptimalactivelearningthroughsamplingestimationoferrorreduction.InICML,2001.[21]B.Russell,A.Torralba,K.Murphy,andW.Freeman.La-belMe:adatabaseandweb-basedtoolforimageannotation.IJCV,77:157–173,May2008.[22]B.Settles.Activelearningliteraturesurvey.ComputerSciencesTechnicalReport1648,UniversityofWisconsin–Madison,2009.[23]B.SiddiquieandA.Gupta.Beyondactivenountagging:Modelingcontextualinteractionsformulti-classactivelearn-ing.InCVPR,2010.[24]A.SorokinandD.Forsyth.UtilitydataannotationwithAmazonMechanicalTurk.InCVPRWorkshoponInternetVision,2008.[25]S.VijayanarasimhanandK.Grauman.Multi-levelactivepredictionofusefulimageannotationsforrecognition.InNIPS,2008.[26]S.VijayanarasimhanandK.Grauman.What'sitgoingtocostyou?:Predictingeffortvs.informativenessformulti-labelimageannotations.InCVPR,2009.[27]G.WangandD.Forsyth.Jointlearningofvisualattributes,objectclassesandvisualsaliency.InICCV,2009.[28]Y.WangandG.Mori.Max-marginhiddenconditionalran-domeldsforhumanactionrecognition.InCVPR,2009.[29]Y.WangandG.Mori.Adiscriminativelatentmodelofob-jectclassesandattributes.InECCV,2010.[30]T.-F.Wu,C.-J.Lin,andR.C.Weng.Probabilityestimatesformulti-classclassicationbypairwisecoupling.InJour-nalofMachineLearningResearch,volume5,pages975–1005,December2004.[31]C.ZhangandT.Chen.Anactivelearningframeworkforcontent-basedinformationretrieval.IEEETransactionsonMultimedia,4(2),June2002. ActivelySelectingAnnotationsAmongObjectsandAttributesAdrianaKovashkaSudheendraVijayanarasimhanKristenGraumanUniversityofTexasatAustinadriana,svnaras,graumanWepresentanactivelearningapproachtochooseim-ageannotationrequestsamongbothobjectcategorylabelsandtheobjects'attributelabels.Thegoalistosolicitthoselabelsthatwillbestusehumaneffortwhentrainingamulti-classobjectrecognitionmodel.Incontrasttopreviousworkinactivevisualcategorylearning,ourapproachdi-rectlyexploitsthedependenciesbetweenhuman-nameablevisualattributesandtheobjectstheydescribe,shiftingitsrequestsineitherlabelspaceaccordingly.Weadoptadis-criminativelatentmodelthatcapturesobject-attributeandattribute-attributerelationships,andthendeneasuitableentropyreductionselectioncriteriontopredicttheinuenceanewlabelmighthavethroughoutthoseconnections.Onthreechallengingdatasets,wedemonstratethatthemethodcanmoresuccessfullyaccelerateobjectlearningrelativetobothpassivelearningandtraditionalactivelearningap-proaches.1.IntroductionManystate-of-the-artobjectrecognitionsystemsinte-graterobustvisualdescriptorswithasupervisedlearningalgorithm.Thisbasicframeworkentailshavinghumans“teach”themachinelearneraboutobjectsthroughlabeledexamples,whichmakesthedatacollectionprocessitselfofcriticalimportance.Assuch,recentresearchexploresinterestingissuesingatheringlargedatasetsofWebim-ages[21,24,10,3],miningexternalknowledgesources[19,1,2],creatingbenchmarkchallenges[7],anddevelopingnewmethodstoreducetheexpenseofmanualannotations.Activelearningmethodsinparticularareapromisingwaytofocushumaneffort,asthesystemcanrequestlabelsonlyforthoseinstancesthatappearmostinformativebasedonitscurrentcategorymodels[17,25,26,13,12,23].Inspiteofsuchprogress,however,substantialchallengesremain.Firstofall,mostexistingtechniquesassumethatthelabelsofinterestaretheobjectcategorynames,yetre-centworkshowstheneedtomove“beyondlabels”toevenricherannotationssuchasdescriptiveattributesorrelation- Figure1.Objectandattributelabelsaffectthecurrentmodel'sunderstandingofeachtrainingimageindistinctways.Thisex-ampleillustrateshowthedifferentlabelrequestsabouttheimage(left)willinuencethedifferentcomponentsofthelearnedmodels(right,colorcodedbytypeofimpact).Forexample,whereasget-tingthe`panda'labelmayreduceuncertaintyaboutthatclassandrenethemodel'sdistinctionswithotherbearclasses(top),gettingthe'spotted'labelcouldhaveevengreaterinuence,strengtheningdiscriminabilityforthestripedandspottedattributesalike.shipsbetweenobjects[15,14,6,29,10,11].Attributesarehigh-levelfeaturesthatdescribetraitsofanobjectsuchasphysicalproperties,behavior,oruses;forexample,whileobjectlabelsmightinclude,anddog,attributelabelsmightinclude,orred.Secondly,real-worldapplicationsofobjectrecognitiondemandscalingtoaverylargenumberofcategories,whichatthesurfacesug-geststhatthenumberoflabelsneededmustgrowpropor-tionallywiththenumberoftotalclassesconsidered—evenifoneplanstocollectlabelswithactivelearning.Weproposeanactivelearningapproachtoaddresstheseissues.Themainideaistoactivelyselectimageannota-tionrequestsamongobjectcategorylabelsaswellastheobjects'sharedattributes,soastoacquirethelabelsex-pectedtomostreducetotaluncertaintyformulti-classob-