Content-BasedImageRetrievalwithLIReandSURFonaSmartphone-BasedProductIm - PDF document

Content-BasedImageRetrievalwithLIReandSU
Content-BasedImageRetrievalwithLIReandSURFonaSmartphone-BasedProductImageDatabaseKaiChenandJeanHennebertUniversityofFribourg,DIVA-DIUF,Bd.deP´erolles90,1700Fribourg,Switzerland232K.ChenandJ.HennebertFig.1.GeneraloperationsofaCBIRsystemusingtheFPIDdatabaseManyCBIRsystemshavebeenproposedanddescribedintheliterature,forexam-pleQBIC[4],GIFT[6],andFIRE[3].Inourwork,wehavechosentousetheopensourcejavaLuceneImageRetrieval(LIRe)library1[5].LocalfeatureSURF(Speeded-UpRobustFeatures)[1]isalsoemployedfortheproductidenticationtask.Inthispaper,wereportontheevaluationofLIReandSURFusingtheFribourgProductIm-ageDatabase(FPID)thathasbeenreleasedrecently[2].FPIDisasmartphone-basedimagedatabase.Itcontainsmorethan3’000picturesofconsumerproductscapturedinsupermarketswithvariousregularsmart-phonecameras.UsingtheevaluationprotocolproposedwithFPID,weexploretheperformanceofdifferentpreprocessingandfeatureextractionusingLIReandSURF.Comparedwithglobalfeaturemethods,SURFtakesmoretime.Ontheotherhand,thesizeofimagehasahugeimpactonthetimetakenforSURFapproach.Therefore,imageresizingandfeatureindexingapproachesareusedtospeedupSURFinoursystem.Theexperimentalresultsdemonstratetheeffectivenessoftheproposedmethodfortheproductidenticationtask.Thispaperisorganizedasfollows.WeintroducetheFPIDsmartphone-basedim-agedatabaseandourCBIRproto

colinSection2.InSection3wepresentbaseli
colinSection2.InSection3wepresentbaselineCBIRperformanceaswellasseveralimprovementsthatwecouldobtainthroughtheparametersofthefeatureextractionandpreprocessingoftheimages.Section4presentsconclusionsandfutureworks.2FPIDandCBIREvaluationProtocolTheevaluationofCBIRsystemsrequirestwoelements.First,adatabaseofreferenceimagesmustbeprovidedtogetherwithveriedgroundtruthvaluesforeachimages.Second,anevaluationprotocolmustbeclearlydened,sothatdifferentteamscanruntheiralgorithmsandcomparetheirresults.Fortheworkreportedhere,weusedthe”FribourgProductImageDatabase”(FPID)[2].Thisdatabasehasbeenrecentlyreleasedtothescienticcommunity.Cur-rently,FPIDcontainsmorethan3’000picturesofretailproductsthatcanbefoundin1http://www.semanticmetadata.net/lire/Content-BasedImageRetrievalwithLIReandSURFonFPIDImageDatabase233SwissandEuropeansupermarkets.Thesetofimagescoversabout350productsspreadinto3families:bottledwater,coffee,andchocolate.Eachproducthasatleastoneimageinthedatabaseandthemostpopularproductshaveabout30images.Theimageshavebeencapturedusingvariousmobilephonesindifferentsupermarketswithoutanycon-troloftheillumination.Foridenticalproducts,theimagefeaturesmaythereforeshowalargevariability.Thegroundtruthinformationistheproductlabelexpressedasachar-acterstring,i.e.,iftwoimageshavethesameproductlabel,thentheyareconsidered

asrelevantforaCBIRtask.Theproductlabelis
asrelevantforaCBIRtask.Theproductlabelisacharacterstringuniquelyidentifyingtheproductbrandandmodel.Thegroundtruthalsoincludesthemobilephonebrand/-model,theshopnameanditslocationthatallowsforsomeadvancederroranalysis.SomeimagestakenfromFPIDareillustratedonFigure2.(a)169,Nokian95,ManorFribourg(b)497,Samsungg600,CoopFribourg(c)1052,Sonyericsonw880i,MigrosFribourg(d)1216,Sonyeric-ssonw880i,MigrosFribourg(e)23,nokian95,Mi-grosFribourg(f)2041,NokiaN78,ManorFribourgFig.2.ExampleimagesfromFPIDwiththeimageid,thedevicenameandlocationoftheacquisitionInthispaper,wefollowstrictlytheevaluationprotocolsforproductidenticationthatareproposedwithFPID[2].Theseprotocolsarebasedonasubsetof1200imagesincluding100differentproductswith12imagesperproduct.Fromtheset,disjoint234K.ChenandJ.HennebertsetsTandQfortrainingandqueryaredened.Theprotocolsaresaidtobeclosed-setasallqueryimagesbelongtoaproductcategorythatisrepresentedinthetrainingsets.Inotherwords,theproposedprotocolsdonotevaluaterejectionperformancesofCBIRsystemswhereaqueryimagehaszerocorrespondingrelevantimages.AsillustratedinFigure3,differenttrainingsetsTkaredened,krepresentingthenumberofimagesperproductintheset.Allthetrainingsetsarebalancedwith,e.g.,thetrainingsetT4containingexactly4imagesperproductsforatotalof400images.Listsofimagesareprovidedonthewebsit

eofFPID2forT1,T2,...,T10.Inasim
eofFPID2forT1,T2,...,T10.Inasimilarmanner,aquerysetQ2including200imageswith2imagesperproductisdened.Q2isofcoursedisjointtoallthetrainingsetsTn.Fig.3.FPIDevaluationprotocolsWithFPID,itisproposedtoreportsystemperformanceusingtherecognitionrateRRI(n)consideringthen-bestretrievedsetofimages.TherateRRI(n)iscomputedastheratioofpositivematchesdividedbythetotalnumberofqueries.Amatchiscon-sideredpositivewhen,foragivenqueryimage,thereexistsatleastonerelevantimageintheretrievedn-bestsetofimages.Ifthereisnorelevantimageintheretrievedn-bestsetofimagesthenthisisamiss.Increasingthevalueofninthen-bestretrievedsetofimageswillmakethetaskeasier.Whennisequalto1,therateRRI(1)isactuallyequaltotheprecisionat1orP@1,frequentlymeasuredwhenbenchmarkingCBIRsystems.Fortheexperimentsreportedinthiswork,weusedn{1,2,5,10,15,20}.Inasimi-larmanner,wealsomeasuretherecognitionrateRRP(n)whichistherecognitionrateconsideringthesetofn-bestretrievedproductcategories.Inthiscaseweretrieve,for2http://diuf.unifr.ch/diva/FPID/Content-BasedImageRetrievalwithLIReandSURFonFPIDImageDatabase235agivenqueryimage,thesetofn-bestimagesinwhichwekeeponlyonerepresentingimageperproduct,theotheronebeingdiscarded.3SystemDescriptionandResultsOurCBIRsystemisbasedontheopen-sourcelibraryLIReandSURF.The

LIReli-braryoffersdifferentfeatureextrac
LIReli-braryoffersdifferentfeatureextractionpossibilitiesthatwehaveexploredinthiswork.OneofthedifcultiesofCBIRsystemsisindeedtoselectthemostsuitablefeatureextractiontechniqueregardingthespecicitiesofagiventask.WehavealsoexploredtheimpactofthevalueontheperformancesasdescribedinSection2.Weappliedsomemeaningfulpreprocessingoftheimagesandmethodofchoosingmostsuitablefeaturethathaveleadtoimprovementsoverourbaselineresults.TheSURFisarobustalgorithmforlocal,similarityinvariantrepresentationandcomparison.Itoutper-formedallglobalfeaturesavailableinLIRe,butontheotherhanditsuffersthespeed.Weobservedthatthesizeofimagehasadirectimpactontimetaken.Imageresizingandfeatureindexingapproachesaretakentospeedupthetask.AllourexperimentsareachievedinDALCOHighPerformanceLinuxClusterwith64GBperComputenodeintheUniversityofFribourg.GlobalFeaturesComparison.SeveralglobalfeatureextractionareavailableinLIRe.Figure4showstheperformancesforthedifferentfeaturesandtheevolutionofasafunctionofinthesetofn-bestretrievedimages.InLIRe,imageispresentedinfeaturevectors.Foragivenqueryimage,atrainingsetandfeature,wecomputethedistanceq,i,f,suchthatq,i,fistheEuclideandistancebetweenfeaturevectorofimagesin.Thenwesortimagesinbyascendingorderofthisdistancevalue.Therstimagesinareconsideredasn-bestrelevantimagestoAsexpected,theperformancesincreasewhenisgettingbigger.Wecanals

oob-servethattheimagefeature”MPEG-7edgeh
oob-servethattheimagefeature”MPEG-7edgehistogram”givesthebestperformance.Thesecondbestfeatureisthe”MPEG-7colorlayout”.Suchresultsareactuallymeaningfulifweconsiderthatproductpackagingshavepurposelydifferentshapesandcolorsthatformtheirmarketingidentity.Thetexturefeaturessuchas”Tamura”or”Gabor”seemtocharacterizelessefcientlytheproducts.Overall,theperformancesarenotsomuchsatisfyingwith,forexample,72%measuredwith=10for”MPEG-7edgehistogram”.Wecanalsoobservethattheperformancesareverylowwhentendsto1,whichisaclearindicationofthedifcultyofthetaskwithaprobableexplanationtondinthelargevariabilityofimagecharacteristics.SuitableFeatureSelection.Weobservethatfordifferentkindofproducts,samefea-turehasdifferentperformance.Inordertondthemostsuitablefeature,foragivenqueryimageimagesin,thedistanceisre-denedasq,i=minq,i,fwhereisthefeaturesetavailableinLIRe.Aftersortingthedistancesofascendingorder,rstimagesareconsideredastherelevantimagesto.TheresultsshowninFigure5aindicateusthismethodimprovetheperformanceslightly.Productsvs.ImageRecognition.WeshowonFigure5bthecomparisonofratesforthetestusing”Combineglobalfeatures”approachwhichis236K.ChenandJ.HennebertFig.4.evolutionusingfordifferentfeaturesdescribedinprevioussection.Asexpected,ratesaresystematicallyhigherasthetaskiseasier.Fromtheresultswecanobserveanincreaseof6%between(20)(20).Similarresu

ltshavebeenobservedforallfeatures.ImageC
ltshavebeenobservedforallfeatures.ImageCropping.Inourtask,wecanreasonablyassumethatmostoftheinformationtoidentifytheproductsislocatedatthecenteroftheimage.Theouterpartsare,formostoftheFPIDimages,showingtheoorofthesupermarket.Wethereforeattemptedtoremovetheseouterpartsbyimplementingaproportionalcropping.AsillustratedinFigure6a,wedenethecroppedareaasarectanglewherethetopleftcoordinateisdenedbywidthheightwhereisexpressedasapercentageofwidthandheightthatisremoved.Thebottomrightcoordinateiscomputedinasimilarmanner.Figure6bshowstheevolutionof(10)asafunctionusingour”Combineglobalfeature”method.Interestingly,weseeasignicantgainofperformanceupto86%,85%,85%recognitionratewhen60%,30%,10%oftheouterpartsoftheimagesareremoved.LocalFeatures.SURFisascaleandrotationinvariantdetectoranddescriptor.TheSURFalgorithmcontainsthreesteps:(1)interestpointsdetection.(2)buildingthedescriptorassociatedwitheachinterestpoints.(3)descriptormatching.Thersttwostepsarerelyonscale-pacerepresentation,andonrstandsecondorderdifferentialoperators.Allthethreestepsarespeed-upbyusingintegralimageandboxlters.FordetailsofSURF,wereferto[1].Table1givestheresultsonwithcroppingfactor.TheresultsindicatethatSURFimprovetheperformancedrastically,(3)increasefrom52%to94%.Content-BasedImageRetrievalwithLIReandSURFonFPIDImageDatabase237(b)Fig.5.evolutionusingfor”combineglo

balfeatures”and”MPEG-7edgehis-togram”.5b
balfeatures”and”MPEG-7edgehis-togram”.5b:(10)(10)ratesevolutionforusingasapproach”combineglobalfeatures”However,comparetootherglobalfeaturesmethods,SURFsuffersfromthetimeofinterestpointsextractionandmatching.Foragivenqueryimageandtrainingsettheaveragetimeusedforglobalfeaturesislessthanseconds,ontheotherhand,byapplyingSURFmethod,ittakesaboutseconds.InordertoreducethetimetakenforSURFmethod,weproportionallyreducetheimagesize.Foragivenimage,suchthatwidthwidth,wherewidthistheresizedimagewidthandistheresizingfactor.Figure7aillustratestheimpactofimagesizeonthetimetaken.TheimpactofimagesizeonperformanceisgiveninFigure7b.Weobservethatreducingimage238K.ChenandJ.Hennebert(a)(b)Fig.6.6a:Proportionalimagecropping.6b:RRP(10)evolutionasafunctionofp,thepropor-tionalcroppingfactor,for(T10,Q2)using”combineglobalfeatures”approach.sizebychoosingw=0.7,westillhaveahighaccuracyRRP(3)=92%comparetoRRP(3)=94%withtheoriginalimagesize,butthetimehasbeenreducedfrom240secondsto158seconds.InspiredbyLIRe,weuseApachLucene3asfeatureindexingtooltospeed-upSURF.ForeachimageinT,werstextracttheinterestpointswithSURF,thenthesepointsaresavedintoaindexbyusingLucene.Withthisapproach,wereducethetimefrom158to80seconds.3http://lucene.apache.org/core/Content-BasedImageRetrievalwithLIReandSURFonFPIDImageDatabase239Tab

le1.Comparison”combineglobalfeatures”me
le1.Comparison”combineglobalfeatures”methodwithSURFN-bestproducts123Combineglobalfeatures0.320.430.52SURF0.720.900.94(a)(b)Fig.7.7a:ImpactofimagesizeontimetakenforSURF.7b:ImpactofimagesizeonperformanceforSURF.4ConclusionWepresentedaproductidenticationsystembasedonCBIR.ExperimentsaremadeonLIRe(anopen-sourceCBIRsystempartofLucene)andSURF.Theperformancesof240K.ChenandJ.HennebertthesystemhavebeenevaluatedusingtheprotocolsproposedwiththeFPIDdatabase.Wehavealsofoundthattheglobalfeaturesbasedoncolorlayoutsandedgehistogramsarethemostperformingoneonourproductidenticationtask.Consideringthatprod-uctpackagingshavepurposelydifferentshapesandcolorsformarketingidentity,suchresultsareprobablymeaningful.SeveralimprovementshavebeenproposedoverthebaselineuseoftheLIResystem,includingproportionalimagecroppingandglobalfea-turescombination.ByusingSURF,productrecognitionperformanceincreasedto94%(withinthen-bestretrievalproducts),tobecomparedwiththe52%obtainedusingbaselinecongurationofLIRewith”MPEG-7edgehistogram”.ByresizingtheimagesandusingLuceneforindexing,foragivenimage,wereducethetimefromseconds,with92%accuracyofproductrecognition(withinthen-bestre-trievalproducts)byresizingfactor.FutureworkswillprobablytoincreasethespeedofSURFinoursystem,sinceforourproductrecognitionsystem,nearly1minuteperqueryistoolongforamobileapp

lication.Anotherinterestingapproachwould
lication.AnotherinterestingapproachwouldbetocombineCBIRwithcamera-basedOCRastherearefrequentlytextsonthelabelsoftheproducts.Acknowledgements.ThisworkwaspartlysupportedbythegrantGreen-TRCSOISNetfromtheUniversityofAppliedSciencesHES-SO//Wallis,bytheHES-SO//FribourgandbytheUniversityofFribourg.References1.Bay,H.,Ess,A.,Tuytelaars,T.,VanGool,L.:SURF:SpeededUpRobustFeatures.ComputerVisionandImageUnderstanding(CVIU)110(3),346–359(2008)2.Chen,K.,Hennebert,J.:TheFribourgProductImageDatabaseforProductIdenticationTasks.In:Chen,K.,Hennebert,J.(eds.)IEEE/IIAEInternationalConferenceonIntelligentSystemsandImageProcessing(ICISIP),pp.162–169(2013)3.Deselaers,T.,Keysers,D.,Ney,H.:FIRE–exibleimageretrievalengine:ImageCLEF2004evaluation.In:Peters,C.,Clough,P.,Gonzalo,J.,Jones,G.J.F.,Kluck,M.,Magnini,B.(eds.)CLEF2004.LNCS,vol.3491,pp.688–698.Springer,Heidelberg(2005)4.Faloutsos,C.,Equitz,W.,Flickner,M.,Niblack,W.,Petkovic,D.,Barber,R.:EfcientandEffectiveQueryingbyImageContent.JournalofIntelligentInformationSystems3,231–262(1994)5.Lux,M.:ContentbasedimageretrievalwithLIRe.In:Proceedingsofthe19thACMInterna-tionalConferenceonMultimedia,pp.735–738(2011)6.Squire,D.M.,M¨uller,W.,M¨uller,H.,Raki,J.:Content-BasedQueryofImageDatabases,InspirationsFromTextRetrieval:InvertedFiles,Frequency-BasedWeightsandRelevanceFeedback.PatternRecognitionLetters,143–1