AlocallyadaptivezoomingalgorithmfordigitalimagesS.Battiato,G.Gallo,F.S - PDF document

AlocallyadaptivezoomingalgorithmfordigitalimagesS.Battiato,G.Gallo,F.StancoDipartimentodiMatematicaedInformatica,UniversitadiCatania,VialeA.Doria6,Catania95125,Italy Inthispaperweaddresstheproblemofproducinganenlargedpicturefromagivendigitalimage(zooming).Weproposeamethodthattriestotakeintoaccountinformationaboutdiscontinuitiesorsharpluminancevariationswhiledoublingtheinputpicture.Thisisrealizedby 1.IntroductionInthispaperweaddresstheproblemofproducinganenlargedpicturefromagivendigitalimage(zooming).This ImageandVisionComputing20(2002)805Ð812www.elsevier.com/locate/imavis Correspondingauthor.E-mailaddresses:fstanco@dmi.unict.it(F.Stanco),battiato@dmi.unict.it(S.Battiato),gallo@dmi.unict.it(G.Gallo). weakness.Section4showsaninterestingextensionoftheproposedstrategyabletoworkdirectlyonBayerdatadata.Alsointhiscaseexperimentsarereportedandbrießydiscussed.Section5concludesthepaper.2.ThebasicalgorithmInthissectionwegiveadetaileddescriptionoftheproposedalgorithm.Theprocedureissummarizedinpseudo-codeinbox1.Firstwedescribethealgorithminthecaseofgrayscalepicture.Forsakeofsimplicitythemagnifyingfactorappliedhereistwoinbothhorizontalandverticaldirections.Thealgorithmworksinfoursuccessivestagesasdescribedinthefollowing.2.1.Firststage:simpleenlargementTheÞrststageisthesimplestoneandrequirestoexpandthesourcepixelsimageontoaregulargridofsizeFig.1).Morepreciselyifdenotesthepixelinthethrowandthcolumnofthesourceimageanddenotesthepixelinthethrowandthcolumninthezoomedpicturetheexpansionisdescribedasamappingaccordingtotheequation:ThemappingleavesundeÞnedthevalueofallthepixelswithatleastoneevencoordinate(whitedotsinFig.12.2.Secondstage:llingtheholes(partI)Inthesecondstagethealgorithmscanslinebylinethepixelsinwhosecoordinatesarebotheven(e.g.thepixelsdenotedbyagraydot,labeledwithanFig.2a).Forreference,inthefollowingdescriptionwewillusethecapitallettersasinFig.2a,todenotethepixelssurroundingthepixelandthathavealreadybeenassignedavalueinthepreviousstage(blackdots).Withletterswedenotetheluminancevalueofpixels,respectively.Foreverypixeloneofthefollowingmutualexclusiveconditionsistestedandaconsequentialactionistaken:Uniformity:ifthevalue4isassignedtopixelEdgeintheSWNEdirection:ifthevalue2isassignedtopixelEdgeintheNWSEdirection:ifthevalue2isassignedtopixelEdgeintheNSdirection:if0thevalue2topixel2topixelandleavethevalueatpixelundeÞned.EdgeintheEWdirection:if0thevaluetopixel2topixelandleavethevalueatpixelundeÞned.denotesuitablethresholdvaluesthataredescribedinSection3.Attheendofthisstagetheremaybeseveralpixels,aswellasseveralpixels,leftwithanundeÞnedvalue.SuccessivescanningofthezoomedpicturewilltakecareoftheseÔholesÕ. Fig.1.ThepictureshowstheÞrststageofzooming(enlargement). Fig.2.(a)Thelayoutofthepixeltoreconstruct;(b)and(c)thelayoutandthenotationreferredinthedescriptionofthealgorithm.S.Battiatoetal./ImageandVisionComputing20(2002)805–812 2.3.Thirdstage:llingtheholes(partII)InthethirdstagethealgorithmscanslinebylineimagelookingforpixelsleftundeÞnedinthepreviousstageandwithatleastoneoddcoordinate.ThesearethepixelsdenotedbythegraydotwithlabelFig.2bandc.Thetwocasesillustratedinthetwosub-pictureswillbetreatedsimilarlyinthefollowingdiscussion.ForreferencewewillusethelettersasinFig.2bFig.2c),todenotethepixelsaboveandbelow(leftandright,respectively)ofpixelandthathavealreadybeenassignedavalueintheexpansionstage(blackdots).Observethatthevaluesofpixelsmay,ormaynot,havebeenassignedinthepreviousstage.Iftheyhavebeenassignedtheywillbereferred,respectively,with.Accordinglythealgorithmconsidersthefollowingtwocasesandtakeconsequentialactions:orXhavenotbeenassignedavalue.Inthiscasethealgorithmchecksifwiththesameintroducedinthepreviousstage.Ifitisso,thevalueofthepixelissetequaltootherwisethevalueofpixelisleftundeÞned.andXhavebothbeenassignedavalue.Inthiscasethealgorithmchecksforthepresenceofaverticalorahorizontaledge.Thefollowingsub-casesarise:ÐPresenceofanedgeindirectionInthiscasethevalue2isassignedtopixelÐPresenceofanedgeindirectionAB.Inthiscasethevalue2isassignedtopixelÐNoneofabove.NoactionistakenandthevalueofisleftundeÞned.AttheendofthisthirdstageallthepixelswhosespatialdependencefromtheneighborhoodvaluesisÔsimpleÕhavebeenassignedavalue.Usingtheinformationgatheredinsofar,inthenextstagetheremainingÔholesÕareeventuallyÞlled.2.4.Thenalstage:rebinningTheÞnalstageofthealgorithmscansoncemorepicturelookingforpixelswithundeÞnedvalueandÞxtheholeusingasuitablyweightedmeanvalueasdescribedbelow.Preliminarilythegrayvaluescale(typicallyrangingfrom0to255)issubdividedintoarougherscaleofwhereisasuitable,userselectedintegervalue.ThebinincludesthegrayvaluesrangingfromuptoThealgorithmÞrstlooksforeverypixel,withbothevencoordinatesandwithanundeÞnedvalue.InthiscasethepixelssurroundingasinFig.2afallwithinatleastoneandatmostfourbins.Therepresentativevaluesofthesebins(typicallythemedian)areaveragedtoproduceavaluefor.Observethatinthiswaythemorefrequentvaluesareweightedless:thisÔtrickÕguarantees,asshownintheexperiments,abetterdetailpreservationinthezoomedpicture.Inordertopreservevisualsharpnessisfundamentalreducingasmuchaspossiblethelow-passoperation.Itisalsoimportanttonotethatthevaluesconsideredtoassignavaluetoarecomingfromtheoriginalvaluesinthesourceandhencearenotaffectedbythesmoothingoperationsdoneinthethreepreviousstages.Eventuallythealgorithmlooksforeverypixel,withatleastoneoddcoordinatewhosevalueisstillleftundeÞned.InthiscasethesamerebinningproceduredescribedforpixelsisperformedstartingfromthevaluesFig.2bandc.Observethatatthisstep,allfourofthesevalueshavebeenalreadyset.TheproposedapproachrequiresOstepstozoomoutofafactor2adigitalpictureofpixels.Asforspacerequirementthealgorithmrequiresonlythestoragespaceforthezoomedpicture.ThesamecomplexityOstepsisrequiredbyclassicalzoomingalgorithmswhereindeedthemultiplicativecon-stantsaresensiblygreater.Thisisparticularlytrue,havingastrongimpactontheoverallperformance,whenthedimensionsoftheinputimageareextremelylarge(e.g.3/4MpixelacquiredbyaHighQualityDigitalCamera).3.ZoomingcolorpicturesThebasicalgorithmdescribedaboveforgrayscalepicturescanbeeasilygeneralizedtothecaseofRGBcoloreddigitalimages.Todosowetakeadvantageofthehighersensitivityofhumanvisualsystemtoluminancevariationswithrespecttochrominancevalues.Henceitmakessensetoallocatelargercomputationalresourcestozoomluminancevalues,whilechrominancevaluesmaybeprocessedwithasimplerandmoreeconomicalapproach.ThissimplestrategyisinspiredbyanaloguestechniquesusedbyclassicallossyimagecompressionalgorithmslikeJPEGand/orJPEG2000[13,16,21]vastlyimplementedinalmostdigitalstillcameraengines.Accordinglyweproposetooperateasfollows:TranslatetheoriginalRGBpictureintotheYUVcolormodel.ZoomtheluminancevaluesYaccordingwiththebasicalgorithmdescribedabove.ZoomtheUandVvaluesusingasimplerpixelreplicationalgorithm.BacktranslatesthezoomedYUVpictureintoanRGBTheresultsobtainedwiththisbasicapproacharequalitativelycomparablewiththeresultsobtainedusingbicubicinterpolationoverthethreecolorchannels.Fromthecomputationalpointofview,itisimportanttonotehownoS.Battiatoetal./ImageandVisionComputing20(2002)805–812 signiÞcantdifferenceintermsoftimingresponsehasbeenobservedbetweenthesimpleapplicationofourapproachtothethreeRGBplanesandtheapproachdescribedabove(RGBÐYUVconversion,YzoomingU,Vreplication,YUVÐRGBconversion).Yet,inrealapplications(DSC,3GMobilephone,É)thezoomingprocessinsidetypicalImageGenerationPipelineifpresentisrealizedjustbeforecompressionsion:theYUVconversionisalwaysperformedasacrucialsteptoachievevisuallosslesscompression.Inthiscasethecolorconver-sionitselfdoesnotintroducefurthercomputationalcosts.4.ExperimentalresultsThevalidationofazoomingalgorithmrequirestheassessmentofthevisualqualityofthezoomedpictures.Fig.3showstwoexamplesofzoomingpicturesobtainedwiththeproposedalgorithm.Unfortunatelythisqualitativejudgmentinvolvesqualitativeevaluationofmanyzoomedpicturesfromalargepoolofhumanobserversanditishardtobedoneinasubjectiveandpreciseway.Forthisreasonseveralalternativequantitativemeasurementsrelatedtopicturequalityhavebeenproposedandwidelyusedintheliterature.TovalidateouralgorithmwehavechosenboththeapproachesproposedinRefs.[11,12,15],classicalmetricsandsubjectivetests.Inparticularwehaveusedthecross-correlationandthePSNRbetweentheoriginalpictureandthereconstructedpicturetoassessthequalityofreconstruction.InourexperimentalcontestwehaveÞrstcollectedatestpoolof30grayscalepictures.ForeachimageinthissetwehaveÞrstperformedthefollowingoperations:Ðreductionbydecimation:Ðreductionbyaveraging:Thesizereductiontechniqueadoptedmayhaveinßu-encesonthequalityofthezoomedpicture.Startingfromwehaveobtainedthezoomedimageslistedasfollows:,thepicturezoomedbyafactor2usingasimplepixelreplicationtechnique;,thepicturezoomedbyafactor2usingasimplepixelreplicationtechnique;,thepicturezoomedbyafactor2usingbicubicinterpolation;,thepicturezoomedbyafactor2usingbicubicinterpolation;,thepicturezoomedbyafactor2usingouralgorithm;,thepicturezoomedbyafactor2usingouralgorithm.Wehavechosensimplereplicationandbicubicinter-polationasthetwocomparingstonestoassessthequalityofourtechnique.Itisgenerallyacceptedthatreplicationprovidestheworstquality-wisezoomingalgorithmwhilebicubicinterpolationisconsideredoneofthebestoptionsavailable.Itshouldalsobeobservedthatourtechniquerequiresthechoiceoftwothresholdparametersthatcouldbeinprinciple,user-selected.WehaveperformedourtestswithaÞxeddefaultsettingofthethresholds(Thesethresholdvaluescouldbechoseninamoreproperwayusinglocaladaptivemeasuresofactivityinalocalneighborhood(e.g.localvarianceand/orlocaledgesensingsensing).Ourheuristicvalueshavebeenselectedbecausetheyprovidedinpreliminaryexperimentsthebestzoomingquality.Preliminaryexperiments,moreover,haveshownthatsmallvariationsintheseparametersdonotproducelargequalitychangesinthezoomedpicture.Letthecross-correlationcoefÞcientbetweentwopictures,be: wheredenote,respectively,theaveragevalueofpicturedenote,respectively,widthandlength,inpixels,ofimages.Noticethatcross-correlationcoefÞcientsisbetween0and1.ThemorethecoefÞcientapproaches1,thebetterthereconstructionquality.Foreverypair(),(),(),(),( Fig.3.Twoexamplesofzoomedpicturesobtainedwiththeproposedalgorithm.(a)and(b)relativetoanaturalscene,(c)and(d)relativetoaprintedtext.S.Battiatoetal./ImageandVisionComputing20(2002)805–812 )wehavecomputedtherelativecross-correlationcoefÞcientsThecoefÞcientshavebeenaveragedoverthetestpooltoobtaintheaveragecoefÞcients:ThevaluesobtainedarereportedinTable1Thetablepromptlyshowsthatthecross-correlationcoefÞcientsobtainedwiththeproposedtechniquearebetterorveryclosetotheanalogouscoefÞcientsobtainedusingbicubicinterpolationwhenthezoomingisdonefromadecimatedpicture.ThisprovesthebetterorequalabilityofouralgorithmtoÞllinwithproperdetailsthemissinginformationinthelargerpicture.Settingto0thescoreforthepixelreplicationalgorithmandto1thescoreforbicubicinterpolation,theproposedtechniquescoresavalueof1.286545.Theresultsinthecaseofasmallerpictureobtainedbyaveragingareessentiallyinconclusivebecauseinthiscasethecross-correlationcoefÞcientisnotagoodindicatorofreconstructionquality.Theresults,indeed,seemtoprizethepixelreplicationtechniqueabovetheothertwo,evenifitisverywellknownthatthistechniquedoesnotproduceperceptuallygoodimages.Thisapparentlycontradictoryresultonlycomesfromthestatisticalpropertiesofcross-correlationcoefÞcient.Thisstatisticalindicatorturnsoutclosertoonewheneverasquareoftwotimetwopixelsissubstitutedbyasimplereplicationoftheiraverage.Thebehaviorinthecaseofzoomedpictureobtainedfromaveragedsmallerpictureshouldhencebeassessedusingadifferenttechnique.Tothisaimwecomputedthepercentageofzoomedpixelswhosedifference,inabsolutevaluebetweenthecomputedandtheoriginalintensityisgreaterthanagiventhreshold.Fig.4ashowstheaveragepercentageoferrorsobservedoverthetestpoolasdifferenttolerancethresholdsareconsideredinthecasewhenthesub-sampledimageisobtainedbydecimation.Theproposedtechniqueclearlyoutperformstheothertwo.Fig.4bshowstheaveragepercentageoferrorsobservedoverthetestpoolasdifferenttolerancethresholdsareconsideredinthecasewhenthesub-sampledimageisobtainedbylocalaveraging.Inthiscaseaswelltheproposedtechniquegivesabetterperformance.AnotherclassicqualitymeasureweusedinourvalidationexperimentsistheclassicalPSNR.InarereportedtheaveragePSNRvaluesinDbobtainedfromzoomedimagesreducedbyaveragingordeci-mation.InbothcasesourapproachhasPSNRvaluesveryclosetothebicubicones.Alsointhiscasethepixelreplicationtechniqueintermsofmeasurederrorgivesgoodresults.ThenumericalresultsshowedabovedoesnotgiveusaclearandobjectivecomparativejudgmentwithrespecttotheÞnalqualityofazoomedimages.ItisalsoimportanttonotethattheperceivedqualityisnotnecessarilyequivalenttoÞdelity(i.e.theaccuratereproductionoftheoriginal).Forexample,sharpimageswithhighcontrastareusuallymoreappealingtotheaverageviewer.Indeed,thebestwaytovaluatethevisualqualityofthezoomingproposedalgorithmisthesubjectiveobservation.InFigs.5and6areshownzoomingresultsforthreedifferentimages. Table10.9187750.8999810.8343930.972190.973860.97454 Fig.4.Theaveragepercentageoferrorsobservedoverthetestpoolfordifferenttolerancethresholdswhenthesub-sampledimageisobtained:(a)byaveraging;(b)bydecimation. Table2PSNRvaluesinDbmeasuredoveratestpoolofdigitalimagesbetweenoriginalandzoomedimagesobtainedwithclassicalandproposedtechniquestartingfromimagereducedbydecimationandaveragingOurReplicationBicubicDecimation23.6619.9524.77Average21.8423.0522.19S.Battiatoetal./ImageandVisionComputing20(2002)805–812 5.WorkingonBayerdataThemethodhasbeengeneralizedtoworkwithBayerdataata,acquiredbyCCD/CMOSsensorinalmostdigitalcamcorder.Eachpixel,usingsuitableCFA(ColorFilteringArray)array,preservestheintensityofjustoneofthemany-colorseparations.NoticethatinatypicalBayerpattern,onlyonequarterofthearrayÕspixelsisredorblueandonehalfgreen.Greenischosentohavetwicethenumberofpixelsasredorbluebecauseoureyesaremostsensitivetogreenlight.ThisÞlteringschemeallowsustocapturecolorimages,butsincefourpixelsmustbecombinedtoformonecolordot,theresolutionoftheimageislessthanamonochromeimage.Thereconstructionprocessmustguaranteetherenderingofahighqualityimagesavoidingtypicalartifactsthat,duetotheacquisitionprocess,couldbepresent.ForthisreasonpowerfulandsmartalgorithmsareappliedtoenhancequalityinasortofchainknownasDigitalStillCamerapipeline(Fig.7).TobeabletoworkintheBayerpatterndomainallowsonetomanageÔnoise-freeÕdataandthecomplexitywillbelowerthanworkingonRGBdomain.WorkingintheBayerdomainrequiresalittleefforttoreadaptideasandtechniquestothenewparticularenvironmentbutallowstoimprovesigniÞcantlythequalityoftheÞnalimagereducingatthesametimetheassociatedcomputationalcomplexity. Fig.5.(a)Idealimage;(b)ourzooming;(c)replicationzooming;(d)bicubicinterpolationfromaveragingsub-sampling. Fig.6.(a)Idealimage;(b)ourzooming;(c)replicationzooming;(d)bicubicinterpolationfromdecimationsub-sampling.S.Battiatoetal./ImageandVisionComputing20(2002)805–812 Inordertopreservethedetailsoftheoriginalimages,withoutintroducingvisibleartifacts,theinputBayerimageissplitintothreesub-planes,R,G,Bobtainedretainingthecorrespondingchromaticcom-ponenthaving,respectively,2and2pixels.Theproposedzoomingalgorithmisthenapplied,independently,foreachoneofthesecolor-planes.CombiningtogethertheseintermediateresultsanewzoomedBayerdataimagesisobtained.TheimageobtainedisnotanRGBimagebecausetheproposedalgorithmisnotappliedasacolorreconstructionalgorithm.ItsimplyenlargesaninputBayerpattern,obtaininganewzoomedBayerdatainalocallyadaptivewaypreservingdetailswithoutsmoothingartifacts.Fig.8showstworelatedexamples:startingfromtwoBayerimageswehaveobtainedthecorrespondingzoomedBayerimages,successivelyinterpolatedwithasimplecolorinterpolationalgorithmrithm.WeclaimthatworkingdirectlyintheBayerdomain,beforecolorinterpolationalgorithm,ispossibletoimprovefurtherthequalityoftheÞnalzoomedimage.Furtherexperimentsmustbedoneinordertobeabletomanipulatesuchimagesinamoreproperway.OurpreliminaryresultsseemtosuggestthatitisnecessarytoretaininformationstrictlyrelatedtotheÔlocalÕcolordistribution.6.ConclusionsInthispaperwehaveproposedanewtechniqueforzoomingadigitalpicture,bothingrayscale,inRGBcolorsandinBayerdatadomain.Theproposedtechniquehasbeencomparedaccordingtodifferentperformanceindicatorstobicubicinterpolationandpixelreplicationalgorithm.Theexperimentalresultsshowthattheproposedmethod,whileofnotgreatercomplexitythanbicubicinterpolationprovidesqualitativelybetterresults. Fig.7.TypicalDigitalStillCameraPipeline. Fig.8.(a)OriginalBayerpattern;(b)zoomedBayerpattern;(c)RGBzoomedimage.S.Battiatoetal./ImageandVisionComputing20(2002)805–812 AcknowledgmentsTheauthorsaredeeplygratefultoDrMassimoMancusoofSTMicroelectronicsforprovidingmotivationtothisresearchandconstructivecriticstoanearlyversionofthisalgorithm.ThanksgoalsototheDigitalStillCameragroupofSTMicroelectronicsforprovidingsomeofthetestpictures.References[1]S.Battiato,G.Gallo,F.Stanco,Anewedge-adaptivezoomingalgorithmfordigitalimages,ProceedingsofSignalProcessingandCommunicationsSPC,Marbella,Spain(2000)144Ð149.[2]S.Battiato,M.Mancuso,Anintroductiontothedigitalstillcameratechnology,STJournalofSystemResearch,SpecialIssueonImageProcessingforDigitalStillCamera2(2)(2001).[3]S.D.Bayarakeri,R.M.Mersereau,Anewmethodfordirectionalimageinterpolation,ProceedingsofInternationalConferenceonAcoustics,SpeechandSignalProcessing,Detroit,MI24(1995).[4]B.E.Bayer,Colorimagingarray,USPatent3,971,065-1976.[5]D.F.Florencio,R.W.Schafer,Post-samplingaliasingcontrolforimages,ProceedingsofInternationalConferenceonAcoustics,SpeechandSignalProcessing,Detroit,MI2(1995)893Ð896.[6]K.P.Hong,J.K.Paik,H.JuKim,C.HoLee,Anedge-preservingimageinterpolationsystemforadigitalcamcorder,IEEETrans-actionsonConsumerElectronics42(3)(1996).[7]H.S.Hou,H.C.Andrews,CubicsplinesforimageinterpolationanddigitalÞltering,IEEETransactionsonAcoustics,Speech,SignalProcessingASSP-26(6)(1978)508Ð517.[8]A.K.Jain,FundamentalsofDigitalImageProcessing,Prenctice-Hall,EnglewoodCliffs,NJ,1989.[9]R.G.Keys,Cubicconvolutioninterpolationfordigitalimageprocessing,IEEETransactions.onAcoustics,Speech,SignalProcessing29(6)(1981)1153Ð1160.[10]S.W.Lee,J.K.Paik,ImageinterpolationusingadaptivefastB-splineÞltering,ProceedingsofInternationalConferenceonAcoustics,Speech,andSignalProcessing5(1993)177Ð179.[11]T.M.Lehmann,C.Gonner,K.Spitzer,Survey:interpolationmethodsinmedicalimageprocessing,IEEETransactionsonMedicalImaging18(11)(1999).[12]E.Maeland,Onthecomparisonofinterpolationmethods,IEEETransactionsonMedicalImagingMI-7(1988)213Ð217.[13]M.W.Marcellin,M.J.Gormish,A.Bilgin,M.P.Boliek,AnoverviewofJPEG-2000,ProceedingsofIEEEDCC(2000).[14]D.M.Monro,P.D.WakeÞeld,Zoomingwithimplicitfractals,ProceedingsofInternationalConferenceonImageProcessingICIP971(1997)913Ð916.[15]J.A.Parker,R.V.Kenyon,D.E.Troxel,Comparisonofinterpolatingmethodsforimageresampling,IEEETransactionsonMedicalImagingMI-2(1983)31Ð39.[16]W.B.Pennebaker,J.Mitchell,JPEGStillImageDataCompressionStandard,VanNostrandReinhold,NewYork,NY,1993.[17]E.Polidori,J.L.Dugelay,Zoomingusingiteratedfunctionsystems,NATOASIConferenceonFractalImageEncodingandAnalysis,Trondheim,NorwayJuly(1995).[18]T.Sakamoto,C.Nakanishi,T.Hase,Softwarepixelinterpolationfordigitalstillcamerassuitablefora32-bitMCU,IEEETransactionsonConsumerElectronics44(4)(1998)1342Ð1352.[19]P.V.Sankar,L.A.Ferrari,SimplealgorithmsandarchitectureforB-splineinterpolation,IEEETransactionsonPatternAnalysisMachineIntelligencePAMI-10(1988)271Ð276.[20]S.Thurnhofer,S.Mitra,Edge-enhancedimagezooming,OpticalEngineering35(7)(1996)1862Ð1870.[21]G.K.Wallace,TheJPEGstillpicturecompressionstandard,CommunicationsoftheACM34(4)(1991).S.Battiatoetal./ImageandVisionComputing20(2002)805–812