On the left is the original image with one horizontal and one vertical seam In the middle the energy function used in this example is shown the magnitude of the gradient along with the vertical and horizontal path maps used to calculate the seams By ID: 52305
Download Pdf The PPT/PDF document "Seam Carving for ContentAware Image Resi..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
SeamCarvingforContent-AwareImageResizingShaiAvidanMitsubishiElectricResearchLabsArielShamirTheInterdisciplinaryCenter&MERL Figure1:Aseamisaconnectedpathoflowenergypixelsinanimage.Ontheleftistheoriginalimagewithonehorizontalandoneverticalseam.Inthemiddletheenergyfunctionusedinthisexampleisshown(themagnitudeofthegradient),alongwiththeverticalandhorizontalpathmapsusedtocalculatetheseams.Byautomaticallycarvingoutseamstoreduceimagesize,andinsertingseamstoextendit,weachievecontent-awareresizing.Theexampleonthetoprightshowsourresultofextendinginonedimensionandreducingintheother,comparedtostandardscalingonthebottomright.Effectiveresizingofimagesshouldnotonlyusegeometriccon-straints,butconsidertheimagecontentaswell.Wepresentasim-pleimageoperatorcalledseamcarvingthatsupportscontent-awareimageresizingforbothreductionandexpansion.Aseamisanop-timal8-connectedpathofpixelsonaimagefromtoptobot-tom,orlefttoright,whereoptimalityisdenedbyanimageenergyfunction.Byrepeatedlycarvingoutorinsertingseamsinonedirec-tionwecanchangetheaspectratioofanimage.Byapplyingtheseoperatorsinbothdirectionswecanretargettheimagetoanewsize.Theselectionandorderofseamsprotectthecontentoftheimage,asdenedbytheenergyfunction.Seamcarvingcanalsobeusedforimagecontentenhancementandobjectremoval.Wesupportvariousvisualsaliencymeasuresfordeningtheenergyofanim-age,andcanalsoincludeuserinputtoguidetheprocess.Bystoringtheorderofseamsinanimagewecreateimages,thatareabletocontinuouslychangeinrealtimetotagivensize.CRCategories:I.3.0[ComputingMethodologies]:ComputerGraphicsGeneral;I.4.10[ComputingMethodologies]:ImageProcessingAndComputerVisionImageRepresentationKeywords:Imageresizing,Imageretargeting,Imageseams,Content-awareimagemanipulation,Displaydevices 1IntroductionThediversityandversatilityofdisplaydevicestodayimposesnewdemandsondigitalmedia.Forinstance,designersmustcreatedif-ferentalternativesforweb-contentanddesigndifferentlayoutsfordifferentdevices.Moreover,HTML,aswellasotherstandards,cansupportdynamicchangesofpagelayoutandtext.Nevertheless,uptodate,images,althoughbeingoneofthekeyelementsindigitalmedia,typicallyremainrigidinsizeandcannotdeformtotdiffer-entlayoutsautomatically.Othercasesinwhichthesize,oraspectratioofanimagemustchange,aretotintodifferentdisplayssuchascellphonesorPDAs,ortoprintonagivenpapersizeorresolu-Standardimagescalingisnotsufcientsinceitisoblivioustotheimagecontentandtypicallycanbeappliedonlyuniformly.Crop-pingislimitedsinceitcanonlyremovepixelsfromtheimagepe-riphery.Moreeffectiveresizingcanonlybeachievedbyconsider-ingtheimageandnotonlygeometricconstraints.Weproposeasimpleimageoperator,weterm,thatcanchangethesizeofanimagebygracefullycarving-outorin-sertingpixelsindifferentpartsoftheimage.Seamcarvingusesanenergyfunctiondeningtheimportanceofpixels.Aseamisaconnectedpathoflowenergypixelscrossingtheimagefromtoptobottom,orfromlefttoright.Bysuccessivelyremovingorinsert-ingseamswecanreduce,aswellasenlarge,thesizeofanimageinbothdirections(seeFigure1).Forimagereduction,seamselec-tionensuresthatwhilepreservingtheimagestructure,weremovemoreofthelowenergypixelsandfewerofthehighenergyones.Forimageenlarging,theorderofseaminsertionensuresabalancebetweentheoriginalimagecontentandthearticiallyinsertedpix-els.Theseoperatorsproduce,ineffect,acontent-awareresizingofWeillustratetheapplicationofseamcarvingandinsertionforas-pectratiochange,imageretargeting,imagecontentenhancement,andobjectremoval.Furthermore,bystoringtheorderofseamre-movalandinsertionoperations,andcarefullyinterleavingseamsin bothverticalandhorizontaldirectionswedenemulti-sizeimagesSuchimagescancontinuouslychangetheirsizeinacontent-awaremanner.Adesignercanauthoramulti-sizeimageonce,andtheclientapplication,dependingonthesizeneeded,canresizetheim-ageinrealtimetottheexactlayoutorthedisplay.Seamcarvingcansupportseveraltypesofenergyfunctionssuchasgradientmagnitude,entropy,visualsaliency,eye-gazemovement,andmore.Theremovalorinsertionprocessesareparameterfree;however,toallowinteractivecontrol,wealsoprovideascribble-baseduserinterfaceforaddingweightstotheenergyofanimageandguidethedesiredresults.Thistoolcanalsobeusedforauthor-ingmulti-sizeimages.Tosummarize,ourmaincontributionsareasfollows:Deneseamcarvingandpresentitsproperties.Presentalgorithmforimageenlargingusingseaminsertions.Useseamsforcontent-awareimagesizemanipulations.Denemulti-sizeimagesforcontinuousimageretargeting.2BackgroundImageresizingisastandardtoolinmanyimageprocessingappli-cations.Itworksbyuniformlyresizingtheimagetoatargetsize.Recently,thereisagrowinginterestinimageretargetingthatseekstochangethesizeoftheimagewhilemaintainingtheimportantfea-turesintact,wherethesefeaturescanbeeitherdetectedtop-downorbottom-up.Topdownmethodsusetoolssuchasfacedetectors[ViolaandJones2001]todetectimportantregionsintheimage,whereasbottom-upmethodsrelyonvisualsaliencymethods[Ittietal.1999]toconstructavisualsaliencymapoftheimage.Oncethesaliencymapisconstructed,croppingcanbeusedtodisplaythemostimportantregionoftheimage.Suhetal.[2003]pro-posedautomaticthumbnailcreation,basedoneitherasaliencymaportheoutputofafacedetector.Thelargeimageisthencroppedtocapturethemostsalientregionintheimage.Similarly,Chen[2003]consideredtheproblemofadaptingimagestomobiledevices.Intheirapproachthemostimportantregionintheimageisautomaticallydetectedandtransmittedtothemobiledevice.Liuetal.[2003]alsoaddressedimageretargetingtomobiledevices,suggestingtotradetimeforspace.Givenacollectionofregionsofinterest,theyconstructanoptimalpaththroughtheseregionsanddisplaythemserially,oneaftertheother,totheuser.Santella[2006]useeyetracking,inadditiontocompositionrulestocropimagesintelligently.Allthesemethodsachieveimpressiveresults,butrelyontraditionalimageresizingandcroppingoperationstoactuallychangethesizeoftheimage.etal.[2003]consideranadaptivegrid-baseddocumentlay-outsystemthatmaintainsaclearseparationbetweencontentandtemplate.Thepagedesignerconstructsseveralpossibletemplatesandwhenthecontentisdisplayedthemostsuitabletemplateisused.Thetemplatescanusedifferentdiscretealternativesofanimageiftheyareprovided,butnospecicreferencetoimageresiz-ingismade.Acompromisebetweenimageresizingandimagecroppingistointroduceanon-linear,datadependentscaling.SuchamethodwasproposedbyLiuandGleicher[2005;2006]forimageandvideoretargeting.ForimageretargetingtheyndtheRegion-Of-Interest(ROI)andconstructanovelFisheye-Viewwarpthatessentiallyap-pliesapiecewiselinearscalingfunctionineachdimensiontotheimage.ThiswaytheROIismaintainedwhiletherestoftheimageiswarped.Theretargetingcanbedoneininteractiverates,oncetheROIisfound,sotheusercancontrolthedesiredsizeoftheimagebymovingaslider.Intheirvideoretargetingworktheyuseacom-binationofimageandsaliencymapstondtheROI.Thentheyuseacombinationofcropping,virtualpanandshotcutstoretargetthevideoframes.etal.[2005]proposedanautomatic,non-photorealistical-gorithmforretargettinglargeimagestosmallsizedisplays.Thisisdonebydecomposingtheimageintoabackgroundlayerandfore-groundobjects.Theretargetingalgorithmsegmentsanimageintoregions,identiesimportantregions,removesthem,llstheresult-inggaps,resizetheremainingimage,andre-inserttheimportantregion.Therstsolutiontothegeneralproblemofwarpinganimageintoanarbitraryshapewhilepreservinguser-speciedfeatureswasre-centlyproposedbyGaletal.[2006].Thefeature-awarewarpingisachievedbyaparticularformulationoftheLaplacianeditingtech-nique,suitedtoaccommodatesimilarityconstraintsonpartsofthedomain.Sincelocalconstraintsarepropagatedbytheglobalopti-mizationprocess,notalltheconstraintscanalwaysbesatisedatonce.Ouralgorithmisdiscrete,socarvingasingleseamhasnoaffectontherestoftheimage.Theuseofseamsforimageeditingisprevalent.Agarwalaetal.[2004]describeaninteractiveDigitalPhotomontagesystemthatndsperfectseamstocombinepartsofasetofphotographsintoasinglecompositepicture,usingminimaluserassistance.Jiaetal.[2006]proposedDrag-and-DropPastingthatextendsthePoissonImageEditingtechnique[Perezetal.2003]tocomputeanoptimalboundary(i.e.seam)betweenthesourceandtargetimages.Rotheretal.[2006]developedAutoCollage,aprogramthatautomaticallycreatesacollageimagefromacollectionofimages.Thisprocessrequires,amongotherthings,ndingoptimalboundaries,orseams,betweenmanyimagefragments.Noneoftheabovemethodsdis-cusstheproblemofimageretargeting.AnotableexceptionistheworkofWangandCohen[2006]thatproposestosimultaneouslysolvemattingandcompositing.Theyallowtheusertoscalethesizeoftheforegroundobjectandpasteitbackontheoriginalback-ground.Zometetal.[2005]evaluatedseveralcostfunctionsforseamlessimagestitchingandconcludedthatminimizingannormbetweenthegradientsofthestitchedimageandthegradi-entsoftheinputimagesperformedwellingeneral.Computingtheseamcanbedoneinavarietyofways,includingDijkstra'sshortestpathalgorithm[1998],dynamicprogramming[2001]orgraphcutscutsChangingthesizeoftheimagehasbeenextensivelystudiedintheeldoftexturesynthesis,wherethegoalistogeneratealargetextureimagefromasmallone.Efrosetal.[2001]ndseamsthatminimizetheerrorsurfacedenedbytwooverlappingtexturepatches.Thisway,theoriginalsmalltextureimageisquiltedtoformamuchlargertextureimage.Thiswaslaterextendedtohan-dlebothimageandvideotexturesynthesisbyKwatraetal.al.thatshowedhowtoincreasethespaceandtimedimensionsoftheoriginaltexturevideo.Asforobjectremoval,Bertalmioetal.[2000]proposedanim-ageinpaintingmethodthatsmoothlypropagatesinformationfromtheboundariesinwards,simulatingtechniquesusedbyprofessionalrestorators.Patchbasedapproaches[Drorietal.2003;Criminisietal.2003;Bertalmioetal.2003]useautomaticguidancetodeter-minesynthesisordering,whichconsiderablyimprovesthequalityoftheresults.Andrecently,Sunetal.[2005]proposedaninter-activemethodtohandleinpaintingincaseofmissingstrongvisualstructure,bypropagatingstructurealongused-speciedcurves. (a)Original (b)Crop (c)Column (d)Seam (e)Pixel (f)OptimalFigure2:Resultsof5differentstrategiesforreducingthewidthofanimage.(a)theoriginalimageanditsenergyfunction,(b)bestcropping,(c)removingcolumnswithminimalenergy,(d)seamremoval,(e)removalofthepixelwiththeleastamountofenergyineachrow,andnally,(f)globalremovalofpixelswiththelowestenergy,regardlessoftheirposition.Figure3showstheenergypreservationofeachstrategy.3TheOperatorOurapproachtocontent-awareresizingistoremovepixelsinaju-diciousmanner.Therefor,thequestionishowtochosethepixelstoberemoved?Intuitively,ourgoalistoremoveunnoticeablepixelsthatblendwiththeirsurroundings.Thisleadstothefollowingsim-energyfunctionthatwasusedinmanyguresinthispapersuchasFigures1,6,5,8,11,12,13(weexploreotherenergyfunctionsinsubsection3.2):)= ¶xIj+j¶ Givenanenergyfunction,assumeweneedtoreducetheimagewidth.Onecanthinkofseveralstrategiestoachievethis.Forin-stance,anoptimalstrategytopreserveenergy(i.e.,keeppixelswithhighenergyvalue)wouldbetoremovethepixelswithlowesten-ergyinascendingorder.Thisdestroystherectangularshapeoftheimage,becausewemayremoveadifferentnumberofpixelsfromeachrow(seeFigure2(f)).Ifwewanttopreventtheimagefrombreakingwecanremoveanequalnumberoflowenergypixelsfromeveryrow.Thispreservestherectangularshapeoftheimagebutde-stroystheimagecontentbycreatingazigzageffect(Figure2(e)).TopreserveboththeshapeandthevisualcoherenceoftheimagewecanuseAuto-cropping.Thatis,lookforasub-window,thesizeofthetargetimage,thatcontainsthehighestenergy(Figure2(b)).Anotherpossiblestrategysomewhatbetweenremovingpixelsandcroppingistoremovewholecolumnswiththelowestenergy.Still,artifactsmightappearintheresultingimage(Figure2(c)).There-fore,weneedaresizingoperatorthatwillbelessrestrictivethancroppingorcolumnremoval,butcanpreservetheimagecontentbetterthansinglepixelremovals.Thisleadstoourstrategyofseamcarving(Figure2(d))andthedenitionofinternalseams.Formally,letbeanimageanddeneaverticalseamtobe:jisamappingmapping1;:::;;1;:::;.Thatis,averticalseamisan8-connectedpathofpixelsintheimagefromtoptobot-tom,containingone,andonlyone,pixelineachrowoftheimage(seeFigure1).Similarly,ifisamappingmapping1;:::;;1;:::;thenahorizontalseam))jThepixelsofthepathofseam(e.g.verticalseam)willthere-forebe.Notethatsimilartotheremovalofaroworcolumnfromanimage,removingthepixelsofaseamfromanimagehasonlyalocaleffect:allthepixelsoftheimageareshiftedleft(orup)tocompensateforthemissingpath. Figure3:Imageenergypreservation.Acomparisonofthepreser-vationofcontentmeasuredbytheaveragepixelenergyusingvedifferentstrategiesofresizing.TheactualimagescanbeseeninFigure2.Thevisualimpactisnoticeableonlyalongthepathoftheseam,leavingtherestoftheimageintact.Notealsothatonecanreplacetheconstraintj1withj,andgeteitherasimplecolumn(orrow)for0,apiecewiseconnectedorevencompletelydisconnectedsetofpixelsforanyvalue1Givenanenergyfunction,wecandenethecostofaseamas)=)=)).Welookfortheoptimalseamminimizesthisseamcost:)=))Theoptimalseamcanbefoundusingdynamicprogramming.Therststepistotraversetheimagefromthesecondrowtothelastrowandcomputethecumulativeminimumenergyforallpossibleconnectedseamsforeachentry)=)+))Attheendofthisprocess,theminimumvalueofthelastrowinwillindicatetheendoftheminimalconnectedverticalseam.Hence,inthesecondstepwebacktrackfromthisminimumentryontondthepathoftheoptimalseam(seeFigure1).Thedenitionforhorizontalseamsissimilar.3.1EnergyPreservationMeasureToevaluatetheeffectivenessofthedifferentstrategiesforcontent-awareresizing,wecanexaminetheaverageenergyofallofpixels (a)Original (b)e1 Entropy HoG (e)SegmentationandFigure4:Comparingdifferentenergyfunctionsforcontentawareinanimage duringresizing.Randomlyremovingpix-elsshouldkeeptheaverageunchanged,butcontent-awareresizingshouldraisetheaverageasitremoveslowenergypixelsandkeepsthehighenergyones.Figure3showsaplotofthechangeinaver-ageenergywhilechangingtheimagewidthofFigure2usingthevedifferentstrategiesoutlinedabove.Asexpected,removingthelowenergypixelsinascendingordergivestheoptimalresult.Thisiscloselyfollowedbypixelremoval.Butbothmethodsdestroythevisualcoherenceoftheimage.Croppingshowstheworstenergypreservation.Columnremovaldoesabetterjobatpreservingen-ergy,butstillintroducevisualartifacts.Seamcarvingstrikesthebestbalancebetweenthedemandsforenergypreservationandvi-sualcoherency.Thisgraphresultsarecharacteristictomanyimagesingeneral.3.2ImageEnergyFunctionsWehaveexaminedseveralpossibleimageimportancemeasuresfoundinliteratureastheenergyfunctiontoguideseamcarving.Wehavetestedboth-normofthegradient,saliencymeasure[Ittietal.1999],andHarris-cornersmeasure[HarrisandStephens1988].Wealsousedeyegazemeasurement[DeCarloandSantella2002],andtheoutputoffacedetectors.Figure4comparestheresultsoftheerror,entropy,segmentation,andHistogramofGradients(HoG).Theentropyenergycomputestheentropyovera99windowandaddsitto.Thesegmentationmethodrstsegmentstheimage[Christoudiasetal.2002]andthenappliestheerrornormontheresults,effectivelyleavingonlytheedgesbetweensegments.Finally,HoGisdenedasfollows:HoG)= ¶xIj+j¶ ¶yIj HoG)) Figure5:Comparingaspectratiochange.Fromlefttorightinthebottom:theimageresizedusingseamremovals,scalingandHoG))istakentobeahistogramoforientedgradientsateverypixel[DalalandTriggs2005].Weusean8-binhistogramcomputedovera1111windowaroundthepixel.Thus,takingthemaximumoftheHoGatthedenominatorattractstheseamstoedgesintheimage,whilethenumeratormakessurethattheseamwillrunparalleltotheedgeandwillnotcrossit.HoGwasalsousedinFigures9and10.Asexpected,nosingleenergyfunctionperformswellacrossallimagesbutingeneraltheyallaccommodateasimilarrangeforre-sizing.Theyvaryintherateatwhichtheyintroducevisualartifactsandthepartsoftheimagetheyaffect.WefoundeitherHoGtoworkquitewell.4DiscreteImageResizing4.1AspectRatioChangeAssumewewanttochangetheaspectratioofagivenimage.Thiscanbeachievedsimplybysuccessivelyremovingverticalseamsfrom.Contrarytosimplescaling,thisoperationwillnotalterimportantpartsoftheimage(asdenedbytheenergyfunction),andineffectcreatesanon-uniform,content-awareresizingoftheimage(Figure5).Thesameaspectratiocorrection,fromcanalsobeachievedbyincreasingthenumberofrowsbyafactorof(Figure6).Theaddedvalueofsuchanapproachisthatitdoesnotremoveanyinformationfromtheimage.Wediscussourstrategyincreasinganimagesizeindetailsinsub-section4.3.4.2RetargetingwithOptimalSeams-OrderImageretargetinggeneralizesaspectratiochangefromonedimen-siontotwodimensionssuchthatanimageofsizewillberetargetedtosizeand,forthetimebeing,weassumethat.Thisbegsthequestionofwhatisthecorrectorderofseamcarving?Removeverticalseamsrst?Horizontalseamsrst?Oralternatebetweenthetwo?Wedenethesearchfortheoptimalorderasanoptimizationofthefollowingobjective Figure6:AspectratiochangeofpicturesoftheJapanesemasterUtagawaHiroshige.Inbothexamplestheoriginalimageiswidenedbyseaminsertion.+(=(=(isusedasapa-rameterthatdetermineifatstepweremoveahorizontalorvertical2f)=Wendtheoptimalorderusingatransportmapthatspecies,foreachdesiredtargetimagesize,thecostoftheoptimalsequenceofhorizontalandverticalseamremovaloperations.Thatis,entryholdstheminimalcostneededtoobtainanimageofsize.Wecomputeusingdynamicprogramming.Startingat)=0welleachentrychoosingthebestoftwooptions-eitherremovingahorizontalseamfromanimageof1orremovingaverticalseamfromanimageof)=)+)))+)))denotesanimageofsize))))arethecostoftherespectiveseamremovaloperation.Westoreasimple1-bitmapwhichindicateswhichofthetwooptionswaschosenineachstepofthedynamicprogramming.Choosingaleftneighborcorrespondstoaverticalseamremovalwhilechoosingthetopneighborcorrespondstoahorizontalseamremoval.Givenatargetsizewebacktrackfromandapplythecorrespondingremovaloperations.Figure7showsanexampleofdifferentretar-getingstrategiesonanimage.4.3ImageEnlargingTheprocessofremovingverticalandhorizontalseamscanbeseenasatime-evolutionprocess.Wedenoteasthesmallerimage Figure7:Optimalorderretargeting:Onthetopistheoriginalim-ageanditstransportmap.Givenatargetsize,wefollowtheoptimalpath(whitepathon)toobtaintheretargetedimage(toprow,right).Forcomparisonweshowretargetingresultsbyalter-natingbetweenhorizontalandverticalseamremoval(toprow,left),removingverticalseamsrst(bottomrow,left),andremovinghor-izontalseamsrst(bottomrow,right)createdafterseamhavebeenremovedfrom.Toenlargeanimageweapproximatean`inversion'ofthistimeevolutionandinsertnew`articial'seamstotheimage.Hence,toenlargethesizeofanbyonewecomputetheoptimalvertical(horizontal)seamandduplicatethepixelsofbyaveragingthemwiththeirleftandrightneighbors(topandbottominthehorizontalcase).Usingthetimeevolutionnotation,wedenotetheresultingimageas.Unfortunately,repeatingthisprocesswillmostlikelycreateastretchingartifactbychoosingthesameseam(Figure8(b)).Toachieveeffectiveenlarging,itisimportanttobalancebetweentheoriginalimagecontentandthearticiallyinsertedparts.Therefore,toenlargeanimageby,wendtherstseamsforremovalandduplicatetheminordertoarriveat(Figure8(c)).Thiscanbeviewedastheprocessoftraversingbackintimetorecoverpixelsfromalargerimagethatwouldhavebeenremovedbyseamremovals(althoughitisguaranteedtobethecase).Duplicatingalltheseamsinanimageisequivalenttostandardscaling(seeFigure8(e)).Tocontinueincontent-awarefashionforexcessiveimageenlarging(forinstance,greaterthan50%),webreaktheprocessintoseveralsteps.Eachstepdoesnotenlargethesizeoftheimageinmorethanafractionofitssizefromthepre-viousstep,essentiallyguardingtheimportantcontentfrombeingstretched.Nevertheless,extremeenlargingofanimagewouldmostprobablyproducenoticeableartifacts(Figure8(f)).4.4ContentAmplicationInsteadofenlargingthesizeoftheimage,seamcarvingcanbeusedtoamplifythecontentoftheimagewhilepreservingitssize.Thiscanbeachievedbycombiningseamcarvingandscaling.Topre-servetheimagecontentasmuchaspossible,werstusestandard (a) (b) (c) (d) (e) (f) Figure8:Seaminsertion:ndingandinsertingtheoptimumseamonanenlargedimagewillmostlikelyinsertthesameseamagainandagainasin(b).Insertingtheseamsinorderofremoval(c)achievesthedesired50%enlargement(d).Usingtwostepsofseaminsertionsof50%in(f)achievesbetterresultsthanscaling(e).In(g),acloseviewoftheseamsinsertedtoexpandgure6isshown. Figure9:Contentamplication.Ontheright:acombinationofseamcarvingandscalingampliesthecontentoftheoriginalimagescalingtoenlargetheimageandonlythenapplyseamcarvingonthelargerimagetocarvetheimagebacktoitsoriginalsize(seeFigure9).Notethatthepixelsremovedareineffectsub-pixelsoftheoriginalimage.4.5SeamCarvinginthegradientdomainTherearetimeswhenremovingmultipleseamsfromanimagestillcreatesnoticeablevisualartifactsintheresizedimage.Toover-comethiswecancombineseamcarvingwithPoissonreconstruc-tion([Perezetal.2003]).Specically,wecomputetheenergyfunc-tionimageasbefore,butinsteadofremovingtheseamsfromtheoriginalimageweworkinthegradientdomainandremovetheseamsfromthederivativesoftheoriginalimage.Attheendofthisprocessweuseapoissonsolvertoreconstructbacktheimage.Figure10showsanexampleofthistechnique.4.6ObjectRemovalWeuseasimpleuserinterfaceforobjectremoval.Theusermarksthetargetobjecttoberemovedandthenseamsareremovedfromtheimageuntilallmarkedpixelsaregone.Thesystemcanauto-maticallycalculatethesmalleroftheverticalorhorizontaldiame-ters(inpixels)ofthetargetremovalregionandperformverticalorhorizontalremovalsaccordingly(Figure11).Moreover,toregaintheoriginalsizeoftheimage,seaminsertioncouldbeemployedontheresulting(smaller)image(seeFigure12).Notethat,contrary Figure10:SeamCarvinginthegradientdomain.Theoriginalimage(topleft)isretargetedusingstandardtechnique(topright)andinthegradientdomain(bottomright).Zoomincomparisonisshownonbottomleft. Figure11:Simpleobjectremoval:theusermarksaregionforre-moval(green),andpossiblyaregiontoprotect(red),ontheoriginalimage(seeinsetinleftimage).Ontherightimage,consecutivever-ticalseamwereremoveduntilno`green'pixelswereleft. Figure12:Objectremoval:ndthemissingshoe!(originalimageistopleft).Inthisexample,inadditiontoremovingtheobject(oneshoe),theimagewasenlargedbacktoitsoriginalsize.Notethatthisexamplewouldbedifculttoaccomplishusingin-paintingortexturesynthesis. Figure13:Animagewithitsverticalandhorizontalseamindex,coloredbytheirindexfromblue(rstseams)tored(lastseams).topreviousobjectremovaltechniques[Drorietal.2003;Criminisietal.2003;Bertalmioetal.2003],thisschemealtersthewholeim-age(eitheritssizeoritscontentifitisresized).Thisisbecauseboththeremovedandinsertedseamsmaypassanywhereintheimage.5Multi-sizeImagesSofarwehaveassumedthattheuserknowsthetargetsizeaheadoftime,butthismightnotbepossibleinsomecases.Consider,forexample,animageembeddedinawebpage.Thewebdesignerdoesnotknow,aheadoftime,atwhatresolutionthepagewillbedisplayedandtherefore,cannotgenerateatargetimage.Inadifferentscenario,theusermightwanttotrydifferenttargetsizesandchoosetheonemostsuitableforhisorherneeds.Seamcarvingislinearinthenumberofpixelsandresizingisthere-forelinearinthenumberofseamstoberemoved,orinserted.Onaverage,weretargetanimageofsize400500to100100inabout2.2seconds.However,computingtensorhundredsofseamsinrealtimeisachallengingtask.Toaddressthisissuewepresentarepresentationofmulti-sizeimagesthatencodes,foranimageof,anentirerangeofretargetingsizesfrom11toandevenfurtherto,when.Thisinforma-tionhasaverylowmemoryfootprint,canbecomputedinacoupleofsecondsasapre-processingstep,andallowstheusertoretagetanimagecontinuouslyinrealtime.Fromadifferentperspective,thiscanbeseenasstoringanexplicitrepresentationofthetime-evolutionimplicitprocessofseamre-movalsandinsertions.Consider,rst,thecaseofchangingthewidthoftheimage.Wedeneanindexmapofsizeencodes,foreachpixel,theindexoftheseamthatremovedit,i.e.,)=meansthatpixelwasremovedbythe-thseamre- Figure14:Retargetingtheleftimagewithalone(middle),andwithafacedetector(right). Figure15:RetargetingtheBuddha.Atthetopistheoriginalimage,acroppedversionwheretheornamentsaregone,andascaledver-sionwherethecontentiselongated.Usingsimplebottomupfea-turedetectionforautomaticretargetingcannotprotectthestructureofthefaceoftheBuddha(Bottom,left)andthisisachallengingimageforfacedetectorsaswell.Byaddingsimpleuserconstraintstoprotecttheface(Bottom,middle)orthefaceandower(Bottom,right),betterresultsareachieved.moval(Figure13).Togetanimageofwidth,weonlyneedtogather,ineachrow,allpixelswithseamindexgreaterthanorequalThisrepresentationsupportsimageenlargingaswellasreduction.Forinstance,ifwewanttosupportenlargingoftheimageuptosize,weenlargetheimageusingseaminsertionproceduretoasimilartoSection4.3.However,insteadofaveraging-thseamwithit'stwoneighbors,wedonotmodifytheoriginalimagepixelsintheseam,butinsertnewpixelstotheimageastheaverageofthe-thseamanditsleft(orright)pixelneighbors.Theinsertedseamsaregivenanegativeindexstartingat1.Conse-quently,toenlargetheoriginalimageby,weuseexactlythesameprocedureofgathering(fromtheenlargedimage)allpixelswhoseseamindexisgreaterthan))=,andgetanimageofsize)=Computingahorizontalindexmapforimageheightenlargingandreductionisachievedinasimilarmanner(seeFigure13).How-ever,supportingbothdimensionresizingwhilecomputingindependentlywillnotwork.Thisisbecausehorizontalandverti-calseamscancollideinmorethanoneplace,andremovingaseaminonedirectionmaydestroytheindexmapintheotherdirection.Moredetailscanbefoundintheappendix.However,asimplewaytoavoidthisistoallowseamremovalinonedirection,andusede- Figure16:Exampleswhenresizingusingseamsfails:imagesthataretoocondensed(left)orwherethecontentlayoutpreventsseamstobypassimportantparts(right).Insuchcasesthebeststrategywouldbetousescaling.generateseams,i.e.rowsorcolumns,intheotherdirection.Notethatalthoughretargetingamulti-sizeimagetoanysizeisinstanta-neous,duetotheadditionalconstraints,theresultingimagewouldbedifferentthantheonecreatedintheoptimalorderinducedbytheimplicittime-evolvingprocessofsubsection4.2.Thereaderisreferredtotheattachedvideoforexampleofcontinuouslyresizingmulti-sizeimagestovarioussizesinreal-time.6LimitationsAlltheexamplesshownsofarinthispaperwerecomputedau-tomatically,butourmethoddoesnotworkautomaticallyonallim-ages.Thiscanbecorrectedbyaddinghigherlevelcues,eitherman-ualorautomatic.Forexample,inFigure14theerrorfunctionfails,butcombinedwithafacedetectorwegetmuchbetterresults.Figure15showsanexampleofaddinguserconstraints.Othertimes,notevenhighlevelinformationcansolvetheproblem.Wecancharacterizetwomajorfactorsthatlimitourseamcarvingapproach.Therstistheamountofcontentinanimage.Iftheimageistoocondensed,inthesensethatitdoesnotcontain`lessimportant'areas,thenanytypeofcontent-awareresizingstrategywillnotsucceed.Thesecondtypeoflimitationisthelayoutoftheimagecontent.Incertaintypesofimages,albeitnotbeingcon-densed,thecontentislaidoutinamannerthatpreventstheseamsfrombypassingimportantparts(Figure16).7ConclusionsandFutureWorkWepresentedanoperatorforcontent-awareresizingofimagesus-seamcarving.Seamsarecomputedastheoptimalpathsonasingleimageandareeitherremovedorinsertedfromanimage.Thisoperatorcanbeusedforavarietyofimagemanipulationsin-cluding:aspectratiochange,imageretargeting,contentamplica-tionandobjectremoval.Theoperatorcanbeeasilyintegratedwithvarioussaliencymeasures,aswellasuserinput,toguidetheresiz-ingprocess.Inaddition,wedeneadatastructureformulti-sizeimagesthatsupportcontinuousresizingabilityinrealtime.Therearenumerouspossibleextensionstothiswork.Wewouldliketoextendourapproachtootherdomains,therstofwhichwouldberesizingofvideo.Sincetherearecaseswhenscalingcanachievebetterresultsforresizing,wewouldliketoinvestigatethepossibilitytocombinethetwoapproaches,specicallytodenemorerobustmulti-sizeimages.Wewouldalsoliketondabetterwaytocombinehorizontalandverticalseamsinmulti-sizeimages.AcknowledgmentsWewouldliketothankFredoDurandandthegraphicsgroupatMITforreviewinganearlyversionofthiswork.WethankStarkDraperfornarratingthevideo.WethankEricChanfortheuseofthewaterfallimage,andnumer-ousickr()membersformak-ingtheirimagesavailablethroughcreativecommonrightshttp://creativecommons.org/):crazyegg95(Buddha),Gustty(CoupleandSurfers),JeffKubina(Capitol),mykaul(Han-nukaandCar),o2ma(Vase),sigs66(LongbeachandTwopeoplenearsea).Wealsothanktheanonymousreviewersandrefereesfortheircomments.ReferencesGARWALA,A.,DONTCHEVA,M.,AGRAWALA,M.,DRUCKERS.,COLBURN,A.,CURLESS,B.,SALESIN,D.,ANDOHENM.2004.Interactivedigitalphotomontage.ACMTrans.Graph.,3,294302.ERTALMIO,M.,SAPIRO,G.,CASELLES,V.,ANDALLESTERC.2000.Imageinpainting.InProceedingsofACMSIGGRAPHERTALMIO,M.,VESE,L.,SAPIRO,G.,ANDSHER,S.2003.Simultaneousstructureandtextureimageinpainting.InProc.IEEEConferenceonComputerVisionandPatternRecognitionOYKOV,Y.,ANDOLLY,M.-P.2001.Interactivegraphcutsforoptimalboundary®ionsegmentationofobjectsinn-dim-ages.InInternationalConferenceonComputerVision,(ICCV)vol.I,105112.HEN,L.,XIE,X.,FAN,X.,M,W.,ZHANG,H.,ANDHOUH.2003.AvisualattentionmodelforadaptingimagesonsmallMultimediaSystems9,4,353364.HRISTOUDIAS,C.,GEORGESCU,B.,ANDEER,P.2002.Syn-ergisminlow-levelvision.In16thInternationalConferenceonPatternRecognition,vol.IV,150155.RIMINISI,A.,PEREZ,P.,ANDOYAMA,K.2003.Objectre-movalbyexemplar-basedinpainting.InInIEEEConferenceonComputerVisionandPatternRecognition,417424.ALAL,N.,ANDRIGGS,B.2005.Histogramsoforientedgradi-entsforhumandetection.InInternationalConferenceonCom-puterVision&PatternRecognition,vol.2,886893.AVIS,J.1998.Mosaicsofsceneswithmovingobjects.InPro-ceedingsofCVPRARLO,D.,ANDANTELLA,A.2002.Stylizationandab-stractionofphotographs.InProceedingsofSIGGRAPH,769RORI,I.,COHEN-O,D.,ANDESHURUN,Y.2003.Fragment-basedimagecompletion.InProceedingsofACMSIGGRAPHFROS,A.A.,ANDREEMAN,W.T.2001.Imagequiltingfortexturesynthesisandtransfer.InSIGGRAPH2001,ComputerGraphicsProceedings,ACMPress/ACMSIGGRAPH,E.Fi-ume,Ed.,341346.AL,R.,SORKINE,O.,ANDOHEN-O,D.2006.Feature-awaretexturing.InEurographicsSymposiumonRendering ARRIS,C.,ANDTEPHENS,M.1988.Acombinedcornerandedgedetector.InProceedingsofthe4thAlveyVisionConferenceTTI,L.,KOCH,C.,ANDEIBUR,E.1999.Amodelofsaliency-basedvisualattentionforrapidsceneanalysis.PAMI20,11,ACOBS,C.,L,W.,SCHRIER,E.,BARGERON,D.,ANDALESIN,D.2003.Adaptivegrid-baseddocumentlayout.InProceedingsofACMSIGGRAPH,838847.IA,J.,SUN,J.,TANG,C.-K.,ANDHUM,H.-Y.2006.Drag-and-droppasting.InProceedingsofSIGGRAPHUHN,H.W.1955.Thehungarianmethodfortheassignmentproblem.InNavalResearchLogisticQuarterly,2:8397.IU,F.,ANDLEICHER,M.2005.AutomaticImageRetargetingwithFisheye-ViewWarping.InACMUIST,153162.IU,F.,ANDLEICHER,M.2006.VideoRetargeting:Automat-ingPanandScan.InACMinternationalconferenceonMultime-,241250.IU,H.,XIE,X.,M,W.,ANDHANG,H.2003.Automaticbrowsingoflargepicturesonmobiledevices.ProceedingsoftheeleventhACMinternationalconferenceonMultimedia,148EREZ,P.,GANGNET,M.,ANDLAKE,A.2003.PoissonimageACMTrans.Graph.22,3,313318.OTHER,C.,BORDEAUX,L.,HAMADI,Y.,ANDLAKE,A.2006.Autocollage.InProceedingsofSIGGRAPH2006ANTELLA,A.,AGRAWALA,M.,DARLO,D.,SALESIN,D.,ANDOHEN,M.2006.Gaze-basedinteractionforsemi-automaticphotocropping.InACMHumanFactorsinComput-ingSystems(CHI),771780.ETLUR,V.,TAKAGI,S.,RASKAR,R.,GLEICHER,M.,ANDOOCH,B.2005.AutomaticImageRetargeting.InIntheMobileandUbiquitousMultimedia(MUM),ACMPress.UH,B.,LING,H.,BEDERSON,B.B.,ANDACOBS,D.W.2003.Automaticthumbnailcroppinganditseffectiveness.InUIST'03:Proceedingsofthe16thannualACMsymposiumonUserinterfacesoftwareandtechnology,ACMPress,NewYork,NY,USA,95104.UN,J.,YUAN,L.,JIA,J.,ANDHUM,H.2005.Imagecom-pletionwithstructurepropagation.InProceedingsofACMSIG-V.KWATRA,A.SCHDL,I.E.G.T.,ANDOBICK,A.2003.Graphcuttextures:Imageandvideosynthesisusinggraphcuts.ProceedingsofSIGGRAPHIOLA,P.,ANDONES,M.2001.Rapidobjectdetectionusingaboostedcascadeofsimplefeatures.InCoferenceonComputerVisionandPatternRecognition(CVPR)ANG,J.,ANDOHEN,M.2006.SimultaneousMattingandMicrosoftResearchTechnicalReport,MSR-TR-OMET,A.,LEVIN,A.,PELEG,S.,ANDEISS,Y.2005.Seam-lessimagestitchingbyminimizingfalseedges.IEEETransac-tionsonImageProcessing15,4,969977. Figure17:Thehorizontalandverticalindexmapsoftheimageongure13.Colorindexgoesfromblue(rstseams)tored(lastseams).AConstructingConsistentIndexMapsComputinganhorizontalindexmapandverticalindexmapdependentlyformulti-sizeimagewillnotwork.Wesaythatifeveryhorizontalseamintersects(ortouches)allseamindexesandeveryverticalseamintersectsallhor-seamindexes.Consistencyassuresthatremovingaseaminanydimensionwillremoveexactlyonepixelfromallseamsintheotherdimension,retainingtheindexmapstructure.Ifconsistencyisnotmaintained,thenafterremovingonehorizontalseamwemightbeleftwithverticalseamswithdifferentnumberofpixelsandtherectangularstructureoftheimagewillbedestroyed.Asidefromlimitingseamstoberowsorcolumnsinone,orbothdimensions,wepresenthereanotherapproach.Weuseonlytem-porally0-connectedseams,i.e.seamsthatarespatiallyconnectedontheoriginalsizeimage.Forsuchseams,theonlypossi-bleviolationofconsistencybetweenthemapscanoccurindiagonalseamsteps.Ourmethodrstcomputestemporally0-connectedseamsinonedirection,andthenimposestheconstraintsonthediagonalwhencomputingtheseamsintheotherdirection.Tounderstandwhytheonlyviolationofconsistencyoccursindi-agonals,assumewithoutlossofgenerality,thatsomeverticalseam2f:::violatestheconsistencyconstraint.Thismeansitmusttouchsomehorizontalseam2f:::morethanonce.Denotethosepixelswhereseammeetsseam.Sincearepartofaverticalseam,theycannotbeinthesamerow.How-ever,theyarealsopartofthehorizontalseam,andcannotbeinthesamecolumn.Letusexaminetherectangledenedbyinitscorners.seamsmustbeconnectedinsidethisrectangleandtheybothtouchit'scorners.However,oneisaverticalseamandtheotherahorizontalseam.Theonlypossibilityforthistohappenisthattherectangleisinfactasquare,andbothseamspassthroughitsdiagonal.Notethattheaboveclaimreliesonthefactthatallseamsarecon-nectedintheoriginalimage,whichisnottrueifweusenon0-connectedseams.However,becauseweareusing0-connectedseams,wecansimultaneouslycomputeoftheminonedirec-tion.Withoutlossofgenerality,for0-connectedverticalseamsweexamineallpairsofrowsoftheoriginalimageindependently.Foreachsuchpairwendtheoptimalsetof1-edgepathslinkingallpixelsofonerowtoallpixelsofthenextrow.Theglobalmultipleseampathsfromthetopoftheimagetoitsbottomwouldsimplybetheconcatenationofthose1-edgepaths.Findingthebest1-edgepathsbetweenapairofrowsissimilartoaweightedassignmentproblemwhereeachpixelinonerowiscon-nectedtoitsthreeneighboringpixelsintheotherrow.WeusetheHungarianalgorithm[Kuhn1955]tosolvethisweightedassign-mentproblem.Oncewendtheseamsinonedirection,werepeattheprocessintheotherdirection,butwemaskouteverydiagonaledgethatwasalreadyusedbyanyoftherstdirectionseams.Thisguaranteesthattheseamsintheseconddirectionwillbeconsistentwiththerstdirection(Figure17).