SemiCoupled Dictionary Learning with Applications to Image SuperResolution and PhotoSketch - PDF document

toSCDL.Themodelselectioncaneffectivelyseparatedataintodifferentclusterssothatineachclusterastablelinearmappingbetweenthetwostylescanbelearned.Differentfromthepreviousmethodswhichdoclusteringinthesignaldomain,theproposedmodelselectionperformsclusteringinthestyle-specicsparsedomains,aimingatenhancingthestyleconversioncapability.Therestofthepaperisorganizedasfollows.Section2reviewsrelatedworks.Section3presentstheSCDLframe-work.Section4presentsthealgorithm.Section5conductsexperimentsandSection6concludesthepaper.2.RelatedWorksVariouscross-styleimagesynthesisproblems,suchasimageanalogies[11],texturesynthesis[8,16],andmul-timodalfacerecognition[26,13,22],havebeenproposedandstudied.Inthispaper,wefocusontheproblemsofim-agesuper-resolutionandphoto-sketchsynthesis,andthuswemainlyreviewthemethodsonthesetwoapplications.Imagesuper-resolutionaimstoreconstructahighreso-lution(HR)imagefromitslowresolution(LR)counterpart.Therearemainlytwocategoriesofsuper-resolutionmeth-ods.Intherstcategory,theLRimageisdown-sampledfromablurredversionoftheHRimage[10].Theblur-ringkernelisknown(orcanbeestimated)andusedintheHRimagereconstructionprocess.Thisisbasicallyanin-verseproblemwithanimagingmodelavailable.Inthesecondcategory,oftentheLRimageismodeledasthedi-rectlydown-sampledversionoftheHRimage.Weconsid-erthesecondcaseinthispaper,andthesuper-resolutionproblemcanbeviewedasanimageinterpolationproblem[12,29,15,21].Manyimageinterpolationmethods,in-cludingtheclassicalbi-cubicinterpolator[12]andtheedgeguidedinterpolators[15,29],interpolatethemissingHRpixelastheweightedaverageofitslocalneighbors.Thedifferencebetweenthesemethodsliesinhowtheweightsaredetermined.In[29]theautoregressivemodelisusedtoexploittheimagelocalcorrelationforeffectiveimageinter-polation.In[21],aseriesoflinearinverseestimatorsofHRimagearecomputedbasedondifferentpriorsontheimageregularity.Theseestimatorsarethenmixedinaframeoverpatchesofcoefcients,providingasparsesignalrepresen-tationunderl1-normminimizationweightedbythesignalregularityineachpatch.Inlawenforcement,wemayhavetocomparemug-shotphotostoasketchdrawnbyanartistbasedontheverbaldescriptionofthesuspect.Inaddition,sincenearinfrared(NIR)imagingisrobusttoilluminationchanges,itisoftenusedinoutdoorfaceimageacquisition,andmatchingfaceimagesunderNIRandvisiblelightsisnecessary.TangandWang[24]usedeigentransformtolearnmappingsbetweendifferentimagestyles.Theirmethodisbasedontwoimpor-tantassumptions:transformationbetweendifferentstylescanbeapproximatedasalinearprocess,andfacescanbereconstructedfromtrainingsamplesbyPCA.Thismethodworkswellinfacehallucination[25].However,duetothelimitationsofPCA,thetwoassumptionscanhardlyholdforimagestylesbetweenwhichthemappingsarehighlynonlinear.Anotherfamilyofcross-styleimagemodelingmethodsistoconstructahiddensubspace[22,13].Thissubspaceaimstomaximizecorrelationsofdifferentimagestylessothatimagesofdifferentstylesprojectedintothesubspacearehighlycorrelated.Onerepresentativeworkiscanonicalcorrelatedanalysis,whichhasbeenwellusedinmulti-modalfacerecognitiontasks[14].However,canoni-calcorrelatedanalysisaimsatpreservingcorrelationordis-criminativeinformationinsteadofreconstructiveinforma-tion,anditmaynotleadtohighlyaccurateimagerecon-structionacrossstyles.Toovercomethesedrawbacks,LinandTang[16]proposedanovelcoupledsubspacelearningstrategytolearnimagemappingsbetweendifferentstyles.Theyrstutilizedcorrelativecomponentanalysistondthehiddenspacesforeachstyletopreservecorrelativeinforma-tion,andthenlearnedabidirectionaltransformbetweentwosubspaces.Naturalimagepatchescouldbesparselyrepresentedbyanover-completedictionaryofatoms.Recently,sparsecoding(orsparserepresentation)anddictionarylearninghaveproventobeveryeffectiveinimagereconstruction[20,9,6,5],whilethedictionaryplaysanimportantroletosuccessfullyaccomplishsuchtasks.Learningadictio-naryfromexampleimagepatcheshasbeenattractingmuchinterest,andsomerepresentativemethodshavebeenpro-posed,suchasK-SVD[1],superviseddictionarylearning[19],onlinedictionarylearning[17],etc.In[27],Yangetal.usedacoupleddictionarylearningmodelforimagesuper-resolution.TheyassumedthatthereexistcoupleddictionariesofHRandLRimages,whichhavethesamesparserepresentationforeachpairofHRandLRpatch-es.Afterlearningthecoupleddictionarypair,theHRpatchisreconstructedonHRdictionarywithsparsecoefcientscodedbyLRimagepatchovertheLRdictionary.InourproposedSCDL,thisstrongregularizationof“samesparserepresentation”isrelaxedforcross-styleimagesynthesis,andamorestablecross-stylemappingcanbelearnedinthesparsedomain.3.Semi-coupleddictionarylearning3.1.ProblemformulationTheimagecross-stylesynthesisproblemcanbeformu-latedasfollows:givenanimagexofstylesx,howtore-covertheassociatedimageyofstylesyofthesamescene?Thedifcultiesofthiskindofproblemsvarywithimagestyles.SupposethatalltheimagesinstylesxformaspaceXandimagesinstylesyformaspaceY,andthereexists Figure1.Flowchartoftheproposedsemi-coupleddictionarylearning(SCDL)basedimagecross-stylesynthesis.amappingf(
)fromXtoY:y=f(x).Ifthemappingisinvertibleandknown,wecansimplytransformbetweenxandy.Unfortunately,inmostcasesthiskindoftransformisinvertibleandhardtolearndirectly.Sinceeachpairofimagesindicatethesamescene,itisreasonabletoassumethatthereexistsahiddenspacewherethestylescanbeconvertedtoeachother.There-fore,somecoupledsubspace/dictionarylearningmethods[16,27]havebeenproposed,andtheyassumethatinthecoupledsubspacetherepresentationcoefcientsoftheim-agepairshouldbestrictlyequal.However,thisassumptionistoostrongtoaddresstheexibilityofimagestructuresindifferentstyles.Inthispaper,werelaxthisassumptionandassumethatthereexistsadictionarypairoverwhichtherepresentationsoftwostyleshaveastablemapping.S-incethepairofdictionariesisnotrequiredtobefullycou-pled,wecalltheproposedmethodsemi-coupleddictionarylearning(SCDL).InSCDL,weemploydictionariestoseekforthestructuralhiddenspacesandthemapping.Oncethedictionarypairandmappingarelearned,cross-styleimagesynthesiscanbeperformed,andthesynthesisproceduresareillustratedinFig.1.DenotebyXandYthetrainingdatasetsformedbytheimagepatchpairsofstylessxandsy.Weproposetomin-imizetheenergyfunctionbelowtondthedesiredsemi-coupleddictionariesaswellasthedesiredmapping:minfDx;Dy;f(
)gEdata(Dx;X)+Edata(Dy;Y)+ Emap(f(x);y)+Ereg(x;y;f(
);Dx;Dy)(1)whereEdata(
;
)isthedatadelitytermtorepresentdatadescriptionerror,Emap(
;
)isthemappingdelitytermtorepresentthemappingerrorbetweenthecodingcoefcientsoftwostyles,andEregistheregularizationtermtoregular-izethecodingcoefcientsandmapping.Notethatintheproposedmodel,thecodingcoefcientsofXandYoverDxandDywillberelatedbyamappingf(
).Thetwodic-tionaries(DxandDy)andthemappingfunctionf(
)willbejointlyoptimized.Onespecialbutimportantcaseisthatthemappingf(
)islinear,andthentheframeworkinEq.1canbeturnedintothefollowingdictionarylearningandridgeregressionproblem:minfDx;Dy;WgkX�Dxxk2F+kY�Dyyk2F+ ky�Wxk2F+xkxk1+ykyk1+WkWk2Fs.t.kdx;ikl21;kdy;ikl21;8i(2)where ;x;y;Wareregularizationparameterstobal-ancethetermsintheobjectivefunctionanddx;i;dy;iaretheatomsofDxandDy,respectively.Theobjectivefunc-tioninEq.2isnotjointlyconvextoDx;Dy;W.How-ever,itisconvexw.r.t.eachofthemifothersarexed.Therefore,wecandesignaniterativealgorithmtoalterna-tivelyoptimizethevariables.In[27],themappingtrans-formWispredenedasanidentitymatrixandthecodingcoefcientsxandyareassumedthesame.Thismod-elactuallyapproximatesf(
)asaconformalmappingonthecoupleddictionaries.However,forcomplexdatawithinvertiblemapping,thismodelislimitedtoreconstructtheimagestructuresacrossdifferentstyles.Incomparison,ourproposedSCDLmodelrelaxesthecouplingofdictionariesbyallowingmappingerrorsbetweencodingcoefcients.3.2.TrainingTotackletheenergy-minimizationinEq.2,weseparatetheobjectivefunctioninto3sub-problems,namelysparsecodingfortrainingsamples,dictionaryupdatingandmap-pingupdating.First,weneedtoinitializethemappingWanddictionarypair.Wcanbesimplyinitializedasthei-dentitymatrix.TherearemanywaystoinitializeDxandDysuchasrandommatrix,PCAbasis,DCTbasis,etc.Us-ingl1-minimization,thesparsecodesxandycanthenbeobtained.NotethatmappingbyWisassumedtobelin-ear,andthebidirectionaltransformlearningstrategycanbeadoptedtolearntransformsfromxtoyandfromytoxsimultaneously.WithsomeinitializationofWanddictionarypairDxandDy,wecancalculatethesparsecodingcoefcientsxandyasfollows:minfxgkX�Dxxk2F+ ky�Wxxk2F+xkxk1minfygkY�Dyyk2F+ kx�Wyyk2F+ykyk1(3)