A fair benchmark for image watermarking systems M - PDF document

AfairbenchmarkforimagewatermarkingsystemsM.KutterandF.A.P.PetitcolasSignalProcessingLaboratory,SwissFederalInstituteofTechnology,Ecublens,1015Lausanne,SwitzerlandTheComputerLaboratory,UniversityofCambridge,PembrokeStreet,CambridgeCB23QG,UnitedKingdomElectronicImaging'99.SecurityandWatermarkingofMultimediaContents,vol.3657,SansJose,CA,USA,25{27January1999.TheInternationalSocietyforOpticalEngineering.ABSTRACTSincetheearly90'sanumberofpaperson\robust"digitalwatermarkingsystemshavebeenpresentedbutnoneofthemusesthesamerobustnesscriteria.Thisisnotpracticalatallforcomparisonandslowsdownprogressinthisarea.Toaddressthisissue,wepresentanevaluationprocedureofimagewatermarkingsystems.Firstweidentifyallnecessaryparametersforproperbenchmarkingandinvestigatehowtoquantitativelydescribetheimagedegradationintroducedbythewatermarkingprocess.Forthis,weshowtheweaknessesofusualimagequalitymeasuresinthecontextwatermarkingandproposeanovelmeasureadaptedtothehumanvisualsystem.Thenweshowhowtoecientlyevaluatethewatermarkperformanceinsuchawaythatfaircomparisonsbetweendierentmethodsarepossible.Theusefulnessofthreegraphs:\attackvs.visual-quality,"\bit-errorvs.visualquality,"and\bit-errorvs.attack"areinvestigated.Inadditionthereceiveroperatingcharacteristic(ROC)graphsarereviewedandproposedtodescribestatisticaldetectionbehaviorofwatermarkingmethods.Finallywereviewanumberofattacksthatanysystemshouldsurvivetobereallyusefulandproposeabenchmarkandasetofdierentsuitableimages.Keywords:digitalwatermarking,benchmark,evaluation,qualitymetric,robustness1.INTRODUCTIONAtthebeginningof1990theideaofdigitalwatermarking,embeddingimperceptibleinformationintoaudiovisualdata,hasemerged.Sincethenworldwideresearchactivitieshavebeenincreasingdramaticallyandtheindustrialinterestindigitalwatermarkingmethodskeepsgrowing.Therstacademicconferenceonthesubjectwasorganisedin1996.Digitalwatermarkshavemainlythreeapplicationelds:datamonitoring,copyrightprotection,anddataauthentication.TherstwatermarkingmethodswereproposedfordigitalimagesbyCaronni8,9in1993,althoughearlierpublicationsalreadyintroducedtheideaoftaggingimagestosecretlyhideinformationandensureownershiprights.42,41Sincethen,theideaofdigitalwatermarkinghasbeenextendedtootherdatasuchasaudioandvideo.ForrecentoverviewsofdigitalwatermarkingmethodsthereaderisreferredtoAnderson,Aucsmith,andSwansonetal.Besidesdesigningdigitalwatermarkingmethods,animportantandoftenneglectedissueaddressespropereval-uationandbenchmarking.Thisnotonlyrequiresevaluationoftherobustness,butalsoincludessubjectiveorquantitativeevaluationofthedistortionintroducedthroughthewatermarkingprocess.Onlyfewauthors(e.g.,Brau-dawayorKutteretal.)reportquantitativeresultsontheimagedegradationduetothewatermarkingprocess.Ingeneral,thereisatradeobetweenwatermarkrobustnessandwatermarkvisibility.Hence,forfairbenchmarkingandperformanceevaluationonehastoensurethatthemethodsunderinvestigationaretestedundercomparableconditions.Inthispaperweproposeawaytoevaluateandcompareperformancesof\robust"invisiblewatermarkingsystems.InSection2weredenethegenericwatermarkingschemeandidentifyimportantparametersandvariables.DistortionmetricsandattacksonwatermarksaredescribedinSection3andSection4,respectively.InSection5weproposedierentgraphsusefulforperformanceassessment.OurbenchmarkprocedureandanimagedatabaseareintroducedinSection6. Furtherauthorinformation:M.K.:E-mail:Martin.Kutter@ep .chF.A.P.P.:E-mail:Fabien.Petitcolas@cl.cam.ac.uk 2.DIGITALWATERMARKING:FRAMEWORK,DEFINITIONSANDPARAMETERSInordertoidentifyimportantwatermarkingparametersandvariables,werstneedtohavealookatthegenericwatermarkingembeddingandrecoveryschemes.Inthefollowingweusethesamenotationforsetsandtheirelements;thedierenceshouldbeobvioustothereader.Figure1illustratesthegenericembeddingprocess.Givenanimage,awatermarkandakey(usuallytheseedofarandomnumbergenerator)theembeddingprocesscanbedenedasamappingoftheform:andiscommontoallwatermarkingmethods.ThegenericdetectionprocessisdepictedinFigure2.Itsoutputiseithertherecoveredwatermarkorsomekindofcondencemeasureindicatinghowlikelyitisforagivenwatermarkattheinputtobepresentintheimageunderinspection.Thereareseveraltypesofwatermarkingsystems.Theyaredenedbytheirinputsandoutputs:Privatewatermarkingsystemsrequireatleasttheoriginalimage.TypeIsystems,extractthewatermarkfromthepossiblydistortedimageandusetheoriginalimageasahinttondwherethewatermarkcouldbeinTypeIIsystems(e.g.,9,10,35)alsorequireacopyoftheembeddedwatermarkforextractionandjustyielda`yes'or`no'answertothequestion:doescontainthewatermarkItisexpectedthatthiskindofschemewillbemorerobustthantheotherssinceitconveysverylittleinformationandrequiresaccesstosecretmaterial.Semi-privatewatermarkingdoesnotusetheoriginalimagefordetection()butanswersthesamequestion.Theonlyuseofprivateandsemi-privatewatermarkingseemstobeevidenceincourttoproveownershipandcopy-controlinapplicationssuchasDVDwherethereaderneedstoknowwhetheritisallowedtoplaythecontentornot.Alargenumberofthecurrentlyproposedschemesfallinthiscategory.5,20,26,27,45,48,55Publicwatermarking(alsoreferredtoasblindwatermarking)remainsthemostchallengingproblemsinceitrequiresneitherthesecretoriginalnortheembeddedwatermark.Indeedsuchsystemsreallyextractofinformation(thewatermark)fromthewatermarkedimage:15,16,21,23,40,56Publicwatermarkshavemuchmoreapplicationsthantheothersandwewillfocusourbenchmarkonthesesystems.Actuallytheembeddingalgorithmsusedinpublicsystemscanalwaysbeusedintoaprivateoneimprovingrobustnessatthesametime.Thereisalsoasymmetricwatermarking(orpublickeywatermarking)whichhasthepropertythatanyusercanreadthewatermark,withoutbeingabletoremoveit.Aftergroupingthedierentsystems,wecannowidentifyimportantparametersandvariables.Amountofembeddedinformation{Thisisanimportantparametersinceitdirectlyin uencesthewa-termarkrobustness.Themoreinformationonewantstoembed,theloweristhewatermarkrobustness.Theinformationtobehiddendependsontheapplication.Inordertoavoidsmallscaleproprietarysolutions,itseemsreasonabletoassumethatonewantstoembedanumbersimilartotheoneusedforISBN(roughly10digits)orbetterISRC(roughly12alphanumericcharacters).Ontopofthis,oneshouldalsoaddtheyearofcopyright,thepermissionsgrantedontheworkandratingforit.Thismeansthatroughly70bitsofinformationshouldbeembeddedinanimage.Thisdoesnotincludeextrabitsaddedforerrorcorrectioncodes.Watermarkembeddingstrength{Thereisatradeobetweenthewatermarkembeddingstrength(hencethewatermarkrobustness)andquality.Increasedrobustnessrequiresastrongerembedding,whichinturnincreasesthevisualdegradationoftheimages. InternationalStandardBookNumberingInternationalStandardRecordingCode Watermark(W)Stego-image(I)Secret/publickey(K) image(IFigure1.Genericdigitalwatermarkembeddingscheme.Watermark(W)originalimage(I)Test-image(ISecret/publickey(K) Watermarkdetection confidencemeasureFigure2.Genericdigitalwatermarkrecoveryscheme.Sizeandnatureofthepicture{Althoughverysmallpicturesdonothavemuchcommercialvalue,awatermarkingsoftwareneedstobeabletorecoverawatermarkfromthem.Thisavoidsa\Mosaic"attackonthemandallowstiling,usedveryofteninWebapplications.ForprintingapplicationshighresolutionimagesarerequiredbutonealsowantstoprotecttheseimagesaftertheyareresampledandusedontheWeb.Photographersandstockphotocompanieshavegreatconcernsabouthavingtheirworkstolenandmostofthemstillrelyonsmallimages,visiblewatermarksandeven\rolloverjavascripts"toreduceinfringement.Furthermorethenatureoftheimagehasalsoanimportantimpactonthewatermarkrobustness.Veryoftenmethodsfeaturingahighrobustnessforscannednaturalimageshaveasurprisinglyreducedrobustnessforsyntheticimages(e.g.,computergeneratedimages).Afairbenchmarkshoulduseawiderangeofpicturesizes,fromfewhundredtoseveralthousandspixels,anddierentkindofimages.Secretinformation(e.g.,key){Althoughtheamountofsecretinformationhasnodirectimpactonthevisualdelityoftheimageortherobustnessofthewatermark,itplaysanimportantroleinthesecurityofthesystem.Thekeyspace,thatistherangeofallpossiblevaluesofthesecretinformation,mustbelargeenoughtomakeexhaustivesearchattacksimpossible.Thereadershouldalsokeepinmindthatmanysecuritysystemsfailtoresisttoverysimpleattacksbecauseofbadsoftwareengineering.1,333.VISUALQUALITYMETRICSAsmentionedintheprevioussection,thewatermarkrobustnessdependentsdirectlyontheembeddingstrength,whichinturnin uencesthevisualdegradationoftheimage.Forfairbenchmarkingandperformanceevaluation,thevisualdegradationduetotheembeddingisanimportantandunfortunatelyoftenneglectedissue.Sincethereisnouniversalmetric,wereviewinthissectionthemostpopularpixelbaseddistortioncriteriaandintroduceonemetricwhichmakesuseoftheeectinthehumanvisualsystem(HVS).3.1.PixelBasedMetricsMostdistortionmeasuresorqualitymetricsusedinvisualinformationprocessingbelongtothegroupofdierencedistortionmeasuresTherstpartofTable3.1liststhemostpopulardierencedistortionmeasures.Thesesmeasureareallbasedonthedierencebetweentheoriginal,undistortedandthemodied,distortedsignal.Thesecondpartofthesametableshowsdistortionmeasuresbasedonthecorrelationbetweentheoriginalandthedistortedsignal.ForacomparativestudyofthemeasurestheinterestedreaderisreferredtoEskiciogluandFisher.Nowadays,themostpopulardistortionmeasuresintheeldofimageandvideocodingandcompressionaretheSignaltoNoiseRatio(SNR),andthePeakSignaltoNoiseRatio(PSNR).TheyareusuallymeasuredindecibelsSNR)=10logSNRTheirpopularityisverylikelyduetothesimplicityofthemetric.However,itiswellknownthatthesedierencedistortionmetricsarenotcorrelatedwithhumanvision.ThismightbeaproblemfortheirapplicationindigitalwatermarkingsincesophisticatedwatermarkingmethodsexploitinonewayortheothertheHVS.Usingtheabovemetrictoquantifythedistortioncausedbyawatermarkingprocessmightthereforeresultinmisleadingquantitativedistortionmeasurements.Furthermorethesemetricsareusuallyappliedtotheluminanceandchrominancechannelsofimages.Ifthewatermarkingmethodsworkinthesamecolor-space,forexampleluminancemodication,thisdoesnotposeaproblem.Onthecontrary,ifthemethodsusedierentcolorspaces,thesemetricarenotsuitable. Thesescriptsareusedtodisplayimagesinsuchawaythattheyarereplacedbyanotherimage(typicallyacopyrightsign)whentheusermovesthecursoronittosaveit.Contrarytopopularbelief,thisdoesnotprovideanysecurity.Dependingonthetypeofwatermarking,thekeyspacecanbeorasubsetof DierenceDistortionMetrics MaximumDifference =max AverageAbsoluteDifference =1 XjIm;n�~Im;nj Norm.AverageAbsoluteDifference NAD MeanSquareError MSE MNX�Im;n�~Im;n
2 NormalisedMeanSquareError NMSE -Norm Lp= 1 XjIm;n�~Im;njp!1 LaplacianMeanSquareError LMSE SignaltoNoiseRatio SNR PeakSignaltoNoiseRatio PSNRmax ImageFidelity �X�Im;n�~Im;n
2=Xm;nI2m;n CorrelationDistortionMetrics NormalisedCross-Correlation =XIm;n~Im;n=Xm;nI2m;n CorrelationQuality =XIm;n~Im;n=Xm;nIm;n Others StructuralContent =XI2m;n=Xm;n~I2m;n GlobalSigmaSignaltoNoiseRatio GSSNR where 1 block block SigmaSignaltoNoiseRatio SSNR SSNRwhereSSNR=10log (b�~b)2 SigmatoErrorRatio SER 1 block HistogramSimilarity where)istherelativefrequency oflevelina256levelsimage. Note:Table1.Commonlyusedpixelbasedvisualdistortionmetrics.representsapixel,whosecoordinatesarem;n),intheoriginal,undistortedimage,andrepresentsapixel,whosecoordinates(m;n),inthewatermarkedimage.GSSNRSSNRandrequirethedivisionoftheoriginalandwatermarkedimagesintoblocksofpixels(e.g.,44pixels).MoredetailsaregiveninNunes. Rating Impairment Quality 5 Imperceptible Excellent 4 Perceptible,notannoying Good 3 Slightlyannoying Fair 2 Annoying Poor 1 Veryannoying Table2.ITU-RRec.500Qualityratingsonascalefrom1to5.3.2.PerceptualQualityMetricsTheweaknessesofthepixel-baseddistortionmetricshavebeenknownforalongtime.Inrecentyearsmoreandmoreresearchconcentratesondistortionmetricsadaptedtothehumanvisualsystembytakingvariouseectintoaccount.46,51{53Inthispaper,wemakeuseofadistortionmetricproposedbyvandenBrandenLamprechtandFarrell.TheperceptualqualitymeasureexploitsthecontrastsensitivityandmaskingphenomenaoftheHVSandisbasedonamulti-channelmodelofthehumanspatialvision.Computingthemetricinvolvesthefollowingsteps:coarseimagesegmentation,decompositionofthecodingerrorandtheoriginalimageintoperceptualcomponentsusinglterbanks,computingthedetectionthresholdforeachpixelusingtheoriginalimageasmasker,dividingthelterederrorbythedecisionthreshold,poolingoverallcolorchannels.TheunityforthemetricisgiveninunitsabovethresholdalsoreferredtoasJustNoticeableDierence(JND).Theoverallmetric,MaskedPeakSignaltoNoiseRatio(MPSNR)isthengivenby:MPSNR=10log (1)whereisthecomputeddistortion.SincethisqualitymetricdoesnothaveexactlythesamemeaningastheknowndB's,itisreferredtoasvisualdecibels(vdB).Anormalisedqualityratingisoftenmoreuseful.WeusetheITU-RRec.500qualityrating.Theratingiscomputedas: (2)whereisthemeasureddistortionandanormalisationconstant.isusuallychosensuchthataknownreferencedistortionmapstothecorrespondingqualityrating.Table2liststheratingsandthecorrespondingvisualperceptionandquality.TheITUratinghasseveraladvantages,suchasnotblowingupfornotdistortedimages,overtheMPSNRqualitymetricandishencemoresuitableforthewatermarkingpurpose.Thesoftwaretocomputethepresenteddistortionmetricisavailablefornoncommercialusageandcanbedownloadedathttp://ltswww.epfl.ch/mpqm/&#x-480;4.POSSIBLEATTACKSONWATERMARKSWeproposeherealistofattacksagainstwhichwatermarkingsystemcouldbejudged.Wedonotmakeadierencebetweenintentionalandunintentionalprocessing.JPEGcompression{JPEGiscurrentlyoneofthemostwidelyusedcompressionalgorithmsforimagesandanywatermarkingsystemshouldberesilienttosomedegreeofcompression.Geometrictransformations{Horizontal ip{Manyimagescanbe ippedwithoutloosinganyvalue.Althoughresilienceto ippingisusuallystraightforwardtoimplementonlyveryfewsystemsdosurviveit. {Rotation{Smallanglerotation,oftenincombinationwithcropping,doesnotusuallychangethecom-mercialvalueoftheimagebutcanmakethewatermarkun-detectable.Rotationsareusedtorealignhorizontalfeaturesofanimageanditiscertainlytherstmodicationappliedtoanimageafterithasbeenscanned.Forbenchmarkingweproposetocroptherotatedimagesothatthereisnoneedtoaddaxedcolorbordertoit.{Cropping{Insomecases,infringersarejustinterestedbythe\central"partofthecopyrightedmaterial,moreovermoreandmoreWebsitesuseimagesegmentation,whichisthebasisofthe\Mosaic"attack.Thisisofcourseanextremecaseofcropping.{Scaling{Aswenoticedearlier,thishappenswhenaprintedimageisscannedorwhenahighresolutiondigitalimageisusedforelectronicapplicationssuchasWebpublishing.ThisshouldnotbeneglectedaswemovemoreandmoretowardWebpublishing.Scalingcanbedividedintotwogroups,uniformandnon-uniformscaling.Underuniformscalingweunderstandscalingwhichisthesameinhorizontalandverticaldirection.Non-uniformscalingusesdierentscalingfactorsinhorizontalandverticaldirection(changeofaspectratio).Veryoftendigitalwatermarkingmethodsareresilientonlytouniformscaling.{Deletionoflinesorcolumns{Thiswasourrstattackonsomecopyrightmarkingsystemsandisveryecientagainstanystraightforwardimplementationofspread-spectrumtechniquesinthespatialdomain.Removingsamplesatregularintervalsinapseudorandomsequence(1)(henceshiftingthenextones)typicallydividesbytheamplitudeofthecrosscorrelationpeakwiththeoriginalsequence.{Generalisedgeometricaltransformations{Ageneralisedgeometricaltransformationisacombina-tionofnon-uniformscaling,rotation,andshearing.{Randomgeometricdistortions(StirMark){Thesedistortionsweredetailedinanearlierpaper32,33andwesuggestedthatimage-watermarkingtools,whichdonotsurvivethemshouldbeconsideredunac-ceptablyeasytobreak.{GeometricdistortionswithJPEG{Rotation,andscalingalonearenotenoughtheyshouldbealsotestedincombinationwithJPEGcompression.Sincemostartistswillrstapplythegeometrictransformationandthensavetheimageinacompressedformatitmakessensetotestrobustnessofwatermarkingsystemtogeometrictransformationfollowedbycompression.Howeveranexhaustivetestshouldalsoincludethecontrarysinceitmightbetriedbywillfulinfringers.Itisdiculttochoseaminimal\qualityfactor"forJPEGasartifactquicklyappear.Howeverexperiencefromprofessionalsshowsthat\qualityfactors"downto70%arereasonable.ArtistsseemtouseJPEGextensivelyaswellasresizing.Enhancementtechniques{Lowpassltering{Thisincludeslinearandnon-linearlters.Frequentlyusedltersincludemedian,Gaussian,andstandardaveragelters.{Sharpening{Sharpeningfunctionsbelongtothestandardfunctionalitiesofphotoeditionsoftware.Theselterscanbeusedasaneectiveattackonsomewatermarkingschemesbecausetheyareveryeectiveatdetectinghighfrequencynoiseintroducedbysomedigitalwatermarkingsoftware.MoresubtleattacksarebasedontheLaplacianoperator:initssimplestversiontheattackedimageis)whereisthestrengthoftheattack.{Histogrammodication{Thisincludeshistogramstretchingorequalisationwhicharesometimesusedtocompensatepoorlighteningconditions.{Gammacorrection{Veryfrequentlyusedoperationtoenhanceimagesoradaptimagesfordisplay,forexampleafterscanning.{Colorquantisation{ThisismostlyappliedwhenpicturesareconvertedtotheGraphicsInterchangeFormat(GIF)extensivelyusedforWebpublishing.Colorquantisationisveryoftenaccompaniedbyditheringwhichdiusestheerrorofthequantisation.{Restoration{Thesetechniquesareusuallydesignedtoreducetheeectsofspecicdegradationprocessesbutcouldalsobeusedwithoutprioriknowledgeofthenoiseintroducedbythewatermarkingsystem. Parameter GraphType VisualQuality Robustness Attack Bits Robustnessvs.attack variable variable Robustnessvs.visualquality variable variable xed Attackvs.visualquality variable variable ROC xed xed/variable Table3.Dierentgraphsandcorrespondingvariablesandconstants.Noiseaddition{Additivenoiseanduncorrelatedmultiplicativenoisehavebeenlargelyaddressedinthecommunicationtheoryandsignalprocessingtheoryliterature.Authorsoftenclaimthattheircopyrightmarkingtechniquessurvivethiskindofnoisebutmanyforgettomentionthemaximumlevelofacceptablenoise.Printing-scanning{Thisprocessintroducesgeometricalaswellasnoise-likedistortions.Statisticalaveragingandcollusion{Giventwoormorecopiesofthesameimagebutwithdierentmarks,itshouldnotbepossibletoremovethemarksbyaveragingtheseimagesorbytakingsmallpartsofallimagesandreassemblingthem.Over-marking{Inthiscasetheattackerneedsspecialaccesstothemarkingsoftware.Currentcommercialimplementationswillrefusetoaddawatermarkifanotherisalreadyembedded.Consequentlytheattacksneedtobypassthetestimplementedinthesoftware.Howevermanufacturershavefullaccesstothemarkingsoftwareandcanperformthistestwithoutanydiculty.Oracleattack{Whenapublicdecoderisavailable,anattackercanremoveamarkbyapplyingsmallchangestotheimageuntilthedecodercannotnditanymore.Atheoreticalanalysisofthisattackandapossiblecountermeasure(randomisingthedetectionprocess)havebeenpresentedrecently.Onecouldalsomakethedecodingprocesscomputationallyexpensive.Howeverneitherapproachisreallysatisfactoryintheabsenceoftamper-resistanthardware.Weconsiderthislisttobeaminimumforwatermarkingtesting.Randomnon-linearimperceptiblegeometricdistortionsarestillverychallengingandsolutionshavenotbeendiscussedyet.5.PERFORMANCEEVALUATIONANDREPRESENTATIONInordertoproperlyevaluatetheperformanceofwatermarkingschemesandallowfaircomparisonbetweendierentschemes,thetestsetupconditionsareofhighimportance.Inthissectionthepossibleevaluationtoolsareoutlined,togetherwiththetestsetupandconditions.Table3listsusefulgraphs,togetherwiththevariableandxedparametersforcomparison.Forallevaluationstrategiesitisveryimportanttoperformthetestsusingdierentkeysandavarietyofimageswithchangingimagesizeandnature.Theresultsshouldthenbeaveragedandplotted.Ifperformanceevaluationonindividualimagesisrequired,forexamplefordirectperformancecomparisonoftwomethodsforoneimage,itisstillveryimportantthatalltestsarerepeatedseveraltimes,usingdierentkeys.Inthefollowing,attackreferstoanyattackoftheprevioussection.Thetermrobustnessdescribesthewatermarkresistancetotheseattacksandcanbemeasuredbythebit-errorratewhichisdenedastheratioofwrongextractedbitstothetotalnumberofembeddedbits.ThevisualqualityistheresultofadistortionmetricsuchasMPSNRInordertoillustratetheusefulnessoftheproposedgraphs,weimplementedacomparativescenariofortwosimplewatermarkingmethods.Bothmethodsarebasedonspread-spectrummodulation,butindierentdomains.Onemethodusesthespatialdomainwhiletheothermethodusesamulti-resolutionenvironment(threelevelwavelettransformwithDaubechies6taplters).Thesystemsuseasecretkeywhichservesasseedforapseudorandomnumbergeneratorusedtogeneratethespreadspectrumsequences.Asrobustnessmeasureweusethebit-errorrate,themetricforthevisualqualityistheratingintroducedinSection3.2,andtheattackisJPEGcompression.Alltestswereperformedonthe512512,24-bitcolourversionoflena.Eachtestwasrepeatedusingeachtimearandomlychosenkey.Thewatermarklengthis100bits. 40 50 60 70 80 90 100 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 JPEG-QualityBit-Error Rate Multi-Resolution Figure3.Bit-errorvs.attackgraphforspread-spectrummodulationinaspatialandmulti-resolutionenvironment.Thevisualqualitywasxedto45.Itisclearlyvisible,thatthemulti-resolutionapproachhasahigherrobustness. 4 4.5 5 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Visual QualityBit-Error Rate Multi-Resolution Figure4.Bit-errorvs.visualqualitygraphforspread-spectrummodulationinaspatialandmulti-resolutionenvironment.TheattackwasxedtoJPEGcompressionwith75%quality.Againthemulti-resolutionapproachshowssuperiorperfor-5.1.Bit-Errorvs.AttackStrengthGraphOneofthemostimportantgraphsrelatesthewatermarkrobustnesstotheattack.Usuallythisgraphshowsthebit-errorrateasafunctionoftheattackstrengthforagivenvisualquality.Severalpapershaveusedthisgraph,unfortunatelywithoutexplicitlyreportingthevisualimagequality.Thisevaluationallowsdirectcomparisonofthewatermarkrobustnessandshowstheoverallbehaviourofthemethodtowardsattacks.Figure3showsthisgraphforourexample.Eachtestwasrepeated10times,usingdierentkeys,andthevisualqualityratingwasxedto45.Itisclearlyvisible,thatforagivenvisualqualitythemulti-resolutionschemehassuperiorperformance.5.2.Bit-Errorvs.VisualQualityGraphThe\bit-errorvs.visualquality"graphshowstherelationshipbetweenthebit-errorandthevisualimagequalityforaxedattack.Foragivenattack,thisgraphcanbeusedtodeterminetheexpectedbit-errorforadesiredvisualquality.Thismightbeespeciallyusefultodeterminetheminimalvisualqualityforadesiredbit-errorrateunderagivenattack.Figure4showsthegraphforourexample.Thetestwasrepeated10timesforeachimage,usingadierentkey.TheattackwasJPEG-compressionat75%quality.Theindividualresultswerethenaveragedandplotted.Wecaneasilydeterminethemaximalachievablevisualqualitysuchthat,forexample,thebit-errordoesnotexceedadesiredvalue.Inaddition,thesamegureclearlyillustratesthatforadesiredbit-errorratethemulti-resolutionwatermarkingschemeallowshighervisualqualities.5.3.Attackvs.VisualQualityGraphThe\attackvs.visualquality"graphillustratesthemaximumallowableattackasafunctionofthevisualqualityforagivenrobustness.Thisgraphallowsimmediateevaluationoftheallowablewatermarkattackforgivenvisualqualities.Thisisespeciallyusefulifthevisualqualityrangeisgivenandthecorrespondingmaximalallowabledistortion,i.e.watermarkattack,needstobeevaluated.Furthermorethisgraphisveryusefulincomparingdierentwatermarkingmethodssinceitfacilitatesimmediaterobustnesscomparisonsforagivenvisualimagequalityataxedbit-errorrate. Figure5showsthegraphforourexample.Thebit-errorratewasxedto01andeverytestwasrepeated5timesusingadierentkey.Thegraphclearlyshowsthesuperiorperformanceofthemulti-resolutionapproach.Foragivenvisualquality,thespatialwatermarkingalgorithmrequiresmuchhighercompressionqualities.5.4.ReceiverOperatingCharacteristicGraphsGivenanyimageawatermarkdetectorhastofullltwotasks:decideifthegivenimageiswatermarkedanddecodetheencodedinformation.Theformercanbeseenashypothesistestinginthatthewatermarkdecoderhastodecidebetweenthealternativehypothesis(theimageiswatermarked)andthenullhypothesis(theimageisnotwatermarked).Inbinaryhypothesistestingtwokindsorerrorscanoccur:acceptingthealternativehypothesis,whenthenullhypothesisiscorrectandacceptingthenullhypothesiswhenthealternativehypothesisistrue.ThersterrorisoftencalledTypeIfalsepositiveandtheseconderrorisusuallycalledTypeIIerrororfalsenegativeReceiverOperatingCharacteristic(ROC)graphsareveryusefulinassessingtheoverallbehaviorandreliabilityofthewatermarkingschemeunderinspection.Usuallyinhypothesistesting,ateststatisticiscomparedagainstathresholdtodecideforoneortheotherhypothesis.Comparingdierentwatermarkingschemeswithaxedthresholdmayresultinmisleadingresults.ROCgraphsavoidthisproblembycomparingthetestusingvaryingdecisionthresholds.TheROCgraphshowstherelationbetweenthetruepositivefraction(TPF)onthey-axisandfalsepositivefraction(FPF)onthex-axis.Thetruepositive-fractionisdenedas:TPF (3)whereisthenumberoftrue-positivetestresults,andisthenumberfalsenegativetests.Thefalse-positivefractionisdenedas: (4)whereisthetotalnumberoffalse-positivetestresults,andisthenumberoftruenegativetestresults.Inotherwords,theROCgraphshowsTPF-FPFpairsresultingfromacontinuouslyvaryingthreshold.Anoptimaldetectorhasacurvethatgoesfromthebottomleftcornertothetopleft,andthentothetoprightcorner.Thediagonalfromthebottomleftcornertothetoprightcornerdescribesadetectorwhichrandomlyselectsoneortheotherhypothesiswithequalprobability.Hence,thehigherthedetectoraccuracy,themoreitscurveapproachesthetopleftcorner.Oftentheintegralunderthecurveisusedasadetectorperformancemeasure.Togeneratethesegraphs,thesamenumberofwatermarkedandnon-watermarkedimagesshouldbetested.Iftheoverallperformanceofwatermarkingmethodsistobeevaluated,testsshouldincludedavarietyofattackswithvaryingparameters.Figure6showstheROCgraphforourexample.Eachtestwasrepeated10timesusingadierentkeyandthevisualqualitywassetto45.TheattackwasJPEG-compressionandthequalityfactorwasvariedfrom30%to100%instepsof5%.Thetwocurvesinthegraphshow,thatthemulti-resolutionschemefeatureshigherdetectionreliability.Furthermoreitisinterestingtonote,thatthespatialdomainapproachhasatendencytorejectwatermarkedimages.6.ABENCHMARKAswenoticedintheintroductionanumberofbroadclaimshavebeenmadeaboutthe\robustness"ofvariousdigitalwatermarkingmethod.Unfortunatelythecriteriaaswellasthepicturesusedtodemonstratetheseclaimsvaryfromonesystemtotheotherandrecentattacks24,25,32,33showthattherobustnesscriteriausedsofarareinadequate:JPEGcompression,additiveGaussiannoise,lowpasslteringrescaling,andcroppinghavebeenaddressedinmosttheliterature5,9{12,15,18{20,22,23,26,27,35,36,43{45,48,49,54,55butspecicdistortionssuchasrotationhavebeenrarelyad-dressed.21,29Insomecasesthewatermarkissimplysaidtobe\robustagainstcommonsignalprocessingalgorithmsandgeometricdistortionswhenusedonsomestandardimages."MostofthepotentialattacksdetailedinSection4areactuallyimplementedintothelatestversionofStirMarkgivenawatermarkedimage,StirMarkwillapplythesetransformationswithvariousparameters.Thentheoutputimagescanbetestedwithwatermarkdetectionorextractionprograms.Thefullprocesscanbeautomatedusingasimplebatchle. 4 4.2 4.4 4.6 4.8 5 30 40 50 60 70 80 90 100 Visual QualityJPEG-Quality Multi-Resolution Figure5.Attackvs.visualqualitygraphforspreadspectrummodulationinaspatialandmulti-resolutionenvironment.Thebit-errorratewassetto01.Thecurvesclearlyshowthatthemulti-resolutionapproachaccommodateslargercompressionratiosforagivenvisualquality. 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 False Positive RateTrue Positive Rate Multi-Resolution Figure6.ROCgraphforspreadspectrummodu-lationinaspatialandmulti-resolutionenvironment.Thecurvecorrespondingtothemulti-resolutionap-proachisclosertothetopleftcorner,whichindicatesitsuperiorperformance.6.1.ImageDatabaseItisimportanttotestanimagewatermarkingsoftwareonmanydierentimagesandforfaircomparisonthesamesetofsampleimagesshouldalwaysbeused.Picturescanbeinterestingfromthesignalprocessingpointofview:textured/smoothareas,size,synthetic,withstraightedges,sharp,blur,brightness/contrast,etc.Theyshouldalsocoverabroadrangeofcontentsandtypes.Itisimpossibletogetanexhaustivelistofclassesofpicturesandstockphotocompanieshavealotofdicultiestosetupasatisfactoryindex.Howeveronecanatleastretainthemainthemesthatarecommonamongtheselibrariesandthatareusedveryofteninthepressinordertokeepawiderangeofkindofpictures:colors,textures,patterns,shapes,lightning.Someimagedatabasesalreadyexistforimageprocessingresearch.TheUSC-SIPIImageDatabaseisanexampleofsuchdatabasewhereonecanndthe\classics:"lena,baboon,peppers,etc.Usingthesedatabasesforresearchondigitalwatermarkingandindeedcopyrightprotectionissomewhathypocriticalas\someoftheimagesinthedatabasewerescannedfromcopyrightedmaterial"andthe\originofmanyisunknown."Consequentlywetriedtondawiderangeofotherphotographersandgottheauthorisationtousethemfreelyforresearchonwatermarking(includingpublicationinproceedingsorjournals)aslongascreditisgiventothephotographer.6.2.RatingandProcedureWesuggestthefollowingprocedureforthetest.Foreachimageintheset:Embedawatermarkwithstrongeststrengthwhichdoesnotintroduceannoyingeects:thequalityratingde-nedinSection3.2shouldbeatleast4.InthecaseofprivatewatermarkingoftypeI,semi-publicwatermarkingandpublic-watermarkingsystemembedan80-bitwatermark.ApplytheStirMarkbenchmarktoproduceasetofdistortedversionoftheimage.Trytodetect(80%)orrecover(all)thewatermark. Forthisrstversionofourbenchmarkwedidnotputanyweightingonthepossibleattacks.Awatermarkextractedordetectedsuccessfully{dependingonthetypeofwatermarking{givesonemark.Foreachtypeofattackthesemarksareaveragedtogiveamarkoutof20andtheresultingmarksareaveragedtogivethenalmark.Adetailedlistofattacks,oursetofimagesandsomeresultsarepresentedat:http://www.cl.cam.ac.uk/~fapp2/watermarking/benchmar&#x-480;k/7.CONCLUSIONInthispaperweaddressedtheissueofhowtoperformfairbenchmarkingandperformanceevaluationofdigitalwatermarkingmethods.Inarstpartwehaveshownthatforafaircomparisonbetweendierentmethods,thevisualdegradationoftheimageshastobetakenintoaccount.Wehavereviewedavarietyofcommonlyuseddistortionmetrics.ThedrawbackofthesedistortionmetricsisthattheyarenotcorrelatedtotheHVS.WethereforepresentedanotherdistortionmetricwhichisadaptedtotheHVSandhencemoresuitablefordigitalwatermarking.Inadditionthemetricallowscomparisonevenifthedistortionisinadierentcolorchannel.DistortionmetricssuchasthePSNRarenotsuitableforthisbecausetheygiveadistortionvalueforallcolorchannels,forexampleand.Thenwelookedathowtoevaluatetheperformanceofdierentwatermarkingmethodsinaresearchenvironment.Furtherworkcouldtrytoimprovethismeasurebytakingintoaccountpossibleminornonlineargeometricdistortions.Wehaveproposedfourdierentgraphswhichcanbeusedtoevaluateindividualperformanceandallowfaircomparisonbetweendierentmethods.WealsoproposedtheuseofROCgraphs,whichareveryusefulinassessingtheoverallstatisticaldetectionbehaviorofwatermarkingmethods.Theusefulnessofallgraphshasbeendemonstratedbycomparingtwodierentwatermarkingmethods.Asmentioned,theintroducedperformanceevaluationisveryusefulinascienticenvironmentsinceoneneedsfullaccesstothealgorithmsandtheirparameters.Howeverinacommercialenvironmentthisisoftennotthecase.Wethereforeproposeagenericbenchmarktestwhichcanbeusedtoevaluatewatermarkingmethodswithoutgoingintotechnicaldetails.Thebenchmarkteststherobustnessofthewatermarkingmethodsusingavarietyofattacksanddistortions.Theresultisasinglenumberbetween0and20whichdescribestheoverallperformanceofthemethods.Thehigherthenumber,thebettertheperformance.Howeveritisimportanttonotethatevenforthisbenchmarkingthedistortionhastobetakenintoaccount.Thismeansthatthemethodsshouldbeparametrisedsuchthatthevisualdegradationisthesameforalltestedmethods.Asabasisforourbenchmarkandinordertolimitthenumberofattackstobetested,weenhancedSirMarktoincludeasetofpre-denedtypicalattacks(rotation,scaling,colourquantisation,etc.)andbetterrandomgeometricdistortions.Furthermoreitisimportantthatalltestsarerunseveraltimes,usingdierentkeysanddierentimages.Wethereforeproposeasetoftest-imagestobeusedfortheevaluationofwatermarkingmethods.Theseimagearefreelyusableforresearchpurposeonlyaslongascreditisgiventotheartist.AcknowledgementsTherstauthorthanksProfessorEbrahimi,SwissFederalInstituteofTechnology,Lausanne,forintroducinghimtothepresentedtopicandisgratefulforthetechnicaldiscussions,insightsandhints.ThesecondauthorisgratefultotheIntelCorporationfornancialsupportunderthegrant\RobustnessofInformationHidingSystems."SomeoftheideaspresentedherewereclariedbydiscussionwithRossJ.Anderson,UniversityofCambridgeandJean-LucDugelay,InstitutEurecom.Theauthorswouldliketoexpresstheirthankstoalltheartistsandphotographswhocooperatedwiththembygivingtherighttousetheirworkandbyprovidingusefulfeedback.REFERENCES1.RossJ.Anderson.Whycryptosystemsfail.CommunicationsoftheACM,37(11):32{40,November1994.2.RossJ.Anderson,editor.Informationhiding:rstinternationalworkshop,volume1174ofLectureNotesinComputerScience,IsaacNewtonInstitute,Cambridge,England,May1996.Springer-Verlag,Berlin,Germany,ISBN3-540-61996-8. SuchmeasurecouldbebasedontheGSSNRasitisblock-basedandtheseminordistortionswouldnotaecttheblocksverymuch.StirMarkwasoriginallywrittenbyMarkusG.Kuhntotestrobustnessagainstbilinearrandomgeometricdistortions. 3.DavidAucsmith,editor.InformationHiding:SecondInternationalWorkshop,volume1525ofLectureNotesinComputerScience,Portland,Oregon,USA,1998.Springer-Verlag,Berlin,Germany,ISBN3-540-65386-4.4.R.BarnettandD.E.Pearson.FrequencymodeLRattackoperatorfordigitallywatermarkedimages.ElectronicsLetters,34(19):1837{1839,September1998.5.MauroBarni,FrancoBartolini,VitoCappellini,andAlessandroPiva.ADCT-domainsystemforrobustim-agewatermarking.SignalProcessing,66(3):357{372,May1998.EuropeanAssociationforSignalProcessing(EURASIP).6.Anonymous(&#x-540;zguan.bbs@bbs.ntu.edu.tw).LearncrackingIV{anotherweaknessofPictureMarc.news:tw.mirroredonhttp://www.cl.cam.ac.uk/~fapp2/watermarking/image_watermarking/digimarc_,August1997.IncludesinstructionstooverrideanyDigimarcwatermarkusingPictureMarc.7.GordonW.Braudaway.Resultsofattacksonaclaimedrobustdigitalimagewatermark.InvanRenesse.ISBN0-8194-2556-7,ISSN0277-786X.8.G.Caronni.Ermittelnunauthorisierterverteilervonmaschinenlesbarendaten.Technicalreport,ETHZurich,Switzerland,August1993.9.GermanoCaronni.Assuringownershiprightsfordigitalimages.InH.H.BruggermannandW.Gerhardt-Hackl,editors,ReliableITSystems(VIS'95),pages251{263.ViewegPublishingCompany,Germany,1995.10.IngemarJ.Cox,JoeKilian,TomLeighton,andTalalShamoon.Asecure,robustwatermarkformultimedia.InAnderson,pages183{206.ISBN3-540-61996-8.11.IngemarJ.CoxandMattL.Miller.Areviewofwatermarkingandtheimportanceofperceptualmodeling.InRogowitzandPappas.ISBN0-8194-2427-7,ISSN0277-786X.12.J.F.Delaigle,C.DeVleeschouwer,andB.Macq.Watermarkingalgorithmbasedonahumanvisualmodel.SignalProcessing,66(3):319{335,May1998.EuropeanAssociationforSignalProcessing(EURASIP).13.JanaDittmann,PetraWohlmacher,PatrickHorster,andRalfSteinmetz,editors.MultimediaandSecurity{WorkshopatACMMultimedia'98,volume41ofGMDReport,Bristol,UnitedKingdom,September1998.ACM,GMD{ForschungszentrumInformationstechnikGmbH.14.AhmetM.EskiciogluandPaulS.Fisher.Imageaualitymeasuresandtheirperformance.IEEETransactionsonCommunication,43(12):2959{2965,December1995.15.FrankHartungandBerndGirod.Watermarkingofuncompressedandcompressedvideo.SignalProcessing66(3):283{301,May1998.EuropeanAssociationforSignalProcessing(EURASIP).16.AlexanderHerrigel,JosephJ.K.O'Ruanaidh,HolgerPetersen,ShelbyPereira,andThierryPun.Securecopyrightprotectiontechniquesfordigitalimages.InAucsmith,pages169{190.ISBN3-540-65386-4.17.MartyKatz.DigitalwatermarksoftenfailonWebimages.TheNewYorkTimes,11November1997.18.E.KochandJ.Zhao.Towardsrobustandhiddenimagecopyrightlabeling.InWorkshoponNonlinearSignalandImageProcessing,pages452{455,NeosMarmaras,Greece,June1995.IEEE.19.DeepaKundurandDimitriosHatzinakos.Arobustdigitalimagewatermarkingmethodusingwavelet-basedfusion.InInternationalConferenceonImageProcessing,pages544{547,SantaBarbara,California,USA,October1997.IEEE.20.DeepaKundurandDimitriosHatzinakos.Digitalwatermarkingusingmultiresolutionwaveletdecomposition.InternationalConferenceonAcoustic,SpeechandSignalProcessing(ICASP),volume5,pages2969{2972,Seattle,Washington,USA,May1998.IEEE.21.MartinKutter.Watermarkingresistingtotranslation,rotation,andscaling.InProceedingsofSPIEInternationalSymposiumonVoice,Video,andDataCommunications,November1998.22.MartinKutter,F.Jordan,andFrankBossen.Digitalwatermarkingofcolorimagesusingamplitudemodulation.JournalofElectronicImaging,7(2):326{332,April1998.23.GerritC.Langelaar,JanC.A.vanderLubbe,andReginaldL.Lagendijk.Robustlabelingmethodsforcopyprotectionofimages.InIshwarK.SethinandRameshC.Jain,editors,StorageandRetrievalforImageandVideoDatabaseV,volume3022,pages298{309,SanJose,California,USA,February1997.TheSocietyforImagingScienceandTechnology(IS&T)andtheInternationalSocietyforOpticalEngineering(SPIE),SPIE.ISBN0-8194-2433-1,ISSN0277-786X.24.Jean-PaulM.G.LinnartzandMartenvanDijk.Analysisofthesensitivityattackagainstelectronicwatermarksinimages.InAucsmith,pages258{272.ISBN3-540-65386-4.25.MauriceMaes.Twinpeaks:Thehistogramattackonxeddepthimagewatermarks.InAucsmith,pages290{305.ISBN3-540-65386-4. 26.GianlucaNicchiottiandEnnioOttaviano.Non-invertiblestatisticalwaveletwatermarking.In9thEuropeanSignalProcessingConference(EUSIPCO'98),pages2289{2292,IslandofRhodes,Greece,8{11September1998.ISBN960-7620-05-4.27.N.NikolaidisandI.Pitas.Robustimagewatermarkinginthespatialdomain.SignalProcessing,66(3):385{403,May1998.EuropeanAssociationforSignalProcessing(EURASIP).28.PauloRobertoRosaLopesNunes,AbrahamAlcaim,andMaraReginaLabutoFragosodaSilva.Qualitymeasuresofcompressedimagesforclassicationpurposes.TechnicalReportCCR-146,IBMBrasil,RioScienticCenter,P.O.Box4624,20.0001RiodeJaneiro,Brazil,October1992.29.JosephJ.K.O'RuanaidhandThierryPun.Rotation,scaleandtranslationinvariantspreadspectrumdigitalimagewatermarking.SignalProcessing,66(3):303{317,May1998.EuropeanAssociationforSignalProcessing(EURASIP).30.AdrianPerrig.Acopyrightprotectionenvironmentfordigitalimages.Diplomadissertation,EcolePolytechniqueederaledeLausanne,Lausanne,Switzerland,February1997.31.FabienA.P.Petitcolas.Weaknessofexistingwatermarkingschemes.http://www.cl.cam.ac.uk/~fapp2/watermarking/image_watermarking&#x-530;/,October1997.32.FabienA.P.PetitcolasandRossJ.Anderson.Weaknessesofcopyrightmarkingsystems.InDittmannetal.,pages55{61.33.FabienA.P.Petitcolas,RossJ.Anderson,andMarkusG.Kuhn.Attacksoncopyrightmarkingsystems.InAucsmith,pages218{238.ISBN3-540-65386-4.34.FabienA.P.PetitcolasandMarkusG.Kuhn.StirMark2.http://www.cl.cam.ac.uk/~fapp2/watermarking/,November1997.35.ChristineI.PodilchukandWenjunZeng.Digitalimagewatermarkingusingvisualmodels.InRogowitzandPappas,pages100{111.ISBN0-8194-2427-7,ISSN0277-786X.36.StephaneRocheandJean-LucDugelay.Mecanismesdesecuritlieslatransmissiondesimages.InCOompressionetREprsentationdesSignauxAudiovisuels(CORESA'97),Issy-les-Moulineaux,France,March1997.37.BerniceE.RogowitzandThrasyvoulosN.Pappas,editors.HumanVisionandElectronicImagingII,volume3016,SanJose,California,USA,February1997.TheSocietyforImagingScienceandTechnology(IS&T)andtheInternationalSocietyforOpticalEngineering(SPIE),SPIE,ISBN0-8194-2427-7,ISSN0277-786X.38.KhalidSayood.IntroducationtoDataCompression,chapter7,page142.MorganKaufmannPublishers,1996.39.MitchellD.Swanson,MeiKobayashi,andAhmedH.Tewk.Multimediadata-embeddingandwatermarkingtechnologies.ProceedingsoftheIEEE,86(6):1064{1087,June1998.40.MitchellD.Swanson,BinZu,andAhmedH.Tewk.Robustdatahidingforimages.In7thDigitalSignalProcessingWorkshop(DSP96),pages37{40,Loen,Norway,September1996.IEEE.41.K.Tanaka,Y.Nakamura,andK.Matsui.Embeddingsecretinformationintoaditheredmultilevelimage.InProceedingofthe1990IEEEMilitaryCommunicationsConference,pages216{220,September1990.42.K.Tanaka,Y.Nakamura,andK.Matsui.Embeddingtheattributeinformationintoaditheredimage.SystemsandComputersinJapan,21(7),1990.43.A.Z.Tirkel,C.F.Osborne,andT.E.Hall.Imageandwatermarkregistration.SignalProcessing,66(3):373{383,May1998.EuropeanAssociationforSignalProcessing(EURASIP).44.A.Z.Tirkel,G.A.Rankin,R.M.vanSchyndel,W.J.Ho,N.R.A.Mee,andC.F.Osborne.Electronicwatermark.InDigitalImageComputing,TechnologyandApplications(DICTA'93),pages666{673,MacquarieUniversity,Sidney,1993.45.DimitriosTzovaras,NikitasKaragiannis,andMichaelG.Strintzis.Robustimagewatermarkinginthesubbandordiscretecosinetransformdomain.In9thEuropeanSignalProcessingConference(EUSIPCO'98),pages2285{2288,IslandofRhodes,Greece,8{11September1998.ISBN960-7620-05-4.46.ChristianJ.vandenBrandenLambrechtandJoyceE.Farrell.Perceptualqualitymetricfordigitallycodedcolorimages.InProceedingofEUSIPCO,pages1175{1178,Trieste,Italy,September1996.47.RudolfL.vanRenesse,editor.OpticalSecurityandCounterfeitDeterrenceTechniquesII,volume3314,SanJose,California,USA,January1998.TheSocietyforImagingScienceandTechnology(IS&T)andtheInternationalSocietyforOpticalEngineering(SPIE),SPIE,ISBN0-8194-2556-7,ISSN0277-786X.48.R.G.vanSchyndel,A.Z.Tirkel,andC.F.Osborne.Adigitalwatermark.InInternationalConferenceonImageProcessing,volume2,pages86{90,Austin,Texas,USA,1994.IEEE. 49.G.Voyatzis,N.Nikolaidis,andI.Pitas.Digitalwatermarking:anoverview.In9thEuropeanSignalProcessingConference(EUSIPCO'98),pages9{12,IslandofRhodes,Greece,8{11September1998.ISBN960-7620-05-4.50.AllanG.Weber.Theusc-sipiimagedatabse:Version5.http://sipi.usc.edu/services/database/Database.,October1997.SingalandImageProcessingInstituteattheUniversityofSouthernCalifornia.51.S.J.P.Westen,R.L.Lagendijk,andJ.Biemond.PerceptualimagequalitybasedonamultiplechannelHVSmodel.InProceedingofICASP,volume4,pages2351{2354,1995.52.StefanWinkler.Aperceptualdistortionmetricfordigitalcolorimages.InProc.ICIP,volume3,pages399{403,Chicago,IL,October1998.53.StefanWinkler.Aperceptualdistortionmetricfordigitalcolorvideo.InSPIEProceedingsofHumanVisionandElectronicImaging,volume3644,SanJose,CA,January1999.54.RaymondB.WolfgangandEdwardJ.Delp.Awatermarkfordigitalimages.InInternationalConferenceonImagesProcessing,pages219{222,Lausanne,Switzerland,September1996.IEEE.55.RaymondB.WolfgangandEdwardJ.Delp.Awatermarkingtechniquefordigitalimagery:furtherstudies.InInternationalConferenceonImaging,Systems,andTechnology,pages279{287,LasVegas,Nevada,USA,30June{3July1997.IEEE.56.J.ZhaoandE.Koch.Embeddingrobustlabelsintoimagesforcopyrightprotection.InInternationalCongressonIntellectualPropertyRightsforSpecialisedInformation,KnowledgeandNewTechnologies,Vienna,Austria,August1995.57.MarkH.ZweigandGregoryCampbell.Receiver{operatingcharacteristics(ROC)plots:Afundamentalevalu-ationtoolinclinicalmedicine.ClinicalChemisitry,39(4):561{577,1993.