Bravo Department of Computer Science Dartmouth College Hanove r NH 03755 USA Psychology Department Rutgers University Camden NJ 081 02 USA ABSTRACT While historically we may have been overly trusting of photo graphs in recent years there has been a ID: 82241
Download Pdf The PPT/PDF document "Image Forensic Analyses that Elude the H..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
ImageForensicAnalysesthatEludetheHumanVisualSystemHanyFaridaandMaryJ.BravobaDepartmentofComputerScience,DartmouthCollege,HanoverNH03755,USAbPsychologyDepartment,RutgersUniversity,Camden,NJ08102,USAABSTRACTWhilehistoricallywemayhavebeenoverlytrustingofphotographs,inrecentyearstherehasbeenabacklashofsortsandtheauthenticityofphotographsisnowroutinelyquestioned.Becausethesejudgmentsareoftenmadebyeye,wewonderedhowreliablethehumanvisualsystemisindetectingdiscrepanciesthatmightarisefromphototampering.Weshowthatthevisualsystemisremarkablyineptatdetectingsimplegeometricinconsistenciesinshadows,re\rections,andperspectivedistortions.Wealsodescribecomputationalmethodsthatcanbeappliedtodetecttheinconsistenciesthatseemtoeludethehumanvisualsystem.Keywords:PhotoForensics1.INTRODUCTIONInanattempttoquellrumorsregardingthehealthofNorthKorea'sleaderKimJong-Il,theNorthKoreangov-ernmentreleasedaseriesofphotographsintheSpringof2008showingahealthyandactiveKimJong-Il.ShortlyaftertheirreleasetheBBCandUKTimesyreportedthatthephotographsmighthavebeendoctored.Oneofthereportedvisualdiscrepancieswasaseeminglyincongruousshadow.Becausesuchclaimsofinauthenticityareoftenmadebyeye,wewonderedhowreliablethehumanvisualsystemisindetectingdiscrepanciesthatmightarisefromphototampering.Inmanyways,thevisualsystemisremarkable,capableofhyperacuity,1rapidsceneunderstanding,2androbustfacerecognition.3Inotherarenas,however,thevisualsystemcanbequiteinept.Forexample,thevisualsystemcanbeinsensitivetoinconsistenciesinlighting,4viewingposition,5andcertainjudgmentsoflightnessandcolor.6Thereisalsosomeevidencethatobserverscannotreliablyinterpretshadows,7re\rections,8andperspectivedistortion.9Theselastthreecuescanprovideevidenceofphoto-tampering,andso,astheexampleaboveillustrates,itisimportanttounderstandhowwellobserverscanutilizethesecues.Herewereportthreeexperimentsthatcomparetheperformanceofhumanobserverswiththatofcomputationalmethodsindetectinginconsistenciesinshadows,perspectiveandre\rections.2.SHADOWS2.1HumanPerformanceFigure1(a)isarendered3-Dsceneilluminatedbyasinglelightthatproducescastshadowsonthegroundplaneandbackwall.Panel(b)ofthis gureisthesamescenewiththelightmovedtoadi erentlocation.Panel(c)isacompositecreatedbycombiningthebackwallfrompanel(a)andthegroundplanefrompanel(b)tocreateascenewithshadowsthatareinconsistentwithasinglelight.Onehundredandfortyrenderedsceneswerecreatedsuchthatthecastshadowswereeitherconsistentorinconsistentwithasinglelight.Fortheconsistentscenes,thelightwaspositionedeitherontheleftorrightsideoftheroomandinoneofninedi erentlocationsthatvariedindistancefromthegroundplaneandfromthebackwall.Fortheinconsistentscenes,thebackwallsfromsceneswithdi erentlightingwereinterchanged.Twentyobserverswereeachgivenunlimitedtimetojudgewhethertheoriginalandcompositesceneswereconsistentwithasinglelight.Theirperformancewas Contact:farid@cs.dartmouth.eduandmbravo@camden.rutgers.edu\'Fakephoto'revivesKimrumours",BBC,November12,2008y\KimJongIl:digitaltrickeryoranamazingrecoveryfromastroke?",UKTimes,November7,2008. Figure1.Thecastshadowsinpanels(a)and(b)areconsistentwithasinglelight.Thesceneinpanel(c)isacompositeofthebackwallfrompanel(a)andthegroundplanefrompanel(b).Inthiscase,thecastshadowsareinconsistentwithasinglelight.Theyellowlinesinpanel(d)connectpointsoneachobjecttotheirprojectedshadow.Theintersectionoftheselinesisthe2-Dlocationofthelight.nearlyperfect(95:5%)forinconsistentscenesthatwerethecombinationoflightsfromoppositessidesoftheroom(i.e.,thecastshadowsraninoppositedirections).Forallothercases,however,observeraccuracywasnearchance(52:8%).Theaverageresponsetimewas4:9seconds,indicatingthatobserversspentareasonableamountoftimeinspectingeachscene.2.2GeometryofShadowsAlthoughobservershaveconsiderabledicultydetectinginconsistenciesincastshadows,thereisasimpleimage-basedtechniqueformakingthisjudgment.Sincelighttravelsinastraightline,apointontheshadow,itscorrespondingpointontheobject,andthelightsourcealllieonasingleline.Therefore,thelightsourcewillalwayslieonalinethatconnectseverypointonashadowwithitscorrespondingpointonanobject,regardlessofscenegeometry.Inanimage,theprojectionoftheselineswillalwaysintersectatthe2-Dprojectionofthelightposition.Inpractice,therearesomelimitationstothisgeometricanalysisoflightposition.Caremustbetakentoselectappropriatelymatchedpointsontheshadowandtheobject;thisisbestachievedwhentheobjecthasadistinctshape(e.g.,thetipofacone).Ifthedominantlightisthesun,thenthelinesmaybenearlyparallel,makingthecomputationoftheirintersectionvulnerabletonumericalinstability.Forexample,Figure2(a)isanauthenticimagewherethesunisdirectlyabovethevehicle.Theyellowlines,connectingshadowandobject,arenearlyparallel,makingitdiculttodetermineiftheselinesintersectatasinglepoint.Ontheotherhand,theimageinFigure2(b)mustbeafakebecausethelinesclearlydiverge.Eveniftheintersectionisdicultto Figure2.An(a)authenticand(b)fakeimage.Theyellowlines,connectingshadowandobject,shouldintersectatthelocationofthelight.Thediverginglinesontherightrevealthatthecastshadowsareinconsistentwithasinglelight.compute,thisanalysiscanstillbeemployedtodetermineifanobject'sshadowisinconsistentwithasinglelight.Wealsonotethatthissimplegeometricanalysiscouldbeusedtoconditiontheestimationoflightdirectionasdescribedin.12,133.PLANARPERSPECTIVE3.1HumanPerformanceFigure3(a)isarenderedscenewiththreeplanarsurfacestexture-mappedwiththesame2-Dimage.Panel(b)ofthis gureisthesamescenewiththetexturemapontheleft-mostpanelskewedrelativetothecentralpanel.Threeplanarsurfaceswereplacedinthecon gurationshowninFigure3,withonlyoneofthethreeplanestexture-mappedwithoneofsiximagesoffamiliarobjects.Theimagewastexture-mappedwithnodistortionorwithhorizontalskew.Thehorizontalskewwascreatedbyapplyingtheanematrix[1s;01]totheimage,wheretheamountofskewsvariedfrom2to8.FortheleftplaneinFigure3,anegativeskewcounteractedsomeoftheperspectiveplanardistortion,whileapositiveskewexaggeratedtheperspectivedistortion.Ontheotherhand,apositiveskewwouldhaveexaggeratedtheperspectivedistortion.Twentyobserverswereeachshownexamplesofundistortedandskewedimages,andinstructedthattheirtaskwouldbetodetermineifanimagewasskewed.Inthecenterpanelcondition,therewasminimalperspectivedistortionandweexpectedallobserverstoaccuratelydetectskewintheimage.Nonetheless, veobserversperformedbelow70%overallandtheirdatawereeliminatedfromthisanalysis.Theremainingobserversshowedtheexpectedpatternofresultsforthecenterpanel,reliablydetectinglargepositiveandnegativeskews(centerpanelofFigure4).Ofinterestishowtheseobserversperformedwhenthepanelwasviewedobliquelyandtheimagewassubjectedtoperspectivedistortions(leftandrightpanels).Herethepatternofresultsisasymmetricaldependingonwhethertheskewintheimageexaggeratedorcounteractedtheperspectivedistortion.Whentheskewexaggeratedtheperspectivedistortion(positiveimageskew,leftpanel;negativeimageskew,rightpanel)performancewasgood.Whentheskewscounteractedtheperspectivedistortion,performancewasatorbelowchance.Thatis,observerswereunabletodetectevenalargeskewwhenthee ectsofperspectivedistortedcounteractedtheskew.Theseresultssuggestthatobserversunderestimateorpossiblyevenignorethee ectsof Figure3.Theimageoneachoftheplanesinpanel(a)arethesame.Theimageontheleft-mostplaneinpanel(b)isaskewedversionoftheimageonthecentralplane,asshownintherecti edimagesbelow. -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 0 20 40 60 80 100 skew'skew' responses (%) -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 0 20 40 60 80 100 skew'skew' responses (%) -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 0 20 40 60 80 100 skew'skew' responses (%) Figure4.Skewestimationaccuracyforeachofthreeplanarpanels(left,center,right,respectively)showninFigure3.perspectiveprojection.Averagingoverallconditions,observeraccuracyfortheleftandrightpanelwas63:1%and64:4%,comparedto82:6%onthecenterpanel.Theaverageresponsetimewas2:8seconds,indicatingthatobserversspentareasonableamountoftimeinspectingeachscene.3.2GeometryofPlanarPerspectiveObserversroutinelyunderestimatetheamountofdistortioncausedbyplanarperspectivedistortion.Fromacomputationalpointofview,theperspectivetransformationofaplanarsurfaceisrelativelyeasytomodelandestimate.14,15Themappingfrompointsin3-Dworldcoordinatesto2-Dimagecoordinatescanbeexpressedbytheprojectiveimagingequation:~x=P~X,wherethe34matrixPembodiestheprojectivetransform,thevector~Xisa3-Dworldpointinhomogeneouscoordinates,andthevector~xisa2-Dimagepointalsoinhomogeneouscoordinates.Inthecasewhenalloftheworldpoints~Xlieonasingleplane,thetransformreducestoa33planarprojectivetransformH,alsoknownasahomography:~x=H~X;(1)wheretheworld~Xandimagepoints~xarenowrepresentedby2-Dhomogeneousvectors.InordertoestimatethehomographyH,webeginwithacrossproductionformulationofEquation(1):~x[H~X]=00@x1x2x31A240@h1h2h3h4h5h6h7h8h91A0@X1X2X31A35=0:(2)Evaluatingthecrossproductyields:0@x2(h7X1+h8X2+h9X3) x3(h4X1+h5X2+h6X3)x3(h1X1+h2X2+h3X3) x1(h7X1+h8X2+h9X3)x1(h4X1+h5X2+h6X3) x2(h1X1+h2X2+h3X3)1A=0:(3)Thisconstraintislinearintheunknownelementsofthehomographyhi.Re-orderingthetermsyieldsthefollowingsystemoflinearequations:0@000 x3X1 x3X2 x3X3x2X1x2X2x2X3x3X1x3X2x3X3000 x1X1 x1X2 x1X3 x2X1 x2X2 x2X3x1X1x1X2x1X30001A0BBBBBBBBBBB@h1h2h3h4h5h6h7h8h91CCCCCCCCCCCA=0A~h=0:(4) Figure5.Thefaceofeachcigaretteboxinpanel(a)isrecti edusingtheknowndimensionsoftheboxface.Showninpanels(b)and(c)aretherecti edimages,revealinganinconsistencyinthecartooncharacterandtext.Amatchedsetofworld~Xandimage~xcoordinatesappearstoprovidethreeconstraintsontheeightunknownelementsofH(thehomographyisde neduptoanunknownscalefactor,reducingthenumberofunknownsfromninetoeight).TherowsofthematrixA,however,arenotlinearlyindependent(thethirdrowisalinearcombinationofthe rsttworows).Assuch,thissystemprovidesonlytwoconstraintsintheeightunknowns.Therefore,atotaloffourormorepointswithknownworldandimagecoordinatesarerequiredtoestimatethehomography.Fromthesepoints,standardleast-squarestechniquescanbeusedtosolvefor~h:theminimaleigenvalueeigenvectorofATAistheunitvector~hthatminimizestheleast-squareserrorkA~hk2.TheinversehomographyH 1isappliedtotheimagetoremoveplanarperspectivedistortion.Figure5(a)isaforgeryofourcreation{twoboxesofMarlborocigarettesweredoctoredtoread\Marlborokids"withanimageofthecartooncharacterTweetyBird.Onbothboxes,the\kids"textandthecharacterweremanuallyadjustedtogivetheappearanceofcorrectperspective.Figure5(b)and(c)aretheresultsofplanarrecti cationbasedontheknownshapeoftherectangleonthefrontofthebox(111=1631=8inches,determinedbymeasuringanactualboxofcigarettes).Notethatafterrecti cationthetextandcharacterontheboxesareinconsistentwithoneanother,clearlyrevealingtheimagetobeafake.4.REFLECTIONS4.1HumanPerformanceFigure6(a)isarendered3-Dscenecontainingaredconeandamirror.Panel(b)ofthis gureisthesamescenewiththeconedisplacedrelativetothemirror.Panel(c)isacompositecreatedbyreplacingthecorrectre\rectioninpanel(a)withthatfrompanel(b)tocreateaphysicallyimpossiblescene.Three-dimensionalrenderedscenesweregeneratedsuchthatthere\rectionwaseitherconsistentorinconsistentwiththescenegeometry,Figure6.Thesceneswererenderedwiththeviewerinoneofthreelocationsrelativetothere\rectivemirror,either10(nearlyfronto-parallel)or60relativetothemirror.Foreachviewingdirection,theobject(redcone)wasmovedtooneofthreelocationsalongthegroundplane.Theinconsistentscenesweregeneratedbycombiningthere\rectionfromonescenewiththeobjectfromanother,alwaystakenfromsceneswiththesameviewingdirection.Twentyobserverswereeachpresentedwiththesescenes(14consistentand28inconsistent)andgivenunlimitedtimetodetermineifthere\rectionineachwascorrect.Theaverageaccuracyoverallviewingconditionswasonly55:7%,slightlybetterthanchance.Theaverageresponsetimewas7:6seconds,indicatingthatobserversspentareasonableamountoftimeinspectingeachscene.4.2GeometryofRe\rectionsObserverswerelargelyunabletopredictthelocationofanobject'sre\rectioninaplanarmirror.Thisfailuremightnotseemsurprisinggiventhatthetaskrequiresknowledgeof3-Dscenegeometry.However,ifthere\rective Figure6.There\rectionsintheplanarmirrorinpanels(a)and(b)areconsistentwiththescenegeometry.Thesceneinpanel(c)isacompositeoftheobjectfrompanel(a)andthere\rectionfrompanel(b).Inthiscase,there\rectionandscenegeometryareinconsistentTheyellowlinesinpanel(d)constrainthelocationoftheobjectrelativetothere\rection.surfaceisplanarwithknowndimensions,thereissucientinformationintheimagetoconstraintherelationshipbetweenanobjectanditsre\rection.Thelawofre\rectionstatesthatalightrayre\rectsfromasurfaceatanangleofre\rectionrequaltotheangleofincidencei,measuredwithrespecttothesurfacenormal.Assumingunit-lengthvectors,thedirectionfromthere\rectiontotheobject~Zcanbedescribedintermsoftheviewdirection~Vandsurfacenormal~Nas:~Z=2cos(i)~N ~V=2(~VT~N)~N ~V:(5) Inordertoestimatetheviewdirectionandsurfacenormalinacommoncoordinatesystem,wemust rstdeterminethehomographyHthatmapsthere\rectiveplanarsurfacefromworldtoimagecoordinates.ThisestimationisthesameasthatdescribedinSection3.Onceestimated,thehomographyisfactoredas:H=K(~r1~r2~t);(6)whereisascalefactor,thematrixKembodiestheinternalcameraparameters,andwheretherigidbodytransformationfromworldtocameracoordinatesisspeci edbyatranslationvector~t,andarotationmatrix Rwhose rsttwocolumnsare~r1and~r2.Thefullrotationmatrixis(~r1~r2~r1~r2):Ifweassumethatthecamerahasunitaspectratioandzeroskew(i.e.,squarepixels),andthattheprinciplepointistheimagecenter,thentheintrinsicmatrixsimpli esto:K=0@f000f00011A;(7)wherefisthefocallength.SubstitutingthisintrinsicmatrixintoEquation(6)gives:H=0@f000f00011A(~r1~r2~t):(8)Left-multiplyingbyK 1yields:0@1 f0001 f00011AH=(~r1~r2~t)0@1 f0001 f00011A0@h1h2h3h4h5h6h7h8h91A=(~r1~r2~t)(9)Because~r1and~r2arethe rsttwocolumnsofarotation(orthonormal)matrix,theirinnerproduct,~rT1~r2,iszero,leadingtothefollowingconstraint:240@1 f0001 f00011A0@h1h4h71A35T240@1 f0001 f00011A0@h2h5h81A35=0(h1h4h7)0@1 f20001 f200011A0@h2h5h81A=0:(10)Thefocallengthisestimatedbysolvingtheabovelinearsystemforf:f=r h1h2+h4h5 h7h8:(11)Theadditionalconstraintthat~r1and~r2areeachunitlength,~rT1~r1=~rT2~r2,canalsobeusedtoestimatethefocallength.14Thescalefactorisdeterminedbyenforcingunitnormonthecolumnsoftherotationmatrix:=1 kK 1~h1k=1 k~r1k(12)where~h1isthe rstcolumnofthematrixH.Withthehomographyfactored,thedesiredviewdirectionandsurfacenormalcaneachbeestimatedinacommoncoordinatesystem.Theviewdirection,incameracoordinates,isgivenby:~V=1 K 1H(XY1)T;(13)where(XY)isthelocationofthere\rectioninworldcoordinates.Thesecoordinatesaredeterminedby rstapplyingH 1totheimage,inordertoplanarrectifythere\rectivesurface.Apoint(XY)onthere\rectionisthenselected.Withoutlossofgenerality,weassumethattheworldplaneispositionedatunitlengthfromthe Figure7.FlamingosfromMiami'sMetroZooseekshelterfromHurricaneGeorges(JoeCavaretta/AssociatedPress/Sept.1998).Ontherightisamagni edviewofthe\ramingosandtheirre\rection.Theyellowlinesshowthatthepositionofthere\rectionsareconsistentwiththescenegeometry.origin(i.e.,Z=1).Thesurfacenormalinworldcoordinatesis(00 1)T(i.e.,alongtheZ-axisandfacingtheorigin),andincameracoordinates:~N=R(00 1)T:(14)With~Vand~Nestimated,thedirection~ZtotheobjectisthendeterminedfromEquation(5).Notethatallofthesedirectionsandnormalsarespeci edinthecameracoordinatesystem.Assuch,theycaneachbeprojectedintoimagecoordinatesandusedtodetermineifanobjectanditsre\rectionareconsistent.Speci cally,intheoriginalimage,alineisdrawnfromapointonthere\rectiontoK(~V+~Z),Figure6(d).Note,however,thatthisconstraintdoesnotuniquelyde nethelocationoftheobject,asthere\rectionisconsistentwithanobjectanywherealongtheconstraintline.Figure7isaseeminglyimprobable,albeitauthenticimage.Therightpanelisamagni edviewofthe\ramingosandtheirre\rection.Becausethetilessurroundingthemirroraresquare,theywereusedtoestimatetheaspectratioofthemirror.Thisknownaspectratioof0:65wasusedtoestimatethehomographyH,fromwhichtheobjectlocationofare\rectionwasestimated.Figure7aretwosuchestimates,wheretheyellowlinesconnectpointsinthemirrorwiththeircorrespondingreal-worldlocations.Anyinconsistenciesintheselocationscouldbeusedasevidenceoftampering.5.DISCUSSIONThehumanvisualsystemis,attimes,remarkablyineptatdetectingsimplegeometricinconsistenciesthatmightresultfromphototampering.Wedescribedthreeexperimentsthatshowthatthehumanvisualsystemisunabletodetectinconsistenciesinshadows,re\rections,andplanarperspectivedistortions.Atthesametime,wehavedescribedcomputationalmethodsthatcanbeappliedtodetecttheinconsistenciesthatseemtoeludethehumanvisualsystem.Theseresultssuggestthatcareshouldbetakenwhenmakingjudgmentsofphotoauthenticitybasedsolelyonvisualinspection.ACKNOWLEDGMENTSThankstoChristopherKapsalesforhishelpwiththedatacollection.ThisworkwassupportedbyagiftfromAdobeSystems,Inc.,agiftfromMicrosoft,Inc.,andagrantfromtheNationalScienceFoundation(CNS-0708209). REFERENCES[1]Westheimer,G.andMcKee,S.,\Visualacuityinthepresenceofretinal-imagemotion,"JournaloftheOpticalSocietyofAmerica65,847{850(1975).[2]Potter,M.,\Short-termconceptualmemoryforpictures,"JournalofExperimentalPsychology:HumanLearningandMemory2,509{522(1976).[3]Sinha,P.,Balas,B.,Ostrovsky,Y.,andRussell,R.,\Facerecognitionbyhumans:19resultsallcomputervisionresearchersshouldknowabout,"ProceedingsoftheIEEE94(11),1948{1962(2006).[4]Ostrovsky,Y.,Cavanagh,P.,andSinha,P.,\Perceivingilluminationinconsistenciesinscenes,"Percep-tion34,1301{1314(2005).[5]Vishwanath,D.,Girshick,A.,andBanks,M.,\Whypictureslookrightwhenviewedfromthewrongplace,"NatureNeuroscience10(8),1401{1410(2005).[6]Adelson,E.H.,[TheNewCognitiveNeurosciences,2ndEdition],ch.LightnessPerceptionandLightnessIllusions,339{351,MITPress(2000).[7]Jacobson,J.andWerner,S.,\Whycastshadowsareexpendable:Insensitivityofhumanobserversandtheinherentambiguityofcastshadowsinpictorialart,"Perception33(11),1369{1383(2004).[8]Bertamini,M.,Spooner,A.,andHecht,H.,\Predictingandperceivingre\rectionsinmirrors,"JournalofExperimentalPsychology:HumanPerceptionandPerformance39,982{1002(2003).[9]Bravo,M.andFarid,H.,\Textureperceptiononfoldedsurfaces,"Perception30(7),819{832(2001).[10]Ritschel,T.,Okabe,M.,Thormahlen,T.,andSeidel,H.-P.,\Interactivere\rectionediting,"ACMTrans.Graph.(Proc.SIGGRAPHAsia2009)28(5)(2009).[11]Zhang,W.,Cao,X.,Zhang,J.,Zhu,J.,andWang,P.,\Detectingphotographiccompositesusingshadows,"in[IEEEInternationalConferenceonMultimediaandExpo],1042{1045(2009).[12]Johnson,M.andFarid,H.,\Exposingdigitalforgeriesbydetectinginconsistenciesinlighting,"in[ACMMultimediaandSecurityWorkshop],(2005).[13]Johnson,M.andFarid,H.,\Exposingdigitalforgeriesincomplexlightingenvironments,"IEEETransac-tionsonInformationForensicsandSecurity3(2),450{461(2007).[14]Hartley,R.andZisserman,A.,[MultipleViewGeometryinComputerVision],CambridgeUniversityPress(2004).[15]Johnson,M.andFarid,H.,\Metricmeasurementsonaplanefromasingleimage,"Tech.Rep.TR2006-579,DepartmentofComputerScience,DartmouthCollege(2006).