DidyketalPerceptuallymotivatedRealtimeTemporalUpsamplingof3DContentforHighrefreshrateDisplays Figure2SimulationofholdtypeblurAnanimationsequencewiththesampleframeasshownontheleftisdisplayed ID: 395409
Download Pdf The PPT/PDF document "Didyketal./Perceptually-motivatedReal-ti..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Didyketal./Perceptually-motivatedReal-timeTemporalUpsamplingof3DContentforHigh-refresh-rateDisplaysThegoalofthisworkistocombatickeringandblurartifactswhilemaintainingtheoriginalappearanceofthe3Dscenerenderingusinghigh-refresh-ratedisplays.Onesolutionistocombathold-typebycreatingmanyframes,butthisleadstoanoverhead,makingperformanceoftheframesynthesisacrucialaspect.ThisisexpectedtobeanevenmoreimportantissueforthenewlyappearingSuper-HDdisplays(40962160resolution).Oursolutionproducesadditionalframesatasceneindependentcost,whichalsomakesitwell-suitedtoincreaseperformance.Ourcontributionisatemporalup-samplingscheme,tai-loredtowardstherequirementsofhigh-refresh-ratehold-typedisplaysandthecapabilitiesformoderngraphicshardwareatthesametime.Weexploittheinformationondepth,occlu-sionandthree-dimensionalstructure,madeavailableviatheGPUaswellasperceptualndingstooutperformpureimage-basedup-sampling.WediversifysubsequentframesintermsofspatialfrequencycontenttoaccommodateimagefusioncharacteristicsintheHVS.Therequiredstepsaresimple;donotintroducelagintothevideostreamandcanextrapolateoneorevenmultipleframes.Weperformapsychophysicalstudythatevaluatestaskperformanceandcomparesperceivedqualityforfulltemporalresolution,lowtemporalresolution,andtemporallyup-sampledvideostreams.Theseadvantagesmakeourprocessacheapandmore-suitedalternativetopro-ducing120Hzcontentdirectly.Thispaperisorganizedasfollows.Sections2,3presentselectedaspectsofblurperception.Wereviewrelatedworkonhold-typeblurreductionaswellastemporalup-samplingrequiredtoincreaseimagerefreshratesinSection4.OurapproachisdescribedinSection5,anditsimplementationinSection6.Results,includingaperceptualstudy,arepresentedinSection7.WeconcludeinSection8.2.TemporalPerceptionofBlurandSharpnessImagesharpnessisanimportantfactorwhichdecidesuponperceivedimagequality[Jan01].Perceptualstudiesclearlyindicate[CF03,LGK06]thatpeoplepreferimageswithin-creasedcontrastatedges.Thisisoftenachievedbyapplyingimagesharpeningtechniquessuchasunsharpmasking.Notsurprisingly,blurintheimageisconsideredanartifactandisparticularlyannoyingwhenpresentintheregionsofinter-estthatattractvisualattentionsuchasmovingobjects.Blurperceptionisacomplexphenomenon,whichisaffectedbycharacteristicsoftheHVSsuchastemporalintegrationintheretina,eyemotion,andvisualillusions.Itisoftenassumedthattemporalintegrationofinforma-tionbytheHVSfollowsBloch'slaw[GT86]whichstatesthatthedetectabilityofstimuliwithsimilarspatialcharac-teristicsdependsolelyontheirenergy,i.e.theproductofluminanceandexposuretime.Inthecontextofhigh-refresh-ratedisplays,wheresubsequentimagesarefusedbytheHVS,thissuggeststhatafeaturedisplayedwithenhancedintensityinasingleframeisperceivedinthesamewayasthesamefeaturepresentintwoframes,eachwithahalvedintensity.Weexploitthisobservationtomaintainalocalaveragepowerineachfrequencycomponentandamplifyhighfrequenciesinfullyrenderedframestocompensatefortheblurred(warped)framesthatfollow.NotethatBloch'slawisvalidonlyuptosomecriticalduration(around4010msdependingonspatialfrequency[GT86]),whichismetbyhigh-refresh-ratedisplays.Forlongerdurationsonlypixelluminancematters.Motionblurnaturallyarises,whenretinalimagesofob-jectsmoverelativelytotheretina,whichmaybecausedbytheactualobjectmotion,eyemotion,orboth.Sincephotore-ceptorsintheretinaintegratesignalovertimebyananalogytotheniteexposuretimeincameras,theretinalimageac-quiredinsuchconditionsisblurred.Aswementionedintheintroduction,thesmoothpursuiteyemotionstabilizesamov-ingobjectonthefovea,whichisefcientmostlyforsimplemotionswithconstantvelocityoverpredictabletrajectories.However,eyetrackinghasitslimitations[Dal98].Forlowangularvelocitiesbelow0.15deg/sthedrifteyemovementinterfereswiththesmoothpursuiteyemotion.Similarly,forvelocitieshigherthan80deg/strackingbecomesimpossible.Aninterestingvisualillusionistheso-calledmotionsharp-ening[RRV74].Surprisingly,theHVSseemstobeequippedwithamotiondeblurringmechanismwhichmaycausemov-ingblurredimagestoappearsharperthantheirstaticcounter-part.E.g.,WesterinkandTeunissen[WT95]haveobservedthatforvelocitieshigherthan15–20deg/stheperceivedsharp-nessofimagesblurredwitha6-pixel-widelterappearssimilartotheoriginalsharpimagesundergoingthesamemotion.TakeuchiandDeValois[TV05]investigatedthemotionsharpeningeffectbyinterleavingsharpandblurredframes.Theviewerscouldnotseeanydifferenceinthevideosharpnesseveniftwothirdsoftheimageshadbeenblurred,buttheycomplainedaboutickeringforlowrefreshrates.Thisideahassuccessfullybeenexploitedinvideocompres-sionandtransmissionapplications[FB08],whereselectedframeshavebeenlteredoff-linetoreducetherequiredband-width.Inthiswork,weexploittheseeffecttoproposeanefcientimagewarpingtechniquethatenablesicker-freehigh-refresh-raterendering.3.PerceptionofDisplaysSofar,inourdiscussionofblurperception,weassumedthatimagesofmovingobjectsareperfectlyreproducedintermsofmotionsmoothness,meaningthatsignaltransitionsatretinalphotoreceptorsfollowreal-worldobservationconditions.Thisassumptionisnotvalidfortheexistingdisplaytech-nology.Inthissection,wefocusontoday'spredominanthold-typeLCDdisplays.Theyexhibittwoprominentformsofblur:responsetimeblurandhold-typeblur[PFD05].Botharenotpresentinimpulse-typeCRTdisplays,forwhichotherdrawbacksexist,suchasickering,lowerbrightness,andreducedcontrast[KV04].c 2009TheAuthor(s)Journalcompilationc 2009TheEurographicsAssociationandBlackwellPublishingLtd. Didyketal./Perceptually-motivatedReal-timeTemporalUpsamplingof3DContentforHigh-refresh-rateDisplays Figure2:Simulationofhold-typeblur.Ananimationse-quencewiththesampleframeasshownontheleftisdis-playedsimultaneouslywith60and120HzrefreshrateonaSamsung2233RZ120Hzdisplay.Theeffectivevelocityofhor-izontalmotionasseenonthescreenisthesameinbothcases.Theuser'staskistoadjusttheblurinthesequencerefreshedwith120Hzuntilthelevelofblurmatchesthe60Hzsequence.Theaverageoutcomeofsuchanexperimentisshownontheright.Inotherwords,thesequenceofblurredframes(right)at120Hzarevisuallyequivalenttothesequenceofsharpframes(left)displayedat60Hz.blurandrefertheinterestedreadertotheextensivesurveybySungetal.[SPW02].Hold-TypeBlurCompensationManymodernTVsetsfea-tureincreasedrefreshratessuchas100and200Hz(respec-tively120and,240HzfortheNTSCstandard)asameanstoreducehold-typeblur.Intermediateframesareaddedinter-nallyrelyingonlyonastandard(low-framerate)broadcastingsignal.Fengetal.[FPD08]provideanextensivesurveyofexistingmethodsforhold-typeblurreduction.Thesimplestoptionisblackdatainsertion(BDI)whichmeansinterleavingoriginalframeswithblackframestore-ducetheholdtime.Thiscomesattheexpenseofickering(bestvisibleinbrightatregions),brightnessreduction,andcolordesaturation.Originalframescanalsobeduplicatedbyaddingblurredcopies[CKL05],butthiscausesvisibleghostingasnomotioncompensationisperformed.Backlightashing(BF)[PFD05,Fen06]isanefciental-ternativeandovercomestheLCresponsetimeproblemsbyashingthebacklight(500and600HzTVssetsareavail-able).Inmoderndevices,backlightsarebuiltoutofhundredsofLEDs,thatareashedonlyaftertheLCDreachesitstargetlevel.Althoughhelpfulforresponse-timeblur,thisapproachispronetovisibleickeringandreducesbrightnessduetoshorterbacklightdutycycles.NotethatBFandBDIessen-tiallymimicimpulse-typedisplays,suchasCRT.They,hence,reintroducedrawbacksofolderdisplays.Frameratedoublingbyinterpolation(FRT)[Kur01]de-rivesin-betweenframes.Eachpairoforiginalframesisinter-polatedalongtheopticalowtrajectories.Opticalowisadifcultproblem,pronetoartifactswhichaffectthequalityofin-betweenframes.Ourexperimentalinvestigations,whichweconductedonamodernTVsetus-ingahighspeedcamera(1,200frames-per-second),revealedthatsuchalgorithmstendtofailforocclusions,highvelocitymotion,andhighlytexturedregions.Toavoidvisibleerrorsopticalowisautomaticallydeactivatedincaseofdoubtsandoriginalframesaresimplyreplicated,atexpenseofin-creasinghold-typeblur.Evensuchprecautionsdonothelpinallsituationsandobjectionableartifactsstillcanappearforsomerealisticscenarios.Intheexecutableaccompanyingtothispaperweshowapathologicalcase,whichcannotbehandledbythetestedTVset.Inthisrespectourtechniqueisquitegeneralandadditionallybenetsfromhigh-qualitymotionowderivedfrom3Dsceneandcameradata.Anotherdrawbackis,thatinterpolationnaturallyresultsinatimelagwhichisnotaproblemforbroadcastingap-plications,butcannotnecessarilybetoleratedforgamesorotherinteractiveapplications.Ifaninputarrivesbetweentwoframes(nomatterthedisplayfrequency),theinterac-tionisvisualizedonlyinthenextframe.Thus,60Hzre-actwith30Hzintheworstcase.Perceptualexperimentsshowedthatsubjectscoulddetectdelaysintheinteractionevenbeyond90Hz[LWC02].Interestingly,itissometimesstatedthatbeyond60Hz,noperformanceincreaseispossi-ble[LWC02].Thisdependsstronglyonthetaskanddisplay.Ourstudyshowsthatindynamicenvironmentshigherrefreshratesdohaveanimportantimpact.Infact,temporalvisuallagscanbeperceivedasastrongdistractionforsomecross-modalities,asstudiedforaudio[DS80],haptics[LMM00]orphysics[OD01].Motion-compensatedinverseltering(MCIF)[KV04]isasoftwarealternativetosuppresshold-typeblur.Motionvec-torsarelocallycomputedusingaspace-timerecursivesearchblockmatcher.Itisthenassumedthateyetrackinglocallyfollowsthesemotionvectorsandeachframeissharpenedusingalocal1Dlter,whichisorientedalongthemotiondirection.Thesharpeningstrengthischosentocompensateforhold-typeblur.Theeffectivenessofthismethodislimitedbythefactthathold-typeblurremoveshighestfrequenciesirrespectivelyoftheirenhanced(sharpened)contrast.Also,theamountofsharpeningthatisrequiredforperfectblurcom-pensationwouldleadtoextremelterband-passproperties,whichisnotfeasibleduetopossibleintensityclipping.TVsetmanufacturersalsousehybridsolutionsthatcom-bineBFandFRTathigherrefreshrates,butdetailsonsuchcustomsolutionsarenotpublished.NotethatourapproachcouldbecombinedwithselectedtechniquessuchasBF.TemporalUp-SamplingDisplayingmoreframesisoneop-tiontoremovehold-typeblur.Generatingtheseframesre-latestomorphing,aclassiccomputergraphicsproblem,seeWolberg'ssurvey[Wol98].Wemorphframesusing3Din-formationgeneratedasaby-productofGPUrenderingandexploitperceptualndingstocompensateforinaccuracies.Stichetal.[SLW08]addressperceptualeffectsintempo-ralup-samplingofimagesequences.Theyshowthathigh-qualitymovingedgesareakeyfeaturetotheHVSandthatc 2009TheAuthor(s)Journalcompilationc 2009TheEurographicsAssociationandBlackwellPublishingLtd. Didyketal./Perceptually-motivatedReal-timeTemporalUpsamplingof3DContentforHigh-refresh-rateDisplays Figure3:Ourpipeline,fromlefttoright:Toextrapolateone(ormultiple)in-betweenframes,weusemotionowtowarpthepreviouslyshadedresultintoanin-betweenframe,thatisthenlocallyblurredtohideartifactscausedbymorphingfailures.Finally,wecompensateforthelosthigh-frequenciesduetothisblurbyaddingadditionalhighfrequencieswherenecessary.ghosting,andcomprisededgesharpness,botharisingfromhold-typeblur,canbeastrongdistraction.Theyimprovetheperceivededgequalitybymakingtheirmovementmorecoherentovertimeviainterpolation.Liuetal.[LGJA09]usecontent-preservingwarpstostabi-lizevideo.Stabilizedframesarewarpedsuchthattheoriginalimagecontentremainsintact.Wecanrelyonblurinsteadofcontent-preservingwarps.Thelatterarecomputationallyexpensiveandnotrequiredathighframerates.Up-samplingforvideoshasrecentlybeenaddressedbyMahajanetal.[MHM09].Theirworkisverywell-suitedforasingledisocclusion,butitrequiresfullknowledgeoffu-tureframesandcanbecomputationallyexpensive(reportingseveralordersofmagnitudemorecomputepower).Asmentionedbefore,theseinterpolationsintroducealag,whichcanbeanissueforreactivity.Anotheradvantageis,thatourup-samplingoperatortakesafewmilliseconds,makingitsuitableforreal-timepurposes,whereastheotherhere-mentionedmethodstargetofinevideo.Inthecontextof3Dinteractiveapplications,up-samplinghasbeenaddressedbyMarketal.[MMB97].Theyre-projectshadedpixelsexploitinginformationfromthedepthbuffer.Suchre-useofshadedsamplesisalsothebasisoftheRen-derCache[WDP99]whichiseffective,e.g,inglobalillu-minationwheresamplesareveryexpensive.MorerecentlyNahabetal.[NSL07]proposedanewcachingscheme,ex-changingforwardforreversemapping.5.OurApproachInthissectionwegiveahigh-levelintroductiontoourup-samplingscheme,whileitsimplementationisdetailedinSection6.Ourapproachiseffective,improvingqualitybyusing3Dinformation(i.e.occlusions)andbyexploitingthelimitationsoftheHVS(i.e.insertionofblurathighfrequenciesisnotdetectable).5.1.PipelineToreducehold-typeblurweproposetoup-samplethestreamofrenderedimagesusingapipelineasdepictedinFig.3.Informationextractedduringrenderingallowsustoimprovethequalityofin-betweenframesandacceleratetheircompu-tation.Toextrapolateone(ormultiple)in-betweenframes,weusemotionowtowarpthepreviouslyshadedresultthatisthenblurred[CKL05]whererequired.Theadditionalblurallowsustohideartifactsifmorphingfailsandmakesex-trapolationsufcientlyaccurate.Thelosthighfrequenciesarecompensatedforbyaddingadditionalhighfrequencieswhereneeded.MotionowContrarytomotionowfromvideos,wecanextractmotionowduringrendering.Thegraphicscardhasknowledgeaboutobjectdisplacements,whichisdifferentfromspecialdisplaystocombathold-typeblurbecausetheyneedtoreverseengineerimperfect3Dmotionviaopticalow.Bytakingthedifferenceinpositionforeveryvertex,wecancomputeperfectmotionowandrasterizeitintoabuffer.Whilehigher-ordermotionmodelsarepossible,alinearassumptionprovedsufcientinourtests.MorphingMorphingtakestheoriginalframeandmapsev-erypixelintoitsnewpredictedposition,butthiscanbecostly.Inourimplementation,wemakethismappingpiecewiselin-earbymappingasubsetofpixels–agrid–andextrapolatingthedeformationoverthisgrid.Morphingmightmapmultiplesourcepixelstoasingledestinationpixel.Wecanresolvesuchambiguities,byrelyingondepth,extractedjustlikethemotionow.Notethatsuchinformationisnotavailabletoimage-basedapproachessuchasthoseusedbyTVmanufacturers.Wewillshowthatblurcanremoveinconsistenciestoalargeextent,butmorphingaxedresolutionregulargridcanleadtosignicantartifactsthatarenoteasilyxable.c 2009TheAuthor(s)Journalcompilationc 2009TheEurographicsAssociationandBlackwellPublishingLtd.