From Reference Frames to Reference Planes MultiView Parallax Geometry and Applications - PDF document

FromReferenceFramestoReferencePlanes:Multi-ViewParallaxGeometryandApplicationsM.Irani,P.Anandan,andD.WeinshallDept.ofAppliedMathandCS,TheWeizmannInst.ofScience,Rehovot,Israel,MicrosoftResearch,OneMicrosoftWay,Redmond,WA98052,USA,InstituteofComputerScienceHebrewUniversity91904Jerusalem,IsraelThispaperpresentsanewframeworkforanalyzingthege-ometryofmultiple3Dscenepointsfromuncalibratedimages,basedondecomposingtheprojectionofthesepointsontheimagesintotwostages:(i)theprojectionofthescenepointsontoa(realorvir-tual)physicalreferenceplanarsurfaceinthescene;thiscreatesavirtualimageŽonthereferenceplane,and(ii)there-projectionofthevirtualimageontotheactualimageplaneofthecamera.Thepositionsofthevirtualimagepointsaredirectlyrelatedtothe3Dlocationsofthescenepointsandthecameracentersrelativetothereferenceplanealone.Alldependencyontheinternalcameracalibrationparametersandtheori-entationofthecameraarefoldedintohomographiesrelatingeachimageplanetothereferenceplane.Bi-linearandtri-linearconstraintsinvolvingmultiplepointsandviewsaregivenaconcretephysicalinterpretationintermsofgeometricrela-tionsonthephysicalreferenceplane.Inparticular,thepossibleduali-tiesintherelationsbetweenscenepointsandcameracentersareshowntohavesimpleandsymmetricmathematicalforms.Incontrasttotheplane+parallax(p+p)representation,whichalsousesareferenceplane,theapproachdescribedhereremovesthedependencyonareferenceim-ageplaneandextendstheanalysistomultipleviews.Thisleadstosim-plergeometricrelationsandcompletesymmetryinmulti-pointmulti-viewduality.Thesimpleandintuitiveexpressionsderivedinthereference-planebasedformulationleadtousefulapplicationsin3Dsceneanalysis.Inparticular,simplertri-focalconstraintsarederivedthatleadtosimplemethodsforNewViewSynthesis.Moreover,theseparationandcompactpackingofcameracalibrationandorientationintothe2Dprojectiontransformation(ahomography)allowsalsopartialreconstructionusingpartialcalibrationinformation.Keywords:Multi-pointmulti-viewgeometry,uncalibratedimages,newviewsynthesis,dualityofcamerasandscenepoints,plane+parallax,trilinearity. M.IraniissupportedinpartbyDARPAthroughARLContractDAAL01-97-K-0101 1IntroductionTheanalysisof3Dscenesfrommultipleperspectiveimageshasbeenatopicofconsiderableinterestinthevisionliterature.Giventwocalibratedcameras,theirrelativeorientationscanbedeterminedbyapplyingtheepipolarconstrainttotheobservedimagepoints,andthe3Dstructureofthescenecanberecoveredrelativetothecoordinateframeofareferencecamera(referredtohereasthereferenceframe…e.g.,see[13,6]).ThisisdonebyusingtheepipolarconstraintandrecoveringtheEssentialMatrixŽEwhichdependsontherotationbetweenthetwocameras.ConstraintsdirectlyinvolvingtheimagepositionsofapointinthreecalibratedviewsofapointhavealsobeenderivededIfthecalibrationofthecamerasisunavailable,thenitisknownthatre-constructionisstillpossiblefromtwoviews,butonlyuptoa3Dprojectivetransformation[4].Inthiscasetheepipolarconstraintstillholds,buttheEssen-tialMatrixisreplacedbytheFundamentalMatrixŽ,whichalsoincorporatestheunknowncameracalibrationinformation.The3Dscenepoints,thecameracentersandtheirimagepositionsarerepresentedin3Dand2Dprojectivespaces(usinghomogeneousprojectivecoordinates).Inthiscase,thereferenceframereconstructionmayeitherbeareferencecameracoordinateframe[8],orasde-“nedbyasetof5basispointsinthe3Dworld[14].Acompletesetofconstraintsrelatingtheimagepositionsofmultiplepointsinmultipleviewshavebeende-rived[5,15].Alternatively,givenaprojectivecoordinatesystemspeci“edby5basispoints,thesetofconstraintsdirectlyrelatingtheprojectivecoordinatesofthecameracenterstotheimagemeasurements(in2Dprojectivecoordinates)andtheirdualconstraintsrelatingtotheprojectivecoordinatesofthe3Dscenepointshavealsobeenderived[2,20].Alternatively,multipleuncalibratedimagescanbehandledusingtheplane+parallaxŽ(P+P)approach,whichanalyzestheparallaxdisplacementsofapointbetweentwoviewsrelativetoa(realorvirtual)physicalplanarsurfaceinthescene[16,12,11].Themagnitudeoftheparallaxdisplacementiscalledtherelative-a
nestructureŽin[16].[12]showsthatthisquantitydependsbothontheHeightŽanditsdepthrelativetothereferencecamera.Sincetherelative-a
ne-structuremeasureisrelativetoboththereferenceframe)andthereferenceplane(through),werefertotheP+Pframe-workalsoasthereference-frame+reference-planeformulation.TheP+Phasthepracticaladvantagethatitavoidstheinherentambiguitiesassociatedwithestimatingtherelativeorientation(rotation+translation)betweenthecameras;thisisbecauseitrequiresonlyestimatingthehomographyinducedbythereferenceplanebetweenthetwoviews,whichfoldstogethertherotationandtranslation.Also,whenthesceneis”atŽ,thematrixestimationisunstable,whereastheplanarhomographycanbereliablyrecovered[18].Inthispaper,weremovethedependencyonthereferenceframeoftheanal-ysisofmulti-pointmulti-viewgeometry.Webreakdowntheprojectionfrom3Dto2Dinto2operations:theprojectionofthe3Dworldontothe2Dreference,followedbya2Dprojectivetransformation(homography)whichmaps thereferenceplanetotheimageplane.GiventhevirtualimagesŽformedbytheprojectionontothereferenceplane,wederivealgebraicandgeometricrela-tionsinvolvingtheimagelocationsofmultiplepointsinmultipleviewsinthesevirtualimages.Thepositionsofvirtualimagepointsaredirectlyrelatedtothe3Dlocationsofthescenepointsandthecameracentersrelativetothereferenceplanealone.Alldependencyontheinternalcameracalibrationparametersandtheorientationofthecameraarefoldedintohomographiesrelatingeachimageplanetothereferenceplane.WeobtainastructuremeasurethatdependsontheheightsofthescenepointsrelativetothereferenceplaneInthispaper,wederiveacompletesetdualrelationshipsinvolving2and3pointsin2and3views.Onthereferenceplanethemulti-pointmulti-viewgeometryissimpleandintuitive.Theserelationsaredirectlyrelatedtophys-icalpointsonthereferenceplanesuchastheepipoleandthedual-epipoleoleWeidentifythesepoints,andalsotwonewentitiescalledthetri-focallinedualtrifocal-linewhichareanalogoustotheepipoleandthedual-epipolewhenconsideringthree-viewandthree-pointgeometriesonthereferenceplane.Structuressuchasthefundamentalmatrixandthetrilineartensorhavearathersimpleformanddependonlyontheepipoles,andnothingelse.Thesymmetrybetweenpointsandcamerasiscomplete,andtheycanbesimplyswitchedaroundtogetfromtheepipolargeometrytothedual-epipolargeometry.Thesimpleandintuitiveexpressionsderivedinthereference-planebasedformulationinthispaperleadtousefulapplicationsin3Dsceneanalysis.Inparticular,simplertri-focalconstraintsarederived,andtheseleadtosimplemethodsforNewViewSynthesis.Also,theseparationandcompactpackingofcameracalibrationandorientationintothe2Dprojectiontransfor-mation(ahomography)thatrelatestheimageplanetothereferenceplane,leadstopotentiallypowerfulreconstructionandcalibrationalgorithms.Forinstance,basedonminimalpartialdomaininformation,partialcalibrationandpartialreconstructioncanbeachieved.Thisisalsobrie”ydiscussedinthispaper.Theremainderofthispaperisorganizedasfollows:Section2introducesournotations,anddescribesthetwo-viewgeometricandalgebraicconstraints(bi-linearityandparallax)inthereferenceplanerepresentation.Section3describesduality(betweenscenepointsandcameracenters)onthereferenceplane.Sec-tion4examinestherelationsinvolving3viewsandthecorrespondingdualre-lations.Section5discussesapplicationsofthisrepresentationandshowsinitialresultsforoneparticularapplication,namelynew-viewsynthesis.2TwoViewGeometryontheReferencePlaneFigure1illustratesthetwostagedecompositionoftheimageformationprocess.Figure1ashowstheprojectionofonescenepointfromtwocameracentersontothereferenceplane.Figure1bshowsthere-projectionfromtheplanetooneofthecameraimageplanes(thereferenceframeŽ).Inthisandinallsubsequent“guresinthispaper,weadoptthefollowingnotations:denotesscenepointsin3D,denotescameracenters;i,j,kareindicesusedforscenepoints(e.g., (a) ispitpts PiCsCtHiHsHtPlane
pis H CsCt Reference Reference Image Plane pitpts Fig.1.Thereferenceplanerepresentation:(a)theprojectionofthepointsontothereferenceplaneitself,removingthedependencyontherefer-enceimageplane.istheepipole,andtheredlineistheepipolarline.(b)re-projectionofthereferenceplaneimageontoareferenceimageframe(cameratŽ).)andr,s,tareindicesusedforcameracenters(e.g.,).Also,denotestheprojectionofthescenepointthroughcameracenter.Itistheintersectionoftheraywiththereferenceplane.Similarlyistheintersectionofwiththereferenceplane.Wede“neasthevirtual-imagesŽofonthereferenceplanefromcamerasrespectively.Wede“netheintersectionofastheepipole.Weusetodenotetheepipole.NotethatthelocationoftheepipoleonthereferenceplaneisisindependentoftheorientationsandtheinternalcalibrationparametersoftheToderivethealgebraicconstraintsinvolvingmultiplepointsinmultipleviews,wede“neacoordinatesystem(x,y,Z)relativetothereferenceplane,where(x,y)areparalleltoisperpendiculartoit.Forpointsonthereferenceplane,wede“ne=1,forotherpointswede“ne+1,denotestheheight(i.e.,theperpendiculardistance)ofthepointfromtheplane.Thus,),where+1,denotesthe3Dcoordi-natesofthescenepoint.Similarly,where+1,and,where+1.Thepointsonthereferenceplanearetheintersectionsofthelines,andwiththereferenceplane HiŠHtHiytŠHtyi HiŠHt1pis=
xityit1=
HixsŠHsxi HiŠHsHiysŠHsyi HiŠHs1(1)pts=
xtsyts1=
HsxtŠHtxs HsŠHtHsytŠHtys Notethattheexpressionsgivenabovedonotinvolveanyofthecamerainternalcalibrationparametersortheorientationsoftheimageplanes.Alsonotethat thereisonlyasingleepipole,whichisunlikethecaseofthereference-framebasedformulation,whichinvolvestwoepipoles,oneoneachimageframe.Thepointsonthereferenceplanearerelatedtotheircor-respondingpointsonanimageplane(e.g.,areferenceimage)viaasingle2Dprojectivetransformation,whichisthehomographybetweenthatimageplaneandtheplane.Figure1bshowsthere-projectionontothereferenceimage„thepointsp,p,andaretheprojectionsoftheimagepoints,andtheepipolerespectively.Therearetwobasicresultsconcerningtwoviewsofapointasobservedonthereferenceplane.The“rstistheexpressionfortheparallaxŽonthereferenceplane,andthesecondisthebilinearconstraintinvolvingthetwoimagelocationsofthescenepointandtheepipole.Thesearedescribedbelow.ParallaxontheReferencePlane:GiventheexpressionsinEquations1and2,itcanbeeasilyveri“edthat Notethatthisexpressionforparallax(Equation3)involvesonlytheheightsofthescenepointandofthecameracentersrelativetothereferenceplane.Itdoesnotincludeanyquantitiesrelativetoanyofthecameracoordinatesystems(e.g.,thereferenceframe)suchasasbefore.Also,theparallaxdoesnotdependonthex,ylocationsofeitherthecameracentersorthescenepointTheBilinearConstraint:Equation3impliesthat,andarecollinearSimilartothede“nitionoftheepipolarlineontheimageplane,thelinecontain-ingthesethreepointsonistheintersectionoftheepipolarplanecontaining,and.Thus,thisistheepipolarlineasobservedonthereference-plane.ThecollinearityofthesethreepointscanbeexpressedasistheFundamentalMatrixŽ.Asopposedtothereferenceframebasedformulation,wherethefundamentalmatrixdependsonthecamerarotationsandtheinternalcalibrationparametersofthecamera,hereitdependsonlyontheepipole.Moreover,theepipoleisinthehere,whereas,itisinthestandardformulation. Theexpressionfor intheP+Pcasecanberelatedtothecurrentexpressionasfollows:Consideracameracenteredat,whoseimageplaneistheplane,anditsopticalaxiscoincideswiththedirection.Then WhathappenswhentheepipolegoestoInEquation2,itcanbeseenthatwhen,theepipolegoesto.Inthiscase,andtheexpressionforparallaxcanberewrittenas:( Inotherwords,alltheparallaxvectorsareparalleltoeachother(i.e.,meetat).TheFundamentalMatrixWecan,ofcourse,unifythe“niteandthein“nitecasebyusing2Dprojectivenotations.However,inthispaperwechoosetouse2DEuclideancoordinaterepresentations,inordertoemphasizethephysicalmeaningofthevariousobservedandderivedquantities.Moreover,theparallaxexpressioninEquation3,whichinvolvesmetricrelationsismeaninglessinaprojectivecoordinaterepresentation.Also,when,thengotorespectively.Thisoccurswhen,fortuitously,theplaneischosentobeparalleltotheopticrayfromthescenepointtooneofthecameras.Inthiscase,thecorrespondingimagepointcannotbeobservedonthereferenceplane,andouranalysisdoesnotapply.3DualityontheReferencePlaneInthissection,wederiveasetofdualrelationsonthereference-planebyswitch-ingtherolesofcameracentersandscenepointsaswaspreviouslydonein[2,Considertwopointsandonecameracenter.Considerthein-tersectionoftherayswiththereferenceplaneFigure2a).Theseoccurrespectivelyat.InamanneranalogoustotheepipolarplaneŽ(de“nedby2cameracentersandascenepoint),wede-“netheplanecontaining(2scenepointsandacameracenter)astheepipolarplaneŽ.Bythesameanalogy,wede“neitsintersectionwith(i.e.,thelineconnecting)astheepipolarlineŽ,andastheepipoleŽ.Notethatthedual-epipole,thedual-epipolarlines,andthedual-epipolarplanesrelatetoapairofscenepointsovermultipleviews,inthesamewaytheepipole,theepipolarlines,andtheepipolarplanesrelatetomultiplescenepointsoverapairofviews.Byapplyingthedualityofscenepointsandcameracenters,wecanderivethedualofthebilinearconstraintandtheparallaxexpressionsinalgebraicform.Theyare:DualParallax:),where DualBilinearityConstraint:=0, (a) ijpjt Pi PjCtpitHjHtHiPlane
jtpjs Cs PjPiCtpts pitpispijPlane
Fig.2.Dualityonthereferenceplane:(a)thedual-epipolargeometryassociatedwithtwopointsinoneview.istheepipole,andthebluelinegoingthroughistheepipolarline.(b)bothsetsofepipolarlines(showninred)anddual-epipolarlines(showninblue)thatarisewhenconsideringtwopointsintwoviews.is(de“nedas)theFundamentalMatrixŽ.Thedualityofthebilinearconstrainthasbeenpreviouslyexplored-e.g.,Carlsson[2]andWeinshall,etal.[20]derivedualbilinearandtrilinearrelationsintermsoftheprojectivecoordinaterepresentationsofthescenepoints,cameracenters,andimagepoints.Here,however,wederivetheserelationsinthecontextofthereferenceplaneimages,andprovidephysicalmeaningtothedualrelations.Also,IraniandAnandan[9]pointedoutthedualepipoleinthecontextoftheplane+parallaxrepresentation.Inthatcase,sincetheprojectiononacameraimageplane(referenceframeŽ)isincludedintheformulation,thereexistsanasymmetryinthevariousconstraintsandtheirdualconstraints.Here,completesymmetryisachievedbyprojectingalltheobservedquantitiesontothereferenceplaneitself.Figure2bcompletesthepicturebyconsideringtwopoints()intwoviews().Thiscon“gurationgivesrisetoonesetofepipolarlines(corre-spondingtoeachscenepoint)goingthroughtheepipole,andonesetofepipolarlines(correspondingtoeachcamera)goingthroughtheepipole4ThreeViewGeometryontheReferencePlaneInthissectionweextendourtreatmenttothreeviews.[5]showsthattherearenoindependentconstraintsthatcanbederivedinmorethanthreeviews.Inthissectionwepresentageometricinterpretationofthethree-viewconstraintsintermsofphysicalquantitiesonthereferenceplane.Wederivethealgebraicthree-viewconstraintsandshowthattheyhaveaverysimplemathematicalform.Wewillalsoshowthatthetensor-representationoftheseconstraintsin thereference-planehasaverysimplemathematicalformwhencomparedtothetensorsinthestandardformulations[15,7,5].4.1GeometricObservationsFigure3showsthreeviews,andofapointasprojectedontothereferenceplane.Thenewcameracenterislabeledas,andthetwonewepipolesas irpts PiCsCt pitpisCrpsrprtPlane
Fig.3.Geometryofthreeviewsonthereferenceplane–a3DThe3redlinesaretheepipolarlinesofpairsofviews,andtheturquoiselinethetrifocal-line .The6pointsonlieon4linesformingacompleteTakenpairwiseatatime,thethreeviewsgiverisetothreeepipolarcon-straints:arecollinear. arecollinear. arecollinear. Thereis,however,afourthcollinearityconstraint,namely:Theepipolesarecollinear Thislineissimplytheintersectionoftheplanecontainingthethreecameracenters(seeFigure3).Thisplaneisreferredtoasthetri-focalplane.Basedonthisde“nitionwede“nethelineconnectingthethreeepipolesasthetri-focallineŽ.Thefactthatthesixpointslieonfourlinesisfundamentaltotheprojectionofthreeviewsofapointontoareferenceplane.Notethatthis“gureonthe Notethatgeometricallythis“gureisidenticaltoFigure2b,butthelabelingofthepointisdi
erent.ThescenepointinFigure2bhasbeenreplacedbyacameracenter.Infact,thisisbecauseofthecompletesymmetrybetweenscenepointsandcameracentersinourrepresentation. plane(Figure4a)isknownasthecompletequadrilateralŽandplaysacentralroleinplaneprojectivegeometry[3].Giventhethreecameras,everypointinthesceneformsatriangle(e.g.,withvertices.Di
erentpoints(e.g.,indexedi,j,etc.)willformdi
erenttriangles,allofwhichsharethesametri-focalline(seeFigure4b).Inotherwords,allthesetrianglesareperspectivefromthetri-focalline4.2TheTrifocalRatioEachpairofviewsfromthethreeviewsprovidesanexpressionforparallaxsimilartoEquation3.Forexample,consider: (HtŠHi)Hs(pisŠpts)pirŠpit=Hi(HtŠHr) Fromtheseequationswecaneliminate)toobtain: rst rst HsHr .Theaboveequationistrueuptoasignchange.Notethatrstdoesnotdependonthepoint.Inotherwords,foreveryscenepoint,thelocationsofitsimagefromthethreeviewsonthereferenceplaneisrelatedbythesameEquation5.ThisconstraintisfurtherexploredinSection4.3.GiventwoimagesŽofthepoint,e.g.,,andthecorre-spondingepipoles,,wecandeterminethelocationofthethirdimageŽbyintersectingthetwoepipolarlines(seeFigure4a).Thereare,however,twocasesinwhichthethreeepipolarlinescollapseintoasingleline(andhence,theirintersectionisnotunique).Thesearethesamesituationsnotedin[5,15],buthereweexamineitinthecontextofthereference-planeimages.The“rstisthecasewhenthethreecameracentersarecollinear(seeFigure5)-inthiscasethethreeepipolescollapseintoasinglepoint(denotedinFigure5).Thethreeepipolarlinesalsocollapseintoasingleline,andcannotbedeterminedbytheintersectionoftheepipolarlines.However,giventhecommonepipolerstcanberecoveredfromusingEquation5.Infact,inthiscase,rstisthecrossratioofthesefourpoints(thethreeimagepointsandtheepipole). Itisknowninplane-projectivegeometrythatiftwotrianglesareperspectivefromalinetheyarealsoperspectivefromapoint[3]…thisistheconverseoftheDesarguesTheorem.GiventhetwotrianglescorrespondingtoasinFigure4b,thenthepointofperspectivityisinfactthedual-epipole (a)ir pts pitpispsrprt ts pitpispirpsrprt pjspjrpjt Fig.4.Threeviewgeometryonthereferenceplane:(a)thecompletequadrilateralformedbytheimageofasinglepointinthreeviewsandthetrifocal-line(shownasadashedline)containingthethreeepipoles.(b)di
erenttrianglesduetodi
erentscenepointssharethesametrifocal-line.Anotherinterestingcaseiswhenthescenepointliesonthetri-focalplaneŽofthethreecameras.Inthiscasethethreeimagepointsalllieonthetri-focallineitself,i.e.,onceagainthethreeepipolarlinescollapseontothetri-focalline.Hencewecannotusetheintersectionofepipolarlinesto.Inthiscasetoo,canbedeterminedbyEquation5,usingrstTheratiorsthasaspecialsigni“cance.Ifweconsiderthetri-focalline,wecanshow(byreplacinginEquation3)that:rst)(6)(Hence,thenametrifocal-ratioŽ.)Inotherwords,inthegeneralcase:rst ||psrŠpts||=||pisŠpit|| ||pisŠpts||||pirŠp Notethatinthesingularcase,whentheepipolescollapse,theratioofthedis-tancesbetweentheepipoles(thetopequation)isunde“ned,butthebottomequationisstillvalidandcanbeused.4.3TheTrifocalTensorReturningtoEquation5,wecanwritedowncomponentequalitiesasfollows: xisŠxts=yisŠyit rst rst Bytakingtwooftheseequalitiesatatimeandcross-multiplyingbythedenomi-natorswecangetsixlinearconstraints.Ofthesetwoarethesameasthebilinear(epipolar)constraintsinvolvingonlytwoviewsatatime.Theotherfour,whichinvolveallthreeviewsare:rstrstrstrst)(9) pite = pts = psr = prtpirpis Fig.5.TheEpipolarLinesCollapseNotethatthesethreeviewconstraintsareactuallyonlybilinearintheimagelocationsofthescenepoint(asopposedtothetrilinearconstraintsin[15])).Thisisbecausebyconsideringtheprojectionofthepointsonthereferenceplaneitself,weeliminatethehomographiesinducedbetweentheviews(whichappearin[16]).ThetrilinearformsgiveninEquation9canbeuni“edintoasingletensorequationinamanneranalogousto[17]:rst)=0(10),2indicatetherowindicesofofŠ10xis])4.Basedonfurtheralgebraicmanipulation,Equation10canberewrittenas:,20=rstrstfollowsthestandardde“nition:=1ifand0otherwise.rstis33tensorrst=((rstIntheaboveequations,(,etc.denotethe“rst(i.e.,),thesecond(i.e.,),andthethird(i.e,1)componentsof,etc.Similarly(rstdenotestheentryindexedbya,b,cintheTensor. Notethatasin[17],istheverticallineonpassingthroughisthehorizontallineonpassingthrough.Similarlyaretheverticalandhorizontallinesonpassingthrough.Also,asin[17]therelationshipsinEqua-tion10arevalidforanylinepassingthroughandanyotherlinepassingthrough.Inotherwords,Equation10capturesthesamepoint-line-linerelationshipde-scribedin[17]and[5]. Notethattheelementsofrstdependonthetwoepipolesrst.Thisisincontrasttothegeneralformofthetrifocaltensor…forexample,thetrilineartensorin[15]alsodependsonthehomographiesduetotheplanebetweenthedi
erentcamerasandthetensordescribedin[5]whichdependsonthecameraprojectionmatrices.AsinthecaseoftheFundamentalMatrixinourformulation,theepipolesareexplicitwithintheTensor,whereasinthegeneralformulation,thetensorisimplicitlyrelatedtotheepipole.GiventheTensorrstwecanrecoverthetwoepipolesandthetrifocal-rationrst;usingEquation6wecanrecoverthethirdepipole4.4DualsoftheThreeViewConstraints3ScenePoints+1Camera:Asinthecaseoftwo-viewanalysis,thedualitybetweenscenepointsandcameracentersalsoappliestothree-viewanalysis.ByswitchingtherolesofscenepointsandcameracentersinFigure3(i.e.,)wecanderivenewconstraintsinvolvingonecameracenterandthreepoints.Theresultinggeometriccon“gurationisalsocompletequadrilateral,butwithadi
erentlabelingofthepoints.Figure6aindicatesthelabelingcorrespondingtooneviewofthreepoints.Inthiscasetrifocal-linecontainsthedual-epipoles,and.Thethree-viewconstraintgiveninEquation5isreplacedby ijk ijk HjHk ,isthetothetrifocal-ratiorst.Dualtotheotherformsofthethree-viewconstraints,(e.g.,Equation9)canalsobeobtainedbythesamesubstitutionofindices(i.e,t,ti,sj,rleadingtothe-tensorform:ijk=((ijkandthecorrespondingconstraintset:,20=ijkwherethede“nitionsofthe23matricesareanalogoustothede“nitionsgivenearlier.Notethatthethethree-viewconstraintsandtheirdual,thethree-pointconstraintsarecompletelysymmetric.Thedual-tensordependsonthedual-epipolesandthetothetrifocal-ratioOtherCombinationsof3+1Points:Thecompletesymmetrybetweenscenepointsandcameracentersimpliesthatwecanarbitrarilychoosethelabel(eitherasascenepointorasacameracenter)foreachofthefour3DpointsinFig-ure3.Sofar,wehaveconsideredtwochoices:3cameracenter+1scenepoint , (a)ki pjk pjtpktpitpij itpispjtpjs pijpts Fig.6.DualstotheThreeViewGeometry:(a)thecompletequadri-lateralformedby3points+1cameracenter .(b)thequadrilateralformedby2points+2cameras .Notethattheepipolar-lines(thinlines)intersectatanepipole,thedual-epipolarlines(thicklines)intersectatadual-epipole,andanepipolarlineintersectsadual-epipolarlineatanimage-pointand3scenepoints+1cameracenter .Thebasicstructureisthatfourpointsaredividedintoagroupof3andasinglepoint.Wecanobtainotherdualsbychoosingthefourpointstoconsistof2cameracenters+2scenepoints groupingthemas2cameracentersandascenepoint+1scenepoint oras2scenepointsandacameracenter+1cameracenter InFigure6bweshowtheresultingquadrilateralcorrespondingtothe“rstofthesegroupings.Sincethecon“gurationshowninthis“gureisbasedon2cameracentersand2scenepoints,thesixpointsonthequadrilateralconsistoffourimagepoints,oneepipole,andonedual-epipole.Notethatthetwoepipolarlinesintersectatanepipole,thetwodual-epipolarlinesintersectatadual-epipoleandeachepipolarlineintersectseachdual-epipolarlineatanimagepoint.Unlikethe3Dworld,wheretherearetwotypesofpoints,cameracentersandscenepoints,onthereference-plane,therearethree-typesofpoints…epipoles,dual-epipoles,andimagepoints.Eachoftheseformthecenterofaradial“eldoflinesthatgothroughthatpoint,allthreehavecompletelydual-roleson5ApplicationsThesimpleandintuitiveexpressionsderivedinthereference-planebasedformu-lationinthispaperleadtousefulapplicationsin3Dsceneanalysis.Inparticular,thesimpler(bilinear)tri-focalconstraintswiththeidenti“edtri-focalratioleadtoasimplemethodforNewViewSynthesis.Initialexperimentalresultsareshowninthissection.Moreover,theseparationandcompactpackingofthecameracalibrationandorientationintothe2Dprojectiontransformation(ahomography)thatrelatestheimageplanetothereferenceplane,leadstopotentiallypowerfulreconstructionandcalibrationalgorithms.Thesearebrie”ydiscussedinthissection.5.1NewViewGenerationUsingtheThree-ViewConstraintsInthissectionweshowthatthereference-planebasedformulationprovidesasimpleandintuitivewaytogeneratenewviewsfromagivensetofviews.We“rstshowsomeresults,followedbyanexplanationhowtheywereob- Figure7aand7bdisplaytwoimagestakenbyahand-heldcamera.Thescenecontainedtoyswhichwereplacedonarugonthe”oor.Thecameratranslatedandrotatedbetweenthetwoviews.The3Dparallaxe
ectsduetothecameratranslationareapparentinthechangeofthe2Ddistance(ontheimage)betweentheclownshatandtheupper-rightcorneroftherug.Figure7cisanewsynthesizedviewofthescene,asifobtainedfromacamerapositionedfarthertotheleftofthescenerelativetothetwooriginalviews(androtated,tocompensateforthetranslation).Notethesmallerdistancebetweentheclownshatandthecorneroftherug.Forcomparisonandveri“ca-tion,Figure7dshowsanviewobtainedfromthesameviewingdirectionandorientation.Also,notethedi
erencesbetweentheactualandsynthesizedview.Thereareimagedistortionswherethe”owwasinaccurate(atdepthdis-continuities,e.g.,ontherugaroundtheclownshead,andneartheearsofthesmallerdoll).Also,thesynthesizedviewismissingtheleftpartoftherug,asthisportionoftherugwasnotviewedinanyofthe2inputimages. (b) (d) (c) Fig.7.NewViewSynthesis.(a)and(b)showtwoimagestakenbyahand-heldcamera.Thecameratranslatedandrotatedbetweenthetwoviews.The3Dparallaxe
ectsduetothecameratranslationareapparentbythechangeinthe2Ddistance(ontheimage)betweentheclownshatandtheupper-rightcorneroftherug.(c)Anewsynthesizedviewofthescene,Notethesmallerdistancebetweentheclownshatandthecorneroftherug.(d)anviewobtainedfromthesameviewingdirectionandorientation.Notethedi
erencesbetweentheactualandsynthesizedview:Thereareimagedistortionswherethe”owwasinaccurate(e.g.,ontherugaroundtheclownshead,andneartheearsofthesmallerdoll).Also,thesynthesizedviewismissingtheleftpartoftherug,asthisportionoftherugwasnotviewedinanyofthe2inputimages,(aandb). Belowisabriefdescriptionofhowthesynthesizedviewwasgenerated.Toworkdirectlywithquantitiesonthereferenceplanewouldrequirepartialcal-ibrationinformationabouttheinputviews.Butasexplainedbelow,newviewsynthesisispossibleevenwithoutsuchinformation.StepI:Oneofthetwoinputimages(cameraŽ)is“rstwarpedtowardstheotherinputimage(cameraŽ;thereferenceimage)viaa2Dprojectivetrans-formationtoaligntheoftheplaneinthetwoinputimageistheplaneoftherug,inourcase).Thecorresponding2Dprojectivetransfor-mationiscomputedautomatically,withoutanypriororadditionalinformation,usinga2Dregistrationtechniquedescribedin[10].Thismethodlocksontoa2Dparametrictransformationbetweenapairofimages,eveninthepresenceofmovingobjectsorotheroutliers(suchasthetoys,inourcase).Formoredetailssee[10].Notethataftersuch2Dwarping,thetwoplane-stabilizedimagesareinfullalignmentinallimageregionswhichcorrespondtotherug,andaremisalignedinallother(i.e.,out-of-plane)imagepoints(i.e.,thetoys).Thefartherascenepointisfromtheplanarsurface(rug),thelargeritsresidualmisalignment.Werefertotheseasplanar-parallaxdisplacements(see[11,12,9]).Notethattheplane-stabilizedsequenceisinfacta2Dre-projectionofthecorrespondingvirtualimagesŽonthereferenceplaneontothereferenceim-ageplane,(SeeFigure1.b).Therefore,aquadrilateralŽonwillprojecttoaquadrilateralŽontheimageplane;di
erenttrianglesoncorrespondingtodi
erentscenepointsandsharingacommontri-focallinewillpreservethisrela-tiononthereferenceimageplane.Itcanbeshownthatforanyquadrilateral,thereexistssomerstsuchthatEquation(7)holds.Infact,itcanbeshownrst HsHr isthecomponentofthetranslationbetweencamerastheoptical()axisofthereferencecamera.Similarlyforthethirdcameraareasbefore(i.e.,heightsrelativetoStepII:Dense”owisestimatedbetweenthetwoplane-stabilizedimages(us-ingthemethoddescribedin[12]).Notethatafterplanestabilization,the”ow“eldbetweenthetwoimagesreducestoaradialepipolar“eldcenteredattheepipole(seeEquation(3);seealso[11,12,9]).Thecancellationoftheplaneho-mographyremovesalle
ectsofcamerarotationandchangesincalibration.Thisallowstocomputethe”ow“eldbetweenaplane-stabilizedimagepairmorereli-ablythangeneral”ow,asitisconstrainedtosatisfyaglobalepipolarconstraint.StepIII:Weestimatetheepipole()fromtheradial”ow“eldbetweenthetwoinputplane-stabilizedimages.Wethenspecify:(i)theepipole(e.g.,)betweenthereferenceimageandtheplane-stabilizedŽimage,(ii)atri-focalratiorstinthe referenceframe.Giventhevirtualtri-focalratiorst,thevirtualepipoletheactualepipole,andthedense”ow“eldbetweenthetwoplane-stabilizedimages(betweensandcorrespondings),wecanestimateallimagepointsinthevirtual(plane-stabilized)image(namely,alls)usingEquation8.Thevirtualtri-focalratiorstandthevirtualepipolecaneitherbespec-i“eddirectly(e.g.,viaEquation(14)),orelsebyspecifyingthelocationoftwoormoreimagepointsinthevirtual(plane-stabilized)view,andestimatethemaccordingly.StepIV:Notethatthesynthesizedplane-stabilizedimageisthesameforcameracenteredat.Inotherwords,itisindependentoftheinternalparame-tersandtheorientationofthatcamera.Byspecifyingahomographythatrelatestheimageplaneofthevirtualcameratothestabilizedimagefromthereferenceview,wehavethecomplete”exibilitytogenerateanimageobtainedbyerasituatedat.Thisisdonebythesynthesizedplane-stabilizedimageviathecorresponding2Dprojectivetransformation.5.23DReconstructionandCameraCalibrationGivenuncalibratedimages,anyapproachforobtainingEuclidean(orAne)reconstructionrequiressometypeofcalibration.Oneofthebene“tsofourap-proachisthatthisprocessisfactoredintotwoseparatestages,eachofwhichhasasimpleandintuitivesolution.First,giventheinputimages,thevirtual-imagesŽonmustbedetermined.ThiscanbedonebytakingadvantageoftheP+Pmethod[12]…(i)determinetheplanarhomographyforbetweenarbitrarilychosenreferenceimageandeachotherimage,and(ii)determinethehomographybetweenthereferenceimageand.NotethattheparallaxEqua-tion3isvalideveniftheimagelocationsonareknownonlyuptoa2Dtransformation.Thismeansthatjustbyindicatingtwosetsofparallellines(thatareindi
erentorientations),the3DHeightsrelativetothereferenceplanecanberecovered.Fromthisinformation,theparallaxmagnitude(inEqua-tion3)canbedetermined.(Notethatbyspecifyingthe2Dcoordinatesoffourpointsonthereferenceplane,thehomographycanfullydetermined,leadingtoEuclideanreconstruction.)Givenandtheheightofone3DscenepointrelativetotheHeightsofallotherpointscanbedetermineduptoaglobalscalefactor.Boththesecalibrationstepsaresimpleandintuitiveandrequireminimalspeci“cationofinformation.Theresultingreconstructioniswithrespecttothereferenceplaneanddoesnotinvolvethecamerareferenceframes.Theforegoingoutlineforareconstructionmethodassumesthatthecor-respondencesofeachpointacrossthemultiple-viewscanbeestimated.Thisinvolvescomputingtheparallax”ow-“eld(s),andtheepipole(s)…thesecanbedoneinthesamemannerasdescribedin[11,12].Itisworthnoting,however,theremovaloftheplanarhomographyallowstheparallaxcomputationtobemorerobustandaccurate[11,12]. 1.S.AvidanandA.Shashua.Novelviewsynthesisintensorspace.InIEEECon-ferenceonComputerVisionandPatternRecognition,pages1034…1040,San-Juan,June1997.2.S.Carlsson.Dualityofreconstructionandpositioningfromprojectiveviews.InWorkshoponRepresentationsofVisualScenes,1995.3.H.S.MCoxeter,editor.ProjectiveGeometry.SpringerVerlag,1987.4.O.D.Faugeras.Whatcanbeseeninthreedimensionswithanuncalibratedstereorig?InEuropeanConferenceonComputerVision,pages563…578,SantaMargaritaLigure,May1992.5.O.D.FaugerasandB.Mourrain.Onthegeometryandalgebraofthepointandlinecorrespondencesbetweennimages.InInternationalConferenceonComputer,pages951…956,Cambridge,MA,June1995.6.OlivierFaugeras.Three-DimensionalComputerVision…AGeometricViewpointMITPress,Cambridge,MA,1996.7.RichardHartley.Linesandpoinsinthreeviews…auni“edapproach.InDARPAImageUnderstandingWorkshopP,1994.8.RichardHartley.EuclideanReconstructionfromUncalibratedViews.InApplica-tionsofInvarianceinComputerVision,J.L.Mundy,D.Forsyth,andA.Zisserman(Eds.),Springer-Verlag,1993.9.M.IraniandP.Anandan.Parallaxgeometryofpairsofpointsfor3dsceneanalysis.EuropeanConferenceonComputerVision,Cambridge,UK,April1996.10.M.Irani,B.Rousso,andS.Peleg.Computingoccludingandtransparentmotions.InternationalJournalofComputerVision,12(1):5…16,January1994.11.M.Irani,B.Rousso,andP.peleg.Recoveryofego-motionusingregionalignment.IEEETrans.onPatternAnalysisandMachineIntelligence,19(3):268…272,March12.R.Kumar,P.Anandan,andK.Hanna.Directrecoveryofshapefrommultipleviews:aparallaxbasedapproach.InProc12thICPR,1994.13.H.C.Longuet-Higgins.Acomputeralgorithmforreconstructingascenefromtwoprojections.Nature,293:133…135,1981.14.R.Mohr.AccurateProjectiveReconstructionInApplicationsofInvarianceinComputerVision,J.L.Mundy,D.Forsyth,andA.Zisserman,(Eds.),Springer-Verlag,1993.15.A.Shashua.Algebraicfunctionsforrecognition.IEEETransactionsonPatternAnalysisandMachineIntelligence,17:779…789,1995.16.A.ShashuaandN.Navab.Relativeanestructure:Theoryandapplicationto3dreconstructionfromperspectiveviews.InIEEEConferenceonComputerVisionandPatternRecognition,pages483…489,Seattle,Wa.,June1994.17.A.ShashuaandP.AnanadanTrilinearConstraintsrevisited:generalizedtrilinearconstraintsandthetensorbrightnessconstraint.IUW,Feb.1996.18.P.H.S.Torr.MotionSegmentationandOutlierDetection.PhDThesis:ReportNo.OUEL1987/93,Univ.ofOxford,UK,1993.19.M.SpetsakisandJ.Aloimonos.Auni“edtheoryofstructurefrommotion.DARPAImageUnderstandingWorkshop,pp.271-283,Pittsburgh,PA,1990.20.D.Weinshall,M.Werman,andA.Shashua.Shapedescriptors:Bilinear,trilinearandquadlinearrelationsformulti-pointgeometry,andlinearprojectivereconstruc-tionalgorithms.InWorkshoponRepresentationsofVisualScenes,1995.