Un rolling the shutter - PowerPoint Presentation

1. Unrolling the shutter: CNN to correct motion distortionsVijay Rengarajan, Yogesh Balaji, A.N. RajagopalanIndian Institute of Technology MadrasImage Processing and Computer Vision lab, Department of Electrical Engineering, IIT Madras

2. Camera Motion Causes Rolling Shutter DistortionsMobile phonesDrone camerasStreetview captureMotion blurLens distortions

3. Sequential Exposure of Rolling Shutter0timeGlobal shutter CCD image sensorExposure closeExposure openExposure timete0Top row Bottom row timeTd Total line delayteTop row Bottom row All pixels expose at the same timeRolling shutter CMOS image sensorEach row starts exposing sequentially

4. Rolling Shutter Distortions are GeometricDifferent rows see the scene at different poses of the moving cameraEven short exposure causes distortionsxzyxzytx translationrz rotationSceneCaptured imageSceneCaptured imagetimetime

5. Correct Rolling Shutter Distortions from a Single ImageDisturbs visual appealAffects scene inferenceSingle image ambiguity curved building or rolling shutter effect?

6. Prior Works on Rolling Shutter CorrectionRengarajan et al. CVPR (2016) for urban scenesHeflin et al. Conf. Biometrics (2010) for facesRolling shutterCurvaturesCorrectedRolling shutterCorrected using facial featuresRingaby and ForssenCVPR (2010)IJCV (2012)Grundmann et al.ICCP (2012)Video rolling shutter correctionUse frame-to-frame point correspondencesNeedA single method that can be used for different classes of imagesDifferent levels of features to correct extract motion and to discard feature outliers

7. Let machines extract desired featuresConvolutional Neural NetworkInput Rolling shutter image 256x256x3Output Translation and rotation (tx,rz) 15 tx and 15 rz motion samples of equally spaced rowsTrain for different classes of imagesMotion FittingDistortion CorrectionCorrected ImageRolling shutter distorted imageCNNFeature ExtractionMotion EstimationDistortion CorrectionCorrected ImageRolling shutter distorted imageExisting approachMotion fittingPolynomial trajectory to get tx and rz for each rowDistortion correctionInverse warping based on row-wise motion123

8. VanillaCNN with square filtersVanilla Convolutional Neural NetworkMotion Mean Squared Errortx and rz at 15 rowsVanillaCNNTranslations only Translations and rotationsCorrected byVanillaCNNDistorted imageDistorted imageCorrected byVanillaCNN12

9. Ideas for a new architectureInitial feature extractionFeature combinationAny better ideas?Along rows : motion constancyAlong columns : temporal motion

10. Use long filters for RowColCNN Motion Mean Squared ErrorCaptures information in rows earlyCaptures information along time dimension earlyFilter h x w x c

11. Use long filters for RowColCNN Motion Mean Squared ErrorCaptures information along row dimension earlyCaptures information along time dimension earlyTraining Data GenerationGenerate random polynomial camera trajectoryApply on undistorted imageCorrectionGet camera motion values from CNNFit a polynomial trajectory and get motion at all rowsCorrect distorted image using target-to-source mappingDatasetsChessboard Urban scenes Faces 7k 300k 250k Sun Oxford Zurich LFW Corrected byVanillaCNNCorrected byRowColCNN

12. Correction Results of RowColCNN

13. Learning excels in challenging conditionsGeometry-basedLearning-basedDistorted inputRengarajan et al. (2016) fail due to tree branches which are naturally curvedHeflin et al. 2010RowColCNNRowColCNNRengarajan et al. 2016Heflin et al. (2010) fail due to wrong estimation of facial features in varied illumination conditions

14. New CNN filter shapes inspired by applicationNew learning-based method for single image rolling shutter correctionCNN learns image to motion mapping Long filters in CNN architecture for rolling shutter exposureapvijay.github.io/rs_rect_cnnPoster 21 AM