Vision-based 3d bicycle tracking using deformable part model and interacting multiple - PowerPoint Presentation

Slide1

Vision-based 3d bicycle tracking using deformable part model and interacting multiple model filter

Presentation by Jonathan Kaan DeBoyPaper by Hyunggi Cho, Paul E. Rybski and Wende Zhang

1Slide2

Motivation

Build understanding of surroundingDetect vulnerable road users (VRU)

B

icyclist

Motor cyclistsPedestriansChange appearances very drastically and a very short timeCurrentlyLIDAR and RADARThis paper focuses vision implementation

2Slide3

Vision Based Detection

Higher resolution view of the worldColor, texture, shape, contoursLow costProcessor intensive—complicated backgrounds to extract from

Error prone to lightening changes, object shape,

etc.

3Slide4

Other Forms of Detection

Planar LIDARLow resolution RADAROnly RCS/LCS is available

High cost

Easy on

processingNot as susceptible to noise4Slide5

3 Major Contributions

5Slide6

1st – Model

Three-component bicycle modelBicycle Kinematics have restrictive constraints on movement

Unlike pedestrians

Two motion models

IMM estimatorExtended Kalman FilterPosition and orientation in vehicle coordinates6Slide7

2nd – Tracking

Extension of single bicycle trackingRao-Blackwellized Particle FilterParticle filter for data association

IMM for each bicycle tracking

7Slide8

3rd – Bicycle Dataset

First public domain bicycle datasetAvailable for anyone to conduct bicycle tracking research

8Slide9

Pedestrian Detection

9Slide10

Single Template

Originally showed better performanceCapture whole human body with detection windowHaar wavelet with polynomial SVMDense Histogram of Oriented Gradient (HOG) then linear SVM

10Slide11

Part-Based

Began to look more promisingFlexible and rich modelsCaptures the pattern of each part—handles various appearancesDivide body into 4 parts: head, legs, left arm, right arm

Polynomial SVM fed into a classifier

Scale Invariant Feature Transform (SIFT)

11Slide12

Pedestrian Tracking

Statistical Probabilistic MethodsExtended Kalman FilterParticle FilterAlpha Beta Filter

Constant velocity

12Slide13

Bicycle Detection

Object detection and tracking by detectionRun every frameDetector Virtual Sensor2D bounding boxes

Different classes of bicycles

Road, mountain, etc.

13Slide14

Deformable Parts Model

14Slide15

Deformable Part Representation

Star structured part-based model with:

Root filter

Capturing overall shape of an object (2

nd row)Part filtersCapture the appearance of each part of an object (3rd row)Deformation parametersDeviation from ideal location (4

th row)Score = Root filter score + part filter score (from best possible placement) – deformation cost

15Slide16

Efficient Matching Process

Dynamic programingGeneralized distance transformsHuge optimization model for matchingImportant to use fast method for a detection task

16Slide17

Latent SVM Training Process

Train a mixture of star models from bounding box ground truthOptimization task with two sets of

variables

Beta is vector of model parametersZ’s are latent valuesPhi(x,z) is feature vectorStar model examplebeta is root (+) parts (+) deformation costsZ is specification of object configurationPhi(x,z) is concatenation of sub windows

17Slide18

Bicycle Detector as a Virtual Sensor

Monocular video camera – sequence of imagesGenerates set of bounding boxes for potential bicyclesEssentially a sequence of measurements at time step

k

Measurement

is fed to Kalman filterbi is: y coordinates of top (t) and bottom (d) border of box, x coordinates of left (l) and right (r) borders, and an index of its view (v)18Slide19

Number of Viewpoints

Our camera is moving, detector must accountTradeoff between increasing number of models and reducing time complexity.8

view-based

bicycle detector was used

Paired and trained by symmetric counterpart19Slide20

Multiple Bicycle Tracking with IMM Algorithm and a Rao-Blackwellized Particle Filter

Bicycle detector is very processor demandingNeed reliable tracking algorithm that is certain of its uncertainty for tracking



Extended

Kalman Filter20Slide21

Alpha-Beta Filter

Tracks based off incoming data and previous velocityVelocity is updated based on a weighted sumPrevious prediction (acquired from previous data points)

Current data point

Alpha and Beta are typically set weights

21Slide22

Extended Kalman Filter

Real motion modeled by simple motion modelsLinearized nonlinear perspective projection (not extended)

Assume flat ground

Tracking:

PredictUpdate22Slide23

Bicycle Motion Model Set

Bicycles have unique kinematicsDifficult to measure when the object is in a rough bounding boxInstead comprehensive experiment results lead to moving mass with constant velocity model

To improve performance,

Add simplified versions of bicycle’s kinematics

Use a well-known IMM filter23Slide24

Interactive Multiple Model (IMM) Filter

Multiple motion models representing dynamic behaviorsManeuveringSeveral motion models ran in parallelEstimates a state through weighted sum of several filter results

24Slide25

Model Set

Constant Velocity (CV)Simplified Bicycle (SB)

Point

model and 3D bounding

box25Slide26

Bicycle Measurement Model

Need a way to map image space to vehicle coordinates (state space)One representative point (middle of bottom line of 2D bounding box)

26Slide27

Extension to Multiple Bicycle Tracking

27Slide28

Rao-Blackwellized Particle Filter (RBPF)

Multiple measurements – no information on number of bicycles that existBreak down huge state estimation into smaller problemsAnalytical solutions and

particle

filter solution

Rao-Blackwellized Monte Carlo Data Association RBMCDA algorithmBayesian factorizationSeparate posterior into number of bicycles and a tracking problem28Slide29

Experimental Results

29Slide30

The Experiment

BossDARPA Urban Challenge winner of 20078 view-based bicycle modelAnalyze the statistics of bicycle detection responses

Deformable Parts

Model

30

https://www.cmu.edu/news/image-archive/Boss.jpgSlide31

Detection Performance

Building bicycle model357 positive training samples3300 negative samplesThree component model

8 viewpoints

Frontal, rear, four diagonal, left, right

Precision Recall (PR) CurveDocuments search engine31Slide32

Tracking Performance

6 Video sequences3 Stationary vehicle, 3 movingRMS error compare between CV and CV+SB in IMM filter

32Slide33

33Slide34

Conclusions and Future Work

Deformable Parts Based Model to detect bicycles3 Part Bicycle ModelTwo motion models: CV and SB

Sent to EKF to estimate position and velocity in vehicle coordinate system

IMM tracking algorithm

Extended with Rao-Blackwellized Particle Filter for multiple bicycles tracksFutureNew measurement mapping function to extract more info from 2D bounding box34