Context Neelima Chavali - PowerPoint Presentation

Slide1

Context

Neelima

Chavali

ECE6504 02/21/2013Slide2

Roadmap

Roadmap

Introduction

Paper1

Motivation

Problem statement

Approach

Experiments & Results

Paper 2

Experiments

Comments

Questions?Slide3

What is context?

Any data or meta-data not directly produced by the presence of an object

Nearby image data

Context

Derek

HoeimSlide4

What is context?

Any data or meta-data not directly produced by the presence of an object

Nearby image data

Scene information

Context

Context

Derek HoeimSlide5

What is context?

Any data or meta-data not directly produced by the presence of an object

Nearby image data

Scene information

Presence, locations of other objects

Tree

Derek HoeimSlide6

How do we use context?

What is this?

Now can you tell?Slide7

How is context used in

Compuer

VisionSlide8

An Empirical study of Context in Object detection:

Santosh

Duvvala

,

Derek

Hoiem

, James Hays, Alexei Efros, Martial HebertPaper 1Slide9

Motivation

Lack of standardization

Little agreement about what constitutes “context”

Isolate contribution of contextual informationSlide10

Problem statement

Evaluate context

on the challenging PASCAL VOC 2008 dataset using state-of-the-art object detectors

Analyze

–different

sources of context

–different

uses of contextNovel algorithms using geographic context, and local pixel contextSlide11

Sources and Uses of Context

Sources:

local pixel context, 2D scene gist, 3D geometric, semantic

,

geographic, photogrammetric, cultural

Uses:

Object

presence, location, size, spatial supportSantosh DuvvalaSlide12

Approach

Local Appearance detector: Deformable parts model

Slide13

Object presence

gist

:

Torralba

Oliva

2003

geom

context:

Hoiem

et al

. 2005

im2gps

: Hays and

Efros

2008

Santosh DuvvalaSlide14

Semantic and Geographic context

Santosh DuvvalaSlide15

Object location

Santosh DuvvalaSlide16

Object size

Santosh DuvvalaSlide17

Combining Contexts

Santosh DuvvalaSlide18

Object Spatial SupportSlide19

Experimental Results and Analysis

Detection results on PASCAL VOC 2008

testsetSlide20

Results

Change in accuracy with respect to Size and occlusionSlide21

Confusion matricesSlide22

Object Graphs for Context-aware category discovery:

Yong

Jae Lee and Kristen

Grauman

Paper 2Slide23

Motivation

Unlabeled Image Data

Discovered categories

1) reveal structure in very large image collections

2) greatly reduce annotation time and effort

3) Mitigate the biases.

23

Yong Jae Lee and Kristen GraumanSlide24

Can seeing previously learned objects in novel images help to discover

new

categories?

1

3

4

2

Main idea

24

Can you identify the recurring pattern?

Yong Jae Lee and Kristen GraumanSlide25

Problem Statement

Discover novel categories that occur amidst known objects within un-annotated images

25

Yong Jae Lee and Kristen GraumanSlide26

drive-way

sky

house

?

grass

Context-aware visual discovery

grass

sky

truck

house

?

drive-way

grass

sky

house

drive-way

fence

?

?

?

?

26

Context in supervised recognition:

[

Torralba

2003], [

Hoiem

et al. 2006], [He et al. 2004], [

Shotton

et al. 2006], [

Heitz

&

Koller

2008], [

Rabinovich

et al. 2007], [

Galleguillos

et al. 2008], [

Tu

2008], [Parikh et al. 2008], [Gould et al. 2009], [

Malisiewicz

&

Efros

2009], [

Lazebnik

2009]

Yong Jae Lee and Kristen GraumanSlide27

Approach Overview

27

Learn category models for some classes

Detect unknowns in unlabeled images

Describe object-level context via

Object-Graph

Group regions to discover new categories

Yong Jae Lee and Kristen GraumanSlide28

Learn “Known” Categories

Offline: Train region-based classifiers for

N

“known” categories using labeled training data.

sky

road

building

tree

28

Detect Unknowns

Object-level Context

Discovery

Learn Models

Yong Jae Lee and Kristen GraumanSlide29

Identifying Unknown Objects

Input: unlabeled pool of novel images

Compute multiple-segmentations for each unlabeled image

29

Detect Unknowns

Object-level Context

Discovery

Learn Models

e.g., [

Hoiem

et al. 2006], [Russell et al. 2006], [

Rabinovich

et al. 2007]

Yong Jae Lee and Kristen GraumanSlide30

P(class | region)

b

ldg

t

ree

s

ky

r

oad

P(class | region)

b

ldg

t

ree

s

ky

r

oad

P(class | region)

b

ldg

t

ree

s

ky

r

oad

P(class | region)

b

ldg

t

ree

s

ky

r

oad

Prediction: known

Prediction: known

Prediction: known

High entropy

→

Prediction:unknown

For all segments, use classifiers to compute posteriors for the

N

“known” categories.

Deem each segment as “known” or “unknown” based on resulting entropy.

30

Identifying Unknown Objects

Detect Unknowns

Object-level Context

Discovery

Learn Models

Yong Jae Lee and Kristen GraumanSlide31

31

Previous Work:

[

Scholkopf

2000], [

Markou

& Singh 2003], [

Weinshall

et al. 2008]

Image

GT known/unknown

Multiple-Segmentation Entropy Maps

unknowns

building

tree

knowns

sky

road

Identifying Unknown Objects

Detect Unknowns

Object-level Context

Discovery

Learn Models

Yong Jae Lee and Kristen GraumanSlide32

An unknown region within an image

0

Closest nodes in its object-graph

2a

2b

1b

1

a

3a

3b

Consider spatially near regions

above

and

below

,

record distributions for each known class.

S

b t s r

1a

above

1b

below

H

1

(s)

b t s r

b t s r

H

0

(s)

0

self

g(s)

= [ ,

, ,

]

H

R

(s)

b t s r

b t s r

Ra

above

Rb

below

1

st

nearest region

o

ut

to

R

th

nearest

b t s r

0

self

Object-Graphs

Detect Unknowns

Object-level Context

Discovery

Learn Models

32

Yong Jae Lee and Kristen GraumanSlide33

Object-Graphs

Average across segmentations

N

posterior prob.’s per pixel

b t s r

b t s r

N

posterior prob.’s per

superpixel

b t s r

b t s r

Obtain per-pixel measures of class posteriors on larger spatial extents.

33

Detect Unknowns

Object-level Context

Discovery

Learn Models

Yong Jae Lee and Kristen GraumanSlide34

g(s

1

)

= [

: , , : ]

b t g r

above

below

H

R

(s)

H

1

(s)

above

below

b t g r

b t g r

b t g r

g(s

2

)

= [

:

, , : ]

b t g r

above

below

H

R

(s)

H

1

(s)

above

below

b t g r

b t g r

b t g r

Object-graphs are very similar



produces a strong match

Known classes

b: building

t: tree

g: grass

r: road

34

Object-Graph matching

Detect Unknowns

Object-level Context

Discovery

Learn Models

building

?

road

building / road

building

/ road

tree

/

road

building

?

road

building

/ road

Yong Jae Lee and Kristen GraumanSlide35

Unknown Regions

Clusters from region-region affinities

35

Detect Unknowns

Object-level Context

Discovery

Learn Models

Yong Jae Lee and Kristen GraumanSlide36

Object Discovery Accuracy

Four datasets

Multiple splits for each dataset; varying categories and number of

knowns

/unknowns

Train 40% (for known categories), Test 60% of data

Textons

, Color histograms, and pHOG

Features

MSRC-v2

PASCAL 2008

Corel

MSRC-v0

36

Yong Jae Lee and Kristen GraumanSlide37

37

MSRC-v2

PASCAL 2008

Corel

MSRC-v0

Object Discovery Accuracy

Yong Jae Lee and Kristen GraumanSlide38

Comparison with State-of-the-art

Russell et al., 2006: Topic model (LDA) to discover categories among multiple segmentations using appearance only.

Significant improvement over existing state-of-the-art.

38

MSRC-v2

Yong Jae Lee and Kristen GraumanSlide39

MAP comparision

Yong Jae Lee and Kristen Grauman

39Slide40

Example Object-Graphs

building

sky

road

unknown

40

Color in

superpixel

nodes indicate the predicted known category.

Yong Jae Lee and Kristen GraumanSlide41

Conclusions

Discover new categories in the context of those that have already been directly taught.

Substantial improvement over traditional unsupervised appearance-based methods.

Future work

: Continuously expand the object-level context for future discoveries.

41

Yong Jae Lee and Kristen GraumanSlide42

Experiments

42Slide43

10 Unknowns: Randomly selected groups

43Slide44

10 Unknowns: Manmade objects

44Slide45

15 Unknowns: Known Animals

45Slide46

5 Unknowns : animals

46Slide47

Acknowledgements

Dr. Yong Lee

Dr. Devi Parikh

47