ECE6504 02212013 Roadmap Roadmap Introduction Paper1 Motivation Problem statement Approach Experiments amp Results Paper 2 Experiments Comments Questions What is context Any data or metadata not directly produced by the presence of an object ID: 717475
Download Presentation The PPT/PDF document "Context Neelima Chavali" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Context
Neelima
Chavali
ECE6504 02/21/2013Slide2
Roadmap
Roadmap
Introduction
Paper1
Motivation
Problem statement
Approach
Experiments & Results
Paper 2
Experiments
Comments
Questions?Slide3
What is context?
Any data or meta-data not directly produced by the presence of an object
Nearby image data
Context
Derek
HoeimSlide4
What is context?
Any data or meta-data not directly produced by the presence of an object
Nearby image data
Scene information
Context
Context
Derek HoeimSlide5
What is context?
Any data or meta-data not directly produced by the presence of an object
Nearby image data
Scene information
Presence, locations of other objects
Tree
Derek HoeimSlide6
How do we use context?
What is this?
Now can you tell?Slide7
How is context used in
Compuer
VisionSlide8
An Empirical study of Context in Object detection:
Santosh
Duvvala
,
Derek
Hoiem
, James Hays, Alexei Efros, Martial HebertPaper 1Slide9
Motivation
Lack of standardization
Little agreement about what constitutes “context”
Isolate contribution of contextual informationSlide10
Problem statement
Evaluate context
on the challenging PASCAL VOC 2008 dataset using state-of-the-art object detectors
Analyze
–different
sources of context
–different
uses of contextNovel algorithms using geographic context, and local pixel contextSlide11
Sources and Uses of Context
Sources:
local pixel context, 2D scene gist, 3D geometric, semantic
,
geographic, photogrammetric, cultural
Uses:
Object
presence, location, size, spatial supportSantosh DuvvalaSlide12
Approach
Local Appearance detector: Deformable parts model
Slide13
Object presence
gist
:
Torralba
Oliva
2003
geom
context:
Hoiem
et al
. 2005
im2gps
: Hays and
Efros
2008
Santosh DuvvalaSlide14
Semantic and Geographic context
Santosh DuvvalaSlide15
Object location
Santosh DuvvalaSlide16
Object size
Santosh DuvvalaSlide17
Combining Contexts
Santosh DuvvalaSlide18
Object Spatial SupportSlide19
Experimental Results and Analysis
Detection results on PASCAL VOC 2008
testsetSlide20
Results
Change in accuracy with respect to Size and occlusionSlide21
Confusion matricesSlide22
Object Graphs for Context-aware category discovery:
Yong
Jae Lee and Kristen
Grauman
Paper 2Slide23
Motivation
Unlabeled Image Data
Discovered categories
1) reveal structure in very large image collections
2) greatly reduce annotation time and effort
3) Mitigate the biases.
23
Yong Jae Lee and Kristen GraumanSlide24
Can seeing previously learned objects in novel images help to discover
new
categories?
1
3
4
2
Main idea
24
Can you identify the recurring pattern?
Yong Jae Lee and Kristen GraumanSlide25
Problem Statement
Discover novel categories that occur amidst known objects within un-annotated images
25
Yong Jae Lee and Kristen GraumanSlide26
drive-way
sky
house
?
grass
Context-aware visual discovery
grass
sky
truck
house
?
drive-way
grass
sky
house
drive-way
fence
?
?
?
?
26
Context in supervised recognition:
[
Torralba
2003], [
Hoiem
et al. 2006], [He et al. 2004], [
Shotton
et al. 2006], [
Heitz
&
Koller
2008], [
Rabinovich
et al. 2007], [
Galleguillos
et al. 2008], [
Tu
2008], [Parikh et al. 2008], [Gould et al. 2009], [
Malisiewicz
&
Efros
2009], [
Lazebnik
2009]
Yong Jae Lee and Kristen GraumanSlide27
Approach Overview
27
Learn category models for some classes
Detect unknowns in unlabeled images
Describe object-level context via
Object-Graph
Group regions to discover new categories
Yong Jae Lee and Kristen GraumanSlide28
Learn “Known” Categories
Offline: Train region-based classifiers for
N
“known” categories using labeled training data.
sky
road
building
tree
28
Detect Unknowns
Object-level Context
Discovery
Learn Models
Yong Jae Lee and Kristen GraumanSlide29
Identifying Unknown Objects
Input: unlabeled pool of novel images
Compute multiple-segmentations for each unlabeled image
29
Detect Unknowns
Object-level Context
Discovery
Learn Models
e.g., [
Hoiem
et al. 2006], [Russell et al. 2006], [
Rabinovich
et al. 2007]
Yong Jae Lee and Kristen GraumanSlide30
P(class | region)
b
ldg
t
ree
s
ky
r
oad
P(class | region)
b
ldg
t
ree
s
ky
r
oad
P(class | region)
b
ldg
t
ree
s
ky
r
oad
P(class | region)
b
ldg
t
ree
s
ky
r
oad
Prediction: known
Prediction: known
Prediction: known
High entropy
→
Prediction:unknown
For all segments, use classifiers to compute posteriors for the
N
“known” categories.
Deem each segment as “known” or “unknown” based on resulting entropy.
30
Identifying Unknown Objects
Detect Unknowns
Object-level Context
Discovery
Learn Models
Yong Jae Lee and Kristen GraumanSlide31
31
Previous Work:
[
Scholkopf
2000], [
Markou
& Singh 2003], [
Weinshall
et al. 2008]
Image
GT known/unknown
Multiple-Segmentation Entropy Maps
unknowns
building
tree
knowns
sky
road
Identifying Unknown Objects
Detect Unknowns
Object-level Context
Discovery
Learn Models
Yong Jae Lee and Kristen GraumanSlide32
An unknown region within an image
0
Closest nodes in its object-graph
2a
2b
1b
1
a
3a
3b
Consider spatially near regions
above
and
below
,
record distributions for each known class.
S
b t s r
1a
above
1b
below
H
1
(s)
b t s r
b t s r
H
0
(s)
0
self
g(s)
= [ ,
, ,
]
H
R
(s)
b t s r
b t s r
Ra
above
Rb
below
1
st
nearest region
o
ut
to
R
th
nearest
b t s r
0
self
Object-Graphs
Detect Unknowns
Object-level Context
Discovery
Learn Models
32
Yong Jae Lee and Kristen GraumanSlide33
Object-Graphs
Average across segmentations
N
posterior prob.’s per pixel
b t s r
b t s r
N
posterior prob.’s per
superpixel
b t s r
b t s r
Obtain per-pixel measures of class posteriors on larger spatial extents.
33
Detect Unknowns
Object-level Context
Discovery
Learn Models
Yong Jae Lee and Kristen GraumanSlide34
g(s
1
)
= [
: , , : ]
b t g r
above
below
H
R
(s)
H
1
(s)
above
below
b t g r
b t g r
b t g r
g(s
2
)
= [
:
, , : ]
b t g r
above
below
H
R
(s)
H
1
(s)
above
below
b t g r
b t g r
b t g r
Object-graphs are very similar
produces a strong match
Known classes
b: building
t: tree
g: grass
r: road
34
Object-Graph matching
Detect Unknowns
Object-level Context
Discovery
Learn Models
building
?
road
building / road
building
/ road
tree
/
road
building
?
road
building
/ road
Yong Jae Lee and Kristen GraumanSlide35
Unknown Regions
Clusters from region-region affinities
35
Detect Unknowns
Object-level Context
Discovery
Learn Models
Yong Jae Lee and Kristen GraumanSlide36
Object Discovery Accuracy
Four datasets
Multiple splits for each dataset; varying categories and number of
knowns
/unknowns
Train 40% (for known categories), Test 60% of data
Textons
, Color histograms, and pHOG
Features
MSRC-v2
PASCAL 2008
Corel
MSRC-v0
36
Yong Jae Lee and Kristen GraumanSlide37
37
MSRC-v2
PASCAL 2008
Corel
MSRC-v0
Object Discovery Accuracy
Yong Jae Lee and Kristen GraumanSlide38
Comparison with State-of-the-art
Russell et al., 2006: Topic model (LDA) to discover categories among multiple segmentations using appearance only.
Significant improvement over existing state-of-the-art.
38
MSRC-v2
Yong Jae Lee and Kristen GraumanSlide39
MAP comparision
Yong Jae Lee and Kristen Grauman
39Slide40
Example Object-Graphs
building
sky
road
unknown
40
Color in
superpixel
nodes indicate the predicted known category.
Yong Jae Lee and Kristen GraumanSlide41
Conclusions
Discover new categories in the context of those that have already been directly taught.
Substantial improvement over traditional unsupervised appearance-based methods.
Future work
: Continuously expand the object-level context for future discoveries.
41
Yong Jae Lee and Kristen GraumanSlide42
Experiments
42Slide43
10 Unknowns: Randomly selected groups
43Slide44
10 Unknowns: Manmade objects
44Slide45
15 Unknowns: Known Animals
45Slide46
5 Unknowns : animals
46Slide47
Acknowledgements
Dr. Yong Lee
Dr. Devi Parikh
47