S Bhattacharya R Sukthankar MShah University of Central Florida Intel labs amp CMU Motivation Related work Quality enhancement framework Visual aesthetics Aesthetic appeal assessment ID: 397575
Download Presentation The PPT/PDF document "Photo-Quality Enhancement based on Visua..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Photo-Quality Enhancement based on Visual Aesthetics
S. Bhattacharya*, R.
Sukthankar
**,
M.Shah
*
*University of Central Florida, **Intel labs & CMUSlide2
MotivationSlide3
Related work
Quality enhancement framework
Visual aesthetics
Aesthetic appeal assessment
Enhancement through recomposition Experimental results Conclusions Future directions
OutlineSlide4
Low-level (dehazing
etc.)
Related Work
Domain-specific (face beautification etc.)
T.
Leyvand,et
al.
, “Data-Driven Enhancement of Facial Attractiveness”, ACM SIGGRAPH 08
K. He,
J.Sun
X. Tang, “Single Image Haze Removal using Dark Channel Prior”, CVPR 09Slide5
Overview
Enhanced Image
Input Image
Enhancement Engine
Assessment Engine
Aesthetic Features
Image Semantics, Aesthetic Features
Appeal Prediction
Recomposition
Aesthetic ModelSlide6
Visual Aesthetics: Rule of Thirds
Motivated by Renaissance Paintings…
Rule of thirds:
Subject of interest is aligned to
one of the stress points
Professional photographs also abide this:
http://howtophotography.org/wp-content/uploads/2010/06/rule-of-thirds-photo2.jpg
http://hoocher.com/Joseph_William_Turner/Joseph_William_Turner.htmSlide7
Visual Aesthetics: Golden Ratio
http://hoocher.com/Joseph_William_Turner/Joseph_William_Turner.htm
Divine proportion:
Horizon divides sky and sea/land according to golden ratio.
http://www.dptips-central.com/rules-of-composition.html
An example professional photographic composition:
~1.618k
Sky
Sea
Sky
Land
kSlide8
Single subject
Compositions
(384)
Modeling Aesthetics: Dataset
Landscapes/Seascapes (248)
http://www.flickr.comSlide9
Single subjects
Modeling Aesthetics: User study
Landscapes/Seascapes
http://www.flickr.com
1
2
15
14
…
Rank Assignment between 1-5
Ground Truth Appeal FactorsSlide10
Modeling Aesthetics: User study
1.76
4.23
Poorly rated images
Best rated imagesSlide11
Modeling Aesthetics: User study
Appeal Factor Intervals
User Agreements
Good Compositions
Poor compositionsSlide12
Modeling Aesthetics: Features
(a)
Relative Foreground Location (Rule of Thirds)
Visual Attention Center
Stress PointSlide13
Modeling Aesthetics: Features
(b)
Visual weight deviation from Golden Ratio (Divine Proportion)
Y
k
Y
gSlide14
Experiments (Assessment)
Learn Support Vector Regression models
Prediction accuracy:
Single subject compositions ~ 87%
Landscapes/Seascapes ~ 91%
Smooth mapping between
Appeal factor and Aesthetic Features
Relative Foreground Location
Visual Weight Deviation Slide15
Spatial RecompositionSlide16
Why Cropping does not work?
Optimal CropSlide17
Recomposition: Algorithm I
Input Image
Labeled Elements
Semantic Segmentation
Single Subject?
Optimal Object Placement
Spatial
Recomposition
In-paintingSlide18
Semantic Segmentation
Input Image
Geometric Context Classifier*
*D.
Hoiem
, A.A.
Efros
, and M. Hebert, "Geometric Context from a Single Image",
ICCV
2005
Post Processing
Sky
Support
Horizon
Segmented ForegroundSlide19
Optimal Object Placement
Find
x
that Maximizes Appeal
Intensity
Term
Labeled Image
Support Neighborhood
Gradient Term
s.t
. neighbors stay “like neighbors”
+Slide20
Optimization (Example)
PAF = 3.31
PAF = 3.68
Semantic constraint prevents this
PAF = 3.22
Original Image
PAF = 4.53
Optimal Solution
XSlide21
Perspective Scaling
Scaling
Factor
Vanishing Point
Optimal location
Visual Attention Center
Scaled ForegroundSlide22
Inpainting Foreground Hole
Inpaint
Hole
Yunjun
Zhang.
Jiangjian
Xiao. Mubarak Shah
, “
Region Completion in a Single Image”,
EUROGRAPHICS 04Slide23
Recomposition: Algorithm 2
Input Image
Labeled Elements
Semantic Segmentation
Land/Sea
scape
?
Visual Weight Balancing
Optimally Crop/ExpandSlide24
Ratio of Current extents
Balancing Visual Weights
h
=
vertical extent of the balanced image
Solve for
h
(sign of
h
determines crop/expansion)
Y
k
Y
g
Y
k
+h
Y
gSlide25
Experimental Results
Horse is moved to a more visually pleasing location
Scaled appropriately
Appeal increases by 64%
Single Subject Composition
Before
Recomposition
After
RecompositionSlide26
Results
Before
After
PAF = 2.45
PAF = 4.29
PAF = 3.98
PAF = 4.46Slide27
Results
Before
After
PAF = 3.13
PAF = 4.19
PAF = 4.02
PAF = 4.34Slide28
Results
Before
After
PAF = 3.77
PAF = 4.25
PAF = 3.92
PAF = 4.11Slide29
Results
Before
After
PAF = 4.06
PAF = 4.68
PAF = 2.71
PAF = 3.26Slide30
Optimally cropped support region to increase weights for sky
Appeal factor increased by 51%
Visual
weight balancing
Results
Before
Recomposition
After
RecompositionSlide31
Balancing Visual weights
Before
After
PAF = 3.83
PAF = 4.02
PAF = 3.92
PAF = 4.38Slide32
Balancing Visual weights
Before
After
PAF = 4.02
PAF = 4.71
PAF = 4.17
PAF = 4.49Slide33
Not Perfect
Algorithm says nice, humans: otherwise
PAF = 2.34
Fa
= 2.41 (Ground Truth)
Before
PAF = 3.63
Fa
= 2.54 (Ground Truth)
AfterSlide34
Summary: Optimal Placement
Before
After
Increased
# of
Highly
rated Images
De
creased
# of
Poorly
rated ImagesSlide35
Summary: Visual Weights
Before
After
Increased
# of
Highly
rated Images
De
creased
# of
Poorly
rated ImagesSlide36
Conclusion
Intelligent photo
recomposition
Can also be used for aesthetic filteringEasy to use practical toolSlide37
Future Work
Synthesizing ideal image from many photos of the same scene
Recomposition
for videosSlide38
Questions?