Gannon University Department of Computer and Information Science Advisor Dr Sreela Sasi Automated Colorization of Grayscale Images Introduction Image Colorization 2 Automated Colorization of Grayscale Images ID: 345845
Download Presentation The PPT/PDF document "Christophe Gauge" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Christophe Gauge
Gannon University
Department of Computer and Information Science
Advisor: Dr. Sreela Sasi
Automated Colorization of Grayscale ImagesSlide2
IntroductionImage Colorization
2
Automated Colorization of Grayscale ImagesSlide3
Introduction (contd.)Digital Image Colorization
3
Automated Colorization of Grayscale ImagesSlide4
Introduction (contd.) Applications of Image Colorization
4
Automated Colorization of Grayscale ImagesSlide5
+
=
+
=
Previous Research
Image
Colorization
5
Automated Colorization of Grayscale ImagesSlide6
Current ResearchProcess Workflow
Texture-based Segmentation
Image
Image
Sample Image
Feature Extraction
Color Descriptors
Texture Descriptors
New Grayscale
Image
Texture-based Segmentation
Feature Extraction
Texture Descriptors
Texture Matching
Colorization Process
Database
6
Automated Colorization of Grayscale ImagesSlide7
Image SegmentationImage segmentation:
Is the partitioning of an image into homogeneous regions based on a set of characteristics.
Is a key element in image analysis and computer vision.
7
Automated Colorization of Grayscale ImagesSlide8
Image Segmentation (contd.)
Clustering:Is one of the methods available for image segmentation.
Is a process which can be used for classifying pixels based on similarity according to the pixel’s color or gray-level intensity.
8
Automated Colorization of Grayscale ImagesSlide9
Image Segmentation (contd.)Despite the substantial amount of research performed to date, the design of a robust and efficient clustering algorithm remains a very challenging problem
9
Automated Colorization of Grayscale ImagesSlide10
Color-based Image SegmentationComposite Image
10
Automated Colorization of Grayscale ImagesSlide11
Color-based Image Segmentation
Composite
Image
with salt & pepper noise added
11
Automated Colorization of Grayscale ImagesSlide12
Texture-based Image Segmentation
12
Automated Colorization of Grayscale ImagesSlide13
Workflow ProcessTexture-Based Image Segmentation
Original Image
Filtered Image
Filtered Image
Filtered Image
Filtered Image
…
Feature Image
Feature Image
Feature Image
Feature Image
…
Feature Image
Blobs
Gabor Filters
Energy Computation
Segmentation
Add, mean smoothing, normalization
13
Automated Colorization of Grayscale ImagesSlide14
14
Automated Colorization of Grayscale Images
Image Segmentation
Multi-Channel Filtering - Gabor TransformSlide15
Previous Research
(contd.)
Texture-Based Segmentation
15
Automated Colorization of Grayscale ImagesSlide16
16
Automated Colorization of Grayscale Images
Image Segmentation
Normalized Sum of Gabor ResponsesSlide17
Current ResearchProcess Workflow
Texture-based Segmentation
Image
Image
Sample Image
Feature Extraction
Color Descriptors
Texture Descriptors
New Grayscale
Image
Texture-based Segmentation
Feature Extraction
Texture Descriptors
Texture Matching
Colorization Process
Database
17
Automated Colorization of Grayscale ImagesSlide18
Previous Research
(contd.)
Clustering and Feature Extraction
18
Automated Colorization of Grayscale ImagesSlide19
Previous ResearchThe K-means algorithm has been used for a fast and crisp “hard” segmentation.
The Fuzzy set theory has improved this process by allowing the concept of partial membership, in which an image pixel can belong to multiple clusters.
This “soft” clustering allows for a more precise computation of the cluster membership, and has been used successfully for image clustering and segmentation.
19
Automated Colorization of Grayscale ImagesSlide20
The Fuzzy C-means clustering (FCM) algorithm [1] is a widely used method for “soft” image clustering. However, the FCM algorithm is computationally intensive.
It is also very sensitive to noise because it only iteratively compares the properties of each individual pixel to each cluster in the feature domain.
Previous Research
(contd.)
[1]
James C.
Bezdek
, Pattern Recognition with Fuzzy Objective Function Algorithms. New York: Plenum, 1981.
20
Automated Colorization of Grayscale ImagesSlide21
Image Segmentation
Modified Fuzzy C-means Clustering
21
Automated Colorization of Grayscale ImagesSlide22
Previous Research (contd.)Fuzzy C-means clustering (FCM) Algorithm
22
Automated Colorization of Grayscale ImagesSlide23
Previous Research (contd.)FCM Pseudo-code
Step 1 Set the number c of clusters, the fuzzy parameter m, and the stopping condition εStep 2 Initialize the fuzzy membership values µ
Step 3 Set the loop counter b
= 0Step 4 Calculate the cluster centroid
values using (3)Step 5 For each pixel, compute the membership values using (4) for each clusterStep 6 Compute the objective function
A. If the value of A between consecutive iterations < ε then stop, otherwise set b=
b+1 and go to step 4
23Automated Colorization of Grayscale ImagesSlide24
[2]
Stelios
Krinidis
and Vassilios Chatzis
, "A Robust Fuzzy Local Information C-means Clustering Algorithm," Image Processing, IEEE Transactions on, pp. 1-1, 2010.
Previous Research
(contd.)Modified Fuzzy C-means clustering with Gki
factorIn order to improve the tolerance to noise of the Fuzzy C-means clustering algorithm, Krinidis and
Chatzis [2] have proposed a new Robust Fuzzy Local Information C-means Clustering (FLICM) algorithm by introducing the novel Gki factor. The purpose of this algorithm is to adjust the fuzzy membership of each pixel by adding local information from the membership of neighboring pixels.
24Automated Colorization of Grayscale ImagesSlide25
Previous Research (contd.)Modified Fuzzy C-means clustering with Gki
factor
Sliding window of size 1 around the
i
th
pixel
The
G
ki
factor is obtained by using a sliding window of predefined dimensions:25Automated Colorization of Grayscale ImagesSlide26
Previous Research (contd.)Modified Fuzzy C-means clustering with Gki
factor
The G
ki factor is calculated by using the following equation:
26
Automated Colorization of Grayscale ImagesSlide27
Current AlgorithmModified Fuzzy C-means clustering with novel Hik factor
This algorithm is further improved by including both the local spatial information from neighboring pixels and the spatial Euclidian distance of each pixel to the cluster’s center of gravity.
In this research, the algorithm is also extended for clustering of color images in the Red-Green-Blue (RGB) color space.
27
Automated Colorization of Grayscale ImagesSlide28
Current Algorithm (contd.)
Illustration of the new
H
ik
factor displaying the spatial Euclidian distance to the center of gravity of each cluster
28
Automated Colorization of Grayscale ImagesSlide29
Current Algorithm (contd.)Process Workflow
Customize Parameters
Calculate cluster membership values
Compute
G
ki
Readjust membership values
Compute
H
ki
Compute objective functionDefuzzification and clustering
-
Image
Calculate cluster
centroid
29
Automated Colorization of Grayscale ImagesSlide30
Current Algorithm (contd.)Modified Fuzzy C-means Clustering
30
Automated Colorization of Grayscale ImagesSlide31
Simulation and Results
Synthetic Grayscale Test Image
31
Automated Colorization of Grayscale ImagesSlide32
Natural test image
FCM segmentation
with 5 clusters
FCM segmentation
using the modified FCM algorithm
with 5 clusters,
G
ki
window=1 and
H
ik
Simulation and Results
Natural Test Image
32
Automated Colorization of Grayscale ImagesSlide33
Simulation and Results
Synthetic Grayscale Test Image
Synthetic 4-color test image
with added salt and pepper noise
FCM clustering
FCM clustering
with
G
ki
window=1 and with
H
ik
FCM clustering
with
G
ki
window=5 and with
H
ik
33
Automated Colorization of Grayscale ImagesSlide34
Synthetic 4-color test image
with added salt and pepper noise
FCM clustering
FCM clustering
with
G
ki
window=1 and with
H
ik
FCM clustering
with
G
ki
window=5 and with
H
ik
Simulation and Results
Synthetic Color Test Image
34
Automated Colorization of Grayscale ImagesSlide35
Image SegmentationClustering Demo
35
Automated Colorization of Grayscale ImagesSlide36
Modified Fuzzy C-means Clustering
Summary
In this research, the FCM with the
G
ki
factor is modified using the
H
ik
factor, and the algorithm is extended for the clustering of color images. The use of the sliding window in the
Gki factor improves the segmentation results by incorporating local information about neighboring pixels in the membership function of the clusters. However, this resulted in a significant increase in the number of calculations required for each iteration for each pixel, and can be given by
36Automated Colorization of Grayscale ImagesSlide37
Modified Fuzzy C-means Clustering
Summary (contd.)
By combining the
G
ki
and the
H
ik
factors, this modified FCM algorithm considerably reduced the number of iterations needed to achieve convergence. The tolerance to noise of the Fuzzy C-means algorithm is also greatly increased, allowing for an improved capability to obtain coherent and contiguous segments from the original image.
37Automated Colorization of Grayscale ImagesSlide38
Modified Fuzzy C-means Clustering
Summary (contd.)
However, because of the radial nature of the spatial Euclidean distance to the cluster’s center of gravity, this new method may not be as effective for images containing circular shapes, or for images where the cluster’s center of gravity are close to each-other.
In this research, the FCM is extended for the clustering of color images in the RGB color space. The effectiveness of this algorithm may be tested for images in other color spaces also.
38
Automated Colorization of Grayscale ImagesSlide39
Current ResearchProcess Workflow
Texture-based Segmentation
Image
Image
Sample Image
Feature Extraction
Color Descriptors
Texture Descriptors
New Grayscale
Image
Texture-based Segmentation
Feature Extraction
Texture Descriptors
Texture Matching
Colorization Process
Database
39
Automated Colorization of Grayscale ImagesSlide40
40Automated Colorization of Grayscale Images
Sample Color ImagesSlide41
41
Automated Colorization of Grayscale Images
Image Segmentation
Normalized Sum of Gabor ResponsesSlide42
Image Segmentation
Feature Extraction
42
Automated Colorization of Grayscale ImagesSlide43
Image Segmentation
Feature Extraction
(contd.)
43
Automated Colorization of Grayscale Images
Blob Filtering for color and texture extraction.Slide44
44
Automated Colorization of Grayscale Images
Texture and Color database
Image Segmentation
Feature
Extraction
(contd.)Slide45
45Automated Colorization of Grayscale Images
Current Research
Process Workflow
Texture-based Segmentation
Image
Image
Sample Image
Feature Extraction
Color Descriptors
Texture Descriptors
New Grayscale
Image
Texture-based Segmentation
Feature Extraction
Texture Descriptors
Texture Matching
Colorization Process
DatabaseSlide46
46Automated Colorization of Grayscale Images
Grayscale Image ProcessingSlide47
47Automated Colorization of Grayscale Images
Current Research
Process Workflow
Texture-based Segmentation
Image
Image
Sample Image
Feature Extraction
Color Descriptors
Texture Descriptors
New Grayscale
Image
Texture-based Segmentation
Feature Extraction
Texture Descriptors
Texture Matching
Colorization Process
DatabaseSlide48
48Automated Colorization of Grayscale Images
Previous Research
Visual descriptors
Visual
descriptors are descriptions of the visual features of the contents
of images.They
describe elementary characteristics such as the shape, color, and texture.
MPEG-7 is a multimedia content description standard. It was standardized in ISO/IEC 15938 (Multimedia content description interface
).This description is associated with the content itself, to allow fast and efficient searching for material that is of interest to the user.
MPEG-7 is formally called Multimedia Content Description Interface. Thus, it is not a standard which deals with the actual encoding of moving pictures and audio, like MPEG-1, MPEG-2 and MPEG-4. It uses XML to store
metadata.Slide49
49Automated Colorization of Grayscale Images
Previous Research
Visual descriptors
http://chatzichristofis.info/?page_id=213
The
Img(Rummager) application was developed in the Automatic Control Systems & Robotics Laboratory at the Democritus University of Thrace-Greece
.The application can execute an image search based on a query image, either from XML-based index files, or directly from a folder containing image files, extracting the comparison features in real time. Slide50
Previous Research
(contd.)
Content-Based Image Retrieval
50
Automated Colorization of Grayscale ImagesSlide51
MPEG-7 EHD
Fuzzy Spatial BTDH
ADS
51
Automated Colorization of Grayscale Images
Previous Research
(contd.)
Content-Based Image RetrievalSlide52
Image Descriptors used:MPEG-7 Homogeneous Texture Descriptor: Edge Histogram
Descriptor (EHD).
CCD for Medical Radiology Images: Brightness and Texture Directionality Histogram (BTDH)Fuzzy rule based scalable composite descriptor (BTDH) is a compact composite descriptor that can be used for the indexing and retrieval of radiology medical images. This descriptor uses brightness and texture characteristics as well as the spatial distribution of these characteristics in one compact 1D vector. The most important characteristic of the proposed descriptor is that its size adapts according to the storage capabilities of the application that is using it. This characteristic renders the descriptor appropriate for use in large medical (or gray scale) image databases.
Simulation Results
(contd.)
Content-Based Image Retrieval (CBIR)
52
Automated Colorization of Grayscale ImagesSlide53
Simulation Results
(contd.)
Content-Based Image Retrieval (CBIR)
53
Automated Colorization of Grayscale ImagesSlide54
54Automated Colorization of Grayscale Images
Current Research
Process Workflow
Texture-based Segmentation
Image
Image
Sample Image
Feature Extraction
Color Descriptors
Texture Descriptors
New Grayscale
Image
Texture-based Segmentation
Feature Extraction
Texture Descriptors
Texture Matching
Colorization Process
DatabaseSlide55
The RGB color space is defined by the three chromaticities of the red, green, and blue additive primaries, and can produce any chromaticity that is the triangle defined by those primary colors.
The YCbCr
color space is used in video and digital photography systems.
Y is the luma (luminance ) component and
Cb and Cr are the blue-difference and red-difference chroma
components.
Simulation Results (contd.)
Image Colorization
55
Automated Colorization of Grayscale ImagesSlide56
56
Automated Colorization of Grayscale Images
Image from Wikipedia
Simulation Results
(contd.)
Image ColorizationSlide57
Simulation Results
(contd.)
Colorization
57
Automated Colorization of Grayscale ImagesSlide58
Conclusion and Future Work
New and innovative method
Automating example-based colorization
Combines several state-of-the-art techniques
Reasonably accurate results were obtained
Several of the steps require custom parameters
computationally very intensive
Texture retrieval needs improvement
Complex textures containing multiple colors
Anisotropic diffusion for preserving strong edge informationCombining these techniques in order to automatically colorize grayscale images is a viable option
58Automated Colorization of Grayscale ImagesSlide59
Conclusion and Future Work (contd.)
I
mages segmentation and clustering methods computationally
very intensive, P
rocessing
time for each 600x450 sample color image
took 20
minutes on a quad-core Intel 2.6 GHz processor
.Texture retrieval methods still need to be improved for scale and rotation invarianceStore
more complete color descriptors to accommodate more complex textures containing multiple colors. Anisotropic diffusion could also be used to smooth the Gabor response images while preserving strong edge information.
Testing conducted as part of this research proved that the ability to combine these techniques in order to automatically colorize grayscale images is a viable option. 59Automated Colorization of Grayscale ImagesSlide60
References
[1]
Anat
Levin,
Dani
Lischinski
, and
Yair
Weiss, "Colorization using optimization,"
ACM Transactions on Graphics, vol. 23, no. 3, p. 689–694, 2004.[2]R. Irony, D. Cohen-Or, and D. Lischinski, "Colorization by example," in
Eurographics Symposium on Rendering, 2005, p. 277–280.[3]
Ashikhmin M., Mueller K. Welsh T., "Transferring Color to Greyscale Images,".
[4]
X., Wan L., Qu Y., Wong T., Lin S., Leung C., Heng P. Liu, "Intrinsic colorization,"
ACM Trans. Graph.
, vol. 27, no. 5, p. 152, 2008.
[5]
Malik J. Perona P., "Preattentive texture discrimination with early vision mechanisms,"
J. Opt. Soc. Am. A
, vol. 7, no. 5, May 1990.
[6]
A. K. Jain and F.
Farrokhnia
, "Unsupervised texture segmentation using
Gabor filters
,"
Pattern Recognition
, vol. 24, no. 12, pp. 1167-1186, 1991.
[7]
Seo
Naotoshi
, "Texture Segmentation using Gabor Filters," University of Maryland, College Park, MD, Project ENEE731 , 2006.
[8]
Xiaoming
Hu
,
Xinghui
Dong,
Jiahua
Wu, Ping
Zou
Junyu
Dong, "Texture Segmentation Based on Probabilistic Index Maps," in
International Conference on Education Technology and Computer
, 2009, pp. 35-39
.
60
Automated Colorization of Grayscale ImagesSlide61
References (contd.)
[9]
X Muñoz, J
Freixeneta
, X
Cufı́a
, and J
Martı́a
, "Strategies for image segmentation combining region and boundary information,"
Pattern Recognition Letters, vol. 24, no. 1-3, pp. 375-392, January 2003.
[10]James C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms. New York: Plenum, 1981.[11]
Chuang Keh-Shih, Tzenga Hong-Long, Chen Sharon, Wu Jay, and Chen Tzong-Jer, "Fuzzy c-means clustering with spatial information for image segmentation," Computerized Medical Imaging and Graphics, vol. 30, no. 1, pp. 9-15, January 2006.[12]
Zhou Huiyu, Schaefer Gerald, Sadka
Abdul H., and
Celebi
M.
Emre
, "Anisotropic Mean Shift Based Fuzzy C-Means Segmentation of
Dermoscopy
Images,"
IEEE Journal of Selected Topics in Signal Processing
, vol. 3, no. 1, pp. 26-34, February 2009.
[13]
Stelios
Krinidis
and
Vassilios
Chatzis
, "A Robust Fuzzy Local Information C-means Clustering Algorithm,"
Image Processing, IEEE Transactions on
, pp. 1-1, 2010.
[14]
Gauge
Christophe
and
Sasi
Sreela
,
"Automated Colorization of
Grayscale
Images Using Texture Descriptors and a Modified Fuzzy C-Means Clustering,“
Journal of Intelligent Learning Systems and Applications (JILSA), Vol. 4 No. 2, 2012, pp. 135-143, DOI: 10.4236/
jilsa
.
61
Automated Colorization of Grayscale ImagesSlide62
62Automated Colorization of Grayscale Images
Questions?