organs at risk in brain cancer context via a deep learning classification scheme Jose Dolz Researcher Engineer at AQUILAB PhD s tudent at U1189 INSERM Little bit of history ID: 919087
Download Presentation The PPT/PDF document "Towards automatic segmentation of the" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
Slide1
Towards automatic segmentation of the organs at risk in brain cancer context via a deep learning classification scheme
Jose Dolz Researcher Engineer at AQUILABPhD student at U1189 INSERM
Slide2Little bit of history
2
SUMMER Project
Initial Training Network project funded by the European Commision
Seven partners
from different countries
It aimed at developing a
S
oftware
for the
U
se of
M
ulti-
Modality images in External Radiotherapy
My
task
:
Segmentation
of the
medical
structures in
brain
cancer
Slide3Little bit of history
3
AQUILABIt was the coordinator of the SUMMER
project.It is a company based in Lille.Quality assurance and evaluation software for medical imaging and radiotherapy.
INSERM U-1189
Onco
Thai
«
Laser
Therapies Assisted by Image for
Oncology
»
It develops
minimally invasive therapies using laser light. Among several research programs for localized cancer it is focused on Glioblastoma
Slide4Structure1
Introduction2
State of the art of segmentation methods
3Our main contributions
4
Methods
and
materials
5
Experiments and results
6
7
Discussion
Conclusion
Slide5Structure1
Introduction2
State of the art of segmentation methods
3Our main contributions
4
Methods
and
materials
5
Experiments and results
6
7
Discussion
Conclusion
Slide6Brain cancer
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Incidence
Cancer
is a leading cause of death and disability
worldwide:
14.1
million of new cancer cases
8.2
million deaths in
2012.Brain tumors:Second most common cause of cancer death
in men aged 20 to 39.
Fifth
most common cause of cancer among
women
aged
20 to 39.
Brain tumors can be categorized in benign and malignant.
Both types
are potentially disabling and life
threatening.
Consequently, both types require
treatment
.
4
Slide7Brain tumor treatment
Introduction
State of the artContributions
Materials
Results
Discussion
Conclusion
Radiation
therapy
(RT)
Medical application of
ionizing radiation to control malignant cells by damaging
their
DNA.
The three primary techniques:External or
conventional
RT
Internal
RT or
brachytherapy
Stereotactic
radiosurgery
(SRS)
During
RT,
high
dose of radiation
is
delivered
to
brain
cells
.
However
surrounding
healthy
tissue, i.e.
organs
at
risk
(
OARs
),
might
receive
part of
this
radiation.
Knowing
the exact volume and location of
OARs
is
of crucial importance on the RTP to
constrain
the
risk
of
severe
toxicity
.
Motivation of
imaging
in RT
Segmentation of the
OARs
is
required
in all of
these
techniques.
5
Slide8Need of automatization of the OARs segmentation process
Introduction
State of the artContributions
Materials
Results
Discussion
Conclusion
Delineation
is
manually performed by experts, or with very few machine
assistance.
Highly
time consumingThere exist significant variation between contours produced by different experts.
Medical
specialists spend a substantial
portion
of their
time to
image segmentation.
Furthermore,
recent
investigations
have
shown that the effects of inter-variability in delineating OARs have
a
significant
dosimetric
impact.
Nelms
et al. 2012,
Int
J
Radiat
Oncol
Biol
Phys
.
We expect that
by automatizing this process it is possible to
achieve:
a
more
repeatable set
of contours
that
can be agreed upon by the majority of
oncologists.
A reduction of the
time taken to perform this
step.
6
Slide9Structure1
Introduction
2State of the art of segmentation methods
3Our main contributions
4
Methods
and
materials
5
Experiments and results
6
7
Discussion
Conclusion
Slide10Introduction
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Image segmentation in the
medical
field
Medical
image segmentation distinguishes itself from
conventional image
segmentation tasks and still remains generally
challenging.Challenges in medical image segmentationMany medical imaging modalities generate very noisy and blurred images due to their intrinsic imaging
mechanisms
Medical
images may be relatively poorly
sampled (Partial Volume effects)
Some tissues share
similar intensity levels with
nearby regions
, leading to a lack of strong edge
information
along the
boundaries
.
Segmentation must
ideally
be
performed
in 3D.
7
Slide11Methods
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Methods
: atlas-
based
The
definition of an appropriate atlas or a set
of appropriate
atlases remains still an open
question.
Our experience reveal the need
of manual
editing or correction of the
automatic
contours.
The success of the atlas propagation highly depends on the registration step
.
Time and accuracy.
8
Slide12Methods
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Methods
:
statistical
models
(
SMs
)
A
major limitation of
SMs
is the
time
needed for model
construction
.
If
number
of
samples
is
not
sufficient
,
there
is
a
risk
of
overfitting
.
A precise initialization is required.
If the initial position is too distant from the
object to segment,
in
terms of translation
, rotation or scale, this can lead to
a poor segmentation.
In addition to the
shape
, the
appearance
can
also
be
modeled
.
9
Slide13State of the art of segmentation methods
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Methods
:
deformable
models
(
DMs
)
They are modeled by internal
and
external
forces.
Main
properties
:
No training
or previous knowledge is
required
by
deformable
models
.
They evolve
to fit into the desired
shape, showing
more flexibility than other
methods.
Definition
of
stopping criteria might become hard to achieve, and it depends on
the
characteristics
of the
problem
.
10
Slide14State of the art of segmentation methods
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Methods
: machine
learning
Common
features
include
:
Intensity
-
based
information.
Spatial information.
Probability
values.
Among
the
typical
techniques:
Artificial
neural networks (ANN)
Support
vector
machines (SVM).
11
Slide15State of the art of segmentation methods
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Artificial
neural networks
12
A
number
of simple
computational
units
are
connected
together
to
compute
a more
complex
function
.
Overfitting
if
training
goes on too long
,
n
ot enough training data.
ANNs often converge on
local minima
rather than global minima
.
Parametric models:
Fixed size.
Less parameters than SVM and easier to control.
Multi-class classification.
Slide16State of the art of segmentation methods
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Support
vector
machines
13
The basic idea of SVM is to construct an optimal
hyperplane
which gives maximum separation margin between two
classes.
Non-Parametric model:
Consists
of a set of support
vectors:
selected from the training
set
with a weight for each.
Classification slower.
Only two-class classification.
Slide17Summary and motivations
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Structures presenting large variations with respect to the training
dataset are not well handled by most presented methods.
Machine
learning techniques have
demonstrated
to
outperform other
, more traditional, approaches in segmenting brain structures.Recent developments of medical imaging acquisition techniques have led to an
increase of
complexity on the analysis of images.
Analysis where
large amount of data is compelled.
On
this context, we believe
that machine
learning techniques perfectly suit to deal with these new challenges
.
However, a new area of Machine Learning has recently emerged with
the intention
of moving machine learning closer to one of its original
purposes:
Artificial
Intelligence
.
14
Deep
Learning
Dolz
et al. 2015, IRBM
Slide18Structure1
Introduction
2State of the art of segmentation methods
3Our main contributions
4
Methods
and
materials
5
Experiments and results
6
7
Discussion
Conclusion
Slide19Introduction
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Advantages
of
deep
networks respect to
shallow
ones
Computational
efficiency.Typical machine learning approaches
must
be generally preceded by a feature selection step,
where most
discriminative features are privileged for a given problem
.
This is
not
usually needed
in deep learning-based classification schemes
.
Ability to automatically learn features from
data
With
the inclusion of layer-
wise
training,
they
do not
get
stuck
in local minima.
Hinton et al.
2006,
Neural Comput
Features extraction
Learning
Features selection
15
Slide20Deep Learning
Introduction
State of the art
ContributionsMaterials
Results
Discussion
Conclusion
Auto encoder (AE)
An auto-encoder is typically a
feedforward
neural network that aims to learn a compressed and distributed representation of a given input
.
16
An
AE
is
trained to minimize the discrepancy between the input data and its reconstruction.
Slide21Deep Learning
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Auto encoder (AE). Reconstruction
example
17
Slide221
2
Deep
Learning
Introduction
State of the art
Contributions
Materials
Results
Discussion
Conclusion
Denoised
auto encoder (DAE)
An auto-encoder with
N
inputs and encoding of dimension, at least
N
, could learn the identity function, which may lead to just a copy
function of the input.
To avoid this, some noise is randomly added in the input before the reconstruction.
This makes the
DAE more
robust than the
AE.
Vincent et al., 2008
J. Mach.
Learn
.
Res
.
3
Comparison
Input
Corrupted input
Representation
(Hidden)
Reconstruction
18
Slide23Deep Learning
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Stacking
DAEs
DAEs can be stacked to build deep network which has more than one hidden layer.
Each layer is trained at a time (i.e. layer-wise).
Output from layer k-1 is used as input for layer k.
Input
(x)
Output
(x’)
Unsupervised
Learning
19
Slide24Deep Learning
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Stacking
DAEs
DAEs can be stacked to build deep network which has more than one hidden layer.
Each layer is trained at a time (i.e. layer-wise).
Output from layer k-1 is used as input for layer k.
Input
(x)
x’
y
Unsupervised
Learning
19
Slide25Deep Learning
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Stacking
DAEs
DAEs can be stacked to build deep network which has more than one hidden layer.
Each layer is trained at a time (i.e. layer-wise).
Output from layer k-1 is used as input for layer k.
Input
(x)
Unsupervised
Learning
y
y’
v
19
Slide26Deep Learning
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Stacking
DAEs
DAEs can be stacked to build deep network which has more than one hidden layer.
Each layer is trained at a time (i.e. layer-wise).
Output from layer k-1 is used as input for layer k.
Input
(x)
Unsupervised
Learning
v
v’
w
19
Slide27Deep Learning
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Stacking
DAEs
DAEs can be stacked to build deep network which has more than one hidden layer.
Each layer is trained at a time (i.e. layer-wise).
Output from layer k-1 is used as input for layer k.
Input
(x)
Supervised
Learning
v
w
y
Labels
Fine
tunning
19
Slide28Training the deep network
Introduction
State of the art
ContributionsMaterials
Results
Discussion
Conclusion
Training
AC-PC
Alignment
Probability
map
Create
ROI
Voxel
pruning
Features
extraction
Features
scaling
Training
20
Obtention of
masks
Slide29Using the deep network
Introduction
State of the art
ContributionsMaterials
Results
Discussion
Conclusion
Classification
AC-PC
Alignment
Apply
ROI
Voxel
pruning
Features
extraction
Features
scaling
Classification
Post-
processing
21
Slide30Features used for classification
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Gradient
features
Vertical and Horizontal
Orientation
5
5
Features
array
for 1
voxel
inside
ROI
MRI
5x5x3 = 75
features
22
Slide31Features used for classification
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Contextual
features
For
each
voxel
32 patches
Continuous
value
Discrete
value
Voxel
intensity
Mean
patch
intensity
23
2x32 = 64
features
Bai et al., 2015
MedIA
.
Features
array
for 1
voxel
inside
ROI
Slide32Features used for classification
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Features
from
texture
analysis
Mean
Kurtosis
Wavelet
1st
Variance
Skewness
Energy
Entropy
Wavelet
2nd
1
1
Voxel
value on
each
image
24
Features
array
for 1
voxel
inside
ROI
Slide33Features used for classification
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Geodesic
Distance
Transform
Map
To
encourage spatial regularization and contrast-sensitivity
.
It exploits the
ability of seed-expansion
to
fill
contiguous, coherent regions
without
regard to
boundary
length
.
25
Features
array
for 1
voxel
inside
ROI
Slide34Features used for classification
Introduction
State of the art
ContributionsMaterials
Results
Discussion
Conclusion
3D Local Binary Texture Pattern
To catch neighborhood appearance with the fewest number of features, Local Binary Patterns (LBP) are investigated
.
Idea is to give a pattern code to each
voxel
26
4
features
Montagne et al., 2013
BIOSIGNALS
Pattern code:
Texture code:
Features
array
for 1
voxel
inside
ROI
Slide35Structure1
Introduction
2State of the art of segmentation methods
3Our main contributions
4
Methods
and
materials
5
Experiments and results
6
7
Discussion
Conclusion
Slide36Validation
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Design of validation protocol
Validation of medical image processing methods is of crucial
importance
their
performance
can
have an impact on the
performance of the larger systems in which they are embedded
Definition
of a standard protocol for validation may therefore have a high
relevance to facilitate
the
complete and accurate reporting of validation studies and
results
the
comparison of such studies and
results.
Jannin
et al. 2006, IJCARS
27
Slide37Validation
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Validation objective
In the clinical context of segmentation of OARs of brain cancer patients undergoing RT or radio
surgery
a segmentation method based on
a
stack of
denoised
AEsfed by a wide range of image-based featuresextracted from MR-T1 images
is able to segment those OARs
with
an accuracy that is significantly better than other state-of-the-art
methods
and that lies between experts variability
28
Slide38Imaging Data
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Dataset
MRI – T1 from 15 patients that underwent
Leksell Gamma
Knife
Radiosurgery
Resolutions
: 1x1x1 and 0.8203x0.8203x1
mm3 Several
MRI acquisition
systems
Several
pathologies:
Trigeminal neuralgia
Metastases
Brainstem
cavernoma
29
Slide39Training the deep network
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Pre
-
processing
Anterior
comisure
-
Posterior
comisure
(AC-PC)
alignment
Resampling
to 1 x 1 x 1 mm
3
Image
normalization
Creating
probability
map
30
Slide40Imaging Data
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Reference
contours
Manual
contours
from
up to 4 experts
were
collectedReference contours have been obtained by using the concept of probability maps:
Threshold of 50 and 75% from the max values
50 % (3 contours available)
75 % (4 contours available)
Contours used in the training
s
et to generate probability maps.
31
Slide41Evaluation
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Leave
-one-out-cross-validation
Training
Testing
15 Patients
Metrics
Spatial
overlap
Spatial distance
Efficiency
Dice
similarity
coefficient (DSC)
Sensitivity
Specificity
Volume
differences
Hausdorff distances (HD)
Processing
time
Statistical
analysis
ANOVA tests
were
conducted
over the
results
32
Slide42Structure1
Introduction
2State of the art of segmentation methods
3Our main contributions
4
Methods
and
materials
5
Experiments and results
6
7
Discussion
Conclusion
Slide43Experiments set-up
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
OARs
groups and classifier settings
33
OARs
groups
Classifier settings
Classifier
SDAE
SVM
Features
sets
Group
A
Classical
(
Intensity
,
probability
and spatial
based
)
Proposed
(
Probability
, spatial, GDTM and 3D-LBTP)
Group
B
Classical
(
Intensity
,
probability
and spatial
based
)
Textural
(
Classical
+ Textures)
Augmented
(
Classical
+
Gradient +
Contextual
)
Proposed
(All
together
)
SDAE
1
SDAE
2
SDAE
1
SDAE
Augmented
SDAE
Textural
SDAE
AE-FV
Slide44Results
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Dice
similarity
coefficients (DSC)
Automatic
Manuals
34
Group A
Group A
Group B
Group B
Slide45Results
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Hausdorff distances (HD)
Automatic
Manuals
35
Group A
Group A
Group B
Group B
Slide46Results
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Relative volume
differences
(
rVD
)
Automatic
Manuals
36
Group B
Group B
Group A
Group A
Slide47Sensitivity and specificity
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
37
Slide48Results
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Comparison
across
manual
contours
38
Left
Optic
nerves
Right
Optic
nerves
Slide49Results
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Segmentation times
Segmentation times reported by SDAE are about 2-3 orders of magnitude smaller than SVM
39
Slide50Discussion
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Comparison
among
automatic
settings
The
proposed
deep learning classification scheme outperformed all classifier settingsDifferences were statistically significant.
It drastically decreased segmentation times.
40
An accuracy that is significantly better than other state of the art methods
The
proposed
deep
learning
classification scheme
outperformed all classifier settings
Differences were statistically
.
It drastically decreased segmentation times.
Slide51Discussion
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Comparison
with
manual
annotations
The
segmentation
error we have obtained is comparable to the inter-rater difference observed when contours are delineated without time constraints.Statistical
analysis
point
out that
differences among all of them
were not generally statistically significant
.
41
And that lies between experts variability
The
segmentation
error
we
have obtained is
comparable to the inter-rater difference
observed
when contours
are delineated without time constraints
.
Statistical
analysis
point
out that
differences among all of them
were not generally statistically significant
.
Slide52Discussion
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
It is important to note that similarity metrics are very sensitive in
small
organs
.
Even
in the worst cases, where DSC was
above 0.55-0.60
for all the organs analyzed, the automatic contours can be
considered as a good approximation of the reference0.80
0.84
0.64
0.60
42
Slide53Discussion
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Comparison
with
other
works
Our
method outperforms presented methods in most cases:
Similarity
metrics
Spatial-
based
metrics
Segmentation times are considerably decreased.
No
need
for
complex
pre
-registration
process
.
No
need
for
initialization
for
each
single structure.
43
Slide54Structure1
Introduction
2State of the art of segmentation methods
3Our main contributions
4
Methods
and
materials
5
Experiments and results
6
7
Discussion
Conclusion
Slide55Discussion
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
44
Main contributions:
Use of auto-encoders to a new application.
Proposition of features not employed in segmentation.
Alternative way to create the features array in deep learning.
Slide56Future work
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Include
other image sequences in both contouring and
learning/classification
steps
.
Limited
datasets
.
Data augmentation.Automatic learning of best network architecture and parameters.From
functional
prototype to commercial
product
:
Optimization
.
Interface & User interaction engineering
Clinical
validation in a
larger
dataset
.
45
Slide57Structure1
Introduction
2State of the art of segmentation methods
3Our main contributions
4
Methods
and
materials
5
Experiments and results
6
7
Discussion
Conclusion
Slide58Conclusion
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
46
The objective of this thesis was
to
propose an approach
to segment OARs as alternative to existing
methods that
addresses
their limitations.
Particularly, a segmentation
scheme based on a deep learning
technique
was
suggested
.
Features
typically
employed
in machine
learning
segmentation are
extended
by
proposed
features
.
Results were compared against SVM and manual contours.
Our method
outperformed
SVM in this particular application
and
lay into the variability
of the experts.
Although this
work is not pioneering on the evaluation
of automatic
segmentation of OARs in the context of brain
RTP, it presents
important improvements respect to the
others.
Slide59My Future
Post-doctoral researcher at LIVIA department, ETS de Quebec, Montreal (Canada)
Deep
learning (CNNs) applied to medical semantic segmentationMulti MR modality for brain
parcellationMulti MR modality for Glioblastoma
segmentationMRI for cardiac images.CT/MRI for
spine
segmentation
Slide60Aknowledgements
Aknowledgements
The research leading to these results has received funding from the
European Union Seventh Framework
Programme
(FP7-PEOPLE-2011-ITN)
under grant agreement PITN-GA-2011-290148
.
SUMMER Project Consortium
Slide61End
Slide62End
Slide63Discussion
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
In RTP context, a method that is capable of managing deformations
and unexpected
situations on the OARs is highly desirable
.
The
employed
dataset
contained some cases where tumors inside the brainstem changed its texture properties.
Tumor was surrounded
Tumor was not surrounded
Slide64Software
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Workflow of the connection between MSVS and MATLAB
Slide65Experiments set-up
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
Parameters
settings for SDAE
Slide66CNNs
Introduction
State of the artContributions
MaterialsResults
Discussion
Conclusion
CNNs