Sebastian Sotela Introduction CS2Net Methodology Experimental Setup Results and Discussion ERNet Methodology Experimental Setup Results and Discussion Personal Review Takeaways References ID: 1042587
Download Presentation The PPT/PDF document "Curvilinear Structure Segmentation" is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.
1. Curvilinear Structure SegmentationSebastian Sotela
2. IntroductionCS2-NetMethodologyExperimental SetupResults and DiscussionER-NetMethodology Experimental SetupResults and DiscussionPersonal ReviewTakeawaysReferencesTable of Contents
3. Curvilinear Structures in Medical ImagesAutomated Segmentation crucial forUnderstandingDiagnosisTreatmentGeometrical or topological changes can determine pathologiesDiseasesKeratitisStrokesRetinal hematologic disordersModalitiesMRACCMOCTA January 19, 2023Computer Aided Medical ProceduresSlide 3Images (tow row) and their manual annotations of curvilinear structures (bottom row) in dierent medical imaging modalities. From the left to right column: Retinal color fundus image; Retinal optical coherence tomography angiogram (OCTA); Corneal confocal microscopy (CCM) image; Optical coherence tomography (OCT) and Brain MRA. Note that the manual annotations of OCTA and CCM are made at a centerline level, and thecerebral vasculatures are visualized in 3D by maximum intensity projection.
4. ChallengesResearch has focused on 2D segmentation more than in 3DFilter based methods depend on domain knowledge and manual tuningDeep learning methods focus on one single image modalitySparse manual annotationsUnbalanced distribution of voxels of edgesFailure to identify microvasculatureJanuary 19, 2023Computer Aided Medical ProceduresSlide 43D volume (top row) and their manual annotations of vessel-like structures(bottom row) in two sample medical imaging modalities. Left: Circle of Willis in MRA;Right: Olfactory projection fibers in two-photon microscope image.
5. What has been done so far?Old fashioned filter methodsEnhance curvilinear structure while suppressing noise and background pixelsExamplesActive contour based methodsHessian matrix based filtersMulti oriented filtersDeep learning methodsR2U-NetRecurrent neural network embedded into the U-shaped networkVessel segmentationDANet: dual attention networkSegmentation of natural imagesUpsamples attention features in the last layerUception (3D)U-Net architecture with inception modulesConsiderable GPU memory resources usage January 19, 2023Computer Aided Medical ProceduresSlide 5
6. CS2-NetU shaped encoder decoder architectureWith Residual blocksDual self attention mechanismsSpatialChannelDeals with 2D and 3D in a unified manner6 different image modalitiesJanuary 19, 2023Computer Aided Medical ProceduresSlide 6
7. CS2-Net: Methodology 2D NetworkJanuary 19, 2023Computer Aided Medical ProceduresSlide 7
8. CS2-Net: Methodology 2D NetworkSAB: Spatial Attention BlockModels spatial relationships between the features of any 2 pixelsCaptures the edge information of tree like structures in horizontal and vertical directionsCAB: Channel Attention BlockEnhance the contrast between class dependent features to help improve expressivenessLoss Function:Binary Cross Entropy lossJanuary 19, 2023Computer Aided Medical ProceduresSlide 8
9. CS2-Net: Methodology3D NetworkJanuary 19, 2023Computer Aided Medical ProceduresSlide 9
10. CS2-Net: Methodology 3D NetworkSAB: Spatial Attention BlockCAB: Channel Attention BlockLoss Function:Since the labels in 3D are sparse and have few high quality annotations Weighted cross entropy lossAdjust learning bias between vascularity and backgroundDice coefficient lossEnsure micro-cerebrovascular segmentationJanuary 19, 2023Computer Aided Medical ProceduresSlide 10
11. CS2-Net: Experimental Setup 2D NetworkSTARE, IOSTAR and OCTA used k-fold cross validationSTARE has image from half pathological and half healthy individualsAdam optimizerPoly learning rate policyData augmentation for trainingRandom croppingContrast enhancementRandom rotation Random flippingJanuary 19, 2023Computer Aided Medical ProceduresSlide 11
12. CS2-Net: Experimental Setup 2D NetworkMetricsAccuracyCorrect predicted pixels divided into the total amount of pixelsSensitivityCorrect vessel predicted pixels over total amount of ground truth vessel pixelsSpecificityCorrect background predicted pixels over total amount of ground truth background pixelsAUC: Area under the ROC curveTradeoff between sensitivity/“true positive rate“ and false positive rate (errors on bacground pixel prediction)p valuesFor statistical analysis, p<0.05 is considered statistically significantFalse discovery rate just on corneal nerve fiber tracingPixels incorrectly predicted as fiber over all pixels predicted as fiber January 19, 2023Computer Aided Medical ProceduresSlide 12Wikipedia (2022)
13. CS2-Net: Experimental Setup 3D NetworkReal dataset (public)MIDAS Brain MRAHealthy 18 to 60 years25 males25 femalesCircle of WillisTriangular polygonal surfaces voxelized with softwareBatch size 2Synthetic datasetsSynthetic136 volumesVascuSynthGaussian noise added with different variances for more realistic resultsVascuSynth 1: 20VascuSynth 2: 60VascuSynth 3: 100Batch size 6January 19, 2023Computer Aided Medical ProceduresSlide 13
14. CS2-Net: Experimental Setup 3D Network*Same settings for training except for maximum iterations (200)MetricsTrue Positive Rate/ SensitivityFalse Positive RateFalse positive voxels over ground truth background voxelsFalse Negative RateFalse negative voxels over ground truth vessel voxelsSegmentation RatesThe lower the values the better the performanceOver segmentation rateUnder segmentation rateDice coefficientNot appropriate for MRA volumes since many voxels have no labelsJanuary 19, 2023Computer Aided Medical ProceduresSlide 14
15. CS2-Net: Results and discussion 2D NetworkJanuary 19, 2023Computer Aided Medical ProceduresSlide 15Retinal vessel segmentation results of three randomly selected imagesfrom three dierent datasets by R2U-Net, DANet and our proposedCS2-Net respectively.
16. CS2-Net: Results and discussion 2D NetworkVessel segmentation performances in different metrics of different methods over three retinal fundus datasetsJanuary 19, 2023Computer Aided Medical ProceduresSlide 16
17. CS2-Net: Results and discussion 2D NetworkJanuary 19, 2023Computer Aided Medical ProceduresSlide 17Results of dierent methods for vessel segmentation of dierent images in dierent imaging modalities. From the left to right column: the originalimages, labels, and segmentation results of U-Net, Attention U-Net, DANet and the proposed CS2-Net, respectively. From the top to bottom row: OCTA,CORN-1 and OCT RPE Layer, respectively.
18. CS2-Net: Results and discussion 2D NetworkJanuary 19, 2023Computer Aided Medical ProceduresSlide 18
19. CS2-Net: Results and discussion ROC CurvesJanuary 19, 2023Computer Aided Medical ProceduresSlide 19
20. CS2-Net: Results and discussion 3D NetworkJanuary 19, 2023Computer Aided Medical ProceduresSlide 203D renderings of curvilinear structure segmentation results of animage in the MRA dataset. From the left to right column: a MIP view of asample MRA image, the segmentation of ground truth, the 3D U-Net andthe proposed CS2-Net respectively.
21. CS2-Net: Results and discussion 3D NetworkJanuary 19, 2023Computer Aided Medical ProceduresSlide 213D renderings of curvilinear structure segmentation results of differentmethods over Synthetic and VascuSynth. The first column showsvolumes with the dierent levels of noise (2 = 20 for Synthetic, 2 = 60for VascuSynth-2 and 2 = 100 for VascuSynth-3). Segmentation results ofdierent methods in the second to right column: ground truth, 3D U-Netand the proposed CS2-Net, respectively. The green boxes in dierent rowsshow an enlarged view of the local segmentation results.
22. CS2-Net: Results and discussionJanuary 19, 2023Computer Aided Medical ProceduresSlide 22Attention maps of dierent methods in the intermediate layers ofthe decoding parts. (a) the attention maps of the proposed CS2-Net in differentdecoding layers on dierent datasdets: DRIVE, STARE, IOSTAR,CORN-1, OCT-A, and OCT RPE datasets, respectively. D1 D4 displaythe attention maps representing the incremental refinement in curvilinearstructure segmentation; (b) the enlarged local intermediate attention mapsof dierent methods: U-Net, DANet, and CS2-Net.The output of the proposed CSAM on a randomly selected imagefrom the MIDAS dataset. From the left to right: the original volume, thepredicted probabilities of voxles as curvilinear structure before and afterapplying the proposed CSAM, respectively.
23. CS2-Net: ConclusionsJanuary 19, 2023Computer Aided Medical ProceduresSlide 23Curvilinear structure segmentation Essential step in automated diagnosisRemains a challengeCS2-Net outperformed state of the art methodsGreat potential for computer aided diagnosisAI deep learning models should be tried in different application to maximize impactImprovementsWith neighborhood or continuity constrains discard diseased cells marked as vessel-like structuresSimplify 3D architecture to reduce resource consumption
24. ER Net MethodologyJanuary 19, 2023Computer Aided Medical ProceduresSlide 24Encoder decoder architectureREAM: Reverse edge attention moduleFSM: Feature selection moduleEdge Reinforced Loss
25. ER Net MethodologyJanuary 19, 2023Computer Aided Medical ProceduresSlide 25REAM: Reverse edge attention moduleEmbedded between adjacent layers of the encoderIntersection of the foreground and background of different layersThe edge feature is discovered by increasing the weight of voxels on the edge partThe feature map after concatenation at the same position has more edge information
26. ER Net MethodologyJanuary 19, 2023Computer Aided Medical ProceduresSlide 26FSM: Feature selection moduleFeature fusion via simple channel stacking may lead to redundancyAdaptively selects effectiveRecovery features from decoderEncoding features from encoder2 layer FC with max(0, s(x)) as first activationThe final attention weights encode interdependency between different feature channelsX represents channel wise product+ represents weighted channel sum
27. ER Net MethodologyJanuary 19, 2023Computer Aided Medical ProceduresSlide 27Edge reinforced lossDice Similarity CoefficientHarmonic mean of precision and recallUsed as threshold to switch between lossMask supervised LossDice LossMeasures the non overlapping ratio between the prediction and the ground truthInsensitive to the number of foreground or background voxelsAlleviates class imbalance issueEdge refined lossDice loss Edge loss: Measures edge dissimilarity between prediction and ground truthEdge binary cross entropy lossDoes not consider global structureEdge dice lossAims to optimize global structureAuxiliary term is a regularizerKappa and tau are learnable parameters that restrict each otherThe remaining parameter is empirically set to 1The edge probabilities and labels are obtained using a 3x3x3 Laplacian operator
28. ER Net Experimental SetupJanuary 19, 2023Computer Aided Medical ProceduresSlide 28Datasets (public)2 CerebrovascularMIDAS100 MRA volumesHealthy volunteers18 -60 yearsCategoriesMIDAS I50 manual annotations of Circle of WillisPatches of 224x208x64MIDAS IIIntra cranial vasculature (center line + radius)42 manual annotationsPatches of 96x96x96 randomly cropped from the volume90 degree rotation for data augmentation2 Nerves DIADEM: Digital reconstruction of axonal and dendritic morphologyOPF: Olfactory projection fibers 9 separate Drosophila (fly) olfactory axonal projections image stacks2 channel confocal microscopyNL1A: Neocortical layer 1 axons16 image stacks involving numerous axonal trees2 photon laser scanning microscopy in vivoEach image stack represents 1 tile in a mosaic that contains all the fibers
29. ER Net Experimental SetupJanuary 19, 2023Computer Aided Medical ProceduresSlide 29General training settingsPyTorch frameworkSingle GPU RTX3090Normalization of the intensities of voxelsK fold cross validation5,5,4, and 3 respectiveleyAdam optimizerInitial learning rate 0.0001Weight decay 0.0005Poly learning rate policyPower of 0.9Maximum epoch 1000Lambda in ER loss (threshold for applying edge reinforced loss)0.8, 0.6, 0.8, and 0.7 respectively
30. ER Net Experimental SetupJanuary 19, 2023Computer Aided Medical ProceduresSlide 30MetricsSensitivitySpecificityDSCAverage Hausdorff distanceTakes voxel location into accountReflects the edge error of segmentation
31. ER Net Results and discussionJanuary 19, 2023Computer Aided Medical ProceduresSlide 31Segmentation results on cerebrovascular datasets by different methods: V-Net (Milletari et al., 2016), 3D U-Net (Çiçek et al., 2016), CS2-Net (Mou et al., 2020) and 3DU-Net++ (Zhou et al., 2019).
32. ER Net Results and discussionJanuary 19, 2023Computer Aided Medical ProceduresSlide 32Segmentation results on nerve datasets by different methods: V-Net (Milletari et al., 2016), 3D U-Net (Çiçek et al., 2016), CS2-Net (Mou et al., 2020) and 3D U-Net++ (Zhouet al., 2019).
33. ER Net Results and discussionJanuary 19, 2023Computer Aided Medical ProceduresSlide 33Examples of attention feature map. Three encoders (E1, E2, and E3) and decoders (D1, D2, and D3) accordingly represent the output feature map in the encoding anddecoding path of the network, respectively.
34. ER Net ConclusionsJanuary 19, 2023Computer Aided Medical ProceduresSlide 34ER NetDetects edges betterCaptures microstructureImproves connectivityOutperformed state of the art methodsPotential for automated diagnosisIncomplete manual annotation make MIDAS II and NL1A yield poorer resultsImprovementsThreshold lambda in ER Loss needs automated adjustment based on datasetSegmentation is not enough for practical purposesMust integrate quantification of biomarkers like density, length, tortuosity and caliberAdopt one of the following to avoid supervised learning:Semi supervised learningTransfer learningActive learning (query user)Data augmentation with GAN
35. Personal Review ComparisonJanuary 19, 2023Computer Aided Medical ProceduresSlide 35CS2-NetBothER-Netoptimization method Adam learning rate policy poly data augmentation for some datasets image modalitiesColor Fondus, OCTA, OCTMRA, Confocal Microscopy dimensions2D3D attention mechanismsCAB, SAB REAM, FSMloss functionsBCE,WCEDice LossEdge refined lossperformance Outperforms CS2-Net on all its metricsdatasetsprivate, syntheticpublic base architecture Encoder-decoder with residual blocks metricsAccuracy,area under the ROC, p values, FDR, FPR, FNR, OR,URSensitivity, specificity, dice coefficientAverage Hausdorff distanceNoise added to some datasets
36. Personal ReviewJanuary 19, 2023Computer Aided Medical ProceduresSlide 36Open questionsWould the 3D CS2-Net attention block yield the same results if the K vector was not for the x dimension?Will the complete integration of quantification of biomarkers happen in the next 5 years?Why not start donating data for these kind of studies?Applicability to other fieldsMorphological changes in the valves of the heart generate arrhythmia. This kind of technology can help detect this unwanted changes.WeaknessesIn none of the works there is a quantification of time to train the network or complexity of the algorithmSpecificity is surprisingly used as a metric on both works even though is not that relevant for unbalanced semantic segmentationThere are no results in other kind of curvilinear structures like bronchi and bronchiolesSome hyperparameters are empirically set with no further analysis StrengthsVisual demonstration are all over both worksBoth outperformed state of the art methods
37. TakeawaysJanuary 19, 2023Computer Aided Medical ProceduresSlide 37Attention improves significantly semantic segmentation problemsResearch should focus on improving deep learning methods for 3D imagesEdges are a important part curvilinear structure segmentation to guarantee continuityDeep learning methods should be agnostic of image modalityAny new AI development should be tried in solving different problemsResource consumption should be addressed for 3D volume problems
38. ReferencesJanuary 19, 2023Computer Aided Medical ProceduresSlide 38Mou, L., Zhao, Y., Fu, H., Liux, Y., Cheng, J., Zheng, Y., Su, P., Yang, J., Chen, L., Frangi, A.F., et al., 2020. CS2-Net: Deep learning segmentation of curvilinear structures in medical imaging. Med. Image Anal. 101874.Xia, L., Zhang, H., Wu, Y., Song, R., Ma, Y., Mou, L., Liu, J., Xie, Y. , Ma, M., Zhao, Y., 2022. 3D vessel-like structure segmentation in medical images by an edge-reinforced network Med. Image Anal. 102581.National Cancer Institute. Retrieved from https://www.cancer.gov/publications/dictionaries/cancer-terms/def/cornea Wikipedia (2022). Sensitivity and Specificity. Retrieved from: https://en.wikipedia.org/wiki/Sensitivity_and_specificity