Summary of: “Comparison of thyroid segmentation techniques for 3D ultrasound” - PowerPoint Presentation

1. Summary of: “Comparison of thyroid segmentation techniques for 3D ultrasound” Master Seminar: Deep Learning for Medical Applications Radu Raicea

2. Introduction

3. What is the thyroid gland?The thyroid gland is part of the endocrine system. It regulates hormones released in the body10.Two types of diseases: hyperthyroidism and hypothyroidism.Both create an imbalance of hormones released in the body4.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 3Source: CFCF from wikipedia.org[4] Cleveland Clinic Medical Professional (Ed.). Thyroid disease: Causes, symptoms, risk factors, testing & treatment. Cleveland Clinic, retrieved 2021[10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017

4. How can we diagnose thyroid diseases?Measure the volume of the thyroid regularly9.Measure using ultrasound because it is much cheaper than a CT or MRI, it gives a real-time feedback, and it is more easily available.Conventionally, 2D segmentation methods are used and the volume is estimated using an ellipsoidal shape for each of the thyroid’s lobes10.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 4[5] Fujita, N & Kato, K & Abe, S & Naganawa, S. Variation in thyroid volumes due to differences in the measured length or area of the cross-sectional plane: A validation study of the ellipsoid approximation method using CT images. Journal of Applied Clinical Medical Physics, 2021[9] Poudel, P & Illanes, A & Sheet, D & Friebe, M. Evaluation of Commonly Used Algorithms for Thyroid Ultrasound Images Segmentation and Improvement Using Machine Learning Approaches. Journal of Healthcare Engineering, 2018[10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017Source: Fujita, N & Kato, K & Abe, S & Naganawa, S. [5]

5. Can we do better than 2D?The ellipsoidal method leads to bad estimations because US is hard to interpret and has a high variability in measurements.Instead, we can do a 3D US by collecting a stack of 2D US images and their positions.Manually segmenting each slice, we can construct a 3D model and estimate the volume.This is more accurate than using the ellipsoidal model (for healthy and unhealthy thyroids)10.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 5VolumeSource: Poudel, P & Hansen, C & Sprung, J & Friebe, M. [8]Source: Poudel, P & Hansen, C & Sprung, J & Friebe, M. [8][8] Poudel, P & Hansen, C & Sprung, J & Friebe, M. 3D Segmentation of Thyroid Ultrasound Images using Active Contours. Current Directions in Biomedical Engineering, 2016[10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017

6. Manual segmentation? Is that realistic?No, it would take too much time to manually segment hundreds of slices.Luckily, there are other methods that are semi-automatic or automatic9.The authors recognized previous comparative works done by Zhao et al.11 and Kaur et al.7, but those compared 2D segmentation techniques.This paper compares 3D segmentation techniques, and focuses on both the accuracy of the techniques and their clinical usability.The data is public to encourage future research.Three segmentation methods were compared:Level SetGraph CutFeature-Based Classifier10February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 6[7] Kaur, J & Jindal, A. Comparison of thyroid segmentation algorithms in ultrasound and scintigraphy images. International Journal of Computer Applications, 2012[9] Poudel, P & Illanes, A & Sheet, D & Friebe, M. Evaluation of Commonly Used Algorithms for Thyroid Ultrasound Images Segmentation and Improvement Using Machine Learning Approaches. Journal of Healthcare Engineering, 2018[10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017[11] Zhao, J & Zheng, W & Zhang, L & Tian, H. Segmentation of ultrasound images of thyroid nodule for assisting fine needle aspiration cytology. Health Information Science and Systems, 2013

7. Methodology

8. Level Set MethodTakes an initialization “snake”, which gets transformed to represent the edge of the segmented object8.Requires no further user interaction after initialization.If the segmentation is bad, we can start again with a different initialization.The accuracy of the method depends on the initialization10.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 8The method is called Active Contours without Edges (without edges because US images are noisy and the edges of the thyroid are not clearly defined)10.[8] Poudel, P & Hansen, C & Sprung, J & Friebe, M. 3D Segmentation of Thyroid Ultrasound Images using Active Contours. Current Directions in Biomedical Engineering, 2016[10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017Source: Poudel, P & Hansen, C & Sprung, J & Friebe, M. [8]

9. Graph Cut MethodThe method is called GrabCut, from the OpenCV library.Requires the user to initialize it by drawing a purple contour around the thyroid and a yellow scribble inside.The algorithm segments the slices using the color distribution of the thyroid and the area around it.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 9If a slice is not properly segmented, the user can correct it, which triggers a resegmentation of the other slices.[10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017Source: Wunderling, T & Golla, B & Poudel, P & Arens, C& Friebe, M & Hansen, C. [10]The accuracy of the method depends on the number of times the user corrects the incorrect segmentations10.

10. Feature-Based ClassifierThe classifier used is a decision tree.Features were based on the mean and standard deviation of the color values of a local neighborhood.Initialization is done by clicking on different areas in and outside the thyroid.The tree trains on the neighborhoods and then segments the rest of the slices.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 10[10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017If the segmentation is incorrect, the user can select more neighborhoods and retrain the decision tree.Source: Normalized Nerd on YouTubeThis method leads to noisy regions, so the authors did some post-processing on the final segmentation10.All the previously mentioned methods require the user to initialize them, so they are not fully automatic.

11. Can we segment the thyroid without user interactions?Yes, here are some methods based on Convolutional Neural Networks!February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 11They both require a large data set to train on.[2] Chen, F & Tinghui, Y & Hong, S & Songyuan, T & Jian, Y. MDIFNet: Multiscale Distant Information Fusion Network for Thyroid Segmentation in 3D Ultrasound Image. 6th International Conference on Multimedia Systems and Signal Processing (ICMSSP 2021), 2021[6] Gulame, M & Dixit, V & Suresh, M. Thyroid nodules segmentation methods in clinical ultrasound images: A review. Materials Today: Proceedings, 2021 [9] Poudel, P & Illanes, A & Sheet, D & Friebe, M. Evaluation of Commonly Used Algorithms for Thyroid Ultrasound Images Segmentation and Improvement Using Machine Learning Approaches. Journal of Healthcare Engineering, 2018MDIFNet: 3D U-Net: Source: Poudel, P & Illanes, A & Sheet, D & Friebe, M. [9]Source: Chen, F & Tinghui, Y & Hong, S & Songyuan, T & Jian, Y. [2]They require no user interactions.They segment rapidly, since they are already trained6.

12. Experimental Setup

13. What is the data? How was it collected?The authors collected 16 freehand-tracked 3D ultrasound records of healthy thyroids10.Freehand-tracked is when the probe is guided by the user’s hand and the position of the probe is tracked by sensors1.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 13The ultrasound collection was performed by a medical expert from MeVisLab using multiple sweeps using the GE ML6-15 probe and the GE Logiq E9 XDclear 2.0 device.Source: Peter Van Ooijen from researchgate.net[1] Cenni, F [Science in the Break]. 3D Freehand Ultrasonography: a video tutorial. YouTube, 2021[10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017The sixteen 3D ultrasounds were manually segmented by the medical expert, making them the “ground truth” for the comparison10.

14. How do we compare the methods with the ground truth?Two criteria were used for comparison:Accuracy of the methodClinical usability (user effort and waiting time for the computations)To compare the accuracy, a similarity statistic called the Dice coefficient was used.A Dice coefficient of 1 indicates identical segmentations, and a Dice coefficient of 0 indicates completely different segmentations.To compare the clinical usability, the number of user interactions, their duration and the computation time were used10.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 14[10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017

15. Results and Discussion

16. How accurate were the three semi-automatic methods?February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 16The Dice coefficient distribution shows a similar median performance among the methods, at around 0.7.The noticeable difference is in the range of Dice coefficients10.Source: Wunderling, T & Golla, B & Poudel, P & Arens, C& Friebe, M & Hansen, C. [10][10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017Source: Wunderling, T & Golla, B & Poudel, P & Arens, C& Friebe, M & Hansen, C. [10]We can see an example of the segmentation done by each of the three methods on one slice.White: manual (ground truth)Red: level setGreen: graph cutBlue: feature-based classifier

17. What can we notice in the range of Dice coefficients? The level set and feature-based classifier have some very under- and over-segmented thyroids, which happened especially in thin thyroid glands and in the area of the isthmus.The graph cut method had the biggest range of Dice coefficients. This was due to its accuracy being dependent on the amount of user corrections10.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 17[10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017Source: Wunderling, T & Golla, B & Poudel, P & Arens, C& Friebe, M & Hansen, C. [10]

18. What about the clinical usability?The level set method has a very short user interaction time, but computation time was relatively long.Due to bad segmentation, the algorithm had to be restarted 6.7 times on average.The graph cut method had the highest amount of user interaction.The average time spent interacting with each record was 36 seconds.A good balance between accuracy and interaction time was to perform a correction every 10 slices, or 2mm.The feature-based classifier also had a short interaction time, taking 13 seconds on average per record.The computation time was short, so even when the algorithm had to be reinitialized, it was not a significant time increase in the total time10.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 18[10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017

19. Are the automated methods any better?The data sets used in the CNN papers are different, but we can get an appreciation.In the 3D U-Net paper, their level set and graph cut methods had a DC of around 0.78, whereas the 3D U-Net had a DC of around 0.889.In the MDIFNet paper, they did not compare it with methods we’ve seen in previous slides, but the MDIFNet comes on top of the other methods, with a DC of 0.93.Also interesting is that, its experiment contained thyroids with nodules in the data set, which are more clinically relevant than healthy thyroids2.Both methods are fully automated and require no user interaction. The segmentation computation is also quite rapid, considering that the CNN is already trained2, 9.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 19[2] Chen, F & Tinghui, Y & Hong, S & Songyuan, T & Jian, Y. MDIFNet: Multiscale Distant Information Fusion Network for Thyroid Segmentation in 3D Ultrasound Image. 6th International Conference on Multimedia Systems and Signal Processing (ICMSSP 2021), 2021[9] Poudel, P & Illanes, A & Sheet, D & Friebe, M. Evaluation of Commonly Used Algorithms for Thyroid Ultrasound Images Segmentation and Improvement Using Machine Learning Approaches. Journal of Healthcare Engineering, 2018

20. What can we conclude?Older and semi-automatic methods for segmenting the thyroid gland perform about the same.Their accuracy depends heavily on the initialization, and hence the user interaction.For some, the computation time is high.The user interaction is frequent, even if short overall10.The newer and automatic methods perform better than the semi-automatic ones.Some, like the MDIFNet, perform extremely well even in thyroids with nodules2.They do not require any user interaction.The computation time is low.However, they need a large and diverse data set to train on2, 9.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 20[2] Chen, F & Tinghui, Y & Hong, S & Songyuan, T & Jian, Y. MDIFNet: Multiscale Distant Information Fusion Network for Thyroid Segmentation in 3D Ultrasound Image. 6th International Conference on Multimedia Systems and Signal Processing (ICMSSP 2021), 2021[9] Poudel, P & Illanes, A & Sheet, D & Friebe, M. Evaluation of Commonly Used Algorithms for Thyroid Ultrasound Images Segmentation and Improvement Using Machine Learning Approaches. Journal of Healthcare Engineering, 2018[10] Wunderling, T & Golla, B & Poudel, P & Arens, C & Friebe, M & Hansen, C. Comparison of thyroid segmentation techniques for 3D ultrasound. Medical Imaging, 2017

21. Review

22. Strengths / Weaknesses+ Good introduction to the thyroid segmentation problem.+ Different types of techniques were presented.+ Accuracy of the techniques were measured using the Dice coefficient, which allows comparisons with other papers (given that the same data is used).+ Comparisons were not limited to the accuracy of the methods, but also their clinical usability (by quantifying the user effort and the wait time during computations).- Lack of unhealthy or deformed thyroids in the data set.- Lack of thyroids with nodules or cysts, which have different textures on an image.- Lack of a clear elapsed time from start to finish of each method; it would have been interesting to have a table to clearly show the average elapsed time for each method.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 22

23. Future ResearchUsing pathological data sets to see if the relative differences between methods are similar.Combining the methods.Trying fully automatic methods like 3D U-Net and MDIFNet on the same data set.Perform the segmentations using multiple medical experts with different levels of experience to see how different initializations and interactions affect the accuracy and the interaction time of each method.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 23

24. Summary / Lessons LearnedThe thyroid gland is a very important organ where disease can heavily impact body functions.These diseases can be diagnosed by measuring the thyroid volume regularly to check for pathological changes.Ultrasound imaging is a cheap, fast, and effective modality for volume estimation.There is a wide range of different types of segmentation techniques to estimate the volume of the thyroid.Fully automatic techniques using CNNs have a higher accuracy than semi-automatic, and slower, techniques.CNN segmentation techniques require a very large and varied data set to train on beforehand, whereas the semi-automatic techniques shown do not require any previous data.February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 24

25. February 3, 2022Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 25Thank you!

26. References[1] Cenni, Francesco [Science in the Break]. 3D Freehand Ultrasonography: a video tutorial. YouTube. Retrieved November 10, 2021, from https://www.youtube.com/watch?v=lMJnpthHP2k. [2] Chen, Feng & Tinghui, Yin & Hong, Song & Songyuan, Tang & Jian, Yang. (2021). MDIFNet: Multiscale Distant Information Fusion Network for Thyroid Segmentation in 3D Ultrasound Image. 6th International Conference on Multimedia Systems and Signal Processing (ICMSSP 2021). Association for Computing Machinery, New York, NY, USA, 22–28. 10.1145/3471261.3471267.[3] Chen, Junying & You, Haijun & Li, Kai. (2020). A Review of Thyroid Gland Segmentation and Thyroid Nodule Segmentation Methods for Medical Ultrasound Images. Computer Methods and Programs in Biomedicine. 185. 1-18. 10.1016/j.cmpb.2020.105329.[4] Cleveland Clinic Medical Professional (Ed.). Thyroid disease: Causes, symptoms, risk factors, testing & treatment. Cleveland Clinic. Retrieved November 10, 2021, from https://my.clevelandclinic.org/health/diseases/8541-thyroid-disease.[5] Fujita, Naotoshi & Kato, Katsuhiko & Abe, Shinji & Naganawa, Shinji. (2021). Variation in thyroid volumes due to differences in the measured length or area of the cross-sectional plane: A validation study of the ellipsoid approximation method using CT images. Journal of Applied Clinical Medical Physics. 2021. 10.1002/acm2.13125. [6] Gulame, Mayuresh & Dixit, Vaibhav & Suresh, M. (2021). Thyroid nodules segmentation methods in clinical ultrasound images: A review. Materials Today: Proceedings. 45. 10.1016/j.matpr.2020.10.259.[7] Kaur, Jaspreet & Jindal, Alka. (2012). Comparison of Thyroid Segmentation Algorithms in Ultrasound and Scintigraphy Images. International Journal of Computer Applications. 50. 24-27. 10.5120/7959-0924.[8] Poudel, Prabal & Hansen, Christian & Sprung, Julian & Friebe, Michael. (2016). 3D Segmentation of Thyroid Ultrasound Images using Active Contours. Current Directions in Biomedical Engineering. 2016. 10.1515/cdbme-2016-0103.[9] Poudel, Prabal & Illanes, Alfredo & Sheet, Debdoot & Friebe, Michael. (2018). Evaluation of Commonly Used Algorithms for Thyroid Ultrasound Images Segmentation and Improvement Using Machine Learning Approaches. Journal of Healthcare Engineering. 2018. 1-13. 10.1155/2018/8087624.[10] Wunderling, Tom & Golla, Björn & Poudel, Prabal & Arens, Christoph & Friebe, Michael & Hansen, Christian. (2017). Comparison of thyroid segmentation techniques for 3D ultrasound. Proc. SPIE 10133, Medical Imaging 2017: Image Processing. 10.1117/12.2254234.[11] Zhao, Jie & Zheng, Wei & Zhang, Li. (2012). Segmentation of Ultrasound Images of Thyroid nodule for Assisting Fine Needle Aspiration Cytology. Health Information Science and Systems. 1. 10.1186/2047-2501-1-5.December 2, 2021Master Seminar: Deep Learning for Medical Applications – Radu RaiceaSlide 26