Analysis of Inter- and Intra-Observer Error Associated with the Use of 3D Laser Scan Data - PowerPoint Presentation

Slide1

Analysis of Inter- and Intra-Observer Error Associated with the Use of 3D Laser Scan Data of the Pubic Symphysis

Jieun Kim*, Ph.D., Florida State UniversityBridget Algee-Hewitt, Ph.D., Stanford Univeristy*Presenting AuthorFebruary 20, 2018NIJ R&D Forensic Symposium

1

Slide2

Introduction

The three shape-based, computational methods by Slice and Algee-Hewitt (2015) and Stoyanova et al. (2015;2017): Use 3D laser scans of the pubic symphysis Target to minimize subjectivity in age estimation by reducing the effects of observer experience in the age-indicator assessment and methodological bias2

Slide3

Introduction

Stoyanova et al. 2015 have shown the improved repeatability of the methods through a single observer error test The computational methods’ performance and reproducibility have not been quantified at the level of multiple observers with different training background and experience levelsParticularly, there is potential for introducing error in the first two steps of data processing: Initial scanning of the pubic symphysisScan editing at different times by different observers. 3

Slide4

Introduction

Importance of quantifying observer error associated with these steps An emerging body of research that utilizes computerized, virtual age indicators and high-dimensional image data (e.g. CT/MRI scans): Villa et al. 2013; 2015; Boyd et al. 2015; Navega et al. 2017; Lottering 2013; 2014; Chiba et al. 2014; Curate et al. 2013, etc. 4

Slide5

Four Hypotheses

Hypothesis 1. The same observer will edit a set of raw scans inconsistently when editing is done repeatedly overtimeHypothesis 2. Scan editing skills will vary between observers based on their training background and/or levels of experience Hypothesis 3. Edited scans with different margin widths left around the pubic symphyseal face will yield different shape measures and age estimates Hypothesis 4. Edited scans with the ventrally protruding pubic tubercle will yield inaccurate age estimates for the VC method5

Slide6

H3: The Effects of Different Margin Widths around the

Symphyseal Face6

6

SB Male Cast Lower Phase V

2mm vs. 4mm vs. 1cm

Region of interest

Slide7

H4: The Effects of Editing with/without the Pubic Tubercle

7A pubic symphysis with a tubercle (semi-circled) extending to the face

(SB Female Cast Upper Phase VI)

Showing two different ventral curvatures depending on presence/absence of the tubercle

(SB Male Cast Upper Phase VI)

Slide8

Materials

3 replicate scans of the 12 Suchey-Brooks’ male casts taken by a single observer (n=36)8

Slide9

Methods

Four observers edited each set of the raw SB scans three timesObservers with various experience levels and training backgroundObserver 1. Biological anthropologist and developer of the scan editing protocolObserver 2. Scientific computing scholar and developer of the methods’ algorithms and software, forAge, (http://morphlab.sc.fsu.edu/software/forAge/index.html)Observer 3. Skeletal biologist with <2 years of experience with scan editingObserver 4. Forensic practitioner new to scan editing 9

Slide10

Methods (cont.)

From the edited scans, x,y,z coordinates were retrieved and subjected to a series of the shape analyses3 sets of shape measures Slice-Algee-Hewitt (SAH) scoresBending energy/thin plate splines (BE/TPS) values Ventral curvature (VC) values 5 sets of final age estimates obtained from single variable & multivariate regression analyses

10

Slide11

H1 & H2: Intra- & Inter-Observer Error Test

Intraclass correlation coefficients (ICC) Two-way random modelAbsolute agreement on single measuresICC guidelines proposed by Cicchetti 1994

11

ICC value

Interpretation

<0.40

Poor reliability

0.40-0.59

Fair reliability

0.60-0.74

Good reliability

0.75-1.00

Excellent reliability

Slide12

H3 & H4: The Effects of Different Editing Conditions on final age estimates

Age estimates vs. Known agesPaired t-test, alpha=0.0512

Slide13

Results: Intra-/Inter-observer Error (H1 & H2)

The resulting ICC values were high and mostly fell within the excellent reliability range (0.75-1.0) This demonstrates that the raw scans were edited consistently within and between observers and the derived shape measures and age estimates were highly reliable among observers. 13

Slide14

14

 Shape measure/Age estimateICC

DF (1,2)

F-Test

Sig. p

95% CI lower

95% CI upper

BE value

0.728

11,22.4

8.42

<0.001*

0.441

0.903

SAH score

0.856

11,22.6

20.5

<0.001*

0.675

0.952

VC value

0.942

11,23.9

48.8

<0.001*

0.859

0.981

BE estimate

0.91

11,23.1

29.7

<0.001*

0.785

0.971

SAH estimate

0.93

11,20

47.5

<0.001*

0.825

0.978

VC estimate

0.942

11,23.8

49

<0.001*

0.859

0.981

VC+BE estimate

0.949

11,24

56.4

<0.001*

0.875

0.984

VC+SAH estimate

0.94511,20.460.3<0.001*0.8610.982

INTRA

-observer error: ICC absolute agreement of within-observer shape measures and age estimates derived from first, second, and third time editing (only showing the ICCs of Observer 1), alpha=0.05

Slide15

15

Shape measure/Age estimateICC

DF (1,2)

F-Test

Sig. p

95% CI lower

95% CI upper

BE value

0.865

35,108

26.8

<0.001*

0.790

0.921

SAH score

0.832

35,27.8

29.3

<0.001*

0.696

0.911

VC value

0.756

35,32.8

18

<0.001*

0.594

0.863

BE estimate

0.829

35,60.4

24.1

<0.001*

0.726

0.902

SAH estimate

0.836

35,32.6

28.7

<0.001*

0.711

0.911

VC estimate

0.746

35,28.7

17.8

<0.001*

0.571

0.859

BE+VC estimate

0.811

35,25

26.2

<0.001*

0.656

0.900

SAH+VC estimate

0.85335,18.438.6<0.001*0.700

0.927

INTER-

observer error: ICC absolute agreement of between-observer shape measures and age estimates, alpha=0.05

Slide16

BE Value* Comparisons between Observers

16Note the proximity of the BE values generated for each cast by the four observers (ICC= 0.865)

*an average of three trial values was used for each observer

Slide17

Comparisons of Final Age Estimates* Generated from the BE values

17*an average of three trial values was used for each observer

All 4 observers yielded consistent age estimates (ICC=

0.829

).

The “X” symbol indicates the documented age for each of the 12 SB casts.

Slide18

Results: The Effects of Different Margin Widths on Age Estimates (H3)

The edited scans with 2mm and 4 mm margins did not produce age estimates significantly different from the known ages.However, 1cm margin produced a significantly different mean for all methods (mean diff. b/t estimated & known ages -14-18yrs, p<0.05), except the VC method (mean diff. -9yrs). 18

Age est.

N

Mean age

Mean age est.

Mean

diff.

SE

Upper 95%

Lower 95%

t

DF

Sig.

BE

12

38.833

24.570

14.263

6.023

27.522

1.005

2.368

11

0.037*

SAH

12

38.833

22.684

16.149

5.476

28.203

4.095

2.949

11

0.013*

VC

11

40.454

31.410

9.044

6.986

24.609

-6.520

1.294

10

0.224

VC+BE

11

40.454

22.935

17.519

6.595

32.215

2.823

2.656

10

0.024*VC+SAH1140.45422.48317.9726.06631.4894.4552.962100.014*

Slide19

Results: The Effects of the Ventral Tubercle on Age Estimates (H4)

Unlike the expectation, the inclusion of the pubic tubercle for the shape analysis did not yield inaccurate age estimates for the VC methodHowever, it did produce statistically significant mean differences for the SAH-score and TPS/BE methods and the two multivariate regression models (p<0.05). 19

Age est.

N

Mean age

Mean age est.

Mean diff.

SE

Upper 95%

Lower 95%

t

DF

Sig.

BE

12

38.833

21.653

17.181

5.281

28.804

5.557

3.253

11

0.008*

SAH

12

38.833

23.140

15.693

3.804

24.066

7.321

4.125

11

0.002*

VC

11

40.455

36.485

3.969

6.390

18.207

-10.268

0.621

10

0.548

VC+BE

11

40.455

23.052

17.403

5.366

29.358

5.447

3.243

10

0.009*

VC+SAH1140.45524.31016.1453.94124.9257.3644.099100.002*

Slide20

Results: The Effects of the Ventral Tubercle on Age Estimates (H4)

A likely reason for this is the fact that the tubercle is located at a different plane from the symphyseal face, and therefore the VC method algorithms are not significantly influenced by the inclusion/exclusion of the feature (mean diff. b/t estimated & known ages -4yrs). However, the methods that assess the complexity of the symphysial face (e.g. SAH-score & TPS/BE methods) may interpret the tubercle as an extra feature to account for, and therefore produce inaccurate (younger, mean diff. b/t estimated & known ages -16-17yrs) age estimates.

As expected, the scans edited without the tubercle yielded age estimates that were not significantly different from the documented ages.

20

Slide21

Discussion & Conclusion

The results show high repeatability of the Slice & Algee-Hewitt and Stoyanova et al.’s computational methods regardless of observer’s level of experience or training background.This further supports using a 3D laser scanner and scanned images to aid in resolving the issue of subjectivity and in standardizing data collection and analysis protocols between observers and institutions. Despite the simulated improper editing of the scans with various margin widths remaining, the computational methods were robust enough to self-correct and produce consistent and accurate age estimates. However, 1cm margin seems to be a threshold where the methods start generating incorrect age estimates.Although the inclusion of the pubic tubercle did not necessarily result in inaccurate age estimates for the VC method, we recommend removing the feature as it may “trick” the BE & SAH-score methods and the two multivariate regression models. 21

Slide22

Acknowledgements

We thank Dr. Judy M. Suchey for providing the documented chronological ages of the SB male casts. This project is funded by a National Institute of Justice grant (2015-DN- BX-K010) awarded to the senior authors, Slice and Algee-Hewitt.

22

Slide23

References

Villa C, Hansen MN, Buckberry J, Cattaneo C, Lynnerup N. Forensic age estimation based on the trabecular bone changes of the pelvic bone using post-mortem CT. Forensic Science International. 2013;233(1):393-402.Villa C, Buckberry J, Cattaneo C,

Lynnerup

N. Reliability of

suchey

‐brooks and buckberry‐chamberlain methods on 3D visualizations from CT and laser scans. American journal of physical anthropology. 2013;151(1):158-63.

Villa C, Buckberry J,

Cattaneo

C, Frohlich B,

Lynnerup

N. Quantitative analysis of the morphological changes of the pubic

symphyseal

face and the auricular surface and implications for age at death estimation. Journal of forensic sciences. 2015;60(3):556-65.

Boyd KL, Villa C,

Lynnerup

N. The use of CT scans in estimating age at death by examining the extent of ectocranial suture closure. Journal of forensic sciences. 2015;60(2):363-9.

Villa C, Buckberry J,

Lynnerup

N. Evaluating osteological ageing from digital data. Journal of Anatomy. 2016.

doi

: 10.1111/joa.12544.

Navega

D, Coelho

JdO

, Cunha E, Curate F. DXAGE: A New Method for Age at Death Estimation Based on Femoral Bone Mineral Density and Artificial Neural Networks. Journal of Forensic Sciences. 2017.

doi

: 10.1111/1556-4029.13582.

Curate F, Albuquerque A, Cunha EM. Age at death estimation using bone densitometry: testing the Fernandez Castillo and Lopez Ruiz method in two documented skeletal samples from Portugal. Forensic Science International. 2013;226(1):296. e1-. e6.

Lottering N, Macgregor DM, Meredith M, Alston CL, Gregory LS. Evaluation of the

suchey

-brooks method of age estimation in an Australian subpopulation using computed tomography of the pubic

symphyseal

surface. Am J Phys

Anthropol

. 2013;150(3):386-99.

doi

: 10.1002/ajpa.22213. PubMed PMID: 23283754.

Lottering N, Reynolds MS, MacGregor DM, Meredith M, Gregory LS. Morphometric modelling of ageing in the human pubic symphysis: Sexual dimorphism in an Australian population. Forensic science international. 2014;236:195. e1-. e11.

Chiba F, Makino Y,

Motomura

A,

Inokuchi

G,

Torimitsu

S, Ishii N, et al. Age estimation by quantitative features of pubic symphysis using multidetector computed tomography. International journal of legal medicine. 2014;128(4):667-73.

Kimmerle

, EH, Prince DA, and Berg GE. Inter‐Observer Variation in Methodologies Involving the Pubic Symphysis, Sternal Ribs, and Teeth. Journal of Forensic Sciences. 2008;53(3): 594-600.

Shirley, NR and Ramirez Montes, PA Age Estimation in Forensic Anthropology: Quantification of Observer Error in Phase Versus Component-Based Methods. Journal of Forensic Sciences. 2015;60(1): 107–111. doi:10.1111/1556-4029.12617.

Slice DE,

Algee

-Hewitt BF. Modeling Bone Surface Morphology: A Fully Quantitative Method for Age-at‐Death Estimation Using the Pubic Symphysis. Journal of Forensic Sciences. 2015;60(4):835-43.

Stoyanova D,

Algee

‐Hewitt BF, Slice DE. An Enhanced Computational Method for Age‐at‐Death Estimation Based on the Pubic Symphysis Using 3D Laser Scans and Thin Plate Splines. American Journal of Physical Anthropology. 2015;158(3):431-40.

Stoyanova D,

Algee

‐Hewitt BF, Kim J, Slice DE. A Fully Computational Framework for Age-at-Death Estimation from the Adult Skeleton: Surface and Outline Analysis of Three-Dimensional Laser Scans of the Pubic Symphysis. Journal of Forensic Sciences.

 2017. doi:10.1111/1556-4029.13439.Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological assessment. 1994;6(4):284.23

Slide24

Cristina Figueroa-Soto, MA

University of TennesseeWaukesha County ME OfficeBiological Anthropologycfiguer1@vols.utk.edu24Bridget Algee-Hewitt, PhD

Stanford University

Skeletal Variation, Human Genetics, Computational Biology, Forensics

bridgeta@stanford.edu

Dennis Slice, PhD

Florida State University

Morphometrics, Scientific Computing

dslice@fsu.edu

Detelina

Stoyanova

, PhD

Florida State University

Scientific Computing

detelinastoyanova@gmail.com

Jieun

Kim, PhD

Florida State University

Biological Anthropology jkim17@fsu.edu