Marker-Based Tracking - PowerPoint Presentation

Slide1

Marker-Based Tracking for Image-Guided Interventions

Ziv YanivSheikh Zayed Institute for Pediatric Surgical Innovation Children's National Medical Center

Last updated: Sep. 17 2012Slide2

Tracking

Continuously determine

the position

and possibly the orientation

of

tools/anatomical

structures relative to a fixed coordinate

system.Slide3

MotivationDisplay a dynamic virtual replica of the real world on screen, facilitating understanding of current spatial relationships between objects and enabling the clinician to predict the consequences of motions.Slide4

MotivationDisplay a dynamic virtual replica of the real world on screen, facilitating understanding of current spatial relationships between objects and enabling the clinician to predict the consequences of motions.Automated tool positioning (robot end effector pose) using a prediction model based on current and previous poses of other tools and anatomical structures. Slide5

MotivationDisplay a dynamic virtual replica of the real world on screen, facilitating understanding of current spatial relationships between objects and enabling the clinician to predict the consequences of motions.Automated tool positioning (robot end effector pose) using a prediction model based on current and previous poses of other tools and anatomical structures.

User interaction. Slide6

ClassificationsFundamental system principle(s)?mechanical, optical (infra-red or visible light), electromagnetic, ultrasonic, inertial [hybrid]. System characteristics?refresh rate and lag, number of objects tracked simultaneously, working volume, wired or wireless tools.Is the system performance effected by the environment?ferromagnetic materials, specific constraints on placement Does the system effect the environment?

introduce imaging artifacts, constraints on physical location Does the system have additional capabilities?Optical visible light systems provide the video stream.Slide7

Ideal Tracking Device

A la Welch and

Foxlin

:

Refresh rate and latency

:

refresh rate of 100Hz with a latency of less than 1ms, regardless of the number of tracked objects.

Concurrency

:

tracks up to 30 sensors concurrently.Working volume: has an effective work volume of 5^3m

(room sized).Obtrusiveness

: sensors are wireless and can function for several hours, all hardware components can be positioned so that they do not restrict the physical access to the patient, and the system does not have any effect on other devices used during the procedure.

Completeness

: sensors are small enough to embed in any tool and provide all six degrees of freedom (6DOF sensors).

Accuracy: resolution less than 0.1mm and 0.1

o.

Robustness

:

not affected by the environment (light, sound, ferromagnetic materials, etc.).

Cost

:

less than $5000.

“Motion Tracking: No Silver Bullet, but a Respectable Arsenal”,

G.

Welch and

E.

Foxlin

,

IEEE

Comput

. Graph. Appl.,

vol. 22(6), pp. 24-38

,

2002.Slide8

Forward/Direct kinematics approach. Encoders at the joints provide the relative translation and rotation.Mechanical Systems

Horsley-Clarke Frame

Faro ArmDa-Vinci RobotRefresh rate and latency++Concurrency

---Working volume-

Obtrusiveness

--Completeness+

Accuracy++Robustness

++Cost-

Pro: Highly accurateCon: Single object at a timeSlide9

Optical Systems (1…n cameras)

Pose estimation based approach

(single calibrated camera):

2D image data

 object pose

Triangulation based approach (two or more calibrated cameras):

nx(2D image data)3D structure  object pose

Camera 1

Camera 2

Camera

nSlide10

Optical SystemsInfra-red –passive markers  system illuminates the environment with IR light which is reflected.active markers  markers are IR-LEDs, emitting light.Visible light – passive markers  reflect visible light already present in the environment.Passive markers are always wireless, active markers are either wired or wireless (battery operated).Common configuration is “outside in”. “inside out” is also useful:

ActiSight

systemTMSlide11

Optical SystemsstereomonocularRefresh rate and latency+++Concurrency++Working volume+

Obtrusiveness+Completeness+++Accuracy+Robustness

+Cost+Refresh rate and latency+++Concurrency++Working volume++

Obtrusiveness-

Completeness

+++Accuracy++Robustness

++Cost-

Refresh rate and latency

+++

Concurrency+++Working volume+++Obtrusiveness

--Completeness

+++Accuracy

+++Robustness

+++Cost---

Pro: Highly accurate

Con: Require line of sightSlide12

Pose Estimation based Approach

(from calibration – interesting but not too useful)

When viewing an object with known geometry we know how to estimate the camera calibration matrix:

We want to obtain the matrices and given

The QR decomposition of a matrix factorizes the matrix into an orthogonal matrix and an upper triangular matrix .

We obtain K and R from P by taking the QR factorization of the inverse 3x3 left

submatrix

:

“Monocular

Model-Based 3D Tracking of Rigid Objects: A Survey”,

V.

Lepetit

, P.

Fua

, Foundations and Trends

in

Computer Graphics and

Vision, vol. 1(1), 2005.Slide13

Pose Estimation based ApproachWhen using planar fiducials we have a closed form solution based on knowledge of the internal camera parameters and the homography (projective linear transformation) between the image plane and marker plane.

The

homography

is estimated using four or more pairs of points

The pose is then obtained from Slide14

Triangulation based ApproachAccurately localize fiducials in images (blob/corner detection).Match corresponding fiducials:Epipole – projection of other camera origin onto image plane.Epipolar line – intersection of image plane and the plane defined by the origins of the two cameras and the viewed point.Given the calibration matrices and the point in one image the corresponding point in the other image is on the

epipolar line (obtained usingthe Essential matrix, not discussed here).Intersect back-projected rays toget the 3D points.Estimate pose using known 3Dstructure and computed one (paired-pointrigid registration a.la. Horn, Arun).

Camera 1

Camera 2

worldSlide15

Ray Intersection (stereo)

In general the rays do not intersect, so we look for the closest distance between them, the common normal (segment that is perpendicular to both):

Given two intersecting rays:

c

ross with

dot with

Similarly get:

m

id or intersection point:Slide16

Optical Systems (additional points)When using stereo, issues of wide vs. short baseline.Require 3 or more markers on tracked object in a known spatial configuration.Clinical standard of care.Slide17

Electromagnetic SystemsInducing current via magnetic field.AC or pulsed DC driven magnetic fields.Slide18

Electromagnetic SystemsSet of coils inside electromagnetic field generator.Miniature coil(s) embedded in tools (5DOF or 6DOF). Pulsed DC: sequence of static fields, pose obtained by measuring magnitude of sensor response to each field – problems with ferrous materials.AC: repeated signal pattern on high frequency carrier wave, pose obtained by amplitudes of the received signals – problems with eddy currents and resulting magnetic fields. There is a wireless transponder based system which is part of a radiotherapy system(Calypso Medical), but most are wired.Slide19

Electromagnetic SystemsRefresh rate and latency+Concurrency+Working volume+Obtrusiveness+ [depends on FG choice]Completeness+

Accuracy+Robustness+Cost+

Pro: Do not require line of sight.Con: Wired, and susceptible to transient distortions.Slide20

Ultrasonic Systems

Trilateration approach.

Compute an emitter’s distance from a set of receivers at known locations. Distance obtained using the time it takes sound to reach the receiver from the emitter (TOF).

m

icrophones mounted

on operating light:

“A Frameless Stereotaxic Operating

Microscope

for Neurosurgery”,

E.

M.

Friets

et al., IEEE

Trans. Biomed. Eng

., vol. 36(6), pp. 608-617

, 1989.

Refresh rate and latency

-

Concurrency

-

Working volume

+

Obtrusiveness

+

Completeness

+

Accuracy

+

Robustness

+-

Cost

-

Pro:

Unobtrusive.

Con:

Low refresh rate.Slide21

Ultrasonic SystemsA set of receivers (n>2) and an emitter yielda set of n quadratic equations:

m1m2m3

pd1

d3

d

2

In 3D when we have 3 receivers we can obtain two linear equations and a quadratic by subtracting the first equation from the other two:

Solve the two linear equations and obtain the x and y coordinates as a function of the z coordinate. Substitute back into the quadratic equation and obtain the z coordinate.

“

Efficient solution and performance analysis of 3-D position estimation by trilateration”, D. E.

Manolakis

,

IEEE

Trans.

Aerosp

. Electron. Syst., vol. 32(4), pp. 1239-1248

, 1996.Slide22

Ultrasonic Systems

Three possible solutions for z: Imaginary solution, no intersection between the three spheres.

One solution, single intersection point.Two solutions, two intersection points.For n>3, create a set of linear equations and solve either in an exact manner (n=4) or using a least squares formulation (pseudo-inverse).Mount 3 or more emitters on tracked object in a known spatial configuration, fire them one after the other with a minimal temporal separation that is greater then the time it takes to reach all receivers.Once all emitters are localized estimate the transformation using analytic paired point registration.Slide23

InertialRelative motion, integration based approach.Estimate change in position from linear acceleration (accelerometer).Estimate change in orientation from angular velocity (gyroscope). Refresh rate and latency+Concurrency+-Working volume+++Obtrusiveness

+Completeness+Accuracy-Robustness+

Cost+Pro: Infinite work volume.Con: Unknown relationship between sensors.Slide24

Fiber optic basedPro: High refresh rateCon: Low accuracyFiber optic bend and twist sensors along the tape.

Refresh rate and latency+Concurrency+-Working volume

+Obtrusiveness+Completeness+Accuracy-Robustness+Cost+Slide25

Tracking for the MassesLow CostMicrosoft Kinect$250 (USD)Leap Motion

$70 (USD)Now:

Near future?:Slide26

Practical AdviceChoosing a device:Depends on task and budget.Use the criteria defining the “ideal tracking device” as a reference. Once in a while check tracking quality.For optimal performance, follow manufacturers instructions (x minutes warm-up time etc.).This would be an inappropriate choice:Slide27

Trackers Galore[www.igstk.org]http://public.kitware.com/IGSTKWIKI/index.php/Supported_Tracking_SystemToolkit supports more than a dozen tracking devices (list is found here):Slide28

Markerless TrackingUse medical imaging modalities for tracking*:X-ray:“Markerless Real-Time 3-D Target Region Tracking by Motion Backprojection From Projection Images”, T. Rohlfing et al.,TMI,24(11), 2005.“Fast tracking of catheters in 2D fluoroscopic images using an integrated CPU-GPU Framework”, W. Wu et al., ISBI 2012. 3D US:“Prediction Based Collaborative Trackers (PCT): A Robust and Accurate Approach Toward 3D Medical Object Tracking”, TMI, 30(11), 2011.

“3D Ultrasound-Guided Motion Compensation System for Beating Heart Mitral Valve Repair”, S. G. Yuen et. al., MICCAI (1) 2008.Endoscopy:“Three-Dimensional Tissue Deformation Recovery and Tracking”, IEEE Signal Processing, P. Mountney et al., 27(4), 2010.

* These are random examples.Slide29

Thank YouIf you’ve always wanted to show friends and family what image-guidance means:

IGI Tutorial:

http://public.kitware.com/IGSTKWIKI/index.php/IGI_TutorialSlide30

Commercial CompaniesMechanicalFaro Technologies [www.faro.com]OpticalNorthern Digital Inc. (NDI) [www.ndigital.com]Claron Technology [www.clarontech.com]Atracsys [www.atracsys.com]AXIOS 3D Services [www.axios3d.de/EN]Advanced Realtime Tracking [www.ar-tracking.com]Vicon [www.vicon.com

]Slide31

Commercial CompaniesElectromagneticNorthern Digital Inc. (NDI) [www.ndigital.com]Ascension technology corporation (pronounced NDI) [www.ascension-tech.com]Polhemus [www.polhemus.com]InertialInterSense [www.intersense.com]XSens [www.xsens.com]Fiber OpticLuna Innovations [www.lunainnovations.com]Measurand [www.measurand.com]Slide32

Commercial CompaniesFor the MassesMicrosoft Kinect [www.microsoft.com/en-us/kinectforwindows]Leap Motion [leapmotion.com]