The Illusion of Presence in Immersive irtual Reality during an fMRI Brain Scan HUNTER - PDF document

The Illusion of Presence in Immersive Virtual Realityduring an fMRI Brain ScanHUNTER G. HOFFMAN, Ph.D.,1,2,3TODD RICHARDS, Ph.D.,3BARBARACODA, M.D.,4ANNE RICHARDS, B.S.,3and SAM R. SHARAR, M.D.4ABSTRACTThe essence of immersive virtual reality (VR) is the illusion it gives users that they are insidethe computer-generated virtual environment. This unusually strong illusion is theorized tocontribute to the successful pain reduction observed in burn patients who go into VR duringwoundcare (www.vrpain.com) and to successful VR exposure therapy for phobias and post-traumatic stress disorder (PTSD). The present study demonstrated for the first time that sub-jects could experience a strong illusion of presence during an fMRI despite the constraints ofthe fMRI magnet bore (i.e., immobilized head and loud ambient noise).127CYBERPSYCHOLOGY& BEHAVIORVolume 6, Number 2, 2003© Mary Ann Liebert, Inc.ANIMMERSIVEVIRTUALREALITY(VR) system typi-cally consists of virtual reality software, ahead-tracking sensor, a helmet-mounted visual dis-play that blocks the patients view of the real world,three-dimensional (3-D) sound effects, and aninput device the subject uses to interact with theenvironment (to navigate through it, shoot snow-balls, and/or manipulate or influence virtual ob-jects). In a typical setup, patients wear a virtualreality helmet that positions two goggle-sizedminiature computer monitor screens near theireyes. Electromagnetic position tracking devices letthe computer know any time the person in VRchanges their head location or orientation. Some-times hand location is tracked with a second sen-sor. The scenery in the virtual world changes as theuser moves their head (e.g., they may see a riverwhen they look down, canyon walls when theylook to either side, and a blue sky when they lookup). Sometimes the patients can physically touchthe virtual objects, using real object props,1or com-puter generated force feedback devices like thepHanTom. The converging multisensory combina-tion of sight, sound, touch, and sometimes taste2and smell gives users a uniquely compelling expe-rience of “being there” in the virtual world. Theessence of immersive virtual reality is the illusion itgives users that they are inside the computer-generated environment, as if it is a place they havegone. This unusually strong illusion is theorized tocontribute to the successful pain reduction ob-served in burn patients who go into VR duringwoundcare3–5and the successful use of virtual real-ity exposure therapy for phobias6,7and posttrau-matic stress disorder (PTSD).8,9fMRI is a powerful new technology for studyingpatterns of brain activity associated with varioustypes of mental activities. In order to study pat-terns of brain activity, stimuli are often presented tomanipulate brain activity. Patients/subjects typi-cally wear prismatic glasses so they can watchslides presented on a rear-projection screen or com-puter screen located outside of the magnet bore.Researchers investigating neural correlates of spa-tial navigation have used “desktop” virtual real-ity.10,11With “desktop” virtual reality (notinvolving 1Human Interface Technology Laboratory, University of Washington, Seattle, Washington.2Department of Psychology, University of Washington, Seattle, Washington.3Department of Radiology, University of Washington School of Medicine, Seattle, Washington.4Department of Anesthesiology, University of Washington School of Medicine, Seattle, Washington. a virtual reality “VR” helmet), participants are un-likely to have a strong illusion of going inside thecomputer-generated virtual reality environment,which they see far away, through the tunnel of thebore hole with only a narrow angle of their visualfield. Researchers have begun exploring whethervirtual reality goggles could be brought inside thescanner. Displaying images close to the patient’seyes helps give patients the illusion of “presence,”the sensation that they are actually inside thecomputer-generated environment, interacting withvirtual objects, instead of merely watching the vir-tual world on a distant computer screen.12And in-creasing the field-of-view of a VR display has alsobeen shown to increase presence.13Awide field-of-view, interactive virtual reality image delivery sys-tem will allow more realistic interactions betweenpatients and stimuli during fMRI scans. And brainactivity patterns elicited by immersive VR stimulimight in some cases be more ecologically validthan brain activity elicited by conventional stimuli.Attempting to elicit a strong illusion of presencefrom subjects during an fMRI is challenging be-cause position sensors cannot be used, the patient’shead must be kept very still during the scans, thescanner makes loud noises, and the patient is lay-ing inside a magnet tube, which they may find dis-tracting. The present study is designed to determinewhether subjects are able to experience the illusionthat they have gone inside the virtual world duringan fMRI brain scan.Hoffman, Richards et al.14recently built a customdisplay into the MR radio frequency head coil toproject high resolution, wide field-of-view stereo-graphic images to subjects during an fMRI brainscan. Using virtual reality during an fMRI scan ischallenging because the strong magnetic field in-terferes with performance of electronic equipment(e.g., CRT screens that use magnetic fields to directelectrons onto a phosphor TV screen), and electric-ity creates electromagnetic fields that can ruin thebrain scan images. Although Hoffman, Richards etal. (submitted) used only non-conductive, non-ferrous materials in their device, the present studyis the first to test whether their fiberoptic magnet-friendly image delivery system causes any interfer-ence, ruining the brain scans.MATERIALS AND METHODSSubjectsSeven healthy human subjects (ages 20–30,median age = 23 years) participated in return formonetary compensation. All participants gave in-formed written consent. This study and consentforms were approved by the University of Wash-ington Human Ethics Committee.Experimental taskEach subject was placed horizontally in the MRIscanner and goggles were used to display the vi-sual stimulus to both eyes in stereo (Fig. 1). Themagnet-friendly fiberoptic virtual reality image de-livery system is described in detail by HoffmanRichards et al.14Instead of mounting miniature com-puterscreens in the helmet (conventional VR hel-met), with the magnet-friendly system, the virtualimages are first converted from electrons to pho-tons via 1024 2768 Infocus LCD projectors locatedoutside the magnet room. After being convertedinto light images, the real-time VR images are mini-fied with lenses and relayed to the patient via two15-foot long optic fiber image guides manufacturedby www.Schottfiberoptics.com. Each 8mm210 mmimage guide is comprised of 800 21000 very thinstrands of glass (fiberoptics) packed very close to-gether into an ordered array. After traveling throughthe 15-ft. long image guides, the light images entercustom-made VR goggles where they are magni-fied and seen by the patient as wide-field-of-viewimages with special VR optic lenses. Only light, noelectricity, reaches the subject’s head. Each subjectsees two independent computer-generated imagesof the virtual world (one for each eye), one imageslightly offset from the other, which their brainsfuse into a single 3-D illusory world with depth,simulating the normal visual depth cue called reti-nal disparity.The VR computer system consisted of a Dell 530workstation with dual 2-gig CPUs, 2 gigs of RAM,a Wildcat 6210 video card, and Windows 2000operating system, coupled with the custom fMRIVR image delivery system, with approximately67degrees horizontal and 29 degrees vertical fieldof view (circular eyepieces), with nearly 100%overlap in the images. The subject kept their headstill, and looked around in the virtual worldbymoving a magnet-friendly trackball. The com-puter quickly updated the virtual environmentpresented to the user by changing the viewpointin VR when the user moved their trackball. Thesubject had the illusion of flying through Snow-World, a virtual environment created withCreatorTM, Alias modelling software packages,and VEGATMdevelopment software from www. MultiGen.com. SnowWorld depicts an icy 3-Dvirtual canyon with a river and waterfalls. The 128HOFFMAN ET AL. subjects shot snowballs at snowmen and igloos,robots and penguins by aiming with their gaze,controlled via trackball and pressing the triggerbutton on the plastic magnet-friendly trackball.The snowballs exploded with animations and 3-Dsound effects on impact.The “on” condition consisted of an unobstructed3D Virtual Reality view of snow world and thiscondition was defined as high-tech virtual reality(Fig. 2). The “off” control condition consisted of thesame snow world but with a white cross that ob-structed part of the view, and this condition wasdefined as low-tech (Fig. 3). In the present study,subjects heard sound effects in both experimentalconditions and viewed SnowWorld through thesame magnet-friendly VR image delivery system.The high- and low-presence conditions were pre-sented to the subjects alternating every 30 sec dur-ing a 6-min fMRI scan.RESULTSImmediately after the fMRI brain scan, the exper-imenter asked each subject two questions over theintercom. Subjects were verbally asked to give anumber from 0 to 10 to indicate how present theyhad felt in the virtual world on the stimuli that didnot have a white cross (the high-tech SnowWorldcondition) and to give another rating of their pres-ence when they were in SnowWorld with the whitecross (the low-tech condition). The question theywere asked was as follows “While experiencing thevirtual world with nowhite cross, to what extentdid you feel like you went inside the virtual world?0 = I did not feel like I went inside at all, 1–4 = mildsense of going inside, 5–6 moderate sense of goinginside, 7–9 = strong sense of going inside, and 10 = Iwent completely inside the computer-generatedvirtual world. “While experiencing the virtualworld withthe white cross, to what extent did youfeel like you went inside the virtual world? 0 = Idid not feel like I went inside at all, 1–4 = mildsense of going inside, 5–6 moderate sense of goinginside, 7–9 = strong sense of going inside, and 10 = Iwent completely inside the computer-generatedvirtual world. They were told they could give frac-tions if they wanted. Their answer did not have tobe a whole number. On the scale from zero to ten,subjects rated presence in the high-tech conditionsignificantly higher than presence in the low-techcondition (mean presence rating = 7.0 for high-techand 4.1 for low-tech), t(6) = 10.59,p0.001, SE =0.28. In fact, each of the seven subjects showedhigher presence in the high tech condition than thelow tech condition.As can be seen from Figure 4, MR anatomicalscans and echoplanar images (both showing brainstructure but not brain function), acquired from fMRI VR PRESENCE129 FIG. 1.Subject in an MR magnet looking into VRgoggles. FIG. 2.Example of Snow World high-presence condition. FIG. 3.Example of Snow World low-presence condition. one of our subjects did not interfere with image ac-quisition. This was also verified with additional“phantom” scans using a head shaped like a ball ofwater. The functional MRI results showing patternsof brain activity associated with high- versus low-presence VR are not presented in this preliminaryreport because of concerns that the results wouldbe misinterpreted. Further (more carefully de-signed) research exploring presence-related brainactivity is needed.DISCUSSIONSubjects reported experiencing a strong illusionof presence in VR via the magnet-friendly VR imagedelivery system, in spite of the constraints of lyingdown with immobilized head in an enclosed envi-ronment (the fMRI bore) with loud knocking noises.Subjects reported a stronger illusion of presence inVR in the high-tech condition than in the low techcondition. And our magnet-friendly fiberoptic imagedelivery system did not interfere with the brainscans. VR could be used during fMRI to study anumber of psychological phenomenon (e.g., fear-related brain activity in phobics and PTSD patientsbefore and after therapy, drug craving of substanceabusers, and patterns of brain activity associatedwith the illusion of presence in virtual reality).Combining VR and fMRI could potentially lead to abetter understanding of the relation between whatpeople are thinking and experiencing, and their as-sociated patterns of brain activity.ACKNOWLEDGMENTSThis research was funded by NIH grantHD40954–01 to S.R.S. and a gift from the Paul AllenFoundation for Medical Research to David Patter-son at Harborview Burn Center. Figure 1 wascreated by cyberartist Duff Hendrickson (www. surrealstudio.com) and is copyrighted by H.G.H. (www.vrpain.com). Figures 2 and 3 were createdby and are copyrighted by H.G.H. Figure 4 was cre-ated and is copyrighted by T.R. Thanks to CecilHayes, Ph.D., Mark Mathis, B.A., and Jeff Magulafor engineering assistance and to Eric Seibel, Ph.D.,for optical engineering assistance. Also thanks toHoward Abrams (www.Howard-3rd.com) for cus-tomizing Snow World for the present study.REFERENCES1.Hoffman, H.G., Garcia-Palacios, A., Carlin, C., et al.(in press). Interfaces that heal: coupling real and vir-tual objects to cure spider phobia. International Jour-nal of Human–Computer Interaction.2.Hoffman, H.G., Hollander, A., Schroder, K., et al.(1998). Physically touching and tasting virtual ob-jects enhances the realism of virtual experiences. Vir-tual Reality: Research, Development and Application3:226–234.3.Hoffman H.G., Doctor J.N., Patterson, D.R., et al.(2000). Use of virtual reality for adjunctive treatmentof adolescent burn pain during wound care: Acasereport. Pain85:305–309.4.Hoffman H.G., Patterson D.R., & Carrougher, G.J.(2000). Use of virtual reality for adjunctive treatmentof adult burn pain during physical therapy: a con-trolled study. Clinical Journal of Pain16:244–250.5.Hoffman, H.G., Patterson, D.R., Carrougher, G.J., etal. (2001). The effectiveness of virtual reality basedpain control with multiple treatments. Clinical Jour- nal of Pain 17:229–235. 6.Rothbaum, B.O., Hodges, L.F., Kooper, R., et al.(1995). Effectiveness of virtual reality graded expo-sure in the treatment of acrophobia. American Journalof Psychiatry152:626–628.7.Garcia-Palacios, A., Hoffman, H. G., Carlin, C., et al.(2002). Virtual reality in the treatment of spider pho-bia: a controlled study. Behaviour Research and Therapy40:983–993.8.Rothbaum, B.O., Hodges, L.F., Ready, D., et al.(2001). Virtual reality exposure therapy for Vietnamveterans with posttraumatic stress disorder. Journalof Clinical Psychiatry62:617–622.9.Difede, J., & Hoffman, H.G. (2002). Virtual realityexposure therapy for World Trade Center Post Trau-matic Stress Disorder: a case report. CyberPsychology & Behavior 5(6):529–536. 10.Shelton, A.L., & Gabrieli, J.D. (2002). Neural corre-lates of encoding space from route and survey per-spectives. Journal of Neuroscience22:2711–2717. 130HOFFMAN ET AL. FIG. 4.Magnetic resonance (left) and echoplanar (right)structural scans indicate that the fiberoptic VR image de-livery system did not interfere with image acquisition. 11.Maguire, E.A., Burgess, N., Donnett, J.G., et al.(1998). Knowing where and getting there: a humannavigation network. Science280:921–924.12.Prothero, J., Hoffman, H.G., Parker, D.E., et al. (1995).Foreground/background manipulations affect pres-ence [On-line]. Available: www.hitl.washington. edu/publications/p-95–3//p-95–3.rtf. 13.Prothero, J., & Hoffman, H. (1995). Widening thefield of view increases the sense of presence in im-mersive virtual environments (R-95-5) [On-line].Available: www.hitl.washington.edu/publications/ r-95–5/. 14.Hoffman, H.G., Richards, T., Magula, J., et al. (sub-mitted). Amagnet-friendly virtual reality fiberopticimage delivery system.Address reprint requests to:Hunter G. Hoffman, Ph.D.,Human Interface Technology LaboratoryUniversity of WashingtonSeattle, WA98195E-mail:hunter@hitL.washington.edu fMRI VR PRESENCE131