Perception & Psychophysics - PDF document

Perception & Psychophysics When auditory and visual events, such as sound burstsand light flashes, occur close together in time but atsomewhat disparate locations, source localization for onemodality interacts with that for the other modality incharacteristic ways (Bermant & Welch, 1976; Bertelson,esting because they may help us to understand the pro- 321Copyright 2000 Psychonomic Society, Inc. The present work was supported by the Ministry of Scientific Re-search of the Belgian French-Speaking Community (Concerted Re-search Actions 91/96-148 and 96/01-2037) and by the Belgian National PAUL BERTELSONand Tilburg University, Tilburg, The NetherlandsTilburg University, Tilburg, The NetherlandsBƒATRICE 322BERTELSON, VROOMEN, while trying to ignore the stimulation in the other modal-ity. Responses characteristically show partial displace-ment in the direction of the irrelevant competing stimu-lus from the other modality (Bermant & Welch, 1976;Bertelson & Radeau, 1981, 1987; Bertelson, Vroomen,Wiegeraad, & de Gelder, 1994; Klemm, 1909; Radeau,1992; Radeau & Bertelson, 1976, 1987; Thomas, 1941;Warren, Welch, & McCarthy, 1981). Most published dem-onstrations concern visual biasing of auditory localiza-tion, but there is evidence for a smaller auditory biasingtelson & Radeau, 1981, Experiment1; Radeau & Bertel-son,1987). A further on-line influence, perceptual observed when a subject mistakenly reports that the dis-mon location (Bertelson & Radeau, 1981, Experiment2;Choe, Welch, Gilford, & Juola, 1975; Jack & Thurlow,1973; Witkin, Wapner, & Leventhal, 1952). Finally, af-tereffects can be observed when subjects perform local-ization for unimodal stimuli after a period of exposure tocross-modal discordance. In comparison with preexpo-sure localization, postexposure localization is typicallydisplaced toward the direction of the previous competingstimulus from the other modality, just as for on-line cross-Sensory Versus CognitiveFactors in VentriloquismMany studies of ventriloquism have used fairly realis-tic situations, simulating real-life bimodal events, such asthe voice of a person speaking and the concurrent sightof his or her face (e.g., Bertelson etal., 1994; Radeau &Bertelson, 1977, Experiments 2Ð3; Warren etal., 1981;Witkin etal., 1952), the sight of whistling kettles (Jack-son, 1953), or the sight of beating drums (Radeau & Ber-telson, 1977, Experiment1). In such situations, the ob-served interactions might, to some degree, originate in theobserverÕs prior knowledge of the simulated situationsand, thus, may have a cognitive rather than a strictly sen-sory basis. However, ventriloquism is also found in sim-plified unfamiliar situations, such as those in which the in-of light. Here, structural properties of the bimodal sen-sory input, such as the degree of temporal synchrony(Choe etal., 1975; Klemm, 1909; Radeau & Bertelson,1987; Thomas, 1941), spatial separation (Bertelson &Radeau, 1981; Choe etal., 1975), or the relative saliencyof the inputs (Radeau, 1985), constrain the cross-modalRadeau and Bertelson (1977) compared the aftereffectsof exposure to simplified versus realistic conflict situa-tions. A voice reading a text was combined, in the realis-ticcondition, with the sight of the speakerÕs face and, inthe simplified condition, with light flashes synchronizedto the amplitude peaks of the speech. Aftereffects of com-parable sizes were produced in the two situations, sug-gesting that cognitive influences from a familiar contextare not required for cross-modal adaptation to occur.On the other hand, Pick, Warren, and Hay (1969) re-ported a result that seems to demonstrate a role for purelycognitive factors in determining on-line cross-modal ef-fects. Their observers, when pointing to the apparentsource of a sound, shifted responses toward the discor-dant location in which a dummy loudspeaker was visible.This result was presumably a reflection of the knowledgethat loudspeakers can deliver sounds. However, in a morerecent experiment (Bertelson & Radeau, 1987; Radeau,1992), subjects exposed to a similar situation showed no visual bias of auditory localization,whereas in a noise bursts and light flashes presented synchronouslybut at discrepant locations), they showed the usual visualinfluence. The fact that the cognitive influence reportedby Pick etal. seems less reliable than that from synchro-nized sensory conflict suggests that the former influencemay reflect postperceptual response biases, which are vari-ablein nature.Taken together, these previous results imply that therole of nonsensory factors in ventriloquism may be verylimited at best. However, Welch and Warren (1980, 1984;see also Warren etal., 1981, and Welch, 1994) proposedthat even the effects of stimulus factors, such as temporalsynchrony and spatial separation, might themselves bemediated through judgmental cognitive processes, ratherthan through sensory processes. This notion received ap-parent support from an application of signal detectiontheory to findings concerning the effect of temporal syn-chrony on reports of audiovisual fusion (Choe etal., 1975).Choe etal. reported that synchrony apparently affecteddecision criteria, rather than perceptual sensitivity (i.e.,), and concluded that ventriloquism was due to acognitive response bias, rather than to a sensory change inperceived location.However, close examination of Choe etal.Õs (1975)method showed that the particular perceptual model theydiscounted was implausible from the outset. Bertelsonand Radeau (1976) argued that a truly sensory effect ofreconciled with the signal detection data. On the otherhand, they did not deny that strategic cognitive factorscan influence responses in some audiovisual conflict sit-uations (as for most phenomena based on observersÕ re-ports). Evidence for a genuinely sensory component tovisual biasing of auditory localization has recently beenreported (Bertelson & Aschersleben, 1998), using a psy-chophysical staircase approach. This showed that the ap-parent location of sound bursts was attracted toward syn-chronous light flashes even when the subject showed noawareness of the spatial discrepancy, so that any judg-mental factors were presumably minimized. VENTRILOQUISM AND DELIBERATE VISUAL ATTENTION323 Possible Role of the Directionof Attention in VentriloquismOur discussion so far has focused on the contrast be-tween sensory accounts of ventriloquism as a genuineperceptual phenomenon and accounts that attribute it tomore cognitive factors. There is, however, a third possi-bility, which has been generally ignored in previous dis-cussions: an involvement of the direction of the generation of ventriloquism. This possibility has be-come apparent in light of recent work on cross-modalDriver & Spence, 1994; Spence & Driver, 1996), as wewill discuss below.Deliberate attention is not a strictly sensory factor,since it can be directed voluntarily under the observerÕsown cognitive control; but neither is it a strictly nonper-ceptual factor, since the attentional state of observers canaffect their perception, not only their judgmental or deci-cognitive penetrabililityof perception still remains controversial with respect totop-down influences from knowledge (see, e.g., Krueger,1989; Pylyshyn, 1999), the fact that deliberate attentionPashler, 1998, for an extensive review; also, Pylyshyn,1999). Thus, considering the possible role of attention inventriloquism should broaden the debate about the un-derlying mechanisms, beyond the simple sensory versuscognitive dichotomy that has been considered in the past. In discussing how attention might influence ventrilo-quism, one must distinguish several different senses inwhich the term gets used. We may attend to onesensory modality versus another, regardless of location(Spence & Driver, 1997b); or to one location versus an-other, regardless of modality (Spence & Driver, 1996).selection. Moreover, for the case of spatial attention,we can use overtsensitive receptors toward a region of interest) or purelycovertmechanisms (which enhance perception at partic-ular locations even though no receptors shift; see Posner,1980). Finally, our attention may be driven by the suddenonset of a salient stimulus, regardless of our intentionsexogenousattention capture); or may be directed delib-erately, on the basis of expectations or task requirementsendogenousattention); see Yantis and Jonides (1990)for examples of this exgogenous/endogenous distinction.There is now considerable evidence suggesting qualitativedifferences between the operation of exogenous versus en-dogenousattention, both in cases of spatial selection (seeSpence & Driver, 1996, 1997a) and of modality selection(Spence & Driver, 1997b). It is, therefore, important tokeep these forms of attention distinct in any theoreticaland its limits are clearly involved inclassical demonstrations of ventriloquism. The fact thatobservers can be influenced by spatial location in onemodality, even when asked to concentrate entirely on stim-ulation in another modality, indicates that modality selec-tion is imperfect. Evidently, the irrelevant modality cannotbe entirely filtered out, since it affects judgments of thetarget modality, despite instructions. Ventriloquism thusreveals a partial failure of endogenous modality selection.On the other hand, the failure is indeed only partial, sinceobservers typically report different locations when in-structed to localize either the visual or the auditory stim-ulation, and the sum of these biases is generally less thanthe actual spatial separation between the sources (Ber-telson & Radeau, 1981; Pick etal., 1969; Radeau, 1985,1992; Warren etal., 1981). In an experiment in which bothunimodal pointing and sameÐdifferent origin judgmentswere recorded on every trial, Bertelson and Radeaufound incomplete bias even on those trials on which per-ceptual fusion was reported. Thus, the reported locationchanges with modality instructions even for fused per-cepts. Mere instructions can apparently influence theweight given to data from the two modalities in the inte-gration operation or, at least, their weight in the readoutfinding is that the aftereffects of exposure to audiovisualdiscrepancy, subsequently observed within either modal-ity when tested alone, can be influenced by having theduring the bimodal exposure phase (Canon, 1970; Ra-Given this evidence for a role of deliberate (albeit im-perfect) attention to one modality versus another in ven-triloquism, one might ask whether deliberate tention to one location rather than another plays anysimilar role. Some recent developments in cross-modal re-searchon spatial attention may be taken to suggest sucha possibility. Although in most of the attention literature,only a single modality at a time has been considered, astudy by Driver and Spence (1994) demonstrated clearaudiovisual links in endogenous spatial attention. Theyshowed that unrelated auditory and visual tasks are com-bined less efficiently as a concurrent dual task when theauditory and visual stimuli have two spatially separategests a difficulty in directing endogenous visual and au-ditory attention to separate locations, rather than to a com-mon location. In a related study, Spence and Driver (1996)reported that endogenous shifts of visual attention tendto be accompanied by corresponding shifts of auditoryattention in the same direction, and vice versa.These findings on cross-modal attention suggest the ex-istence of an obligatory link between spatial attention inhearing and in vision. It thus seems possible that the bi-asing effects of vision on auditory localization, as foundin many cases of ventriloquism, may, in part, reflect suchattentional linkage between the two modalities. After all,real ventriloquists go to considerable lengths in their effortto get the audience to attend visually to the agitated pup-pet. Perhaps the observed influence of concurrent visualevents on auditory localization depends on a shift in at- 324BERTELSON, VROOMEN, tention toward the synchronous visual events. As Spenceand Driver (1996) found, such a shift in visual attentionwould tend to carry auditory attention along with it.for ventriloquism actually involves a second premise be-yond the demonstrated fact (Spence & Driver, 1996,1997a) that endogenous attention tends to shift togetheracross the modalities. One would also have to assume thatthe locations registered within each modality depend, inpart, on the current locus of deliberate attention withinthat modality (so that auditory localization would tend toshift along with attention toward a synchronous visualevent). Several influential models of attention do, in fact,represent stimulus location by means of activity in an at-, which represents the current focusof attention and, in so doing, also represents the perceivedlocation that would be reported for the attended stimulus(see, e.g., Cave & Wolfe, 1990; Treisman, 1988; Treisman& Gelade, 1980). Thus, given current findings on cross-modal attention and current models of how attention mayrelate to localization, the hypothesis that cross-modal linksin spatial attention might underlie ventriloquism seemsworthy of direct investigation.In addition to its potential theoretical importance, thepossible role of spatial attention in the generation of ven-triloquism also has methodological implications for thedesign of experimental tasks. A common procedure inexperiments on ventriloquism is to constrain the gaze ofsubjects via instructions to monitor a particular locationfor the possible occurrence of a prespecified visual target.For an appropriate interpretation of data obtained undersuch conditions, it is important to know whether such amonitoring requirement may, in itself, bias auditory lo-calization toward the area of visual events, owing to aninfluence from the location at which attention is directed.The Present StudyThe experiments reported here test whether the effectof synchronous visual events on the apparent location ofan auditory target may be influenced by the direction ofdeliberate attention toward, rather than away from, thesesynchronous visual events. The primary task was alwaysbursts, which, in Experiment1, could be accompanied bythe synchronous flashing of an irrelevant visual square toone side, to serve as a potential visual attractor for induc-ingventriloquism. In Experiment2, synchronous flashesappeared on both sides, to provide competing visual at-tractors, and one could be more salient than the other. Inboth experiments, the secondary task was to monitor asmall stream of visual events for occasional changes. Thepurpose of this was to manipulate the direction of delib-erate visual attention. In Experiment1, the visual moni-toring task was performed either centrally or laterally atthe location of the flashing attractor square. In Experi-ment2, it was performed at the location of one or the otherof the two lateral squares (i.e., always on one side, butnow in the presence of bilateral visual attractors). The vi-sual monitoring task involved both the deliberate focusingalso, overt gaze toward this location. If our manipulationof visual attention turned out to affect auditory localiza-tion, it would, of course, then be necessary to run furtherexperiments, in order to separate the relative contributionsof covert internal visual attention from those of overt gazefixation. In fact, this necessity did not arise, because weattention (and accompanying gaze) on auditory localiza-tion. All that finally mattered was the position and salienceof the visual flashes presented synchronously with thesounds, implying that ventriloquism is not affected by de-liberate attention but, rather, is determined by automaticsensory interactions.Unilateral Visual AttractorsIn this experiment, each trial involved the presentationof a sequence of sound bursts at one of several locations (asdetermined by stereo presentation), plus the synchronousaddition, they had to monitor for rare small changes to avisual target (catch stimulus), which was located at theIf the direction of deliberate visual attention plays a rolein ventriloquism, a larger shift of auditory localization to-wardthe flashing square should be observed with lateralattention toward that square than with central. Seventeen undergraduate students from Tilburg Uni-versity were paid a small amount for participating in the 1of the experimental study.Apparatus. The subject sat with his or her head on a chinrest ata 55-cm distance from a Commodore 1936 PC with a 20cm high 27cm wide monitor. All the visual input was displayed on this mon-itor. Two small loudspeakers were hidden behind an occluding screenon either side of the monitor. These allowed stereophonic presenta-tion of the target sound bursts. The subject indicated the directionfrom which the sound appeared to emanate by moving the indexfinger of his or her preferred hand to the corresponding locationalong a bowed rod that was fixed flat to the table top in front of thescreen. The main function of this rod was to serve as a distal bound-ary for the excursion of the pointing finger. A horizontal opaquepanel hid the responding hand from the subjectÕs sight. A graduatedscale by the rod allowed the experimenter, who watched the fingerest degree of angle.Procedure. On each trial, six 190-msec segments of 1000Hzsine wave, each separated by 750-msec silent intervals, were pre-sented simultaneously over the two loudspeakers, at a comfortablelistening level and with fade-ins and fade-outs, to avoid clicks.Three apparent locations of the sound were produced by setting therelative amplitude of the sine wave on the left and the right loud- VENTRILOQUISM AND DELIBERATE VISUAL ATTENTION325 speakers, respectively, at 100%Ð100% (middle), 110%Ð90% (left),and 90%Ð110% (right).The center of the screen was occupied by a white circle, 10mm indiameter, which came on at the beginning of each trial (500msec be-fore the onset of the first tone burst) and stayed on continuously untilthe offset of the last burst. On two thirds of the experimental trials, a15mm white square, with its center at a distance of 82.5mm(8.5¼ of visual angle) to the left or the right of the screen center, flashedagainst a dark background in exact synchrony with the six soundattractorsquare was the same throughoutthe six cycles of each trial. Thus, the flashing square provided eithera left or a right visual attractor that might influence auditory localiza-tion. On the remaining third of the experimental trials, no flashingsquare appeared. These no-square trials provided a baseline for esti-mating conflict-free auditory localization. The subject was instructedto perform the finger-pointing response for each trial only after thelast of the six tones and to return the hand immediately thereafter to anear-medial resting position. The intertrial interval was 4 sec.Separate blocks of trials were run under two different attentionconditions. The critical difference between these conditions con-cerned the location of a small visual target, which had to be moni-tored for occasional slight changes in shape. The target always oc-curred in synchrony with the tones and in the same locationthroughout the six cycles of each trial. It was a small (3 3mm)blacksquare, which on catch trials3mm) on either the fifth or the sixth cycle of the trial. The instruc-tions to the subject for these catch trials were to say ÒjaÓ (yes) andto abstain from pointing. Across different half-sessions, the sub-jects had to monitor visually at different locations. In the centraltention condition, the visual target was situated, on all the trials, inthe middle of the central circle. In the lateralthe visual target was presented in the center of the attractor squareon the two thirds of the trials that had such a square and in the mid-dle of the central circle on no-square trials. For the latter trials, thevisual monitoring task was, thus, exactly the same as that for thecorresponding trials in the central attention condition. Thus, direc-tion of attention was only a dummy variable for no-square trials,denoting just the location on which attention was focused for thosetrials in the same half-session that did have an attractor square. Theno-square trials served as baselines to assess the respective biasingeffects of left versus right squares, but played no role in determin-ing the effect of attention direction.To check that monitoring for the catch stimuli effectively focusedattention on the appropriate visual location in the experimentalstudy, a separate control studywas run on 4 subjects (who did nottake part in the main experiment). They had to monitor for the sametype of visual targets as that in the main experimentÑthat is, smallblack squares occasionally turning into black diamonds. The dif-ference from the main experiment was that, in the control study,there were always two such visual streams, one on the central whitecircle and one on a lateral white square. The aim was to test whetherthe other. If so, this would demonstrate the requirement for focusedIn this control study, one block of trials was run under each oftwo conditions, which differed as regards the location (central orThe remaining 25% of the catch stimuli were presented in the otherlocation, in a randomly intermingled sequence. The subjects in thecontrol study were told of these probabilities and were requested toconcentrate on the most frequent location for visual catch events,while attempting to detect all catch events at both locations. Theironly task was to report catch events; although sounds were presentedto make the stimulation as close as possible to that of the mainex-periment, no pointing to the location of these sounds was requested.On each block of 96 trials in the control study, half (48) were catchtrials, on which the critical change of a visual target from a squareto a diamond occurred on either the fifth or the sixth cycle. The re-maining half were no-catch trials. For the 36 catch stimuli occurringin the frequent location, the mean hit rate was 98% (96% for thecentral location and 100% for the lateral). Of the 12 catch stimuliin the infrequent location, none was ever detected by any subject.No false alarms occurred on no-catch trials.These control results show that focusing attention at one of thetwo locations (central vs. lateral), in such a way as to be able to de-tect the target events in our visual monitoring task, made detectionof similar events in the other location impossible. They thereforeimply that the monitoring task focuses visual attention on a singlelocation. It must be noted that this test of the extent to which atten-tion must be focused in our monitoring task is a conservative one,since, in the control study, the subjects were, in effect, requested tomonitor two locations, if possible, whereas in the main experimen-tal study, they were requested to monitor only one visual location.Thus, the subjects in the experimental study should, if anything,have their deliberate visual attention even more focused.Returning to the main experiment, lateral versus central atten-tion was blocked by half-sessions, with order balanced across sub-jects. Each half-session included 90 experimental trialsÑthat is, 10right) crossed with three visual displays (square left, square right,no square), all intermingled with 12 additional catch trials and pre-ceded by 3 additional warm-up trials.Because lateral and central attention conditions were blocked,the subjects knew where to focus their deliberate visual attention.For the half-session in the central condition, they had to visuallymonitor the middle of the central circle throughout. For the half-session in the lateral attention condition, they had to monitor thelateral attractor square when one was presented and the central cir-cle when there was no attractor square. In any case, since the smallvisual target for monitoring occurred at the same location through-out the six cycles of each trial, its location on the first cycle wouldalways inform the subject where to attend visually.To summarize, on each trial in the experimental study, the sub-jects pointed to the perceived location of a sound, which could bepresented at the right, at the left, or centrally. Large attractor squaresventriloquism that might influence auditory localization. At thesame time, the subjects had to monitor small visual targets for oc-casional catch events, either centrally or at the location of the largeattractor square. The control study had shown that this monitoringtask requires attention to one visual location at the expense of theother. If ventriloquism depends on the direction of deliberate visualattention, the biasing effect of an attractor square on auditory lo-calization should be more pronounced when subjects monitored forcatch events at the peripheral location of the visual attractor, ratherthan centrally.Data from 5 subjects were discarded from the analysisbecause they missed more than 3 out of the 12 catch tri-alsunder one of the two monitoring conditions (in fact,under the lateral attention condition for all of them). Forthe remaining 12 subjects, the mean detection hit rate was90% under the central attention condition and 93% underthe appropriate location with equal efficiency under bothlateral and central conditions. 326BERTELSON, VROOMEN, Figure1 shows the mean pointing responses for allcombinations of the three independent variables: soundence of a lateral square on the experimental trials resultedin strong shifts of pointing in its direction, as is shown bythe large (vertical) differences between the three pairs oflines in Figure1. By contrast, any influence of attentioncondition seemed very small and inconsistent (each solidline in Figure1 is very close to the dotted line paired withit, which represents the other attention condition). Aswas discussed earlier, the results for the no-square trials(middle two lines) are not relevant to the question of theeffect of attention direction, since, on those trials, atten-tion was always, in fact, directed to the center. Any differ-encebetween the (dummy) attention conditions on thesetrials could only have arisen because of transfer from thetask performed on the trials with an attractor square withinthe same half-session. The results for the trials with an at-tractor square are the relevant ones, and they show no ef-fect of These results were statistically assessed by a three-waywithin-subjects analysis of variance (ANOVA). The maineffects of sound location [(2,22)= 7.3, = 7.3, F(2,22)= 48.2, .001] were significant,but their interaction was not [1]. There was nomain effect of attention condition (actions with the other variables were all nonsignificanticantsound location: F(2,22)1; attention vi-sual input: F(2,22)1; attention (4,44)= 1.5, n.s.]. The important outcomeis the fact that the visual input not even approach significance. Any influence of the di-rection of deliberate visual attention on ventriloquismwould have shifted the pointing response in opposite di-rections for the trials with the square on the left and forconditions (as compared with central attention), whichshould have produced a significant interaction.In comparison with the control trials with no flashingsquare, pointing shifted on experimental trials in the di-rection of the lateral attractor square. This shift in auditorylocation occurred under both attention conditions and forall sound locations. Thus, the usual immediate biasingef-fect of synchronous but spatially discrepant visual eventson auditory localization was obtained in the present sit-was produced. The main focus of the experiment was any effect of re-quiring deliberate visual attention and the accompany-ing gaze to be directed either at the center of the displayor, instead, at the flashing lateral attractor square thatproduced the cross-modal influence on auditory local-ization. No influence was observed from the direction of Figure1. Experiment1: Mean pointing response for sound localization as a functionof sound location (along the abscissa), visual input (right flashing square, no square, orleft flashing square; depicted by upright triangles at the top of the graph, circular sym-bols in the middle, or inverted triangles at the bottom of the graph, respectively), andvisual attention condition (monitor at the central position, open symbols and dottedlines; monitor at the location of a lateral flashing square, filled symbols and solid lines).Sound localization, as measured by the pointing response, is shown along the with zero representing the center of the scale. Negative values represent pointing to-ward the left, and positive values pointing to the right. VENTRILOQUISM AND DELIBERATE VISUAL ATTENTION327 deliberate(endogenous) visual attention, suggesting thatit plays little or no role in ventriloquism. In the GeneralDiscussion section, we consider whether the flashing at-tractors might have exerted their effect by capturing ex-ogenous attention in an automatic manner, rather than bydepending on endogenous attention. For now, we notemerely that, if this were so, the results of Experiment1would imply that the deliberate direction of endogenousvisual attention cannot modulate capture of exogenous at-tention by peripheral visual events. This would run con-trary to several recent findings in the visual attention lit-erature (see, e.g., Egeth & Yantis, 1997; Yantis & Jonides,1990), which have demonstrated endogenous modulationof capture of exogenous attention. The null effect of attention direction in Experiment1certainly cannot have been due to a failure to manipulatedeliberate visual attention. The analyzed subjects effec-tively directed attention in the way requested by the instructions for the lateral versus central attention con-ditions, as is shown by their catch trial performance. More-over,the results of the control study showed that such fo-cusing of deliberate visual attention on one of the twoHowever, there may be another respect in which Ex-periment1 might have lacked sensitivity. One could arguethat the attraction of auditory localization toward the sin-gle flashing square may have been so powerful that anymore modest influence from deliberate attention may havebeen hidden. If so, it would be premature to conclude thatneverhas any effecton ventriloquism, even though Experiment1 clearly showsdeterminant of the cross-modaleffect. A more sensitive test could be provided by pre-senting flashing visual stimuli of comparable salience onboth sides at the same time and then manipulating whichside must be deliberately attended visually. In sucha sit-uation, we might now find that deliberate attention caninfluence which of two equally competitive visual attrac-torswill come to dominate auditory localization. Exper-iment2 tested this possibility.Bilateral Visual AttractorsThis experiment was run to address the concern thatthe method used in Experiment1 may have been insuf-ficiently sensitive to reveal a role for deliberate spatialvisual attention in ventriloquism, owing to dominance bypowerful bottom-up factors. We now used a situation inwhich, in some of the conditions, two identical visual at-tractors were presented together, one on either side of theauditory input. In this case, no systematic bias of auditorylocalization would be expected to manifest itself frombottom-up visual stimulation. Any subtle influence of de-become more apparent. The experimental manipulationthe left or the right attractor, using the same monitoringtask as that in Experiment1. If we found that the direction of deliberate attentionhad no effect even in such a situation, this would providea much more convincing demonstration of the irrele-vance of deliberate spatial attention to ventriloquism. Onthe other hand, one would also need to show that a situa-tion with bilateral attractors is capable of producing someventriloquism; otherwise, the sensitivity of our methodmight be questioned once again. Accordingly, we also in-troducedfurther bilateral conditions that had a strongervisual attractor on one side and a weakerside, in bottom-up stimulation terms. Radeau (1985)found that the biasing effect of a light flash on localizationsity of the light, with stronger effects from more intenselights. We expected that strength of attraction could sim-ilarly be manipulated in the present situation by varyingthe relative of the visual squares flashed on eachside. We predicted greater attraction of auditory localiza-tion toward the side of the larger square. When the squareswere equivalent in size, any influence of deliberate at-tention on ventriloquism should be revealed by an anal-ogous greater attraction to the side that was attended forthe monitoring task. On the basis of Experiment1, how-ever, we now predicted no such effect from deliberate vi-The apparatus was the same as that in Experiment1. The samesequence of six tones, with the same timing, was presented on eachtrial as before, and the primary task of the subject was again to pointto its apparent source. Only two auditory locations, one left and theother right of the apparent middle, were used this time. The visualinputs were now always bilateral, comprising two squares flashingsimultaneously with the tones on every trial, one on each side, withtheir centers again 82.5mm from the center of the screen. Each15mm) or 5mm). Bycombining these two sizes, four possible visual arrangements weregenerated: small left/big right, or SB, plus BS, SS, and BB. No-square control trials, which had been run in Experiment1, wereomitted. Such trials are not necessary for testing the main hypothe-sis concerning any influence of deliberate attention on ventrilo-quism; in Experiment1, their role had simply been to provide a base-lineof pure auditory localization. To test any influence of deliberatevisual attention on ventriloquism, all that is required is a compari-son of localization responses for a given sound, when paired withconcurrent attractor squares on both sides, with attention focusedon the left versus the right attractor square.The manipulation of deliberate visual attention involved thesame monitoring task as that in Experiment1. The same 3 3mmblack square target was presented on each trial in the center of eitherthe left or the right attractor square. On a catch trial, it became a 33mm diamond on either the fifth or the sixth cycle. Unlike Ex-periment1, which had concerned central versus lateral attentioncondition, we now contrasted left and right visual attention. More-over, whereas attention conditions had previously been blocked, at-tention to the left versus the right was now intermingled within each 328BERTELSON, VROOMEN, block. As the target for the monitoring task was again presented onthe same side throughout the six cycles of each trial, the subject knewfrom the first cycle which side required deliberate visual attention.Just as in Experiment1, the subjects had to point to the apparentlocation of the sounds on all the experimental trials. On catch tri-als, where the small target square changed to a diamond, they hadto withhold the pointing response and give a spoken response(ÒyesÓ) to indicate their detection of the diamond.Crossing the two possible sound locations (left/right), the fourpossible visual inputs (BS, SS, BB, and SB; i.e., the possible com-binations of sizes for the single squares on each side, as was ex-plained earlier), and the two visual attention requirements (moni-toring at the left vs. the right square) yielded 16 different types ofexperimental trial. Each of these was presented twice, randomly in-termingled with an additional four catch trials in each of five blocksof 36 trials each. Blocks were separated by a short pause. The ses-sion started with 16 practice trials. Since the separation between the two possible locations for visualmonitoring (now left vs. right) was twice as large in this experimentas that in Experiment1 (where it had been lateral vs. central), we canassume that visually monitoring one location again precluded attend-ing to the other, as was shown by the control study for Experiment1.Twenty-two undergraduate students from Tilburg Universitywere paid to participate in the -h session. None had taken part inExperiment1.Data from 3 subjects were discarded because they de-tected less than 75% of the catch stimuli. For the remain-ing19, mean hit rate on catch trials was 84.3%. The mean pointing responses were analyzed in a 2 (at-tentiondirection) ANOVA, with all factors being within subjects. The maineffects of sound location [(1,18)= 23.9, = 23.9, F(3,54)= 6.97, .001] were signifi-cant, but that of attention did not even approach signifi-i-F(1,18)1]. The sound location attentioninteraction was also nonsignificant [(1,18)= 1.52, n.s.],as well as the other first-order interactions (visual input is shown in Figure2 (note that the abscissa now denotesstimulation, unlike Figure1). As would be expected,the main effect of sound location was due to rightwardFigure2), as compared with leftward pointing for leftsounds(lower lines with squares). Less trivially, the maineffect of visual input appears to stem from the fact that,in comparison with the conditions with symmetrical vi-asymmetrical conditions (i.e., SB and BS) are shifted inthe direction of the bigger square (i.e., rightward and left-ward, respectively). By in the two symmetrical conditions SS and BB were notsignificantly different [1]. Their mean was usedas a baseline for assessing the significance of any shiftstoward the larger square in the asymmetrical conditions.Both shifts were significant by one-tailed Figure2. Experiment2: Mean pointing response as a function of the different bilat-eral visual input (shown along the abscissa; BS stands for big square on the left, smallsquare on the right, etc.), whether the sound was actually presented from the left (lowertwo lines in the graph, with squares) or from the right (upper two lines with diamonds),and whether visual monitoring was required on the right (filled symbols) or on the left(unfilled symbols). Sound localization, as measured by the pointing response, is shown-axis, with zero representing the center of the scale. Negative values repre-sent pointing toward the left, and positive values pointing to the right. VENTRILOQUISM AND DELIBERATE VISUAL ATTENTION329 against baseline: (18)= 3.56, .001; BS against base-(18)= justified by the fact that the direction of the shifts towardthe larger square had been predicted. As in Experiment1, there was no consistent effect of the(in Figure2, filled symbols with solid lines for attentionto the left). It is particularly noteworthy that there was ab-solutely no trend toward an effect of attention directionfor conditions SS and BB of visual input, which shouldprovide the most sensitive test for any influence of delib-erate attention on ventriloquism, since these conditionshave no bias favoring visual attraction of sound local-ization toward either side in purely bottom-up, stimulus-based terms. Thus, any top-down influence of deliberateattention to one side on ventriloquism would not have tocompete with bottom-up factors in these conditions, un-like Experiment1. Nevertheless, no effect of attention wasobserved even in this most sensitive situation.The apparent trend toward a small influence of attention(apparent at either end of the upper two lines in Figure2)is misleading. In fact, in the comparison of attend-leftwith attend-right for these conditions, the individual datawent in the direction of the overall means for only 10 ofthe 19 subjects, and the overall difference was skewed byThe most important finding of this experiment is thattask to one side or the other had absolutely no reliable ef-fect on sound localization. This was notably the case evenfor trials with symmetrical visual inputs (SS or BB),which should induce no bias toward one or the other sideon a bottom-up basis, thus leaving ample room for anyattentional effect to manifest itself. On the other hand,significant stimulus-driven visual biases on auditory lo-calization were obtained in the two asymmetrical inputconditions, always in the direction of the bigger square.This aspect of the results is important, because it showsthat the experimental situation is sufficiently sensitive tolet visual influences on sound localization manifest them-selves. At a more general level, this finding confirms ourits capacity to bias auditory location, in the same way aspreviously shown for intensity (Radeau, 1985).In passing, we note also that the biases of auditory lo-calization toward the bigger square in conditions SB andBS were much smaller than those toward a single lateralsquare, as obtained in Experiment1: about 1¼, on average,in this study against about 7.5¼ in Experiment1. It seemslikely that, with bilateral visual inputs of different sizes,the observed shift in auditory localization represents thedifference between the opposite attractions exerted by thetwo squares, rather than the total attractive power that thelarger square would have exerted in isolation. This sug-averagingor competitive resolution of at-tractive power may take place between concurrent visualevents, rather than ventriloquism being strictly a winner-As with many previous studies, in Experiment1, itwas found that the apparent location for a sequence ofsound bursts is attracted in the direction of a visual eventflashed synchronously in a discrepant location, the usualeffect. The novel finding was that the sizeof this effect was uninfluenced by directing deliberate vi-sual attention toward the location of the peripheral attrac-tor flash (rather than to the center) in a concurrent visualmonitoring task. Experiment2 showed that directing at-tention deliberately to just one of two identical flashes,presented concurrently on either side, similarly exerted noeven though such conditions of balanced bilateral stimula-tion should be maximally sensitive to any influence fromthe direction of deliberate attention. By contrast, in-creasing the size of one flash relative to the other one didproduce an attraction of auditory localization toward themore salient of two bilateral visual stimuli. This patternof results confirms bottom-up stimulus-based influenceson ventriloquist phenomena, while suggesting that delib-eratespatial attention plays little if any role.Certainly, these results are inconsistent with any hy-pothesis that visual biasing of auditory localization, asfound in ventriloquism, depends principally on where de-bleaccount prior to our study, but the present resultswould seem to disconfirm it. Our findings fit more nat-urally with the alternative notion that ventriloquist phe-nomena reflect automatic interactions between sensorycodes for location, arising at levels of representation thatpreattentive, in the sense that they are uninfluenced bythe current direction of deliberate attention in space.The notion that ventriloquism may arise preattentivelyin this particular sense has received recent support froma demonstration that, in the traditional cocktail-party sit-uation of trying to listen to just one of two competing spo-kenmessages, ventriloquism can help listeners to focuson just the target message. Reisberg (1978) had originallyshown that selective shadowing for one of two concur-rent spoken messages can be improved by adding visuallip-read information for the relevant message. On its own,this result may show only that added visual stimulationcan provide extra information about the content of the rel-evantmessage. However, Driver (1996) recently showedthat seen lip movements can have a further specificallyinfluence on selective listening. He examined shad-owing for one of two concurrent sets of spoken words, de-liveredby the same voice on a single mono loudspeaker,when visual lip-read information was now always pre-sented for the relevant message. He found that shadowingwas facilitated if the moving face of the person speakingwas presented at a discrepant location, away from the 330BERTELSON, VROOMEN, This effect is probably due to a displacement of justthe relevant sounds toward the location of the face, sincethe relevant sounds will be temporally correlated withthe seen articulatory movements, whereas the distractorsounds will not. Ventriloquism in this situation should,therefore, displace the relevant sounds away from thedistractor sounds, toward the moving face. Since this dis-placement can evidently aid selection of the targetsounds from the distractor sounds, to produce more effi-cient shadowing, it must presumably arise accords with the implications of the present study, whereventriloquism was found to be unaffected by the direc-In laboratory situations such as those of the presentexperiment or those studied by Driver (1996), ventrilo-quism leads to mislocalization of auditory sources. How-ever, in the real world, auditory sources typically respondin location to matching visual events, rather thanbeing discrepant (e.g., speech sounds do, indeed, usuallyemanate from synchronously moving lips). Hence, thevarious cross-modal interactions that we have discussedcan be viewed as reflecting mechanisms that should nor-mally be adaptive, tending to favor veridical rather thanillusory spatial perception. Given that auditory localiza-tion is generally rather inaccurate, relative to visual local-ization,it makes sense that visual location informationshould often be weighted more heavily than auditory lo-cation information, even for auditory judgments.The aftereffects found after exposure to spatially dis-crepant auditory and visual stimulation, as was discussedearlier, suggest that auditory and visual spatial represen-tations are continuously being recalibrated against eachother. This presumably serves to optimize their corre-spondence with external reality, by updating their relationwhen consistent discrepancies arise (as has similarly beenargued for the related prism-adaptation literature; seeHeld, 1965). From this perspective, it seems sensible thatventriloquist phenomena should, as we found, be fectedby deliberate spatial attention. If auditory localiza-tion could be affected simply by wherever a person choseto attend visually, the apparent location of a fixed audi-tory source would change every time the person endoge-nously shifted his or her deliberate visual attention.teractions that underlie ventriloquist phenomena (in-cluding their aftereffects) is presumably to keep internalspatial representations in as close an accord with ex-ternal reality as is possible, it would seem far more adap-tive for these cross-modal spatial interactions to be driven passively by stimulus events in a bottom-up fash-ion, rather than being susceptible to wherever a personchooses to attend. Thus, our finding that deliberate spa-influence ventriloquism makesWe should delimit the restricted sense in which we pro-pose that attention does not, and indeed should not, affectventriloquist phenomena. First, our study has dealt onlywas mentioned in the introduction, there are strong indi-cations that audiovisual spatial interactions can be modu-lated to some extent by deliberate attention to one modal-ityversus another, in contrast to what the present studyhas shown for deliberate spatial attention. Second, our study has been concerned only with liberate(or endogenous) spatial attention, directed in ac-cordance with instructions to perform the monitoringtask, and not with reflexive (exogenous) spatial attention,which may be captured by salient events in a stimulus-driven manner (see Egeth & Yantis, 1997, Spence & Dri-ver, 1997a, and Yantis & Jonides, 1990, for reviews ofthe endogenous/exogenous distinction for spatial atten-tion). It remains an interesting question whether the au-tomatic visual influence on auditory localization that wasapparent in the present study (and in many previous ex-periments on the ventriloquist effect) might be mediatedvia exogenousspatial attention capture by the visual at-tractors. The visual attractors in ventriloquist studies typicallyhave abrupt onsets, as required to produce exogenous at-tention capture, according to Jonides and Yantis (1988).However, there are at least three difficulties for any in-terpretation of ventriloquism in terms of such exogenousattention capture alone. First, visual attractors that areasynchronouswith sounds do not bias auditory localiza-tion (Bertelson &Aschersleben, 1998; Klemm, 1909;Radeau & Bertelson, 1987; Thomas, 1941), yet shouldinduce exogenous capture equivalent to that produced byvisual events that are synchronous with the sounds. Sec-ond, the present study has shown that the full ventrilo-quist effect occurs even with deliberate attention di-awayfrom the visual attractor, a situation underwhich Yantis and Jonides (1990) have shown that exoge-nous capture no longer occurs. Thus, any suggestion thatventriloquism depends on capture of exogenous attentionseems rather unlikely, givenexisting evidence. Finally,such an account would rely on an entirely conjectural in-fluence from exogenous visual attention on auditory local-ization. Accordingly, for now, weprefer to stick closer towhat was actually observed and to conclude merely thatstimulus-driven interactions between the visual and theauditory events produce ventriloquism.To summarize, the present experiments confirm ear-lier demonstrations that the visual biasing of auditory lo-calization is strongly affected by stimulus factors, suchas the relative location of auditory and visual events (Ex-periment1) and the relative salience of concurrent visualevents (Experiment2), but they also show, for the firstaffected by the direc-tion of deliberate visual attention. We argue that this find-ingmakes good functional sense. If deliberate changes inthe direction of visual spatial attention were able to shiftauditory localization for a fixed source in the correspond-ingdirection, the senses might continuously recalibrateagainst each other, for apparent discrepancies that were,in fact, caused only by where the person had chosen to VENTRILOQUISM AND DELIBERATE VISUAL ATTENTION331 dogenous spatial attention, we suggest that ventriloquistphenomena are largely driven, in an automatic fashion, bypreattentivespecific sense outlined above. This agrees with DriverÕs(1996) recent claim that ventriloquism may serve a usefulsegmentation function in noisy environments, regardlessof where deliberate attention is currently directed. Finally,we acknowledge that ventriloquism may be subject to adifferent and form of attentional controlÑnamely, a limited ability to weight the modalities differ-entially when making localization judgments. However,deliberate spatial visual attention to different positionsseems to play little or no role in ventriloquism.Bermant, R.I., & Welch, R.B.(1976). The effect of degree of visualÐauditory stimulus separation and eye position upon the spatial in-Perceptual & Motor SkillBertelson, P. (1994). The cognitive architecture behind auditoryÐvisualinteraction in scene analysis and speech identification. Current Psy-chology of CognitionBertelson, P.(1998). Starting from the ventriloquist: The perceptionof multimodal events. In M.Sabourin, F.I.M. Craik, & M.RobertAdvances in psychological science: Vol.1. Biological andcognitive aspects(pp.419-439). Hove, U.K.: Psychology Press.Bertelson, P., & Aschersleben, G. perceived auditory location. Psychonomic Bulletin & ReviewBertelson, P., & Radeau, M.(1976). Ventriloquism, sensory inter-action, and response bias: Remarks on the paper by Choe, Welch, Gil-Perception & PsychophysicsBertelson, P., & Radeau, M. fusion with auditoryÐvisual spatial discordance. Perception & Psycho-physicsBertelson, P., & Radeau, M. auditoryÐvisual conflict: Have top-down influences been overesti-mated here also? Paper presented at the 2nd Meeting of the EuropeanSociety for Cognitive Psychology, Madrid.Bertelson, P., Vroomen, J., Wiegeraad, G., & de Gelder,B. September). Exploring the relation between McGurk interference andventriloquism. In International Congress on Spoken Language Pro-(pp.559-562). Canon, L.K. (1970). Intermodality inconsistency of input and directedattention as determinants of the nature of adaptation. Journal of Ex-perimental PsychologyCave, K.R., & Wolfe, J.M. Cognitive PsychologyChoe, C.S., Welch, R.B., Gilford, R.M., & Juola, J.F.Òventriloquist effectÓ: Visual dominance or response bias? Percep-tion & Psychophysics(1996). Enhancement of selective listening by illusory mis-Nature(1994). Cross-modal synergies in attention.In C.Umiltˆ & M.Moscovitch (Eds.), Attention and performanceXXV: Conscious and nonconscious information processing(pp.311-Egeth, H.E., & Yantis, S.(1997). Visual attention: Control, repre-Annual Review of PsychologyHeld, R. Scientific Ameri-Jack, C.E., & Thurlow, W.R. (1973). Effects of degree of visual as-sociation and angle of displacement on the ÒventriloquismÓ effect.Perceptual & Motor SkillsJackson, C.V. (1953). Visual factors in auditory localization. QuarterlyJournal of Experimental PsychologyJonides, J., & Yantis, S. (1988). Uniqueness of abrupt visual onset inPerception & Psychophysics(1909). Localisation von SinneneindrŸcken bei disparateen bei disparateditional stimulation]. Psychologische Studien(1989). Cognitive impenetrability of perception. ioural & Brain SciencesPashler, H. The psychology of attentionPick, H.L., Jr., Warren, D.H., & Hay, J.C. (1969). Sensory conflictPerception & PsychophysicsPosner, M.I.Quarterly Journal of Ex-perimental PsychologyPylyshyn, Z.W. (1999). Is vision continuous with cognition? The casefor cognitive impenetrability of visual perception. Behavioural &Brain SciencesRadeau, M. dÕune discordance entre la vision et lÕaudition [Adaptation to prismaticdisplacement on the basis of discordance between vision and audition].LÕAnnŽe PsychologiqueRadeau, M.(1985). Signal intensity, task context, and auditoryÐvisualPerceptionRadeau, M.(1992). Cognitive impenetrability in auditoryÐvisual inter-action. In J.Alegria, D.Holender, J.Morais, & M. Radeau (Eds.), alytic approaches to human cognition (pp.41-55). Amsterdam: El-sevier.Radeau, M.(1994). AuditoryÐvisual interaction and modularity. rent Psychology of CognitionRadeau, M., & Bertelson, P. (1974). The after-effects of ventrilo-Quarterly Journal of Experimental PsychologyRadeau, M., & Bertelson, P.(1976). The effect of a textured visualfield on modality dominance in a ventriloquism situation. Perception& PsychophysicsRadeau, M., & Bertelson, P.(1977). Adaptation to auditoryÐvisualdiscordance and ventriloquism in semirealistic situations. Perception& PsychophysicsRadeau, M., & Bertelson, P. (1987). AuditoryÐvisual interaction andthe timing of inputs: Thomas (1941) revisited. Psychological Re-search(1978). Looking where you listen: Visual cues and audi-tory attention. Acta Psychologica(1996). Audiovisual links in endogenouscovert spatial attention. Journal of Experimental Psychology: HumanPerception & Performance(1997a). Audiovisual links in exogenouscovert spatial orientation. Perception & Psychophysics(1997b). On measuring selective attention to anexpected sensory modality. Perception & PsychophysicsThomas, G.J.Journal of Experimental PsychologyTreisman, A.(1988). Features and objects: The fourteenth BartlettQuarterly Journal of Experimental PsychologyTreisman, A., & Gelade, G.(1980). A feature-integration theory ofCognitive PsychologyWarren, D.H., Welch, R.B., & McCarthy, T.J. visualÐauditory ÒcompellingnessÓ in the ventriloquism effect: Impli-cations for transitivity among the spatial senses. Perception &PsychophysicsWelch, R.B. (1994). The dissection of intersensory bias: WeightingCurrent Psychology of Cognition 332BERTELSON, VROOMEN, Welch, R.B., & Warren, D.H. to intersensory discrepancy. Psychological BulletinWelch, R.B., & Warren, D.H.(1984). Intersensory interactions. InK.R. Boff, L.Kaufman, & J.P. Thomas (Eds.), ception and human performance: Vol.1. Sensory processes and per-(chap.25, pp.1-36). New York: Wiley. Witkin, H.A., Wapner, S., & Leventhal, T.tion with conflicting visual and auditory cues. Journal of Experi-mental PsychologyYantis, S., & Jonides, J. (1990). Abrupt visual onsets and selective at-tention: Voluntary versus automatic allocation. Journal of Experi-mental Psychology: Human Perception & PerformanceNOTES1. The term is used here, as it has been by other authors, in apurely descriptive way, to designate an observed influence. The use car-ries no assumption regarding the processing level at which the effectarises and, in particular, no implication as to whether responseare involved, which we will discuss later.2. In another experiment in the same study (Experiment3), reports ofperceptual fusionwere enhanced by realism. On-line effects may bemore susceptible to cognitive influences than are aftereffects, as wehave previously suggested (Bertelson, 1998; Bertelson & Aschersleben,1998; Radeau & Bertelson, 1977). (Manuscript received August 21, 1997;revision accepted for publication October 20, 1998.)