Imagery slides - PowerPoint Presentation

Slide1

Imagery slidesSlide2

Imagery and Memory

Memory Examples: Dual

Code Theory

To recall Y you must first recall X

Windows, doorknob, glasses, other facial features,

global-to-local

But: Something like the same thing happens in recall of alphabet letters and many other memorized lists

Imageability

rating

are more

effective than frequency

of occurrence or

frequency of

co-occurrence in paired-associates learning.Slide3

V

ision

is

clearly involved when images are superimposed onto vision

Many experiments show that when you project an image onto a display the image acts very much like a superimposed display

Shepard & Podgorny (paper folding task…)

Interference effects (Brooks)

Controvercial

Perky

effect:

Perception

or

response bias

?Slide4

Project an image onto a perceived formSlide5

Brooks’ spatial interference study

Respond by pointing to symbols in a table or by saying the words

left

or

rightSlide6

Perception or attention effects?

Many impressive imagery effects can be plausibly attributed to attention

Bisiach

widely-cited finding on visual neglect

Bartolomeo

, P., &

Chokron, S. (2002). Orienting of attention in left unilateral neglect.

Neuroscience and Biobehavioral Reviews, 26(2), 217-234.Dulin, D., Hatwell, Y., Pylyshyn, Z. W., & Chokron, S. (2008). Effects of peripheral and central visual impairment on mental imagery capacity. Neuroscience and Biobehavioral Reviews, 32(8), 1396-1408.Does neglect require vision?Chokron, S., Colliot, P., & Bartolomeo, P. (2004). The role of vision in spatial representations.

Cortex, 40, 281-290.Slide7

We can to some extent control our attended region

Farah, M. J. (1989). Mechanisms of imagery-perception interaction.

Journal of Experimental Psychology: Human Perception and Performance, 15

, 203-211.

Is an image being projected onto a percept, or just a selective attention?Slide8

Shepard & Podgorny experiment

Both when the displays are seen and when the F is imagined, RT to detect whether the dot was on the F is fastest when the dot is at the vertex of the F, then when on an arm of the F, then when far away from the F – and slowest when one square off the F.Slide9

Similarities between perception of visual scenes and ‘perception’ of mental images

Judgments from mental images

Shape comparisons (of states: Shepard & Metzler)

Size

comparisons

(Weber fraction or ratio

effect)What do they tell us about the format of images?

But this applies to nonvisual properties (e.g., price, taste)Slide10

More demonstrations of the relation between vision, imagery (and later action)

Images constructed from descriptions

The D-J example(s)

Perception or inference/guessing

But there are even more persuasive counterexamples we will see later

The two-parallelogram example

Amodal completionReconstruals

: SlezakSlide11

Dynamic imagery

Imagining actions: Paper FoldingSlide12

Mental rotation

Time to judge whether (a)-(b) or (b)-(c) are the same except for

orientation. Time

increases linearly with the angle between them

(Shepard & Metzler, 1971) Slide13

What do you do to judge whether these two figures are the same shape?

When you make it rotate in your mind, does it seem to retain its rigid 3D shape without re-computing it?

Is this how the process looked to you?Slide14

Rotation is neither holistic nor impenetrableSlide15

Mental rotation – the real story

In

mental rotation

the phenomenology motivates the theory of “rotation” – but what the data

actually show

is that,

Mental rotation is only found when the comparison figures are enantiomorphs or if the difference between figure pairs can only be expressed in figure-centric coordinates eg. they are 3D mirror-imagesNo rotation occurs if the figures have landmarks that can be used to identify the relations among their parts.Records of eye movements show that mental rotation is done incrementally: It is not a holistic rotation as often reported. If fact even the phenomenology is not of a smooth continuous rotation.

The “rate of rotation” depends on the conceptual complexity of both the figure and comparison task so that, at least, is not a result of the architecture (Pylyshyn, 1979). There are even demonstrations that it depends on how the subject interprets the figure (Kosslyn, 1994).Slide16

Mental Scanning

Hundreds of experiments have now been done demonstrating that it takes longer to scan attention between places that are further apart in the imagined scene. In fact the relation is linear between time and distance.

These have been reviewed and described in:

Denis, M., &

Kosslyn

, S. M. (1999). Scanning visual mental images:

A window on the mind

. Cahiers de Psychologie Cognitive / Current Psychology of Cognition, 18(4), 409-465.Slide17

Studies of mental scanning

Does it show that images

have

metrical space?

Does this show that images are spatial, or

have

spatial properties, or that they “preserve metrical spatial properties”?

(Kosslyn, S. M., T. M. Ball, et al. (1978). "Visual images preserve metric spatial information: Evidence from studies of image scanning." Journal of Experimental Psychology: Human Perception and Performance 4: 46-60. Slide18

The idea of images being in some sense

spatial

is an interesting and important claim

I will discuss this claim at some length later because it reveals a deep and all-consuming error that runs through all imagery theorizing – by psychologists, neuroscientists and philosophers.

This is in addition to the errors I discussed earlier: The idea that subjects understand the task of imagining something to be the task of pretending they are seeing it, and the idea that certain properties of the world are properties of the image (the intentional fallacy)Slide19

Constructing an image

What determines what the image is like when it is constructed from memory or from knowledge?

After constructing an image can you see novel aspects of the imagined situation?

ExamplesSlide20

Imagine seeing these events unfolding…

You hit a baseball. What shape trajectory does it trace? It is coming towards you: Where would you run to catch it? If you have ever played baseball you would have a great deal of “tacit knowledge” of what to do in such (well studied) cases.

You drop a rubber ball on the pavement. Tap a button every time it hits the ground and bounces.

Plot

height

vs

time.Drop a heavy steel ball at the same time as you drop a light ball (a tennis ball), e.g., from the leaning tower of Pisa. Indicate when they hit the ground. Repeat for different heights.Take a clear glass containing a colored liquid. Tilt it 45º to the left (counter-clockwise). What is the orientation of the liquid?

Examples to probe your intuition and your tacit knowledge

Time since drop

height

What is responsible for the

pattern shown here?Slide21

What color do you see when two color filters overlap?

?Slide22

Where would the water go if you poured it over a full beaker of sugar?

Is there conservation of volume in your image? If not, why not?Slide23

Seeing Mental Images

Do images have size?

Can we say that one image is larger than another?

If so, what properties do we expect the smaller/larger image to have?Slide24

Do mental images have size?

Imagine a very small mouse. Can you see its whiskers?

Now imagine a huge mouse. Can you see its whiskers?Slide25
Slide26

Do this imagery exercise:

Imagine a parallelogram like this one

Now imagine an identical parallelogram directly below this one

Connect each corner of the top parallelogram with the corresponding corner of the bottom parallelogram

What do you see when you imagine the connections?

Did the imagined shape look (and change) like the one you see now?Slide27

Slezak figures

Pick one (or two) of these animals and memorize what they look like. Now rotate it in your mind by 90 degrees clockwise and see what it looks like.Slide28

Slezak

figures rotated 90oSlide29

Space

P 29Slide30

Images and the representation of

s

patial

p

roperties

We need to understand what it could mean for a representation to be

spatial

.At the very least it must mean that there are constraints placed on the form of the representation that do not apply when the representation is not spatial.Slide31

Studies of mental scanning

Does it show that images

have

metrical space?

Does this show that images are spatial or

have

spatial properties or that they “preserve metrical spatial properties”?

(Kosslyn, S. M., T. M. Ball, et al. (1978). "Visual images preserve metric spatial information: Evidence from studies of image scanning." Journal of Experimental Psychology: Human Perception and Performance 4: 46-60. Slide32

The idea that images are in some sense

spatial

is an interesting and important claim

I will return to this claim later because it reveals a deep and ubiquitous error that runs through most (all?) imagery theorizing – by psychologists, neuroscientists and philosophers. This is the error of mistaking

descriptive adequacy

with

explanatory adequacy. Let’s call this conflating, the missing constraint error.This is in addition to the two errors I discussed earlier: Ignoring the fact that the task of imagining something is actually the task of pretending you are seeing it, and The mistaken assumption that certain properties of the world are properties of the image (the intentional fallacy)Slide33

Connecting Images

and Motor actions

Images and visual-motor

phenomena

S-R

Compatibility / Simon

effect

Finke’s imagined wedge-prism goggles Harry’s subitizing-by-pointingSlide34

Another chapter in the imagery debate:

The i

nteraction of images with vision and motor control

One of the properties of mental images that makes them appear spatial is that they connect in certain ways not only with vision, but also with the motor system:

We can point to things in our image

!

We can “project” our images onto perceived space – even space perceived in different modalities. I believe that this observation is the key to understanding the spatial character of images.

This projection does not require a picture to be projected, only the location of a small number of features. Over the past few decades I have been studying a mechanism called a visual index, or a FINST, that is well suited for this task.

Important idea! Slide35

Both vision and visual imagery have some connection to the motor system

There are a number of experiments showing the close connection between images and motor control

*

You can get Stimulus-Response compatibility effects between the location of a stimulus in space and the location of the response button in

space,

Ronald Finke showed that you could get adaptation with the position of the misperceived hand that was similar to adaptation to displacing prism goggles,

Both these findings provide support for the view that the spatial character of images comes from

something being projected onto a concurrently perceived scene and then functioning much as objects of perception.This is the main new idea in Chapter 5 of Things & Places)

an imageimagined Slide36

S-R Compatibility effect with a visual display:

The Simon effect: It is faster to make a response in the direction of an attended objects than in another direction

Response for

A

is faster when YES in on the left in these displaysSlide37

S-R Compatibility effect with an imagined display

RT is faster when the

A

is recalled (imagined)

as being on the left

The same RT pattern occurs for a recalled display as for a perceived oneSlide38

Recall the studies

of mental scanning…

But

the way we compute the time it takes to scan across an image is by imagining something moving across the

real perceived

display. Without

this display

, we could not

use

our time-to-collision computation to compute the time to cross

various distances on the image because there are no actual distances on the image! (Pylyshyn & Cohen, 1999)



We showed that t

he image scanning effect is Cognitively Penetrable

Does this result show that images have spatial properties?

Does this result show that images have metrical properties?Slide39

Using a concurrently perceived room to anchor FINSTs tagged with map labelsSlide40

The Spatial character of images

What does it mean to say that images are spatial?

It means that certain constraints hold among spatial measures (e.g., axioms of geometry and measure theory, such as

triangle inequality, symmetry of distances, Euclidean axioms, Pythagoras’ theorem

…}

That

certain

constraints hold among “distances”, that certain relations can be defined among these distances (e.g., ‘between’, ‘farther than’), that Newtonian Physics holds between the terms that are used in explanations (e.g., distances and time).That mental images and motor control interact with one another to some degree – so you can “point to” objects in your image.Certain visual-motor ‘reflexes’ are automatic or preconceptual

 They are computed within the encapsulated Visual Module Preconceptual motor control is not sensitive to visual illusions, relative to control that is computed by the cognitive (‘seeing as’) system.Slide41
Evidence for a literal spatial display in the brain

I will discuss the proposal that V1 is the imagery display in the brain. But since the conclusion will be that it is not, let’s look at other options.

The problem is to explain such phenomena as the scanning effect or the size effect without assuming a physical display

.

The main alternative to a ‘spatial display’ is something called a

‘functional space’

. This proposal was introduced by

Kosslyn

in his characterization of the ‘depictive’ nature of the image representation.Slide42

Mental images as “depictive”

representations

“A depictive representation is a

type of

picture



,

which specifies the locations and values of configurations of points in a space.The space in which the points appear need not be physical but can be like an array in a computer, which specifies spatial relations purely functionally. That is, the physical locations in the computer of each point in an array are not themselves arranged in an array; it is only by virtue of how this information is “read” and processed that it comes to function as if it were arranged into an array….Depictive representations convey meaning

via their resemblance to an object.When a depictive representation is used, not only is the shape of the represented parts immediately available to appropriate processes , but so is the shape of the empty space … [and] one cannot represent a shape in a depictive representation without also specifying a size and orientation

….”Slide43

Form

vs

Content of images

As in earlier discussion, one must be careful in distinguishing

form

from

content

. We know that there is a difference between the content of images and the content of other (nonimaginal) thought: Images concern sensory appearances while ‘propositions’ can express most* other contents.In attributing a special form of representation to images one should ask whether some symbolic system (e.g., sentences of LOT) would not do. Simplicity (Occam’s Razor

) would then prefer a single format over two, especially if the one format is essential for representing thoughts and inferences [Fodor, J. A. and Z. W. Pylyshyn (1988). "Connectionism and cognitive architecture: A critical analysis." Cognition 28: 3-71.]The most promising contents that might require different forms of representation are those that essentially represent magnitudes. Of the magnitudes most often associated with images are spatial ones.

There has been a long-standing debate in Artificial Intelligence concerning the advantages of logical formats vs other symbol systems vs something completely difference (procedure).Slide44

What is assumed in imagist explanations of mental scanning?

In actual vision, it takes longer to scan a longer distance because real distance, real motion, and real time is involved, therefore this equation holds due to natural law:

Time

=

distance

speed But what ensures that a corresponding relation holds in an image? The obvious answer is: Because the image is laid out in space!But what if that option is closed for empirical reasons?Imagists appeal to a “Functional Space” which they liken to a matrix data structure in which some cells are adjacent to other cells, some are closer and others further away, and to move from one to another it is natural that you pass through intermediate cellsQuestion: What makes these sorts of properties “natural” in a matrix data structure?

Slide45

Thou shalt not cheat

There is no natural law that requires the representations of time, distance and speed to be related according to the motion equation. You could just as easily imagine an object moving instantly or with constant acceleration or with any motion relation you like, since it is your image!

There are two possible reason why the observed relation

Actual Time

=

Representation of distance

Representation of speed typically holds in an image-scanning task:Because subjects have tacit knowledge that this is what would happen if they viewed a real display, orBecause the matrix is taken to be a simulation of a real physical display, as it often is in computer science.

Notice that in the second case the explanation for the Reaction Time comes from the simulated real display and not from the matrix.Slide46

The

missing

constraint in appeals to “space”

in both scanning and mental rotation

What is assumed about the format or architecture of the mental representation in the examples of mental rotation?

According to philosopher Jesse

Prinz

(2002) p 118,“If visual-image rotation uses a spatial medium of the kind Kosslyn envisions, then images must traverse intermediate positions when they rotate from one position to another. The propositional [i.e., symbolic] system can be designed to represent intermediate positions during rotation, but that is not obligatory.”

This is a very important observation, but it is incomplete. One still needs to answer the question: What makes it obligatory that the object must ‘pass through intermediate positions’ when rotating in ‘functional space’, and what constitutes an ‘intermediate position’? These terms apply to the represented world, not to the representation!Slide47

The important distinction between architecture and represented

content

It is only obligatory that a certain pattern must occur if the pattern is caused by fixed properties of the architecture as opposed to being due to properties of what is represented (i.e., what the observer

tacitly

knows about the behavior of what is represented)

If it is obligatory only because the theorist says it is,

score that as a free empirical parameter that any theory can assume.This failure of image theories is quite general – all picture theories suffer from the same lack of principled constraints.Slide48

The important distinction between

descriptive and explanatory adequacy

It is important to recognize that if

we allow one theory to stipulate what is obligatory without there being a principle that mandates it, then any other theory can stipulate the same thing.

Such a theories are unconstrained so they can fit any possible observation – i.e., they are able to

describe

anything but explain nothing. A theory that does not explain why some pattern is obligatory can still be useful the way an organized catalog is useful. It may even list the features according to which it is organized. But it does not give an account of why it is organized that way rather than some other way. To do that it needs to appeal to something constant such as a law of nature or a fixed property of the architecture.Slide49

The important distinction between

descriptive and explanatory adequacy

We have come back to the distinction between architecture- and representation-governed process. Consider the parallel between the apparent obligatory nature of mental rotation and the apparent obligatory nature of the pattern of dots and dashes exhibited by the code box (double spike occurs before single spike except after a long-short-long-short pattern).

Here is another way to look at what it means to

be

obligatory

. Suppose psychological experiments show that reaction time is a linear function of number of items being processed. As with the code box example, what we don’t know is whether this pattern is fortuitous or principled. There are several ways in which we could show that it is principled:

If it can be derived from a basic law of nature (very rarely happens). Note that this is about the pattern of the representation, not the pattern of the world imagined.If it follows from independently motivated assumptions about the architectureIf it holds for other

relevant inputs (where being relevant is theory-dependent)Slide50

How are these ‘obligatory’ constraints realized?

Image properties, such as

size

and

rigidity

are assumed to be inherent in the architecture

(e.g., of the ‘display’)That raises the question of what kind of architecture could possibly enforce rigidity of shape? Notice that there is nothing about a spatial display, let alone a functional space, that makes it obligatory that shape be rigidly maintained as orientation is changed.Such rigidity could not be a necessary property of

the architecture of an image system because we can easily imagine that rigidity does not hold (e.g. imagine a rotating snake!).There is also evidence that ‘mental rotation’ is incremental, not holistic, and the speed of rotation depends on the conceptual complexity of the shape and the comparison task.Slide51

Review: Why do theories that appeal to a functional space not explain imagery behavior

They fall prey to one of the following errors:

The intentional fallacy – confounding properties of the representation with properties of the represented

Task demands – neglecting the fact that subjects are being asked to pretend that they are seeing somethingSlide52

What makes some properties seem “natural” in a matrix but not so natural in a symbolic data structure?

A matrix is generally viewed as a two-dimensional structure in which specifying the x and y values (rows and columns) specifies the location of any cell. But that’s just the way it is conventionally viewed. Rows, columns and cells are not actually

spatial

locations.

In a computer there is no requirement that in getting from one cell to another one

must

pass through any other specified cells nor is there any requirement that there be empty cells between any pairs of cells.

Slide53

What makes some properties “natural” in a matrix while not so natural in a symbolic data structure?

The main reason it is natural to view a matrix as having spatial constraints is that one is tacitly assuming that it

represents

some space. Then it is the

represented space

that has the constraints, not the matrix.

Notice the subtle

succumbing to the intentional fallacy again!Any constraints that the functional space exhibits are constraints extrinsic to the format. Such constraints reside in the external world which the ‘functional space’ represents.

But such extrinsic constraints can be added to any model of scanning, including a propositional one.

Slide54

What warrants the ‘obligatory’ constraint?

But it is no more

obligatory

that the relation between distance, speed and time hold in functional space than in a symbolic (propositional) representation. There is no natural law or principle that requires it. You

could

imagine an object moving instantly or according to any motion relation you like, and the functional space would then comply with that motion since it has no constraints of its own.

So why does it seem natural for imagined moving objects to traverse a ‘functional space’ than a sequence of symbolic representations of locations?There are at least two reasons why a ‘functional space’ might seem more natural than a symbolic representation of space, and both depend on (1) subjective experience and (2) the intentional fallacy.Slide55

Where does the

obligatory constraint come from?

There are at least two reasons why the following equation holds in the mental image scanning task, even though, unlike in the real vision case, it does not follow from a natural law.

Actual Time

=

Representation of distance Representation of speedBecause subjects have tacit knowledge that this is what would happen if they viewed a real display, and they understand the task to be one of reproducing properties of this viewing, orBecause the matrix is taken to be a simulation of real space. In that case the reason that the equation holds is that it is supposed to be simulating real space and the equation holds in real space.

In that case it is not something about the form of the representation that provides the principled constraint, it’s the fact that it is supposed to be simulating real space which is where the obligation comes from. But the same thing can be done for any form of representation.Slide56

Real and

functional space

What is assumed by ‘picture’ accounts of mental imagery experiments, including those involving image scanning, image size and image rotation, is that images have the properties of a real spatial display as viewed by the mind’s eye. This is what provides a principled explanation.

But this explanation carries a number of assumptions, including that images are 2D patterns laid out in real space (presumably on visual cortex). Because the evidence does not support this assumption,

pictorialists

appeal to a “functional” space.

What is a functional space and how does it explain the scanning or image size findings?Slide57

What is “functional space”?

Because ‘functional space’ is cited by almost every imagery theorist, it deserves some attention.

The main value of a functional space is that it has whatever properties we want to assume for it – i.e., it can be made to fit any data. We

stipulate

that it takes longer to scan greater ‘image distances.’ The law relating distance, time and speed does not apply to image distance so we assume it.

For that reason the functional space assumption has no advantage over any other assumption about how space is represented; the properties we assign to functional space can be assigned to any theory. So the concept of functional space does no explanatory work.

The assumption that functional space must have certain obligatory (as Prinz put it) spatial properties, relies on one of several mistakes which result in these properties seeming more “natural” in a functional space.Slide58

Why is it ‘natural’ to assume that functional space is like real space?

There are several reasons why a functional space, such as a matrix data structure, appears to have natural spatial properties (e.g., distances, size, empty places):

Because when we think of functional space, such as a matrix, we think of how we usually interpret it.

A matrix does not intrinsically have distance, empty places, direction or any other such property, except in the mind of the person who draws it or uses it!

Moving from one cell to another does not require passing through intermediate cells unless we stipulate that it does. The same goes for the concept of ‘intermediate cell’ itself.Slide59

Why is it ‘natural’ to assume that functional space is like real space?

Because when we think of a functional space, such as a matrix, we think of it as being a way of

simulating

real space in the model – making it more convenient to build the model which otherwise would require special hardware

This is why we think of some cells as being ‘between’ others and some being farther away. This makes properties like distances seem natural because we interpret the matrix as simulating real space.

In that case we are not appealing to a functional space in explaining the scanning effect, the size effect, etc. The explanatory force of the explanation comes from the real space that we are simulating.

This is just another way of assuming a real space (in the brain) where representations of objects are located in neural space

All the reasons why the assumption of real brain space cannot be sustained in explanations of mental imagery phenomena apply to this version of ‘functional space.’Slide60

Why is it ‘natural’ to assume that functional space is like real space?

Because what we really want to claim is that images are displayed on a real spatial surface – a blackboard. But to model this we would need to build a hardware display.

{An easier way to do this is simply to claim explicitly that there is a display or even simulate one using software

(such as

Kosslyn

, et al. (1979) claim to have done

*)}.This allows us to view some cells as being ‘between’ others and some being farther away. This makes properties like distances seem natural because we interpret the matrix as simulating or standing in for a real spatial display board or screen.In that case we are not appealing to a functional space in explaining the scanning effect, the size effect, etc. The explanatory force of the explanation comes from the real space that we are claiming and simulating. This is just another way of assuming a real space (in the brain) where representations of objects are located in neural space.Slide61

Functional space and explanatory power

There is a notion of explanatory power that needs to be kept in mind. It is best illustrated in terms of models that contain empirical parameters, as in fitting a polynomial curve to data.

The general fact about fitting a model to data is that the fewer parameters that need to be estimated from the data to be fitted, the more powerful the explanation. The most powerful explanation is one that does not have to use the to-be-fitted data to tune the model.

In terms of the current example of explaining results of experiments involving mental imagery, appealing to a “functional space” leaves open an indeterminate number of empirical parameters, so it provides a very weak (or vacuous) explanation.

A literal (brain) space, on the other hand, is highly constrained since it

must

conform to Euclidean axioms and Newtonian physics – otherwise it would not be the space of natural science. But that kind of space implies that images are displayed on a surface in the brain and while that is a logical possibility it is not

an empirical oneSlide62

Explanation and Description

Another way to look at what is going on is to think about the difference between a description and an explanation. The two ways of characterizing a set of phenomena appear similar – they both speak of how things are and how they change (think of the

Code Box

example).

But a description of a system’s behavior can apply to many different types of system with different mechanisms and different causal properties. And the same mechanisms can also produce very different behaviors under different circumstances. Although a general statement of what constitutes scientific explanation and how it differs from description has a long and controversial history, the simple

Code Box

example will suffice to suggest the distinction I have in mind. Slide63

Cognitive Penetrability again

A description states the observed generalizations (

the

observed patterns of

behavior).

The explanation goes beyond this. The

difference is related to the question of how mutable a set of generalizations are and what types of effects can lead to changes in these generalizations.

Causal accounts tend to have a longer time scale and when they change they tend to change according to different sorts of principles than those that describe the patterns.*Notice that we have come back to the criterion of cognitive penetrability. According to this way of looking at the question of explanatory adequacy, an theory meets the criteria of explanatory adequacy if it describes the architecture of the system and its operation. Slide64

A note about time scales and types of changes

“Causal

accounts tend to have a longer time scale and when they change they tend to change according to different sorts of principles than those that describe the patterns

.”

Consider the Code Box example.

Changes that are not architectural tend to occur rapidly – different patterns are observed simply because different topics or words or even languages might be transmitted.

Changes that are architectural require altering which letters or other symbols are transmitted (e.g., they may be numerals) or changing whether the outputs consist of short and long pulses that are interpretable as Morse Code. They require what we might think of as “rewiring”.Slide65
Slide66

A different way of approaching the question of spatial representation

I offer a provisional proposal that preserves some of the advantages of the global spatial display, but assumes that the relevant spatial properties are

in the perceived world

and can be accessed if we have the right

access mechanisms

for

selecting and indexing objects in the perceived worldLet’s call this the Index Projection Hypothesis because it suggests that mental objects are somehow projected onto and associated with perceived objects in real spaceBut this proposal is very different from image-projection because only a few object-labels are projected – not the rich visual properties suggested by the phenomenologySlide67

The

Image Projection

H

ypothesis

This

projection hypothesis

relies on the spatial locations of objects in the concurrently perceived world to meet the conditions outlined earlier. It rests on two assumptions:We have a system of “pointers” (viz, the FINST perceptual index mechanism to be described) by which a small number (n≤4) of objects in the world can be selected and indexed. Indexes provide demonstrative references to individual targets qua individuals, that keep referring to these objects despite changes in their location or any other properties.

When we perceive a scene that contains indexed objects, our perceptual system is able to treat those objects as though they were assigned unique labels. Thus our perceptual system is able to detect configurational properties among the indexed objects.Slide68

The

index

projection

hypothesis

(2)

The hypothesis claims that the subjective impression that we have access to a panorama of detailed high-resolution perceptual information is illusory. What we have access to is only information about selected or indexed objects.We have the potential to obtain other information from more of the scene through the use of our system of perceptual indexes. This is the basic insight expressed in the “world as external memory” slogan or the “situated cognition” approach.

In reasoning using mental images we may assign indexes to perceived objects based on our memory of (or our assumptions about) where certain ‘mental’ objects are located But notice that the memory representation is itself not used in spatial reasoning and therefore need not meet the spatial constraints listed earlier – it can be in some general LOTSlide69

Examples of the projection hypothesis

To illustrate how the projection hypothesis works, first consider index-based projection in the visual modality, where indexes can convert some apparently mental-space phenomena into perceived-space phenomena (more on the non-visual case later)

Examples from some ‘mental imagery” experiments

Mental scanning (Kosslyn, 1973)

Mental image superposition (Podgorny& Shepard, 1978)

Visual-motor adaptation (Finke, 1979)

S-R compatibility to imagined locations (Tlauka, 1998)Slide70

Studies of mental scanning

Often cited to suggest that spatial representations

are literally spatial and

have

metrical properties

Distance on image

Time to “see” feature on imageSlide71

Brain image or index-based projection?

A way to do this task:

Associate places on the memorized map with objects located in the same relative locations in the world that you perceive (e.g., the room you are in)

Move your attention or gaze from one place to another as they are namedSlide72

Using a perceived room to anchor

FINST

s tagged with map labelsSlide73

Using vision with selected ‘labeled’ objects

If you ‘project’ the pattern of map places by indexing objects in the room in front of you that correspond to the memorized relative locations, then you can scan attention from one such indexed object to another. The relation

time = distance



speed

holds because the space you are scanning is the real

physical space in the room.You can also use the indexed objects to infer configurational properties you may not have noticed, despite memorizing the location of objects. e.g.Which 3 or more places on the map are collinear?Which place on the map is furthest North (or South, East, West)?Which 3 places form an isosceles triangle?Such configurational consequence can be detected as opposed to logically inferred, so long as they involve only a few places, because the visual system can examine the indexed objects in the sceneSlide74

Connecting Images and Motor actions

Images and visual-motor phenomena

S-R

Compatibility / Simon

effect

Finke’s

imagined wedge goggles

Harry’s subitizing-by-pointingSlide75

Both vision and visual imagery have some connection to the motor system

There are a number of experiments showing the close connection between images and motor control

*

You can get Stimulus-Response compatibility effects between the location of a stimulus in space and the location of the response button in space,

Ronald Finke showed that you could get adaptation with the position of the misperceived hand that was similar to adaptation to displacing prism goggles,

Both these findings provide support for the view that the spatial character of images comes from

something

being projected onto a concurrently perceived scene and then functioning much as objects of perception.This is the main new idea in Chapter 5 of Things & Places)

an image

imagined

Slide76

In all these cases you only need indexes to a few visual objects located in appropriate places

In all examples we have seen, the results can be explained without appealing to a mental display, if you assume that:

You can index a few visible objects (including texture elements on an apparently plain surface) and

The visual system can treat indexed objects as distinct or

visually labeledSlide77

This story is plausible for visual cases, but how does it work without vision (

e.g., in the dark)

?

We must rely on our remarkable capacity to orient to (point to, navigate towards, …) perceived or recalled objects (including proprioceptive ‘objects’) in space without vision



Call this general capacity our

spatial senseHow can the projection hypothesis account for this apparently world-centered spatial sense without assuming a global allocentric frame of reference?Answer: Just as it does with vision, by anchoring represented objects to (non-visually) perceived objects in the worldSlide78

The

spatial sense and the projection hypothesis

Indexing non-visual ‘objects’ must exploit auditory and proprioceptive signals, and perhaps even preparatory motor programs

(the ‘intentional’ frame of reference proposed by Anderson & Bruneo, 2002; Duhamel, Colby & Goldberg, 1992)

Is there some special problem about proprioceptive inputs that makes them different from visual inputs?Slide79

Is there a problem with

proprioceptive inputs indexing objects the way visual

indexes do?

Unlike visual objects, proprioceptive “objects” are not fixed in an allocentric frame of reference – or are all objects the same?

Notice that in vision and audition, even though static objects are fixed in an allocentric frame of reference, they nonetheless move

relative to sensors

, so their location in an allocentric frame must be updated as the proximal pattern moves

(Andersen, 1999; Stricanne, Anderson & Mazzoni, 1996) The neural implementation of FINST indexes in vision requires an active updating process of some kindMaybe the same updating operation can also yield the sense of “same location in space” for proprioceptive ‘objects’There are good reasons to think that proprioceptive signals may also be given in an allocentric frame of reference! (Yves Rosetti)Slide80

What is the real problem of our sense of space?

In order to solve the problem of how we index objects in the world using proprioceptive inputs we need to solve the problem of how we recognize two such inputs as corresponding to actions (e.g., reaching) towards the

same object

in the world

This is the problem of the

equivalence

of movements, or of proprioceptive inputs, corresponding to the same object – it is the problem that Henri Poincar

é recognized as the central problem of understanding our sense of space (in Poincaré “Why space has three dimensions” Les Dernier Penseés, 1913)Solving the equivalence problem would solve the problem of coordinating signals across frames of referenceThat’s why mechanisms of coordinate transformation are of central importance – they generate the relevant equivalences!Slide81

Assumption

: Coordinate transformations are the basis for the illusory “global frame of reference”

A

coordinate transformation

operation takes a representation of an object relative to one coordinate system – say retinal coordinates – and produces a representation of that object relative to another frame of reference – say relative to the location of a hand in

proprioceptive

or kinematic coordinatesCoordinate transformations define equivalence classes of proprioceptive inputs that correspond to actions (e.g., reaching, eye movements) towards the same object in spaceSuch transformations are well-known and ubiquitous in the brain (especially in posterior parietal cortex and superior colliculus) A consequence of these mechanisms is that, as (Colby & Goldberg, 1999) put it, “Direct sensory-to-motor coordinate transformation obviates the need for a single representation of space in environmental coordinates” (p319)Slide82

Coordinate transformations need not transform

all points in a given frame of reference

Coordinate transformations need not transform all

points

(including points in empty space) or all sensory objects:

Only a few selected objects

need to be transformed at any one time

The computational complexity of coordinate transformations can be made tractable by only transforming selected objects (as is done by matrix operations in computer graphics)This idea is closely related to the conversion-on-demand hypothesis of Henriques et al. (1998) and Crawford et al. (2004).

In the Henriques et al COD proposal, visual information about object locations is held in a gaze-centered frame of reference and objects are converted to motor coordinates when neededSlide83

Coordinate transformations define equivalence classes of gestures which individuate

proprioceptive

objects just the way that FINST indexes do in vision

Coordinate transformations compute equivalence classes of

proprioceptive

signals {s} corresponding to distinct motor actions to individual objects in real space. The equivalence class is given by:

s ≡ s′ iff there is a coordinate transformation between S ↔ S′As in the visual case, only a few such equivalence classes are computed, corresponding to a few distal objects that were selected and assigned an index, as postulated in FINST Theory

We can thus bind several objects of thought to objects in real space (including sensory ‘objects’ perceived in proprioceptive modalities)This can explain the ‘spatial’ character of spatial representations, just the way they did in the purely visual cases illustrated earlierSlide84

Mental imagery and neuroscience

Neuroanatomical

evidence for a

retinotopic

display in the earliest visual area of the brain (V1)

Neural imaging data showing V1 is more active during mental imagery than during other forms of thought

The form of activity differs for small

vs large images in the way that it differs when viewing small and large displaysTranscranial magnetic stimulation of visual areas interferes more with imagery than other forms of thoughtClinical cases show that visual and image impairment tend to be similar (Bisiach, Farah)More recently psychophysical measures of images shows parallels with comparable measures of vision, and these can be related to the receptive cells in V1Slide85

Status of different types of evidence in the debate about the form of mental images

Phenomenology. Is it

epiphenominal

?

Neuroscience evidence for:

Role of vision

Type and location of neural structures underlying images

Are the neural mechanisms for early vision used in imagery?Does neuroanatomy provide evidence for the nature of “depictive” representations.Slide86

Neuroscience has shown that the retinal pattern of activation is displayed on the surface of the cortex

Tootell, R. B., Silverman, M. S., Switkes, E., & de Valois, R. L. (1982). Deoxyglucose analysis of retinotopic organization in primate striate cortex.

Science, 218,

902-904.

There is a topographical projection of retinal activity on the visual cortex of the cat and monkey. Slide87

Problems with drawing conclusions about the nature of mental images from neuroscience data

The capacity for imagery and for vision are known to be independent. Also all imagery results are observed in the blind.

Cortical topography is 2-D, but mental images are 3-D – all phenomena (e.g. rotation) occur in depth as well as in the plane.

Patterns in the visual cortex are in retinal coordinates whereas images are in world-coordinates

Your image stays fixed in the room when you move your eyes or turn your head or even walk around the room

Accessing information from an image is very different from accessing it from the perceived world. Order of access from images is highly constrained.

Conceptual rather than graphical properties are relevant to image complexity (e.g., mental rotation).Slide88

Problems with drawing conclusions about mental images from the neuroscience evidence

Retinal and cortical images are subject to

Emmert’s Law

, whereas mental images are not;

The signature properties of vision (e.g. spontaneous 3D interpretation, automatic reversals, apparent motion, motion aftereffects, and many other phenomena) are absent in images;

A cortical display account of most imagery findings is incompatible with the cognitive penetrability of mental imagery phenomena, such as scanning and image size effects;

The fact that the

Mind’s Eye is so much like a real eye (e.g., oblique effect, resolution fall-off) should serve to warn us that we may be studying what observers know about how the world looks to them, rather than what form their images take.Slide89

Problems with drawing conclusions about mental images from the neuroscience evidence

Many clinical cases can be explained by appeal to tacit knowledge and attention

The ‘tunnel effect’ found in vision and imagery (Farah) is likely due to the patient knowing what things now looked like to her post-surgery

Hemispatial neglect seems to be a deficit in attention, which also explains the “representational neglect” in imagery reported by Bisiach

A recent study shows that imaginal neglect does not appear if patients have their eyes closed. This fits well in the account I will offer in which the spatial character of a mental images derives from concurrently perceived space.

What if colored three-dimensional images were found in visual cortex? What would that tell you about the role of mental images in reasoning? Would this require a homunculus?Slide90

Should we welcome back the homunculus?

In the limit if the visual cortex contained the contents of ones conscious experience in imagery we would need an interpreter to “see” this display in visual cortex

But we will never have to face this prospect because many experiments (including ones by Kosslyn) show that the contents of mental images are conceptual (or, as Kosslyn puts it, contain “predigested information”).

And finally, it is clear to anyone who thinks about it for a few seconds that you can make your image do whatever you want and to have whatever properties you wish.

There are no known constraints on mental images that cannot be attributed to lack of knowledge of the imagined situation (e.g., imagining a 4D cube).

All currently claimed properties of mental images are cognitively penetrable.Slide91

Explaining mental scanning, mental rotation and image size effects in terms of “functional space”

When people are faced with the natural conclusion that the “iconic” position entails space (as in scanning and size effects) they appeal to “functional space”

A Matrix in a computer are often cited as an example

Consider a functional space account of scanning or of mental rotation:

Why does it take longer to scan a greater distance in a functional space?

Why does it take longer to rotate a mental image a greater angle?Slide92

Why do conscious contents misguide us?

The contents that appear in our conscious experience almost always concern what we are thinking

about

and not what we are thinking

with

– with

content rather than form.

The processes that we see unfolding in our mind are almost always attributable to what we know about how the things we are thinking about would enfold, rather than being due to laws that apply to our cognitive architecture. cf Code BoxWe should take seriously the possibility that (almost) all constraints and law-like behaviors of objects of our experience are constraints due to our knowledge rather than of the mental architecture. Notice the mental rotation example and the mistake that Jesse Prinz makes.Slide93

This is what our conscious experience suggests goes on in vision…

KlibanSlide94

This is what the demands of explanation suggests must be going on in vision…Slide95

Imagine this shape rotating

slowly

When you make it rotate in your mind, does it retain its rigid 3D shape without re-computing it? Would you expect to ‘see’ this kind of information process? Does the experience in this case reassure you that the rotation was smooth? Are you sure

something rotated

?

Is this how it looked to you?Slide96

What about the evidence of conscious experience? Is it irrelevant?

I have often been accused of relegating conscious experience to the category of

epiphenomena

–

something that accompanies a process but does not itself have a role in its causation.

But images are not illusory or unnatural, they are quite real. The problem is that people have theories of the causal or information-processing that underlies these phenomena and these theories are almost always false because they assume a simple and obvious mapping from the experience to the computational or brain states so that we can

see

the form of the representation.The connection between conscious experience and information processing is deeply mysterious (it’s the mind-body problem). But one thing we do know is that the sequence of events that unfolds when we imagine something does not reveal causal laws because there are no causal laws of conscious states as conscious states.Slide97

The important distinction between architecture and represented content

It is only obligatory that a certain pattern must occur if the pattern is caused by fixed properties of the architecture, as opposed to being due to properties of what is represented (i.e., what the observer

tacitly

knows about the behavior of what is represented)

If it is obligatory only because the theorist says it is, then score that as a free empirical parameter (a wild card)

If we allow one theory to stipulate what is obligatory without there being a principle that mandates it, then any other theory can stipulate the same thing. Such theories are unconstrained and explain nothing.

This failure of image theories is quite general – all picture theories suffer from the same lack of principled constraintsSlide98

The important distinction between architecture and represented content

It is only obligatory that a certain pattern must occur if the pattern is caused by fixed properties of the architecture, as opposed to being due to properties of what is represented (i.e., what the observer

tacitly

knows about the behavior of what is represented)

If it is obligatory only because the theorist says it is, then score that as a free empirical parameter (a wild card)

If we allow one theory to stipulate what is obligatory without there being a principle that mandates it, then any other theory can stipulate the same thing. Such theories are unconstrained and explain nothing.

This failure of image theories is quite general – all picture theories suffer from the same lack of principled constraintsSlide99

How are these ‘obligatory’ constraints realized?

Image properties, such as size and

rigidity

,

are assumed to be inherent in the system of representation (its architecture)

That raises the question of

what kind of architecture could possibly enforce rigidity of shape

? Notice that neither a spatial display nor a functional space make it obligatory that shape be rigidly maintained as orientation is changed. Only certain physical properties can explain rigidity.Such rigidity could not be part of the architecture of an imagery system because we can easily imagine situations in which rigidity does not hold (e.g. imagine a rotating snake!).There is also evidence that ‘mental rotation’ is incremental, not holistic, and the speed of rotation depends on the conceptual complexity of the shape and the comparison task.Slide100

Aside: What can we conclude from the contents of conscious experience?Slide101

What should we conclude about the role of conscious appearance in cognitive science?

We can’t do without it:

When we ask which line appears longest or which version of an ambiguous figure we see we are asking for a report of conscious content. Much of what we know of how vision works depend on such evidence.Slide102

What should we conclude about the role of conscious appearance in cognitive science?

But we can’t accept it at face value:

If you ask yourself “what am I thinking …?” you raise one of the most mysterious problems in the philosophy of mind. You could not possibly give the correct, or at least an unproblematic answer because it is quite possible that you don’t know what you are thinking! Your experience when thinking is as of speaking or of perceiving – what else could it be?

Every imagined sentence is infinitely ambiguous and understanding it presupposes a huge amount about the context of its utterance. Your experience of speaking cannot be the same as your thought; your thought precedes your imagined speech and is just one of the contents that are expressed. The sentences you imagine, as the sentences you speak, follow

Gricean

conversational

maxims – e.g., don’t state what your listener already knows, state only what is relevant, state only as much as necessary to convey your intentions. Something similar is true of imaging.Slide103
Examples of conscious evidence that leads to false conclusions

Sentence example

Example of mental diagram and what it must assumeSlide104

Do we (or can we) experience our thoughts?

There is much that can be said on this topic, but there is no time for it here. But see:

Grice, H. P. (1975). "Logic and Conversation."

Syntax and Semantics

3: 41-58

.

Hurlburt

, R. T., & Schwitzgebel, E. (2007). Describing Inner Experience? Cambridge, MA: MIT Press.Schwitzgebel, E. (2011). Perplexities of Consciousness. Cambridge, MA: MIT Press.Slide105

What should we conclude about the role of conscious appearance in cognitive science?

But we can’t accept it at face value:

If you ask yourself “what am I thinking …?” you raise one of the most mysterious problems in the philosophy of mind. You could not possibly give the correct, or at least an unproblematic answer because it is quite possible that you don’t know what you are thinking! Your experience when thinking is as of speaking or of perceiving – what else could it be?

Every imagined sentence is infinitely ambiguous and understanding it presupposes a huge amount about the context of its utterance. Your experience of speaking cannot be the same as your thought; your thought precedes your imagined speech and is just one of the contents that are expressed. The sentences you imagine, as the sentences you speak, follow

Gricean

conversational

maxims – e.g., don’t state what your listener already knows, state only what is relevant, state only as much as necessary to convey your intentions. Something similar is true of imaging.Slide106

But there are examples of solving geometry problems easily with imagery

There are many problems that you can solve much more easily when you imagine a layout than when you do not.

In fact many instances of solving problems by imagining a layout that seem very similar to how would solve them if one had pencil-and-paper.

The question of how pictures, graphs, diagrams, etc help in reasoning is very closely related to the question of how imagined layouts function in reasoning. That is not in question. What is in question is what happens in

either

the visual or imagined cases and how images can benefit from this processes even though there is no real diagram.Slide107

How do real visual displays help thinking?

Why do diagrams, graphs, charts, maps, icons and other visual objects help us to reason and to solve problems?

The question why visual aids help is nontrivial and

Seeing & Visualizing

, chapter 8 contains some speculative discussion, e.g., they allow the visual system to:

make certain kinds of

visual inferences

make use of visual demonstratives to offload some of the memory loadCapitalize on the fact that the displays embody the axioms of measure theory and of geometry (which are then inherited by thought)The big question is whether any of these advantages carry over to imaginal thinking! Do mental images have some (or any) of the critical properties that make diagrams helpful in reasoning?Slide108

Visual inferences?

If we recall a visual display it is because we have encoded enough information about its visual-geometrical properties that we can meet some criteria, e.g., we can draw it. But there are innumerably many ways to encode this information that are sufficient for the task (e.g. by encoding pairwise spatial relations, global spatial relations, and so on). For many properties the task of translating from one form to another is much more difficult than the task of visually encoding it – the translation constitutes visual inference.

The visual system generalizes from particular instances as part of its object-recognition skill (all recognition is recognition-as and therefore assumes generalization from tokens to types). It is also very good at noticing certain properties (e.g., relative sizes, deviations from square or circle, collinearity, inside, and so on). These capabilities can be exploited in graphical layouts.Slide109

Memorize this map so you can draw it accuratelySlide110

From your memory:

Which groups of 3 or more locations are collinear?

Which locations are midway between two others?

Which locations are closest to the center of the island?

Which pairs of locations are at the same latitude?

Which is the top-most (bottom-most) location?

If you could draw the map from memory using whatever properties you noticed and encoded, you could easily answer the questions by looking at your drawing – even if you had not encoded the relations in the queries.Slide111

Draw a rectangle. Draw a line from the bottom corners to a point on the opposite vertical side. Do these two lines intersect? Is the point of intersection of the two lines below or above the midpoint? Does it depend on the particular rectangle you drew?

A

B

C

D

m

m’

y

xSlide112

Which properties of a real diagram also hold for a mental diagram?

A mental “diagram” does not have any of the properties that a real diagram gets from being on a rigid 2D surface.

When you imagine 3 points on a line, labeled A, B, and C, must B be

between

A and C? What makes that so? Is the distance AC greater than the distance AB or BC?

When you imagine drawing point C after having drawn points A and B, must the relation between A and B remain unchanged (e.g., the distance between them, their qualitative relation such as above or below). Why?

These questions raise what is known as the

frame problem in Artificial Intelligence. If you plan a sequence of actions, how do you know which properties of the world a particular action will change and which it will not, given that there are an unlimited number of properties and connections in the world?Slide113

Summary of constraints to be met by a system of active spatial representation:

Represent

magnitudes

in a non-numerical way

In this proposal magnitudes are analogue and in the world

Represent holistic

configurations

(enabling some inferences by pattern-matching)Up to 4 indexed objects can be treated as one since they are indexedEngage the motor systemIn this proposal indexed objects can be bound to motor commandsRepresent distances in

3DIndexes bind distal objects so are assumed to be in 3D Capture connectedness and continuity of spacePredicts that empty places are not indexed and continuity only applies when there is a visible physical connection even though the space, being physical has all the properties of real space.