2001 Nature Publishing Group http://neurosci.nature.

com

news and views

Figure and ground in the brain

Nava Rubin
Segregation of an image into figure and ground is an important step in visual processing. Two new papers show that responses in human and monkey brain areas known to be involved in shape perception depend critically on whether a region is perceived as figure or ground.
2001 Nature Publishing Group http://neurosci.nature.com

Study the shapes in Fig. 1a for a moment. Now look at Fig. 1b: which shape was present in 1athe top or the bottom one? Many observers answer top, not noticing that Fig. 1a also contains a region that corresponds to the bottom shape. The reason observers often do not notice this initially is that this region is attributed to the background. The observation that the shape of background regions is often not registered perceptually dates back to the Gestalt psychologist Edgar Rubin1,2, who wrote, what is perceived as figure and what is perceived as ground do not have shape in the same way. In a certain sense, the ground has no shape. The different perceptual quality of figure and ground has profound ecological justification. Imagine rearranging the surfaces in Fig. 1; their own shapes would remain intact, but the shape of the background regions would change entirely. Whereas the shape of a figure is intrinsic to it, the shape of the ground is an accidental outcome of a specific arrangement of figural objects. It therefore makes sense that shape analysis (such as for recognizing objects) should be performed only for figural regions. To encode or analyze the shape of background regions would be a waste of resources. Papers from Baylis and Driver in this issue of Nature Neuroscience3 and Kourtzi and Kanwisher in this weeks issue of Science4 present evidence for a neural correlate of the differential treatment of figure and ground by the brain. Baylis and Driver3 studied the inferotemporal (IT) cortex of awake behaving monkeys, which contains cells whose responses are often selective to specific, complex shapes5. They asked, when a cell is selective to a certain shape, does it matter whether this shape is presented as figure or as ground? Their answer is a resounding yes. The authors devised stimuli that allowed them to reverse whether a particular shape was seen as figure or ground, while making
Nava Rubin is in the Center for Neural Science, New York University, New York, New York 10003, USA. e-mail: nava@cns.nyu.edu

minimal changes to the image. They found that IT cells that produced a vigorous response to a particular shape when it was figure could have a very weak response when a region of nearly identical shape was present in the background. Over the large population of IT cells from which they recorded, there was no correlation between a cells response to the figure versus ground conditions for each shape. This means, for example, that an IT cell that responds strongly to the bottom shape in Fig. 1b typically will not respond to Fig. 1a, where a region of the same shape is present as part of the background. The effect of figure/ground assignment was observed even in the earliest portions of the cells responses, suggesting an intimate coupling between shape selectivity and figure/ground segregation. These new physiological findings are in excellent agreement with the perceptual effects described by Edgar Rubin and extended in several behavioral studies by Baylis and Driver (see references in ref. 2). Because inferotemporal cortex has been implicated in a variety of shape perception and object recognition processes5, it is important to know that monkey IT cells are influenced by figure/ground segregation in a manner similar to human perception. As Baylis and Driver point out, their results indicate that the selectivity of IT responses is not determined simply by the contours in a display. This may seem at odds with the strong emphasis vision

research has put on edges (sharp gradients in luminance, color or texture) as a major source of visual information. The edge doctrine has deep roots, ranging from the pioneering physiology experiments of Hubel and Wiesel6 to perception studies with line drawings and decades of research in computer vision7. Indeed, edges provide important information about the shape of the surfaces they bound. But detecting edges is only a first step toward interpreting images. An edge is only informative about the shape of the front surface (Fig. 1). The region on the other side of the edge typically continues behind the front surface, and its own shape will have nothing to do with the edges shape. This is true not only for the ground (the distant-most, shapeless region, which nothing can come behind), but also when the back region is a surface with a well-defined shape of its own. For example, the redblue border in Fig. 1a defines the shape of the blue, but not the red shape. The classic problem of figure/ground segregation is therefore a special case of the more general problem of determining which of the two sides of an edge is in front. Edgar Rubin used the term belonging together (Zusammmengehrigkeit) to refer to the inseparability of a contour and the region it bounds. In recent years, the problem has been stated as determining the direction of border-ownership: the side in front is said to own the edge8. Baylis and Drivers results indicate that IT cells have information not just about edges, but also

Fig. 1. The assignment of image regions as figure or background has a dramatic effect on their perceived shape. The top shape in (b) is easily found in (a), but the region that corresponds to the bottom shape in (b) is often unnoticed, because it is part of the background (top right corner) in (a). Two papers appearing this week report a neural correlate for this perceptual phenomenon. 857

nature neuroscience volume 4 no 9 september 2001

news and views

2001 Nature Publishing Group http://neurosci.nature.com

about their border-ownership polarity. This allows the neurons to avoid responding to meaningless background regions, and instead reliably encode the shapes of objects. Kourtzi and Kanwisher4 address a similar question in humans, using functional magnetic resonance imaging (fMRI). Brain imaging studies suggest that the lateral occipital complex (LOC), a region in human occipitotemporal cortex, is important for shape perception and object recognition911. Kourtzi and Kanwisher studied whether shape selectivity in this region is also sensitive to figure and ground. Like Baylis and Driver, they manipulated whether a region appeared as figure or ground by making small changes to the image. Stereo was used to reverse the relative depth of two bordering regions, and thus the polarity of border-ownership, by swapping the images to the two eyes8. Kourtzi and Kanwisher used an adaptation technique to investigate the selectivity of LOC neurons to these stimuli. Single-unit and fMRI responses typically adapt to repeated presentations of similar stimuli, becoming smaller with repetition11,12. Selective adaptation can be taken to indicate neuronal selectivity, and is thus a useful tool for fMRI research. Previous work has shown that LOC adapts to repeated presentations of the same shape but not to different shapes, suggesting the presence of neurons tuned to shape. LOC also adapts to translated and scaled versions of an object, but not objects shown from different vantage points, indicating that shape encoding in LOC is invariant to size and location, but not to changes in viewpoint11. Kourtzi and Kanwisher asked whether this adaptation also depended on border-ownership. In their critical comparison, a particular contour was followed by the same contour with border-ownership reversed. They found no detectable LOC adaptation when the repeated stimulus had reversed border-ownershipas if a wholly new stimulus were presented. The magnitude of the effect is striking: after reversing border-ownership, the response amplitude was just as high as when presenting an entirely new contour, fully reflecting the dramatic change in the perceived identity and shape of the figure. Thus, LOC cells seem to behave much like the IT cells studied by Baylis and Driver: a shape is not merely an arrangement of contours; it does not count as a shape unless it is also figural. Given that monkey IT and human LOC are the brain regions most often implicated in shape and object perception, the findings of these two papers3,4 suggest that
858

there may be no place in the brain where shape is represented independently of border-ownership. This is important for theories of shape perception and object recognition. At the same time, these papers leave open the question of where and how border-ownership is computed in the brain. One possibility is that border-ownership is resolved at earlier stages of cortical processing, in areas that provide input to human LOC and monkey IT. Another recent study provides some support for this idea, but at the same time poses new questions. Zhou et al.13 reported that cells sensitive to border-ownership can be found in early monkey visual cortexV1, V2 and V4. They oriented a large square so that one of its edges fell over the minimal receptive field (MRF) of a cell. They centered one edge and then the opposite edge on the cells MRF, thus stimulating the cell with a contour in both cases, but with inverted border-ownership polarity. They found that over 50% of the cells in V2 and V4, and 18% of V1 cells, showed marked differences in their response to the two conditions. These cells therefore appear to signal only edges with particular border ownership assignmentsnot the mere presence of any edge. Can we therefore conclude that border-ownership is computed early in visual cortex, and that the reported behavior of monkey IT cells 3 and human LOC 4 reflect input from early visual areas? Several considerations suggest that the picture may be more complex than that. Resolving border-ownership is a nontrivial process not yet well understood computationally. A central difficulty is the need to combine local and global information. In the experiments of Zhou et al., for instance, the size of the square was much larger than the size of the cells MRF (as mapped by small bar stimuli). The responses of the cells were therefore affected by manipulations that took place in image regions represented by widely distant cells in those early cortical areas. Although it is possible to compute borderownership with a network of small-receptive-field units that mediate information via a cascade of lateral connections14, this is a time-consuming computation for large figures. A key finding of Zhou et al. was that the effects of border-ownership could be observed very early, within 1025 milliseconds from response onset (similar to what Baylis and Driver found). The authors note that V2 mechanisms would be at their limit to resolve border-ownership so quickly for large figures, and that feedback from cells with larger receptive

2001 Nature Publishing Group http://neurosci.nature.com

fields in higher visual cortical areas could facilitate the computations considerably. Moreover, perceptual results showing that object familiarity can have marked effects on figure/ground resolution15 also implicate higher cortical regions in the process. Taken together, computational considerations and experimental results suggest that the resolution of borderownership, and the resulting assignment of image regions as figure or ground, involve computations in early as well as high-level visual areas. Asking how these computations are done by the brain may therefore be more appropriate than asking where they are done. Traces of activity before border-ownership resolution might be found not in a specific area, but in the early activity of the network. Slowing the process down, for example by using stimuli where border-ownership resolution is more difficult, may help in future studies. The striking efficiency with which the visual system assigns border-ownership to edges makes this computational feat hard to unravel, but also points to its central importance.
1. Rubin, E. Visuell wahrgenommene Figuren (Gyldendals, Copenhagen, 1921). 2. Rubin, E. in Visual Perception Essential Readings (ed. Yantis, S.) 225230 (Psychology Press, Philadelphia, 2001). 3. Baylis, G. & Driver, J. Nat. Neurosci. 4, 937942 (2001). 4. Kourtzi, Z. & Kanwisher, N. Representation of perceived object shape by the human lateral occipital cortex. Science (in press). 5. Logothetis, N. K. & Sheinberg, D. L. Annu. Rev. Neurosci. 19, 577621 (1996). 6. Hubel, D. H. & Wiesel, T. N. J. Physiol. (Lond.) 195, 215243 (1968). 7. Marr, D. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information (W.H. Freeman, San Francisco, 1982). 8. Nakayama, K., He, Z. J. & Shimojo, S. in Visual Cognition (eds. Kosslyn, S. M. & Osherson, D. N.) 170 (MIT Press, Cambridge, Massachusetts, 1995). 9. Malach, R. et al. Proc. Natl. Acad. Sci. USA 92, 81358139 (1995). 10. Mendola, J. D., Dale, A. M., Fischl, B., Liu, A. K. & Tootell, R. B. J. Neurosci. 19, 85608572 (1999). 11. Grill-Spector, K. & Malach, R. Acta Psychol. (Amst.) 107, 293321 (2001). 12. Miller, E. K., Li, L. & Desimone, R. Science 254, 13771379 (1991). 13. Zhou, H., Friedman, H. S. & von der Heydt, R. J. Neurosci. 20, 65946611 (2000). 14. Pao, H., Geiger, D. & Rubin, N. Measuring convexity for Figure/Ground separation. Proc. 7th IEEE Intl. Conf. Comp. Vision, 948955 (1999). 15. Peterson, M. A. & Gibson, B. S. Psychol. Sci. 5, 253259 (1994).

nature neuroscience volume 4 no 9 september 2001