Hearing Gesture:

Top

My studies of gesture began serendipitously, as many studies do. Rochel Gelman, my graduate school advisor, had lent me a videotape to use in the Introduction to Developmental Psychology class I was teaching at the University of Chicago. The tape was 1 inch reel-to-reel, which gives those of you who know the history of videorecorders some idea of how long ago this happened (Rochel was the experimenter on the tape and was wearing a mini-skirt, which also dates the event). The tape showed a series of children participating in Piagetian conservation tasks – tasks in which a quantity appears to change but doesn't actually change (e.g., liquid is poured from one container to a differently shaped container) and the child is asked whether the transfer has affected the original quantity. Before age 7 or 8, children are convinced that the amount of water does change when it changes containers, and can give reasoned explanations for their (incorrect) beliefs.

I used the tape in my developmental class every year, and every year I pointed out how striking it was that the children could not conserve quantity. Over time I became less fascinated with the children's non-conserving responses (although my students never tired of them), and I began to look at the tape as well as listen to it. I finally noticed that the children couldn't keep their hands still during their explanations – they gestured constantly. Even the children who had already mastered conservation gestured throughout the task.

I asked Breckie Church, one of my graduate students at the time, to take a look at children's hand movements. We collected our own videotapes of children participating in conservation tasks and, after hours of careful study, Breckie and I were convinced that these movements were not just handwaving. Rather, the children were using their hands to convey substantive information about the task. As an example, one child produced a pouring motion just as she said, "it's a different amount because you poured it."

Not wanting the way we described the gestures to be influenced by the speech we heard during transcription, we next went through the tapes coding speech without gesture (with the picture turned off) and gesture without speech (with the sound turned off). And here we made our most interesting discovery – the observation that underlies all others described in the book. Many times, a child would produce a gesture that conveyed the same information he or she had just articulated in speech – like the pouring gesture in the example given above. However, at other times, a child would give one explanation in speech and a completely different explanation in gesture. For example, the child might say "it's a different amount because you poured it," while gesturing the shape of the container (two C-shaped hands positioned as though holding a round dish). In speech the child had focused on the experimenter's pouring motions, but in gesture the child had conveyed information about the width of the container – the child had produced a gesture-speech "mismatch".

Our second discovery was that these gesture-speech mismatches have cognitive significance. When we gave all of the non-conserving children instruction in conservation, only some of the children profited from our instruction – those who had produced many gesture-speech mismatches in their previous explanations. We thus concluded that gesture not only reveals a child's unspoken thoughts, but it also can give us notice that the child may be ready to learn new things.

The goal of the research program that grew out of these discoveries, and the goal of this book, is to understand and convey the importance of gestures of this sort. When does gesture reveal thoughts that are not expressed in speech? What kind of thoughts does it reveal? Does gesture play an active role in the conversations we have or, even more fundamentally, the thoughts we think? In the book, I try to make the case that gesture can indeed shape both our conversations and our thoughts. In this way, gesture can reveal, and propel, cognitive change.

A gesture-speech match on a math problem -->
	The child mentions the 3 numbers on the left side of the equation in speech, and points at these same 3 numbers in gesture. (i.e., an add-numbers-to-the-equal-sign strategy in both speech and gesture)
	<-- A gesture-speech mismatch on a math problem
The child mentions the 3 numbers on the left side of the equation in speech, but points at all 4 numbers in the problem in gesture. (i.e., an add-numbers-to-the-equal-sign strategy in speech and an add-all-numbers strategy in gesture)

Top

The book is divided into four parts.

• The first section establishes gesture as a tool that we can use to see into a speaker's thoughts. I begin by situating gesture within the vast field of nonverbal communication, and making it clear why gesture (as opposed to other nonverbal acts) is the focus of this book. I provide evidence that gesture is more than just handwaving and can convey substantive information about a task. Moreover, gesture forms an integrated system with speech and must be examined in relation to speech. It is because gesture and speech are so tightly intertwined that, when gesture conveys information that is different from speech, it takes on cognitive significance. It can tell us who is ready to learn and what they know that puts them in this malleable state.



• The second section explores gesture's role in communication. If gesture can reveal unspoken thoughts, those thoughts are then "out there" and can be part of the conversation – assuming, of course, that gesture can be "read" by ordinary people in ordinary circumstances. I show first that everyone can read gesture, young or old, in an experiment or in real-live communication. I also show that how we understand speech is affected by the gestures that accompany that speech. Finally, I consider whether gesture can lead to cognitive change in the gesturer. If speakers reveal their readiness for instruction simply by moving their hands, and if listeners are attentive to those movements and change their responses accordingly, gesture can provide an indirect way for learners to tell their teachers what they need next. Gesture can change the course of learning by influencing the kind of input the learner receives.



• The third section examines gesture's role in thinking. Gesture can convey information to others, but it can also have effects on the speaker. We have all had the experience of finding ourselves gesturing when no one is watching. We may feel sheepish about it but that doesn't seem to stop us. Moreover, many of us have had the experience of seeming to gesture more when we are fumbling for the right word to say. Must we move our hands when we speak? Why? I suggest that gesturing may help us think – by making it easier to retrieve words, easier to package ideas into words, easier to tie words to the real world. If this is so, gesture may contribute to cognitive growth by easing the learner's cognitive burden and freeing resources for the hard task of learning. Moreover, gesture provides an alternate spatial and imagistic route by which ideas can be brought into the learner's cognitive repertoire. That alternative route of expression is less likely to be challenged (or even noticed) than the more explicit and recognized verbal route. Gesture may be more welcoming of fresh ideas than speech.



• The fourth section explores situations in which there is no speech and gesture is forced to take on the full burden of communication. When gesture is produced with speech, it shares the burden of communication with that speech and, I show in the book, takes on an imagistic form that is quite distinct from speech as a consequence. But what happens when gesture is all that a person has? Does gesture change its form when it changes its function? The short answer to this question is "yes" – gesture becomes segmented and language-like in form when it assumes sole responsibility for language functions. I illustrate this point over three timespans – conventional sign systems passed down over generations; gesture systems invented during childhood to serve as the child's primary communication system; and gestures created on the spot by adults asked to use their hands and not their mouths to communicate. Examining gesture when it functions without speech allows us to more completely understand how it functions with speech. I end with a final chapter on what it means to hear gesture.