March 12th 2013 - Ph.D. defence
Ph.D. Defence Committee
Noël NGUYEN Jacques VAUCLAIR Jean-Marc COLLETTA Asli ÖZYÜREK Jean-Luc SCHWARTZ Anne VILAIN Coriandre VILAIN |
|
Abstract
Multimodality of linguistic communication: Gesture/speech interaction and distance encoding in pointing tasks
The aim of this thesis is to characterize speech, manual gesture and language interactions during pointing tasks. The first part of the study is about the production of pointing. The original aspect of this work is to look for distance encoding in the lexical, phonetic and kinematic properties of multimodal pointing. Our results show that these parameters can be related with each other, and underlain by a similar motor behaviour (designating a distant object induces larger gestures, be they vocal or manual), that can be related with the phonological pattern that is used for distance encoding in the world’s lan- guages. Our results also reveal that the vocal and manual modalities act in bidirectional cooperation for deixis, sharing the informational load when used together. The second part of the study explores the development of multimodal pointing in 6-12 year-old children and attests a progressive evolution of speech/gesture interactions. It reveals that distance is preferen- tially encoded in manual gestures, rather than in vocal gestures. It also shows that the cooperative use of speech and gesture in deixis is already at play in children, though with more influence of gesture on speech than the reversed pattern. The third part of the study looks at sensorimotor interactions in the perception of spatial deixis. It reveals that manual gesture plays a role in the production/perception mechanism associated with the semantic processing of language. These results can be related with those of studies on the nature of representations in the processing of linguistic sound units.
Keywords: Multimodality, Deixis, Distance encoding, Production/Perception, Development
Multimodality of linguistic communication: Gesture/speech interaction and distance encoding in pointing tasks
The aim of this thesis is to characterize speech, manual gesture and language interactions during pointing tasks. The first part of the study is about the production of pointing. The original aspect of this work is to look for distance encoding in the lexical, phonetic and kinematic properties of multimodal pointing. Our results show that these parameters can be related with each other, and underlain by a similar motor behaviour (designating a distant object induces larger gestures, be they vocal or manual), that can be related with the phonological pattern that is used for distance encoding in the world’s lan- guages. Our results also reveal that the vocal and manual modalities act in bidirectional cooperation for deixis, sharing the informational load when used together. The second part of the study explores the development of multimodal pointing in 6-12 year-old children and attests a progressive evolution of speech/gesture interactions. It reveals that distance is preferen- tially encoded in manual gestures, rather than in vocal gestures. It also shows that the cooperative use of speech and gesture in deixis is already at play in children, though with more influence of gesture on speech than the reversed pattern. The third part of the study looks at sensorimotor interactions in the perception of spatial deixis. It reveals that manual gesture plays a role in the production/perception mechanism associated with the semantic processing of language. These results can be related with those of studies on the nature of representations in the processing of linguistic sound units.
Keywords: Multimodality, Deixis, Distance encoding, Production/Perception, Development
2009 - 2013
Ph.D. in Cognitive Sciences, Cognitive Psychology, & Neurocognition (Grenoble University)
2008 - 2009
MSc in Engineering of Cognition, Creation, and Learning, speciality Cognitive Sciences (Grenoble Institute of Technology), Honors
2007 - 2008
MSc in Cognitive Psychology (Pierre Mendès France University, Grenoble), Honors
2004 - 2007
BSc in Psychology (Pierre Mendès France University, Grenoble), Honors