Representing discourse referents in speech and gesture

Sammanfattning: The thesis examines the way that speech and gestures are used together to represent referents in discourse. The starting point is the generally acknowledged observation that gestures are a constitutive part of language (Kendon, 2004; McNeill, 1992). Gestures are used in combination with speech in the production of language, and are integrated with speech in the perception of language. But while there is abundant linguistic research on the way that speech varies for the representation of referents as a function of discourse, much less is known about the role that gestures play. The thesis starts with a description of speech patterns. Linguistic research has shown that the way that speakers refer to discourse referents strongly relies on assumptions about the referents’ accessibility or information status (e.g., Ariel, 1988; Arnold, 1998; Chafe, 1994; Givón, 1983). Depending on whether speakers assume a referent to be new/less accessible or given/more accessible (for their addressees), they will vary different parameters on the level of the referential expression or the clause a referent is mentioned in. For instance, speakers will vary richness of expression (lexical NP vs. pronoun), nominal definiteness (indefinite vs. definite NP), the structure of the clause (focusing on the existence of a referent vs. focusing on an event the referent is mentioned in), and grammatical role assignment (subject vs. object). Importantly, gestures too can vary along different dimensions for the representation of discourse referents. They vary in terms of when they are produced, where they are produced, how they are produced, and in terms of what information they express. When gestures are produced refers to the incidence of gestures with some referential expressions used to represent referents but not with others. Where gestures are produced refers to the use of gesture space to create visual anaphoric linkages between different mentions of a referent. How gestures are produced refers to techniques of representation in gestures and is operationalized as gesture viewpoint (McNeill, 1992). And finally, what gestures express refers to whether gestures provide information about the entity itself or an action it is involved in when they accompany discourse referents.Three papers on bimodal discourse production show that there are intricate relationships between referent accessibility, discourse patterns in speech, and variations in gesture for the representation of referents. A fourth paper on bimodal discourse perception further shows that addressees are sensitive to the presence of gestures when they track referents in connected discourse. Together, the four studies contribute to a deeper understanding of the close relationship between speech and gestures on the discourse level, and highlight the multifunctionality of gestures. For language production, two main functions of gestures are identified, a parallel and a complementary function. For gesture perception, a facilitatory function is suggested. The thesis emphasizes the importance of considering gestures in linguistic studies of discourse, as well as considering discourse organizational principles in gesture studies.