A "happy" approach to speechreading : the effects of facial expression, emotional content, and script information on speechreading performance

Sammanfattning: The aim of the present thesis was to study if and how emotions in facial expressions affect speechreading performance. Speechreading is the understanding of spoken messages by watching the speaker's lips, taking facial expressions and situational cues into account. The content of the spoken message and knowledge about the situation where these messages belong (i.e., the script) were also studied. It was shown, in four studies, that congruence between the script and the content of the sentence, and congruence between the content of the sentence and the facial expression, enhanced speechreading performance. It was also shown that a happy facial expression improved speechreading performance. This was found in a happy script but also when the script was neutral and when no script information was presented. The implications from the studies were discussed in relation to current speech perception models. A framework for visual speechreading of emotional sentences was presented. The framework consists of three parallel streams of information processing, a script stream, an emotional content stream and a facial expression stream. Two processes, a perceptual analysis of the lip movements and an on-going checking process of the script and the facial expression are also part of the framework. If the perceptual analysis fits with the facial expression, the emotional signal, and the knowledge activated from the script, comprehension is optimized and the speechreading is successful.