The Influence of Modality Combinations on Communication in Collaborative Virtual Environments

Sammanfattning: Although many studies have been performed on collaboration in multimodal interfaces not many of these have looked specifically on how the supported modalities influence the task solving strategies chosen and the communication between users solving a joint task in collaborative virtual environments. Therefore, the thesis studies performed aimed at shedding light on these aspects of multimodality. The specific research question studied is: How do changes in modality combinations influence employed work strategies, communication during task solving and the task efficiency in collaborative multimodal virtual environments? The studies performed build on theories from HCI, CSCW, human perception and mediated communication and are thus inter-disciplinary in nature. A variety of cases have been studied; collaboration between sighted and visually impaired, task solving in visually demanding environments and to some extent support for achieving medical diagnoses.The research presented in this thesis began with a field study in elementary schools, focusing on collaboration between visually impaired and sighted pupils. The shared environment was in this case a virtual room in which objects could be moved around by means of haptic devices. The results showed a great potential for haptic feedback when it came to supporting collaboration and most of all communication between the participants. A lack of awareness information about mostly the sighted pupils’ actions laid the ground for a follow-up study in which sighted and blindfolded students solved tasks in the same interface. A formal experiment was carried out in this case, comparing a visual/haptic environment with a visual/haptic/audio environment. Results showed that the addition of audio feedback to the visual/haptic environment was beneficial in many respects. Up until now, the focus had been entirely on collaboration between sighted persons and those who cannot see. This is why the next experimental study, based on an abstract gaming environment, aimed at collaboration between sighted persons. Since the earlier studies showed that the combination of modalities clearly matter, this new experiment compared three modality combinations – visual/haptic, visual/audio and visual/haptic/audio. Once again, the results clearly showed that the combination of modalities has an effect on task performance and that it influences collaboration and communication in particular.All studies performed have been subject to both quantitative analysis of performance measures and qualitative analysis of dialogues between collaborators. Even though quantitative data on task performance has played an important role, the main focus has been on qualitative data in all studies performed. The results show that different combinations of modalities influence the collaboration and in particular the communication between two participants solving tasks in different ways in a number of multimodal interfaces. In all cases in which a visual/haptic/audio condition has been compared to a visual/haptic or a visual/audio condition the performance was significantly better in the visual/haptic/audio condition. One of the most important conclusions drawn from the qualitative analysis of dialogues is that both haptic and audio feedback can have communicative properties which influence the dialogue and as a consequence the collaboration.