Multimodal Communication in Virtual and Face-To Gesture Production and Speech Disfluency

Abstract

The COVID-19 pandemic has made online data collection a popular choice. It is important to evaluate howcomparable online studies are to face-to-face studies, particularly in multimodal language research wheremodes of communication significantly impact the results. In this study, we examined individuals' rates andpatterns of speech disfluency and gesture use across face-to-face and online videoconferencing settings asthey described their daily routines (N= 64). We asked whether and how multimodal language is affected acrossdifferent communication settings and gesture use, particularly iconic gestures, is associated with speech fluencyregardless of the context. Our results have showed that the participants' overall disfluency rate was higherfor the speech communicated via videoconferencing than the speech communicated face-to-face. However,the type of disfluencies changed across contexts, such that filled pauses and repairs were more commonin online communication, whereas silent pauses were more common in face-to-face communication. Thesefindings signal an interplay between the cognitive functions of different disfluency types and communicativestrategies. Results indicate that the overall gesture frequency and iconic gesture use were similar in bothsettings. Furthermore, the use of iconic gestures was found to negatively predict the overall disfluency rate,regardless of the setting. This finding suggests that using iconic gestures might facilitate cognitive processes,paving the way for a more fluent speech. This study demonstrates that multimodal language and communicationstrategies may vary across different communication settings and nuanced understanding of the differences inmultimodal language between online and face-to-face communication can be gained using different contexts.The findings contribute to understanding the impact of increasingly widespread online communication onmultimodal language production processes and provide foundation for future research.