Beyond Motion Realism: Linguistically Informed Signing Avatars and Visual Attention in Brazilian Sign Language

Abstract: Automatic translation into sign languages presents challenges extending beyond the realistic reproduction of human movements. This presentation introduces research conducted within the CCD-TAAL project, a partnership between the University of São Paulo and University of Campinas, integrating linguistic analysis, motion capture, and eye-tracking methodologies to advance automatic translation into Brazilian Sign Language (Libras). Our research investigates visual attention and comprehension among Deaf signers exposed to natural signing and translations generated by two commercial systems, Hand Talk and Calibras. These investigations provide a foundation for evaluating the perceptual and linguistic limitations of existing technologies. In parallel, CCD-TAAL is developing a more realistic signing avatar using motion capture data from Deaf signers, aiming to preserve the biomechanical and temporal characteristics of natural signing. Crucially, the project also investigates more sophisticated linguistic annotation frameworks capable of representing complex morphosyntactic and visuospatial phenomena frequently inadequately addressed by automatic translation systems. Particular attention is given to constructed action, classifier constructions, and pronominal reference systems, including manual and gaze-based pointing and their interaction with spatial reference. By combining linguistically informed annotation, human motion modelling, and perceptual evaluation, the project seeks to develop signing avatars that are not only visually realistic but also linguistically accurate and comprehensible.

Speaker bio: Felipe Venâncio Barbosa, PhD is a Professor of Linguistics at the University of São Paulo (USP), Brazil, and Director of the Interdepartmental Centre for Languages (CIL-FFLCH). His research bridges linguistics, cognitive science, and artificial intelligence, focusing on Brazilian Sign Language (Libras), visual language processing, and human interaction with language technologies. His expertise spans psycholinguistics, neurolinguistics, language acquisition, and clinical linguistics, particularly atypical language development and language disorders in deaf signers. As a researcher at the Science for Development Center - Assistive Technologies and Brazilian Sign Language Accessibility (CCD-TAAL), he investigates automatic translation between Portuguese and Libras, signing avatars, and the cognitive mechanisms underlying the perception and comprehension of machine-generated signing. His experimental work employs eye-tracking and other behavioural methodologies to explore visual attention, linguistic processing, and comprehension. His research seeks to integrate insights from human language processing into the development of more natural, linguistically accurate, and accessible AI-driven sign language technologies.