An avatar that communicates in Italian Sign Language
13.03.2025
In Italy, over 40.000 deaf individuals use Italian Sign Language (LIS) as their primary mode of communication, with thousands more learning it for work or personal reasons. Despite this widespread use, there are still limited technological solutions available to make web content accessible to this community.
Together with some collegues, we have been working on a project for the past year, aiming to bridge this gap by creating an AI-powered avatar capable of automatically translating text into LIS. This will extend accessibility to websites, enhancing inclusion for the deaf community.
Our approach differs from other similar solutions developed for different sign languages, as it focuses on the Italian Sign Language, a language with a unique syntax and grammar distinct from Italian. One of the biggest challenges we faced was the lack of well-structured datasets mapping LIS to Italian.
Our solution involves a sophisticated pipeline with the following key steps:
- Speech to Text: The input audio is converted into text (a sentence in Italian).
- Text to LIS Gloss Translation: The converted sentence is transformed into LIS syntax (gloss) using a large language model (LLM).
- Gloss to Pose Lookup: Each gloss is mapped to a corresponding pose sequence, which is the critical step in interpreting sign language.
- Pose Sequence Assembly: The poses are assembled into a smooth sequence, with linear interpolation between signs to ensure natural transitions.
- Avatar Generation: Starting from the pose sequence we generate the Avatar that signs the original sentence.
In the future, we plan to expand the dataset further and refine the avatar generation process to improve accuracy and fluency in the automatic translation to LIS.