Sign-to-Text Translation Models

Google Pixel 11 Has a Camera-Based Sign-to-Text Feature

At the latest Made By Google Event, Google unveiled the Pixel 11 series and other hardware launches, and introduced a breakthrough sign-to-text translation model. The massively multilingual sign-language-to-text (SL2T) translation model powers sign-to-text dictation in Gboard and Live Transcribe on Pixel 11, beginning with American Sign Language to English, and additional languages are set to follow.

In the same way that users can dictate instead of type, this feature lets deaf users sign to their phone to search the web, draft messages or documents, and get assistance from Gemini. With Live Transcribe, users can also sign responses in conversations rather than having to go back and forth, resulting in a more natural experience that doesn't hinge on texting in English. For Privacy, Google promises "SL2T sees sign language as a sequence of pose landmark locations rather than a raw camera feed."

Open at source ↗