Google CEO Sundar Pichai has announced a new artificial intelligence feature for Gboard that converts sign language into written text. This update aims to significantly improve accessibility for people who are deaf or hard of hearing. The new sign-to-text capability launched as part of the "Made by Google" event, integrated into Gboard and Live Transcribe on the new Pixel 11 series of smartphones. The technology allows users to communicate more easily with their phones and with individuals who do not understand sign language.[theprint+3]
Bridging Communication Gaps
The innovative feature is powered by Google DeepMind's sign-language-to-text (SL2T) model. It translates American Sign Language (ASL) captured through the phone's camera into written English text in real time. Users can sign messages, conduct web searches, draft documents, or interact with Google's AI assistant, Gemini, using their natural sign language. The feature is also built into Live Transcribe, enabling users to sign their responses during in-person conversations instead of typing. Google describes this as the sign-language equivalent of voice dictation, making digital interaction more intuitive for sign language users.[theprint+15]
The SL2T model represents a significant leap in accessibility technology, moving sign language translation from research labs into everyday consumer applications. It directly translates from the signer's movements, including facial expressions and physical positioning, which are crucial components of sign languages. This approach avoids an intermediate step of converting signs into written annotations, ensuring a more accurate and comprehensive translation. Google trained the SL2T model using over 100,000 hours of data from more than 50 different sign languages, with about a quarter of that data focused on American Sign Language. This extensive training helps the model understand common structures across various sign languages.[en+9]
Privacy was a key consideration in the development of this feature. The sign-to-text system processes pose landmarks on the device itself. This means that raw camera footage is not sent to Google's servers for translation. Instead, the system identifies geometric coordinates from the signer's movements, and only these coordinates are sent for processing. The original video can then be immediately discarded, helping to protect user privacy. Google has also stated it does not retain logs of what users sign or the text produced unless users explicitly opt into a model evaluation study.[timesofindia+12]
Developed with the Deaf Community
Google developed the SL2T model and the sign-to-text feature in close partnership with the Deaf community. This collaboration ensured that Deaf perspectives were included throughout the entire process, from data collection and testing to evaluation and ongoing development. Sundar Pichai highlighted the importance of this collaboration in a post on X, stating that the feature was "Built in partnership with the Deaf community".[timesofindia+6]
Pichai called the new Gboard update "one of the most important updates" coming out of the Made by Google event. He explained that it can help people who sign ASL communicate more seamlessly with their phones or with those who do not sign. This allows for more natural and direct interaction, overcoming previous communication barriers. Testers for the feature found that signing in ASL was often faster and felt more natural than typing in English. This feedback underscores the potential for the new tool to make everyday smartphone use more intuitive and efficient for sign language users.[timesofindia+9]
The feature is designed to put sign language on par with spoken languages as an input method for smartphones. For many deaf and hard-of-hearing individuals, American Sign Language is their primary language, and typing in English may not be their preferred or most efficient way to communicate. The sign-to-text function addresses this by allowing them to use their native language for digital interactions. This represents a significant step towards more inclusive technology, recognizing sign language as a valid interface for human-computer interaction.[support+5]
Future Plans and Accessibility
The sign-to-text functionality is initially available through Gboard and Live Transcribe on Google's new Pixel 11 series, including the Pixel 11, Pixel 11 Pro, Pixel 11 Pro XL, and Pixel 11 Pro Fold. The Pixel 11 Pro Fold's dual screens are particularly well-suited for real-time translation, with the hands-free form factor making it easier for users to sign. The feature supports both one-handed and two-handed signing, a practical consideration for users holding their phones. Google plans to expand the feature to more devices in the future.[theprint+11]
Beyond American Sign Language, Google also intends to add support for additional sign languages. The company's training data already includes information from over 50 sign languages, laying the groundwork for future expansion. This commitment reflects Google's broader goal of making AI and technology more accessible and inclusive for diverse communities worldwide. The feature is offered at no extra cost, positioning it as a standard accessibility function.[timesofindia+9]
While the sign-to-text feature is a major advancement, Google and its AI Sign Language Advisory Committee describe it as an assistive input tool for low-stakes communication. They caution against using the software to replace qualified human interpreters or to avoid legal accessibility obligations. However, the development paves the way for future advancements, with aspirations for AI that can also generate sign language, creating a continuous circuit between sign, text, speech, and AI. This could redefine how people connect across languages and technologies, offering new possibilities for education, banking, and everyday communication.[mashable+3]
This new Gboard feature marks a significant step in making smartphone technology more accessible and inclusive for the Deaf community.
