Intel, Nvidia, AMD and Microsoft have all invested serious effort in sign language recognition. That work produced real value: libraries of sign language models, purpose-built hardware, and training architectures that did not exist a decade ago.
We are glad it exists. DeafAI builds on top of that foundation rather than pretending to start from nothing.
But there is a gap between the state of that research and something a Deaf person can rely on to talk to a pharmacist. Understanding that gap is the whole thesis of what we are building.
The averaging problem
Standard sign language libraries are built by collecting signing from many people and learning what is common across them. That is the right way to build a general model, and it produces something that performs reasonably on an average of all signers.
The difficulty is that nobody signs like the average.
Ask any fluent signer and they will tell you that signing carries an accent as distinctive as a voice. Regional vocabulary differs across the country. Generations sign differently. Signing space varies in size, speed varies, handshape precision varies. And every signer carries personal vocabulary, above all name signs for the people in their life, that exists in no library because it was invented inside one family or one workplace.
A generic model performs at its best on signers who resemble its training data, and degrades on everyone else. In a demo, the signer usually resembles the training data. In a pharmacy, the user is themselves. That gap between demo and daily life is where most sign language products lose their users.
And the failure mode is unforgiving. A translation tool that is right most of the time but unpredictable about when it is wrong is not a tool anyone will use for a medical appointment. Trust is the product.
Turning the problem into the solution
The conventional response is to fight variation with scale: gather more signers, cover more variation, average harder. That helps, but it is expensive and it never fully arrives, because it is still trying to build one model for everyone.
We took the opposite approach. Rather than treating individual variation as noise to be averaged away, we treat it as the signal.
The DeafAI Virtual Personal Interpreter is built to learn you.
How it works in practice
When someone sets up the app, it presents words and phrases and asks them to sign each to the camera. Those clips are analysed to build a model of that person's individual signing style: their speed, their handshapes, their regional variants, the specific way they produce each sign.
That style model is then applied to standard signs to simulate how that person would produce them, generating large volumes of synthetic training data from a small amount of real signing.
That last step is what makes personalization practical rather than theoretical. Without it, tailoring a model to an individual would demand hours of recording from every single user, which nobody would sit through. By learning the style and then projecting it across a full vocabulary, a manageable setup session produces a personalised model.
Once vocabulary building is complete, the app is ready to interpret that person's signing into spoken English. And because it keeps learning from use, it improves with time rather than plateauing.
Why this changes the outcome
Personalization is not a feature bolted onto the side. It changes the properties that determine whether a tool gets used.
- Accuracy on the signing that actually matters. Not accuracy on a benchmark, accuracy on this user in this conversation.
- Speed. A model with a narrower, better-specified problem can respond faster, and response time is what separates a conversation from an exchange of messages.
- Personal vocabulary. Name signs and household vocabulary become usable, which is a large fraction of what people actually talk about.
- Context tuning. Home, school, work and social settings have different vocabularies, and the tool can be tuned for each.
- Improvement with use. Every session makes the next one better, which inverts the usual pattern where enthusiasm fades as limitations surface.
The virtuous circle
There is a second benefit, and it is the reason this approach compounds.
Every Deaf person who uses a personalised system is, with their participation, contributing to the understanding of how real signing varies across real people. The training data problem that constrains this entire field is precisely a shortage of diverse, well-understood signing data.
An approach that requires individual signing data in order to work, and improves for that individual as a result, aligns the user's interest with the system's improvement. People are not donating data for someone else's benefit. They get a better interpreter, immediately, in exchange.
Where we are
The Virtual Personal Interpreter is in beta testing right now with Deaf users training it on fingerspelling and selected phrases. It is not finished. We are explicit about that because a technology like this has been over-promised before, and we would rather be trusted than impressive.
What stands between a working beta and an app any Deaf person can download is engineering time. That is what donations and grants pay for, and it is the entire reason we ask.
If you sign ASL and want to shape how this works, we are recruiting beta testers. The people who tell us what breaks are doing the most valuable work in this project.
Frequently asked questions
What is a Virtual Personal Interpreter?
It is the DeafAI application: software that uses the camera on a phone, tablet or laptop to interpret a Deaf user's signing into spoken English and text in real time, and converts the hearing person's spoken reply back into text and sign language. It personalises itself to how each individual user signs.
How does DeafAI learn an individual's signing style?
During setup, the app presents words and phrases for the user to sign to the camera. It analyses those clips to model that person's individual signing style, then applies that style to standard signs to generate synthetic training data, producing a personalised model from a manageable amount of real recording.
Why not just train one large model on everyone?
General models are optimised for an average signer who does not exist in practice, so they perform best on people resembling their training data and worse on everyone else. Personalization treats individual variation as useful signal rather than noise to average away.
Is the DeafAI app available now?
The Virtual Personal Interpreter is in beta testing with Deaf users and is not yet publicly available. DeafAI is recruiting Deaf beta testers, and donations fund the engineering work required to reach a public release.
Help us build this.
DeafAI is a 501(c)(3) nonprofit. The Virtual Personal Interpreter is in beta with Deaf users, and donations fund the engineering that takes it public. Every gift is tax deductible.
Donate today