People are often surprised that sign language translation is still an unsolved problem. Speech recognition became reliable years ago. Machine translation between written languages is good enough that most of us use it without thinking. So why is turning ASL into English still hard?
The short answer is that almost every intuition carried over from speech recognition is wrong. Sign language translation is not speech recognition with a camera bolted on. It is a fundamentally harder problem, and understanding why is the first step to building something that actually works.
1. ASL is not English on the hands
This is the misconception that sinks most projects before they start. American Sign Language is not a manual encoding of English. It is a complete, independent natural language with its own grammar, its own syntax, and its own idioms. It is no closer to English than Japanese is.
ASL organises meaning differently. Topic often comes first. Time is frequently established at the beginning of a sentence and then holds until changed. Much of what English does with word order and function words, ASL does with space, movement, and simultaneity.
This means a system that recognises individual signs and strings the English equivalents together in the order they appeared does not produce English. It produces something closer to word salad. Genuine translation between two grammars is required, not transcription.
If a product demo shows single signs being labelled one at a time, you are watching sign recognition, not sign language translation. The gap between those two things is most of the actual problem.
2. The face and body carry grammar, not just emotion
In spoken English, facial expression adds emotional colour to words that carry the meaning. In ASL, facial expression and body posture are grammar. These are called non-manual markers, and they do structural work.
Raised eyebrows can mark a yes-or-no question. Furrowed brows can mark a different kind of question entirely. A slight headshake can negate the sign it accompanies. Mouth morphemes modify meaning in ways that have no manual component at all. Body shifts assign different speakers in reported dialogue.
The same handshape and movement can mean different things depending on what the face is doing. A system watching only hands is not missing tone. It is missing the sentence structure.
3. Space is grammatical
ASL signers set up referents in the space around them. Mention a person and place them to your left, another to your right, and the direction a verb travels between those points tells you who did what to whom.
That spatial agreement can persist across an entire conversation. To resolve a pronoun correctly, a model has to remember that the location to the signer's left was established as a particular person several sentences ago. This is a long-range, three-dimensional coreference problem with no clean analogue in spoken language processing.
4. Signs blend into each other
Speech recognition had to solve coarticulation, where sounds blur into their neighbours. Sign language has the same problem in three dimensions and worse.
Hands do not teleport between signs. They travel, and during that travel they are already forming the next handshape while finishing the last. There is no reliable pause marking where one sign ends and the next begins. A model must segment a continuous, fluid motion stream into discrete linguistic units with no clear boundaries, while the two hands may be doing different things simultaneously.
5. Every signer is different
This is the one we consider most underrated, and it is why we have built our entire approach around it.
Signing varies enormously between individuals. Sources of variation include:
- Regional variation. ASL has regional signs much as spoken languages have regional vocabulary. Signs for common concepts genuinely differ across the country.
- Generational variation. Older and younger signers often use different forms for the same concept.
- Personal style. Size of signing space, speed, handedness, how crisply a handshape is formed. Every signer has an accent.
- Personal vocabulary. Name signs for family, friends, and colleagues exist in no dictionary anywhere. They are invented within a community and used constantly.
- Context. The same person signs differently at home, at work, and with close friends, just as speakers shift register.
A model trained purely on standard sign libraries is trained on an average signer. No such person exists. This is a large part of why generic systems demo impressively and then disappoint in real use: the demo signer looks like the training data and the actual user does not.
6. The training data problem
Modern AI is data-hungry. Speech recognition and text translation were transformed by corpora of staggering size, assembled largely by scraping text and audio that already existed.
Sign language has no equivalent windfall. Video of fluent signing, correctly annotated by people qualified to annotate it, is comparatively scarce and expensive to produce. Annotation is genuinely skilled work requiring fluency in the language.
The result is that sign language models are typically trained on datasets orders of magnitude smaller than those behind the language technology people are used to. That single fact explains a great deal about the current state of the field.
What honest progress looks like
None of this means the problem is unsolvable. It means the shortcuts do not work, and it means anyone promising a finished universal sign language translator should be asked hard questions.
The Deaf community has good reason for scepticism here. There is a long history of technology built for Deaf people rather than with them: products designed by people who did not sign, tested by people who did not sign, and marketed with claims that did not survive contact with a fluent user. Well-meaning does not mean useful.
So what does a serious approach look like?
Build with Deaf people, not for them. Deaf signers testing the system, shaping the roadmap, and sitting on the board. This is a commitment we have written down, because it is the difference between a product that works and a product that demos.
Treat translation as translation. Handle ASL grammar as a distinct grammar rather than pretending it maps word by word onto English.
Solve the variation problem head on. Rather than fighting the fact that every signer is different, learn the individual. That is the core of our personalization approach: the app learns how you sign, then works for you specifically.
Scope honestly. Start narrow, be clear about what works today, and expand from a foundation that actually functions. Our app is in beta with real Deaf users precisely because that is the only way to find out what breaks.
The problem is hard. It is also the right problem, and being clear-eyed about the difficulty is what makes it possible to make real progress on it.
Frequently asked questions
Why can't AI translate sign language as well as it transcribes speech?
Sign language translation is a fundamentally different and harder problem. ASL has its own grammar rather than following English word order, uses facial expression and body position as grammatical elements, encodes meaning in three-dimensional space, has no clear boundaries between signs, varies substantially between individuals, and has far less training data available than speech or text.
Is American Sign Language just English with hand signs?
No. ASL is a complete natural language with its own grammar and syntax, structurally independent from English. A system that recognises individual signs and outputs the English equivalents in the same order does not produce correct English.
Why do facial expressions matter in ASL?
In ASL, facial expressions and body posture are grammatical, not merely emotional. Non-manual markers can indicate questions, negation, and other structural information. A system that tracks only the hands misses part of the sentence structure.
Why does personalization matter in sign language AI?
Signing varies by region, generation, and individual, and every signer uses personal vocabulary such as name signs that appear in no dictionary. A model trained only on standard libraries is optimised for an average signer who does not exist, which is why generic systems often perform worse in real use than in demonstrations.
Help us build this.
DeafAI is a 501(c)(3) nonprofit. The Virtual Personal Interpreter is in beta with Deaf users, and donations fund the engineering that takes it public. Every gift is tax deductible.
Donate today