Machine Translation for Sign Languages

cs.CV Ozge Mercanoglu Sincan, Anton Pelykh, Edward Fish, Harry Walsh, JianHe Low, Karahan Sahin, Oline Ranum, Sobhan Asasi, Steven Emery, Richard Bowden · Oct 1, 2026
Local to this browser
What it does
Sign language machine translation remains far behind spoken-language MT because sign languages exploit simultaneous manual and non-manual channels, spatial grammar, and three-dimensional construction. This survey reviews the technical arc...
Why it matters
This survey reviews the technical arc from isolated recognition to end-to-end translation and production, embedding every stage in sign language linguistics and deaf-community ethics. The authors argue that progress must be judged not by...
Main concern
This is an unusually comprehensive and ethically grounded survey. The authors—an interdisciplinary hearing and deaf team—succeed in balancing computer vision advances with linguistic and stakeholder perspectives, producing a review that...
Community signal
0
0 up · 0 down
Sign in to vote with arrows
AI Review AI reviewed
Plain-language introduction

Sign language machine translation remains far behind spoken-language MT because sign languages exploit simultaneous manual and non-manual channels, spatial grammar, and three-dimensional construction. This survey reviews the technical arc from isolated recognition to end-to-end translation and production, embedding every stage in sign language linguistics and deaf-community ethics. The authors argue that progress must be judged not by benchmark scores alone but by whether deaf users actually accept and understand the output, making the manuscript as much a piece of technology assessment as a call for sociotechnical reform.

Critical review
Verdict
Bottom line

This is an unusually comprehensive and ethically grounded survey. The authors—an interdisciplinary hearing and deaf team—succeed in balancing computer vision advances with linguistic and stakeholder perspectives, producing a review that reads as a call for sociotechnical rigor rather than a simple literature inventory. Its central thesis, that optimizing benchmark metrics without community validation targets the wrong objective, is convincingly threaded through every section. The manuscript is descriptive rather than evaluative in places, but its framing around data sovereignty and community governance sets a higher standard for the field.

“Systems that achieve high scores but produce signing that deaf users find unnatural or incomprehensible have not solved the sign language machine translation problem–they have merely optimized for the wrong objective.”
Mercanoglu Sincan et al. · Conclusion
“Across regions, participants emphasized the importance of DHH-led design, citing the risk of harm when DHH communities are excluded from technological decision-making.”
What holds up

The linguistic primer in Section 2 is essential reading for vision researchers, clearly explaining simultaneity, classifier constructions, and spatial reference. The ethics section moves beyond generic AI principles to concrete governance requirements—accessible informed consent in native sign languages, fair remuneration, and NIL policies for avatar creation—grounded in deaf community position papers. Table 1 provides a valuable, compact overview of datasets across modality, language, and scale. The technical history is thorough, tracing the shift from gloss-based pipelines to gloss-free transformers and diffusion-based production without losing sight of why glosses were problematic in the first place.

“accessible, informed consent provided in a native sign language, not just written consent forms”
Mercanoglu Sincan et al. · Section 3 (Ethics and Safety)
“Gloss representations abstract away much of the richness of signing, including non-manual features, non-lexical signs, spatial modulation, and depicting construction”
Mercanoglu Sincan et al. · Section 4.3
Main concerns

As a review, the paper necessarily summarizes claims from primary sources without independent experimental verification, and it occasionally lapses into cataloguing architectures rather than critically synthesizing which design choices actually improve generalization. The authors note that gains are often tightly coupled to specific datasets, yet they do not perform a quantitative meta-analysis or systematically compare reported results under controlled conditions. Future directions in Section 6 are sensible—better evaluation, more conversational data, and modeling of non-manual features—but largely rehearse established gaps rather than proposing specific hypotheses or prioritization. The manuscript also lacks an explicit literature selection protocol, which makes it difficult to assess whether the coverage is systematic or shaped by author familiarity.

“Reported gains are often tightly coupled to specific datasets, domains, and recording conditions”
Mercanoglu Sincan et al. · Introduction
“Despite this growth, a disconnect remains between what systems achieve on benchmarks and what they deliver in practice”
Mercanoglu Sincan et al. · Conclusion
Evidence and comparison

The evidence base is admirably broad, drawing on computer vision, sign language linguistics, and deaf studies. Comparisons with spoken-language MT are consistently explicit about modality differences—simultaneity, spatial grammar, and the absence of abundant text corpora—rather than treating sign languages as noisy speech. However, the technical evidence relies on reported benchmark numbers from individual papers without controlling for data leakage, compute, preprocessing pipelines, or annotation quality, an issue the authors acknowledge but do not correct. By contrast, the ethical argumentation is stronger because it synthesizes multiple independent community sources (WFD, BDA, COSET) and at least one global survey (Atwell et al.).

“In contrast to spoken-language machine translation, where datasets often contain tens or hundreds of millions of sentence pairs, existing datasets are orders of magnitude smaller”
Mercanoglu Sincan et al. · Section 5 (Technical Challenges)
“The metrics used to measure progress tell us whether output matches a reference, but not whether it would be understood or accepted by a deaf user”
Mercanoglu Sincan et al. · Conclusion
Reproducibility

Because this is a review, reproducibility concerns shift from experimental replication to the transparency of synthesis. The authors do not state a systematic search or inclusion protocol, so it is unclear whether the surveyed literature is exhaustive or selectively curated. More importantly, the paper itself highlights why reproducing state-of-the-art sign language systems is so difficult: many core preprocessing steps—pose extraction, signer detection, background removal—introduce uncontrolled error, and benchmark datasets vary in annotation granularity from gloss-level to sentence-level only. The authors note that evaluation metrics like BLEU penalize valid paraphrases and that no agreed metric exists for production quality, meaning independent researchers cannot easily verify whether newer methods are genuinely better.

“Many factors impact inference performance, including visual pre-processing such as pose extraction, active signer identification, background removal, and cropping”
Mercanoglu Sincan et al. · Section 5
“BLEU and similar n-gram overlap metrics penalize valid paraphrases and alternative word orders, which is a particular concern for sign-to-text translation given the flexible phrase structure of sign languages”
Mercanoglu Sincan et al. · Section 5
Abstract

Sign language machine translation has progressed substantially over the past decade, evolving from isolated sign recognition to end-to-end translation systems. Advances in pose estimation, transformer architectures, and large-scale dataset collection have driven progress, yet challenges remain. Datasets are limited compared to spoken-language resources; evaluation metrics inadequately capture the linguistic quality of output; and models must capture the simultaneous, multi-layered, and three-dimensional structure of sign languages. This manuscript provides a comprehensive review that seeks to balance technical challenges with stakeholder considerations. We examine the linguistic properties that make sign languages computationally unique, trace the evolution of recognition, translation, and production systems, and analyze ongoing technical challenges. Crucially, we address ethical considerations around data governance, community involvement, and appropriate use. Drawing on interdisciplinary perspectives spanning computer vision, sign language linguistics, and deaf studies, our analysis emphasizes that continued progress requires sustained collaboration across these fields and with deaf communities.

Challenge the Review

Pick a starting point or write your own. Challenges run in the background, so you can keep reading while the AI investigates.

No challenges yet. Disagree with the review? Ask the AI to revisit a specific claim.