Reading the ASL alphabet, in the browser
An end-to-end computer-vision project: a MobileNetV2 transfer model that classifies the 26 static letters of the American Sign Language alphabet, exported to ONNX and run entirely client-side. This page is the honest story of how it works and where it falls short.
The problem
What this project is
The American Sign Language alphabet is fingerspelled with 26 distinct static hand shapes (A–Z). The goal here is narrow and concrete: given a single image of a hand, predict which letter it is — and do it fast, privately, and on-device, with no server in the loop.
It is built as a full ML-engineering lifecycle rather than a one-off notebook: dataset ingestion and stratified splitting, augmentation-aware training, rigorous confusion-matrix evaluation, explainability, ONNX export with a numerical parity test, and a static, in-browser demo. The point is to show the whole pipeline done carefully — including being honest about its limits.
Architecture
From pixels to a predicted letter
The classifier is a MobileNetV2 backbone pretrained on ImageNet with its final classifier head replaced by a 26-way head. The ImageNet backbone is frozen for a short warm-up, then the whole network is fine-tuned end-to-end at a 10× lower learning rate. One deliberate choice in the augmentation pipeline: no horizontal flip — ASL signs are not flip-invariant, since pairs like b/d and p/q are mirror images of each other.
Model
In-browser inference flow
Accuracy, honestly
Benchmark vs. real world
Two numbers, told honestly. The headline figures below are benchmark numbers, measured on the held-out test split of the deployed model's training union (Marxulia + aliciiavs + Hemg, ~21k images, 26 classes A–Z). They are real and reproducible with make eval-real — but they describe performance on data that resembles the training set, not the messy real world (see the honest cross-dataset number below).
96.9%
Held-out test accuracy
26 classes A–Z, 3,170 test images (3-source merge)
0.969
Macro F1
averaged equally across all 26 classes
97.3%
Validation accuracy
best epoch during fine-tuning
The honest cross-dataset number
MeasuredThe benchmark above is inflated: train and test images come from the same uniform dataset, so the test set looks like the training set. The number that actually matters is cross-dataset accuracy — evaluated on a different dataset (EitanG98/asl_letters, different signers and real backgrounds) the model never trained on. It is 59.8% on the 24-letter A–Y headline (J and Z are dynamic motion signs a single frame can't capture), or 55.5% across all 26 classes. Reproduce with make eval-realworld-diverse-hemg.
That number was earned by data diversity, the only lever that moved it: single-source 33.4% → add a multi-signer dataset 47.6% → add a third source 55.5%. Preprocessing tricks, augmentation, calibration, and architecture swaps were all measured and found not to help — documented as honest negatives in the repo's docs/.
A robustness experiment, reported honestly
We retrained the same MobileNetV2 with aggressive domain augmentation (wide crops/rotations, strong lighting and contrast jitter, random grayscale, blur, and erasing) aimed at the cluttered-webcam case — reproducible with make train-robust.
Measured on the same held-out test split, the robust model scored 92.3% versus the then-baseline's ~97% — heavy augmentation cost ~4.5 points on the benchmark. Once we had a real cross-dataset test set, augmentation proved neutral there too: it was data diversity, not augmentation or architecture, that moved the honest number. The deployed model is now the diverse multi-source one — chosen because it measured better on the cross-dataset gate, not because we hoped it was better.
Explainability
What is the model looking at?
Grad-CAM highlights the image regions the network leaned on for its prediction — a sanity check that the model attends to the hand, not the background. Below is the real saliency overlay for class V, produced by src/gradcam.py.
This is one real overlay, not a full gallery — overlays for any class can be generated locally with make gradcam.
In-browser inference
How it runs without a server
There is no inference backend. The trained model is exported to ONNX and executed in your browser tab, so every prediction happens locally.
Runs with onnxruntime-web
The model is exported to ONNX and executed by onnxruntime-web, which runs the network via a WebAssembly (WASM) backend — with a WebGPU path where the browser supports it.
Frames never leave your device
Webcam frames and uploaded images are preprocessed and classified entirely in the browser tab. No pixels are uploaded anywhere — the inference is fully client-side and private.
No server, static deploy
The whole site is a static export: plain HTML, JS, and the .onnx weights served as files. There is no inference backend to run, scale, or pay for.
Low latency, on-device
Because there is no network round-trip, predictions update locally as fast as the device can run the forward pass — no API calls per frame.
Under the hood
Tech stack
- Modeling & training
- PyTorch 2.xtorchvisionMobileNetV2 (transfer learning)AdamW + cosine LRCUDA / Apple-Silicon MPS / CPU
- Evaluation & explainability
- scikit-learn (confusion matrix, F1)Grad-CAMdistribution-shift suite
- Serving & in-browser inference
- ONNX exportonnxruntime-web (WASM / WebGPU)MediaPipe hand crop
- Web showcase
- Next.js 15 (App Router)React 19TypeScript (strict)Tailwind CSS
- Engineering
- GitHub Actions CIpytest + coverage gateruff + black + mypy
Honesty
Ethics & limitations
A model is only as trustworthy as the caveats shipped with it. These are drawn directly from the project’s model card.
Limitations
Not an accessibility or medical tool
This is an educational / portfolio demonstration of a computer-vision pipeline. It is not a certified accessibility, communication, medical, legal, or otherwise safety-critical product and must not be relied on as one.
Alphabet only, static frames
It classifies single still frames of the ASL alphabet. It does not recognize continuous signing, words, or grammar, and the motion-based letters (J, Z) are inherently ambiguous as single frames.
Benchmark optimism
The training data is fairly homogeneous, so a random split can leak near-duplicate frames across train and test. The benchmark accuracy is therefore optimistic; a group-aware split would lower it.
Shared hand shapes get confused
Letters that share a hand shape are the usual error sources — P/Q, V/W, and M/N/S are classic confusions.
Ethical considerations
Demographic representation
The training data appears to feature a narrow range of skin tones and a single signing environment, so the model will likely underperform for under-represented skin tones and settings. Any production or accessibility use would require a diverse, consented, group-split dataset and per-group fairness evaluation.
Respect for the Deaf community
ASL is a complete language with its own grammar; alphabet fingerspelling is a small part of it. This project should not be presented as "translating ASL." Accessibility tools should be built with the Deaf and hard-of-hearing community, not merely for it.
Honest reporting
Benchmark accuracy is reported alongside its leakage caveat specifically to avoid overstating real-world capability.
More