The first photo a user sent us was a nightmare. Curved plastic, glare from fluorescent lights, their thumb covering half the ingredient list. The text read "INGRED1ENTS: wat3r, sug4r..." and our OCR system dutifully reported those exact characters. We had work to do.
That was eighteen months ago. Today, HalalLens processes thousands of ingredient photos daily, extracts text with 94% accuracy, and does it all in under a second. This is how we got there.
Why We Ditched Cloud OCR
Our first prototype used Google Cloud Vision. Great accuracy, easy integration, reasonable pricing. One problem: it took 2-3 seconds per image. Users would snap a photo, wait, wait some more, and then give up. Our analytics showed 40% abandonment during that loading screen.
There was also the privacy question. Every ingredient photo going to Google's servers meant we were building a detailed record of what products our users were considering buying. Even anonymized, that felt wrong.
We switched to Google ML Kit—the on-device version. The image never leaves your phone. Processing happens in 150-250 milliseconds. That switch alone cut our abandonment rate in half.
The Pipeline
When you snap a photo, here's what happens in that half-second:
| Step | What Happens | Time |
|---|---|---|
| Capture | Camera grabs the frame | ~100ms |
| Quality Check | Is it bright enough? Sharp enough? | ~50ms |
| Preprocessing | Boost contrast, fix exposure if needed | ~100ms |
| ML Kit OCR | Neural network extracts text | ~200ms |
| Clustering | Find the ingredients (not the brand name) | ~50ms |
Total: under 600 milliseconds. Users see results before their thumb leaves the shutter button.
Finding the Ingredients
A product label has a lot of text. Brand name. Marketing claims. Nutrition facts. "Best by" dates. We only care about the ingredient list, and finding it automatically is harder than it sounds.
We use DBSCAN clustering—a fancy way of saying "group text blocks that are close together." Ingredients typically appear in a dense block with consistent font size. The brand name is off by itself in big letters. Nutrition facts have a distinctive table structure.
The clustering algorithm looks for:
• Dense blocks of similarly-sized text
• Headers like "Ingredients:" or "CONTAINS:"
• Comma-separated patterns
• Consistent language within the block
It works about 92% of the time. When it fails, it's usually because the label designer did something creative—ingredients running vertically, split across two sides of the package, that kind of thing.
When OCR Gets It Wrong
Even good OCR makes mistakes. "gelatin" becomes "ge1atin" (one instead of L). "sugar" becomes "suqar" (Q instead of G). These errors are predictable—they follow patterns based on how similar letters look.
We maintain a correction dictionary:
• 0 → o (zero to letter O)
• 1 → l or i
• rn → m
• cl → d
When a word doesn't match any known ingredient, we try these substitutions before giving up. "ge1atin" fails to match, we try "gelatin"—match found.
For more complex errors, we're training a T5 model on 287,000+ pairs of OCR output and corrected text from OpenFoodFacts. It learns the specific ways that ingredient labels get misread and how to fix them.
Confidence Scoring
ML Kit gives us a confidence score for each word. Anything below 80% gets flagged. We show these uncertain words to users with a yellow highlight: "We're not sure about this one—can you verify?"
Better to ask than to silently misread "pork" as "pcrk" and tell someone their food is halal.
The goal isn't perfect OCR. It's knowing when we're not perfect.
