The Unsolved Challenge of Cleaning OCR Text from Food Packaging

Every day, millions of Muslims around the world pick up a food product, flip it over, and squint at the tiny ingredient list printed on the back. The text is often crammed into a narrow panel, wrapped around curved surfaces, printed in low contrast, and sometimes partially obscured by folds, glare, or damage. For a human, reading these labels is already a challenge. For a machine — for our OCR pipeline at HalalLens — it is a battlefield we have not yet won.

This article is an honest look at where we stand. We have built a system that can scan a food label, extract text, identify individual ingredients, and determine their halal status — all in a matter of seconds. It works remarkably well on clean, flat, well-lit labels. But real-world packaging is rarely clean, flat, or well-lit. The gap between laboratory accuracy and kitchen-counter reality is where our hardest unsolved problems live, and we think our users deserve to know exactly what those problems are.

Why Food Packaging Is Uniquely Difficult for OCR

Optical Character Recognition has come a long way. Modern OCR engines can read printed documents with near-perfect accuracy. But a food label is not a document. It is a three-dimensional object photographed under uncontrolled conditions by a phone camera held at an arbitrary angle. The text may be printed in four-point font on a shiny metallic wrapper, with the ingredient list continuing around a corner that is physically impossible to capture in a single photograph.

Consider a typical chocolate bar. The ingredient list might read: "Sugar, cocoa butter, whole milk powder, cocoa mass, emulsifier (E471), natural flavors, citric acid." On paper, that is straightforward. On a curved, reflective wrapper photographed under fluorescent grocery-store lighting, the OCR engine might return: "Sugar, coc0a butt3r, whol€ milk powdr, cocoa ma55, emulsifi€r (E47I), natural f1avors, citr1c ac1d." Every single garbled character is a potential misidentification — and in halal verification, a misidentification can mean the difference between a product being flagged as safe or suspicious.

The Cleaning Problem: From Raw OCR to Usable Ingredients

Even when our OCR engine successfully extracts text from a label, the raw output is rarely ready for halal analysis. The text needs to be "cleaned" — a deceptively simple word for an enormously complex process. Cleaning means separating the ingredient list from surrounding text like nutritional information, allergen warnings, and marketing copy. It means splitting a long comma-separated string into individual ingredients. It means recognizing that "E471" and "emulsifier (E471)" and "mono- and diglycerides of fatty acids" all refer to the same substance.

This is where things get genuinely hard. Food labels do not follow a universal format. In Europe, ingredients are typically listed in descending order of weight, separated by commas, with allergens in bold. In the United States, the format is similar but allergen labeling rules differ. In Southeast Asia, labels might appear in Malay, Indonesian, Arabic, and English on the same package. In Scandinavian countries, you might find Norwegian, Swedish, Danish, and Finnish ingredient lists stacked on top of each other, with E-numbers used inconsistently.

Our pipeline currently uses a combination of rule-based parsing and LLM-assisted extraction. The rule-based system handles well-formatted labels reliably: find "Ingredients:", split on commas, normalize each entry. But it fails on edge cases — and in the real world, edge cases are the majority. Nested parenthetical ingredients like "chocolate coating (sugar, cocoa butter, gelatin, glycerin)" must be parsed correctly to identify that gelatin and glycerin are sub-ingredients of the coating, not standalone items. Getting the parenthetical nesting right is surprisingly fragile.

The Multilingual Maze

HalalLens serves users across Scandinavia, Southeast Asia, the Middle East, and Western Europe. A single product scanned in a Norwegian grocery store might have its ingredients listed in Norwegian, Swedish, Danish, and Finnish — sometimes with an Arabic or English section as well. Our OCR engine must not only read the text correctly but also determine which language it is looking at, because the same physical word can mean different things in different languages.

The real nightmare begins when OCR errors interact with multilingual parsing. If the engine misreads a Norwegian "ø" as an "o", the word might accidentally become valid in another language — but with a completely different meaning. Our NLP cleaning pipeline attempts to detect the source language first and then apply language-specific normalization rules, but this detection itself is unreliable when the OCR text is garbled. It becomes a circular problem: you need clean text to detect the language, but you need the language to clean the text.

We have experimented with multiple approaches: running OCR with explicit language hints, using ensemble models that vote on the most likely reading, and post-processing with LLMs that can reason about what a garbled word was "probably" supposed to be. Each approach improves accuracy incrementally, but none has solved the fundamental challenge. A product like E120 (carmine) might be listed as "karmin" in Norwegian, "karmiini" in Finnish, and "cochineal" in English — all on the same label, all needing to be recognized as the same halal-critical ingredient.

Where We Stand Today — And What Comes Next

We want to be transparent about our current accuracy. On well-lit, flat, high-contrast English-language labels, our pipeline achieves strong results — correctly identifying and classifying the vast majority of ingredients. On multilingual, curved, or damaged packaging, accuracy drops noticeably. We compensate for this with confidence scoring: when the system is uncertain about an OCR reading or an ingredient identification, it flags the result and encourages the user to double-check manually.

Our roadmap for improving OCR cleaning includes several promising directions. First, we are exploring vision-language models — AI systems that can look at the raw image and understand context, not just individual characters. Instead of reading "ge1atin" and trying to fix it after the fact, a vision-language model might recognize from the visual context that this is an ingredient list and the word is almost certainly "gelatin." Second, we are building a growing database of known OCR misreadings — a fuzzy matching layer that maps common errors like "E47l" back to "E471" based on patterns we have seen across thousands of scans. Third, we are investing in user feedback loops, allowing our community to correct OCR errors which then train the system to avoid similar mistakes in the future.

None of these solutions will be perfect. The fundamental challenge of reading tiny, curved, multilingual text from a phone camera photograph may never be fully solved — it is a problem at the intersection of optics, typography, linguistics, and cultural variation that pushes against the limits of current technology. But we believe that honest communication about these limitations, combined with steady incremental improvement, is the right approach. Every garbled label we process teaches our system something new. Every user correction makes the next scan more accurate.