Documents lie about being documents
Building a multimodal context engine — scanned PDFs, screenshots, charts, audio — and why 'just extract the text' is where pipelines go to die.
The dirty secret of enterprise RAG is that the corpus is not text. It claims to be text. It has file extensions that suggest text. But open a random sample of what an organization actually uploads and you’ll find scanned contracts photographed at an angle, invoices as JPEGs inside PDFs, org charts where all the meaning is in the boxes and arrows, Excel sheets pasted into Word as screenshots, and the occasional voice note that someone swears contains the decision from March.
If your ingestion is “parse PDF, extract text, chunk, embed,” a shocking fraction of your corpus enters the index as either nothing or garbage — and retrieval failures downstream get misdiagnosed as model problems.
Building the Context Engine — the ingestion layer behind both my day-job platform and promptev — taught me to treat ingestion as its own product. A few hard-won rules:
Route by what the file is, not what it says it is
First pass on any file: detect what it actually contains. Digital-native text goes down the cheap path. Scanned or image content goes to OCR — we use Azure Document Intelligence for structured documents (it’s genuinely good at tables and key-value layouts) and a vision model for the weird stuff: photos of whiteboards, charts, UI screenshots. Audio gets transcribed with speaker turns preserved.
The routing itself is boring. Having it at all is what separates pipelines that survive real uploads from ones that don’t.
Tables are not prose
Flatten a table into a sentence stream and you’ve destroyed it — “Q3” is now four hundred tokens away from the number it labels. We extract tables into structured form and store a dual representation: a text rendering for embedding, the structure for when the answer needs actual cells. Same for charts: a vision model writes a description of what the chart shows (“headcount flat, attrition doubling since June”), because that sentence is what someone will ask about. Nobody queries for pixel coordinates.
Provenance or it didn’t happen
Every extracted fragment keeps its origin: file, page, region, extraction method, confidence. When OCR misreads a contract amount — and at volume, it will — provenance is the difference between a two-minute “here’s the source page” answer and a trust crisis. Low-confidence extractions get flagged at ingestion, not discovered in an incident.
Extraction gets evals too
The eval discipline everyone applies to generation, almost nobody applies to ingestion. We keep a golden set of nasty real-world files — the angled scan, the merged-cell monstrosity, the two-column layout with footnotes — and every change to the ingestion pipeline runs against it. Extraction regressions are silent by nature; the golden set is the only reason we catch them before users do.
None of this is glamorous. All of it is why answers cite the right numbers. The model gets the credit, but the pipeline does the work.