Document Extraction with VLMs: PDFs and Scans to Structured JSON

What you need to know

Every business runs on documents that were never meant for a computer to read — an invoice a supplier keyed into their own template, a scanned KYC packet a customer photographed on a phone, a UK bank statement exported to PDF with the numbers locked inside a rendered table. Turning that mess into clean, structured JSON your systems can act on is one of the oldest problems in enterprise software, and for the first time it is genuinely, boringly solvable. Vision-language models (VLMs) read a page the way a person does — layout, tables, stamps, handwriting and all — and hand you fields instead of pixels. The catch is that a model confident enough to read a smudged total is also confident enough to invent one, so the engineering that matters is no longer the reading. It is the scaffolding around it.

This guide is the scaffolding. It is a reference pipeline you can build once and point at Indian GST invoices, UK financial-services statements, purchase orders, insurance forms or anything else: parse the layout, extract against a strict schema, validate, and route the uncertain cases to a human. Here is the shape of it before we go deep:

• Schema first, always. Define the output contract as a typed model and make the model fill it in — never parse free-form prose back into fields.
• Choose the engine per document, not per project. Cloud Document AI, a frontier VLM and a self-hosted open VLM each win on different documents and at different price points.
• Constrain the output. Use native JSON-Schema modes where they exist, and tool-use contracts where they do not.
• Assume hallucination. Confidence-gate every field, prefer null over a guess, and use multi-backend consensus voting to catch invented values.
• Evaluate properly. Field precision and recall for the values, Tree-Edit-Distance Score (TEDS) for tables — never a single "accuracy" number.

Why VLMs changed document extraction

For two decades the standard document pipeline was optical character recognition (OCR) followed by rules: run Tesseract or a cloud OCR engine, get a bag of words with bounding boxes, then write regular expressions and per-vendor templates to pull the invoice number out of the top-right and the total out of the bottom. This works, and where layouts are fixed it still works well. The problem is generalisation. Every new supplier, every rotated phone photo, every two-column layout or second-language form needs a new template, and maintenance cost grows faster than coverage. A team in Pune onboarding a hundred new vendors, or a UK lender ingesting statements from thirty different banks, ends up maintaining more template code than product code.

Vision-language models collapse that. They read the page as an image and reason over layout directly, so a single prompt handles invoices from vendors it has never seen, in formats no one templated. The benchmark that tracks this is OmniDocBench, now the de-facto standard for document parsing. As of the June 2026 leaderboard, which ranks 16 models, GLM-OCR leads the general-purpose pack at 94.62, with Gemini 3.1 Pro at 90.33 and Claude Opus 4.6 at 87.1; a sub-1.3B open model, PaddleOCR-VL, tops the board at around 96.33. Those headline numbers are worth reading with a sceptical eye — the highest scores are vendor self-reported and have not been independently reproduced, so treat the leaderboard as a shortlist generator, not a purchase order.

The more important lesson hides underneath the ranking. On clean printed text the field is bunched: Azure Document Intelligence lands around 96 per cent, with GPT-5, Gemini 2.5 Pro, Google Vision and Amazon Textract all around 95 per cent. When your documents are crisp, model choice barely matters and you should optimise for cost. The gaps open up precisely where business documents live — tables, scans, handwriting and multilingual pages — which is why the rest of this guide spends most of its time there. If you have already built retrieval on top of extracted documents, the same discipline that makes chunking and embedding strategies reliable applies here: structure beats cleverness.

The reference pipeline: parse, extract, validate, review

A document-extraction system that survives contact with production has four stages, and the value is in keeping them separate so each can be swapped, measured and gated independently.

Stage one — layout-aware parse. Turn the raw PDF or image into a normalised, model-friendly representation: page images at a sensible resolution, plus a text or Markdown layer with table structure preserved. Tools such as Docling, PyMuPDF4LLM, Unstructured and Mistral OCR live here; Mistral OCR (model mistral-ocr-2512) processes up to 2,000 pages per minute on a single GPU and returns Markdown with HTML table reconstruction, which is exactly the shape a downstream extractor wants. This stage also handles rotation, deskew, page splitting and de-duplication.

Stage two — structured extract. Hand the parsed page and a strict schema to a VLM and get back a typed object. This is the stage where schema-first prompting, covered in the next two sections, does the heavy lifting.

Stage three — validate. Re-validate the returned object against the same schema, then run business rules: does the line-item sum match the stated total, is the GSTIN 15 characters, is the invoice date within a plausible range, is the sort code six digits. Cheap deterministic checks catch a surprising share of model errors before a human ever sees them.

Stage four — review. Route by confidence. High-confidence, rule-passing documents flow straight through; anything below your threshold, or anything where two extraction backends disagree, lands in a human review queue. The reviewer's corrections become golden-set data for your evals. This is the same architecture used for any high-stakes generation task, and it pairs naturally with the retrieval patterns in our production RAG hybrid-retrieval guide when extraction feeds a downstream agent.

Choosing your engine: cloud Document AI vs frontier VLM vs self-hosted open VLM

There is no single best engine, only the best engine for a given document at a given volume. The three broad classes trade accuracy, cost and control against each other. Pricing as of July 2026 is summarised below — read the accuracy column as a profile, not a promise, because the numbers move with document type.

Two practical notes. First, the cost columns are not comparing like with like: pre-built OCR that only transcribes text is an order of magnitude cheaper than form or field extraction that also structures the output, so map your requirement carefully before quoting a number to finance. Second, the self-hosted economics are compelling only at scale — the ~167× per-page saving assumes you are keeping GPUs busy; for a few thousand documents a month, a managed API is cheaper once you count engineering time. Most mature pipelines end up hybrid: a cheap engine for the easy majority, a stronger one held in reserve for the documents your confidence gate flags as hard.

Prompting a VLM for structured JSON: responseSchema vs tool-use contracts

The single most important design decision is to make the schema the contract, not the prompt. You never want the model to return prose that you then parse with a regex — that reintroduces the fragility VLMs were meant to remove. Instead, define the output as a typed model once, export it to JSON Schema, and use the strongest constraint the provider gives you.

The mechanics differ by provider, and the difference is worth knowing. Gemini 2.5 and later support a native responseSchema (JSON-Schema) mode that constrains the generation to your structure directly. As of April 2026, Claude does not expose a native json_schema response format; the recommended path is tool-use, where you pass your schema as a tool's input_schema and that becomes the enforced contract — an approach that achieves higher schema-validity than asking for prompt-shaped JSON. Both routes start from the same Pydantic-style model, so you write the contract once and adapt only the call. The example below extracts an invoice with a nullable GSTIN, the kind of field an Indian pipeline needs and a UK one leaves empty.

from pydantic import BaseModel, Field
from typing import Literal
import anthropic

# 1. Define the contract ONCE as a typed model.
class LineItem(BaseModel):
    description: str
    quantity: float
    unit_price: float
    line_total: float

class Invoice(BaseModel):
    invoice_number: str = Field(description="Supplier's invoice ID")
    invoice_date: str | None = Field(default=None, description="ISO 8601 date, null if absent")
    supplier_gstin: str | None = Field(default=None, description="15-char GSTIN if present, else null")
    currency: Literal["INR", "GBP", "USD", "EUR"]
    line_items: list[LineItem]
    total_amount: float

client = anthropic.Anthropic()

# 2. Claude has no native json_schema mode (as of April 2026): pass the
#    schema as a TOOL so input_schema becomes the enforced contract.
resp = client.messages.create(
    model="claude-opus-4-6",
    max_tokens=2000,
    tools=[{
        "name": "emit_invoice",
        "description": "Return the extracted invoice as structured data.",
        "input_schema": Invoice.model_json_schema(),
    }],
    tool_choice={"type": "tool", "name": "emit_invoice"},
    messages=[{
        "role": "user",
        "content": [
            {"type": "document", "source": scanned_pdf_page},   # the page
            {"type": "text", "text":
                "Extract every field. If a field is not visible on the page, "
                "return null. Never guess a value you cannot read."},
        ],
    }],
)

# 3. The tool_use input is guaranteed to match the schema SHAPE...
raw = next(b.input for b in resp.content if b.type == "tool_use")
invoice = Invoice.model_validate(raw)   # ...but re-validate as a second defence.

# --- Gemini equivalent: native responseSchema (JSON-Schema mode) ---
# model.generate_content(
#     [pdf_part, "Extract the invoice."],
#     generation_config={"response_mime_type": "application/json",
#                        "response_schema": Invoice})

Three details make this robust. The schema forces optional fields to be nullable so an absent value is a legal answer, not a reason to hallucinate. The prompt explicitly instructs the model to return null rather than guess — a small phrase that measurably reduces invented fields. And the returned object is re-validated with model_validate, because tool-use guarantees the shape but not that total_amount equals the sum of the line items. For a deeper treatment of the prompt patterns, see our guides on structured-output prompting patterns and on reliable JSON with constrained decoding, both of which generalise beyond documents. Purpose-built extraction models fit here too: NuExtract3, a self-hostable 4B extraction VLM, takes a template and returns structured output directly, and IBM's Granite-Docling — a 258M-parameter, Apache-2.0 VLM — targets the parse stage on commodity hardware.

The hard parts: tables, multi-page documents, handwriting and multilingual scans

The easy 80 per cent of document extraction is solved. The remaining 20 per cent is where projects succeed or quietly fail, and it clusters into four hard problems.

Tables are the biggest differentiator

Tables are the hardest sub-task and the widest gap between engines. Mistral OCR reports 96.6 per cent table accuracy against AWS Textract's 84.8 per cent; yet on a separate purchase-order comparison Textract hit 82 per cent line-item detection while Google Document AI's table parser dropped to around 40 per cent. Read those two facts together: table accuracy is document-type-specific, not a fixed ranking. The engine that wins on your invoices may lose on your bank statements. This is precisely why you never trust a single leaderboard number — you benchmark on your own tables, and you score them with a table-aware metric (TEDS), covered in the evals section, rather than raw text accuracy.

Multi-page documents strain context and recall

Long documents introduce two failure modes: the model runs out of attention and misses entities late in the document, and it loses the relationship between a value on page three and the header that defines it on page one. Large models are not immune — one study found Pixtral 12B at 17 to 26 per cent entity recall where a smaller, better-targeted model reached 38 per cent. The mitigation is architectural: split by logical unit (one invoice, one form) rather than by arbitrary page count, extract per unit, and reconcile. Do not assume a bigger context window fixes recall.

Handwriting and multilingual scans

Handwriting, stamps, low-resolution phone photographs and mixed-script documents — a Hindi-and-English delivery note, a Welsh-and-English utility bill — are where general OCR degrades and reasoning-capable VLMs earn their cost. This is also where hallucination peaks, because the model is filling gaps it cannot actually read. The right posture is defensive: lower your confidence threshold for these classes, prefer null aggressively, and route more of them to human review. NuExtract3's edge here is instructive — it beat GPT-4o specifically on messy scans by returning null instead of a confident fabrication.

Failure modes and how to catch them

Because the failures are silent — well-formed JSON that happens to be wrong — you have to design detection in, not bolt it on. The table below maps the failure modes that actually occur in production to how you catch each one and what you do about it.

Confidence scoring and human-in-the-loop

No extraction pipeline should aim for zero human involvement; it should aim to spend human attention only where it changes an outcome. Confidence scoring is the mechanism. Because most VLMs do not hand you a calibrated per-field probability, you construct confidence from signals you can observe: agreement between backends in the consensus vote, whether business rules pass (line items sum to the total, a date parses, a checksum validates), whether required fields came back non-null, and how the document class historically performs in your evals. Combine those into a per-document score and set a threshold.

Above the threshold, the document flows straight through. Below it — or wherever two backends disagree — it enters a review queue where a person confirms or corrects the fields. Two design rules make this pay off. First, calibrate the threshold to your risk: a low-value expense receipt can flow through at a lower confidence than a KYC document that gates account opening, so an Indian fintech onboarding customers and a UK lender approving a mortgage will set very different bars for the same field type. Second, capture every human correction as labelled data — it is the cheapest golden-set expansion you will ever get, and it feeds directly into the evals that tell you when a lower threshold, or a cheaper engine, becomes safe. This is the same feedback loop that makes any judged system improve; our evaluation-suite guide on golden sets and judges covers the general machinery.

Evals: golden sets, field precision and recall, TEDS and regression gates

You cannot manage what you measure with a single number, and "accuracy" is exactly that trap for document extraction. A document has fields and it has tables, and they fail differently, so you score them separately.

Field-level precision and recall. For the scalar fields — invoice number, total, dates, tax IDs — measure precision (of the fields you returned, how many were right) and recall (of the fields that were present, how many you found) against a golden set of hand-labelled documents that mirrors your real distribution: your vendors, your languages, your scan quality. Precision and recall answer different questions — a hallucination-prone model tanks precision, a model that gives up on messy scans tanks recall — and averaging them into one score hides the trade-off you most need to see.

Tables with TEDS. Table structure is scored with Tree-Edit-Distance Score (TEDS), which compares the predicted table tree to the ground-truth tree and is distinct from raw text character- or word-error rate (CER/WER). A model can transcribe every cell's text perfectly and still get the structure wrong — a merged cell, a shifted column — which CER would miss entirely and TEDS catches. Frameworks such as SO-Bench (structural-output evaluation) formalise this alongside per-field metrics. Report TEDS for tables and precision/recall for fields as separate lines, never as one blended "accuracy".

Regression gates. The evals only protect you if they run automatically. Freeze a golden set, compute field precision/recall and TEDS on every model or prompt change, and fail the build if any metric drops below its gate — the same CI discipline you would apply to any other quality-critical system. When you swap the frontier VLM for a cheaper open model to cut cost, the gate is what tells you whether table TEDS fell off a cliff before the change reaches a customer's KYC packet. Because the leaderboard scores are vendor self-reported and unreproduced, your own regression suite on your own documents is the only benchmark that actually governs what ships.

Data protection: DPDP and UK GDPR for scanned documents

The documents this pipeline touches are, almost by definition, full of personal data — an Indian invoice carrying a customer's GSTIN and address, a KYC packet with a PAN or Aadhaar reference, a UK financial-services statement with names, account numbers and sort codes. That places the whole flow inside a data-protection regime on both sides of the AITC audience, and it is worth being precise rather than hand-wavy about it. This is not legal advice; take your own.

In India, the Digital Personal Data Protection (DPDP) Act governs how you process digital personal data — the image you send to a model, the fields you extract, and the copies you keep are all in scope. For UK and EU data, UK GDPR and the EU framework apply the same way. The practical controls are familiar and mostly architectural: process on a lawful basis and minimise what you send, so if the extraction only needs the invoice total you do not ship the whole page of personal data; prefer a data-processing agreement, a region-locked deployment, or a self-hosted open VLM for the most sensitive classes, which is a strong reason to keep a self-hosted engine in your hybrid stack; redact or tokenise identifiers the pipeline does not need; and treat captured images and extracted fields as regulated assets — access-controlled, retention-limited, and never left lying in application logs or a shared notebook. The engine-routing you built for accuracy doubles as a compliance control: sensitive documents can be pinned to the self-hosted path so the image never crosses a border.

The bottom line

Document extraction stopped being a template-maintenance problem and became a reliability-engineering one. Vision-language models read invoices, forms and scans the way a person does, which removes the per-vendor brittleness that made the old pipelines so costly — but they will confidently invent a field before they will admit they cannot read one. So the work that matters is the scaffolding: define the output as a schema and make the model fill it in with a native responseSchema mode or a tool-use contract; choose your engine per document, because cloud Document AI, frontier VLMs and self-hosted open models each win on different pages and price points; assume hallucination and catch it with confidence gating, null-preference and multi-backend consensus voting; and measure with field precision/recall and TEDS, gated in CI, rather than a single accuracy number lifted off a self-reported leaderboard. Build the interface, not the model — the model is a swappable part, and the rigour that keeps a KYC pipeline in Bengaluru or a statement-processing pipeline in Leeds trustworthy lives in the process, so it survives every model swap you make.