Fine-Tune a Vision-Language Model with LoRA for Document AI

When prompting a VLM stops being enough: the fine-tune decision for document AI

There are two ways to build document AI with vision-language models, and they are different enough to deserve separate guides. The first is to use an off-the-shelf VLM — frontier or open — with a strict output schema, null-preference prompting, confidence gating and a human review queue. We covered that pipeline end to end in our guide to document extraction with VLMs, and for most teams it is the right place to start: no training run, no dataset to label, and a model swap is a config change. This article is the counterpart — what to do when that approach plateaus and you decide to train the model itself.

The plateau has a recognisable shape. Your error analysis shows mistakes concentrating on your documents specifically: the model keeps misreading the CGST/SGST split on a particular family of GST invoices, or drops the box numbers on an HMRC form layout it has clearly never seen, or handles ninety suppliers well and two critical ones badly. Prompt iterations stop moving the metric. The model drifts from your JSON schema often enough that repair logic has become a subsystem. Or the economics flip: at hundreds of thousands of pages a month, per-page API pricing dwarfs the cost of a small self-hosted model. And sometimes the decision is made for you — a KYC pipeline in Mumbai or a lender in Leeds whose data-residency rules mean the documents never leave your own GPUs anyway, so the only question is how good your local model can get.

Fine-tuning a small open VLM answers all four, and the general framework for making this call is our fine-tune decision ladder. The table below compresses it for the document case.

One prerequisite is non-negotiable: an evaluation harness before a training run. If you cannot measure field-level accuracy on a held-out set, you cannot know whether the fine-tune helped, and you will be tempted to judge it by the loss curve — a mistake we deal with in its own section below. The base model we use as the running example is Qwen2.5-VL, which ships in 3B, 7B and 72B sizes and whose technical report explicitly targets "robust structured data extraction from invoices, forms, and tables" — the report's document-parsing training data even represents layouts in an HTML-based format with layout boxes. The recipe transfers to other open VLMs with minor renaming.

Building the dataset: image–JSON pairs, layout diversity and label hygiene

A document fine-tune is only as good as its dataset, and a document dataset is a set of image–JSON pairs: a page image on one side, the exact structured output you want on the other. Three properties matter far more than raw size.

Layout diversity beats volume

The model already knows how to read; what it is learning is your mapping from page to schema. If nine hundred of your thousand training invoices come from three suppliers, you have taught it three templates and it will stumble on the fourth. Inventory your layout families first — supplier templates on the India side, form versions on the UK side — and aim for coverage across all of them, plus the ugly tail: rotated phone photos, low-contrast scans, stamps over text, bilingual pages. Published runs suggest the total needed is modest once diversity is handled: a documented Datature fine-tune of Qwen2.5-VL used 1,250 samples, and gains from low-thousands of pairs are typical for a single document domain. If real documents are scarce, synthetic pages can fill layout gaps — with the filtering discipline from our guide to synthetic data for fine-tuning, because training a document model on unfiltered model-generated labels is how you bake hallucinations in permanently.

Label hygiene is the highest-leverage hour you will spend

Every label error is a lesson you are paying GPU-hours to teach. Validate every ground-truth JSON against the schema before it enters the corpus, and run cheap deterministic checks: line items must sum to the total, a GSTIN must be 15 characters, a UK sort code six digits, dates must parse. Reject anything that fails rather than fixing it silently — a rejected label is a labelling-process bug worth finding. Just as important is the split: hold out entire suppliers or form versions, never random pages. A random split leaks layouts between train and validation and flatters every number you compute later.

As of July 2026, TRL's SFTTrainer consumes vision datasets natively: each record carries an images column and a conversational messages column, and the trainer handles the multimodal preprocessing on the fly. The script below builds exactly that, with the hygiene gates and the supplier-held-out split built in.

import json, random
from pathlib import Path
from datasets import Dataset
from PIL import Image
from pydantic import BaseModel, Field, ValidationError

class LineItem(BaseModel):
    description: str
    quantity: float
    unit_price: float
    line_total: float

class InvoiceLabel(BaseModel):
    invoice_number: str
    invoice_date: str | None = None          # ISO 8601, null if absent
    supplier_name: str
    supplier_gstin: str | None = Field(default=None)   # 15-char GSTIN (India)
    vat_number: str | None = Field(default=None)       # GB-prefixed VAT no. (UK)
    currency: str
    line_items: list[LineItem]
    total_amount: float

PROMPT = ("Extract this document as JSON matching the agreed invoice schema. "
          "Use null for any field not visible on the page. Never guess.")

def build_records(label_dir):
    records, rejected = [], 0
    for label_path in sorted(Path(label_dir).glob("*.json")):
        image_path = label_path.with_suffix(".png")
        if not image_path.exists():
            rejected += 1; continue
        try:
            label = InvoiceLabel.model_validate(json.loads(label_path.read_text()))
        except (ValidationError, json.JSONDecodeError):
            rejected += 1; continue                     # label hygiene gate 1
        if abs(sum(li.line_total for li in label.line_items)
               - label.total_amount) > 0.01:
            rejected += 1; continue                     # gate 2: totals reconcile
        records.append({
            "images": [Image.open(image_path).convert("RGB")],
            "messages": [
                {"role": "user", "content": [
                    {"type": "image"},
                    {"type": "text", "text": PROMPT}]},
                {"role": "assistant", "content": [
                    {"type": "text", "text": label.model_dump_json()}]},
            ],
            "supplier": label.supplier_name,
        })
    print(f"kept {len(records)}, rejected {rejected}")
    return records

records = build_records("data/labels")

# Split by SUPPLIER, never by page — random splits leak layouts.
random.seed(17)
suppliers = sorted({r["supplier"] for r in records})
random.shuffle(suppliers)
holdout = set(suppliers[: max(1, len(suppliers) // 10)])

def strip(rs):   # keep only the columns TRL expects
    return [{k: r[k] for k in ("images", "messages")} for r in rs]

train_ds = Dataset.from_list(strip([r for r in records if r["supplier"] not in holdout]))
eval_ds  = Dataset.from_list(strip([r for r in records if r["supplier"] in holdout]))

LoRA + 4-bit quantisation: fitting a 7B VLM on a single consumer GPU

Full fine-tuning of a 7B VLM means holding the weights, gradients and optimiser states in memory at once — tens of gigabytes before the first image is processed. The combination that collapsed this requirement is the same one that did it for text LLMs, covered from first principles in our LoRA and QLoRA recipe: freeze the base model, quantise it to 4-bit NF4 so a 7B's weights occupy roughly 5 GB instead of 15, and train only small low-rank adapter matrices attached to the projection layers. Two published data points anchor what this buys you on real hardware. The Datature run above fine-tuned Qwen2.5-VL with LoRA rank 4 on a single RTX 4070 with 12 GB of VRAM — 4-bit NF4, double quantisation, bfloat16 compute, batch size 1, ten epochs over 1,250 samples in about three hours. Roboflow's JSON-extraction tutorial ran a Qwen2.5-VL fine-tune on a 16 GB T4 in free-tier Colab. The Hugging Face VLM fine-tuning cookbook, which trains the older Qwen2-VL-7B on ChartQA with more headroom for batch size and resolution, recommends an A100. The table below puts working numbers around the three Qwen2.5-VL sizes; the weight figures are arithmetic, the fine-tune columns are illustrative and anchored on those published runs.

The wide fine-tune ranges are not hedging — they are the resolution effect, which is the single most important VLM-specific fact in this whole recipe and gets its own treatment in the failure-modes section. In short: Qwen2.5-VL processes pages at native resolution, every 28×28 pixel patch becomes one visual token, and a full A4 scan at high DPI can dwarf your text tokens. Memory is a function of your images, not just your model.

The configuration below is a sensible starting point for document extraction. Rank 16 with alpha 32 is a middle setting — the Datature run showed rank 4 on attention projections alone is enough to move a 12 GB card's needle, while ranks of 16–32 across attention and MLP projections are the common ceiling before returns diminish. The target module names match the Qwen2.5 language decoder; the vision tower stays frozen, which is what you want — the encoder already reads documents, and the mapping you are teaching lives in the decoder.

import torch
from transformers import BitsAndBytesConfig
from peft import LoraConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)

lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    # Optional (peft >= 0.14): keep the adapter off the vision tower
    # explicitly, since matching is by module-name suffix:
    # exclude_modules=r".*visual.*",
    task_type="CAUSAL_LM",
)

Training with TRL's SFTTrainer: collators, image tokens and hyperparameters

This is the part of the stack that has changed most, and where most tutorials you will find are out of date. As of July 2026, TRL's SFTTrainer (v1.8 docs) supports vision-language models natively: give it a dataset with an images column and conversational messages, and it loads the model's AutoProcessor as the processing class, defaults to a built-in DataCollatorForVisionLanguageModeling, and preprocesses multimodal samples on the fly. The hand-rolled collator that every 2024–2025 tutorial reproduced is no longer required for the standard path. You can also pass quantization_config and peft_config straight to the trainer, which handles k-bit training preparation for you.

from trl import SFTConfig, SFTTrainer

args = SFTConfig(
    output_dir="qwen25vl-7b-invoice-lora",
    max_length=None,                 # VLM rule: never truncate image tokens
    per_device_train_batch_size=1,
    gradient_accumulation_steps=8,   # effective batch 8
    num_train_epochs=2,
    learning_rate=1e-4,              # adapter-appropriate; ~5x the full-FT default
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    gradient_checkpointing=True,
    logging_steps=10,
    eval_strategy="steps",
    eval_steps=100,
    save_strategy="steps",
    save_steps=100,
    save_total_limit=2,
    load_best_model_at_end=True,
    metric_for_best_model="eval_loss",
    greater_is_better=False,
)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-VL-7B-Instruct",
    args=args,
    train_dataset=train_ds,
    eval_dataset=eval_ds,
    peft_config=lora_config,         # from the previous snippet
    quantization_config=bnb_config,  # QLoRA: 4-bit base + trainable adapter
)
trainer.train()
trainer.save_model()                 # writes the adapter, not 7B of weights

Three of those lines deserve explanation. max_length=None is the one VLM-specific rule the TRL docs flag in a warning of their own: sequence truncation can silently cut image tokens out of a sample, which either crashes training or — worse — trains the model on pages it cannot fully see. Leave truncation off and control sequence length through image resolution instead. learning_rate=1e-4 follows the adapter-training guidance in the TRL docs — around 1e-4 for LoRA, against a 2e-5 default for full fine-tuning — because only the new low-rank parameters are learning. And gradient_checkpointing=True (TRL's default) trades roughly a third of your step speed for the activation memory that makes batch-of-1-plus-accumulation work on a 12 GB card.

It is still worth knowing what the collator does under the hood, because the moment you need per-sample control — custom augmentation, mixed single- and multi-page samples — you will write one, and because label masking is where silent bugs live. A VLM training batch interleaves text tokens with image placeholder tokens that the processor expands to match each page's visual patches. Loss must never be computed on those positions. The classic pattern, from the Hugging Face cookbook lineage:

from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct",
    min_pixels=256 * 28 * 28,        # floor: keep small crops readable
    max_pixels=1024 * 28 * 28,       # cap: bound visual tokens per page
)

def collate_fn(examples):
    texts = [processor.apply_chat_template(ex["messages"], tokenize=False)
             for ex in examples]
    images = [ex["images"] for ex in examples]
    batch = processor(text=texts, images=images,
                      return_tensors="pt", padding=True)
    labels = batch["input_ids"].clone()
    labels[labels == processor.tokenizer.pad_token_id] = -100
    for tok in ("<|image_pad|>", "<|vision_start|>", "<|vision_end|>"):
        labels[labels == processor.tokenizer.convert_tokens_to_ids(tok)] = -100
    batch["labels"] = labels
    return batch

# To use it: pass data_collator=collate_fn to SFTTrainer. For VLMs, TRL
# skips dataset pre-tokenisation automatically, so the collator sees the
# raw {"images", "messages"} records.

The -100 value is the ignore index for cross-entropy: padding and vision tokens are excluded from the loss, so the model is graded only on the JSON it writes. The special-token names are Qwen's; other model families use different vision markers, which is exactly why the built-in collator is the better default when you do not need the control. The min_pixels/max_pixels pair on the processor is Qwen-specific and is your resolution throttle — more on that below.

Evaluating document extraction: field-level accuracy, not loss curves

A falling eval loss tells you the model is getting better at predicting your label tokens. It does not tell you whether the invoice total is right. Token-level loss weighs every token equally, so a model can shave loss by perfecting JSON punctuation while still misreading total_amount — the one field your finance integration actually consumes. The metric that governs a document fine-tune has to be field-level: parse the model's output, compare it field by field against gold, and report precision (of the values it emitted, how many were correct) and recall (of the values present in the document, how many it found) — separately, because they fail differently. A hallucination-prone model tanks precision; a model that gives up on hard scans tanks recall; a single blended "accuracy" hides which problem you have. This is the same discipline as any eval suite — golden sets, versioned runs, regression gates, per our evaluation-suite guide — with a document-shaped scorer. Roboflow's tutorial scores its extraction with normalised edit distance, which is a reasonable soft metric for long strings; the loop below uses exact match after normalisation for scalars, which is stricter and closer to what a downstream system experiences.

import json
from decimal import Decimal

SCALAR_FIELDS = ["invoice_number", "invoice_date", "supplier_name",
                 "supplier_gstin", "vat_number", "currency", "total_amount"]

def norm(v):
    if v is None:
        return None
    if isinstance(v, (int, float)):
        return str(Decimal(str(v)).quantize(Decimal("0.01")))
    return " ".join(str(v).split()).lower()

def score_document(pred_text, gold):
    try:
        pred = json.loads(pred_text)
    except json.JSONDecodeError:
        return {"parse_ok": False}
    tp = fp = fn = 0
    per_field = {}
    for f in SCALAR_FIELDS:
        g, p = norm(gold.get(f)), norm(pred.get(f))
        if g is None and p is None:
            match = None                     # correctly absent; not scored
        elif p is not None and p == g:
            tp += 1; match = True
        else:
            if p is not None: fp += 1        # emitted a wrong/invented value
            if g is not None: fn += 1        # missed a value that was there
            match = False
        per_field[f] = match
    return {"parse_ok": True, "tp": tp, "fp": fp, "fn": fn, "fields": per_field}

def aggregate(results):
    parsed = [r for r in results if r["parse_ok"]]
    tp = sum(r["tp"] for r in parsed)
    fp = sum(r["fp"] for r in parsed)
    fn = sum(r["fn"] for r in parsed)
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall    = tp / (tp + fn) if tp + fn else 0.0
    return {
        "parse_rate": len(parsed) / len(results) if results else 0.0,
        "field_precision": round(precision, 4),
        "field_recall": round(recall, 4),
        "field_f1": round(2 * precision * recall / (precision + recall), 4)
                    if precision + recall else 0.0,
    }

Note the convention: a wrong non-null value costs both precision and recall, because the model simultaneously emitted a bad value and missed the good one — state your convention in the eval README, because comparisons across runs are meaningless if it shifts. Report parse_rate as a first-class metric too: it is your format-drift alarm. For line-item tables, extend the loop to align rows (by description similarity or order) and compute a row-level F1; for full table structure, a tree-edit-distance metric is the standard, as covered in the extraction guide. And always slice by held-out supplier — an aggregate number that averages ninety easy suppliers against ten hard ones tells you nothing about the ten.

Common failure modes: resolution limits, hallucinated fields, format drift

Resolution: the failure that looks like a model problem but is a pixels problem

Qwen2.5-VL's headline feature is native dynamic resolution: rather than squashing every page to a fixed square, the vision transformer processes images at their real size, resizing height and width to multiples of 28, splitting the page into 14-pixel patches and merging each 2×2 group of patches into one visual token — so every 28×28 pixel area of your scan costs one token, and the token count scales with page area. That design, per the technical report, is what lets it read documents so well; it is also a trap in both directions. Feed a full A4 page at 300 DPI and the visual token count explodes — memory climbs, training slows, and long invoices push against practical sequence limits. Downscale too aggressively and 8-point line items blur below legibility, at which point the model does not fail loudly; it guesses. Set the processor's max_pixels so your smallest critical text stays readable at the resulting scale (for dense invoices, err higher), fix the same value for training and inference, and treat any accuracy gap between crisp and downscaled documents in your eval slices as a resolution symptom before you blame the model.

Hallucinated fields survive fine-tuning

Fine-tuning reduces hallucination on in-distribution documents; it does not eliminate it, and on out-of-distribution pages a fine-tuned model can be more confidently wrong, because you have taught it that a complete, well-formed JSON is always the answer. Keep every defence from the prompting pipeline: nullable schema fields, an explicit "null, never guess" instruction in the training prompt itself (so the behaviour is learnt, not just requested), deterministic business-rule checks after parsing, and confidence gating with human review for the documents that fail them. Include genuinely unreadable and partially occluded pages in the training set with honest nulls in the labels — a model that has never seen a null target will never emit one.

Format drift and forgetting

A fine-tuned extractor can drift in two directions. Format drift — deviating from your JSON schema — should approach zero on in-distribution pages if your labels were byte-consistent, which is why the dataset section insisted on it; monitor parse_rate in production and treat a decline as retraining trigger number one. The subtler risk is catastrophic forgetting: push a small VLM hard on one invoice family and its general document ability erodes, so the KYC pack it used to handle acceptably now fails. Keep an out-of-domain slice in your eval set to detect it, and if it bites, the mitigations — lower rank, fewer epochs, mixing a slice of general document data into training — are covered in our catastrophic forgetting playbook.

Serving the adapter: merge vs hot-swap, and cost per 1,000 pages

Training leaves you with an adapter of tens to a couple of hundred megabytes, and a decision: fold it into the base weights, or serve it as a separate, swappable component.

Merging is one call — load the base in full precision, apply the adapter, merge_and_unload(), save. The result is an ordinary checkpoint with zero adapter overhead that any serving stack can load. Two rules: merge into the bf16 base, never the 4-bit training base, and re-run the field eval on the merged artefact — the adapter was learnt against a quantised base, so tiny numerical shifts are expected and occasionally material.

import torch
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from peft import PeftModel

base = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct", dtype=torch.bfloat16)
merged = PeftModel.from_pretrained(
    base, "qwen25vl-7b-invoice-lora").merge_and_unload()
merged.save_pretrained("qwen25vl-7b-invoice-merged")
AutoProcessor.from_pretrained(
    "Qwen/Qwen2.5-VL-7B-Instruct").save_pretrained("qwen25vl-7b-invoice-merged")

Hot-swapping keeps the base model resident once and loads adapters per document type or tenant — inference servers in the vLLM family support serving multiple LoRA adapters over a shared base on supported model architectures (check the current support matrix for your exact multimodal model before committing). Ten merged 7B models need ten times the weights; ten adapters share one base. That is the whole trade-off:

On cost, resist quoting anyone else's number and compute your own — it is one line of arithmetic: cost per 1,000 pages = (seconds per page ÷ 3,600) × 1,000 × GPU hourly rate, with seconds-per-page measured on your own pages at your serving resolution, since visual token count drives latency. As a deliberately illustrative shape: if a 7B extractor averages 2–4 seconds per page on a mid-range inference GPU, 1,000 pages is roughly 0.6–1.1 GPU-hours — at the low single-digit dollars per hour such GPUs cost across providers, that lands around a pound, or low hundreds of rupees, per thousand pages, before batching improves it. Throughput engineering — continuous batching, serving-time quantisation — can move that several-fold; the levers are in our guides to cutting self-hosted serving costs and, if you would rather not run the GPU at all, serverless GPU platforms, which suit spiky document workloads well. For residency-bound pipelines, the same maths applies to an L4- or A10-class instance in AWS Mumbai (ap-south-1) or London (eu-west-2) — the point of self-hosting a fine-tuned 7B is precisely that the marginal page costs GPU-seconds, not API fees.

India + UK use cases: GST invoices, KYC packs, HMRC forms

The recipe is generic; the value shows up when it meets a specific pile of paper, and the AITC audience sits on two of the world's best piles.

GST invoices (India). A tax invoice under GST carries a predictable field set — supplier and recipient GSTINs, HSN/SAC codes per line, the CGST/SGST/IGST split, place of supply — rendered across thousands of supplier templates, frequently as phone photographs of thermal prints. This is close to the ideal fine-tuning target: a stable schema, huge layout variance, and deterministic validation for free (a GSTIN has a fixed 15-character structure, and the tax split must reconcile arithmetically with the taxable value). Those checks belong in your label-hygiene gate, your business-rule validator and your eval — the same rule, reused three times. A fine-tuned 3B or 7B handling the standard 90 per cent locally, with hard scans escalated to a stronger model, is a well-trodden architecture for accounts-payable automation in India.

KYC packs (India and UK). Onboarding bundles — identity documents, proof-of-address utility bills, bank statements — are the clearest data-residency case on either side: this is exactly the document class teams are least comfortable posting to a third-party API, and a self-hosted fine-tune keeps the images inside your VPC in Mumbai or London. They are also multi-document, so the win compounds: one adapter (or one per document class, hot-swapped over a shared base) classifies and extracts the whole pack. Train honest nulls hard here — an invented passport number is a compliance incident, not a typo — and keep the human review queue as the backstop regardless of how good the eval numbers get. The obligations under India's DPDP Act and UK GDPR that apply to processing these documents are covered in the extraction guide's data-protection section, and none of them get lighter because the model is yours.

HMRC forms (UK). Structured UK tax and payroll paperwork — self-assessment pages, P60s and P45s, CIS statements — is a bounded family of layouts with boxed, numbered fields, which makes it the fastest dataset of the three to cover well: the layout variance is across form versions and print quality rather than thousands of templates. The trap is version skew — box numbering and layout shift between tax years, so label the form version in your dataset, hold one version out entirely, and check the eval slice for it before trusting the model on next year's print run. The same logic extends to Companies House filings, pension statements and the rest of the UK's document estate.

In all three cases the pattern is identical: schema plus validation rules exist already, layouts vary, volume is high, and the documents are sensitive. That is the fine-tune sweet spot.

The bottom line

Fine-tuning a vision-language model for documents is no longer a research project — it is a weekend-sized engineering task with a well-marked path. Decide with evidence, not enthusiasm: prompt first, and reach for training only when errors concentrate on your formats, drift will not die, or economics and residency demand your own model. Spend your effort where it compounds — layout-diverse image–JSON pairs, ruthless label hygiene, a supplier-held-out split. The training run itself is the easy part now: 4-bit NF4 plus LoRA fits a 7B on a 12–16 GB GPU, and as of July 2026 TRL's SFTTrainer handles the multimodal collation that used to fill half of every tutorial. Judge the result by field-level precision and recall on held-out layouts, never the loss curve; keep the null-preference, validation and review scaffolding from the prompting pipeline, because a fine-tuned model still hallucinates on the pages that matter most; and serve merged for simplicity or hot-swapped when adapters multiply. Then point it at the paper your market actually produces — GST invoices in Pune, KYC packs in Mumbai and Manchester, HMRC forms in Leeds — and let the per-page economics do the arguing.