PII Redaction in RAG Pipelines: Pre-Index and Pre-Response Controls

What you need to know

• PII enters a RAG system at three points — at ingest when documents are indexed, at retrieval when chunks land in the prompt, and at generation when the model emits an identifier. Redacting only the model's output covers one point and leaves the other two open.
• Redaction must happen before embedding, not after. An embedding computed from raw personal data can be partially reconstructed by embedding-inversion attacks, and a vector is far harder to "erase" than a row of text. Clean the text first, then embed.
• Two stages do the work. Stage 1 is pre-index detection and redaction with named-entity recognition plus rule-based recognisers. Stage 2 is a pre-response scan that catches anything the model still manages to surface.
• Recall beats precision for a redactor. A missed Aadhaar or NHS number is a potential breach; an over-redacted sentence is a minor loss of context. Measure both, per entity type, and optimise the trade-off deliberately.
• Compliance is a design constraint, not a bolt-on. Data minimisation and the right to erasure under India's DPDP Act 2023 and UK GDPR both assume you can find and delete a specific person's data — across embeddings, caches and logs, not just the source file.

As of July 2026, most teams shipping retrieval-augmented generation treat personal-data protection as an output problem: run the model, scan the answer, ship it. That is the wrong mental model. A RAG pipeline is a data-copying machine — it takes a document, splits it, embeds each piece into a vector store, retrieves the relevant pieces into a prompt, and lets a model reason over them. Personal data touches every one of those copies. This guide walks through a two-stage redaction architecture that treats ingest and response as separate control points, gives concrete detection recognisers for both the Indian and UK markets, and maps the design back to what the DPDP Act and UK GDPR actually require. It assumes you already have a working RAG stack; if you are still deciding how to split and embed your documents, our guide to chunking and embedding strategies for production RAG is the place to start, because the chunk boundary is also where a lot of PII detection succeeds or fails.

The threat model: where PII flows into a RAG system

Before writing a single recogniser, be precise about where personal data actually enters and leaves the pipeline. There are three distinct points, and each needs its own control.

1. At ingest — documents carry PII into the index

Source documents are the obvious entry point: support tickets, contracts, CVs, medical letters, KYC files. When you chunk and embed them, every identifier in the text is baked into two places — the stored chunk text and the embedding vector computed from it. This is the most dangerous point precisely because it is the most permanent: a vector is a lossy, opaque blob that is genuinely hard to search, audit or selectively delete after the fact.

2. At retrieval — chunks land in the prompt

Even if your stored text is clean, retrieval assembles chunks into a prompt that is sent to a model — often a third-party API in another jurisdiction. If any PII survived ingest, this is where it crosses a trust boundary and, potentially, a border. Retrieval is also where an adversarial query can try to pull specific personal records, which is why PII controls and prompt-injection defence-in-depth belong in the same design conversation.

3. At generation — the model emits an identifier

Finally, the model can produce PII in its answer: repeating an email it was shown, or in rarer cases surfacing something it inferred or memorised. This is the point most teams guard, and it is worth guarding — but on its own it is the weakest of the three, because it fails open whenever the model phrases an identifier in a form your filter did not anticipate.

Stage 1 — Pre-index: detect and redact before you embed

Stage 1 runs inside your ingestion pipeline, after text extraction and before embedding. It has two jobs: detect the personal data, and transform it — either masking it irreversibly or replacing it with a reversible token. The workhorse for the open-source stack in 2026 is Microsoft Presidio, which combines a spaCy or transformer NER model for free-text entities such as names and locations with configurable regex "recognisers" for structured identifiers, and pairs the detector (AnalyzerEngine) with a transformer (AnonymizerEngine).

Detection: NER for names, rules for structured identifiers

The two detection strategies are complementary and you need both. NER catches the things regex cannot — a person's name, an organisation, a place — because those have no fixed format. Rule-based recognisers catch the things NER is unreliable on: structured identifiers with a checksum or a fixed shape, such as an Aadhaar number, a National Insurance number or a credit-card number. Relying on regex alone is the classic failure: it sails through emails and card numbers while completely missing "please refund Priya Menon at the Andheri branch".

Your recogniser set has to be market-aware. An Indian document carries Aadhaar, PAN and Indian mobile formats; a UK document carries NHS numbers, National Insurance numbers and UK postcodes; and both carry the universal identifiers — email, phone, credit card, IBAN. The table below is a starting recogniser matrix for a dual-market pipeline.

The checksum column matters more than it looks. Aadhaar uses the Verhoeff algorithm, NHS numbers and IBANs use modulo checks, and card numbers use Luhn. Validating the checksum after the regex match slashes false positives — an arbitrary 12-digit string is not an Aadhaar number, and refusing to redact random digit runs keeps your recall metrics honest and your retrieval context intact.

Redacting with Presidio

Here is a compact Stage 1 pass. It registers custom recognisers for two market-specific identifiers, analyses a chunk, and anonymises it — masking the high-risk structured identifiers irreversibly while replacing names with a stable placeholder. In production you would move the vault write out of this function, but the shape is the point.

from presidio_analyzer import AnalyzerEngine, PatternRecognizer, Pattern
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig

analyzer = AnalyzerEngine()          # loads spaCy NER + built-in recognisers
anonymizer = AnonymizerEngine()

# --- Market-specific rule recognisers -------------------------------
aadhaar = PatternRecognizer(
    supported_entity="IN_AADHAAR",
    patterns=[Pattern(name="aadhaar", regex=r"\b\d{4}\s?\d{4}\s?\d{4}\b", score=0.4)],
    context=["aadhaar", "uidai", "uid"],   # nearby words raise confidence
)
ni_number = PatternRecognizer(
    supported_entity="UK_NINO",
    patterns=[Pattern(name="nino",
        regex=r"\b[ABCEGHJ-PRSTW-Z]{2}\s?\d{2}\s?\d{2}\s?\d{2}\s?[A-D]\b", score=0.5)],
    context=["national insurance", "ni number", "nino"],
)
analyzer.registry.add_recognizer(aadhaar)
analyzer.registry.add_recognizer(ni_number)

def redact_for_index(text: str) -> str:
    """Detect PII and return chunk text safe to embed and store."""
    results = analyzer.analyze(
        text=text,
        entities=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER",
                  "CREDIT_CARD", "IBAN_CODE", "IN_AADHAAR", "UK_NINO"],
        language="en",
    )
    # Irreversibly mask the high-risk structured identifiers;
    # replace names with a typed placeholder so retrieval still
    # knows "a person" was mentioned without knowing who.
    operators = {
        "IN_AADHAAR":  OperatorConfig("replace", {"new_value": "<AADHAAR>"}),
        "UK_NINO":     OperatorConfig("replace", {"new_value": "<NINO>"}),
        "CREDIT_CARD": OperatorConfig("mask",
            {"masking_char": "*", "chars_to_mask": 12, "from_end": False}),
        "PERSON":      OperatorConfig("replace", {"new_value": "<PERSON>"}),
        "DEFAULT":     OperatorConfig("replace", {"new_value": "<REDACTED>"}),
    }
    return anonymizer.anonymize(
        text=text, analyzer_results=results, operators=operators
    ).text

# In the ingestion loop: redact BEFORE the embedding call
clean = redact_for_index(chunk_text)
vector = embed(clean)             # nothing to invert — the PII never reached the model
store.upsert(id=chunk_id, values=vector, metadata={"text": clean,
             "pii_tags": ["PERSON", "IN_AADHAAR"], "tenant": tenant_id})

Reversible tokenisation versus irreversible masking

The example above masks irreversibly, which is the right default when nothing downstream needs the real value. But some applications do need it back — a support assistant that must actually email the customer, or a reconciliation step that matches on order ID. For those, use format-preserving, reversible tokenisation: replace priya@example.com with a synthetic token of the same shape, and keep the token-to-value mapping in a separate, access-controlled vault — never in the vector store, and never in the same database the model can reach through a tool.

Most mature pipelines mix the two per entity type, exactly as the recogniser table's "default strategy" column suggests: reversibly tokenise the handful of fields the product genuinely needs to act on, irreversibly mask everything else. Tag each chunk with the entity types it originally contained (the pii_tags metadata above) so you can apply chunk-level access control — a support-tier user might be allowed to retrieve chunks that contained order IDs but never chunks that contained health data.

Stage 2 — Pre-response: scan the output before it leaves

Stage 1 makes the stored data safe. Stage 2 is the safety net for everything that slips through — a rare identifier your recognisers missed, or PII the model reconstructs from partial context. It runs on the model's completion, before that completion is returned to the user or logged. The cheap first line is a deny-list re-scan: run the same detector over the output and re-redact anything it flags.

def scan_response(text: str) -> tuple[str, list[str]]:
    """Re-scan model output; re-redact any PII that survived Stage 1."""
    results = analyzer.analyze(
        text=text,
        entities=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER",
                  "CREDIT_CARD", "IBAN_CODE", "IN_AADHAAR", "UK_NINO"],
        language="en",
    )
    if not results:
        return text, []

    leaked_types = sorted({r.entity_type for r in results})
    safe = anonymizer.anonymize(
        text=text, analyzer_results=results,
        operators={"DEFAULT": OperatorConfig("replace",
                                             {"new_value": "<REDACTED>"})},
    ).text

    # Log the EVENT, never the PII itself: count and types only.
    log.warning("pii_leak_blocked",
                extra={"types": leaked_types, "count": len(results),
                       "request_id": request_id})   # no raw values, ever
    return safe, leaked_types

For higher-stakes applications, add a second-pass LLM-judge behind the deterministic scan. A small model prompted only to answer "does this text contain personal data that identifies a specific individual? list the spans" catches subtle leakage that regex and NER miss — an identifier described rather than stated ("the patient in bed four, the one whose wife rang on Tuesday"). Treat the judge as an additional recall layer, never as the only layer: it is non-deterministic and, like any model, can be talked around, so it sits behind the deterministic scan rather than replacing it.

Evaluating redaction: measure recall, per entity type

You cannot manage what you do not measure, and "we run Presidio" is not a measurement. Build a labelled golden set: a representative sample of your real documents with every PII span annotated by type. Then score detection with standard precision and recall — but weight them deliberately.

• Recall is the metric that protects you. A false negative — a missed identifier — is a potential reportable breach. A false positive — an over-redacted phrase — costs you a little retrieval context. For a redactor those are not symmetric, so tune toward high recall and accept some over-redaction.
• Report per-entity-type, never a single blended score. A detector can post an impressive overall F1 while quietly missing 30% of NHS numbers, because names dominate the count and drag the average up. The blended number hides exactly the failure that matters most.
• Red-team with obfuscated PII. Real documents contain identifiers with spaces, dots, or words spelled out ("nine one two three…"), OCR noise, and values split across a line break. Add these to the golden set and watch recall fall; that gap is your true exposure. The same adversarial mindset from our red-teaming and adversarial safety evals guide applies directly here.

Re-run the golden set in CI on every change to a recogniser, model version or chunking strategy. Detection quality regresses silently — a spaCy model upgrade can shift name recall by several points — and a redaction pipeline that quietly degrades is worse than none, because it manufactures false confidence.

Compliance mapping: DPDP Act 2023 and UK GDPR

Two-stage redaction is not just good engineering; it is what makes the legal obligations tractable. The mapping below is deliberately general — read your own sector's rules and take advice for your specifics — but the direction of travel is accurate for both markets.

India — Digital Personal Data Protection Act 2023

India's Digital Personal Data Protection Act 2023 builds on principles that map cleanly onto a RAG pipeline. Data minimisation and purpose limitation mean you should only ingest and retain the personal data your feature actually needs — Stage 1 redaction is minimisation made concrete, stripping identifiers a retrieval assistant has no reason to hold. The Act also gives a data principal the right to correction and erasure of their personal data, which for a RAG system means you must be able to locate and delete a specific individual's data from the vector store, not merely from the source document. Designing erasure as a fan-out across embeddings, caches and logs is enormously easier if most of those stores only ever held redacted text. Our DPDP Phase 2 AI compliance playbook goes deeper on the operational side.

United Kingdom — UK GDPR and the Data Protection Act 2018

Under UK GDPR and the Data Protection Act 2018, the relevant hooks are lawful basis and data minimisation for what you ingest, and the right to erasure under Article 17 for what you must be able to remove. The UK Information Commissioner's Office publishes practical guidance on both; the ICO's UK GDPR resources are the authoritative reference. The same erasure fan-out applies — embeddings, retrieval caches and logs are all "copies" for the purposes of a deletion request. If your pipeline also crosses borders — an Indian data source answered by a model hosted in the UK or the US, or vice versa — pair this guide with our walk-through of data-residency routing for DPDP and GDPR, because redaction and residency are two halves of the same obligation.

Keep records. Both regimes reward being able to show what you detect, how you transform it, and how you handle a deletion request. The redaction-event logs from Stage 2 (types and counts, never values) plus a documented golden-set evaluation are exactly the kind of evidence that demonstrates a defensible, minimised pipeline.

Architecture recap and the pitfalls that undo it

Put together, the pipeline is a clean two-stage sandwich around your existing RAG stack. At ingest: extract text, detect PII (NER + rules), redact or tokenise, then embed and store the clean text plus PII tags — and only then upsert to the vector store. At query time: retrieve, assemble the prompt, call the model, then scan the completion, re-redact anything that leaked, log the event without the data, and return. The token vault sits off to the side in its own trust domain, and every store — vectors, caches, logs — is wired into a single erasure fan-out.

The failure modes are predictable, and almost all of them are variations on "we redacted, but not everywhere or not soon enough":

• Redacting after embedding. The cardinal sin. The vector is already computed from raw PII and is hard to erase; the horse has left. Redact before the embedding call, always.
• Regex-only detection. Structured identifiers get caught; names, locations and free-text mentions of people sail straight through. You need NER as well as rules.
• Irreversible masking when the app needs the value. You mask an email at ingest, then discover the assistant is supposed to actually contact the customer. Decide reversibility per entity type up front.
• Forgetting that logs and caches hold PII too. Prompt logs, completion logs, retrieval caches and evaluation datasets are all copies. A redaction system that writes the PII it caught into an error tracker has simply relocated the breach.
• Deleting the source document but leaving the embeddings. An erasure request satisfied only against the source file is not satisfied. The vectors, caches and logs still hold the person's data.
• Trusting a single blended metric. A strong overall F1 can mask a badly missed high-risk entity type. Score per type, and weight recall.

Get the two stages right and PII protection stops being a frightening audit item and becomes a boring, testable property of your pipeline — which is exactly what you want it to be. If you are serving multiple customers from one stack, layer this on top of proper multi-tenant isolation, because redaction and tenant isolation solve different halves of "the wrong person must never see this data".