Vector DB Index Tuning: HNSW, IVF and Zero-Downtime Migrations

What you need to know

• This is a tuning guide, not a selection guide. Which database to run is a separate question — our vector database selection guide covers that. Here we assume the database is chosen and the job is making its index behave.
• Every index choice is a trade on one triangle: recall, latency and memory. You cannot maximise all three at once; you buy one with another. Tuning is deciding where on that triangle you want to sit.
• HNSW is the graph-based default. Three knobs — M, efConstruction and efSearch — set the balance. Higher values buy recall and pay in memory, build time and query latency respectively.
• IVF and product quantisation are the memory levers. IVF partitions the space so you search only part of it; PQ compresses vectors up to roughly 97% at a measurable recall cost that a re-ranking pass largely recovers.
• Measure recall@k against latency, on your own data. Vendor defaults are a starting point, not an answer. Sweep the parameters, plot the curve, pick the knee.
• Reindex without downtime using blue-green. Build the new index beside the old one, backfill, shadow-read, then flip an alias. Keep the old one warm for instant rollback.

The recall, latency and memory triangle

Approximate nearest-neighbour (ANN) search exists because exact search does not scale. A brute-force scan compares your query against every stored vector; at a few million vectors that is already too slow for an interactive product, and at a billion it is hopeless. Every ANN index is a structured shortcut that trades a little accuracy for a large speed-up — and the accuracy you give up is measured as recall, the fraction of the true nearest neighbours the index actually returns.

The unavoidable fact is that recall, query latency and memory footprint form a triangle you cannot escape. Push recall up and you either search more of the structure (more latency) or store richer structure (more memory). Shrink memory with compression and recall falls unless you spend extra latency re-ranking. Speed queries up by searching less and recall drops. As the OpenSource Connections write-up on navigating recall and performance puts it, the entire discipline of index tuning is choosing your position on that surface deliberately rather than accepting whatever a default hands you.

The reason this matters commercially is that the three corners map to three different bills. Latency is a user-experience and, at scale, a compute bill. Memory is a RAM bill — and for an in-memory graph index on a large corpus, that is the single largest line item. Recall is a quality bill paid by whoever is downstream: a retrieval-augmented generation pipeline that only sees 85% of the right documents produces worse answers no matter how good the model is. A team in Bengaluru serving a consumer app on tight unit economics and a team in London serving a regulated search product will weight these three differently, but both are trading on the same triangle. Naming which corner you are optimising for is the first real decision.

Tuning HNSW: M, efConstruction and efSearch

Hierarchical Navigable Small World (HNSW) is the default index in pgvector, Qdrant, Milvus, Weaviate and most managed services, and for good reason: it delivers high recall at low latency, needs no separate training step and handles incremental inserts gracefully. It builds a layered proximity graph and walks it greedily from a sparse top layer down to a dense bottom one. Three parameters govern its behaviour, and understanding what each one buys is most of the battle.

M is the maximum number of neighbour connections each node keeps. More connections make the graph better at finding true neighbours, so recall rises — but memory grows roughly linearly with M, because every edge is stored. Per Milvus and Zilliz guidance, recall improves strongly up to around M=32 and shows diminishing returns beyond 64. A common starting point is M=16, raised to 32 for workloads where recall matters and the memory is available.

efConstruction is how wide the search is while building the graph — how many candidate neighbours are considered before the edges are fixed. A higher value produces a better-connected graph and higher achievable recall, at the cost of build time. Crucially it does not change the final index size, because it only affects which edges get chosen, not how many. Typical values run from 100 for fast builds to 200–400 for production indexes that will be queried far more often than they are rebuilt.

efSearch (called ef or hnsw.ef_search in some systems) is the size of the dynamic candidate list at query time. This is your live recall-versus-latency dial and the one you will touch most, because it can be changed per query without rebuilding anything. A widely cited illustration: raising efSearch from 100 to 400 might lift recall from roughly 90% to 98% while roughly doubling query latency. The right value is the smallest one that clears your recall floor.

Because the graph is fixed at build time by M and efConstruction but efSearch is free to vary, the correct way to tune HNSW is to build a small number of candidate graphs and then sweep efSearch across each, measuring recall@k and latency at every point. Here is a compact, provider-agnostic sweep. build_index() and search() stand in for whichever library you use — FAISS, hnswlib, pgvector via SQL, or a Qdrant client.

import time
import numpy as np

def recall_at_k(approx_ids, truth_ids, k):
    """Fraction of the true top-k neighbours the index actually returned."""
    hits = 0
    for approx, truth in zip(approx_ids, truth_ids):
        hits += len(set(approx[:k]) & set(truth[:k]))
    return hits / (len(truth_ids) * k)

def exact_ground_truth(data, queries, k):
    """Brute-force kNN: the reference the approximate index is graded against.
    Fine on a sample of a few thousand queries; do not run on the full corpus."""
    truth = []
    for q in queries:
        dists = np.linalg.norm(data - q, axis=1)   # L2; use cosine if that is your metric
        truth.append(np.argsort(dists)[:k])
    return truth

def sweep_hnsw(data, queries, k=10,
               m_values=(16, 32),
               ef_construction_values=(100, 200),
               ef_search_values=(32, 64, 128, 256, 512)):
    """Build a few graphs, then sweep the query-time knob across each.
    Returns rows of (M, efC, efS, recall@k, p50_ms, p95_ms) to plot."""
    truth = exact_ground_truth(data, queries, k)
    rows = []
    for m in m_values:
        for ef_c in ef_construction_values:
            index = build_index(data, M=m, efConstruction=ef_c)  # your library here
            for ef_s in ef_search_values:
                index.set_ef_search(ef_s)
                latencies, approx_ids = [], []
                for q in queries:
                    t0 = time.perf_counter()
                    ids = index.search(q, k=k)
                    latencies.append((time.perf_counter() - t0) * 1000.0)
                    approx_ids.append(ids)
                lat = np.array(latencies)
                rows.append({
                    "M": m, "efConstruction": ef_c, "efSearch": ef_s,
                    "recall@k": round(recall_at_k(approx_ids, truth, k), 4),
                    "p50_ms": round(float(np.percentile(lat, 50)), 2),
                    "p95_ms": round(float(np.percentile(lat, 95)), 2),
                })
    return rows

# Pick the row with the lowest p95 latency that still clears your recall floor.
# rows = sweep_hnsw(data, queries)
# best = min((r for r in rows if r["recall@k"] >= 0.95), key=lambda r: r["p95_ms"])

The shape of the output is the point. You are not looking for the highest recall or the lowest latency in isolation; you are looking for the row where recall first clears your floor at the lowest p95 latency. That is the knee of the curve, and it is almost never the vendor default.

IVF and product quantisation: the memory levers

HNSW is excellent until memory becomes the binding constraint. An in-memory graph over hundreds of millions of high-dimension vectors can demand a very large, very expensive RAM footprint. When that bill dominates, the inverted-file (IVF) family and product quantisation (PQ) are the tools that bring it down.

IVF partitions the vector space into nlist clusters (Voronoi cells) using k-means during a training step. At query time you only search the nprobe clusters nearest to the query, not the whole corpus. This is a direct latency-versus-recall dial: a small nprobe is fast but risks missing neighbours that sit just across a cell boundary; a larger nprobe searches more cells and recovers them at higher cost. Guidance from practitioners is that probing enough cells to cover roughly 5–10% of the dataset tends to reach high recall without paying full-scan latency. nlist is commonly set in the region of the square root of the number of vectors as a starting heuristic, then tuned.

Product quantisation attacks the memory corner directly. It splits each vector into sub-vectors and replaces each with the nearest entry from a small learned codebook, storing a byte-sized code instead of full floats. The compression is dramatic: per Pinecone's FAISS series and Milvus, a 128-dimension float32 vector of 512 bytes can shrink to about 8 bytes — roughly a 97–98% reduction. That compression is lossy, so raw recall falls. The standard remedy is a two-stage search: use the compressed IVF-PQ index to fetch a generous candidate set quickly, then re-rank those candidates against full-precision vectors. With that re-ranking pass, a well-tuned IVF-PQ pipeline typically recovers to around 90–95% recall while keeping most of the memory saving. NVIDIA's cuVS IVF-PQ tuning notes walk through the same refine-then-rank pattern on GPU.

These three approaches are not rivals so much as points on the same triangle, and the honest way to choose is to lay them side by side.

Measuring recall@k against latency

You cannot tune what you do not measure, and vector search has a specific trap: the index will happily return an answer at any parameter setting, so nothing breaks loudly when recall quietly collapses. The only defence is to grade the approximate index against exact truth on representative queries — the discipline every serious tuning effort is built on.

The method is straightforward. First, take a sample of real production queries — a few thousand is usually enough — and compute their exact nearest neighbours by brute force. That set is your ground truth. Then, for each candidate parameter configuration, run the approximate index over the same queries and compute recall@k: the fraction of each query's true top-k neighbours the index actually found. Record recall alongside p50 and p95 latency for that configuration, exactly as the sweep code above does. Finally, plot recall on one axis and latency on the other, and read off the knee: the point past which more latency buys almost no recall.

Three things make this measurement trustworthy. Use real queries, not the stored vectors themselves, because query distribution rarely matches document distribution. Report a tail latency such as p95, not just the mean, because a good average hides the slow queries that actually annoy users. And recompute ground truth whenever your embeddings change — a new embedding model or a re-embedded corpus invalidates every previous recall number. If you are still choosing that embedding model, our guide to building your own retrieval benchmark pairs naturally with this step, and if the index feeds a RAG system, the production RAG hybrid-retrieval guide shows where recall@k sits in the wider pipeline.

Hot and archival tiering

Most large corpora are not queried uniformly. A minority of vectors — recent documents, popular products, active tickets — absorb the overwhelming majority of searches, while a long tail is touched rarely if ever. Paying for a single high-recall, in-memory HNSW index across the whole corpus means paying premium RAM prices to keep cold data instantly searchable that almost nobody searches. Tiering fixes that mismatch.

The pattern mirrors classic storage hierarchies. A hot tier holds the frequently accessed vectors in an HNSW index tuned for low latency and high recall — this is where the money and the performance both go. A warm or archival tier holds the long tail in an IVF-PQ index, compressed and cheaper per vector, accepting slightly higher latency and a re-ranking step because those queries are rare and less latency-sensitive. Queries hit the hot tier first; only searches that need broader or older coverage fan out to the archive. A UK marketplace might keep the last ninety days of listings hot and everything older on a compressed archive in the same region; an Indian support platform might keep open and recently closed tickets hot and years of resolved tickets archived. The architecture is identical; only the boundary moves.

Two operational details decide whether tiering pays off. The promotion and demotion policy — when a vector moves between tiers — should be driven by access patterns and age, and it needs to run continuously, not as a one-off. And placement matters for latency and for data-residency rules: keeping each tier in the region its users and its compliance regime require (AWS Mumbai for Indian data, London or Ireland for UK and EU data) avoids both a cross-region latency penalty and a governance problem. Because that colder tier is exactly where compression lives, it is also where a botched reindex does the most damage — which brings us to the migration.

Zero-downtime reindex and migration

Sooner or later you will have to rebuild an index in place: a new embedding model changes the vectors, you switch index type from IVF-PQ to HNSW as a tier grows hot, you re-tune M and must rebuild the graph, or you migrate from one engine to another. Doing this by dropping the old index and building a new one means a window where search returns nothing or returns garbage — unacceptable for anything user-facing. The answer is a blue-green migration, the same discipline you would apply to a database schema change or a model upgrade.

The shape is: stand up the new index (green) beside the live one (blue); backfill it from your source of truth; keep it current with the live write stream; verify it with shadow reads; then flip traffic atomically via an alias or router, keeping blue warm for rollback. The critical insight is that the source of truth is your primary datastore or embedding pipeline — never the old index. Rebuilding from the index you are trying to replace propagates whatever staleness or corruption prompted the migration.

import time

class BlueGreenMigration:
    """Zero-downtime vector reindex. 'blue' is live; 'green' is the new index.
    Reads always serve from the alias; writes fan out to both once green exists.
    Nothing is destructive until an explicit, verified cutover."""

    def __init__(self, store, alias="search"):
        self.store = store          # your vector DB client
        self.alias = alias          # what the application queries
        self.blue = store.resolve(alias)   # current live index/collection
        self.green = None

    def build_green(self, config):
        """1. Create the new index beside the old one. No traffic yet."""
        self.green = self.store.create_index(config)   # new M/efC, or IVF-PQ, etc.
        # From here, dual-write so green never falls behind live data.
        self.store.enable_dual_write(targets=[self.blue, self.green])

    def backfill(self, source_iter, batch_size=1000):
        """2. Backfill green from the SOURCE OF TRUTH, not from blue.
        Re-embed here if the migration is driven by a new embedding model."""
        batch = []
        for record in source_iter:                     # primary DB / object store
            batch.append(self.store.to_vector(record))
            if len(batch) >= batch_size:
                self.green.upsert(batch)
                batch.clear()
        if batch:
            self.green.upsert(batch)

    def shadow_read(self, queries, k=10, floor=0.95):
        """3. Mirror live queries to green WITHOUT serving its results.
        Compare against blue (or exact truth) before trusting it."""
        agree = 0
        for q in queries:
            blue_ids = set(self.blue.search(q, k=k))
            green_ids = set(self.green.search(q, k=k))
            agree += len(blue_ids & green_ids) / k
        overlap = agree / len(queries)
        return overlap, overlap >= floor

    def cutover(self):
        """4. Atomic flip. The alias now points at green; blue stays warm."""
        self.store.repoint_alias(self.alias, self.green)
        self.blue, self.green = self.green, self.blue   # green is now live
        # Keep the old index for a grace period so rollback is one call.

    def rollback(self):
        """Instant revert if post-cutover metrics regress."""
        self.store.repoint_alias(self.alias, self.blue)

    def finalise(self, grace_seconds=86400):
        """5. Only after the new index has proven itself, retire the old one."""
        time.sleep(grace_seconds)       # in practice, gate on live metrics
        self.store.enable_dual_write(targets=[self.blue])   # stop writing to retired
        self.store.drop_index(self.green)                   # the old 'green' == old blue

Two safeguards turn this from a script into a safe procedure. Shadow reads are the gate: mirror a slice of live traffic to the green index, compare its results against blue (or against exact ground truth), and refuse to cut over until overlap or recall clears your floor. This is the same technique as a canary deploy for a model — our guide to shadow and canary deploys covers the pattern in depth. And the cutover itself must be atomic: flip a single alias or router pointer so every query moves in one instant, rather than draining connections across a window where some hit blue and some hit green. Keep blue warm for a grace period afterwards so a regression is a one-call rollback, not a rebuild.

Pitfalls to avoid

1. Shipping vendor defaults untested. Defaults are tuned for a generic benchmark, not your corpus. Sweep efSearch or nprobe against a ground-truth set before you trust any recall number.
2. Grading the index against stored vectors instead of real queries. Query distribution differs from document distribution; a recall figure measured on the wrong distribution flatters the index and misleads the tune.
3. Adopting PQ without a re-rank stage. Compressed-only recall looks alarming and is not representative. The two-stage fetch-then-re-rank pattern is what makes PQ viable — plan for it up front.
4. Rebuilding graph parameters casually. M and efConstruction are set at build time; changing them means a full rebuild, so treat them as deliberate, infrequent decisions and do the rebuild as a blue-green migration.
5. Reindexing from the old index. Always backfill from the primary source of truth. Rebuilding from the index you are replacing carries its staleness forward and defeats the point of the migration.
6. Ignoring data residency when tiering or migrating. Moving vectors between tiers or regions can move personal data across a border. Keep each tier in the region its users and its regulator require.

Where this leaves you

Vector index tuning is not a dark art; it is disciplined trading on a three-cornered budget. Name your recall floor and latency ceiling first. Reach for HNSW by default and tune it with a parameter sweep, holding M and efConstruction steady while you find the efSearch that clears your floor at the lowest tail latency. Move to IVF and product quantisation when memory is the binding cost, and pair PQ with a re-ranking pass so its compression does not quietly eat your recall. Tier hot and cold data so you are not paying premium RAM for vectors nobody searches. And when the time comes to rebuild — a new embedding model, a new index type, a re-tune — do it blue-green: build beside the old, backfill from the source of truth, shadow-read until it clears the gate, and flip an alias with the old index kept warm behind you. These patterns are provider-agnostic by design, whether you run pgvector, Qdrant, Milvus or FAISS. The engines will keep changing; the triangle will not.