The AI Engineer Interview, Decoded: The 5 Question Clusters and How to Answer "Design a RAG System"

How a 2026 AI engineering interview is structured

If you have not interviewed for an AI engineering role recently, the format has settled into something more predictable than the field's pace might suggest. As of June 2026, five question clusters cover roughly 90 per cent of what you will be asked across a typical loop. The remaining ten per cent is company-specific flavour — a proprietary tool, a particular domain, a culture round — but the spine is consistent enough that you can prepare against it directly rather than hoping to wing it.

The five clusters are: LLM and transformer fundamentals; RAG architecture; agentic systems; prompting and evaluation; and system design for LLM-backed products. What unites them is a single underlying test. Interviewers are not checking whether you can recite a definition or whether you have read a paper. They are checking whether you can move fluently between the theory underneath a technique and the production reality of shipping it. A candidate who can explain what a KV cache is but cannot say why it matters for serving cost will lose to one who connects the two in a sentence. The reward is for traversal, not recall.

That has a practical consequence for how you prepare. For every concept below, hold two facts ready: the mechanism, and the trade-off it forces in production. The table below maps the five clusters to what each one is really probing, so you can audit your own readiness before you read the detail.

Cluster	Typical opener	What it actually tests
1. LLM and transformer fundamentals	"Explain how a KV cache speeds up inference."	Whether you understand model internals well enough to reason about cost, latency and quality, not just call an API.
2. RAG architecture	"Design a RAG system for a customer-support chatbot."	Whether you can design a multi-stage retrieval pipeline and name the failure mode at each stage.
3. Agentic systems	"Build an agent that completes a multi-step task with tools."	Whether you understand planning, tool use and looping — and, crucially, how agents fail.
4. Prompting and evaluation	"How would you measure whether this system is any good?"	Whether you can define and operationalise quality, not just produce output.
5. System design for LLM-backed products	"Design an LLM-powered enterprise search product."	Whether you can balance requirements, model choice, latency, cost and safety under real constraints.

The five clusters cover roughly 90 per cent of 2026 AI engineering interview loops. The remaining share is company-specific.

Cluster 1 — LLM and transformer fundamentals

This cluster establishes whether you understand the machine underneath the API. It is rarely the deciding round on its own, but a weak showing here colours everything after it: if you cannot reason about how a model works, an interviewer will discount your later answers about systems built on top of one. The good news is that the question set is finite and well-trodden.

Expect questions on context windows and their limits, on decoding strategies, on what a KV cache is and why it helps inference, and on the difference between few-shot prompting and chain-of-thought. A 2026-flavoured favourite is to ask why Direct Preference Optimisation has replaced Proximal Policy Optimisation at most frontier labs — a quick probe of whether you understand alignment beyond the acronyms.

Sample questions and crisp answers

"What is a KV cache and why does it speed up inference?" During generation, a transformer attends over every previous token at each step. Without caching, the keys and values for all prior tokens are recomputed on every new token — quadratic, wasteful work. The KV cache stores the key and value tensors already computed for earlier positions, so each new token only computes its own key and value and reuses the rest. The pay-off is much faster decoding; the cost is memory that grows with sequence length, which is exactly why long contexts and large batch sizes strain serving budgets. Naming that memory trade-off is what separates a strong answer from a textbook one.

"Walk me through the main decoding strategies." Greedy decoding takes the highest-probability token each step and is fast but repetitive. Beam search keeps several candidate sequences alive and is useful where there is a single best answer, less so for open-ended generation. Top-k sampling restricts choices to the k most likely tokens; top-p, or nucleus sampling, restricts them to the smallest set whose cumulative probability exceeds p, adapting to how confident the model is. The temperature knob scales the distribution before sampling. The practical point to volunteer: decoding choice is a quality-versus-diversity lever you tune per use case, not a fixed setting.

"Why did DPO replace PPO at most frontier labs?" PPO-style reinforcement learning from human feedback requires training a separate reward model and then running on-policy reinforcement learning against it — a pipeline that is fiddly to stabilise and expensive to run. DPO optimises directly on preference pairs with a simple classification-style loss, removing the explicit reward model and the reinforcement-learning loop. The result is simpler, more stable and cheaper to train, which is why most frontier labs now favour it for preference tuning. The honest coda — that reinforcement-style methods still earn their keep for harder, multi-step objectives — shows you understand it as a trade-off, not a fashion.

Cluster 2 — RAG architecture: a worked "Design a RAG system" answer

If there is one question to rehearse until it is automatic, this is it. "Design a RAG system for a customer-support chatbot" is among the most common opening system-design prompts in 2026, reported across many companies. Close variants ask you to design LLM-powered enterprise search, or a generative-AI document-processing pipeline that ingests unstructured data — emails, PDFs, scanned images — for a workflow such as claims processing. The shape of a strong answer is the same in every case: walk the pipeline stage by stage, and at each stage name both the design decision and the failure mode an interviewer will probe.

Do not start by talking about models. Start by clarifying the corpus, the freshness requirement and what "correct" means for this domain — then walk the pipeline. The reference architecture below is the backbone of a good answer. Speak to every row.

Stage	Talking point	Failure mode the interviewer probes
1. Ingest & chunk	Parse the source documents and split them into retrievable units; chunk on semantic boundaries, not arbitrary character counts, and carry metadata such as source and section.	Chunks too large dilute relevance; too small lose context. Bad chunking is the most common silent killer of RAG quality.
2. Embed	Encode chunks with an embedding model chosen for the domain and language mix; keep query and document embeddings in the same space.	A generic embedding model on specialised jargon retrieves plausible-looking but wrong passages.
3. Vector store	Index the embeddings for approximate nearest-neighbour search; pick the index and filters to match scale and metadata-filtering needs.	Index choices that trade recall for speed too aggressively quietly drop relevant documents.
4. Retrieve (hybrid)	Combine dense vector search with sparse keyword search so you catch both semantic matches and exact terms like product codes or error strings.	Pure vector search misses exact-match terms; pure keyword search misses paraphrases. Customer-support corpora need both.
5. Rerank	Re-score the top retrieved candidates with a cross-encoder reranker so the best few passages, not merely the closest vectors, reach the model.	Skipping reranking leaves marginal passages in context, raising cost and inviting wrong answers.
6. Generate	Prompt the model with the reranked context and a clear instruction to answer only from it and to say when it cannot.	Without grounding constraints the model fills gaps from parametric memory — the classic confident-but-wrong answer.
7. Evaluate	Measure the system continuously, separating retrieval quality from answer quality.	No evaluation means no way to know whether a change helped or hurt — and no way to catch a regression.

The detail that signals seniority is at the evaluation stage, so make it explicitly: separate retrieval metrics from answer metrics. Retrieval is judged by whether the right passages were fetched at all — recall and precision at k. Answer quality is judged by faithfulness to the retrieved context and relevance to the question. The two fail for different reasons and are fixed in different places: if retrieval recall is high but answers are wrong, the problem is generation or grounding; if answers are wrong because the right passage was never fetched, no amount of prompt tuning will save you. Conflating the two is the most common reason candidates cannot debug their own pipeline when pushed.

If you want to go deeper on the retrieval side before an interview, our coverage of the 2026 agentic-RAG papers on hierarchical retrieval is a good way to show you are reading past the basics.

Cluster 3 — Agentic systems

Agentic questions test whether you understand systems that plan, call tools and loop towards a goal rather than answering in a single shot. The canonical prompt asks you to design an agent for a task that genuinely needs several steps — typically five to ten tool calls — such as researching a question across sources, or completing a workflow that touches a search tool, a calculator and an external API in sequence. The interviewer wants to see that you can describe the control loop, the tool interface and, above all, how the thing fails.

Cover the essentials: how the agent decides which tool to call and with what arguments; how it incorporates each tool result before deciding the next step; how it knows when it is finished; and what stops it looping forever. Then volunteer the failure modes, because that is where the round is won. Agents fail in characteristic, debuggable ways, and being able to name them is the strongest signal you have actually built one. The table below lists the failures interviewers most often ask you to diagnose.

Failure mode	What it looks like	How you would address it
Infinite or repeated loops	The agent calls the same tool with the same arguments over and over, never converging.	Add a step budget, detect repeated states, and force a stop-or-summarise decision when progress stalls.
Wrong tool selection	The agent reaches for a tool that cannot solve the sub-task, or invents arguments it does not have.	Tighten tool descriptions and schemas; validate arguments before execution; return clear errors the agent can recover from.
Context overflow	Accumulated tool outputs blow past the context window, and earlier reasoning is silently dropped.	Summarise or prune intermediate results; keep a compact working memory rather than appending everything.
Compounding errors	A small early mistake propagates, so each subsequent step builds on a wrong premise.	Add verification or reflection steps at checkpoints; let the agent re-plan when a result looks inconsistent.
Silent partial failure	A tool returns an empty or malformed result and the agent proceeds as if it succeeded.	Make tools fail loudly with typed errors; treat unexpected output as a branch, not a no-op.

A sober note that earns trust: agents are still markedly less reliable on long-horizon tasks than single-shot generation, and pretending otherwise is a red flag to anyone who has shipped one. If you want grounding in how today's systems are actually built and where they stand, our look at the 2026 agent frameworks and SDKs and the reality check in why agents still score low on reasoning benchmarks are worth having in your head before the round.

Cluster 4 — Prompting and evaluation (the cluster people freeze on)

Here is the cluster that decides more loops than candidates expect. People are comfortable on architecture, and they enjoy talking about prompting — the craft of it is satisfying and there is always a clever technique to mention. Then the interviewer asks the quiet question: "How would you actually measure whether this is any good?" — and a surprising number of otherwise strong candidates freeze. In 2026, evaluation is frequently the most important part of a production AI system, and an inability to talk about it credibly is one of the most common reasons a promising loop ends in a no.

The prompting half is the easier half. Be ready to discuss few-shot examples, chain-of-thought, structured output and the broader move towards context engineering — the discipline of designing what information the model sees, when, and in what form. Our piece on how context engineering replaced prompt engineering in 2026 is a useful frame for that part of the conversation. But treat prompting as table stakes and budget most of your energy for evaluation.

What a strong evaluation answer sounds like

A convincing answer has structure. It starts by defining quality for the specific task, then names concrete tooling and method. Something like this: "First I'd define what 'good' means here — for a support assistant that's faithfulness to the source documents, relevance to the question, and not answering when it shouldn't. Then I'd build a held-out evaluation set of representative queries with reference answers, and run it as a regression test on every change so I can see whether a tweak helped or quietly hurt. For automated scoring I'd use a framework such as RAGAS for retrieval-and-answer metrics, or DeepEval for assertion-style tests, and I'd use an LLM-as-judge for the more subjective dimensions — calibrated against a sample of human ratings so I trust it. I'd track retrieval metrics and answer metrics separately, because they fail for different reasons. And I'd never ship a change that regresses the evaluation set, even if it looks better on a handful of cherry-picked examples."

That answer works because it is operational. It names the metrics, the tooling — RAGAS, DeepEval, LLM-as-judge — the held-out set and the regression discipline, and it ties them to a definition of quality rather than reaching for tools in the abstract. If you can speak about evaluation like that, you will be ahead of most of the field.

Cluster 5 — System design for LLM-backed products

The final cluster zooms out from any single technique to the whole product. The prompt is broad by design — "Design an LLM-powered enterprise search tool", or "Design a document-processing pipeline for claims" — and the interviewer is watching how you impose structure on an open problem under real constraints. The winning move is to have a reusable framework you can apply to any such question, so you are never staring at a blank page.

A reusable six-step framework

Walk these six steps out loud, in order, narrating your decisions:

• 1. Clarify requirements. Who uses this, at what volume, with what latency and accuracy expectations, and what is the cost of a wrong answer? Pin these down before designing anything — the answers reshape every later choice.
• 2. Data and retrieval. What is the knowledge source, how fresh must it be, and how will the system ground its answers in it? For most enterprise products this is a RAG pipeline; say so and reuse the stages from Cluster 2.
• 3. Model choice and routing. Which model tier for which request? A common, credible pattern is to route easy queries to a smaller, cheaper model and escalate hard ones to a larger model, balancing quality against cost per request.
• 4. Evaluation. How will you know it works, and how will you catch regressions? Bring in the held-out set, the metrics and the regression discipline from Cluster 4 — never leave this step implicit.
• 5. Latency and cost. Where are the bottlenecks, and what do you cache, batch or stream? Tie this back to concrete mechanisms — the KV cache, prompt caching, streaming responses — to show the fundamentals connect to the product.
• 6. Safety and guardrails. How do you handle prompt injection, data leakage, harmful output and the cases where the system should refuse or escalate to a human? In an enterprise context this is not optional polish; it is a requirement.

The reason this framework scores well is that it forces you to connect every other cluster into one coherent answer: fundamentals feed model choice and latency, RAG feeds retrieval, and evaluation runs through the middle. Apply it consistently and even an unfamiliar prompt becomes a matter of filling in six familiar boxes.

The prep that actually works

You cannot fake your way through these clusters by reading, and the interviewers know it. The single most reliable preparation is to build small, complete systems and break them on purpose, so that your answers come from memory of real behaviour rather than from documentation. As of 2026, the difference between a doc-derived answer and an experience-derived one is audible within a question or two — the experienced candidate reaches for the failure mode unprompted, because they have seen it.

Three build exercises map almost one-to-one onto the clusters. First, stand up a simple RAG pipeline over a corpus you know, then deliberately break it: feed it bad chunking, swap in a mismatched embedding model, turn off reranking, and watch precisely how the answers degrade. That single exercise arms you for most of Cluster 2 and half of Cluster 4. Second, run an agent on a task that genuinely needs five to ten tool calls, and let it fail — you will meet the loops, the context overflow and the compounding errors first-hand, which is Cluster 3. Third, run a small fine-tune on a custom dataset; even a modest one teaches you what fine-tuning is actually for, and what it costs, which deepens Cluster 1.

Your shipped work answers half of these

Notice the pattern across all five clusters: the answers that win are the ones drawn from systems the candidate has actually shipped and debugged. The evaluation answer, the RAG failure modes, the agent loops — these are not things you can convincingly invent. Candidates who have built and broken real systems answer them from experience, and it shows. The hard part is no longer just doing the work; it is making that work visible to the people deciding whether to interview you in the first place.

That is what a Verified Builder profile on AI Tech Connect is for. It is where your shipped work — the RAG pipeline, the agent, the fine-tune, the evaluation harness — becomes legible to hiring managers across India and the UK who are actively looking for exactly these skills. Founding Builder spots are limited and still open as of June 2026; claiming one now puts your real work in front of the people doing the hiring, before the interview clusters ever come up.