Vibe Coding to Production: The QA Discipline That Separates Demos From Products

The short version

• QA is the missing half of vibe coding. A systematic review of 101 practitioner sources, presented at ICSE 2026, found quality assurance is the most frequently overlooked dimension of AI-assisted coding — many builders skip testing entirely or delegate the checking back to the model that wrote the code.
• Demos and products fail under different conditions. A demo runs one user down the happy path on your machine. A product survives bad inputs, concurrency, cheap Android phones and next-week persistence. AI-generated code is optimised for the first and silent about the second.
• There is a ladder, and you can climb it. Autonoma's five-level Vibe Coding Maturity Model runs from Level 1 (no QA at all) to Level 5 (fully automated QA on every deployment). Level 3 is the minimum for real users; Level 4 is where production lives.
• The highest-leverage hour is before the first prompt. Specification engineering — a structured brief defining scope, constraints and acceptance criteria — eliminates whole categories of generated-code problems before they exist.
• Three failure modes account for most deaths: context decay between prompts 10 and 20, hand-rolled authentication, and the scale cliff where code that was fine at demo scale falls over under real traffic.

Vibe coding won the argument in 2025. By mid-2026 nobody serious disputes that a founder in Bengaluru or a product designer in Manchester can describe an app in plain English and have working software the same day. The argument that remains — the one that decides whether that software becomes a product or a cautionary screenshot — is about everything that happens after the demo works. And on that question the evidence is uncomfortable.

Researchers behind a grey literature review presented at ICSE 2026's Software Engineering in Practice track analysed 101 practitioner sources and extracted 518 firsthand accounts of vibe-coding behaviour. Their headline finding: a speed–quality trade-off paradox. Builders are drawn in by speed and the flow of instant results, yet most describe the output as fast but flawed — and quality assurance is the single most overlooked dimension of the whole workflow. Many skip testing altogether. Others accept model output unmodified. A striking number delegate the checking back to the same AI that generated the code, which is roughly the QA equivalent of marking your own exam. This article is about the discipline that fixes it — what you should actually do, in order, before you ship.

Why vibe-coded demos die in production

Start with an honest account of why the demo worked. It worked because demo conditions are the most generous conditions software will ever meet: one user, who is also the author, walking the single path they had in mind, with fresh state, on a fast machine, over good Wi-Fi. Every one of those conditions is quietly load-bearing, and production removes all of them at once.

Real users type emoji into the name field, paste 40,000-character documents into the search box, double-click the pay button, hit the back button mid-checkout and open the app on a three-year-old Android over patchy 4G on the Delhi Metro or a rural Cumbrian branch line. Real usage means two people editing the same record at the same moment. Real time means sessions that expire, caches that go stale, and data that must still be correct when the user comes back on Thursday. None of these conditions appears in a demo, so nothing about a successful demo tells you the software survives them.

AI generation makes this gap wider, not narrower, for a structural reason: a model optimises for the conditions described in the prompt, and prompts describe features, not failure. "Build me an invoicing tool for my agency" says nothing about input validation, indexes, rate limiting or row-level security, so the generated code frequently has none of them — and the absence is invisible, because the happy path works beautifully. The ICSE 2026 review found precisely this pattern in the wild: rapid perceived success up front, with the flaws surfacing only when someone other than the author uses the thing. The practitioner shorthand for the result is "fast but flawed", and the flaw is not in any single line of code. It is in the missing half of the workflow — the half that checks.

Here is the builder-angle framing that should reorganise your week: the distance between a demo and a product is not more features. It is QA discipline. Features are now nearly free; verification is the scarce skill. Which, incidentally, is why "I shipped this to real users and it held up" has become the single most differentiating line on a portfolio — it is the claim a weekend demo cannot fake.

The five-level QA maturity model

One of the more useful mental models in the 2026 practitioner literature is the Vibe Coding Maturity Model, a five-level ladder published by the QA tooling firm Autonoma. It is one vendor's framework rather than an industry standard, and the levels below are our own reading of it — but it describes how robust your QA workflow is relative to how fast you generate code, and its virtue is honesty: it does not tell you to do everything, it tells you where you are and what one level up looks like.

Two things about the ladder are worth internalising. First, most vibe coders sit at Level 1 — not because they decided to, but because Level 1 is the default the tools hand you. Generation ends, a deploy button appears, and nothing in the workflow asks whether the software was ever verified. Second, the levels are cumulative and each one is a bounded, learnable step. Moving from L1 to L3 requires no new tooling at all — just a checklist and the self-discipline to run it. Moving from L3 to L4 mostly means asking your coding agent to write the tests it should have written anyway, then wiring them into CI so they run without you. If your product has an AI layer of its own — prompts, agents, RAG — the same ratchet applies to model behaviour, and our guide to running evals in CI covers that half of the story.

The trap to avoid is moral rather than technical: treating the ladder as an accusation instead of a map. Level 1 is genuinely fine for a Sunday-afternoon prototype. The failure is not being at Level 1 — it is being at Level 1 while onboarding paying users, which is precisely the combination the ICSE review found over and over in the wild.

Specification engineering: the QA you do before the first prompt

The counterintuitive finding of the 2026 practitioner literature is that the highest-leverage QA practice happens before any code exists. Specification engineering — writing a structured, unambiguous brief that defines scope, constraints, architecture preferences and acceptance criteria before any AI tool is invoked — has largely displaced "prompt engineering" as the skill that separates professional output from slop. The logic is straightforward: a model fills every gap in your brief with a plausible guess, and plausible guesses are where bugs come from. A tight specification does not make the model smarter; it removes the room it has to be wrong. Front-loading effort into the spec eliminates entire categories of generated-code problems that no amount of downstream testing would catch cheaply.

The element that matters most for QA is the acceptance criteria: a checkable list of statements that define, before generation starts, what "done" will mean. Written well, they do double duty — they constrain the generation and they become your test plan verbatim. Here is the shape of a brief that earns its keep:

# Feature brief: CSV bulk-import for invoices

## Scope
IN:  upload CSV (max 5 MB), validate rows, preview errors,
     import valid rows, email summary to the uploader.
OUT: XLSX support, scheduled imports, dedup against existing
     invoices (separate ticket INV-88).

## Constraints
- Stack: Next.js + Supabase. Auth is Supabase Auth (managed).
  Do NOT build custom auth or sessions.
- All tables have RLS enabled; imports scoped to the caller's org_id.
- Parse server-side; never trust client-validated data.

## Acceptance criteria (each becomes a test)
- AC1: a 5,001-row file imports in under 30 s and shows a row count.
- AC2: a row with a malformed date is rejected with a line-numbered
       error; valid rows in the same file still import.
- AC3: a 6 MB upload is refused before parsing with a clear message.
- AC4: re-uploading the same file twice does not create duplicates
       within this import (idempotency key per upload).
- AC5: a user from org A can never see or affect org B's rows
       (RLS test, not a UI test).
- AC6: works on Chrome + Safari, and on a mid-range Android
       over throttled 3G.

Notice what that brief quietly does. AC5 forces a security property the happy path would never exercise. AC3 and AC4 encode hostile-input thinking before the model can omit it. The OUT list removes the tangents an agent would otherwise wander down. And every criterion converts one-to-one into either an automated test (L4) or a checklist line (L3). If you want the full discipline this is a slice of — treating a structured spec as the sovereign artifact your agent builds from, with tooling from Spec-Kit to Kiro to Cursor Plan Mode — we have covered it end to end in our guide to spec-driven development with AI coding agents; this article deliberately stays on the QA side of that line.

The three failure modes that kill vibe-coded apps

Across the 2026 practitioner accounts, three failure patterns recur so consistently that you should treat them as scheduled events rather than surprises. Each has a distinct signature and a distinct fix.

1. Context decay: the wall between prompts 10 and 20

Every long vibe-coding session degrades, and practitioner write-ups typically place the wall between roughly prompt 10 and prompt 20, once the app has accumulated enough moving parts that the model's working memory of the project frays. The symptoms are unmistakable: the agent contradicts decisions it made earlier, re-implements a function that already exists, or asks whether you would like a users table when it built one itself eight prompts ago. From that point, every additional prompt is applied to a model whose picture of your app is partly fictional — which is how one broken fix becomes five.

The fix is not to push through; it is to summarise and restart. When you feel the decay, stop, have the agent write a short recap of the current state — what exists, what is decided, what is next — into a file in the repo, then open a fresh session that loads only that recap. You lose nothing that matters and shed the accumulated confusion. Teams running long agentic builds formalise this into checkpoint discipline: commit at every working state so you can roll back to the last good one instead of excavating a haunted session. Our piece on dynamic Claude Code workflows goes deeper on steering long runs without losing the thread.

2. Authentication: where sessions go to die

If context decay is the chronic disease, custom auth is the acute one. Practitioner reports are blunt that hand-rolled authentication — JWT rotation, OAuth flows, session management — is where vibe-coding sessions most reliably break beyond repair. The reason is architectural: auth touches every other system in the app, so a subtle mistake made mid-session does not stay local. It propagates into every route, every fetch, every guard, and the model's attempts to patch one symptom create three more. Security compounds the stakes — a plausible-looking auth flow that is subtly wrong is worse than one that visibly fails.

The fix is to refuse the fight. Use a managed provider — Supabase Auth, Clerk, Auth0, Firebase Auth — which the models have seen thousands of times in training data and which reduces auth to configuration rather than invention. Put "do not build custom auth" in the constraints section of your brief, as the example above does, so the decision is made before the agent can unmake it.

3. The scale cliff: fine at 100 users, gone at 10,000

The third failure mode is the quietest, because nothing is wrong until suddenly everything is. AI-generated code has a habit of running comfortably at demo scale and falling over once real traffic arrives — where the cliff sits depends entirely on your workload, but the mechanisms are consistent: N+1 query patterns that were milliseconds at demo scale, missing indexes, no rate limiting, no caching, unbounded list endpoints, and row-level security that was never switched on because the demo had one user who owned everything. A production checklist for AI-generated Next.js and Supabase code reads like a list of exactly these omissions: enable RLS on every table, validate all input server-side, parameterise queries, rate-limit API routes.

As of July 2026, Next.js plus Supabase is among the most widely recommended stacks for AI code generation — both are heavily represented in training data, and many app-builder tools default to the pair — which is precisely why the scale cliff deserves respect: the stack makes starting so easy that teams forget what they are holding. As one production architecture deep-dive puts it, most Supabase incidents are not Supabase failing — they are teams treating Postgres-with-batteries like a magic box and never opening the SQL editor until real traffic arrives. The fix is not to change stack; it is to schedule the cliff. Before launch, run one honest load test at ten times your expected traffic, read the slow-query log, and add the indexes and limits the model never knew you would need.

The pre-deployment checklist

Everything above converges on a concrete artefact: the checklist you run, in full, before anything ships to a stranger. This is Level 3 of the maturity model in executable form, and it costs an hour. The four pillars come straight from the practitioner canon — real device, every flow, bad inputs, persistence — with the production basics AI most often omits bolted on.

• Real device, real network. Not your laptop. A mid-range Android over mobile data — the median device for an Indian user and far more common in the UK than London product teams assume. Throttle to 3G in DevTools and watch what loading actually feels like.
• Every user flow, end to end. Not the flow you demoed — all of them, including sign-out, password reset, cancel, refund, delete account and the empty states a brand-new user sees. If a flow exists, a user will find it in week one.
• Bad inputs, deliberately. Empty fields, absurdly long strings, emoji and non-Latin scripts, files that are too big or the wrong type, double-clicked submit buttons, back-button replays, and expired sessions resubmitting a stale form. You are not trying to break it; you are confirming your users cannot.
• Persistence and time. Create data, log out, close the browser, come back the next day on a different device. Is everything still there, still correct, still yours — and only yours?
• Security floor. RLS enabled on every table and tested from a second account; no API keys or secrets in client-side code; rate limits on every write endpoint.
• One load test. Ten times expected traffic against the two heaviest endpoints; read the slow-query log afterwards and fix what it shows you.

Then make the checklist self-enforcing by promoting its most important lines into CI — which is the L3-to-L4 move. Your coding agent will happily write the tests from your acceptance criteria; your job is to make them a gate rather than a suggestion:

# .github/workflows/qa-gate.yml
name: qa-gate
on:
  pull_request:
  push:
    branches: [main]

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22, cache: "npm" }
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm test -- --coverage        # unit + contract (AC1-AC5)
      - run: npx playwright install --with-deps chromium
      - run: npm run test:e2e              # critical flows on a real browser
      # Deploys are blocked unless every step above is green.

The point of the file is not the specific commands — swap in your own — but the property it creates: from now on, the QA runs whether or not you remember it, whether or not you are tired, whether or not the launch is exciting. That property is the entire difference between Level 3 and Level 4.

Making it a repeatable discipline

A checklist you ran once is a story; a checklist you run every time is a discipline. The difference is structural, and three habits build it.

Ratchet one level per project. Do not attempt to jump from L1 to L5 in a weekend — you will produce theatre, not QA. Take whatever you ship next from its current level to the one above and hold it there. A solo builder in Pune whose side project moves from spot-checks to a written pre-deploy checklist has made a bigger real-world reliability gain than a team that bolts a flaky end-to-end suite onto a codebase nobody trusts. The ladder is climbed a rung at a time, and each rung is cheap.

Put the discipline in files, not in memory. The brief template, the checklist and the CI workflow should all live in the repository, version-controlled next to the code they govern. This is what makes the practice survive contact with a deadline — and it is what makes it transfer. The same written-down discipline is exactly what lets you point coding agents at riskier work later: our guide to migrating legacy codebases with AI agents is, at bottom, this article's checklist pointed at someone else's ten-year-old code.

Close the loop after launch. Level 5 is not more pre-release testing; it is admitting that some failures only exist in production and instrumenting for them — error tracking, a couple of alerts on the endpoints that matter, and a rollback path you have actually rehearsed. When an incident happens, the mature move is to convert it into a new acceptance criterion so the same class of failure can never ship silently again. Over a few cycles this compounds into the thing users experience as "this product is solid" — which no individual feature ever produces.

And say the quiet part about careers out loud. In a market where anyone can generate a demo, the credential that moves hirers in Bengaluru and London alike is evidence of the full arc: specified, built, tested, shipped, survived. A production project with real users is the one line on a profile that AI cannot inflate, because the users are the proof. If you have carried something across the demo-to-product gap this guide describes, that project — with its uptime, its user count, its boring green CI badge — belongs on your Builder profile, where the people hiring can actually find it.