From Vibe Coding to Spec-Driven Development to Loop Engineering
On a Friday afternoon, someone on the ops team builds a dashboard in an hour. They describe it to a coding agent, click around, ask for a couple of changes, and share the link. Everyone loves it. On Monday a billing service renames one field, the dashboard quietly shows zero revenue for three regions, and nobody notices until Wednesday, because the only test the dashboard ever had was a person looking at it on Friday.
Nobody did anything wrong. The dashboard was built the right way for a prototype, and then it stopped being a prototype without anyone deciding that it had.
Every team that builds seriously with AI walks the same road, usually without noticing. It starts with a chat window and a good feeling. If the work survives contact with production, it ends somewhere much more disciplined. This post describes the three stages we see teams pass through, what each one is good at, what it looks like with the tools of late 2026, and the specific failure that forces the jump to the next.
The three names get used loosely. They are not moods. They are three different answers to one question: who is responsible for correctness, and how is it checked? The answer changes as the stakes rise.
The short version
| Stage | The unit of work | Who checks correctness | Typical tools in 2026 | Breaks when |
|---|---|---|---|---|
| Vibe coding | A prompt | A human, looking at the output | Claude Code, Codex, Cursor, Claude Design, app builders | You can't hold the whole system in your head |
| Spec-driven development | A specification | The spec and its acceptance tests | Spec Kit, Kiro, Tessl, plan modes in coding agents | The spec is right but the run is unreliable |
| Loop engineering | A loop | The system, verifying and correcting itself | Agent harnesses, eval suites, CI, Ralph-style loops, graders | The verifier is weaker than the generator |
Read top to bottom, it is a story about moving the job of checking out of a person's head, into a document, and then into a running system.
Each arrow is labelled with the failure that forces the move. That failure is the most useful thing to learn about each stage, so every section below ends with it.
Stage one: vibe coding
Where the term came from
Andrej Karpathy named it in a post on February 2, 2025: a new kind of coding "where you fully give in to the vibes, embrace exponentials, and forget that the code even exists." He later called it a throwaway shower thought. It named a movement anyway, because it described something millions of people had just started doing.
Simon Willison drew the line that keeps the term useful. If you review, test, and understand every line the model writes, you are not vibe coding. You are programming with an assistant. Vibe coding specifically means accepting code you have not read, and judging it only by what it does on screen.
What it looks like now
In early 2025 vibe coding meant a chat window and a single file. By late 2026 it means an agent working inside a real repository, running the app, and showing you the result. A product manager prototypes a feature on the real product. A designer explores five layouts in Claude Design before lunch. A founder builds an internal tool by talking to Codex. The loop is the same as it always was:
You: Build a page that lists our open roles from the careers API.
Agent: Writes about 200 lines, starts the dev server, opens the page.
You: Scrolls and clicks. Close. Make the cards clickable and sort by team.
Agent: Revises the code and reloads.
You: Looks. Ship it.
It is great for a prototype, a throwaway script, a UI you can see with your own eyes, and the first exploratory hour of any project. Feedback is immediate and being wrong costs you one more prompt. In a talk at Sequoia's AI Ascent in April 2026, Karpathy put the value precisely: vibe coding raises the floor. People who could not build software at all now can.
Where it breaks
The reason it works is also the reason it doesn't scale: the only verifier is a person looking at the screen. That is fine when "correct" means "looks right in the browser." It fails as soon as correctness depends on something you can't see at a glance.
Karpathy gave a good example from building his own app, MenuGen. The agent linked a user's payments to their account by matching email addresses instead of a stable user id. Everything looked right on screen. It would have broken the first time someone changed their email. No amount of clicking around would have revealed it, because the flaw was in the design, not the UI.
The things a screen cannot show you fall into a few groups:
| What you can't see | Example | What eventually catches it |
|---|---|---|
| Wrong identity or data model | Users matched by email instead of id | A design review, or an incident |
| Silent regressions | A change in one place breaks a page you didn't reopen | Automated tests |
| Contracts with other systems | A renamed field upstream shows zero revenue | Contract tests, monitoring |
| Rare inputs | The 10,000th request, the empty list, the emoji in a name | Property tests, evals on real traffic |
| Quality rot | Defensive code piling up, deprecated APIs, duplicated helpers | Code review, linters |
That last row is what Karpathy meant on the Dwarkesh Podcast in October 2025 when he called a lot of current agent output "slop": code that works today and costs you next quarter.
The tell that you've outgrown vibe coding: you find yourself re-checking the same things by hand every time you make a change, because you no longer trust that a small edit didn't break something out of view. That manual re-checking is a test suite. You just haven't written it down yet.
Stage two: spec-driven development
The idea
Spec-driven development moves the source of truth out of the conversation and into a document. Instead of steering the model turn by turn, you write down what "correct" means: the behaviour, the constraints, and the acceptance criteria. The model implements against that. The spec becomes the thing you review and version, and the code becomes an output of it.
The shift is small to describe and large in practice. In vibe coding you review code, or nothing. In spec-driven development you review the spec, and let the tests review the code.
A spec doesn't have to be heavy. Often it fits on one screen:
Behaviour
- Fetch open roles from the roles API.
- Show role title, team, location, and an apply link.
- Sort by team, then by title, both alphabetically.
Constraints
- Roles whose status is not "open" must never be shown.
- If the API is unreachable, show the last cached response with a "stale" badge.
Acceptance criteria
- AC-1: A closed role in the test data never appears on the page.
- AC-2: With the API returning an error, the stale badge appears.
- AC-3: Sorting is stable and matches the expected order.
Now correctness lives in two places you can point at: the constraints (what must be true) and the acceptance criteria (how you'll know). The model can write the implementation, rewrite it, or a different model can, and the tests decide whether the result is acceptable.
Three levels of commitment
In October 2025, Birgitta Böckeler at Thoughtworks published the clearest map of this space. She separated three levels, and the difference matters because most teams think they are at a higher level than they are:
| Level | What happens to the spec | What you edit | Example tools |
|---|---|---|---|
| Spec-first | Written before coding, then left behind once the feature ships | Code, after the first version | Kiro (requirements, then design, then tasks) |
| Spec-anchored | Kept and updated as the feature changes | Spec and code, together | Spec Kit aims here (constitution, specify, plan, tasks) |
| Spec-as-source | The only artifact humans edit; code is generated and read-only | Spec only | Tessl is exploring this |
Most teams in 2026 practise spec-first while believing they practise spec-anchored. You can tell by asking when the spec was last edited. If the answer is "when the feature was built," the spec is a historical document and the code is the real source of truth again.
What the tools add
GitHub's open-source Spec Kit, released in September 2025, turned the process into commands a coding agent follows: write a project constitution (the rules every change must obey), specify the feature, produce a technical plan, break it into small tasks, then implement. Amazon's Kiro builds the same idea into an editor, with requirements, design, and task documents for each feature. Plan modes in Claude Code and Codex are the lightweight version: the agent proposes a plan, a human approves it, and only then does code change.
Willison's October 2025 essay "Vibe engineering" is a useful checklist of what has to surround a spec for it to work: automated tests, advance planning, documentation the model can read, version control discipline, CI, code review, manual QA, and preview environments. None of these are new. Agents amplify whatever engineering practice you already have, including a weak one.
Where it breaks
Böckeler's review found the tools' limits quickly. The workflows can be overkill for small changes. More importantly, agents often ignore parts of a spec or over-apply it, even with detailed instructions and checklists. A spec does not make a model obey it.
The deeper issue is that a spec describes a destination, not a journey. It tells you what a correct result looks like. It says nothing about what to do when the run that is supposed to produce that result is unreliable, which is exactly the situation once an agent is doing the work at runtime, against inputs you didn't write test data for.
Karpathy summed up the shift in one line at AI Ascent: traditional software automates what you can specify; AI automates what you can verify. A spec is necessary. The verifier is what makes it true.
Stage three: loop engineering
The idea
Loop engineering is what you build when the thing doing the work is an AI system and you are not in the chair. The spec is still there, because you still need to know what correct means. But the thing that checks the work, catches the mistakes, and tries again now has to be part of the system, because you're asleep, or handling ten thousand runs at once.
The unit of work is no longer a prompt or a spec. It is a loop: generate a candidate, verify it against the spec, and if it fails, feed the failure back and try again, until it passes or the budget runs out.
The loop itself is trivial. Everything hard lives inside the verifier and the feedback.
The loop in the wild
The simplest version became famous under a silly name. In May 2025 Geoffrey Huntley described the Ralph Wiggum loop: run a coding agent in a shell loop, feed it the same prompt file every time, and don't let it stop until the task is done. "Ralph is a bash loop," as he put it. The key detail is where progress lives. It is not in the model's context window, which gets thrown away. Progress lives in files and git history, so each fresh run picks up where the last one stopped. The pattern is crude, and it works better than it has any right to.
Anthropic's engineering team published the grown-up version in November 2025, in "Effective harnesses for long-running agents." Their harness for multi-hour coding work has four habits worth copying:
- Set up before building. A first session writes the scaffolding: a setup script, a progress file, and the full list of features to build.
- One feature at a time. Each session takes one item from the list, instead of trying to build everything and running out of context halfway.
- Leave a clean state. Every session ends with a commit and a progress note, so the next session starts from something known.
- Test like a user before saying "done." The agent drives a browser through the feature end to end, which catches bugs that reading the code would miss.
Look at those four habits and you'll see a good engineer's routine, written down so an agent can follow it.
The verifier has to be real
A loop is only as honest as its weakest check. A verifier that approves everything is worse than none, because it produces false confidence at scale. Verifiers come in layers, from cheap and certain to expensive and fuzzy:
| Verifier | Checks | Strength | Weakness |
|---|---|---|---|
| Types, linters, schemas | Structure | Instant and certain | Say nothing about behaviour |
| Unit and end-to-end tests | Behaviour on known inputs | Repeatable | Only cover what someone thought of |
| State checks | Did the world change correctly: row written, refund issued, page updated | Checks outcomes, not claims | Need access to the system of record |
| Model-based graders | Fuzzy quality: tone, completeness, relevance | Cover what code can't | Can be wrong, need calibrating |
| Humans | Anything | Final judgement | Slow, expensive, inconsistent |
Anthropic's January 2026 guide, "Demystifying evals for AI agents," gives practical advice for building these layers. Start with 20 to 50 tasks drawn from real failures rather than waiting for a perfect suite. Prefer checking the resulting state over checking what the agent said it did. Combine code-based graders, model graders, and human calibration. And read the transcripts regularly, because a grader can be broken in ways only a human reading the runs will notice.
Ilya Sutskever raised the warning that belongs next to all of this. On the Dwarkesh Podcast in November 2025, he pointed at the gap between how well models score on evals and how much less impressive they are in real use, and suggested that training aimed at evals is part of the reason. The same thing happens inside your loop. An agent optimising against your verifier will find the verifier's blind spots. Treat the eval suite as something that drifts out of date, and keep adding the failures you see in production.
Feedback has to be specific
"Failed" teaches the model nothing. "Role r_912 had status closed but appeared on the page in position 3" tells it exactly what to fix. In practice, the quality of the feedback message is the quality of the loop. When a loop thrashes (the same failure several times in a row), look at the failure text before you look at the model.
The loop needs a budget and an exit
Real loops cost money and can spin. Bounded attempts, then a clean handoff to a human with the whole trace, is not a failure mode. It is the design. This is Karpathy's "autonomy slider" from his June 2025 talk at YC's AI Startup School ("Software Is Changing (Again)"): products should let people choose how much autonomy to give, and pull it back when something goes wrong. On the same podcast where he called agent output slop, he called this "the decade of agents," not the year. Loops with good exits are how you get value during that decade without pretending agents are finished.
Why this is the stage that ships. Most AI projects that stall do so not because the models are weak, but because they are wrapped like demos: one attempt, no verifier, and a human implicitly in the loop who quietly isn't there at 3 a.m. Loop engineering is what a demo becomes when you make the checking explicit and the human optional.
The stages absorb each other
Loop engineering doesn't discard the earlier stages. It contains them.
The spec supplies the criteria the verifier checks against. The vibe coding instinct (generate fast, react to what you see) becomes the inner generate step, except the thing reacting is now the verifier instead of a person.
This is also why Karpathy's 2026 name for the discipline, agentic engineering, fits alongside ours. He describes it as raising the ceiling where vibe coding raises the floor: the engineer writes specs, supervises plans, inspects diffs, builds evaluation loops, and keeps the judgement and taste the agents lack. His line from AI Ascent is the one to remember: "You can outsource your thinking, but you can't outsource your understanding."
How we think about it at CatalEx
Running agents in production is loop engineering as a full-time job. The platform features we talk about map directly onto parts of a well-built loop:
| CatalEx capability | The part of the loop it is |
|---|---|
| Self-correcting agents catch and fix their own mistakes mid-run | The fail, diagnose, and retry edge, made reliable |
| Self-metriced agents help choose and track the criteria they're judged on | The acceptance criteria, kept current instead of frozen on day one |
| Self-evolving agents improve from each outcome | Feedback from thousands of loops shaping how the next run is produced |
None of it is exotic once you see it as one control loop drawn at different sizes. The demo is the inner box. The spec is the criteria. The loop is the product.
Where you actually are
You don't pick a stage as an identity. You pick it for each piece of work.
- Exploring, prototyping, building something you can see? Vibe code it. Speed is the point, and you are a good enough verifier for things that fit on a screen.
- Building something that must stay correct as it changes, with more than one person or model touching it? Write the spec. Move correctness out of your head and into acceptance criteria.
- Putting an AI system in charge of doing the work unattended, at scale, against inputs you didn't anticipate? Engineer the loop. Make checking and correcting part of the system, because you won't be there.
The mistake is never being at an early stage. Vibe coding a prototype is correct. The mistake is staying at a stage after the work has quietly moved past it:
| The mistake | What it looks like | The fix |
|---|---|---|
| Shipping a vibe-coded prototype as a product | The Friday dashboard is still running in March | Decide explicitly when something stops being a prototype, then write its spec |
| Treating a spec as proof | "The spec says closed roles are hidden," with no test that checks it | Every acceptance criterion gets a test |
| A spec nobody updates | The spec describes last quarter's feature | Edit the spec first on every change, or admit it is spec-first |
| A loop with a soft verifier | Every run passes, and customers still hit bugs | Add state checks and production failures to the eval set |
| A loop with no exit | An agent retries for six hours and burns the budget | Cap attempts and hand off with the trace |
Each stage exists to fix the specific way the previous one lies to you once the stakes go up. Know which lie you can currently afford, and build the next loop before you need it.
Watch and read
The ideas above come from a small number of talks and essays. These are worth your time in full.
Talks and podcasts
- Andrej Karpathy: Software Is Changing (Again), YC AI Startup School, June 2025. Software 3.0, partial autonomy, and the autonomy slider. The best single hour on how products should share control with AI.
- Andrej Karpathy: AGI is still a decade away, Dwarkesh Podcast, October 2025. An honest account of where coding agents fall short, and why he calls this the decade of agents rather than the year.
- Andrej Karpathy at Sequoia AI Ascent 2026, April 2026 (his own written summary). Vibe coding versus agentic engineering, specify versus verify, and the MenuGen example.
- Ilya Sutskever: We're moving from the age of scaling to the age of research, Dwarkesh Podcast, November 2025. Why models look better on evals than in real use, and what that means for anyone relying on benchmarks.
- Ilya Sutskever: Sequence to sequence learning, what a decade, NeurIPS 2024 Test of Time talk. Short, and the origin of "pre-training as we know it will end," which set up the shift toward agents and reasoning.
Essays and guides
- Andrej Karpathy's original "vibe coding" post, February 2025. Where the term started.
- Simon Willison: Not all AI-assisted programming is vibe coding, March 2025. The definition that keeps the term useful.
- Simon Willison: Vibe engineering, October 2025. The practices that make agent-written code trustworthy.
- Birgitta Böckeler: Understanding Spec-Driven Development: Kiro, spec-kit, and Tessl, martinfowler.com, October 2025. The three levels of spec-driven development, and an honest review of the tools.
- GitHub: Spec-driven development with AI, September 2025. The introduction to Spec Kit and its constitution, specify, plan, tasks workflow.
- Anthropic: Building effective agents, December 2024. The workflow patterns, including the evaluator-optimizer loop that underlies stage three.
- Anthropic: Effective harnesses for long-running agents, November 2025. How to keep an agent making progress across many sessions.
- Anthropic: Demystifying evals for AI agents, January 2026. How to build the verifier: graders, tasks, and transcripts.
- Geoffrey Huntley: Ralph Wiggum as a "software engineer", 2025. The bash loop that made loop engineering concrete for a lot of people.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.