Driving a coding agent from a spec
This block explains how to drive a coding agent from a written spec, which parts of that process really hold the agent to it, and where the process is a waste of time.
A Northwind engineer asks a coding agent to "add partial refunds to the admin panel". An hour later there is a pull request of 1,900 lines with 47 green tests, all written by the agent. The code also refunds shipping on partial refunds, which finance never allowed. The tests pass, because the same misreading wrote them.
A very literal contractor
Imagine telling a contractor "redo the kitchen" and leaving for a month. They are skilled and they do not ask questions, so they fill every gap with a guess. You find out which guesses were wrong when the cabinets are already on the wall.
So you write a work order: what you want, what must not change, and how you will know it is done. You look at the drawings before anyone drills. And you bring your own tape measure, because the contractor saying "it's level" does not make it level.
A spec is the work order for a coding agent. The term is still loose. One engineer's attempt at a definition, from her hands-on review of these tools, is a structured, behaviour-oriented document or set of documents in natural language that guides the agent. Here is one for Northwind:
# Partial refunds in the admin panel
Why: staff can only refund whole orders, so they issue
store credit by hand for damaged items.
Who: support staff with the refunds role.
Must
R1. Staff can refund from 1 cent up to the order's remaining
refundable total.
R2. Shipping is refundable only with a full refund.
R3. Refunds of one order can never exceed what was paid,
even when two are submitted at the same moment.
Must not
N1. No change to the customer-facing support agent.
Done when
A1. Given order 4417 paid 34900 with 10000 already refunded,
a refund of 24901 returns 422 and calls no payment API.
A2. Given two simultaneous refunds of 20000 on that order,
exactly one succeeds.You write the why and the limits. Only you know what finance allows. GitHub's explanation of why this helps is that a vague prompt forces the model to guess at potentially thousands of unstated requirements.
Must not draws the edge of the work, so anything outside it can be rejected in review.
Done when is written so that a program can check it. Each line names a state, an action and a result.
Who does what at each stage
Spec Kit and Kiro differ in detail, but they share one shape: the agent writes a document, and you check it before the next one exists.
GitHub's Spec Kit runs /speckit-specify, /speckit-plan, /speckit-tasks and /speckit-implement (names in GitHub Copilot's default syntax), after a once-per-project constitution of standing rules. Its current README adds /speckit-converge, repeated with implement until it reports Converged, and says to review each result before continuing. Kiro produces requirements.md, design.md and tasks.md. Both end in a list of separate tasks.
The spec does not bind the agent
A spec is more text for the model. In that review's trials from September 2025, an agent treated notes about existing classes as new requirements and generated duplicates, and it followed other instructions too eagerly. Spec Kit's checklists inside the spec files are read by the AI too, so nothing guarantees that they are respected.
What holds is a check that runs as code. The check has to come from the spec and not from the agent's understanding of it.
In the Reflexion paper, on MBPP Python problems with GPT-4, 16.3% of the solutions that passed all the model's own tests were still wrong. On HumanEval Python the rate was 1.4%. Those are small functions from 2023, so read it as evidence that the risk is real, and not as a rate for your repository.
Anthropic reported a related failure with long-running agents: absent explicit prompting, Claude tended to mark features complete after unit tests or curl, while the feature did not work end to end. For a web app, asking it to test through browser automation as a user would helped. Its harness kept requirements in a JSON file that the agent was told to change only in one field. Here is such an entry, rewritten for Northwind:
{
"id": "A1",
"description": "Refund above the remaining total returns 422",
"steps": [
"Open order 4417 with 10000 already refunded",
"Submit a refund of 24901",
"Check that no payment call was made"
],
"passes": false
}Anthropic chose JSON because the model was less likely to overwrite it than Markdown. As the post describes it, that rule was an instruction in the prompt. Your CI can enforce it: compare the file before and after, fail if anything other than passes changed, and accept true only with a passing command behind it.
How to review what comes back
One engineer's rule, from a post linked below, is that the person submitting a change must show that it works: commands with their output, and a test that fails if the change is reverted. Agents can be made to produce this too, and you remain the one accountable.
Order | What you read | Why then |
|---|---|---|
1 | Spec and plan | A mistake here is cheapest to fix |
2 | Edited or deleted tests | The fastest way to get green is to change the test |
3 | Evidence: commands, output, a test that fails on revert | Without it, the diff is a claim |
4 | Small diffs against the spec | Checks that nothing outside the spec crept in |
When not to write one
The same reviewer asked Kiro to fix a small bug and got 4 user stories with 16 acceptance criteria. On a mid-sized feature with Spec Kit, which she did not finish, she judged that plain AI-assisted coding would have taken no longer, and preferred reviewing code to that much Markdown. The tools have moved since: Spec Kit now has a separate bug-fixing extension, and Kiro has bugfix specs and a Quick Spec. The judgment is still yours. A spec pays off on a feature with unclear edges. A one-line bug needs a failing test.
How long a spec lives is a separate choice. The review names three levels: spec-first uses the spec for the task at hand, spec-anchored keeps it for later changes to the feature, and spec-as-source makes it the only file people edit.
A kept spec must be maintained, or it starts to lie. Model-driven development, an older practice of generating code from formal models, never took off for business applications, because of its overhead and constraints. The reviewer wonders whether generating code only from specs will end up with those downsides plus the model's non-determinism, which she saw when generating code several times from one spec.
Conclusion
- Write the why, the limits and checkable acceptance criteria. Review each stage before the next one.
- A spec guides the agent. Only checks that run hold it, and they must come from the spec.
- Read specs, edited tests and evidence before the diff.
- Match the ceremony to the size of the change.