Verification, not vibes

How to tell when a coding agent has actually finished

Every harness that runs a coding agent eventually has to answer one question: is this task done? The three signals you reach for first all fail, in ways that are quiet enough to cost you an afternoon.

Disclosure: I build Ordewell, and this page is how it answers that question. The method is not a product feature though. It is a convention you can put in any harness in an afternoon, and the pitfalls below will bite you in any language, so they are worth reading even if you never run my tool.

The three obvious signals, and why they fail

1. The terminal went quiet. An interactive agent TUI does not close when the work is finished. It stays open, waiting for you to say something else. Silence means the model stopped printing, which is also what it does while it thinks, waits on a tool, or gives up.

2. The process exited cleanly. Exit code 0 means the program ended without an error. An agent running out of context, hitting a stop condition, or deciding the task was already satisfied exits 0 too. The exit code tells you the process finished, not the work.

3. The agent said it was done. This is the one that hurts, because it is confident. Models announce success over a build that does not compile, in a tidy paragraph, with a summary of changes that were only partly made. When a coordinator agent supervises worker agents, a version of this scales up: a model reading a model's report and ruling on it.

A report, not a paragraph

Every agent Ordewell runs is connected to a small tool server that ships with Ordewell. The task prompt ends with one instruction: when the work is finished, call task_complete with status done and a summary of what you did. If you can't finish, call it with blocked or failed and say why. The call also takes the handoff the next tasks will read: numbered findings with their evidence, artifacts, a commit message for a change task, and an optional long report stored with the attempt.

Three details make that call trustworthy:

• It is bound to the attempt. Each attempt gets its own credential for the tool server. A call from a retried attempt, another task, or a session that outlived its run cannot settle this one.
• It has to be connected, but a miss no longer fails the task. Ordewell checks the tools are attached right after the agent starts, and restarts the agent once if they aren't. If they still aren't, the task is held: the reason sits on its badge, nothing of it runs, and its dependents are not blocked. A runner that could never report is a start that never happened, not a verdict.
• The handoff is useful. What the agent leaves in the call (a summary for the next reader, numbered findings with their evidence, artifacts, and a commit message for a change task) becomes the context the next tasks receive, with a long report stored beside the attempt for whoever pulls it.

The same server carries checkpoint, which pauses a task and asks you to approve or reject before it goes further. That turns "review the diff at the end" into "decide at the risky step".

What we tried first

Earlier versions of Ordewell did this with a printed marker: a unique token per task that the agent was asked to print on its last line, and that Ordewell watched for in the terminal output. It worked, and it taught us why it shouldn't be the answer.

A terminal is a rendering protocol, not a document. Colour codes, box drawing borders and soft wrapping split a token across writes, and full screen TUIs repaint a spinner in the middle of it. We ended up flattening the stream and replaying cursor moves to reconstruct the screen, just to read one line. The token also had to be kept out of the prompt itself, and out of anything handed to the next task, or the wrong task could pass on someone else's evidence.

A tool call has none of those problems. It arrives as structured data, from a known attempt, with a status and a reason attached. As of 0.8 it is the only thing that completes a task, and printed markers are no longer read at all.

When the report never comes

The goal is to make that state unmistakable. The session ended, maybe the model even claimed the work in its last message, and no report arrived. That is a failed verification, with the exit code kept beside it as separate evidence. These are the messages the task card can carry:

Verified: the runner reported completion through task_complete. Task completed successfully.

Failed verification: the runner exited cleanly but never called task_complete.

Note what the exit code does not do: it never overturns the report. A clean exit without a report is still a failure, and a blocked or failed report ends the task with the reason the agent gave.

Honest limits

• A report proves a claim, not correctness. It shows the agent finished and said so. It does not show the tests pass, the build compiles, or the change is right. Nothing here replaces reviewing the diff.
• An agent can report without doing the work. The report is evidence that the session ended on purpose, not proof of the result. A task specific check in the plan, like running the test suite as its own task, is the answer to that, and it is a real gap in the method rather than a footnote to it.
• It needs a tool server. This only works for agents that can attach one. Claude Code, Codex and OpenCode all can, which is why they are the agents Ordewell runs.
• Manual overrides exist and should feel uncomfortable. Marking a task complete by hand records that no automatic verification was performed. It is honest, and it should read like a warning.

If you want it already wired

Ordewell attaches the tool server, binds each attempt, runs one real coding agent session per task, and settles each one on its report. That is the whole loop:

ordewell plan --goal "Add rate limiting to the public API"
ordewell run

The planner reads your repo without changing it, so planning needs no API key: a coding agent you already pay for can be the planner. Install with npm install -g ordewell, or read the docs first. The source for everything on this page is in the repo.