Verification, not vibes

How to tell when a coding agent has actually finished

Every harness that runs a coding agent eventually has to answer one question: is this task done? The three signals you reach for first all fail, in ways that are quiet enough to cost you an afternoon.

Disclosure: I build Ordewell, and this page is how it answers that question. The method is not a product feature though. It is a convention you can put in any harness in an afternoon, and the pitfalls below will bite you in any language, so they are worth reading even if you never run my tool.

The three obvious signals, and why they fail

1. The terminal went quiet. An interactive agent TUI does not close when the work is finished. It stays open, waiting for you to say something else. Silence means the model stopped printing, which is also what it does while it thinks, waits on a tool, or gives up.

2. The process exited cleanly. Exit code 0 means the program ended without an error. An agent running out of context, hitting a stop condition, or deciding the task was already satisfied exits 0 too. The exit code tells you the process finished, not the work.

3. The agent said it was done. This is the one that hurts, because it is confident. Models announce success over a build that does not compile, in a tidy paragraph, with a summary of changes that were only partly made. When a coordinator agent supervises worker agents, a version of this scales up: a model reading a model's report and ruling on it.

The fix is to stop asking for an opinion and start watching for evidence. A task carries a unique token. The prompt asks for that token on the final line. The harness watches the runner's output for it. Token present means pass. Token missing means fail, loudly, even when the session exited cleanly.

It cannot be a simple string search

Here is the part that surprised me. Watching for a token in terminal output sounds like output.includes(token). It is not, and the reason is that a PTY stream is a rendering protocol, not a document.

The raw bytes carry ANSI colour codes, OSC sequences, box-drawing gutter characters from the TUI's borders, cursor moves, and line erases. Interactive TUIs soft-wrap long lines, so a single token can arrive split across two writes with an escape sequence in the middle. A raw string search misses it.

The first pass is a flatten: strip escapes, strip the box-drawing range, strip all whitespace. Dropping whitespace sounds reckless until you notice the token itself has none, so the flattened view can be scanned safely.

scan, flattened

// strip ANSI/OSC escapes, box-drawing gutters, then every space
function flattenTerminalOutput(raw) {
  return raw
    .replace(ANSI_OR_CTRL_RE, /* escapes, control bytes */ "")
    .replace(/[─-▟]/g, "")   // TUI borders and gutters
    .replace(/\s+/g, "");
}

That handles the byte-stream case. It does not handle the screen case.

A full-screen TUI is not a stream

Some agent TUIs paint fragments at absolute cursor positions and repaint unrelated widgets between those writes. Frame a spinner, then finish the token on a row that was already partly drawn. Flattened chronologically, the spinner lands in the middle of the token and nothing matches:

what the bytes look like

row 9:  <<<ORDEW
row 23: ⟳ spinner repaint
row 9:  ELL_DONE_9f3c…>>>

So there is a second pass: replay the cursor and erase sequences to reconstruct the small screen the user is actually looking at, then scan that. The chronological flatten stays as the fallback for plain piped output and for soft-wrapped lines.

Two scans, one verdict. If either sees the token, the task passed. Being strict here costs throughput for no gain, because a false negative is the failure that erodes trust.

The three rules that make the token trustworthy

Rule 1: one token per task. Not per plan, not per run. Generate it when the task is created, keep it on the task, and scan for that exact string. A shared token across tasks means task six passes on task two's evidence.

Rule 2: the instruction must not contain the token. Whatever you append to the prompt gets echoed into the session, and you are scanning that session's output. A literal token in the instruction would settle the task before the work started. So the instruction asks the model to assemble the token from two pieces, and the assembled form never appears in the text you send:

what the runner is asked for

When you have fully completed this task, print one final line containing
only the completion marker. Build it by writing `<<<ORDEWELL_`
immediately followed by `DONE_<task token>>>`, joined into a single
unbroken token, with no space, quote, or any other character between
the two parts.

Rule 3: break tokens in anything you hand to the next task. Tasks receive their predecessors' notes and summaries. If a finished task's captured output still carries a live token and that text becomes context for the next one, the model can echo it back and settle the wrong task. One replacement when the notes are assembled is enough: the token prefix gets a hyphen spliced into it, and it can never match again.

When the token never shows up

The mission is to make that state unmistakable. The session ended, the model claimed the work, and no token arrived. That is a failed verification, with the exit code kept beside it as separate evidence rather than as a verdict in its own right. These are the two messages the card can carry:

verdict

Verified: completion marker detected in agent output. Task completed successfully.

Failed verification: agent exited cleanly but did not emit the completion marker.

Note what the exit code does not do: it never overturns the token in either direction. A clean exit after a missing token is still a failure. A non-zero exit after a seen token is still a pass, because the agent did the work and then something unrelated tripped on the way out.

It is also worth giving long tasks a mid-run checkpoint. The same output scan can carry a second token type that pauses the task and asks you to approve or reject before it goes further, which turns "review the diff at the end" into "decide at the risky step".

Honest limits

A marker proves a claim, not correctness. It shows the session finished and said so. It does not show the tests pass, the build compiles, or the change is right. Nothing here replaces reviewing the diff.
The scan is bounded. Ordewell scans the recent tail of the output buffer, not the whole session, so a token printed an hour before the session ends can in principle sit outside the window. That is a deliberate trade for not re-flattening an unbounded buffer on every write.
Trimming output can hide evidence. If your harness caps or truncates what it keeps, you can hide your own token. Keep the tail, drop the middle.
Manual overrides exist and are humiliating. Marking a task complete by hand records that no automatic verification was performed. It is honest, and it should feel like a warning.
This is a convention, not a proof. A model that prints the token without doing the work defeats it. A task-specific smoke check in the plan is the answer to that, and it is a real gap in the method, not a footnote to it.

If you want it already wired

Ordewell generates one token per task, appends the instruction above, runs one real coding-agent session per task, and verdicts each one by evidence. That is the whole loop:

bash

ordewell plan --goal "Add rate limiting to the public API"
ordewell run

The planner reads your repo read-only, so planning needs no API key: a coding agent you already pay for can be the planner. Install with npm install -g ordewell, or read the docs first. The source for everything on this page is in the repo.