Agent evaluation and assurance

Most commonly because there was never an evaluation, only a demonstration. Without a ground-truth set, agreed acceptance criteria and a release gate someone outside the build team can hold, the production decision becomes a negotiation with no evidence to settle it, and negotiations without evidence default to no. AI pilots that never reach production tend to share three further properties: only the final answer was scored rather than the agent's trajectory, so nobody knows how it behaves on the cases it has not seen; nothing was designed to monitor it after go-live, so operations cannot say what they would be accepting; and no named person was ever asked to own the live system, its run cost and its incidents. All four are addressable before a line of code is written, and all four are expensive to fix once a pilot has already been declared a success.

Separate the two questions that are usually tangled together: is it good enough, and who is receiving it. For the first, write the acceptance criteria the business owner would actually sign, then measure the existing system against them honestly. That is a two to three week piece of work and it either produces a gate you can pass or a specific, costed list of what is missing, which is far better than the ambient sense that the pilot was fine. For the second, name the person who will own the live system, its run budget and its incidents, and get them to agree the criteria before you re-test. Pilot purgatory is nearly always the second problem wearing the costume of the first: teams keep improving a system that nobody has been asked to take.

Scoring the sequence of steps an agent takes (which tools it chose, what it retrieved, how many steps it used, what it cost, and when it decided to stop) rather than only the final output. It matters because an agent can produce a correct answer through a process that is unsafe, unrepeatable or prohibitively expensive, and output-only scoring makes that invisible until production.

Yes, and it is one of the more useful things an external partner is for. An internal team recommending a delay is arguing against a date their own management committed to, with their next promotion in the room. Tenhaw has recommended against release at least once on every engagement it has run, usually where a launch date was being defended rather than a readiness assessment being made. If a supplier has never told you not to ship, that is information about the supplier rather than about your programmes.

Confidence derived from provenance rather than from the model's own certainty score. Where the data came from (which enrichment source, corroborated by an independent search) gives you a defensible basis for routing even before a labelled set exists. It is not a substitute for ground truth and we would still push to build one, but it means an organisation with no appetite for a labelling exercise is not stuck with nothing.

Someone outside the team that built the agent, with the standing to block a release. Where the build team owns the benchmark, the benchmark tends to describe what the system already does. This is an accountability question rather than a tooling one, and it is settled during operating-model design by mapping who decides, on what evidence, and who can overrule them.

The same criteria that gated the release, so degradation is comparable rather than anecdotal, plus cost and latency distributions and the rate at which humans override the agent. Override rate is often the most useful early signal, because it moves before accuracy metrics do and it is measured on real decisions.

Four parts, and they are more ordinary than the phrase suggests. A dataset: real cases with the answer a qualified person would accept, held under version control and grown every time something goes wrong in production. A runner that executes the system against every case reproducibly, pinning the model version, the prompts, the retrieval corpus and the tool permissions, because a result you cannot reproduce is an anecdote. Scorers, which are a mixture of exact checks, rule-based assertions and, for open-ended output, a model acting as judge. And a report a non-engineer can read, showing the score against the threshold, what regressed since the last run, and what each run cost and how long it took. The UK AI Security Institute's open-source Inspect framework is built around exactly that shape, datasets, solvers and scorers, which is a reasonable sanity check on any design somebody presents to you. The part people underestimate is the dataset, because it is the only part that cannot be bought.