Guardrails, hallucination and accuracy control

You do not stop it, you bound it, and the difference is the whole design. Recent work argues models produce confident false statements because training and evaluation reward guessing over admitting uncertainty, so the lever available to you is architectural rather than a better model. Four things do most of the work. Constrain the output, so a schema or a required citation makes whole classes of invention impossible. Ground the answer in retrieved passages the reader can open, so a wrong answer is checkable rather than merely fluent. Give the system a supported way to say it does not know, and measure how often it uses it. And route by consequence, so the cases where being wrong is expensive reach a person. Accuracy improvements help at the margin; those four change what a wrong answer costs.

Runtime controls placed around a model to bound what it can be asked, what it can retrieve, what it can say and what it can do. They sit outside the model rather than inside its training, which is what makes them changeable without retraining and inspectable by someone who is not an engineer. In practice a serious stack has four layers: input, retrieval, output and action. The important thing to understand about them is that a guardrail toolkit is programmable, so it enforces whatever policy you write and nothing else. Buying one without writing the policy is buying a dependency.

No, and treating one as though it were is a common way to mislead a risk committee without meaning to. A system prompt is an instruction in the same channel as the input that may be trying to override it, which is why prompt injection is the first entry on the OWASP risk list for language model applications, and why system prompt leakage is an entry in its own right. A prompt is a useful way to shape behaviour and a poor way to prevent it. Anything that must never happen belongs in a schema, a permission, a filter outside the model, or a system that is simply not reachable.

Be careful of anyone who answers that with a number before seeing your data, because the answer depends on what the exception path costs. Ask instead for four commitments that are checkable. A named evaluation set built from your cases, with the definition of a correct answer agreed by the person who owns the decision. A threshold per case type rather than one headline figure, since the easy cases will otherwise carry the average. A stated abstention behaviour, so you know what the system does when it should not answer. And the trajectory measures, cost, latency and override rate, alongside accuracy. A supplier willing to be held to those is a better sign than one quoting 95%.

Yes, for the right job. The published work on model-graded evaluation found strong judges reaching over 80% agreement with human preference, which is roughly the agreement rate between humans, so it is a reasonable way to score at a volume no human panel could. It also identified position, verbosity and self-enhancement biases, meaning a judge can prefer the first answer it sees, the longer answer, and answers resembling its own. So use it for triage, regression detection and ranking, not as the gate. Keep a human-scored sample every cycle to detect drift in the judge, and be deliberate about whether the judge and the system share a model family.

The long tail and the disagreements, in that order, and neither is a model problem. Week one and two are the happy path, which is where a demo lives. What surfaces afterwards is the document that is a scan of a fax, the record with a field the specification never mentioned, the case where two experienced people give different correct answers, and the tool that returns success while doing nothing. On our insurance engagement the gap-and-contradiction pass over the requirement corpus surfaced ambiguities the business had not realised were ambiguous, and resolving them took a conversation rather than a rebuild, which is the cheap version of finding out. The limit of that answer: our agentic work is proofs of concept rather than production systems, so we can speak to weeks three to eight of a build, and month fourteen of a live agentic service is outside our own experience. The client engagement is confidential, so specifics beyond this are a conversation under NDA rather than a web page.

Sometimes, at the margin, and it will not change the shape of the problem. A stronger model typically improves the average and leaves you with the same questions: what happens on the cases it still gets wrong, how you would know, and who is accountable when it does. It also resets your evaluation, because behaviour changes in both directions on an upgrade and prompts tuned to the old model frequently perform worse on the new one. The organisations that get value from upgrades are the ones with an evaluation set to run them against, which is an argument for building that first rather than an argument against upgrading.