Guardrails, hallucination and accuracy control

You do not stop it, you bound it, and the difference is the whole design. Recent work argues models produce confident false statements because training and evaluation reward guessing over admitting uncertainty, so the lever available to you is architectural rather than a better model. Four things do most of the work. Constrain the output, so a schema or a required citation makes whole classes of invention impossible. Ground the answer in retrieved passages the reader can open, so a wrong answer is checkable rather than merely fluent. Give the system a supported way to say it does not know, and measure how often it uses it. And route by consequence, so the cases where being wrong is expensive reach a person. Accuracy improvements help at the margin; those four change what a wrong answer costs.

Runtime controls placed around a model to bound what it can be asked, what it can retrieve, what it can say and what it can do. They sit outside the model rather than inside its training, which is what makes them changeable without retraining and inspectable by someone who is not an engineer. In practice a serious stack has four layers: input, retrieval, output and action. The important thing to understand about them is that a guardrail toolkit is programmable, so it enforces whatever policy you write and nothing else. Buying one without writing the policy is buying a dependency.

No, and treating one as though it were is a common way to mislead a risk committee without meaning to. A system prompt is an instruction in the same channel as the input that may be trying to override it, which is why prompt injection is the first entry on the OWASP risk list for language model applications, and why system prompt leakage is an entry in its own right. A prompt is a useful way to shape behaviour and a poor way to prevent it. Anything that must never happen belongs in a schema, a permission, a filter outside the model, or a system that is simply not reachable.

Prompt injection is text that reaches the model as an instruction when it was meant to be read as data, either typed by the person using the system or planted in content the system later retrieves. It is the first entry on the OWASP risk list for language model applications, and it works for the reason above: instructions in a prompt are advisory and sit in the same channel as the input trying to override them, so a control a paragraph of text can talk its way past is not a control. The defence is architectural and it is layered. Prefer constraint to correction, because a model required to return output matching a schema cannot invent a field, a model answering only from retrieved passages with a citation can be checked by the reader, and a model with no tool for an action cannot take it. Handle retrieved passages as data rather than as instructions. Scope the tools and permissions the agent can reach so that a successful injection cannot do much, which is the control that still holds when the others fail. And carry adversarial documents in the evaluation set alongside the ordinary ones, because in an enterprise corpus the injected text does not have to come from an attacker: a well-meant instruction in a document template will do it.

Be careful of anyone who answers that with a number before seeing your data, because the answer depends on what the exception path costs. Ask instead for four commitments that are checkable. A named evaluation set built from your cases, with the definition of a correct answer agreed by the person who owns the decision. A threshold per case type rather than one headline figure, since the easy cases will otherwise carry the average. A stated abstention behaviour, so you know what the system does when it should not answer. And the trajectory measures, cost, latency and override rate, alongside accuracy. A supplier willing to be held to those is a better sign than one quoting 95%.

Yes, for the right job. The published work on model-graded evaluation found strong judges reaching over 80% agreement with human preference, which is roughly the agreement rate between humans, so it is a reasonable way to score at a volume no human panel could. It also identified position, verbosity and self-enhancement biases, meaning a judge can prefer the first answer it sees, the longer answer, and answers resembling its own. So use it for triage, regression detection and ranking, not as the gate. Keep a human-scored sample every cycle to detect drift in the judge, and be deliberate about whether the judge and the system share a model family.

The long tail and the disagreements, in that order, and neither is a model problem. Week one and two are the happy path, which is where a demo lives. What surfaces afterwards is the document that is a scan of a fax, the record with a field the specification never mentioned, the case where two experienced people give different correct answers, and the tool that returns success while doing nothing. On our insurance engagement the gap-and-contradiction pass over the requirement corpus surfaced ambiguities the business had not realised were ambiguous, and resolving them took a conversation rather than a rebuild, which is the cheap version of finding out. The limit of that answer: our agentic work is proofs of concept rather than production systems, so we can speak to weeks three to eight of a build, and month fourteen of a live agentic service is outside our own experience. The client engagement is confidential, so the specifics stay unpublished.