Guardrails and accuracy, answered in full.

You do not stop it, you bound it, and the difference is the whole design. Recent work argues models produce confident false statements because training and evaluation reward guessing over admitting uncertainty, so the lever available to you is architectural rather than a better model. Four things do most of the work. Constrain the output, so a schema or a required citation makes whole classes of invention impossible. Ground the answer in retrieved passages the reader can open, so a wrong answer is checkable rather than merely fluent. Give the system a supported way to say it does not know, and measure how often it uses it. And route by consequence, so the cases where being wrong is expensive reach a person. Accuracy improvements help at the margin; those four change what a wrong answer costs.

Runtime controls placed around a model to bound what it can be asked, what it can retrieve, what it can say and what it can do. They sit outside the model rather than inside its training, which is what makes them changeable without retraining and inspectable by someone who is not an engineer. In practice a serious stack has four layers: input, retrieval, output and action. The important thing to understand about them is that a guardrail toolkit is programmable, so it enforces whatever policy you write and nothing else. Buying one without writing the policy is buying a dependency.

No, and treating one as though it were is a common way to mislead a risk committee without meaning to. A system prompt is an instruction in the same channel as the input that may be trying to override it, which is why prompt injection is the first entry on the OWASP risk list for language model applications, and why system prompt leakage is an entry in its own right. A prompt is a useful way to shape behaviour and a poor way to prevent it. Anything that must never happen belongs in a schema, a permission, a filter outside the model, or a system that is simply not reachable.

Prompt injection is text that reaches the model as an instruction when it was meant to be read as data, typed by a user or planted in retrieved content. It heads the OWASP risk list for language model applications, and it works because a prompt is advisory, sharing a channel with the input trying to override it. The defence is architectural and layered. Prefer constraint to correction, since a schema stops an invented field, a citation lets the reader check a claim, and an absent tool cannot be called. Treat retrieved passages as data, not instructions, and scope what the agent can reach, the control that holds when others fail. Keep adversarial documents in the evaluation set. A well-meant instruction in a document template injects as surely as an attacker's.

Be wary of anyone quoting a number before seeing your data, because it depends on what the exception path costs. Ask instead for four checkable commitments. A named evaluation set built from your cases, with a correct answer defined by whoever owns the decision. A threshold per case type rather than one headline figure, since the easy cases otherwise carry the average. A stated abstention behaviour, so you know what it does when it should not answer. And cost, latency and override rate alongside accuracy. A supplier willing to be held to those is a better sign than one quoting 95%. Tenhaw's own commitment is measurable value every month, with a month that delivers none reported as a failed month.

Yes, for the right job. The published work on model-graded evaluation found strong judges reaching over 80% agreement with human preference, which is roughly the agreement rate between humans, so it is a reasonable way to score at a volume no human panel could. It also identified position, verbosity and self-enhancement biases, meaning a judge can prefer the first answer it sees, the longer answer, and answers resembling its own. So use it for triage, regression detection and ranking, not as the gate. Keep a human-scored sample every cycle to detect drift in the judge, and be deliberate about whether the judge and the system share a model family.

The long tail and the disagreements, in that order, and neither is a model problem. Weeks one and two are the happy path, which is where a demo lives. On a London insurance engagement, Tenhaw ran a gap-and-contradiction pass over the requirement corpus that surfaced ambiguities the business had not realised were ambiguous, resolved in conversation rather than a rebuild. What surfaces later is the document that is a scan of a fax, the record with a field the specification never mentioned, the case where two experienced people give different correct answers, and the tool that returns success while doing nothing. That build is a proof of concept, not a production system, so month fourteen of a live service is outside Tenhaw's experience.

Sometimes, at the margin, and it will not change the shape of the problem. A stronger model typically improves the average and leaves you with the same questions: what happens on the cases it still gets wrong, how you would know, and who is accountable when it does. It also resets your evaluation, because behaviour changes in both directions on an upgrade and prompts tuned to the old model frequently perform worse on the new one. The organisations that get value from upgrades are the ones with an evaluation set to run them against, which is an argument for building that first rather than an argument against upgrading.

Yes, and it should be designed to. If the only supported output is an answer, the system will produce an answer, which is how a confident guess reaches a user. Make abstention a first-class result with a route attached: a person, a fallback, or a plainly worded decline. Then measure it and split the number in two. Correct refusals, where the system genuinely could not answer safely, and wasteful refusals, where it could have. A system that never abstains is not safe, it is unmeasured, and the ratio between those two figures is one of the more informative things you can put in front of an operations team.

Red teaming is adversarial testing. People deliberately try to make the system say or do something it should not, so the failures are found before a user or an attacker finds them. The published work treats it as a scaled, continuing activity rather than a pre-launch workshop. One study released a dataset of 38,961 red team attacks, which is a result you only obtain by testing at volume and over time. So run it continuously. Keep a standing adversarial set that grows with every incident and near miss, refresh it on every model upgrade, and have it owned by someone who does not benefit from the release.

It is largely a false choice, because a rail product enforces the policy you write and nothing else. Published toolkits are programmable by design. They run at runtime, sit outside the model and stay inspectable, which is the useful property and also the catch. Somebody still has to write the rules, in your language, about your business. Tenhaw published its own, 72 engineering rules on GitHub, graded by RFC 2119 severities and enforced by an agent. Buy the runtime if it saves engineers work, and build three things yourself regardless: the accuracy contract, the evaluation set that shows the rails hold, and the standing adversarial set. The strongest controls are not rail products at all. A schema, a permission and an absent tool do more.

Three lists, agreed with the business owner and the second line before anything is built. What must never happen, which becomes a hard constraint enforced outside the model wherever possible, often in the data layer or a tool permission rather than by a model at all. What must be escalated to a person, which becomes the routing design and the resourcing question. What is merely undesirable, which becomes a metric rather than a gate. Most teams find while writing it that the second list is where all the money is. That is roughly a week of work with your risk function, it does not need a supplier, and it makes every later tooling decision obvious.

Not on its own, and not as the sole basis for deciding what a person reviews. A self-reported certainty tells you how sure the model is, which is not the same as how likely it is to be right. On an engagement in the London insurance market, Tenhaw scored confidence from three inputs instead: the provenance of each extracted value, the model's own certainty, and an independent search-based cross-check. Provenance is what makes the routing explainable to the person who owns the outcome, and explainable routing is what lets a business accept an accuracy figure it would otherwise reject. That was batch document work with internal users rather than a customer-facing channel.

Yes, and it is a normal place to start. Rails sit outside the model rather than inside its training, so they can be changed without retraining and read by someone who is not an engineer, which is what makes a retrofit tractable. Tenhaw has built these accuracy controls into delivered proofs of concept rather than against live customer traffic. Begin with the accuracy contract rather than the code, because the usual finding in an existing stack is a toolkit with no written policy behind it, which is a dependency rather than a control. Then design the worst failures out with a schema or a scoped permission, place rails at input, retrieval, output and action, and build the evaluation set that shows the change held.

Give them things they can inspect rather than a paragraph asserting testing was thorough. Tenhaw hands the rails and the evaluation set to your own reviewers to test rather than reporting on its own work. Three artefacts carry most of it. The written accuracy contract, so what must never happen is stated rather than implied. The rails themselves, which run outside the model and stay readable, so a risk function can form its own view without taking an engineer's word for it. And a standing adversarial set that grows with every incident and near miss, owned by someone who does not benefit from the release, with the results of its last run.

Use benchmarks to shortlist, never to settle it. A 2025 paper argues that models produce confident false statements because training and evaluation reward a guess over an admission of uncertainty, and that part of the fix is changing how existing benchmarks score uncertainty rather than adding more hallucination tests. As a procurement point, a leaderboard that scores a guess and an admission of not knowing identically is rewarding the exact behaviour you are trying to design out, so a high score can hide the failure mode you care about. What settles the choice is your own evaluation set, built from your cases, with a threshold per case type rather than one headline figure.

No, and a review step applied to everything tends to become a formality rather than a control. Route by consequence instead, sending the cases that are expensive to get wrong to a person and letting the rest through with a check the reader can perform, such as a citation they can open. On a proof of concept for a London specialty insurance business, Tenhaw routed records to a reviewer on a confidence score rather than sending every extraction to a person. The accuracy contract decides the split rather than the technology. Where volumes are low and stakes are high, reviewing everything is a reasonable starting point, provided you know what it costs.

Week one is the accuracy contract rather than code: what must never happen, what gets escalated to a person, what you will merely measure. Then take one workflow and design the failures out before filtering them out, so a schema, a scoped permission or a tool that simply does not exist removes whole classes of error. Then put the four rails around it: input, retrieval, output and action. In parallel, build an evaluation set from your own cases with a correct answer defined by whoever owns the decision, and start the adversarial set that grows with every near miss. Tenhaw staffs a first month like that with two or three senior people led by James Rooney, and agrees the exit date at kickoff.