# Tenhaw guides and operating methods Tenhaw is a UK AI consultancy and AI delivery partner based in London. We build AI products and workflows, design operating models and develop your team's capability. We also lead programmes across your teams and suppliers. Index: https://tenhaw.com/llms.txt Full corpus: https://tenhaw.com/llms-full.txt Illustrative resources are teaching examples, separate from delivered client evidence. Cite the native page below. ## Document intelligence to business intelligence Source: https://tenhaw.com/guides/document-intelligence-to-business-intelligence Last updated: 2026-11-05 { "slug": "document-intelligence-to-business-intelligence", "metaDescription": "Turn documents into structured, source-linked data and usable business intelligence. A practical guide grounded in Tenhaw's two-week PPL proof of concept.", "name": "Document intelligence to business intelligence", "tagline": "Turn source documents into data your business can check and use.", "summary": "Intelligent document processing turns PDFs, scans, email and forms into structured data with a trace back to the source. A usable business-intelligence workflow also needs agreed definitions, exception review and a semantic layer. At Placing Platform Limited, the London market digital placing platform, Tenhaw built a working Azure document-to-BI proof of concept in two weeks during a six-month engagement.", "theme": "patterns", "evidenceBasis": "delivered", "evidenceStatement": "We have delivered this pattern end to end, on Azure, on a client engagement, as a working proof of concept rather than a production system. What follows is written from that work rather than from vendor documentation. The provider notes below set out where the clouds genuinely differ.", "demandSignal": "Document work is usually where an organisation meets agentic AI first, because the source material already exists, nothing is customer-facing, and the value is easy to describe to a board. It is also the pattern where the published engineering literature is bluntest about the gap between a demo and a system: a study of three production retrieval implementations concluded that validating one is only feasible during operation, and that the robustness of such a system evolves rather than being designed in at the start. That is the four failures below, arrived at from the other direction.", "whyItStalls": [ { "heading": "The demo extracted beautifully, and it became a proof of concept that never shipped", "body": "Extraction accuracy on a curated sample tells you almost nothing about accuracy on the long tail: the scanned fax, the amended schedule, the document where the important number is in a footnote. This is the commonest shape of a proof of concept that never shipped. The programme discovers that 95% accuracy is unusable for a process that requires a defensible number, nobody designed the exception path, and the work stops in the gap between impressive and usable." }, { "heading": "Nobody agreed what a correct answer is, so accuracy became an argument", "body": "Two experienced underwriters will disagree about what a document says. If you have not established ground truth with the people who own the decision, you cannot measure the system, and every accuracy conversation becomes an argument about the benchmark rather than the model. Pilots stuck in that argument do not fail a test, they simply never get one, which is why they can sit unresolved for quarters." }, { "heading": "The extraction works, the BI layer is not ready, and the programme stalls there", "body": "This is the most common failure and the least discussed. You now have structured data with no semantic layer, no agreed definitions, and no governance, so the business gets confident answers drawn from the wrong table. The bottleneck moves from the model to the data foundations, and the programme is not resourced for it, so a technically successful pilot sits waiting on a data programme nobody has funded." }, { "heading": "Expert review became the bottleneck, because it was built as a queue", "body": "Every serious implementation keeps humans in the loop. Most implement it as a review queue that is worked in order, which means scarce expert time is spent uniformly across easy and hard cases. Routing by confidence and consequence is what makes the economics work, and it is usually retrofitted after the queue becomes the bottleneck. Until it is, the business case shows a saving that the operation cannot feel." } ], "approach": [ { "heading": "Extract to markdown first, then narrow to the fields that matter", "body": "The pipeline we ran starts by using a language model to turn each source document into a markdown representation of itself, and only then narrows to the key fields. Extracting to an intermediate readable form first means the extraction is inspectable by a human and re-runnable when the field list changes, rather than being a black box from PDF to database column.", "links": [ { "label": "The pipeline stage by stage, from the insurance proof of concept", "href": "/case-studies/specialty-insurance-agentic-lead#pipeline" } ] }, { "heading": "Normalise, then enrich, then add semantics", "body": "Extracted fields are normalised into consistent formats, enriched against third-party APIs, and then given semantic enhancement (context and thematic grouping) so downstream logic is working with meaning rather than strings. Each stage is separable, which matters because the enrichment and semantic stages are where accuracy problems are usually diagnosable." }, { "heading": "Requirements as a machine-readable corpus", "body": "The document types, the fields, the validation rules and the edge cases go into structured markdown before any build, and we run the gap-and-contradiction pass over them. On the insurance engagement this surfaced ambiguities the business had not realised were ambiguous, and resolving them took a conversation rather than a rebuild." }, { "heading": "Build against the whole requirement set, paired with your engineers", "body": "The build follows our published AI-engineering-first method: a blank repository, high-level prompts against the full corpus, a security review roughly every fifth prompt, and pair-programming throughout. On the insurance engagement the client's own engineer finished a two-week build 70% confident they could run the process unaided." }, { "heading": "Score confidence by source, not just by model certainty", "body": "On the same engagement we scored confidence using the provenance of the data, which third-party enrichment source it came from, alongside model certainty and a search-based cross-check. A model's own confidence is a weak signal on its own; combining it with where the data came from is what makes routing decisions defensible to the people who own the outcome." }, { "heading": "Land it in business logic and a dashboard, not a database", "body": "The final stages apply business logic to interpret the enriched, semantically grouped data and surface it in an internal dashboard. This is the step that distinguishes document intelligence from document extraction, and it is the one most often descoped when a programme runs late, which is how organisations end up with a populated table nobody uses." }, { "heading": "Treat the semantic layer as in scope, not someone else's problem", "body": "Structured output is worthless if the BI layer answers confidently from the wrong table. Agreed definitions, a governed semantic layer and lineage back to the source document are part of the work, because without them the programme delivers extraction and not intelligence.", "links": [ { "label": "Taking the whole process agentic rather than the step", "href": "/guides/end-to-end-agentic-workflow-implementation" } ] } ], "providerNotes": { "heading": "Where the cloud actually matters, and where it does not", "body": "The hard parts of this pattern (ground truth, exception design, human routing, semantic layer, lineage) are provider-independent, and they are where programmes fail. The provider-specific decisions are the document-understanding service and its handling of layout and tables, how identity and permissions propagate from the source repository through to the warehouse, and what the residency position is for the documents themselves. We have delivered this on Azure. On AWS and Google Cloud the architecture shape is the same and the service choices differ; an engagement there applies the method rather than repeats a delivery, and we say so at the point of engagement." }, "engageVia": { "slug": "agentic-proof-of-concept", "label": "Agentic Proof of Concept" }, "faq": [ { "question": "How long does a document intelligence proof of concept take?", "answer": "Two to four weeks for a working proof of concept against one document type and one real workflow. On an engagement with Placing Platform Limited (PPL), Tenhaw delivered a working proof of concept extracting information from PDFs into business intelligence on Azure in two weeks, ground the business had been circling for roughly a year. Production is a separate phase with its own scope and a dedicated team." }, { "question": "Why do document extraction projects stall after the demo?", "answer": "Usually for one of four reasons: accuracy on a curated sample does not survive the long tail and no exception path was designed; nobody established what a correct answer is, so every accuracy discussion becomes an argument about the benchmark; the extraction works but the BI layer has no semantic layer or governance, so the business gets confident answers from the wrong table; or human review was built as a queue rather than routed by confidence and consequence, so expert time becomes the bottleneck." }, { "question": "Our document AI proof of concept worked and never shipped. What now?", "answer": "Start by working out which of the three gaps you are in, because they need different money and different people. Tenhaw does that diagnosis before scoping a build. If accuracy collapsed on the long tail, the missing piece is exception routing rather than a better model. If nobody can agree what a correct answer is, the next work is a ground-truth set built with the people who own the decision, and it is a fortnight rather than a phase. If the extraction is fine and the numbers are not trusted, you are waiting on a semantic layer and lineage, a data programme with a different sponsor and a different budget line. A proof of concept that never shipped is rarely blocked on the model." }, { "question": "What accuracy is good enough for document intelligence?", "answer": "There is no universal number, and quoting one is a warning sign. The right question is what the exception path costs. A process that tolerates review can run at accuracy that would be unacceptable for straight-through processing. Design the routing first, by extraction confidence and business consequence, and the accuracy target falls out of it rather than being asserted up front." }, { "question": "Do we need to fix our data platform before doing this?", "answer": "Not before a proof of concept, and yes before production. Whether the extraction is viable at all is the cheaper question to answer first, so run it against the estate as it stands. Tenhaw did exactly that for a London insurance market business, taking PDFs to business intelligence on Azure in a two-week proof of concept, ahead of any data platform work. But structured output with no semantic layer, agreed definitions or lineage produces confident answers from the wrong table, so data foundations belong in the productionisation scope rather than being discovered during it. That scope usually sits with a different sponsor and a different budget line, which is where programmes with working extraction and untrusted numbers get stuck." }, { "question": "What is intelligent document processing?", "answer": "Intelligent document processing, or IDP, also called document intelligence, turns unstructured source material such as PDFs, scans, email and forms into structured data that lands in a warehouse and drives business intelligence. It is where most enterprises meet agentic AI first, because the source material already exists, nothing is customer-facing, and the value is easy to describe to a board. It is also the pattern most likely to stall between a convincing extraction demo and a number the business will actually act on, which is why exception handling, ground truth and the BI layer matter as much as the model." }, { "question": "How do you establish ground truth for document extraction?", "answer": "Build a ground-truth set with the people who own the decision, before anyone argues about models. Two experienced underwriters will disagree about what a document says, so if the decision owners have not agreed what a correct answer is, every accuracy conversation becomes an argument about the benchmark rather than the system. Done properly it is a fortnight of work rather than a phase, built from real documents, including the long tail of scanned faxes, amended schedules and important numbers buried in footnotes, with the experts' disagreements resolved into agreed answers. Skip it and the pilot never fails a test, it simply never gets one, which is how document programmes sit unresolved for quarters." }, { "question": "What does a good document extraction pipeline look like?", "answer": "Separable stages, in a deliberate order. First use a language model to turn each document into a markdown representation of itself, so the extraction is inspectable by a human and re-runnable when the field list changes, then narrow to the fields that matter. Normalise those fields into consistent formats, enrich them against third-party APIs, then add semantic enhancement, context and thematic grouping, so downstream logic works with meaning rather than strings. Finally apply business logic and surface the result in a dashboard, not just a populated table. Keeping the stages separable matters because enrichment and semantics are where accuracy problems are usually diagnosable." }, { "question": "How do you stop human review becoming the bottleneck in document AI?", "answer": "Route work by confidence and consequence instead of building a first-in, first-out queue. Every serious implementation keeps humans in the loop, but most implement it as a queue worked in order, which spends scarce expert time uniformly across easy and hard cases until review becomes the bottleneck and the business case shows a saving the operation cannot feel. Score confidence using the provenance of the data, which enrichment source it came from, alongside model certainty and a search-based cross-check, because a model's own confidence is a weak signal on its own. Then send experts only the cases where low confidence meets real consequence." }, { "question": "Does it matter which cloud we use for document intelligence?", "answer": "Less than the vendors suggest. Tenhaw has delivered this pattern end to end on Azure, using Claude, Codex and Azure OpenAI. A PPL insurance proof of concept took PDFs to business intelligence in two weeks, ground the business had circled for roughly a year, built inside the client's own Azure estate under its identity and access controls. The parts where programmes actually fail, ground truth, exception design, human routing, the semantic layer and lineage, are provider-independent and transfer intact. What genuinely differs is the document-understanding service and how it handles layout and tables, how permissions propagate from the source repository through to the warehouse, and the residency position. On AWS or Google Cloud those service choices are scoped with your own platform engineers and priced into the engagement." }, { "question": "We already have an OCR tool. What would document intelligence add?", "answer": "Everything that happens after the text comes off the page. Optical character recognition and template extraction solve the easy half, which is getting the characters out. Tenhaw builds the other half. Document intelligence carries those fields on: normalised into consistent formats, enriched against third-party sources, grouped semantically so downstream logic works with meaning rather than strings, then interpreted by business logic and surfaced somewhere people actually decide with. That last step is what separates document intelligence from document extraction, and it is the one most often descoped when a programme runs late, which is how organisations end up with a populated table nobody uses. If your current tool reads a stable template accurately and somebody already acts on the output, keep it." }, { "question": "How much does an intelligent document processing project cost?", "answer": "In two pieces, and it is the second one budgets tend to miss. Tenhaw's Agentic Proof of Concept against one document type and one real workflow is a fixed £20,000 to £55,000 over two to four weeks, set by the complexity of the workflow and the state of the data underneath it. Productionising it is a separate phase, four to six weeks with a dedicated team, and an Agentic Build Team is £70,000 to £85,000 a month. Then there are the data foundations. A governed semantic layer, agreed definitions and lineage back to the source document usually sit with a different sponsor and a different budget line, and programmes that meet that late stall with the extraction working and the numbers untrusted." }, { "question": "Which document type should we start with?", "answer": "One document type, one real workflow, and a workflow whose output somebody actually decides with. Document work is usually where an organisation meets agentic AI first because the source material already exists, nothing is customer-facing, and the value is easy to describe to a board, so the temptation is to pick the tidiest document type in the building. Resist it. Accuracy on a curated sample tells you almost nothing about the scanned fax, the amended schedule, or the important number sitting in a footnote. Choose something with real exceptions and a named destination for the data, or the proof of concept proves the wrong thing." }, { "question": "Who from the business needs to be involved in a document AI project?", "answer": "Four groups, and missing any one of them is the usual cause of a stall. The people who own the decision, because ground truth has to be agreed with them and two experienced experts will disagree about what a document says. Your own engineers, because Tenhaw pairs with them throughout the build and the point is that your team can run it afterwards. Whoever owns the warehouse and the semantic layer, since agreed definitions and lineage are usually a different sponsor with a different budget line. And the operations people whose review time you are about to route by confidence and consequence instead of a first-come queue." }, { "question": "Will it handle tables and awkward layouts in our PDFs?", "answer": "Layout and table handling is one of the genuinely provider-specific decisions in this pattern, so test it on your own documents rather than taking a vendor demo on trust. The pipeline shape helps, and Tenhaw ran this one on Azure for a London insurance market client. Each source document is first turned into a markdown representation of itself, which keeps what the model read inspectable by a human, and only then is it narrowed to the fields that matter. When a layout defeats it, you want that arriving as low confidence and routing to a person rather than as a confident wrong value in a dashboard. Design that exception path before anyone argues about which document-understanding service wins." }, { "question": "What happens when a document format changes after go-live?", "answer": "Less than you would fear, if the pipeline was built in separable stages. Each document is first turned into a markdown representation of itself and only then narrowed to the fields that matter, so extraction stays inspectable by a human and re-runnable when the field list changes rather than being a black box from PDF to database column. A shape the pipeline has not seen before tends to arrive as low confidence, which is exactly what routing by confidence and consequence exists for. Validating a system like this is only feasible during operation, the published engineering literature says, and its robustness evolves rather than being designed in at the start." }, { "question": "Do our documents leave our own cloud during processing?", "answer": "Not if the pipeline is deployed the way Tenhaw deploys it, on your infrastructure and under your policies. The delivered version ran end to end on Azure inside a London insurance market client's own regulated estate, with UK data residency the default and the EU available where that is the requirement. The residency question worth settling early is narrower than most people expect, coming down to which region the document-understanding service processes a file in and how identity and permissions propagate from the source repository through to the warehouse. Both are genuinely provider-specific, and both are far cheaper to answer before any documents move." }, { "question": "How do we trace a dashboard number back to the source document?", "answer": "Design for it, because a sceptical reviewer asks for it first and it is not free. Three mechanisms do the work. Extracting each document into a readable markdown form before narrowing to fields keeps the extraction inspectable by a human rather than a black box. Confidence is scored using the provenance of the data, meaning which third-party enrichment source a value came from, alongside model certainty and a search-based cross-check, so you can say why a number was trusted. And lineage back to the source document belongs in the build alongside the semantic layer, because without it the programme has delivered extraction and not intelligence." } ], "citations": [ { "name": "Barnett et al., Seven Failure Points When Engineering a Retrieval Augmented Generation System (2024)", "url": "https://arxiv.org/abs/2401.05856", "note": "Three case studies across research, education and biomedical domains. The finding quoted above, that validation is only feasible during operation and robustness evolves rather than being designed in, is from the abstract." }, { "name": "Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)", "url": "https://arxiv.org/abs/2307.03172", "note": "Why the position of an extracted passage in the prompt changes the answer, and why a long context window is not a substitute for narrowing to the fields that matter." } ] } ## Agent evaluation and assurance Source: https://tenhaw.com/guides/agent-evaluation-and-assurance Last updated: 2026-11-05 { "slug": "agent-evaluation-and-assurance", "name": "Agent evaluation and assurance", "metaDescription": "Design AI agent evaluation around real cases, trajectory checks and a clear release decision. Practical records, source research and operating ownership.", "tagline": "Give your release decision evidence the business can inspect.", "summary": "Agent evaluation tests whether a system meets agreed cases and operating limits. Build a versioned case set, score the actions as well as the final answer, and give an accountable person the evidence to accept or block a release. Monitoring then helps the live owner notice when behaviour changes.", "theme": "control", "evidenceBasis": "approach", "evidenceStatement": "We have built evaluation and confidence scoring into the proofs of concept we have delivered, including validation and scoring of entity-resolution output on a specialty insurance engagement, and we have recommended against release at least once on every engagement we have run. What follows is the method we would bring to an enterprise-wide agent assurance function across a portfolio of systems.", "evidenceNextStep": { "heading": "So what should you do about it", "body": "Keep the assurance function yourselves. Your second line and internal audit should own the gate, because the independence is the whole point of it and it is not something to buy from the firm that built the system. Use us for the build and for designing the evidence, so the ground-truth set, the trajectory scoring, the release criteria and the monitoring arrive as artefacts your own reviewers can hold us to rather than as a report about our work. And if what you actually need is the assurance function itself stood up rather than reasoned about, say so on the call: that is a real piece of work, and we will tell you whether we are the right firm for it." }, "demandSignal": "Evaluation has stopped being a research topic and become infrastructure, which tells you where the expectation is heading. The UK AI Security Institute publishes an open-source evaluation framework, Inspect, built around three parts, datasets, solvers and scorers, and ships more than two hundred evaluations inside it. The published work on using a strong model as the judge reports over 80% agreement with human preference, about the level at which humans agree with each other, while naming the position, verbosity and self-enhancement biases you have to design around. The methods exist and are documented. What almost no organisation has is the same rigour pointed at its own workflows, and that gap is where the production decision stalls.", "whyItStalls": [ { "heading": "Pilot purgatory starts with a demonstration instead of a measurement", "body": "Someone senior watched it work and was impressed. That is not an evaluation, and it does not transfer. When the same system is asked to pass a production gate, there is no benchmark, no baseline and no agreed definition of good, so the gate becomes a negotiation rather than a test. Programmes stuck in pilot are almost always stuck precisely here: nobody doubts the system, and nobody can produce the evidence that would let a named person carry the risk of switching it on." }, { "heading": "Only the final answer is scored", "body": "Agents take a sequence of steps: choosing tools, retrieving, deciding when to stop. A system can produce a correct answer through a process that is unsafe, expensive or unrepeatable. Scoring only the output hides that, and it is why systems that tested well fail differently in production." }, { "heading": "Evaluation is owned by whoever built it", "body": "Self-marked homework. Where the build team owns the benchmark, the benchmark tends to describe what the system already does. Assurance needs a separate owner with the standing to block a release, which is an organisational question rather than a tooling one." }, { "heading": "The pilot succeeded and then went nowhere, because nobody owned what came next", "body": "A successful pilot creates an orphan. The sponsor who funded an experiment is rarely the person who will carry a live system, its run cost, its incidents and its audit trail, and in most organisations nobody has been asked to. So a pilot succeeded and then went nowhere is usually not a technical verdict at all: it passed, and there was no named owner, no run budget and no slot in a delivery roadmap to receive it. We treat naming that receiver as part of the pilot rather than as a conversation for afterwards, because the answer changes what is worth building.", "links": [ { "label": "Designing the target operating model that makes the fifty-first rollout cheap", "href": "/guides/target-operating-model-for-an-ai-native-organisation" } ] }, { "heading": "Nothing watches it after go-live", "body": "Model behaviour drifts, retrieval corpora change, and the input distribution moves. Without online monitoring against the same criteria used offline, degradation is discovered by users. Most programmes budget for pre-production evaluation and nothing for the two years afterwards." } ], "approach": [ { "heading": "Define what good means before anything is built", "body": "The same discipline as pricing an outcome in currency: if you cannot state the acceptance criteria, you cannot manage the work. We establish the ground-truth set and the pass thresholds with the business owner during design, and they become the release gate rather than a retrospective justification." }, { "heading": "Measure confidence alongside accuracy, and derive it from provenance", "body": "Accuracy and confidence are the two measures we work to, and they answer different questions. On the insurance engagement, confidence was derived from which third-party source the data came from, combined with model certainty and a search-based cross-check, not from the model's self-reported score alone. Provenance-derived confidence is what makes routing defensible, because you can explain to the business owner why a given record was escalated." }, { "heading": "Score the trajectory, not just the answer", "body": "Tool selection, retrieval quality, step count, cost and stopping behaviour are all measured, because a right answer reached the wrong way is a production incident waiting to happen. This mirrors how we treat delivery: flow and process are measured, not just the output at the end." }, { "heading": "Put the gate in someone else's hands, and be willing to be that person", "body": "Assurance sits with a named owner outside the build team, with the authority to hold a release. It is the same accountability mapping as any other consequential decision: who decides, on what evidence, and who can overrule them. Part of what an external partner is for is being the person who can say no without worrying about their next promotion. We have recommended against release at least once on every engagement we have run, usually when a date was being defended rather than a readiness assessment." }, { "heading": "Design the online case to match the offline one", "body": "The criteria that gate the release are the criteria monitored in production, so degradation is comparable rather than anecdotal. Alerting thresholds and the rollback decision are agreed before go-live, not during the first incident." }, { "heading": "Make the evidence the artefact, and name the regime each artefact answers", "body": "Evaluation output is written for the audiences that will ask for it (risk, audit, and the regulator) rather than reconstructed under pressure later. If the evidence is a by-product of the process it is cheap, and if it is assembled retrospectively it is expensive and thin. For a UK insurer it is worth being specific about which regime each artefact is for, because that decides whether the same work gets done once or four times. The model and decision inventory, holding prompts, retrieval corpora, tool permissions and model versions as versioned artefacts with named owners, is what a model risk management framework asks for, and SS1/23 is the UK statement of that expectation for banks; insurers sit outside its formal scope and get asked about it anyway. Outcome measures emitted by the system as it runs, rather than reconstructed from logs a quarter later, are the Consumer Duty evidence for any workflow that touches a customer outcome, because that regime is written around acting to deliver good outcomes for retail customers and monitoring whether you did, rather than around any particular mechanism. Degradation monitoring alongside availability monitoring, and a fallback that is exercised rather than documented, are what an impact tolerance under the operational resilience regime actually rests on. And field-level provenance back to the source document or API is what the data quality expectations under Solvency II and Solvency UK come down to in practice for the actuarial function. One evaluation design can answer all four, but only if it is specified that way at the start.", "links": [ { "label": "What each of those regimes requires of an agent", "href": "/sectors/financial-services" }, { "label": "Generating the evidence as a by-product", "href": "/guides/ai-governance-and-regulatory-evidence" } ] } ], "engageVia": { "slug": "agentic-design-team", "label": "Agentic Design Team" }, "faq": [ { "question": "Why do AI pilots never reach production?", "answer": "Most commonly because there was never an evaluation, only a demonstration. Without a ground-truth set, agreed acceptance criteria and a release gate someone outside the build team can hold, the production decision becomes a negotiation with no evidence to settle it, and negotiations without evidence default to no. Three failures usually sit underneath: only the final answer was scored rather than the agent's trajectory, nothing was designed to monitor it after go-live, and no named person was asked to own the live system, its run cost and its incidents. Tenhaw pairs its builds with the receiving team for that reason. On a PPL insurance proof of concept the client's engineer paired throughout and reported 70% confidence that they could run the process unaided." }, { "question": "What do we do when a pilot succeeded and then went nowhere?", "answer": "Separate the two questions usually tangled together, whether it is good enough and who is receiving it. For the first, write the acceptance criteria the business owner would sign, then measure the existing system against them honestly, which is a two to three week piece of work that produces either a gate you can pass or a costed list of what is missing. For the second, name the person who will own the live system, its run budget and its incidents, and get them to agree the criteria before you re-test. Tenhaw names that receiver during the pilot rather than afterwards. Pilot purgatory is usually the second problem wearing the costume of the first, because teams keep improving a system nobody has been asked to take." }, { "question": "What is trajectory evaluation for AI agents?", "answer": "Scoring the sequence of steps an agent takes (which tools it chose, what it retrieved, how many steps it used, what it cost, and when it decided to stop) rather than only the final output. It matters because an agent can produce a correct answer through a process that is unsafe, unrepeatable or prohibitively expensive, and output-only scoring makes that invisible until production." }, { "question": "Should a consultancy be willing to recommend against its own release?", "answer": "Yes, and it is one of the more useful things an external partner is for. An internal team recommending a delay is arguing against a date their own management committed to, with their next promotion in the room. Tenhaw has recommended against release at least once on every engagement it has run, usually where a launch date was being defended rather than a readiness assessment being made. If a supplier has never told you not to ship, that is information about the supplier rather than about your programmes." }, { "question": "What do you measure when there is no ground truth?", "answer": "Confidence derived from provenance rather than from the model's own certainty score. Tenhaw did exactly this on the PPL engagement, scoring entity-resolution output by which third-party source the data came from, combined with model certainty and a search-based cross-check. Where the data came from gives you a defensible basis for routing before any labelled set exists, and it is explainable to the business owner, which a self-reported score is not. It is not a substitute for ground truth, and building one is still worth pushing for, but an organisation with no appetite for a labelling exercise is not stuck with nothing." }, { "question": "Who should own AI agent evaluation?", "answer": "Someone outside the team that built the agent, with the standing to block a release. Tenhaw builds systems and designs the evidence but does not sell the gate. An engagement either side can end on thirty days' notice is the wrong permanent home for a decision your second line has to keep making. Where the build team owns the benchmark, the benchmark tends to describe what the system already does. This is an accountability question rather than a tooling one, and it is settled during operating-model design by mapping who decides, on what evidence, and who can overrule them." }, { "question": "What should we measure after an agent is live?", "answer": "The same criteria that gated the release, so degradation is comparable rather than anecdotal, plus cost and latency distributions and the rate at which humans override the agent. Override rate is often the most useful early signal, because it moves before accuracy metrics do and it is measured on real decisions." }, { "question": "What does an evaluation harness for an agent actually consist of?", "answer": "Four parts, and they are more ordinary than the phrase suggests. A dataset of real cases with the answer a qualified person would accept, held under version control and grown every time something goes wrong in production. A runner that executes the system against every case reproducibly, pinning the model version, the prompts, the retrieval corpus and the tool permissions, because a result you cannot reproduce is an anecdote. Scorers, which are a mixture of exact checks, rule-based assertions and, for open-ended output, a model acting as judge. And a report a non-engineer can read, showing the score against the threshold, what regressed since the last run, and what each run cost and how long it took. The UK AI Security Institute's open-source Inspect framework is built around exactly that shape, datasets, solvers and scorers, which is a reasonable sanity check on any design somebody presents to you. The part people underestimate is the dataset, because it is the only part that cannot be bought." }, { "question": "How do you build a ground-truth dataset for evaluating AI agents?", "answer": "Start with real cases pulled from the workflow itself, each paired with the answer a qualified person would accept, and build the set with the business owner rather than leaving it to the engineers. Agree the pass thresholds during design, so the set becomes the release gate rather than a retrospective justification. Keep it under version control alongside the prompts, model versions and tool permissions it exercises, and grow it every time something goes wrong in production, because live failures are exactly the hard cases the original set was missing. Runners and scorers can be adopted from open-source frameworks; a labelled set that reflects your own workflow is the part you have to make yourselves." }, { "question": "What is the difference between monitoring and evaluating an AI agent?", "answer": "Monitoring tells you the system is running. Evaluation tells you whether it is right. An agent can be fully available and quietly getting worse, because model behaviour drifts, retrieval corpora change and the input distribution moves, and none of that shows up on an uptime dashboard. The fix is to carry the offline evaluation criteria into production, running degradation monitoring alongside availability monitoring, with alerting thresholds and the rollback decision agreed before go-live rather than during the first incident. Most programmes budget for evaluation before release and nothing for the two years afterwards, which is why degradation is so often discovered by users rather than by the team that built the system." }, { "question": "How do we get started with agent evaluation and assurance?", "answer": "Start during design rather than after a demonstration, because the whole value is in agreeing what good means before there is a system to defend. Tenhaw agrees the ground-truth set and the pass thresholds with the business owner during design so they become the release gate, and publishes the method for doing it in full, free to adopt without hiring anyone. The four parts are offline evaluation against that set, trajectory scoring of the steps the agent takes rather than only its final answer, online monitoring once it is live, and a gate someone outside the build team can hold. The runner, the scorers and the report are ordinary engineering once the definition is settled." }, { "question": "What happens if the agent fails its evaluation thresholds?", "answer": "You get a specific, costed list of what is missing, which is a better position than a vague sense that the system is nearly there. A fail usually names a class of cases the system handles badly rather than condemning the idea, and those cases go into the dataset so the next run measures the fix rather than the impression. Hold the release, or ship the cases that pass and route the rest to a person until a later run clears them. That choice is a business decision rather than an engineering one. Failing a threshold and missing a date are different problems, and the second should never be solved by quietly moving the first." }, { "question": "Do we need all this for a low-risk internal AI tool?", "answer": "Not the full apparatus. Scale it to consequence, so a tool that drafts text a person reads and edits before it goes anywhere does not need an independent gate owner or evidence mapped to four regulatory regimes. Tenhaw grades its own engineering rules the same way, publishing 72 of them on GitHub with an RFC 2119 severity on each rather than equal weight. The light tier still needs a definition of good agreed before anything is built, a small set of real cases with the answer a qualified person would accept, and something watching for degradation, because model behaviour drifts whether or not the use case is risky. The heavier version earns its cost when a decision touches a customer, money or a regulator." }, { "question": "Can one set of evaluation evidence answer more than one regulator?", "answer": "Yes, if it is specified that way at the start; otherwise the same work gets done four times. Tenhaw specifies that mapping at the start, and the client owns the inventory, evidence and code from day one. For a UK insurer, one design can answer four regimes. A model and decision inventory holding prompts, corpora, tool permissions and model versions with named owners is what model risk management asks for, and SS1/23 states that expectation for UK banks; insurers sit outside its formal scope and are asked anyway. Outcome measures emitted as the system runs are the Consumer Duty evidence. Degradation monitoring with an exercised fallback is what an impact tolerance rests on, and field-level provenance to source is the Solvency II data quality expectation." }, { "question": "Does building an evaluation set slow the project down?", "answer": "It moves the argument earlier, to where it is cheap. Tenhaw builds the ground-truth set inside a retainer month that carries a value target, and a month that moves no number the client agreed is reported back as a failed month, so the set is built on the delivery clock rather than beside it. The thresholds are agreed during design alongside acceptance criteria you would be writing anyway. If you cannot state what good looks like, you cannot manage the work. Skip it and the production decision becomes a negotiation with no evidence to settle it, which is where programmes lose quarters. After the first run the harness makes delivery quicker, because a change is scored against every case at once." }, { "question": "Do we need to re-run evaluation when the model version changes?", "answer": "Yes, and it is one of the better arguments for having a harness at all. A run pins the model version alongside the prompts, the retrieval corpus and the tool permissions, so changing any of them gives you a different system and the previous result no longer describes it. Re-run the same dataset, compare against the same thresholds, and read what regressed since the last run. Providers update and retire versions on their own schedule rather than yours, which is why the re-run wants to be a routine job somebody owns rather than a project somebody funds." }, { "question": "What happens to the evaluation harness when the build team leaves?", "answer": "It hands over with everything else. On a Tenhaw engagement the exit date is agreed at kickoff, the final sixty days of a build run as a documented handover taper, and recruiting your permanent team is a stated deliverable of the retainer engagements. The dataset, the runner, the scorers and the report sit under version control in your own repositories, and the criteria that gated the release are the ones already monitored in production, so there is nothing to rebuild. What matters more is that a named person has been asked to own the live system, its run cost and its incidents, because if nobody has, evaluation quietly stops when the project does. Naming that receiver belongs in the pilot rather than afterwards." }, { "question": "Can a non-technical risk team sign off an AI agent?", "answer": "Yes, and designing for that is part of the work. What qualifies someone to hold a release gate is independence and the standing to say no, not the ability to read the code. What the team needs is evidence it can act on: the score against the agreed threshold, what regressed since the last run, and what each run cost and how long it took, written for the audience that will ask for it rather than reconstructed under pressure later. Agree the criteria with risk during design instead of presenting results afterwards, and name the regime each artefact answers, so they are checking evidence rather than translating it." } ], "citations": [ { "name": "UK AI Security Institute, Inspect: a framework for large language model evaluations", "url": "https://inspect.aisi.org.uk/", "note": "Open-source, built by the UK AI Security Institute and Meridian Labs around datasets, solvers and scorers, with more than 200 evaluations included. Cited here as evidence of the shape a serious harness takes, not as a recommendation of a tool." }, { "name": "Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)", "url": "https://arxiv.org/abs/2306.05685", "note": "Source of the over 80% agreement figure, and of the position, verbosity and self-enhancement biases that make a model judge useful for triage and unsafe as a gate." }, { "name": "FCA, the Consumer Duty", "url": "https://www.fca.org.uk/firms/consumer-duty/about", "note": "The Consumer Principle, the cross-cutting rules and the four outcomes, in the regulator's own words. Read it before accepting anyone's account of what the Duty asks of an agent." }, { "name": "FCA, the Senior Managers and Certification Regime", "url": "https://www.fca.org.uk/firms/senior-managers-certification-regime", "note": "The regime behind the claim that accountability resolves to a named individual rather than to a system." } ] } ## AI governance and regulatory evidence Source: https://tenhaw.com/guides/ai-governance-and-regulatory-evidence Last updated: 2026-11-05 { "slug": "ai-governance-and-regulatory-evidence", "name": "AI governance and regulatory evidence", "metaDescription": "Build an AI governance framework with inventory, classification, accountable owners and delivery evidence. Includes a sourced, dated AI Act timeline.", "tagline": "Keep the AI inventory, accountable decisions and evidence connected.", "summary": "An AI governance framework records which systems you use, their purpose and risk, who owns the decisions and how controls are evidenced. Start with that inventory, then use the current legislation and your legal and risk owners to establish the duties for each system.", "theme": "control", "evidenceBasis": "approach", "evidenceStatement": "This is how we would approach it, grounded in a decade of delivery inside regulated organisations including HSBC. We are not a law firm and we do not give legal advice; we design the operating model and the delivery discipline that produces the evidence your legal and risk functions need. Where you need formal legal interpretation, we will say so and work alongside whoever provides it.", "demandSignal": "Regulation (EU) 2026/1744 entered into force on 27 July 2026. It moved the application of Chapter III, Sections 1 to 3, except Article 6(5), to 2 December 2027 for Article 6(2)/Annex III high-risk systems and 2 August 2028 for Article 6(1)/Annex I systems. Its amended Article 4 still requires providers and deployers to take measures supporting the AI literacy of staff and people using systems on their behalf, without guaranteeing any individual's specific level. The final regulation and Commission explanation are linked in Sources. Confirm your system's classification, your role and the duties that follow with the legal owner.", "whyItStalls": [ { "heading": "The board wants an AI plan, and nobody can list the AI already running", "body": "The inventory is the first deliverable and the one that reliably takes three times as long as planned, because AI has entered the organisation through tool purchases, embedded vendor features and individual initiative rather than through a single programme. You cannot classify what you cannot enumerate, and you cannot write a credible plan for a board on top of an estate nobody has counted. Plans written before the count are the ones that get quietly rewritten two quarters later." }, { "heading": "Governance is written as policy, not as process", "body": "A policy that says agent decisions must be auditable does not make them auditable. Where the requirement is not embedded in how work actually flows, evidence has to be reconstructed later from people's memories, which is expensive, thin, and exactly what an assessor is trained to notice." }, { "heading": "Accountability does not resolve to a person", "body": "Under existing regimes such as SM&CR a named individual is accountable for outcomes, and 'the system decided' is not a defence. Where the operating model has not mapped which agent decisions sit under whose accountability, the governance framework has a hole in exactly the place a regulator looks first." }, { "heading": "Second line arrives at the end", "body": "Risk and compliance are brought in to review rather than to design, so they see a finished system and the only lever available is to block it. This is experienced as friction and is usually a sequencing failure rather than an obstruction." } ], "approach": [ { "heading": "Inventory first, and treat it as real work", "body": "Enumerate every AI system, including embedded vendor capability and tools bought outside procurement, before classifying anything. We scope this as a distinct piece of work rather than a precursor, because underestimating it is the most common reason a governance programme is late before it starts." }, { "heading": "Make risk and compliance co-authors, not reviewers", "body": "Second line is in the room during operating-model design, defining the control points rather than assessing them afterwards. In our experience this is the single highest-leverage sequencing decision in a regulated agentic programme, and it costs nothing except being early." }, { "heading": "Map accountability onto people before deployment", "body": "Every class of agent decision is mapped to a named accountable individual with matching authority, with the human-in-the-loop boundary defined by consequence and reversibility. This is the accountability mapping in our operating-model work, applied to the question a regulator will ask first." }, { "heading": "Generate evidence as a by-product", "body": "Audit trails, evaluation results, approval records and change history are outputs of the delivery process rather than a separate documentation exercise. If producing the evidence requires a project, the evidence will be late and thin; if it falls out of how the work already runs, it is close to free." }, { "heading": "Govern it on a cadence", "body": "Inventory, classification and post-market monitoring are reviewed on a fixed rhythm with named owners, in the same way as any other operating cadence. Governance that is reviewed annually describes an organisation that no longer exists." } ], "engageVia": { "slug": "agentic-design-team", "label": "Agentic Design Team" }, "faq": [ { "question": "When do the EU AI Act's high-risk obligations actually apply?", "answer": "Regulation (EU) 2026/1744, Article 1(40)(b), sets application of Chapter III, Sections 1 to 3, except Article 6(5), at 2 December 2027 for systems classified as high-risk under Article 6(2) and Annex III, and 2 August 2028 for those under Article 6(1) and Annex I. The amending regulation entered into force on 27 July 2026. Other provisions have their own dates, so confirm the classification, your organisation's role and the duties for that system with your legal owner. The final regulation and Commission overview are linked in Sources. Tenhaw helps establish the delivery practices, accountability and evidence records that support those decisions." }, { "question": "The board wants an AI plan. What should actually be in it?", "answer": "Five things, and they are all answerable in weeks rather than quarters. An inventory of the AI already running, including embedded vendor features and anything bought outside procurement, because a plan written before the count gets rewritten later. A classification of that inventory by consequence, so the board can see which systems carry real risk rather than a flat list. A named accountable individual per class of decision, which is the first thing a regulator tests and the first thing a board should. A sequence with dates, showing which processes go first and what evidence each one produces as a by-product of the work. And an honest statement of what you cannot yet evidence, because the plans that survive board scrutiny are the ones that name their own gaps before somebody else does. If a stalled pilot is what prompted the board to ask, say so, and say whether it stalled on evidence, on ownership or on the data underneath it, because those three need different money and different people." }, { "question": "Where do AI governance programmes usually go wrong?", "answer": "Four places. The inventory takes far longer than planned because AI entered the organisation through tool purchases and embedded vendor features rather than one programme. Governance is written as policy rather than embedded in process, so evidence has to be reconstructed. Accountability does not resolve to a named person, which is the first thing a regulator tests. And second-line risk arrives to review a finished system rather than to co-design it, so their only available lever is to block." }, { "question": "How do you make agent decisions auditable?", "answer": "By designing the audit trail into the workflow rather than adding logging afterwards: what the agent was asked, what it retrieved, which tools it called, what it decided, which human approved or overrode it, and against which version of the system. The test is whether you could reconstruct a specific decision from six months ago without asking anyone what happened." }, { "question": "Who is accountable when an AI agent makes a mistake?", "answer": "A named individual, and that has to be established before deployment rather than after an incident. Under regimes such as SM&CR accountability cannot rest with a system. In practice this means mapping each class of agent decision to a person with matching authority, and defining explicitly which decisions require human approval based on how consequential and how reversible they are." }, { "question": "What should an AI inventory include, and why does it take so long?", "answer": "Everything that behaves like AI in production, not just the systems a programme built: embedded vendor features, tools bought outside procurement, and anything adopted through individual initiative. It takes so long because AI rarely entered the organisation through one front door, so enumeration means going department by department rather than reading a project list. The inventory reliably takes three times as long as planned, so scope it as work rather than a precursor. Tenhaw's founder co-designed a target operating model across 500 teams at HSBC Global Payment Solutions, so the department-by-department pass is familiar ground. You cannot classify what you cannot enumerate. The effort pays for itself, because a governance plan written before the count is the one that gets quietly rewritten two quarters later." }, { "question": "Should risk and compliance review AI systems or help design them?", "answer": "Help design them. The common failure is one of sequencing. Risk and compliance are brought in at the end to review a finished system, so the only lever left is to block it, which everyone then experiences as friction. Put them in the room during operating-model design, defining the control points rather than assessing them afterwards. In Tenhaw's experience this is the single highest-leverage sequencing decision in a regulated agentic programme, and it costs nothing except being early. Controls designed with the people who will have to defend them also change the evidence, producing records that hold up under assessment rather than documentation reconstructed after the fact." }, { "question": "The EU AI Act deadline moved. Should we pause our AI governance work?", "answer": "Continue with a proportionate inventory and clear decision ownership. The Commission reports amended application dates of 2 December 2027 for Annex III high-risk rules and 2 August 2028 for relevant product-embedded rules, following the Omnibus entering into force on 27 July 2026. The changes include more than timing, so ask the legal owner to verify the current duties for your system. Delivery teams can meanwhile identify purposes, owners, source access, review points and evidence gaps. Those records make it easier to decide what to test or change next without assuming that every use case has the same legal obligations." }, { "question": "How often should AI governance be reviewed?", "answer": "On a fixed rhythm with named owners, in the same way as any other operating cadence, and certainly more often than annually. Governance reviewed once a year describes an organisation that no longer exists, and three things need the cadence. The inventory, because new AI arrives continuously through vendor features and tool purchases rather than through a programme you control. The classification, because a system's consequence shifts as its scope and usage grow. And post-market monitoring, because the obligations on high-risk systems do not end at deployment. Run each review as a working session that updates the record, with a named owner per item, rather than a committee that receives a report." }, { "question": "What evidence should an AI governance programme produce?", "answer": "Audit trails, evaluation results, approval records and change history, produced as outputs of the delivery process rather than as a separate documentation exercise. If producing the evidence requires a project of its own, it will be late and thin, and reconstructed documentation is exactly what an assessor is trained to notice. For a high-risk system under the EU AI Act the record needs to support conformity assessment, technical documentation, risk management, data governance, human oversight, registration and post-market monitoring. Build the requirement into how work actually flows, so the records accumulate as people deliver, and the evidence becomes close to free rather than assembled under deadline." }, { "question": "Is there an FCA AI governance framework we should be following?", "answer": "There is no separate FCA AI rulebook, and the regulator has said it does not intend to write one. Tenhaw designs the operating model rather than the legal opinion, drawing on a decade of delivery inside regulated organisations including HSBC. Under the Senior Managers and Certification Regime, 'the system decided' is not a defence, so an FCA AI governance framework is the regimes you already answer to applied to AI, with individual accountability as its spine. It is four things: an inventory of the AI actually running, vendor features included; a classification of it by consequence; a named accountable individual for each class of agent decision, with matching authority and an approval boundary; and evidence generated by the delivery process rather than reconstructed for inspection." }, { "question": "Do we need a separate AI framework, or can we extend model risk governance?", "answer": "Extend what you have, then close the three gaps it leaves. Scope is the first. Model risk governance sees models that went through a model lifecycle, while AI has arrived through tool purchases, embedded vendor features and individual initiative, so the inventory is built department by department, not read off a register. Accountability is the second, because agents act, and each class of agent decision needs a named individual with matching authority and a human-approval boundary set by consequence and reversibility. Evidence is the third, so audit trails, evaluation results, approval records and change history have to fall out of how work runs. Tenhaw publishes that method in full, and you are free to adopt it without hiring anyone." }, { "question": "Will an AI governance platform give us the evidence a regulator wants?", "answer": "It will store evidence. It will not create it. What an assessor asks for is audit trails, evaluation results, approval records and change history for specific systems and specific decisions, and those are produced by how the work runs rather than by the tool that holds them. Buy a platform and change nothing about delivery and you have a tidy home for documentation reconstructed from people's memories, which is expensive, thin, and exactly what an assessor is trained to notice. Tenhaw sells no platform; its own engineering handbook is 72 rules on GitHub, enforced by an agent rather than filed. Get the requirement embedded in the workflow first, then choose somewhere to keep the output. The order matters more than the product does." }, { "question": "What does it cost to get AI governance in place?", "answer": "It is usually bought one of two ways, and Tenhaw publishes both prices rather than quoting on request. If nobody has counted the estate yet, the AI Readiness Audit is the entry point at £44,000 fixed over four weeks, with the first two weeks spent inventorying what is actually in use across the business, sanctioned or not, and ending in working prototypes rather than a slide pack. If the direction is already clear and what you need is the operating model, accountability mapped onto named people and delivery that generates its own evidence, that is Agentic Design Team work at £35,000 to £55,000 a month for two senior people. The audit is standalone with no obligation to continue, and a recommendation to stop is a valid outcome." }, { "question": "How do you decide which AI systems count as high risk?", "answer": "In two passes. First, a practical triage of your own, classifying the inventory by consequence so a board can see which systems could genuinely harm a customer, a colleague or the firm, rather than reading a flat list of everything running. Second, the legal test against the EU AI Act's own categories, which is where counsel earns the fee, because the classification has to be defensible to somebody else rather than comfortable for you. You cannot classify what you have not counted, so enumerate before either pass. And keep the classification as a live record, because a system that was low-consequence at pilot scale rarely stays that way." }, { "question": "Does ISO 42001 certification cover us for the EU AI Act?", "answer": "No, because they answer different questions. ISO/IEC 42001 certifies that you run a management system for AI: policies, roles, controls and review. The EU AI Act asks what a particular system evidences, and for a high-risk system that means conformity assessment, technical documentation, risk management, data governance, human oversight, registration and post-market monitoring. The certificate is genuinely useful, because the disciplines overlap heavily and both want a current, defensible record, but it does not discharge a per-system obligation. Firms running 42001 properly tend to find the per-system work smaller rather than unnecessary, because the inventory, the classification and the review cadence already exist." }, { "question": "Will AI governance slow our delivery down?", "answer": "Built into the work it costs very little, and bolted on afterwards it costs a great deal. The expensive version is a policy saying agent decisions must be auditable while nothing in the workflow changes, so the records get assembled under deadline from memory. Where the audit trail, the evaluation results and the approval record are outputs of the delivery process, they accumulate while the team works. Tenhaw's work at Globelynx cut lead time by 60% inside six months, which is the direction disciplined delivery moves in. What people usually mean when they say governance slowed them down is second line arriving to review a finished build, with blocking as the only lever left. That is a sequencing failure rather than a control problem." }, { "question": "Do we need a law firm or a delivery partner for AI governance?", "answer": "Usually both, doing different jobs. A law firm tells you how the EU AI Act and your own regulator's rules apply to your systems: the formal interpretation, the classification you can defend, the contractual position with vendors. That is not Tenhaw. It is not a law firm and it does not give legal advice. What Tenhaw designs is the operating model and the delivery discipline that produce the evidence your legal and risk functions need, which is the part no legal opinion delivers: the inventory, accountability mapped onto named people, and audit trails and approval records generated as a by-product of the work rather than assembled under deadline." } ], "citations": [ { "name": "Regulation (EU) 2026/1744, final Official Journal text", "url": "https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX%3A32026R1744", "note": "Official Journal 24 July 2026, checked 5 November 2026. Article 1(5) replaces AI literacy duties; Article 1(40)(b) amends high-risk application dates; Article 4 governs entry into force." }, { "name": "European Commission, AI Act framework and application timeline", "url": "https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai", "note": "Primary overview checked 5 November 2026, with a link to the final AI Omnibus text and current high-risk dates." }, { "name": "European Commission, AI Omnibus enters into force", "url": "https://digital-strategy.ec.europa.eu/en/news/ai-omnibus-enters-force", "note": "Published 27 July 2026. Confirms entry into force, amended high-risk dates and the simplification of the previous company AI-literacy requirement." }, { "name": "EU AI Act, original Regulation2024/1689", "url": "https://eur-lex.europa.eu/eli/reg/2024/1689/oj/eng", "note": "Original legal text. Read it with the amending legislation linked by the Commission, not as an unchanged current timetable." } ] } ## Agent identity and access Source: https://tenhaw.com/guides/agent-identity-and-access Last updated: 2026-11-05 { "slug": "agent-identity-and-access", "name": "Agent identity and access", "metaDescription": "Give AI workflows bounded identities, permissions and a tested revocation route. Compare the authority, approval and evidence each action needs.", "tagline": "Give each workflow the authority it needs, with a way to take it away.", "summary": "Agent identity and access management gives a workflow its own identity, bounded permissions, an action trail and a revocation route. Start with the actions it may read, propose or execute. Your IAM and workflow owners can then agree the credentials, resource grants and approval checks and test those boundaries together.", "theme": "control", "evidenceBasis": "approach", "evidenceStatement": "This is how we would approach it, grounded in how we actually run pilots and in delivery inside regulated environments. Our own practice is to run pilots locally against mocked services, or in a dedicated hosted environment with synthetic data rather than real customer records, with the CISO involved from design concept and agreeing the scope up front. Where deep identity engineering is required we would expect to work alongside your IAM function or a specialist.", "demandSignal": "The population this belongs to is already enormous and already ungoverned, before a single agent is added to it. A survey of 2,930 IT and security leaders, published as the 2026 Identity Security Landscape, puts the ratio at 109 machine identities for every human and reports that 96% of human users hold access beyond what their role requires. Agents are the newest entrants to that population and the first that decide for themselves what to do with the access they hold.", "whyItStalls": [ { "heading": "The agent inherits a person's credentials", "body": "The fastest way to ship a pilot is to run the agent as the developer or as a shared service account. It works immediately, it is invisible in the logs, and it means every action the agent takes is attributed to a human who did not take it. Unpicking this after the fact is far more expensive than designing it in." }, { "heading": "Permissions are scoped to the agent, not to the task", "body": "An agent that needs to read one system for one workflow is given broad standing access because that is simpler. The blast radius of a prompt injection or a reasoning error then equals the entire permission set rather than the task at hand. One agent like this is survivable. It is also the thing that quietly stops organisations scaling AI agents across the enterprise: at thirty agents nobody can answer what any of them is permitted to do, and the security function's only remaining lever is to slow the whole programme down." }, { "heading": "Retrieval quietly bypasses source permissions", "body": "The most common serious failure in enterprise retrieval. Documents are indexed with the ingestion account's privileges, so the assistant will answer any user from any document it indexed. Permissions must propagate through to query time, and vendor quick-starts rarely do this." }, { "heading": "There is no revocation story", "body": "Nobody can answer what happens when an agent misbehaves at three in the morning. If the answer involves finding the person who deployed it, you do not have a control, and that will be the finding." } ], "approach": [ { "heading": "Do not use real customer data to prove the concept", "body": "Our default is to run pilots locally against mocked services, or in a dedicated hosted environment, using synthetic data built to exercise the real use cases. It removes the hardest approval from the fastest-moving phase of the work, and it means the identity and access design can be got right before anything sensitive is in scope. It also tends to be why our CISO conversations are short." }, { "heading": "Bring the CISO in at design concept, not at review", "body": "In our engagements the security function is involved from the design concept and agrees the limited scope up front. That is the difference between security being a co-author and security being the last gate before a date, and it is almost entirely a sequencing choice rather than a cost." }, { "heading": "Give every agent its own identity from day one", "body": "Distinct, enumerable identities per agent, never shared with a human and never a general-purpose service account. This is cheap at design time and expensive to retrofit once actions have accumulated against the wrong principal." }, { "heading": "Scope permissions to the task, and make them expire", "body": "Least privilege applied at the granularity of the workflow rather than the agent, with time-boxed credentials. The design question is what this agent needs for this task for this long, which is also the question that makes the blast radius calculable." }, { "heading": "Propagate source permissions through to query time", "body": "Retrieval respects the permissions of the asking user against the source system, not the privileges of whatever indexed the corpus. We treat this as a design requirement rather than a hardening step, because retrofitting it usually means rebuilding the index." }, { "heading": "Log actions against the agent, and link them to the human", "body": "Every action attributable to a specific agent identity, and traceable to the human accountable for that agent under the operating model. This is where identity design and accountability mapping meet, and where governance evidence comes from without a separate exercise." }, { "heading": "Design the kill switch before you need it", "body": "A named person, a documented mechanism and a tested path to revoke an agent's access immediately. Untested revocation is not a control, and this is the question we would expect a CISO to ask first." } ], "providerNotes": { "heading": "Provider divergence is real here", "body": "Unlike most patterns, identity is genuinely different across the hyperscalers: the agent identity primitives, how they federate with an existing enterprise directory, and how far permission propagation is supported natively rather than built. The design principles above are provider-independent; the implementation is not, and we would expect to work alongside your existing IAM function rather than around it." }, "engageVia": { "slug": "agentic-design-team", "label": "Agentic Design Team" }, "faq": [ { "question": "Should AI agents have their own identities?", "answer": "Yes, and from the first pilot rather than at production. Tenhaw treats a distinct identity per agent as a day-one design decision on Agentic Design Team work, rather than something added before go-live. Running an agent under a developer's credentials or a shared service account attributes its actions to a human who did not take them, makes agent activity invisible in logs, and is materially more expensive to unpick later than to design correctly at the start." }, { "question": "How do you stop AI retrieval leaking documents to the wrong users?", "answer": "By propagating source-system permissions through to query time, so retrieval respects the permissions of the asking user rather than those of the account that indexed the corpus. This is the most common serious failure in enterprise retrieval, and because it is usually discovered after indexing it often means rebuilding the index. It belongs in the design, not in hardening." }, { "question": "What breaks when you go from one agent to thirty?", "answer": "Everything that was a shortcut becomes a control failure. Shared credentials stop being a tidiness issue and start meaning that no log can attribute an action to a specific agent. Standing permissions granted for convenience add up to a combined blast radius nobody has calculated. Retrieval indexes built with an ingestion account's privileges multiply into a leak surface across every corpus. And revocation, which was one person and a console, becomes a question of who is on call at three in the morning for thirty systems. The organisations that scale AI agents across the enterprise are the ones that made identity, permission propagation, action logging and revocation a shared platform concern before the second agent, rather than solving it per agent thirty times." }, { "question": "What is non-human identity governance?", "answer": "Managing the identities, credentials, permissions and lifecycle of things that are not people (service accounts, workloads and now AI agents) with the same rigour applied to human identity. It has risen sharply up the security agenda because agents that call tools take consequential actions with real authority, and because the population was already unmanaged before agents arrived. A 2026 survey of 2,930 IT and security leaders put machine identities at 109 for every human, and found that 96% of human users already hold access beyond what their role requires." }, { "question": "How do you run an AI pilot without exposing customer data?", "answer": "Run it locally against mocked services, or in a dedicated hosted environment, on synthetic data built to exercise the real use cases rather than on production records. It removes the hardest approval from the fastest-moving phase of the work and lets the identity and access design be settled before anything sensitive is in scope. Combined with static analysis, dependency and secrets scanning in the pipeline and a model-led security review roughly every fifth prompt during the build, proofs of concept produced this way are frequently more compliant than the legacy systems they sit next to. This is Tenhaw's own default on pilot work, with the security function involved from design concept and agreeing the limited scope up front." }, { "question": "When should the security team get involved in an agentic project?", "answer": "At design concept, agreeing the scope, rather than at review. Tenhaw works that way on its own engagements, with the security function involved from the design concept and agreeing the limited scope up front. It is a sequencing choice that costs almost nothing and turns the security function into a co-author of the control design instead of the last gate before a committed date, where its only available lever is to block." }, { "question": "What is the first thing a CISO should ask about an agent deployment?", "answer": "How do we revoke it, who is authorised to do that, and when was it last tested. If the answer involves locating the person who deployed the agent, there is no control, and untested revocation is not a control either. The second question is what the blast radius is if this agent is manipulated, which is answerable only if permissions are scoped to the task rather than to the agent." }, { "question": "What does least privilege look like for an AI agent?", "answer": "Permissions scoped to the task rather than to the agent, with time-boxed credentials that expire. The design question is what this agent needs, for this task, for this long. If the agent is then manipulated through prompt injection or makes a reasoning error, the damage is bounded by the task at hand rather than the agent's entire permission set, which is what makes the blast radius calculable. Broad standing access granted because it is simpler is the shortcut that quietly stops organisations scaling agents later, so Tenhaw treats task-scoped, expiring credentials as a design requirement from the first pilot rather than a hardening step." }, { "question": "Does agent identity work the same across cloud providers?", "answer": "No, and this is one of the few areas where the differences between providers genuinely matter. The agent identity primitives, the way they federate with an existing enterprise directory, and how far permission propagation is supported natively rather than built all vary meaningfully across the hyperscalers. The design principles are provider-independent: a distinct identity per agent, task-scoped credentials that expire, source permissions propagated through to query time, actions logged against the agent, and a tested revocation path. The implementation is not, so the engineering effort, and how much you build rather than configure, will differ depending on which platform you are on." }, { "question": "Can our existing IAM programme handle AI agents?", "answer": "Partly, and the gap is in the definitions rather than the tooling. In most organisations the definition of a privileged user still means a person, while an agent is a non-human identity with real authority, and one that decides for itself what to do with the access it holds. The disciplines your IAM function already runs transfer directly: enumerable identities, least privilege, logging, revocation. What does not transfer is the shared service account habit, which attributes an agent's actions to a principal that did not take them and is expensive to unpick once actions have accumulated. Where deep identity engineering is required, Tenhaw would expect to work alongside your IAM function or a specialist, not around them." }, { "question": "What is the difference between a service account and an agent identity?", "answer": "A service account is a shared, general-purpose principal that runs a fixed job, usually with standing broad privileges and no way to tell from the logs what triggered it. An agent identity is distinct and enumerable, held by one agent, never shared with a human, scoped to the task rather than to the system it touches, and issued with credentials that expire. The distinction matters because an agent, unlike a script, decides for itself what to do with the access it holds, so both the record of what it did and the bound on how far it could go have to be designed in rather than reconstructed afterwards." }, { "question": "What is excessive agency in the OWASP LLM Top 10?", "answer": "LLM06, Excessive Agency, is the entry in the 2025 OWASP Top 10 for large language model applications covering an agent that holds more permission, functionality or autonomy than its task requires, so a manipulated or mistaken model can act well beyond the job in hand. It is the standards-body version of the argument for scoping permissions to the workflow rather than to the agent, with credentials that expire. Its companion, LLM08, Vector and Embedding Weaknesses, covers the retrieval side. Both are useful references when a control design needs to carry weight with a risk committee rather than be argued from first principles." }, { "question": "What is security trimming, and does it work for AI retrieval?", "answer": "Security trimming is the pattern where every indexed document carries an identity field and the search filters results at query time against the group membership of the person asking, so an assistant can only answer from documents that person could already open. Microsoft documents it for Azure AI Search, named here as one provider's worked example rather than as a recommendation of that provider. It works, provided the decision is taken before the corpus is indexed and the filter is evaluated against the source system rather than against a copy of the permissions made at ingestion. What it does not do is bound what the agent may then act on, which is a separate question about task-scoped permissions." }, { "question": "Our agents already run on a shared account. How do we fix it?", "answer": "Start by establishing what that account can actually reach, because the answer is almost always broader than anyone expects, and that permission set is your current blast radius. Tenhaw takes this on as Agentic Design Team work. Then move the agents onto distinct identities of their own before you add the next one, since unpicking attribution once actions have accumulated against the wrong principal is materially more expensive than designing it in. Alongside that, agree who is authorised to revoke an agent's access and test the path. One agent on shared credentials is survivable. The trap is arriving at thirty of them and being unable to say which one took a given action." }, { "question": "Who needs to be involved in designing agent access?", "answer": "Four roles, and the sequence matters more than the headcount. The security function comes in at design concept and agrees the scope, so it co-authors the control design rather than gating it the week before a committed date. Your IAM function, or a specialist alongside it, owns the identity engineering itself, and the sensible posture is to work alongside them rather than around them. The workflow owner is the only person who can say what the agent needs, for this task, for how long. And one named human is accountable for the agent, which includes being explicit about who may switch it off." }, { "question": "Won't a separate identity for every agent create identity sprawl?", "answer": "You will have more identities, and that is the right trade. Sprawl is not really a counting problem so much as an enumeration problem. The machine identity population in most organisations was already large and largely ungoverned before a single agent joined it, and the failure is that nobody can say what any given principal is permitted to do. Distinct identities per agent, scoped to the task and expiring, make the population enumerable and the blast radius calculable. The alternative, one shared account behind everything, means fewer identities and no attribution at all, which is the version that becomes unmanageable at thirty agents rather than at three." }, { "question": "How much does it cost to get agent identity and access right?", "answer": "Less than retrofitting it. Distinct identities and task-scoped permissions are cheap at design time and expensive once actions have accumulated against the wrong principal. Most of what it costs is sequencing rather than engineering, since the decisions that matter are taken before anything is indexed or deployed. Tenhaw publishes the method in full and free to adopt, so a team with a functioning IAM practice can apply this design without hiring anyone, and where deep identity engineering is bought it runs alongside your own IAM function or a specialist rather than around them. Engagements are retainer-shaped, and a month that delivers no measurable value is reported as a failed month." }, { "question": "Where does the 109 machine identities per human figure come from?", "answer": "From the 2026 Identity Security Landscape, an Idira by Palo Alto Networks survey of 2,930 IT and security leaders published in May 2026. It reports 109 machine identities per human and that 96% of human users hold more access than their role requires. These are findings from a vendor-commissioned survey, so cite the edition and sample alongside the figures. The source note records why earlier CyberArk statistics were retired." } ], "citations": [ { "name": "2026 Identity Security Landscape, Idira by Palo Alto Networks, May 2026", "url": "https://www.paloaltonetworks.com/idira", "note": "Vendor-commissioned May 2026 survey of 2,930 IT and security leaders: the source of the 109 to 1 and 96% figures. Earlier CyberArk statistics were retired when the old press release redirected to this edition without carrying those figures." }, { "name": "OWASP Top 10 for Large Language Model Applications (2025)", "url": "https://genai.owasp.org/llm-top-10/", "note": "LLM06 Excessive Agency and LLM08 Vector and Embedding Weaknesses are the entries behind the scoping and retrieval-permission arguments above." }, { "name": "Microsoft, security filter pattern for Azure AI Search", "url": "https://learn.microsoft.com/en-us/azure/search/search-security-trimming-for-azure-search", "note": "A worked example of propagating source permissions to query time: an identity field on every indexed document, filtered against the caller's group membership at search time. Named as one provider's documented pattern, not as a recommendation of that provider." } ] } ## AI-native SDLC and product delivery lifecycle Source: https://tenhaw.com/guides/ai-native-sdlc-and-product-delivery Last updated: 2026-11-05 { "slug": "ai-native-sdlc-and-product-delivery", "name": "AI-native SDLC and product delivery lifecycle", "metaTitle": "AI SDLC: what it is, phases and framework", "metaDescription": "Explore an AI-native SDLC from requirements to human release decisions. Compare responsibilities and inspect the scoped PPL proof-of-concept method.", "tagline": "Connect requirements, AI engineering and human release decisions.", "summary": "An AI-native SDLC changes how a team specifies, builds and verifies software. The requirements form a shared corpus, engineers direct the build, and people assess the result against tests and user journeys. At PPL, Tenhaw used the published method for a working two-week document-to-BI proof of concept. The comparison below shows the responsibilities that change across the lifecycle.", "theme": "patterns", "evidenceBasis": "delivered", "evidenceStatement": "We have published both the delivery lifecycle and the build method in full, and applied the build method on a client engagement where it produced a working proof of concept in two weeks, pair-programmed with the client's own engineer. Rolling out an AI-native SDLC across an engineering organisation of several hundred people is exactly the work our design and build teams are scoped for: you start from a published method and a proof point earned in two weeks, not from a pilot we are inventing on your budget.", "evidenceNextStep": { "heading": "So what should you do about it", "body": "Write your own engineering standard first, explicit enough for an agent to enforce. That is a fortnight of work with your own principals, every later decision hangs off it, and it does not need us. Where we are useful is the design rung: an Agentic Design Team designs the target operating model, the roles defined by the decisions they own, the governance and the adoption plan, at £35,000 to £55,000 a month over two to four months, and it ends on a sequenced plan your own engineers can execute. It is deliberately not the build rung. An Agentic Build Team at £70,000 to £85,000 a month exists to build agentic systems inside your estate, which is a different question from how your lifecycle changes, and if where AI delivers value across the business is still the open question, the AI Readiness Audit answers that one first at a fixed £44,000." }, "demandSignal": "Developer tooling was among the earliest and widest enterprise AI deployments, and the useful research question has moved on from how many engineers use it. The DORA programme's 2025 report on AI-assisted software development concluded that AI acts as an amplifier, magnifying an organisation's existing strengths and weaknesses rather than delivering a uniform uplift. That is the finding worth planning around, because it explains why two organisations buying identical licences get different results, and why the unanswered question in most places is not whether engineers will use AI but why the lifecycle it was bought to change has not changed.", "whyItStalls": [ { "heading": "Adoption plateaus, and more enablement does not move it", "body": "The first cohort adopt because they were always going to. Everyone else adopts when their actual role, measurement and definition of done change. More training does not move that, because awareness was never the constraint, and that is the single most common misdiagnosis we see. An AI programme stalled at the tooling layer looks the same from the outside every time: the licences are bought, a minority of them are in daily use, the demos went well, and the lifecycle they were bought to change has not changed at all.", "links": [ { "label": "What enterprise AI has to mean past the licence layer", "href": "/guides/end-to-end-agentic-workflow-implementation#enterprise-ai" } ] }, { "heading": "Requirements stay in tickets and people's heads", "body": "If the model can only see one file at a time, you have bought a faster autocomplete rather than changed the lifecycle. The leverage comes from giving the model the whole requirement set, which requires the requirements to exist somewhere machine-readable, which is a process change rather than a tooling one." }, { "heading": "Review capacity becomes the new bottleneck", "body": "Generation speeds up and human review does not, so the queue simply moves. Organisations that do not redesign review (what gets automated first-pass, what a human must see, what the standard is) end up with the same throughput and more code to maintain." }, { "heading": "There is no enforceable standard for what good looks like", "body": "A model will happily produce code that passes tests and violates every convention the team holds. Without a standard explicit enough for an agent to enforce, output volume rises and maintainability falls, which shows up two quarters later as slower delivery." } ], "approach": [ { "heading": "Treat requirements as the source code", "body": "Convert the requirement estate into structured markdown, map the relationships, and run the gap-and-contradiction pass before any build. This is steps one to seven of our published build method, and it is the part that changes the economics rather than the part that generates code." }, { "heading": "Move the unit of work from stories to outcome-driven epics", "body": "The concrete change in our own definition of done: epics become outcome-driven and oriented around the behaviour of the system, rather than work being decomposed into stories describing what someone will build. Development then starts by pasting the epic into Claude or Codex in the editor and iterating until both the business objective and the code are met, which only works if the epic actually describes the outcome rather than the task.", "links": [ { "label": "An outcome ticket against a story on the same epic: 54 front-end changes to 1", "href": "/case-studies/agentic-ai-in-insurance-operating-model" } ] }, { "heading": "Replace human authorship with human verification of coverage", "body": "Where a model writes most of the code, the control is no longer reading every line. It is high automated and unit test coverage, performance testing, and manual testing of the key user journeys. If all of those pass, the system is no more at risk than human-written code, the output volume is simply much higher. That is a different assurance model, and a risk function can sign it off because every control in it is testable rather than asserted: coverage, performance and the key user journeys either pass or they do not, and the evidence is on file." }, { "heading": "Make the engineering standard machine-enforceable", "body": "We publish ours as an open-source handbook with stable rule identifiers and RFC 2119 severities, written to be enforced by an agent rather than remembered by a human. An organisation adopting this needs its own equivalent, and agreeing it is a fortnight of work that saves quarters." }, { "heading": "Treat engineer resistance as signal, not obstruction", "body": "Resistance is expected, because change is hard, and we bring people on the journey by sitting with them and showing the value rather than mandating it. The more useful observation is that engineers who push back for a specific technical reason are very often right. Once the objection is understood it can usually be resolved by updating a skill file or an instruction so the model stops doing the thing they objected to, and those engineers then adopt fastest, because their objection was answered rather than overruled." }, { "heading": "Pair rather than hand over", "body": "Capability transfers by building together. On our insurance engagement the client engineer who paired through a two-week build finished it 70% confident they could run the process unaided, the measure we would want across a wider adoption programme rather than counting licence activations.", "links": [ { "label": "The two-week working proof of concept, paired with the client's engineer", "href": "/case-studies/specialty-insurance-agentic-lead" } ] }, { "heading": "Measure adoption as behaviour, not seats", "body": "Licence activation is not adoption. What matters is whether the lifecycle changed: are requirements maintained as a corpus, has review been redesigned, is the standard enforced, and has throughput moved on real work. Our Adoption Lead owns this, and it is measured monthly rather than surveyed annually." } ], "engageVia": { "slug": "agentic-design-team", "label": "Agentic Design Team" }, "faq": [ { "question": "What is an AI-native SDLC?", "answer": "A software delivery lifecycle redesigned around AI doing the research, drafting, scaffolding and first-pass review, rather than one with a coding assistant bolted on. Tenhaw publishes both halves of it, The Tenhaw Way as the product delivery lifecycle and a build method that took a proof of concept from blank repository to working in two weeks with Placing Platform Limited (PPL). In practice it means requirements maintained as a machine-readable corpus rather than as tickets, gaps and contradictions resolved before code is written, review redesigned because generation is no longer the bottleneck, and an engineering standard explicit enough for an agent to enforce." }, { "question": "Why does AI coding tool adoption stall?", "answer": "Because the first cohort adopt for their own reasons and everyone else adopts only when their role, measurement and definition of done change. Additional training and enablement reliably fail to move it, because awareness was never the constraint. Moving it takes lifecycle and operating-model change, so what a story must carry, what review looks like and what good is defined as all have to be settled. The 2025 DORA research points the same way from a different angle, finding that AI amplifies an organisation's existing strengths and weaknesses rather than lifting everyone equally, which is why the same licences produce different outcomes in two different engineering functions." }, { "question": "Does AI-assisted development make code less maintainable?", "answer": "It does if there is no enforceable standard, because a model will produce code that passes tests while violating conventions the team holds, and it will do so faster than humans can review. The mitigation is a standard specific enough to be machine-enforced, applied from the first prompt rather than in review. Tenhaw publishes its own as an open-source handbook with stable rule identifiers and RFC 2119 severities." }, { "question": "How do you handle engineers who resist AI-assisted development?", "answer": "By taking the objection seriously rather than treating it as change resistance. In Tenhaw's experience engineers who push back for a specific technical reason are usually correct. The model is doing something that genuinely violates a standard or a constraint they can see and you cannot, and that is nearly always fixable by updating a skill file or instruction so the behaviour stops. Engineers whose objection gets answered that way tend to adopt fastest, because they were listened to rather than overruled. Sitting with people and showing the value works; mandating it does not." }, { "question": "If AI writes most of the code, how do you know it is safe?", "answer": "By changing the control from authorship to verification. High automated and unit test coverage, performance testing, manual testing of key user journeys, and a published engineering standard the code is generated against. If all of those pass, the system carries no more risk than human-written code, the difference is that output volume is considerably higher. Tenhaw also runs static analysis, dependency and secrets scanning on every commit and a model-led security review roughly every fifth prompt, which in practice makes proofs of concept more compliant than a lot of legacy code." }, { "question": "How do you turn a proof of concept into production software?", "answer": "Decide first whether you are hardening or rebuilding it, and be willing to rebuild. A proof of concept is optimised to answer a question, so it carries shortcuts in identity, error handling and data handling that cost more to unpick than to redo, while the requirement corpus and the evidence it produced are worth keeping. From there it is ordinary: the whole requirement set at once, high automated and unit test coverage, performance testing, key journeys tested by hand, a security review roughly every fifth prompt, and a named owner who agreed the acceptance criteria. Tenhaw took one from a blank repository to working in two weeks with Placing Platform Limited (PPL), pair-programmed throughout with one of the client's own engineers." }, { "question": "How do you measure whether an AI-native SDLC is working?", "answer": "Not by licence activations. By whether the lifecycle actually changed: whether requirements are maintained as a corpus, whether review has been redesigned, whether the engineering standard is enforced, whether throughput moved on real work, and how confident engineers are that they could run the method unaided. That last one is measurable, so ask them, and take the honest number." }, { "question": "Why does code review become the bottleneck in AI-assisted development?", "answer": "Because generation speeds up and human review does not, so the queue simply moves from writing code to reading it. Staffing review up does not fix that. Redesign it instead, deciding what gets automated first-pass, what a human must see, and what standard the code is checked against. The deeper change is moving the control from authorship to verification, so instead of reading every line, a reviewer verifies high automated test coverage, performance results and the key user journeys. Teams that skip this redesign end up with the same throughput as before and more code to maintain." }, { "question": "What replaces user stories in an AI-native SDLC?", "answer": "Outcome-driven epics that describe the behaviour of the system rather than the tasks someone will perform. Tenhaw's own definition of done no longer decomposes work into stories describing what a developer will build. The epic states the business objective and the system behaviour, and development starts by pasting it into Claude or Codex in the editor and iterating until both the objective and the code are met. That only works if the epic genuinely describes the outcome, so the human effort moves up a level, away from specifying tasks and towards resolving ambiguity in the requirement itself." }, { "question": "What does it cost to move an engineering team to AI-native delivery?", "answer": "Less than you might expect, because the first step does not need a consultancy at all. Writing your own engineering standard, explicit enough for an agent to enforce, is a fortnight of work with your own principal engineers, and every later decision hangs off it. Where outside help earns its keep is designing the operating model around it: the roles, the governance and the adoption plan, which runs two to four months and ends on a sequenced plan your own engineers execute. Much of what you need is free either way, since Tenhaw publishes the method in full and its 72-rule engineering handbook as open source, adoptable without hiring anyone." }, { "question": "Where do we start with an AI-native SDLC?", "answer": "Start with the engineering standard, and write it yourselves. An AI-native SDLC hangs off a definition of good that is explicit enough for an agent to enforce rather than for a human to remember, and every later decision follows from it, including what review automates, what a definition of done carries and what the model is allowed to produce. The second move is getting requirements out of tickets and into a machine-readable corpus, then running a gap-and-contradiction pass before anyone builds. Do those two in order, because each makes the next cheaper. The standard tells the model what good looks like, the corpus tells it what to build, and only then is redesigning review worth the effort." }, { "question": "Do AI coding tools actually make a team ship faster?", "answer": "Only if the lifecycle changes with them. The 2025 DORA research on AI-assisted software development found that AI acts as an amplifier, magnifying an organisation's existing strengths and weaknesses rather than lifting everyone by the same amount, which is why two engineering functions buying identical licences get very different results. Generation gets faster on day one. Delivery only gets faster if review is redesigned around what a human must actually see, if the requirements exist somewhere the model can read in full, and if the standard is enforced from the first prompt. Judge it on real work shipped, not on tool telemetry." }, { "question": "Do requirements need rewriting before a model can build from them?", "answer": "Usually yes, and it is the first real piece of work rather than an afterthought. If the model can only see one ticket at a time you have bought a faster autocomplete, because the leverage comes from giving it the whole requirement set at once. In practice that means converting the requirement estate into structured markdown, mapping the relationships between the pieces, and running a gap-and-contradiction pass before any code exists. That pass is usually where a team finds out how much of the specification was living in somebody's head, which is far cheaper to discover there than halfway through a build." }, { "question": "How do you write an engineering standard an agent can enforce?", "answer": "Write it for a machine to apply rather than a human to remember. Tenhaw publishes its own as an open-source handbook of 72 rules, free to take as a structure, and agreeing your version is a fortnight of work with your own principal engineers. The mechanics are stable identifiers on every rule so an agent can cite the one it is applying, RFC 2119 severities so must, should and may are not left to interpretation, and rules specific enough to be checked rather than admired. The content is the part only your own principals can settle, and the argument is worth having, because without it a model will produce code that passes every test and violates every convention you hold." }, { "question": "Does it matter which AI coding tool we standardise on?", "answer": "Less than the lifecycle around it does. Tenhaw runs Claude Code, OpenAI models and Gemini, picking the best tool for the case and running it inside the client's own infrastructure under their policies, and its build work starts by pasting an outcome-driven epic into Claude or Codex in the editor. What decides the result is whether the model can see the whole requirement set, whether an enforceable standard is applied from the first prompt, and whether review has been redesigned around it. Those hold whichever vendor you pick, which is just as well, because the tooling turns over quickly and the operating model does not." }, { "question": "What do developers actually do when the model writes the code?", "answer": "Judgement work, mostly. The human effort concentrates on resolving ambiguity in the requirement, deciding what to proceed without, and owning the last few percent of correctness, which is where a model is weakest and where being wrong costs most. Rather than reading every line, an engineer checks high automated and unit test coverage, performance results and the key user journeys, and keeps the requirement corpus and the engineering standard current as the system changes, which moves the control from authorship to verification. Less typing, more deciding. The typing is what disappears, not the responsibility for what ships." }, { "question": "How long before our own engineers can run this without help?", "answer": "Weeks for the mechanics, longer for the judgement. On a Tenhaw engagement with Placing Platform Limited (PPL), the client engineer who paired through a two-week build finished it 70% confident they could run the process unaided, which is a fair picture of what two weeks of pairing buys: the workflow, the standard, the prompts and the test discipline, with the harder calls still shared. Capability transfers by building together rather than by handover or training, so the pairing has to be real work on your own codebase rather than a demonstration. Two weeks does not make anyone an expert, it makes them able to start." }, { "question": "How long does moving to AI-native delivery actually take?", "answer": "The first change lands inside a fortnight, the design of the operating model around it runs two to four months, and after that it is a rolling programme. The fortnight is agreeing an enforceable engineering standard with your own principal engineers, and teams feel it immediately in what the model is allowed to produce. Tenhaw ran this work for six months with Placing Platform Limited (PPL), including a proof of concept taken from blank repository to working in two weeks. The two to four months covers the roles, the governance, the redesigned review and the adoption plan. It stays a rolling programme because what changes is behaviour rather than tooling, and behaviour changes team by team rather than on a launch date." }, { "question": "What has to change in our SDLC for a Claude Code enterprise implementation to stick?", "answer": "The standard, the definition of done and review, in that order. Tenhaw publishes its own engineering standard as 72 rules with stable identifiers and RFC 2119 severities, and a Claude Code enterprise implementation holds across teams only when yours is that explicit and enforced from the first prompt, which is a fortnight of work with your own principal engineers. Then the definition of done moves to outcome-driven epics, because the epic is what gets pasted into the editor. Then review shifts from reading every line to verifying coverage, performance and the key user journeys. Tenhaw runs all of it inside your infrastructure under your policies, or inside your own enterprise AI tenancy where you have one." } ], "citations": [ { "name": "DORA, State of AI-assisted Software Development 2025", "url": "https://dora.dev/research/2025/dora-report/", "note": "Published by Google Cloud with IT Revolution, GitHub, GitLab, SkillBench and Workhelix. Source of the amplifier finding: AI magnifies an organisation's existing strengths and weaknesses rather than delivering a uniform uplift." } ] } ## Target operating model for an AI-native organisation Source: https://tenhaw.com/guides/target-operating-model-for-an-ai-native-organisation Last updated: 2026-11-05 { "slug": "target-operating-model-for-an-ai-native-organisation", "name": "Target operating model for an AI-native organisation", "metaTitle": "AI target operating model: what agents change", "metaDescription": "Design how people and AI agents work together: roles, decisions, governance and architecture. A practical guide, grounded in product and delivery experience.", "tagline": "Design what agents do, what people decide and how the work improves.", "summary": "A target operating model for an AI-native organisation sets out how people and AI agents work together: the roles, decision rights, governance, workflows and technology that support the business. It makes clear what an agent may do, who checks its work and who remains accountable.\n\nStart with a task your team knows well. Their knowledge of the exceptions and handoffs helps you decide where AI could be useful, what needs human judgement and how to test the new way of working. The Tenhaw Way provides a published starting point for the product and delivery practices, free to use with your own team.", "theme": "patterns", "evidenceBasis": "delivered", "evidenceStatement": "Our experience is in product and delivery operating models. At HSBC, James Rooney co-led the design, piloting and refinement of a model prepared for 500 teams. That was a role held inside the bank and a delivery programme, separate from Tenhaw contracts and from AI transformation. The case record describes a planned 2026 global rollout; it does not establish a completed rollout.\n\nThe Tenhaw Way and its 17 how-to guides are published and used on our engagements. In specialty insurance, a six-month engagement ran from operating-model workshops and a working AI proof of concept to operating guides for Product, Engineering and QA, proven on live work. This supports the practices described here.", "evidenceNextStep": { "heading": "Choose a useful starting point", "body": "Try The Tenhaw Way and its how-to guides with your team. The method is free to read and use; applying it takes your team's time. Start with an outcome, the decisions around it and a way to measure what changes.\n\nIf you need help designing across functions, an Agentic Design Team pairs a senior operating-model lead with an agentic architect. They design the operating model and architecture with your people, typically over two to four months at £35,000 to £55,000 a month excluding VAT. The retainer can end on 30 days' written notice from either party. It finishes with a sequenced build plan that you can execute yourselves, with us or with another supplier.\n\nIf you first need to establish where AI could create value, the AI Readiness Audit is a separate four-week engagement at a fixed £44,000 excluding VAT, led personally by James Rooney." }, "demandSignal": "DORA's 2025 research on AI-assisted software development finds that AI can reinforce both effective practices and existing problems. It points to the organisation's technical and working practices as important conditions for getting value from AI.\n\nThat gives a team a useful question to investigate: what needs to change around the tool? A pilot may work well with its original users, while the next team has different records, approval responsibilities or support needs. Explore those differences with the people involved, alongside the system's performance and access requirements.", "whyItStalls": [ { "heading": "A useful pilot meets a different working day", "body": "The next team may handle different cases, rely on another system or need a different person to approve the result. Model capability, data quality and integration can all matter, alongside roles and governance. Map those differences with the teams before scaling. An operating model gives you a shared design to test and adapt as you learn.", "links": [ { "label": "HSBC's operating-model design and pilot", "href": "/case-studies/hsbc-gps-operating-model" }, { "label": "What a target operating model covers", "href": "/target-operating-model" } ] }, { "heading": "The build changes, and the review needs to catch up", "body": "AI can help people research, draft code, write tests and examine a design. The team still needs to agree how that output will be checked and what evidence permits release. Work through those decisions with developers and product colleagues, including where existing practices already work well. Review skills and clear requirements become part of the change.", "links": [ { "label": "Explore The Tenhaw Way", "href": "/the-tenhaw-way" } ] }, { "heading": "Decision owners need a clear boundary", "body": "An agent that drafts a recommendation needs different authority from one that changes a customer record. State what it may read, propose or change, when a person must approve and how to escalate an exception. Give each decision a human owner with the authority and information to act. Design this with the people doing the work and the relevant risk colleagues before deployment." }, { "heading": "The design needs a trial in everyday work", "body": "A proposed workflow may depend on access the platform cannot provide, or add review work that its users have no time to do. Bring architecture, operational knowledge and adoption into the same design conversation. Try a representative workflow with the team and use what happens to improve the roles, system and support arrangements together." } ], "approach": [ { "heading": "Start with a workflow your team knows", "body": "Pick a result worth improving and walk through how the work happens today. Invite the people who handle it, including those who receive the handover. They know which cases need an extra check and which steps exist because another system is awkward. Use that knowledge to sketch what an agent could do and what people would do around it.\n\nThe Tenhaw Way gives you a published starting point for the product and delivery practices: four values, two delivery modes, approval gates, quarterly roadmaps and seven recurring reviews and activities. Use the free guides to try the relevant practices and fit them to your organisation's controls.", "links": [ { "label": "The Tenhaw Way", "href": "/the-tenhaw-way" }, { "label": "The 17 practical how-to guides", "href": "/the-tenhaw-way/how-to" } ] }, { "heading": "Give the outcome a value you can check", "body": "Agree a monetary target, a baseline, a measurement period and a business owner. Show the assumptions behind the estimate so the team can question and improve them. Time recovered is capacity; it becomes a cash saving only when spending changes.\n\nIn The Tenhaw Way, epics that deliver an outcome carry planned contributions to its target. Compare their total with the target and give any gap an owner. Tech Debt and Bug Budget epics reserve capacity separately. Monthly outcome validation checks what value has been realised, including when it falls short of the plan." }, { "heading": "Choose how each team will build with AI", "body": "The Tenhaw Way describes two delivery modes. In AI-augmented delivery, developers write code with AI support, using outcomes, epics, stories and optional chapters. In AI-native delivery, the model builds and developers direct and verify it. The tracked unit is an outcome ticket at epic level, containing the outcome, its planned value contribution, user journeys and test requirements.\n\nMake the mode a standing choice for each team, based on its work and ability to review and verify the result. Different teams can use different modes within one operating model. A move between them needs practice and evidence; it is not a requirement for every team.", "links": [ { "label": "Explore the AI-native delivery workflow", "href": "/guides/ai-native-sdlc-and-product-delivery" } ] }, { "heading": "Agree what agents may do and what people decide", "body": "Assess decisions by their consequence, observability and reversibility. Frequent tasks that are easy to check and undo may be candidates for bounded automation. Consequential, contested or hard-to-reverse decisions stay with people. Agree the permissions, evidence, approval requirements and escalation route with the relevant risk and operational colleagues, and name the accountable person before deployment.\n\nFor example, an agent could gather approved records and draft a reply to a supplier enquiry. The service team checks whether the records support the answer and decides which replies need approval. Try the awkward cases with them: missing records, conflicting information and a request outside the agreed scope. Their feedback should change the design before it is used more widely." }, { "heading": "Keep a shared basis for approving and reviewing work", "body": "Both product and engineering approve an epic before it leaves Ready for Dev. AI-augmented teams also need at least one product-approved story; AI-native teams put the user journeys and test requirements on the outcome ticket. The AI-native build is complete when those tests pass and the named journeys are demonstrated working, with people verifying the result.\n\nEach delivery roadmap covers one quarter and reserves capacity for tech debt and bugs. Retrospectives run every two weeks; health checks and outcome validation run monthly. Assign the review responsibilities and use what the team learns to adjust the work. Measuring business value can continue beyond the quarter in which the work ships." }, { "heading": "Design the organisation and architecture together", "body": "The operating-model lead and agentic architect work with your product, engineering, operational and risk colleagues. For each proposed change, check the platform, data, integration and access it needs, the person responsible and the support required in everyday use. Those findings should inform both designs.\n\nThe Agentic Design Team produces the target operating model, technical architecture, governance framework and a sequenced build plan. Design is a distinct engagement: your own teams, Tenhaw or another supplier can carry out the subsequent build.", "links": [ { "label": "Scope and pricing for the Agentic Design Team", "href": "/services/agentic-design-team" }, { "label": "What enterprise AI asks of the organisation beyond the first workflow", "href": "/guides/end-to-end-agentic-workflow-implementation#enterprise-ai" } ] }, { "heading": "Let the team's experience shape the next version", "body": "Give adoption a named owner and review it monthly alongside delivery and value. Watch people use the workflow, invite them to question it and check what has become easier or needs more support. Pairing, practice and feedback help people develop the skills to operate and improve the system. Your people function retains responsibility for workforce consultation, redeployment and employment decisions.\n\nOn the specialty-insurance engagement, one client engineer paired through a two-week proof of concept and reported 70% confidence in running the process independently afterwards. That was one person's self-assessment at the end of a prototype build. It is useful feedback to pair with observed use, quality and support needs when planning the next stage.", "links": [ { "label": "How the agentic way of working reached Product, Engineering and QA", "href": "/case-studies/agentic-ai-in-insurance-operating-model" }, { "label": "The working proof of concept behind the 70% figure", "href": "/case-studies/specialty-insurance-agentic-lead" } ] } ], "engageVia": { "slug": "agentic-design-team", "label": "Agentic Design Team" }, "faq": [ { "question": "What is a target operating model for an AI-native organisation?", "answer": "It describes how your organisation works when AI agents perform a meaningful share of the work: its structure, roles, decision rights, governance and ways of working. It makes clear what an agent may do, what a person must approve, who is accountable and how the result will be checked.\n\nFor a product team, that includes choosing how developers use AI, defining approval gates and agreeing the evidence needed before a change ships. The Tenhaw Way publishes the product and delivery core of our model in full, free for you to adopt with your team." }, { "question": "How is this different from a Big Four target operating model?", "answer": "Compare the proposed team, scope, evidence and implementation responsibilities. Big Four services vary by firm and engagement, so the useful comparison is with the proposal you are considering.\n\nTenhaw's Agentic Design Team is two senior practitioners: an operating-model lead and an agentic architect. They design the roles, decision rights, governance and technical architecture together with your people, and produce a sequenced build plan. The service covers design; implementation needs its own delivery plan and resources. You can read and try the published ways of working, The Tenhaw Way, before deciding whether to engage us." }, { "question": "Do we restructure before deploying agents, or after?", "answer": "Test a useful workflow before committing to a broad restructure. Establish the permissions, approval responsibilities and safeguards needed for that pilot, then use what your team learns to design the wider changes before scaling.\n\nA pilot might show that an agent can prepare a reliable first draft, while reviewers still need better access to supporting records. That points to a specific change in work and responsibilities. Map those changes with the people involved, check what the architecture can support and decide which roles or reporting lines, if any, need to change." }, { "question": "What actually changes for a team in an AI-native operating model?", "answer": "The team changes how it specifies, builds and verifies work. In AI-augmented mode, developers write code with AI assistance and work from stories, adding chapters only when a story needs smaller pieces during the build.\n\nIn AI-native mode, the developer directs a model from a complete outcome ticket at epic level. It carries the outcome, its planned currency contribution, key user journeys and test requirements. The developer verifies the change through passing tests, demonstrated journeys and human review.\n\nProduct and engineering approvals remain in both modes. The mode is a standing choice for the team, made with the people doing the work. Completing the build provides release evidence; outcome validation checks the business value afterwards." }, { "question": "Who is accountable when an agent gets something wrong?", "answer": "Name an accountable person with the authority to oversee the agent's work before deployment. The operating model should record the decisions that person owns, what the agent may do, which actions need human approval and where questions or incidents go.\n\nWork through a difficult case with your operations, engineering and risk colleagues. If the agent proposes an incorrect change, who can spot it, stop it and put it right? Give those people the access, records and time they need to carry that responsibility, and test the escalation path alongside the workflow." }, { "question": "What about the people side: roles, spans and layers, workforce transition?", "answer": "Start with the work people do and the decisions they need to make. Explore how AI changes that work with the affected teams, including the exceptions they handle and the expertise they want to develop. Use those findings to design roles, responsibilities and any changes to management spans or layers.\n\nTenhaw's design work includes accountability for agent decisions and a specification for the Head of AI role you may recruit into. Your organisation and people function retain ownership of workforce consultation, redeployment and employment decisions. Bring them into the design early so those responsibilities and the support people need are part of the plan." }, { "question": "How long does target operating model design take, and what does it cost?", "answer": "Tenhaw's Agentic Design Team typically runs for two to four months at £35,000–£55,000 per month, excluding VAT. The pair comprises an operating-model lead holding the interim Head of AI seat and an agentic architect, with partner oversight. Retainers are cancellable on 30 days' written notice either way.\n\nThe outputs are a target operating model, technical architecture, governance framework and a sequenced build plan that your teams, Tenhaw or another supplier can execute. Building the systems is a separate scope. You can also adopt The Tenhaw Way and its how-to guides for free, with your own people taking the work forward." }, { "question": "Why do target operating models fail to get implemented?", "answer": "Implementation becomes difficult when the design asks for capabilities the platform cannot support, leaves delivery responsibilities unclear or gives the people affected little chance to shape how it will work. AI introduces further questions about permissions, human approvals and how agent actions will be checked.\n\nDevelop the operating model and architecture together, then try the proposed ways of working with representative teams. Their feedback should change the design before a wider rollout. Give each next step an owner, capacity and a way to assess progress, so the handover provides a practical basis for implementation." }, { "question": "Will AI fix a weak operating model?", "answer": "AI can help with a task, but unclear priorities or decision rights still need people to resolve them. Faster drafting, for example, may add to an approval queue if nobody has the authority or time to review the output.\n\nTrace one piece of work with the people who handle it. Find where it waits, what gets reworked and which decisions are difficult to make. Then test whether AI, a change in responsibilities or a simpler workflow improves the result. A baseline gives you and your team a way to see what helped." }, { "question": "How do you move a team from AI-augmented to AI-native?", "answer": "Explore the change with the team through a real, bounded build. Practise writing a complete outcome ticket, directing the model, reviewing its changes and verifying the named user journeys. Use what happens to decide whether AI-native delivery suits that team's work and what support it needs.\n\nIf the team adopts the mode, make it a standing way of working, with clear responsibilities for writing, review and verification. Review the team's experience alongside observed performance and the help people still need.\n\nIn our specialty insurance proof of concept, one client engineer paired throughout the two-week build and reported 70% confidence in running the process unaided. That was one person's self-assessment at the end of the build; independent delivery and sustained adoption still need to be observed." }, { "question": "Does an AI target operating model replace the one we already have?", "answer": "It can evolve the operating model you already have. Start by identifying which decisions, responsibilities and workflows change when agents take on part of the work, and keep the arrangements that continue to serve your organisation well.\n\nBring the changes into one coherent model, including any transition arrangements. Different teams can use AI-augmented or AI-native delivery within it, with each holding a standing choice of mode. Shared governance should make ownership and dependencies clear while allowing those differences in how teams build." }, { "question": "Is redesigning the operating model just another reorg?", "answer": "It can include changes to reporting lines, but the design also covers decisions, responsibilities, governance and daily work. Sometimes clearer ownership or a better handoff will address the problem without a new organisation chart.\n\nFor example, if an agent prepares a customer response, the model should say who checks the evidence, who can approve sending it and who handles an exception. Walk through that process with the people who will use and support it. Their experience helps you decide which structural changes would improve the work." }, { "question": "Which functions have to be in the room for an AI-native operating model?", "answer": "Include the people who do the work, the people who support it and those with authority to approve changes. That usually brings together product, engineering or architecture, operations, risk and security, with finance helping define value and the people function owning workforce consultation and transition. An executive sponsor needs the authority to resolve decisions across those groups.\n\nGive affected teams a real role in designing and testing the workflow, with named ownership for adoption. Tenhaw contributes an operating-model lead and an agentic architect. James Rooney provides partner oversight on every engagement and leads audits personally." }, { "question": "How do you know if the problem is your operating model or the technology?", "answer": "Follow one workflow from request to completed result and examine both. Poor output quality, unreliable integrations or missing data point to technical work. Repeated waits for a decision, unclear ownership or approvals that nobody can give point to operating-model work. The two can interact, so check the evidence with the people handling the exceptions.\n\nAt Globelynx, delivery lead times fell by 60% within six months during work on planning, coordination, supplier arrangements and operational insight. The figure was measured with the client from its delivery data. It is our measurement of our own work, has not been independently audited and does not isolate the effect of any one change. It provides an example to investigate, not a forecast for your organisation." }, { "question": "Isn't it too early to redesign the organisation while AI changes so fast?", "answer": "You can define responsibilities and a way to review decisions while the technology develops. Clear accountability, approval requirements, escalation paths and value measures give teams a stable basis for testing new capabilities.\n\nTreat the work delegated to agents as something to revisit with evidence. A task being frequent or easy to reverse is a useful starting point, but you still need to test reliability, permissions, monitoring and recovery. Your teams and risk colleagues can agree when the evidence supports changing that boundary, without committing the whole organisation to one vendor's current features." }, { "question": "Does going AI-native mean fewer approval gates and lighter process?", "answer": "Both delivery modes retain product and engineering approval before an epic leaves Ready for Dev, and human review of the change. AI-augmented teams also need a product-approved story attached; AI-native teams put the user journeys and test requirements on the outcome ticket itself.\n\nThe quarterly roadmap includes Tech Debt and Bug Budget epics with allocated capacity. Retrospectives, health checks and outcome validation keep their published cadence. AI can help prepare work for those decisions, while the people responsible check the evidence. Passing tests and demonstrating journeys establish that the work functions; validating value after release establishes what it achieved." }, { "question": "How do you tell whether a new operating model has actually been adopted?", "answer": "Look at how people carry out the work and ask what they find useful or difficult. Can the team write a usable ticket, apply the approval gates, review an agent's changes and handle an exception? Does feedback lead to changes people can see? Review that evidence alongside releases, quality and the support the team still needs.\n\nKeep adoption and business value visible as separate questions. Outcome work carries currency targets and planned contributions from delivery epics; maintenance epics carry capacity budgets. Monthly outcome validation compares live results with the baseline and measurement plan, recording value realised, not realised or still too early to assess. A team can be using the method while a longer-term benefit is still being measured." }, { "question": "Does an operating model redesign cover the whole business or just delivery?", "answer": "Tenhaw's published model and delivery evidence centre on product and delivery, together with the functions whose decisions enable that work. The Tenhaw Way is free to adopt and is used on every Tenhaw engagement. Redesigning finance, HR and operations across an entire enterprise is a broader scope to agree explicitly.\n\nAt HSBC Global Payment Solutions, James Rooney co-led the design, piloting and refinement of an operating model intended for 500 teams. This was founder track record in a delivery programme, not an AI transformation. The published study records a model designed and tested for a planned 2026 rollout; it does not report a completed global rollout." } ], "citations": [ { "name": "DORA, State of AI-assisted Software Development 2025", "url": "https://dora.dev/research/2025/dora-report/", "note": "Research on how organisational conditions affect AI-assisted software development. It informs the discussion of working practices." } ] } ## Voice agents and conversation intelligence Source: https://tenhaw.com/guides/voice-agents-and-conversation-intelligence Last updated: 2026-11-05 { "slug": "voice-agents-and-conversation-intelligence", "name": "Voice agents and conversation intelligence", "metaDescription": "Choose recorded-call intelligence or a live voice agent. Explore privacy, latency, tool use and handoff, with the evidence for each route clearly scoped.", "tagline": "Choose recorded-call insight or a live agent, then design the right workflow.", "summary": "Conversation intelligence turns recorded calls into structured information for review and analysis. A live voice agent also has to listen, handle interruptions, use tools and hand over to a person. The two routes need different privacy, latency and operating designs. HSBC Voice Insights was a founder-led proof of concept; its administrative-hours figure is a projection, separate from the build-lab's shipped startup voice products.", "theme": "patterns", "evidenceBasis": "delivered", "evidenceStatement": "Both halves rest on real delivery. The HSBC AI Voice Insights work was a proof of concept, led by our founder; its 1.5M+ hour figure was projected rather than realised, and it was not taken to production rollout. The real-time agentic voice experience comes from products shipped through Velocity84, a separate venture of our founder's: startup-scale products without enterprise regulatory constraints.", "demandSignal": "Voice is the channel with the most data already collected and the least structure imposed on it. Most contact centres hold years of recordings captured for quality or regulatory purposes, and can still only answer what customers were calling about anecdotally. That makes conversation intelligence over the recorded estate the lower-risk entry point, because the data already exists and nothing is customer-facing. It also makes the first question a legal one rather than a technical one: recordings collected for one specified purpose cannot simply be reused for a purpose incompatible with it, which is the purpose limitation principle in Article 5(1)(b) of the UK GDPR and the reason these programmes stall in month four rather than month one.", "whyItStalls": [ { "heading": "The recordings exist and the consent position does not", "body": "Most organisations hold years of call recordings captured for quality or compliance purposes. Repurposing them for AI analysis is a different processing purpose, and programmes stall when that is discovered late. It is answerable, but it is a question for month one rather than month four, and it is the commonest reason a voice proof of concept sits finished, demonstrated and unshipped while a data-protection review it should have started with catches up." }, { "heading": "Latency budgets are discovered in production", "body": "Real-time voice is unforgiving. Transcription, reasoning, tool calls and synthesis all have to fit inside a window where a human would have started speaking. Architectures that test fine asynchronously fall apart the moment someone interrupts, and interruption handling is rarely designed up front." }, { "heading": "The handoff is treated as an edge case", "body": "It is the main event. What the human receives when the agent gives up (the context, the transcript, the reason) determines whether customers experience the agent as helpful or as an obstacle. Most implementations design the happy path thoroughly and the handoff barely at all." }, { "heading": "Sentiment is trusted more than it deserves", "body": "Sentiment scoring on real calls is noisy, culturally variable, and frequently wrong on exactly the calls that matter. Using it as a headline metric rather than as one weak signal among several is a reliable way to lose the confidence of the operational team that has to act on it." } ], "approach": [ { "heading": "Start with the recorded estate, not the live channel", "body": "Conversation intelligence over existing recordings is lower risk, uses data you already hold, and produces business value without touching a customer interaction. It also tells you how well the models handle your accents, jargon and line quality before anything is live. At HSBC this meant integrating with the existing contact-centre system and processing inbound handler calls rather than building anything customer-facing." }, { "heading": "Transcribe into a retrievable store, then classify for themes", "body": "The architecture we built transcribed calls and landed them in a vector database, then assessed each call for common themes, account issues and similar recurring categories. That ordering matters: a searchable store first means later analytical questions can be asked of the same corpus without reprocessing, rather than each new question requiring a new pipeline.", "links": [ { "label": "Retrieval and permission-aware access, in full", "href": "/guides/retrieval-rag-and-permission-aware-knowledge-access" } ] }, { "heading": "Settle the data-protection position in month one", "body": "Processing purpose, consent basis, retention and residency for voice data are established before the build, with your DPO involved as a co-author. On any regulated engagement this is a design constraint, not a compliance review at the end." }, { "heading": "Design the latency budget as an architecture requirement", "body": "Each stage (transcription, reasoning, tool calls, synthesis) gets an explicit budget, and the design is tested against interruption from the start. This is the single most transferable lesson from shipping consumer voice products, where users are far less forgiving than internal testers." }, { "heading": "Make the handoff a first-class deliverable", "body": "What the human receives, how the agent decides to escalate, and what the customer experiences at the boundary are specified and tested as carefully as the automation itself. The routing logic here is the same confidence-and-consequence pattern we use for document review." }, { "heading": "Aim at two outputs: the operational one and the analytical one", "body": "The HSBC work produced both, and they serve different people. Operationally, automated agent notes remove manual write-up from the handler's day. Analytically, thematic classification lets the business see what customers were actually calling about, which is a question most contact centres can only answer anecdotally. Programmes that pursue only the analytical output tend to struggle for operational buy-in, because nothing changes for the people whose calls are being analysed." }, { "heading": "Land it somewhere governed, with lineage", "body": "Summaries and themes are only useful if they land somewhere governed, with lineage back to the source conversation. Otherwise you have replaced one unsearchable estate with another, which we would consider a failed engagement." } ], "providerNotes": { "heading": "Where the cloud matters for voice", "body": "Speech recognition quality on your specific accents, jargon and line conditions varies meaningfully between providers and is worth testing rather than assuming, it is the one place we would insist on a bake-off against your own recordings. Real-time orchestration primitives and telephony integration also differ substantially. Data residency for voice is frequently the binding constraint in regulated environments and should be established before any provider selection." }, "engageVia": { "slug": "agentic-proof-of-concept", "label": "Agentic Proof of Concept" }, "faq": [ { "question": "What is conversation intelligence?", "answer": "Turning recorded voice (calls, meetings) into structured, searchable intelligence: summaries, recurring themes, entities, and compliance or risk signals. It is distinct from real-time voice agents, and it is usually the lower-risk starting point because the data already exists, nothing is customer-facing, and it tests how well models handle your actual accents, jargon and line quality before anything goes live." }, { "question": "Can we use our existing call recordings to train or run AI analysis?", "answer": "Often yes, but not automatically. Recordings captured for quality monitoring or regulatory purposes were collected under a specific processing purpose, and analysing them with AI is generally a different one. The consent basis, retention position and residency need establishing before the build rather than during it, because it is answerable, and programmes that leave it to month four lose months." }, { "question": "What makes real-time voice agents hard?", "answer": "Latency and interruption. Transcription, reasoning, tool calls and speech synthesis all have to complete inside the window where a human would have started speaking, and users interrupt constantly. Architectures that work asynchronously fail immediately under those conditions. The handoff to a human is the other hard part, and it is usually designed as an edge case when it is the main event." }, { "question": "Has Tenhaw delivered voice AI?", "answer": "Yes, on both halves of it. At HSBC Tenhaw's founder led an AI Voice Insights proof of concept that integrated with the contact-centre system, transcribed inbound handler calls into a vector database, classified each call for recurring themes, and drove automated agent notes and business intelligence on what customers were actually calling about, with 1.5M+ hours of annual manual administration projected rather than realised. Real-time agentic voice is shipped work at Velocity84, Tenhaw's build lab, which has produced more than twenty agentic products across voice, video, document reading, mobile and go-to-market, at startup rather than enterprise scale. Taking that into a regulated contact centre is scoped as a staged build with your telephony and compliance people in the room, and priced a stage at a time." }, { "question": "Should we start with call analytics or a live voice agent?", "answer": "Start with the recorded estate. Analysing calls you already hold is lower risk because the data already exists, nothing is customer-facing, and it produces business value without touching a live customer interaction. Tenhaw's founder began there at HSBC, integrating with the existing contact-centre system and processing inbound handler calls. It also tells you how well the models handle your real accents, jargon and line quality before anything goes live, which is knowledge you will need either way. Latency, interruption handling and the handoff to a human make a live voice agent a different problem, and one unforgiving of architectures that only worked asynchronously. Prove value on the recordings first, then decide whether the live channel is worth the harder build." }, { "question": "Is sentiment analysis on customer calls reliable?", "answer": "Not reliable enough to be a headline metric. Sentiment scoring on real calls is noisy, culturally variable, and frequently wrong on exactly the calls that matter most. Treat it as one weak signal among several rather than the number the programme reports upwards, because using it as the headline is a reliable way to lose the confidence of the operational team that has to act on it. Thematic classification, showing what customers were actually calling about, is usually the more dependable output, and it answers a question most contact centres can currently only answer anecdotally." }, { "question": "How should a voice agent hand off to a human?", "answer": "Design it as the main event, not an edge case, because the handoff decides whether customers experience the agent as helpful or as an obstacle. Three things need specifying: how the agent decides to escalate, what the human receives when it does (the context, the transcript, the reason the agent gave up), and what the customer experiences at the boundary. Each should be specified and tested as carefully as the automation itself, and the escalation decision works on the same confidence-and-consequence routing logic used for document review. Most implementations design the happy path thoroughly and the handoff barely at all, which is the wrong way round." }, { "question": "How do you choose a speech-to-text provider for a contact centre?", "answer": "Run a bake-off against your own recordings. It is the one place Tenhaw would insist on one. Speech recognition quality on your specific accents, jargon and line conditions varies meaningfully between providers, and it is worth testing rather than assuming from published benchmarks. For real-time work the orchestration primitives and telephony integration also differ substantially, so a transcription score alone does not settle the choice. In regulated environments, establish data residency for voice before any provider selection, because it is frequently the binding constraint and can rule providers out before quality is even measured." }, { "question": "How do you analyse years of call recordings with AI?", "answer": "Transcribe the calls into a retrievable store first, then classify each call for recurring themes, account issues and similar categories. The AI Voice Insights proof of concept Tenhaw's founder led at HSBC used exactly that architecture, integrating with the existing contact-centre system, processing inbound handler calls and landing transcripts in a vector database before thematic classification. The ordering matters, because with a searchable corpus in place, each new analytical question can be asked of the same store without reprocessing, rather than every question requiring a new pipeline. Outputs then need to land somewhere governed, with lineage back to the source conversation, or you have replaced one unsearchable estate with another. Settle the data-protection position in month one, before the build." }, { "question": "What are the benefits of AI call analytics for a contact centre?", "answer": "Two concrete outputs, serving different people. The AI Voice Insights proof of concept Tenhaw's founder led at HSBC produced both, and was projected to remove more than 1.5 million hours of manual administration a year, a projection rather than a realised figure. The operational output is automated agent notes, which remove the manual write-up from a call handler's day. The analytical one is thematic classification, a structured view of what customers were actually calling about, which most contact centres can only answer anecdotally. Aim at both, because programmes that pursue only the analytical output tend to struggle for operational buy-in; nothing changes for the people whose calls are being analysed." }, { "question": "Do call recordings have to stay in the UK to be analysed?", "answer": "They can, and on regulated work they usually should. Tenhaw's default is UK data residency with the EU available, and the pipeline runs on your infrastructure under your policies, not Tenhaw's. Residency for voice is frequently the binding constraint, so it is established before any provider is selected rather than after, because it can rule an otherwise strong speech provider out before quality is even measured. It also sits alongside processing purpose, consent basis and retention as a position settled with your DPO before the build starts, since recordings of customers are personal data and moving them is a decision rather than a deployment detail." }, { "question": "Will an off-the-shelf call analytics tool do, or do we need a build?", "answer": "Often it will. A packaged tool answers packaged questions perfectly well, and if yours are generic, Tenhaw will say so. The build case turns on three things. Whether the categories you need are your own recurring themes and account issues rather than a fixed set someone else chose. Whether the summaries and themes have to land somewhere governed, with lineage back to the source conversation, because outputs nobody can trace have replaced one unsearchable estate with another. And whether the pipeline has to sit inside the contact-centre system you already run, which is how the AI Voice Insights proof of concept Tenhaw's founder led at HSBC was built." }, { "question": "How much does a conversation intelligence pilot cost?", "answer": "Between £20,000 and £55,000, fixed, over two to four weeks. That is Tenhaw's published Agentic Proof of Concept price, and voice work is engaged through it. For that money the scope is usually the recorded estate rather than a live agent: transcribe a slice of your calls into a retrievable store, classify them for recurring themes and account issues, and find out how the models cope with your real accents, jargon and line quality. It ends in something running rather than a slide pack, and it commits you to nothing afterwards. Settle the processing purpose for those recordings inside the same weeks, not later." }, { "question": "Who needs to be involved before we analyse call recordings?", "answer": "Your DPO, as a co-author rather than a reviewer at the end. Tenhaw settles purpose, consent basis, retention and residency with them inside the two to four weeks of an Agentic Proof of Concept, because Article 5(1)(b) of the UK GDPR limits further processing incompatible with the purpose recordings were collected for. Then whoever owns the contact-centre system the pipeline has to integrate with, since that integration is most of the engineering. Then the operational leader whose handlers are on the calls, because their people have to act on whatever comes back. Getting those three in early is what decides whether the work ships or stalls in month four." }, { "question": "Will call handlers see AI call analysis as surveillance?", "answer": "They will if the programme gives them nothing back, and the answer is in the design rather than the communications. The AI Voice Insights proof of concept Tenhaw's founder led at HSBC aimed at an operational output as well as an analytical one, so the same pipeline that tells the business what customers were actually calling about also produces automated agent notes that take the manual write-up out of the handler's day. Do the same, then keep sentiment scoring away from anybody's performance review, because it is noisy and culturally variable and people can tell when they are being measured by a number that is wrong about their hardest calls." }, { "question": "How do you set a latency budget for a voice agent?", "answer": "Give each stage an explicit number and treat the total as an architecture requirement rather than something discovered in production. Transcription, reasoning, tool calls and speech synthesis all have to finish inside the window where a human would already have started speaking, so writing the budget down early is what keeps the design honest once real systems are attached to it. Then test against interruption from the first week, because users interrupt constantly and designs that behave beautifully asynchronously fall apart the moment somebody talks over them. That is the most transferable lesson from shipping consumer voice products, where users are far less forgiving than internal testers." }, { "question": "Won't customers hate talking to an AI voice agent?", "answer": "Mostly they hate being trapped by one. Two design choices decide the experience. The first is interruption, because people talk over an agent constantly and an architecture that only ever worked asynchronously falls apart the moment they do, which makes the whole thing feel like a machine however good the voice sounds. The second is the boundary, the moment the agent gives up and a person takes over. Treat that as the main event and callers experience the agent as helpful. Most of the effort goes into the happy path and almost none into the boundary, which is why so many callers come away feeling obstructed." }, { "question": "Can a voice agent actually do things, or only answer questions?", "answer": "Taking an action is part of the definition. A real-time voice agent holds a live conversation, does something in a system and hands off to a human, so answering questions is only the first part of it. Time is the constraint rather than capability. The tool call has to complete inside the pause a caller will tolerate, which means a slow back-end system quietly decides what the agent can finish while someone is still on the line. Work that out in the architecture instead of discovering it in production, and specify where the agent stops and a person picks up." } ], "citations": [ { "name": "UK GDPR, Article 5(1)(b): purpose limitation", "url": "https://www.legislation.gov.uk/eur/2016/679/article/5", "note": "Personal data must be collected for specified, explicit and legitimate purposes and not further processed in a manner incompatible with those purposes. This is the provision behind the month-one question above. Whether your recordings clear it is a matter for your DPO and, where needed, counsel, not for us." } ] } ## End-to-end agentic workflow implementation Source: https://tenhaw.com/guides/end-to-end-agentic-workflow-implementation Last updated: 2026-11-05 { "slug": "end-to-end-agentic-workflow-implementation", "name": "End-to-end agentic workflow implementation", "metaTitle": "How to scale AI agents across an enterprise", "metaDescription": "Plan enterprise AI implementation from intake to authorised action and a recorded result. Connect engineering, operating ownership and human decisions.", "tagline": "Design the whole process, from arrival to a recorded result.", "summary": "Enterprise AI implementation connects the workflow, the engineering and the people who operate it. Trace intake, evidence gathering, a decision, an authorised action and the resulting record. Tenhaw applies components of this pattern on client work and is rebuilding its own operations with it. The scope map below distinguishes delivered components, internal work in progress and a proposed client programme.", "theme": "patterns", "evidenceBasis": "delivered", "evidenceStatement": "We are running this pattern on our own operations right now. Today our development pipeline is AI-engineering-first, our social content pipeline is semi-automated, and we run research agents doing competitor monitoring and opportunity-gap analysis. Our go-to-market process is being built the same way. The stated goal is that humans are in the loop only where they add value an AI could not. On client work we have delivered components of this pattern, including confidence-scored validation of entity-resolution output with human routing.", "primer": { "id": "enterprise-ai", "navLabel": "What enterprise AI means", "heading": "What enterprise AI means, and why the enterprise half is the hard one", "lede": "Enterprise AI is artificial intelligence doing a real share of an organisation's work, inside its systems, its permissions and its audit trail, rather than a licence per employee sitting beside a process that has not changed. The capability question is largely settled: a competent team can show a model doing the difficult part of a process in a fortnight, and most enterprises have now seen that demonstration. What the phrase is actually pointing at is everything around the fortnight. Who the agent is when it acts, what it is allowed to reach, what evidence it leaves behind, who answers for the decision it took, and whether the next agent inherits any of that or starts from nothing. Those are the parts that decide whether an organisation ends up with one impressive workflow or a different way of working.", "points": [ { "heading": "The unit is a process, not a tool rollout", "body": "Enterprise AI bought by the seat gets measured by the seat, which is how a programme reports strong adoption and unchanged cycle times in the same pack. Assistance makes each step faster, and in most consequential processes the steps were never the majority of the elapsed time: the handoffs, the approvals and the queues were. The version that moves a number leadership already watches is scoped as a complete process, intake through decision through action through record, with one accountable owner and an elapsed-time baseline taken before anything is built. That is a different conversation from a tooling one, and usually a different sponsor." }, { "heading": "The enterprise constraints are the design, not the paperwork", "body": "What makes an AI system enterprise-grade shows up as engineering rather than as policy. The agent needs an identity of its own rather than a developer's credentials, so what it did can be reconstructed months later. Retrieval has to answer from what the person asking is permitted to see, which is a property of the index rather than a filter bolted on at the end. The evidence a governance function will ask for has to fall out of the run rather than be assembled from logs a quarter afterwards. Left until after the pilot, most of those are not hardening: they are a rebuild of the thing you just proved.", "links": [ { "label": "Retrieval that respects the asking user's permissions", "href": "/guides/retrieval-rag-and-permission-aware-knowledge-access" }, { "label": "Why an agent needs an identity of its own", "href": "/guides/agent-identity-and-access" }, { "label": "The evidence a governance function will ask for", "href": "/guides/ai-governance-and-regulatory-evidence" } ] }, { "heading": "The economics live in the exception path and the run cost", "body": "The happy path is typically 70% of volume and 20% of effort, so an enterprise AI case built on it is a case built on the cheap fifth of the work. Route by confidence and consequence, staff the human path deliberately rather than as overflow, and price what the system costs to run as well as to build: inference and retrieval per run, the review the routing actually implies, monitoring, and a re-evaluation every time a model version changes underneath you. A case with a build cost and no run cost has measured its payback against half the bill.", "links": [ { "label": "What a defensible business case holds, and what finance should refuse", "href": "/guides/business-case-for-an-agentic-programme" } ] }, { "heading": "Enterprise scale is a property of the second agent, not the first", "body": "The question the phrase is usually carrying is how this gets from one workflow to twenty, and the answer is not a larger programme. It is that the parts every agent needs, identity and scoped permissions, permission-aware retrieval, an evaluation harness, observability, a deployment path and an agreed boundary with your second line, are pulled out of the first two processes and inherited by everything after them. Built up front, that is a platform programme with nothing a board can see. Rebuilt per agent, every security approval is paid for again and the portfolio stalls at the point where nobody can justify the next business case.", "links": [ { "label": "The six moves, in the order we would take them", "href": "/guides/end-to-end-agentic-workflow-implementation#approach" } ] }, { "heading": "What it looks like in a firm running it on itself", "body": "Our own development pipeline is AI-engineering-first, our social content pipeline is semi-automated, and research agents run competitor monitoring and opportunity-gap analysis while the go-to-market process is rebuilt the same way. The shared layer underneath that is an open-source engineering handbook of 72 rules with stable identifiers and RFC 2119 severities, enforced by an agent rather than remembered by a person, which is what makes the next build cheaper than the last one. The design principle is the one we would hold a client to: humans in the loop where they add value an AI could not." } ] }, "demandSignal": "The gap that matters is not between organisations using AI and organisations not using it. It is between assisting the steps of a process and changing the process, and the evidence that the first does not become the second on its own is now reasonably direct. The DORA programme's 2025 research on AI-assisted software development found that AI amplifies an organisation's existing strengths and weaknesses rather than delivering a uniform uplift, which is the same finding in a different domain: the tooling does not change the process, and the process is what elapsed time is made of.", "whyItStalls": [ { "heading": "Every step has an assistant and the process is exactly as slow as it was", "body": "Every step gets faster and the end-to-end cycle time barely moves, because the waits between steps (handoffs, approvals, queues) were always the majority of elapsed time. This is the most common disappointment in enterprise AI, and it is a process design problem rather than a model one. It is also uncomfortable to report upwards, because the tooling did what it promised and the number the board was given has not moved." }, { "heading": "Every agent is a one-off, so the tenth costs what the first did", "body": "Organisations that set out to scale AI agents across the enterprise usually find there is no substrate underneath: no shared identity model, no evaluation harness, no permission-aware retrieval, no common observability or deployment path. Each new agent therefore repeats the entire cost of the first, including every security and risk approval, and the programme stalls not because any single agent failed but because nobody can justify the twentieth business case. That substrate is not a platform project to be completed first, which is its own way of never shipping. It is the small set of things you deliberately generalise out of the first two processes, on the way through." }, { "heading": "The process nobody owns end to end", "body": "Most consequential business processes cross three or four functions, each owning a segment. Taking the process agentic requires someone with authority over the whole, and if that person does not exist the programme optimises segments and stops.", "links": [ { "label": "Designing the target operating model that makes the fifty-first rollout cheap", "href": "/guides/target-operating-model-for-an-ai-native-organisation" } ] }, { "heading": "Exceptions were never designed", "body": "The happy path is 70% of volume and 20% of effort. Agentic implementations that do not design the exception path early hit a wall where the automated portion is done and the remaining work is harder than before, because the easy cases that used to give staff context have gone." }, { "heading": "Governance was designed for human decisions", "body": "Existing approval thresholds, four-eyes checks and audit expectations assume a human at each point. Running agents through them either creates a bottleneck that removes the benefit, or gets bypassed, which is worse. The controls need redesigning for the new decision profile rather than inheriting." } ], "approach": [ { "heading": "Pick a process, not a use case", "body": "We scope by complete process with a measurable business outcome and an accountable owner, rather than by task. If no single person has authority over the whole process, establishing that is the first piece of work and we will say so before contracting." }, { "heading": "Map the waits, not just the work", "body": "The elapsed-time analysis usually shows the handoffs and approvals dominate. That determines where agents actually create value, and it frequently redirects the programme away from the step everyone assumed was the bottleneck." }, { "heading": "Design the exception path in the first fortnight", "body": "Routing by confidence and consequence, with the human path specified and resourced from the start. The exception path is the part that decides whether the economics work, and designing it late is the most expensive sequencing error in this pattern." }, { "heading": "Rebuild the controls for the new decision profile", "body": "Approval thresholds, four-eyes requirements and audit evidence are redesigned with your second line as co-authors, based on which decisions are consequential and which are cheaply reversible, not inherited from a process that assumed a human at every point." }, { "heading": "Ship a slice end to end before widening", "body": "One complete path through the process, in production, month by month, rather than every step automated to 80% and nothing finished. A narrow slice that runs end to end teaches you more than broad partial coverage, and it is something the board can see." }, { "heading": "Generalise on the second process, not the first", "body": "The pieces every agent will need (identity and scoped permissions, permission-aware retrieval, an evaluation harness, observability, a deployment path, an agreed human-approval boundary) are pulled out into shared components while doing the second process, once you have two real examples to generalise from. Building that platform before the first process is how organisations spend a year shipping nothing, and rebuilding it per agent is how they stall at three. This is the step that decides whether the programme can scale beyond the processes you personally sponsor." } ], "technology": { "heading": "The platform we build on, and where we hold a view", "lede": "Named, because a supplier who will not name the stack is asking you to take the architecture on trust. Each entry carries the same two labels as the guide itself. Delivered means we have shipped on it and you can ask us what broke. Our approach means we hold a position.", "items": [ { "name": "Microsoft Azure and Azure OpenAI", "basis": "delivered", "what": "Microsoft's cloud, and its hosted access to frontier models inside a tenancy you control.", "position": "Every agentic system we have delivered runs here. The two-week document pipeline on a specialty insurance engagement was built from a blank repository on Azure with Claude and Codex, and a separate proof of concept on Azure OpenAI validated and scored entity-resolution output so human review could be routed rather than queued. If your estate is Azure we are repeating a delivery; anywhere else we are applying a method. What we have delivered here is proofs of concept, not a production system." }, { "name": "Amazon Bedrock and Amazon SageMaker", "basis": "approach", "what": "AWS's managed access to foundation models, and its platform for training and serving models.", "position": "The shape of every pattern in this guide holds across clouds: what genuinely changes is the document-understanding service, how identity federates from your source systems, and where the index and the embeddings physically sit. On an AWS estate we would expect to work alongside your own platform engineers rather than lead the platform work, and we would price and scope it that way rather than learn it on your budget." }, { "name": "Google Vertex AI", "basis": "approach", "what": "Google Cloud's managed platform for foundation models and machine learning.", "position": "Same position as AWS. The method transfers, the service choices do not, and anyone telling you a cloud migration is incidental to an agentic programme has not done one." }, { "name": "Kubernetes", "basis": "approach", "what": "The container orchestration platform most large estates already run something on.", "position": "Kubernetes work here runs with your own platform engineers. The position is a sequencing one: do not make your first agentic workflow depend on standing one up. Where you already run a cluster, an agent is another workload on it and that is the cheap answer. Where you do not, a managed runtime gets the first workflow to production sooner, and the platform decision is better taken once you have two real workloads to generalise from rather than nought." }, { "name": "Terraform", "basis": "approach", "what": "Infrastructure defined as code and applied from version control.", "position": "The position is narrower than the tool. An agent's identity, its scoped permissions and its tool list belong in version control beside the rest of the infrastructure definition and reviewed the same way, because a permission granted through a console at four on a Friday is the one nobody can account for at the audit. Whether that is Terraform, Bicep or something else is your platform team's call and not ours." }, { "name": "Apache Airflow", "basis": "approach", "what": "Scheduled orchestration of data pipelines as directed graphs.", "position": "Worth naming because of the distinction it forces. A scheduled graph is the right tool for the deterministic parts of a workflow, ingest, extraction, enrichment, load, and an agent trajectory is the part where the next step is chosen at run time. Most end-to-end agentic workflows are mostly the first thing with the second thing inside them, and programmes that model everything as an agent trajectory pay for non-determinism they never needed, in evaluation cost and in explaining themselves afterwards." }, { "name": "Git, markdown and a model at maximum reasoning", "basis": "delivered", "what": "The build method itself, published in full including the parts that go wrong.", "position": "Every requirement becomes structured markdown held in Git, a model builds a knowledge map across the whole corpus, and a gap-and-contradiction pass runs before a line of code exists. Only then does the build start, from a blank repository, prompted against the full requirement set with a security review roughly every fifth prompt and pair-programmed throughout. Development starts by pasting an outcome-shaped epic into Claude or Codex in the editor. That is delivered: it produced the two-week proof of concept, and the client engineer who paired on the whole fortnight finished it 70% confident they could run the process without us." }, { "name": "GitHub and Azure DevOps", "basis": "delivered", "what": "Where the work is tracked and where the code lives.", "position": "On the specialty insurance engagement we led the migration from Azure DevOps boards to a GitHub-based delivery model with agent-assisted workflows. Named because the delivery substrate decides how much of the agent-assisted workflow is available to you at all, and because it is the least glamorous dependency in an agentic programme and one of the more common blockers." } ] }, "engageVia": { "slug": "agentic-build-team", "label": "Agentic Build Team" }, "faq": [ { "question": "What technology stack does Tenhaw build agentic systems on?", "answer": "Microsoft Azure, with Claude, Codex and Azure OpenAI. On the PPL engagement the document pipeline was built from a blank repository on Azure in two weeks as a working proof of concept, and a separate proof of concept on Azure OpenAI scored and validated entity-resolution output. The build method is Git and markdown rather than a framework. Every requirement becomes structured markdown, a model interrogates the corpus for gaps and contradictions before any code exists, then the build runs against the full requirement set at maximum reasoning, pair-programmed with your engineers. Delivery runs through GitHub, and we led that engagement's migration from Azure DevOps boards. Kubernetes, Terraform and Airflow we take with your platform engineers and price into scope." }, { "question": "Do we need Kubernetes, Terraform or Airflow to run agents?", "answer": "Not for the first workflow, and the right ordering saves a quarter. If you already run a Kubernetes cluster, an agent is another workload on it. If you do not, standing one up puts a platform programme in front of what you were trying to prove, and a managed runtime reaches production sooner. Terraform, or whatever your platform team already uses, matters for a narrower reason than hosting. An agent's identity, its scoped permissions and its tool list belong in version control, reviewed as code, because a permission granted through a console is the one nobody can account for later. Airflow, or any scheduler, keeps the deterministic majority of a workflow deterministic. All three we take alongside your platform engineers and price into scope." }, { "question": "What is an end-to-end agentic workflow?", "answer": "A complete business process (intake, decision, action and record) where agents perform the work and humans govern it, rather than each step being assisted while the overall shape stays the same. The distinction matters because assisting individual steps typically leaves end-to-end cycle time almost unchanged, since the waits between steps were always the majority of elapsed time." }, { "question": "What is enterprise AI?", "answer": "Enterprise AI is artificial intelligence doing a real share of an organisation's work, inside its systems, its permissions and its audit trail, rather than a licence per employee alongside a process that has not changed. In practice the phrase describes four commitments rather than a technology: the agent acts under an identity of its own, so what it did can be reconstructed later; retrieval answers only from what the person asking is permitted to see; the evidence a governance function will want falls out of the run rather than being assembled afterwards; and the decisions agents may take are bounded by consequence and reversibility, agreed with your second line. Enterprise AI that scales adds a fifth: the identity model, the retrieval, the evaluation harness, the observability and the deployment path are shared between agents rather than rebuilt for each one." }, { "question": "How is enterprise AI different from the AI tools our staff already use?", "answer": "Assistants change how a step is done, and enterprise AI changes what the process is. That difference shows up in the number the board is watching. Making every step faster compresses the minority of a cycle time that was ever work, because the majority of elapsed time in most consequential processes is waiting: for a handoff, an approval, a queue, someone's availability. It also shows up in what has to exist around the system. A tool used by a person inherits that person's access, judgement and accountability, and an agent doing the work inherits none of them, so identity, permission-aware retrieval, audit evidence and an agreed human-approval boundary stop being good practice and become the design. Licences are a procurement decision. Enterprise AI is an operating-model one." }, { "question": "Has Tenhaw taken an enterprise process fully agentic?", "answer": "We are partway through doing it to our own operations: the development pipeline is AI-engineering-first, the social content pipeline is semi-automated, research agents run competitor monitoring and opportunity-gap analysis, and the go-to-market process is being rebuilt the same way. On client work we have delivered components of the pattern, including a two-week proof of concept taking PDFs through to business intelligence on Azure for Placing Platform Limited (PPL), and confidence-scored validation of entity-resolution output with human routing. Everything we have built agentically runs on Microsoft Azure, using Claude, Codex and Azure OpenAI. Taking a complete enterprise process end to end is exactly what our build teams are scoped for." }, { "question": "How do you scale AI agents across an enterprise?", "answer": "Deep before broad, generalising on the way through. Take one complete process, with an accountable owner and a measurable outcome, right through to production. A narrow slice that runs end to end teaches more than twenty steps at 80%. Then make the parts every agent needs shared, not rebuilt: identity and scoped permissions, permission-aware retrieval, an evaluation harness, observability, a deployment path, and an agreed boundary with your second line on which decisions need a human. Tenhaw's shared layer is an open-source engineering handbook of 72 rules with stable identifiers and RFC 2119 severities, enforced by an agent rather than remembered by a human. That substrate is why the fifth agent costs less than the first, and its absence is why portfolios stall at three." }, { "question": "Why doesn't AI assistance reduce our cycle times?", "answer": "Because the work was rarely the bottleneck. In most consequential processes the majority of elapsed time is waiting, for a handoff, an approval, a queue, someone's availability. Making each step faster compresses the minority of the timeline. Reducing cycle time requires removing the waits, which is process and operating-model change rather than tooling." }, { "question": "Where do end-to-end agentic implementations usually fail?", "answer": "Five places. Nobody owns the process end to end, so it optimises by segment and stalls at the functional boundary. The exception path was not designed, so the automated portion finishes and the residue is harder than the original job. Governance designed for human decisions either bottlenecks the agents or gets bypassed. Programmes go broad rather than deep, automating every step to 80% and finishing nothing. And nothing is generalised between agents, so every one costs what the first one did and the programme stalls at the point where the next business case cannot be justified." }, { "question": "How do you decide which decisions agents can take?", "answer": "By consequence and reversibility rather than by complexity. Agents take decisions that are high-volume, observable and cheaply reversible; humans retain decisions that are consequential, contested or hard to undo. That boundary is written down per decision class, the escalation path across it is designed, and your second-line risk function co-authors it rather than reviewing it afterwards." }, { "question": "How do you handle exceptions in an agentic workflow?", "answer": "Design the exception path in the first fortnight, because it decides whether the economics work. The happy path is typically 70% of volume and only 20% of effort, so an implementation that only handles the easy cases hits a wall once the automated portion is done and the remaining work is harder than before, because the routine cases that used to give staff context have gone. Route by confidence and consequence, specify the human path from the start, and resource it properly rather than treating it as overflow. Designing it late is the most expensive sequencing error in this kind of work." }, { "question": "Do our existing approval processes work for AI agents?", "answer": "Usually not as they stand, because they assume a human at every decision point. Run agents through approval thresholds and four-eyes checks designed for people and one of two things happens. Either the controls become a bottleneck that removes the benefit of automating, or they get quietly bypassed, which is worse. The fix is to redesign the controls for the new decision profile, with your second-line risk function as co-authors rather than after-the-fact reviewers, based on which decisions are consequential and which are cheaply reversible, instead of inheriting a control set built for a process that no longer exists." }, { "question": "Should we build an AI platform before deploying our first agents?", "answer": "No, and building it first is its own way of never shipping. A platform stood up before any real process runs on it can absorb a year of engineering with nothing a board can see, and its requirements are guesses until real agents exist. Tenhaw's two-week proof of concept for Placing Platform Limited (PPL) took PDFs through to business intelligence with no platform built first. The opposite failure is as common. With nothing shared between agents, the tenth costs what the first did, every security approval is repeated, and nobody can justify the twentieth business case. So ship the first process end to end, then pull the shared pieces out while delivering the second, once there are two real examples to generalise from." }, { "question": "Which process should we take agentic first?", "answer": "Pick a complete process rather than a use case, one with a measurable business outcome, an accountable owner, and enough size that improving it moves a number leadership already watches. Map the elapsed time before committing. In most consequential processes the majority of that time is handoffs, approvals and queues rather than the steps themselves, and the analysis regularly redirects a programme away from the step everyone assumed was the bottleneck. Choose from that picture rather than from a use-case list, and one you can take all the way through intake, decision, action and record rather than four steps out of six. Tenhaw's Globelynx engagement cut lead time by 60% inside six months, the kind of number a first process should move." }, { "question": "Nobody owns our process end to end. Can we still start?", "answer": "Yes, but establishing that ownership is the first piece of work, and Tenhaw says so before contracting rather than halfway through a build. Most consequential processes cross three or four functions, each owning a segment, so taking one agentic needs somebody with authority over the whole. Without that the programme optimises segments, stalls at the first functional boundary, and leaves faster steps inside an unchanged cycle time. Sometimes the answer is a sponsor with real decision rights across those functions, sometimes it is genuine operating model work. Tenhaw's founder co-designed one of those at HSBC Global Payment Solutions, covering 500 teams and a $450M portfolio, piloted rather than rolled out. Either way it is cheaper to settle at the start than once agents are live." }, { "question": "What happens to the team that runs the process today?", "answer": "Their work changes shape rather than disappearing. In an agentic process people govern the work rather than perform it, taking the cases that need judgement, owning the controls the agents run inside, and deciding what happens when something falls outside them. On a PPL insurance proof of concept Tenhaw paired throughout with one of the client's engineers, who reported 70% confidence that they could run the process unaided. Plan for a harder job than the old one, because the routine cases that used to build context and judgement are the ones agents take first. So staff the exception path deliberately rather than as overflow, and keep the people who work exceptions today inside the design, since they know which edge cases genuinely occur." }, { "question": "We are on AWS, not Azure. Can you still deliver this?", "answer": "Yes, with one honest change in who leads the platform work. Everything Tenhaw has delivered agentically runs on Microsoft Azure, using Claude, Codex and Azure OpenAI, and nothing has shipped on Amazon Bedrock, Amazon SageMaker or Google Vertex AI. The shape of the pattern holds across clouds. What genuinely changes is the document-understanding service, how identity federates from your source systems, and where the index and the embeddings physically sit. On an AWS estate Tenhaw would work alongside your own platform engineers instead of leading the platform work, bringing a method it publishes in full and free for anyone to adopt rather than a cloud it is learning on your budget." }, { "question": "Should the whole workflow be an agent, or only part of it?", "answer": "Only part, in most cases. Most end-to-end agentic workflows are mostly a deterministic pipeline (ingest, extraction, enrichment, load) with an agent trajectory inside it, and the agent belongs where the next step is genuinely chosen at run time rather than known in advance. Modelling the deterministic parts as agent decisions buys non-determinism you never needed, and you pay for it twice, once in evaluation because every trajectory has to be tested, and again when somebody asks why a particular case went the way it did. A scheduled graph is dull, cheap to explain and easy to audit. Keep the judgement in the agent and leave the plumbing to the pipeline." }, { "question": "How do we prove the cycle time actually came down?", "answer": "Baseline the elapsed time before anything is built, end to end rather than step by step, using the map that shows where the handoffs, approvals and queues sit. The map is the measurement instrument as well as the design input, because once you know how long a case waited at each boundary before, you can show what the rebuilt process does to the same boundaries rather than argue about how much faster individual steps feel. Then report against it every month. Tenhaw holds itself to that on its own engagements, where a month that delivers no measurable value is reported as a failed month rather than smoothed into the next one." }, { "question": "Do we need a fresh risk sign-off for every agent we deploy?", "answer": "Only for what is genuinely new about it, provided the parts every agent needs are shared rather than rebuilt. Agent identity and scoped permissions, permission-aware retrieval, an evaluation harness, observability and the deployment path are properties of the platform, approved once and then inherited, and they are best pulled out of your first two processes rather than built up front. Hold a standing agreement with your second line about which classes of decision need a human, so that boundary is not renegotiated each time either. What is left to assess for any new agent is the decisions it takes, the data it touches and the tools it can call." }, { "question": "Do you run agentic workflows inside your own business?", "answer": "Yes, and it is where much of this method was proved. Tenhaw's own development pipeline is AI-engineering-first, its social content pipeline is semi-automated, research agents run competitor monitoring and opportunity-gap analysis, and its go-to-market process is being rebuilt the same way. Running the pattern on itself is why Tenhaw can be specific about what breaks rather than about what demonstrates well. The same components are delivered on client work, including confidence-scored validation of entity-resolution output with human routing on a specialty insurance engagement, and Velocity84, Tenhaw's build lab, has produced more than twenty agentic products across voice, video, document reading, mobile and go-to-market. The design principle throughout is humans in the loop where they add value an AI could not." } ], "citations": [ { "name": "DORA, State of AI-assisted Software Development 2025", "url": "https://dora.dev/research/2025/dora-report/", "note": "Source of the amplifier finding quoted above. Its subject is software delivery rather than business process generally, which is worth saying, because we are reading it across." }, { "name": "Budzier and Flyvbjerg, Overspend? Late? Failure? What the Data Say About IT Project Risk in the Public Sector (2013)", "url": "https://arxiv.org/abs/1304.4525", "note": "1,355 public-sector IT projects, average budget $130m over 35 months. Each additional year of duration added about 4.2 percentage points to average cost risk, and 18% of projects were outliers with cost overruns above 25%. The evidence behind shipping a narrow slice rather than a two-year programme." } ] } ## Retrieval, RAG and permission-aware knowledge access Source: https://tenhaw.com/guides/retrieval-rag-and-permission-aware-knowledge-access Last updated: 2026-11-05 { "slug": "retrieval-rag-and-permission-aware-knowledge-access", "metaDescription": "Design enterprise RAG around useful sources and caller permissions. Test relevance, stale grants and conflicting guidance before relying on the answer.", "name": "Retrieval, RAG and permission-aware knowledge access", "metaTitle": "Retrieval, RAG and permission-aware access", "tagline": "Answer from useful sources while keeping access tied to the person asking.", "summary": "Enterprise RAG retrieves relevant passages at question time and uses them to support an answer. Its source filters must respect the caller's permissions, and its answers should expose the supporting evidence. Test missing access, stale grants and conflicting policies alongside relevance. PPL provides evidence of delivered semantic retrieval components; the wider assistant design also needs its own permission and operating tests.", "theme": "engineering", "evidenceBasis": "approach", "evidenceStatement": "We have built retrieval into delivered work. On an engagement with Placing Platform Limited (PPL) we built the semantic layer over extracted document fields, with provenance recorded per field and confidence scored from where each value came from. On the HSBC voice proof of concept our founder led, call transcripts were landed in a vector database and queried for recurring themes. The PPL build is a working proof of concept inside the client's estate. The parts of this guide about permission propagation, index lifecycle and adversarial content are the method we would bring to a permission-aware retrieval estate serving thousands of users across systems that each have their own access model.", "evidenceNextStep": { "heading": "So what should you do about it", "body": "Do not buy a knowledge platform as the first move. The cheapest useful thing is a fortnight spent on the corpus and the permission model: which content is authoritative, who owns it, what its access rules actually are in the source system, and whether those rules can be read at query time. That answer decides most of the architecture and it does not need us. Where we are useful is building the first real retrieval path against it and leaving behind the evaluation set that says whether it works, because that is the artefact almost nobody has and the one that survives every subsequent change of model. If what you need is an enterprise search programme stood up rather than one workflow proved out, say so on the call and we will tell you whether we are the right firm for it." }, "demandSignal": "Retrieval is the oldest idea in this stack and the least well measured. The paper that named the pattern, published in 2020, argued that models storing factual knowledge in their parameters have a limited ability to access and precisely manipulate that knowledge, and that an explicit non-parametric memory addresses it. The engineering literature since is less romantic: a study of three production retrieval systems concluded that validating one is only feasible during operation and that robustness evolves rather than being designed in at the start. Both are arguments for building the measurement before the corpus.", "whyItStalls": [ { "heading": "The index was built with the ingestion account's privileges, so the assistant will answer anyone from anything", "body": "This is the most common serious failure in enterprise retrieval and it is nearly always found after the corpus has been indexed. A crawler runs as a service principal with broad read access, everything it can see goes into one index, and the assistant then answers any user from any document in it. The fix is not a filter added at the end. It is carrying the source system's access control into the index and applying it to the asking user at query time, which usually means rebuilding the index. Design it in, or budget for doing it twice." }, { "heading": "Nobody decided which document is the authoritative one", "body": "Enterprise corpora contain the policy, the superseded policy, three drafts of the next policy, and a slide that summarises all of them incorrectly. Retrieval will return the wrong one with complete composure, because relevance is not authority. Until someone owns the corpus and marks what is current, accuracy work is being done on the wrong problem, and every hour spent tuning chunk sizes is an hour not spent on the thing that is actually wrong." }, { "heading": "Chunking and placement decide the answer, and neither was measured", "body": "How a document is split, and where the retrieved passage sits in the prompt, change the answer materially. Published work on long inputs found that performance is highest when the relevant information appears at the beginning or the end of the context and degrades significantly when a model has to use something in the middle, including in models built for long contexts. A team treating chunking as a default setting is leaving a large and cheap accuracy gain unclaimed, and cannot explain its own failures when asked." }, { "heading": "There is one score, at the end, so nobody can tell a retrieval failure from a generation failure", "body": "If the only measurement is whether the final answer was good, every regression is a mystery. Did the search miss the document, did it return the document and the model ignore it, or did the model have everything it needed and answer badly. Those three have different fixes and different owners, and one end-to-end number cannot separate them. This is the cheapest thing most teams are not doing, and the reason their accuracy conversations go in circles." }, { "heading": "The index is a snapshot and the business is not", "body": "Documents get amended, reclassified and deleted, and people join teams and leave the organisation. Where the index is rebuilt on a schedule and permissions are copied at ingestion time, then for the whole gap between rebuilds the system is answering from a world that no longer exists, including for people whose access has been withdrawn. Nobody notices until an audit or an incident, and both are expensive ways to learn it." } ], "approach": [ { "heading": "Write the questions down before choosing a store", "body": "The first artefact is a list of real questions from the people who will ask them, with the answers they would accept and the documents those answers live in. Fifty of those settle almost every architectural argument, and the same list becomes the first evaluation set. Choosing a vector database before it exists is answering a question nobody has written down.", "links": [ { "label": "Retrieval, fine-tuning or prompting: how to choose", "href": "/guides/rag-fine-tuning-or-prompting" } ] }, { "heading": "Propagate source permissions to query time, and prove it with a test that is meant to fail", "body": "Retrieval respects the permissions of the asking user against the source system, not the privileges of whatever indexed the corpus. In practice that means carrying an access identifier onto every indexed item and filtering on the caller's group membership at search time, which is the security filter pattern the cloud search services document in their own guidance. The control is not the filter, it is the test: a named user who should not see a document asks the question that would surface it, and the pipeline fails if it comes back. That test belongs in the build from week one, not in a penetration test at the end.", "links": [ { "label": "Why the agent needs its own identity too", "href": "/guides/agent-identity-and-access" } ] }, { "heading": "Measure retrieval separately from generation", "body": "Two evaluation sets, scored separately. Retrieval is measured on whether the right passages came back at all, at the depth you actually pass to the model. Generation is measured on whether the answer is right given those passages. Keeping them apart turns a regression from an argument into a diagnosis, and we would build the retrieval set first: it is cheaper to label, it needs no model in the loop to score, and it survives you changing the model.", "links": [ { "label": "What an evaluation harness consists of", "href": "/guides/agent-evaluation-and-assurance" } ] }, { "heading": "Treat chunking, ordering and context budget as tuned parameters", "body": "Chunk boundaries follow the structure of the document rather than a character count, retrieved passages are ordered with the strongest at the edges of the context rather than buried in the middle of it, and the number of passages passed is a decision with a measured cost and a measured benefit rather than a default. All three are cheap experiments once the retrieval evaluation set exists, and guesswork before it does." }, { "heading": "Make every answer carry its source, and score confidence from provenance", "body": "An answer with a link back to the passage it came from is checkable by the person reading it, which changes the risk profile of the whole system more than any single accuracy improvement. On our insurance engagement, confidence came from the provenance of the data, which source each value was drawn from, combined with model certainty and an independent cross-check, rather than from the model's own score. The same principle holds here: where a passage came from is a stronger signal than how sure the model sounds." }, { "heading": "Design the index lifecycle before the first ingest", "body": "Re-index cadence, deletion at source, reclassification, and what happens to a leaver's access are design decisions rather than operational ones. The question to answer on day one is how long a document deleted this morning can still be answered from, and whether that number is acceptable to your data protection officer. If nobody can state the number, it is not a system yet." }, { "heading": "Treat retrieved content as untrusted input", "body": "Anything the system retrieves can carry instructions aimed at the model rather than at the reader. Indirect prompt injection was demonstrated against real deployed applications in 2023 by planting text in content the application would later fetch, and prompt injection is the first entry in the OWASP risk list for language model applications. In an enterprise corpus the injected text does not have to come from an attacker: a well-meant instruction in a document template will do it. So retrieved passages are handled as data, the tools the agent can reach are scoped so a successful injection cannot do much, and the evaluation set carries adversarial documents alongside the ordinary ones.", "links": [ { "label": "Scoping what an agent can actually do", "href": "/guides/mcp-tool-calling-and-system-integration" }, { "label": "Bounding what a wrong answer costs", "href": "/guides/guardrails-hallucination-and-accuracy-control" } ] } ], "providerNotes": { "heading": "Where the cloud matters for retrieval, and where it does not", "body": "The hard parts here are provider-independent: who owns the corpus, which document is authoritative, what the permission model is, and whether an evaluation set exists at all. The genuinely provider-specific decisions are whether document-level access control is native to the search service or something you build on top of it, how identity federates from the source repositories, and where the index and the embeddings physically sit. We have delivered retrieval components on Azure. On AWS and Google Cloud the shape is the same and the services differ." }, "engageVia": { "slug": "agentic-proof-of-concept", "label": "Agentic Proof of Concept" }, "faq": [ { "question": "What is retrieval-augmented generation?", "answer": "Retrieval-augmented generation, or RAG, is a design in which the system searches a body of content at question time and passes the retrieved passages to a language model, which answers from them rather than from what it learned in training. It exists because a model's parametric memory is fixed at training time, cannot be updated for your organisation, cannot be permission-checked, and cannot cite where an answer came from. Retrieval gives you all three: current content, access control at query time, and an answer with a source attached that the reader can open." }, { "question": "Do we need a vector database?", "answer": "Often not for the first workflow, and it is the last decision rather than the first. Plenty of enterprise questions are answered better by keyword search, by a hybrid of keyword and semantic search, or by a filter over structured metadata, and the store you already own may be enough to find out. The decision that matters is the corpus and the permission model. Choose the store after you have a retrieval evaluation set that can tell you whether swapping it changed anything, otherwise you are buying infrastructure on the strength of a demo." }, { "question": "How do we stop RAG answering from documents a user is not allowed to see?", "answer": "By carrying the source system's access control into the index and applying it to the asking user at query time, rather than indexing with a privileged crawler account and hoping. In practice that is an identity field on every indexed item, populated from the source system's access rules and filtered against the caller's group membership on every query, plus a test in the build pipeline where a user who should not see a document asks the question that would return it and the run fails if it does. Retrofitting this after indexing usually means rebuilding the index, which is why it belongs in the design rather than in hardening." }, { "question": "How do you measure whether retrieval is working?", "answer": "Separately from the answer. Build a set of real questions with the passages a qualified person says are needed to answer them, then measure whether those passages come back at the depth you actually pass to the model. That number is your retrieval score, and it can be improved without touching the model. Measure answer quality against the same questions with the correct passages supplied, and you have your generation score. When something regresses, the two scores tell you which half broke, which is the difference between a diagnosis and a fortnight of guessing." }, { "question": "Why does our RAG assistant give confident wrong answers?", "answer": "Usually one of three things, and they are distinguishable if you measure retrieval separately. The right passage was never retrieved, so the model answered from general knowledge and sounded fine doing it. The right passage was retrieved and ignored, which is often a placement problem, since published work found models use information at the beginning and end of a long context far better than information in the middle. Or the corpus genuinely contains the wrong answer, because the superseded policy is still in the index and nothing marks it as superseded. Add a fourth for completeness. The system has no way to say it does not know, so it produces something rather than nothing." }, { "question": "Does a bigger context window remove the need for retrieval?", "answer": "No, and treating it as though it does is an expensive mistake. Published work on long contexts found performance degrades significantly when the relevant information sits in the middle of the input, including in models built for long contexts, so more context is not the same as more attention. Beyond that, a context window does not solve any of the reasons a regulated organisation needs retrieval: permissions still have to be applied per user, content still has to be current, answers still have to cite a source, and every additional token has a price and a latency cost on every single call. Retrieval is what keeps the context small and defensible." }, { "question": "What does a retrieval system cost to run?", "answer": "Any number quoted before seeing your corpus is a guess, Tenhaw's included. The shape of the bill is consistent though, so you can build the estimate yourself: embedding and indexing at ingest, then re-embedding whenever the corpus or the embedding model changes; search per query; the tokens in the retrieved context on every call, usually the largest line and directly controlled by how many passages you pass; the model's output; human review of whatever is routed for it; and evaluation runs, which recur on every model upgrade. Tenhaw publishes its method in full and free to adopt, so this estimate is one you can build without hiring anyone, and it is the number that decides between designs and is almost always the one missing." }, { "question": "How should documents be chunked for a RAG system?", "answer": "Chunk on the document's own structure rather than a fixed character count, so a section or a table survives intact instead of being cut in half, and treat the size as something you tune rather than a default you inherited. Two related settings matter as much. One is where a retrieved passage sits in the prompt, and published work on long contexts found models use information at the beginning and the end far better than anything buried in the middle. The other is how many passages you pass, which has a measured cost as well as a measured benefit. All three are cheap experiments once a retrieval evaluation set exists, and guesswork before it does." }, { "question": "Is prompt injection a risk if all our documents are internal?", "answer": "Yes, and internal corpora are the awkward case, because the instruction does not have to come from an attacker to work. Indirect prompt injection was demonstrated against real deployed applications in 2023 by planting text in content the application would later fetch, and prompt injection is the first entry in the OWASP risk list for language model applications. In an enterprise corpus a well-meant instruction sitting in a document template does the same job by accident. So retrieved passages are handled as data rather than as instructions, the tools the agent can reach are scoped so a successful injection cannot do much, and the evaluation set carries adversarial documents alongside the ordinary ones." }, { "question": "How do you keep a RAG index up to date when documents change?", "answer": "Decide the lifecycle before the first ingest rather than after the first incident. Documents get amended, reclassified and deleted, people change teams and leave, and an index rebuilt on a schedule with permissions copied in at ingestion time spends the whole gap between rebuilds answering from a world that no longer exists, including for people whose access has been withdrawn. Two things carry most of the load. Evaluate permissions against the source system at query time instead of trusting what was copied in, and set re-index cadence, deletion and reclassification deliberately. Agree on day one how long a document deleted this morning can still be answered from, and whether your data protection officer accepts that number." }, { "question": "Should we buy an enterprise knowledge platform before trying RAG?", "answer": "No. Tenhaw builds first retrieval paths over client corpora and puts a fortnight on the corpus and the permission model ahead of any purchase: which content is authoritative, who owns it, what its access rules actually are in the source system, and whether those rules can be read at query time. That work needs no supplier and it decides most of the architecture, including whether a platform would have helped. Buying first usually means paying to discover the same answers in a worse order, because the questions a platform cannot settle for you are exactly the ones that matter. Enterprise search across the whole estate is a different job from one workflow proved out." }, { "question": "Who needs to own the content before an AI assistant can answer from it?", "answer": "The corpus needs a named owner, and the job is deciding which document is authoritative. Enterprise corpora hold the policy, the superseded policy, three drafts of the next one, and a slide that summarises all of them incorrectly. Retrieval returns the wrong one with complete composure, because relevance is not authority, so someone has to mark what is current and what has been withdrawn. You also need whoever controls access in each source system, whose rules have to reach query time, and your data protection officer, who has to accept how long a document deleted this morning can still be answered from. Until those people exist, tuning chunk sizes is work on the wrong problem." }, { "question": "How many questions do you need in a retrieval evaluation set?", "answer": "Fifty real ones settle almost every architectural argument you are about to have. Collect them from the people who will actually ask, and write each row as three things: the question in their words, the answer they would accept, and the document that answer lives in. That list is the design brief and the first evaluation set at the same time, and it is cheap to label because nothing needs a model in the loop to grade it. Write it before you pick a store, an embedding model or a chunking strategy, and those stop being preferences defended in a meeting and become decisions with a number attached." }, { "question": "Can you test a retrieval system properly before it goes live?", "answer": "Only partly, and the literature is candid about it. A study of three production retrieval systems concluded that validating one is only feasible during operation, and that robustness evolves rather than being designed in at the start. That is not a reason to skip testing. It is a reason to ship something narrow to real users early and treat the first months of running it as part of the test. A retrieval score kept separate from the generation score still belongs in the build from week one, and so does a permission test that is meant to fail, where someone who should not see a document asks the question that would surface it." }, { "question": "How do we know an AI answer came from our own documents?", "answer": "Make every answer carry the passage it came from, as a link the reader can open. That one change alters the risk profile of the whole system more than any single accuracy improvement, because a checkable answer fails safely and an unchecked one does not. It also lets a reader see that the source was the superseded policy, which no confidence number would have told them. On an engagement with Placing Platform Limited (PPL), Tenhaw scored confidence from provenance, which source each extracted value was drawn from, combined with model certainty and an independent cross-check. Where a passage came from is a stronger signal than how sure the model sounds, and it is the one a reviewer can argue with." }, { "question": "What can a retrieval proof of concept prove in two to four weeks?", "answer": "Enough to settle the architecture. On a retrieval problem the target is one real path over your own content: a defined corpus, permissions carried from the source system to query time, and answers that cite the passage they came from. Tenhaw took Placing Platform Limited (PPL) from PDFs to business intelligence on Azure in a fortnight, a proof of concept over ground it had circled for roughly a year. The artefact that outlasts the code is the evaluation set, real questions with the passages that answer them, scored separately from the generated answer. That is the thing almost nobody has, and the thing that still tells you the truth after you change the model, the embedding or the store." }, { "question": "Should an AI assistant search all our systems, or start with one?", "answer": "Start with one, and choose the one where the questions actually are. Tenhaw builds retrieval paths one workflow at a time rather than standing up enterprise search across a whole estate. Every repository has its own access model, so each system you add is fresh work rather than a copy of the last. The rules have to be read from that source, carried onto the indexed item and matched to the person asking at query time. Doing that once, properly, on the content that matters most tells you what the second system will cost. Doing it everywhere at once means meeting the hardest permission model late, with an index already built on the wrong assumption and a rebuild in front of you." }, { "question": "Where do the index and embeddings sit if we need UK data residency?", "answer": "Wherever you decide, and decide it explicitly, because embeddings are derived from your documents and inherit their sensitivity. Tenhaw works to UK data residency by default, with the EU available, and builds on your infrastructure under your policies. The genuinely provider-specific choices are whether document-level access control is native to the search service or something you build on top of it, how identity federates from your source repositories, and where the vectors physically live. It has delivered retrieval components on Azure; on AWS and Google Cloud the shape is the same and the services differ. OWASP lists vector and embedding weaknesses among its top risks for language model applications, so treat the index as in scope for classification and review." } ], "citations": [ { "name": "Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)", "url": "https://arxiv.org/abs/2005.11401", "note": "The paper that named the pattern. Source of the argument that models storing knowledge in their parameters have a limited ability to access and precisely manipulate it." }, { "name": "Barnett et al., Seven Failure Points When Engineering a Retrieval Augmented Generation System (2024)", "url": "https://arxiv.org/abs/2401.05856", "note": "Three production case studies. Source of the finding that validating a retrieval system is only feasible during operation and that robustness evolves rather than being designed in." }, { "name": "Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)", "url": "https://arxiv.org/abs/2307.03172", "note": "Performance is highest when the relevant information is at the beginning or end of the context and degrades significantly in the middle, even for models built for long contexts." }, { "name": "Microsoft, security filter pattern for Azure AI Search", "url": "https://learn.microsoft.com/en-us/azure/search/search-security-trimming-for-azure-search", "note": "One provider's documented pattern for document-level access control: an identity field on each indexed document, filtered against the caller's group membership at query time. Named as a worked example, not as a recommendation of that provider." }, { "name": "Greshake et al., Not what you've signed up for: compromising real-world LLM-integrated applications with indirect prompt injection (2023)", "url": "https://arxiv.org/abs/2302.12173", "note": "Demonstrates injecting instructions into data an application is likely to retrieve, against real deployed systems. The reason retrieved content is treated as untrusted input above." }, { "name": "OWASP Top 10 for Large Language Model Applications (2025)", "url": "https://genai.owasp.org/llm-top-10/", "note": "LLM01 Prompt Injection and LLM08 Vector and Embedding Weaknesses are the relevant entries." } ] } ## MCP, tool calling and integrating agents with your systems Source: https://tenhaw.com/guides/mcp-tool-calling-and-system-integration Last updated: 2026-11-05 { "slug": "mcp-tool-calling-and-system-integration", "name": "MCP, tool calling and integrating agents with your systems", "metaTitle": "MCP, tool calling and system integration", "metaDescription": "Choose MCP or direct function calling, then design identity, permissions and action controls. Includes practical examples and dated provider references.", "tagline": "Connect the tools with clear permissions, approvals and operating evidence.", "summary": "MCP is a common interface for connecting AI applications to tools and data. Direct function calling can suit a small, bounded application; a shared MCP interface can help several clients use the same capabilities. Both need explicit authority, input checks and action controls. Start with the decision table, then use the system-specific reference below.", "theme": "engineering", "evidenceBasis": "approach", "evidenceStatement": "We build with tool-calling agents daily. Our own operations run research agents that call external services for competitor monitoring and opportunity-gap analysis, and our development pipeline is AI-engineering-first. On client work, the document pipeline delivered on a specialty insurance engagement calls third-party APIs to enrich extracted fields, so the integration and provenance parts of this are written from delivery. The PPL build is a working proof of concept inside the client's estate. The rest of this guide is the method we would bring to an internal catalogue of MCP servers with an authorisation model behind it. Where deep integration or identity engineering is required we would expect to work alongside your platform and IAM functions rather than around them.", "evidenceNextStep": { "heading": "So what should you do about it", "body": "Settle the authorisation model before the catalogue. One question decides most of the architecture: does a tool call carry the asking user's authority or the agent's own, and does the answer differ between a read and a write. That is a fortnight of design with your platform and identity people in the room, it is worth doing whether or not you ever adopt MCP because the same question appears in any function-calling design, and it is the thing a security review will fail you on. Use us to build the first two integrations properly and to write the contract every later one is held to. If what you actually need is an enterprise integration platform stood up, that is a real piece of work, and we will tell you whether we are the right firm for it." }, "demandSignal": "The standard's own documentation is the demand signal worth reading. The Model Context Protocol describes itself as a standardised way to connect AI applications to external systems, the way USB-C standardised connecting devices, and it ships a security best practices document alongside the specification. That document names confused deputy attacks through OAuth proxies, token passthrough, server-side request forgery during metadata discovery, session hijacking, local server compromise and scope inflation, in specification language: an MCP server must not accept any token that was not explicitly issued for it. A standard that publishes its own attack surface in that much detail is telling you where the work is, and it is not in the wire format.", "whyItStalls": [ { "heading": "The integration takes an afternoon and the authorisation takes a quarter", "body": "Connecting an agent to an API is genuinely easy now, which is the problem: the first version works, so nobody asks whose authority the call carries. The protocol specification is blunt about the anti-pattern, calling token passthrough explicitly forbidden and requiring that a server refuse tokens not issued to it, because the alternative breaks rate limiting, breaks the audit trail, and lets a stolen token use your server as a proxy for exfiltration. Programmes that skip this arrive at the security review with an integration nobody can explain and no cheap way to fix it." }, { "heading": "Every useful tool gets added to one agent until nobody can reason about it", "body": "Tool sprawl is the quiet failure. Each addition is individually sensible, the combined permission set is nobody's decision, and the specification's own guidance on scope minimisation describes exactly where it ends: broad scopes granted up front, a blast radius nobody has calculated, revocation that would disrupt every workflow at once, and users who stop reading consent screens because the list is too long to read. This is also the mechanism that stops an agent portfolio scaling past three or four." }, { "heading": "Nobody separated the tools that read from the tools that act", "body": "A retrieval call and a payment call are the same shape to a model and entirely different to your business. Where that distinction is not explicit in the design, the human approval boundary ends up wherever someone happened to put a confirmation dialogue, rather than being drawn by consequence and reversibility. It is the same boundary question as any other consequential decision and it belongs in the operating model, not in whichever pull request first needed it." }, { "heading": "The API being wrapped was designed for a person with a form in front of them", "body": "Human-facing APIs assume a human pace and a human retry. Agents call them in loops, in parallel, and immediately after a timeout, which surfaces every missing idempotency key, every multi-step write with no transaction boundary and every rate limit nobody had reached before. The failure is rarely dramatic. It is a duplicate record, twice, in a system somebody reconciles monthly." }, { "heading": "A tool description and a tool result are both prompt", "body": "Everything the model reads can steer it, including the text describing what a tool does and the payload a tool returns. Indirect prompt injection was demonstrated against deployed applications by planting instructions in content the application would later fetch, and prompt injection is the first entry in the OWASP risk list for language model applications. Once tools can act, an injection stops being an embarrassing answer and becomes an action taken under your agent's credentials, which is a different conversation with your auditors." } ], "approach": [ { "heading": "Start from the decision, not from the API catalogue", "body": "The design starts with the decisions in the workflow and what each one genuinely needs to see or change, and the tool list falls out of that. Starting from what the systems happen to expose produces an agent with thirty tools, twenty-six of which exist because they were easy, and a permission set nobody can defend." }, { "heading": "Split read from write, and make the write tools boring", "body": "Read tools and write tools are designed and reviewed as different classes. Write tools are single-purpose with narrow typed arguments, carry an idempotency key so a retry cannot double-post, and require explicit confirmation for anything irreversible. There is no general-purpose escape hatch tool, because a tool that can run arbitrary queries or arbitrary commands makes every other permission decision on the page decorative." }, { "heading": "Give the agent its own credentials, scoped to the task and time-boxed", "body": "Each agent gets a distinct identity, never a human's and never a shared service account, with permissions scoped at the granularity of the workflow rather than the agent, and credentials that expire. This is cheap at design time and expensive once actions have accumulated against the wrong principal, and it is the thing that makes the blast radius calculable.", "links": [ { "label": "Agent identity and access in full", "href": "/guides/agent-identity-and-access" } ] }, { "heading": "Validate the audience of every token, and never pass one through", "body": "A server accepts only tokens issued for itself, and exchanges rather than forwards when it needs to call something downstream. The protocol specification states this as a requirement rather than a recommendation, and the reasoning is the part worth carrying to your architecture review: without it the downstream system's logs show the wrong caller, the security controls that depend on token audience are bypassed, and one compromised service becomes access to everything that trusted the same token." }, { "heading": "Version the tool contract, and review descriptions as code", "body": "Tool names, argument schemas and descriptions are versioned artefacts under change control, reviewed like any other code, because a change to a description changes model behaviour as surely as a change to the implementation does. Renaming a tool is a breaking change to a system whose behaviour you have evaluated, and it should fire the same regression run that a model upgrade does." }, { "heading": "Log the call, not just the outcome", "body": "Every tool call records which agent identity made it, which human is accountable for that agent, the arguments, the result, the latency and the token cost. This is where the answer to what it costs per run and how slow it is comes from, it is the raw material for trajectory scoring, and it is the same record a regulator asks for when they want to know what the system actually did. Collect it from the first week, because reconstructing it later is a project.", "links": [ { "label": "Scoring the trajectory rather than the answer", "href": "/guides/agent-evaluation-and-assurance" } ] }, { "heading": "Test the paths nobody demonstrates", "body": "Timeouts, partial writes, rate limits, a tool that has been renamed under you, a downstream system in read-only mode, and a tool that returns hostile content. Those belong in the regression set alongside the happy path, because they are what week three looks like, and because an agent's response to a failed tool call is behaviour you have to specify rather than discover." } ], "technology": { "heading": "The orchestration layer, named, with what we have built on and where we hold a view", "lede": "The frameworks a technical buyer asks about by name, each with the same label the guide carries. Delivered means we have shipped on it. Our approach means we hold a position.", "items": [ { "name": "Native tool calling in the model APIs", "basis": "delivered", "what": "Typed functions declared to the model, which decides which to call and with what arguments. No framework involved.", "position": "This is what we actually build with, daily and on client work. The document pipeline delivered on a specialty insurance engagement calls third-party APIs to enrich extracted fields, and our own operations run research agents that call external services. For one agent and a handful of integrations this is the whole answer, and reaching for a framework first is how a fortnight of work acquires a dependency it will still be carrying in year two." }, { "name": "Model Context Protocol (MCP)", "basis": "approach", "what": "An open standard for connecting AI applications to tools and data sources, so an integration is built once rather than once per assistant.", "position": "The interesting part of adopting it in an enterprise is the authorisation model rather than the wire format. It starts paying when several assistants need the same systems, because the alternative is an adapter per pair. Read the protocol's own security guidance before the design review: it is the document that names token passthrough as forbidden, and that requirement is the one an enterprise architecture actually turns on." }, { "name": "LangChain and LangGraph", "basis": "approach", "what": "Open-source frameworks for composing model calls into applications. LangGraph models the composition as a graph with explicit state.", "position": "The position is about the shape rather than the brand. Agentic work needs explicit, inspectable, resumable state, because a trajectory you cannot reconstruct is one you cannot score, cannot debug and cannot explain to an auditor, and a graph with checkpoints gives you that where a chain of implicit calls does not. What we would resist on any framework is letting the abstraction become the architecture: the model interface, the prompts, the retrieval corpora, the tool contracts and the evaluation sets are your assets and they should survive you replacing whatever composes them. Ask any supplier proposing one what you would be able to change a year later without them." }, { "name": "Microsoft Semantic Kernel", "basis": "approach", "what": "Microsoft's SDK for composing models, plugins and planning inside an application.", "position": "It is the orchestration layer closest to the estate we have actually delivered on. On an Azure estate it is the first thing we would evaluate, and evaluate is the word: against your own tool contracts and your own evaluation set, not against a demonstration." }, { "name": "CrewAI and AutoGen", "basis": "approach", "what": "Frameworks for orchestrating several agents that hand work between each other in defined roles.", "position": "Our position is openly sceptical for a sequencing reason. A multi-agent shape multiplies the number of trajectories you have to evaluate, and it usually arrives before anyone has a ground-truth set for one. Most workflows sold as needing a crew are one agent with a well-designed tool list and a clear stopping rule. Start with one, build the evaluation set, and add agents when a single trajectory demonstrably cannot carry the work. That is a view rather than a finding." } ] }, "namedSystems": { "heading": "The systems agents actually get pointed at, and what each access model implies", "lede": "Compare the interface, authentication and resource permissions separately. The linked provider references were checked on 5 November 2026; validate the target tenant and release before designing against them.", "rows": [ { "system": "Salesforce", "wants": "Accounts, contacts, opportunities and cases: the record an agent summarises, triages or updates.", "auth": "OAuth client credentials run as the configured integration user. Current API setup uses the app and flow configuration supported by your Salesforce release.", "permissions": "The integration user's data and CRUD permissions bound the action. Use a dedicated integration identity and the permission sets needed for this workflow.", "watchOut": "Keep identities attributable and test the exact records and fields the workflow should and should not reach.", "sourceUrl": "https://developer.salesforce.com/blogs/2024/02/invoke-rest-apis-with-the-salesforce-integration-user-and-oauth-client-credentials", "sourceChecked": "5 November 2026" }, { "system": "ServiceNow", "wants": "Incidents, requests, changes and the CMDB: the workflow an agent triages, enriches or routes.", "auth": "Inbound OAuth supports user-facing and unattended grant types. Choose the grant for the actual interaction and instance release.", "permissions": "Define the OAuth API scopes and the platform-side access required for the operation. Test the endpoint, table, field and record boundaries in the target instance.", "watchOut": "A successful token exchange proves authentication, not that the workflow has an appropriate access boundary. Review the integration principal and instance controls with its owner.", "sourceUrl": "https://www.servicenow.com/docs/r/platform-security/authentication/oauth-inbound.html", "sourceChecked": "5 November 2026" }, { "system": "SharePoint and Microsoft 365", "wants": "Policies, procedures and the document estate: the corpus almost every enterprise retrieval build starts from.", "auth": "Entra ID supports delegated and application access. The Selected permission family can be used to limit an application to particular resources.", "permissions": "Selected access requires consent, an explicit resource grant and a token with the relevant scope. Read, write, owner and fullcontrol grants have different consequences.", "watchOut": "An indexing grant is not a guarantee that every caller may receive every indexed passage. Test retrieval against the asking person's current access.", "sourceUrl": "https://learn.microsoft.com/en-us/graph/permissions-selected-overview", "sourceChecked": "5 November 2026" }, { "system": "Snowflake", "wants": "The warehouse an agent queries for a number, and the semantic layer it should be querying instead.", "auth": "Use an appropriate non-password method for service users. Snowflake's published strong-authentication window for all users was August–October 2026; confirm the applied account policy.", "permissions": "Check the workflow identity's role grants, inherited privileges and session context against the specific data operation.", "watchOut": "Do not assume a legacy password or a broad inherited role remains a suitable workflow design. Confirm account state and the allowed tables or views.", "sourceUrl": "https://docs.snowflake.com/en/user-guide/security-mfa-rollout", "sourceChecked": "5 November 2026" }, { "system": "Databricks", "wants": "The lakehouse behind the analytical question, and the pipeline that produced the table the agent is about to trust.", "auth": "OAuth M2M uses a service principal for unattended work. Supported unified authentication handles short-lived token acquisition and refresh.", "permissions": "The principal still needs workspace/resource permissions. Current documentation also describes scoped OAuth secrets; the access token cannot exceed the secret's scopes.", "watchOut": "Scope both the secret and principal to the required operation. Review legacy all-apis secrets and check compute permissions separately.", "sourceUrl": "https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m", "sourceChecked": "5 November 2026" }, { "system": "Workday", "wants": "Worker, organisation and absence data: the system of record behind most HR and finance workflows.", "auth": "Workday REST and Graph APIs use OAuth2.0. Register an API client, then select scopes matching the endpoint functionality.", "permissions": "The public developer documentation covers client scopes and authorised tenants. Confirm the tenant-side security configuration with the Workday administrator for the specific operation.", "watchOut": "Treat deployment to another tenant as a separate access decision: Workday documents an authorised-tenant allowlist. Employment decisions and sensitive records need their own accountable owners.", "sourceUrl": "https://developer.workday.com/documentation/zwx1518028675482", "sourceChecked": "5 November 2026" } ], "caveat": "These are documentation references, not tests or certification of a client tenant. The provider source and review date sit in each row; a new grant, release or workflow is a reason to check the boundary again." }, "providerNotes": { "heading": "Where a standard helps, and where it does not", "body": "MCP standardises the wire between an application and a tool, which removes a real and boring cost: without it, every assistant needs its own adapter for every system, and the count multiplies. What it does not standardise is your authorisation model, your residency position or the semantics of your own systems, and those are the parts that take the time. The protocol's own security guidance is explicit that per-client consent, token audience validation and scope minimisation are the implementer's job. Treat the standard as a saving on plumbing rather than as an answer to the access question, and be sceptical of any supplier who presents it as the second thing." }, "engageVia": { "slug": "agentic-design-team", "label": "Agentic Design Team" }, "faq": [ { "question": "What is the Model Context Protocol?", "answer": "MCP is an open standard for connecting AI applications to external systems: data sources such as files and databases, tools such as search and calculation, and predefined workflows. Its own documentation compares it to USB-C, a standardised connector so that a tool built once can be used by any client that speaks the protocol. Practically, it means an integration with your CRM is built against the standard rather than separately for each assistant, which is a real saving once you have more than one. It is supported across a range of assistants and development tools, and it is a connection standard rather than a security model." }, { "question": "Do we need MCP, or is plain function calling enough?", "answer": "For one agent and three integrations, plain function calling is enough and adding a protocol buys you nothing. MCP starts paying when the same systems have to be reachable by several different agents or assistants, because the alternative is an adapter per pair and a maintenance burden that grows faster than the value. The decision is about how many consumers of an integration you expect, not about capability. Whichever you choose, the authorisation questions are identical. No protocol decides for you whether a call carries the user's authority or the agent's, and that is the more useful thing to notice." }, { "question": "What is the biggest security risk when an agent can call our systems?", "answer": "That the call carries an authority nobody decided on. The specific anti-pattern the protocol specification forbids is token passthrough, where a server accepts a token that was not issued for it and forwards it downstream. Doing so breaks rate limiting and request validation, it makes the downstream logs name the wrong caller, and a stolen token turns your server into a proxy for exfiltration. The second risk is broad standing permissions granted at setup because it was simpler, so a prompt injection through retrieved content or a reasoning error acts with the whole permission set rather than the task's. Both are design decisions taken in week one, and both are expensive to reverse." }, { "question": "How do you stop an agent taking an action it cannot undo?", "answer": "By classifying the tools rather than trusting the model. Tenhaw settles that classification in the operating model rather than in the codebase. Every tool is labelled by consequence and reversibility before it is exposed: read, reversible write, and irreversible or consequential action. The third class either requires a human confirmation that names what is about to happen, or is not exposed to the agent at all and is instead raised as a request for a person to execute. Write tools carry idempotency keys so a retry cannot double-post, and the boundary is written down per decision class with your second line as co-author rather than settled in a code review." }, { "question": "How many tools should one agent have?", "answer": "Fewer than you will be tempted to give it, and the constraint is not the model, it is your ability to state the blast radius. Broad permission sets granted up front expand what a stolen token reaches, make revocation disruptive enough that nobody does it, and turn consent screens into something users click through, which is the argument the protocol's own guidance on scope minimisation makes well. In practice we would rather run three narrow agents with defensible permission sets than one that can do everything, and the audit trail is legible either way." }, { "question": "How do you know what an agent actually did?", "answer": "By logging the tool calls rather than the conclusions. Tenhaw instruments that logging in the first week of a build, because reconstructing it later is a project. Each call records the agent identity, the human accountable for that agent, the arguments passed, the result returned, the latency and the token cost, against the version of the tool contract in force at the time. That record answers the three questions you will be asked after any incident, which are what it was asked, what it did, and under whose authority, and it does so without an archaeology exercise. It is also the input to trajectory scoring, so the same logging pays for itself twice." }, { "question": "What does a tool-calling agent cost per run, and how slow is it?", "answer": "Benchmark numbers would describe Tenhaw's workloads rather than yours, so none are published. Both are measurable from day one if you log them, and the drivers are known: the number of model turns, which rises with the number of tools and falls with a tighter tool set; the tokens in context on each turn, which retrieval design controls; the latency of your own systems, which is usually the dominant term and is not something a model choice fixes; and retries. Tenhaw logs cost and latency per tool call from the first week of a build. Give each stage an explicit budget in the design, measure against it in the build, and get an estimated run cost per candidate workflow before anything is committed to." }, { "question": "How do you connect an agent to Salesforce, ServiceNow, SharePoint, Snowflake, Databricks or Workday?", "answer": "The connector is an afternoon in every one of those. The authorisation model is the quarter, and it differs per system in ways that decide the architecture. Salesforce uses OAuth 2.0 against an external client app, with the JWT bearer flow or the client credentials flow for an unattended agent, and even the client credentials flow requires you to nominate an execution user whose permission sets are the real permission model. ServiceNow resolves an inbound REST call to a platform user, and that user's roles plus table, field and record-level access control rules decide everything, so the agent sees what that user would see in the interface. SharePoint and Microsoft 365 run on Entra ID, where delegated permissions intersect with the signed-in user's own access and application permissions do not, and where a certificate rather than a secret is required for app-only access to the SharePoint APIs. Snowflake is role-based with inheritance, and is retiring single-factor password authentication for service users on a published schedule that completes in the August to October 2026 window. Databricks uses OAuth machine-to-machine for a service principal with one-hour tokens scoped either to the account or to a single workspace. Workday uses an OAuth 2.0 API client registered in the tenant with scopes selected at registration, and a tenant-side security configuration behind it that you should confirm with your own administrator. Each of those authorisation models was read from the vendor's own documentation on 26 July 2026, and vendors change them, sometimes at short notice, so check the source before you design against one." }, { "question": "What is the difference between delegated and application permissions when an agent reads SharePoint?", "answer": "It is the single most consequential design decision in an enterprise retrieval build, and it is usually taken by accident in week one. With delegated permissions the app acts on behalf of a signed-in user and its access is intersected with that user's own, so it can never return a document that person could not already open. With application permissions there is no user in the picture and no intersection, so an app granted a tenant-wide read permission app-only can read every file in the organisation, which is exactly what a crawler is usually given because it is the fastest way to build an index. Between the two sit the Selected scopes, Sites.Selected and the newer Lists, ListItems and Files variants, which grant nothing when consent is given and require an explicit per-resource grant with a role of read, write, owner or fullcontrol, so all three steps have to be completed before the app has any access at all. The design that holds is Selected scopes for what may be indexed, delegated access at query time, and a test in the build pipeline where a named user who should not see a document asks the question that would surface it and the run fails if it comes back. Retrofitting this after indexing usually means rebuilding the index." }, { "question": "Do we need multi-agent AI, or is one agent enough?", "answer": "One is usually enough, and Tenhaw is openly sceptical of the crew for a sequencing reason. A multi-agent shape multiplies the number of trajectories you have to evaluate, and it almost always arrives before anyone has a ground-truth set for a single one. Most workflows sold as needing several agents are one agent with a well-designed tool list and a clear stopping rule, and the tool list is the part that repays the effort. Start with one, build the evaluation set, and add agents when a single trajectory demonstrably cannot carry the work. That sequencing argument is a view rather than a finding. Tenhaw builds that first agent on an Agentic Proof of Concept." }, { "question": "Should we use LangChain, LangGraph, CrewAI, AutoGen or Semantic Kernel?", "answer": "Start with none of them. That is a position rather than a finding. Tenhaw builds on the native tool calling in the model APIs instead, as it did for the document pipeline delivered on a specialty insurance engagement. For one agent and a handful of integrations that native tool calling is the whole answer, and a framework is a dependency you will still be carrying in year two. Where a framework earns its place, the property to buy is explicit, inspectable, resumable state, which a graph with checkpoints gives you and a chain of implicit calls does not, because a trajectory you cannot reconstruct is one you cannot score, debug or explain to an auditor. On an Azure estate, Semantic Kernel is the layer closest to the platform Tenhaw has actually delivered on and the one to evaluate first, against your own tool contracts rather than against a demonstration. Tenhaw is openly sceptical of multi-agent frameworks. A crew multiplies the trajectories you have to evaluate, usually before anyone has a ground-truth set for one. Whichever you pick, keep the model interface, the prompts, the tool contracts, the retrieval corpora and the evaluation sets as your assets, so they survive you replacing the thing that composes them." }, { "question": "Where do we start when connecting an agent to our systems?", "answer": "Settle the authorisation model before you build the tool catalogue. Most of the architecture is decided by one question, whether a tool call carries the asking user's authority or the agent's own, and whether the answer changes between a read and a write. Expect about a fortnight of design with your platform and identity people in the room. It is worth doing whether or not you ever adopt MCP, because the same question turns up in any function-calling design, and it is what a security review will fail you on. Then build the first two integrations properly and write the contract every later one is held to. Tenhaw pair-programs that work with your own engineers, as it did throughout the PPL insurance proof of concept." }, { "question": "Why does every agent action show up as the same Salesforce user?", "answer": "Because the flow has no signed-in user of its own. Salesforce's client credentials flow still requires you to nominate an execution user, and every action the agent takes is attributed to that user, so three agents sharing one execution user leave an audit trail that cannot say which of them updated the record. Give each agent its own integration user, carrying the API Only User permission and holding its privileges through permission sets under the Salesforce API Integration permission set licence, and scope those permission sets to the workflow rather than to the system. Attribution is cheap to design in and expensive to reconstruct once actions have accumulated against the wrong principal." }, { "question": "Is MCP a replacement for our integration platform?", "answer": "No. Without a standard, every assistant needs its own adapter for every system, and MCP takes out that real and boring cost. But the wire is the easy half. It does not standardise your authorisation model, your data residency position or the semantics of your own systems, and those take the time. Tenhaw builds integrations against your authorisation model, and the delivered example is a two-week Azure proof of concept for Placing Platform Limited (PPL). The protocol's own security guidance says plainly that per-client consent, token audience validation and scope minimisation are the implementer's job. Read it as a saving on plumbing, not an answer to the access question, and be sceptical of any supplier who sells it as the second." }, { "question": "What should we ask a vendor selling us an MCP server?", "answer": "Three things. First, which tokens the server will accept. The specification requires a server to refuse any token that was not explicitly issued for it and names token passthrough as forbidden, so a supplier who cannot answer that has not read the document they are selling against. Second, whether the tool catalogue separates reads from writes, and whether it contains a general-purpose tool that can run arbitrary queries or commands, because one of those makes every other permission decision decorative. Third, what you could change a year later without them, since the tool contracts, prompts, retrieval corpora and evaluation sets are your assets and should outlive whatever composes them." }, { "question": "Can an agent just use the APIs we already have?", "answer": "Usually yes, and then usually not quite. Human-facing APIs assume a human pace and a human retry. An agent calls them in loops, in parallel, and immediately after a timeout, which finds every missing idempotency key, every multi-step write with no transaction boundary and every rate limit nobody had reached before. The failure is rarely dramatic, just a duplicate record, twice, in a system somebody reconciles monthly. The answer is rarely a new API estate. It is a thin layer of narrow, single-purpose write tools with typed arguments and an idempotency key so a retry cannot double-post, and explicit confirmation on anything irreversible." }, { "question": "What happens if someone renames a tool the agent uses?", "answer": "You have made a breaking change to a system whose behaviour you have already evaluated, so it should fire the same regression run a model upgrade does. Tool names, argument schemas and descriptions are versioned artefacts under change control, reviewed like any other code, because changing a description changes model behaviour as surely as changing the implementation. A rename underneath a running agent belongs in the regression set next to timeouts, partial writes, rate limits, a downstream system in read-only mode and a tool that returns hostile content. Those are the paths nobody demonstrates, and how an agent responds to a failed call is behaviour you specify rather than discover." }, { "question": "Why can our agent read more in Snowflake than we granted it?", "answer": "Because roles inherit. Snowflake's access control is role-based with object ownership on top, where privileges attach to roles, roles attach to users, roles inherit through a hierarchy, and secondary roles can combine several in one session, so an agent assuming a broad role gets everything beneath it rather than only what was granted to it directly. Inheritance is the trap, and it is why the agent needs its own role. Read what that role inherits before you read what it was granted, and scope it to the workflow rather than to the warehouse. Check how it authenticates while you are there, because Snowflake is retiring single-factor password authentication for service users on a published schedule." } ], "citations": [ { "name": "Model Context Protocol, introduction", "url": "https://modelcontextprotocol.io/docs/getting-started/intro", "note": "The standard's own description: an open standard for connecting AI applications to external systems, with the USB-C comparison quoted above." }, { "name": "Model Context Protocol, security best practices", "url": "https://modelcontextprotocol.io/specification/draft/basic/security_best_practices", "note": "Source of the token passthrough prohibition, the confused deputy and session hijacking attack descriptions, and the scope minimisation guidance. Worth reading in full before any MCP design review." }, { "name": "Greshake et al., Not what you've signed up for: compromising real-world LLM-integrated applications with indirect prompt injection (2023)", "url": "https://arxiv.org/abs/2302.12173", "note": "Why a tool result is treated as untrusted input: instructions planted in retrieved content were used against real deployed applications, including to trigger API calls." }, { "name": "OWASP Top 10 for Large Language Model Applications (2025)", "url": "https://genai.owasp.org/llm-top-10/", "note": "LLM01 Prompt Injection and LLM06 Excessive Agency are the entries that bear directly on tool design." }, { "name": "Salesforce, OAuth 2.0 client credentials flow for server-to-server integration", "url": "https://help.salesforce.com/s/articleView?id=xcloud.remoteaccess_oauth_client_credentials_flow.htm&type=5", "note": "Source of the nominated execution user, the API Only User recommendation and the Salesforce API Integration permission set licence. The note that connected app creation is restricted from Spring '26 in favour of external client apps is from the same documentation set." }, { "name": "ServiceNow, inbound REST API documentation", "url": "https://www.servicenow.com/docs/bundle/zurich-api-reference/page/integrate/inbound-rest/concept/c_RESTAPI.html", "note": "Source of the position that inbound REST authenticates by basic authentication or OAuth and that the caller's roles and access control rules then decide what is returned. Read alongside ServiceNow's own platform security documentation on access control rules." }, { "name": "Microsoft, overview of Selected permissions in OneDrive and SharePoint", "url": "https://learn.microsoft.com/en-us/graph/permissions-selected-overview", "note": "Source of the Selected scopes, the read, write, owner and fullcontrol roles, the three steps required before an app has any access, and the statement that in the delegated scenario the application can never exceed the user's permissions." }, { "name": "Microsoft, overview of Microsoft Graph permissions", "url": "https://learn.microsoft.com/en-us/graph/permissions-overview", "note": "The delegated versus application permission distinction in Microsoft's own words, including the worked example that an app granted Files.Read.All as an application permission can read any file in the organisation." }, { "name": "Microsoft, granting access to SharePoint via Entra ID app-only", "url": "https://learn.microsoft.com/en-us/sharepoint/dev/solution-guidance/security-apponly-azuread", "note": "Source of the certificate requirement: for app-only access to the SharePoint CSOM and REST APIs, the documentation states every other option is blocked and returns access denied." }, { "name": "Snowflake, multi-factor authentication rollout and deprecation timeline", "url": "https://docs.snowflake.com/en/user-guide/security-mfa-rollout", "note": "Source of the phase three window, August to October 2026, in which legacy service users are converted to the SERVICE user type and blocked from password authentication. Read with Snowflake's access control overview for the role hierarchy and secondary roles." }, { "name": "Snowflake, overview of access control", "url": "https://docs.snowflake.com/en/user-guide/security-access-control-overview", "note": "Source of the role-based model with object ownership on top, the role hierarchy that makes inheritance the thing to check, and secondary roles." }, { "name": "Databricks, OAuth machine-to-machine authentication", "url": "https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m", "note": "Source of the service principal client ID and secret exchange, the one-hour token lifetime, and the distinction between account-scoped and workspace-scoped tokens." }, { "name": "Workday, create your API client", "url": "https://developer.workday.com/documentation/zwx1518028675482", "note": "Source of the client ID and secret issued on registration and of selecting one or more scopes for the client's functionality. Workday's customer documentation for the tenant-side security configuration sits behind a login we could not open, which is why the row above stops where it does." } ] } ## Design guardrails for uncertain AI results Source: https://tenhaw.com/guides/guardrails-hallucination-and-accuracy-control Last updated: 2026-11-05 { "slug": "guardrails-hallucination-and-accuracy-control", "name": "Design guardrails for uncertain AI results", "metaDescription": "Design AI guardrails across input, retrieval, output and action. Use practical examples to decide what continues, pauses or needs a human review.", "tagline": "Decide what continues, what needs review and which actions are unavailable.", "summary": "AI guardrails bound what a model-led workflow may receive, retrieve, return and do. Define the consequence of an uncertain or incorrect result, then choose checks and human decisions for that risk. The worked example below connects input, retrieval, output and action controls to a single supplier-record change. Evaluate the ordinary cases and the awkward ones together.", "theme": "engineering", "evidenceBasis": "approach", "evidenceStatement": "We have built accuracy controls into delivered proofs of concept. On an engagement with Placing Platform Limited (PPL) we scored confidence from the provenance of each extracted field, combined with model certainty and an independent search-based cross-check, and routed records for human review on that basis. The layered rail stack this guide describes is the method we would bring.", "evidenceNextStep": { "heading": "So what should you do about it", "body": "Write the accuracy contract before you shop for a guardrail product. Three lists: what must never happen, what must be escalated to a person, and what is merely undesirable. Most teams find while writing it that the second list is where all the money is, and that a good part of the first list is enforceable in the data layer or in a tool permission rather than by a model at all. That is a week of work with the business owner and your risk function, it does not need us, and it makes every later tooling decision obvious. Where we help is building the rails and the evaluation that shows they hold, and handing both to your own reviewers to test rather than reporting on our own work." }, "demandSignal": "The research has moved from treating hallucination as a mystery to treating it as an incentive problem, which is far more actionable. A 2025 paper argues that models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty, and that part of the fix is changing how existing benchmarks score an admission of not knowing rather than adding more hallucination tests. Independently, the OWASP risk list for language model applications now carries misinformation as its own entry, alongside prompt injection, excessive agency and system prompt leakage. Both point the same way: an organisation planning to eliminate wrong answers is planning for something that will not happen, and an organisation planning for them is doing engineering.", "whyItStalls": [ { "heading": "The programme is trying to eliminate hallucination rather than bound it", "body": "Every week spent hunting a model that does not make things up is a week not spent designing what happens when one does. The incentives that produce a confident guess sit in how models are trained and scored, so the mitigation available to you is architectural: constrain what the system can say, make it cite where each claim came from, give it a supported way to decline, and route by consequence. Programmes that treat this as a procurement question, solvable by the next model, are still having the same conversation two quarters later with a larger bill." }, { "heading": "A guardrail product was bought and the policy it enforces was never written", "body": "Runtime rails are programmable by design: the published toolkits describe rails as a way of controlling output, keeping to a dialogue path, staying off certain topics, and they operate outside the model so they stay inspectable. That is the useful property and it is also the catch. Somebody has to write the rules, in your language, about your business, and a toolkit with no policy behind it is a dependency rather than a control. This is the single most common thing we would expect to find when asked to review an existing stack." }, { "heading": "The only guardrail is the system prompt", "body": "Instructions in a prompt are advisory, and they sit in the same channel as the input trying to override them. Prompt injection is the first entry on the OWASP risk list and system prompt leakage is an entry of its own, which between them describe both halves of the problem: the instruction can be overridden and it can be read. A control that a paragraph of user text can talk its way past is not a control, and it is the one most often presented to a risk committee as though it were." }, { "heading": "There is no way for the system to say that it does not know", "body": "If the only supported output is an answer, the system will produce an answer. Abstention has to be a first-class result with a route attached, a person, a fallback, or a plainly worded decline, or the accuracy target is being enforced by optimism. This is also why buying a model on a benchmark score can mislead: a benchmark that scores a guess and an admission of uncertainty identically rewards the behaviour you are trying to design out." }, { "heading": "Red teaming happened once, before launch, run by the people who built it", "body": "The published work treats adversarial testing as a scaled, continuing activity: one study released a dataset of 38,961 red team attacks and reported that models trained with human feedback became harder to attack as they grew, which is a result you only obtain by doing it at volume and over time. An afternoon workshop the week before go-live is not that, and it is what most programmes have. It is also the activity most obviously compromised by being run by the team whose release depends on the outcome." } ], "approach": [ { "heading": "Write the accuracy contract first: never, escalate, tolerate", "body": "Three lists, agreed with the business owner and the second line before anything is built. What must never happen, which becomes a hard constraint enforced outside the model wherever possible. What must be escalated, which becomes the routing design and the resourcing question. What is merely undesirable, which becomes a metric rather than a gate. Almost every argument later in the build is settled by pointing at this document, and its absence is why those arguments do not end." }, { "heading": "Prefer constraint to correction", "body": "A model required to return output matching a schema cannot invent a field. A model answering only from retrieved passages with a citation can be checked by the reader. A model with no tool for an action cannot take it. Each of those removes a class of failure, where an output filter only catches instances of one. Design out first, filter second, and treat the filter as the weaker control of the two." }, { "heading": "Layer the rails, and know what each layer is actually for", "body": "Four places, four different jobs. Input: what may be asked, and what may be pasted in. Retrieval: what may be fetched, and whether the person asking is entitled to see it. Output: what may be said, in what shape, with what attribution. Action: what may be done, by whom, and with what confirmation. Programmable runtime rails of the kind the published toolkits describe sit outside the model and remain interpretable, which matters mainly because a risk function can read them and form its own view.", "links": [ { "label": "Permission-aware retrieval, in full", "href": "/guides/retrieval-rag-and-permission-aware-knowledge-access" } ] }, { "heading": "Make abstention first class, and measure it", "body": "Declining is an outcome the system is designed to produce, not a failure state. Measure how often it happens and split it in two: correct refusals, where the system genuinely could not answer safely, and wasteful refusals, where it could have. A system that never abstains is not safe, it is unmeasured, and the ratio between those two numbers is one of the more informative things you can put in front of an operations team." }, { "heading": "Score confidence from provenance rather than from the model's own certainty", "body": "On our insurance engagement, confidence was derived from which third-party source a value came from, combined with model certainty and an independent search-based cross-check, rather than from the model's self-reported score alone. Provenance is what makes a routing decision explainable to the person who owns the outcome, and explainable routing is what allows a business to accept an accuracy figure it would otherwise reject." }, { "heading": "Use a model as a judge, and know precisely where it lies to you", "body": "Model-graded evaluation is genuinely useful: the published work reports over 80% agreement with human preference, about the rate at which humans agree with each other. The same work names position, verbosity and self-enhancement biases, which is why we would use a judge for triage and regression detection and keep human judgement on the release gate. Two further habits worth adopting: check what happens when the judge and the system under test come from the same family, and keep a human-scored sample every cycle so you can tell when the judge itself has drifted." }, { "heading": "Red team continuously, and not with the build team", "body": "A standing adversarial set that grows with every incident and near miss, refreshed on every model upgrade, and owned by someone who does not benefit from the release. That set is the cheapest insurance in the programme, because each entry is a failure you only pay for once. It also gives your risk function something concrete to inspect rather than a paragraph asserting that testing was thorough.", "links": [ { "label": "Who should own the gate", "href": "/guides/agent-evaluation-and-assurance" } ] } ], "engageVia": { "slug": "agentic-design-team", "label": "Agentic Design Team" }, "faq": [ { "question": "How do you stop an AI agent hallucinating?", "answer": "You do not stop it, you bound it, and the difference is the whole design. Recent work argues models produce confident false statements because training and evaluation reward guessing over admitting uncertainty, so the lever available to you is architectural rather than a better model. Four things do most of the work. Constrain the output, so a schema or a required citation makes whole classes of invention impossible. Ground the answer in retrieved passages the reader can open, so a wrong answer is checkable rather than merely fluent. Give the system a supported way to say it does not know, and measure how often it uses it. And route by consequence, so the cases where being wrong is expensive reach a person. Accuracy improvements help at the margin; those four change what a wrong answer costs." }, { "question": "What are AI guardrails?", "answer": "Runtime controls placed around a model to bound what it can be asked, what it can retrieve, what it can say and what it can do. They sit outside the model rather than inside its training, which is what makes them changeable without retraining and inspectable by someone who is not an engineer. In practice a serious stack has four layers: input, retrieval, output and action. The important thing to understand about them is that a guardrail toolkit is programmable, so it enforces whatever policy you write and nothing else. Buying one without writing the policy is buying a dependency." }, { "question": "Is a system prompt a guardrail?", "answer": "No, and treating one as though it were is a common way to mislead a risk committee without meaning to. A system prompt is an instruction in the same channel as the input that may be trying to override it, which is why prompt injection is the first entry on the OWASP risk list for language model applications, and why system prompt leakage is an entry in its own right. A prompt is a useful way to shape behaviour and a poor way to prevent it. Anything that must never happen belongs in a schema, a permission, a filter outside the model, or a system that is simply not reachable." }, { "question": "What is prompt injection, and how do you defend against it?", "answer": "Prompt injection is text that reaches the model as an instruction when it was meant to be read as data, typed by a user or planted in retrieved content. It heads the OWASP risk list for language model applications, and it works because a prompt is advisory, sharing a channel with the input trying to override it. The defence is architectural and layered. Prefer constraint to correction, since a schema stops an invented field, a citation lets the reader check a claim, and an absent tool cannot be called. Treat retrieved passages as data, not instructions, and scope what the agent can reach, the control that holds when others fail. Keep adversarial documents in the evaluation set. A well-meant instruction in a document template injects as surely as an attacker's." }, { "question": "What accuracy should we ask a supplier to commit to?", "answer": "Be wary of anyone quoting a number before seeing your data, because it depends on what the exception path costs. Ask instead for four checkable commitments. A named evaluation set built from your cases, with a correct answer defined by whoever owns the decision. A threshold per case type rather than one headline figure, since the easy cases otherwise carry the average. A stated abstention behaviour, so you know what it does when it should not answer. And cost, latency and override rate alongside accuracy. A supplier willing to be held to those is a better sign than one quoting 95%. Tenhaw's own commitment is measurable value every month, with a month that delivers none reported as a failed month." }, { "question": "Can you use one language model to check another?", "answer": "Yes, for the right job. The published work on model-graded evaluation found strong judges reaching over 80% agreement with human preference, which is roughly the agreement rate between humans, so it is a reasonable way to score at a volume no human panel could. It also identified position, verbosity and self-enhancement biases, meaning a judge can prefer the first answer it sees, the longer answer, and answers resembling its own. So use it for triage, regression detection and ranking, not as the gate. Keep a human-scored sample every cycle to detect drift in the judge, and be deliberate about whether the judge and the system share a model family." }, { "question": "What actually breaks after week two?", "answer": "The long tail and the disagreements, in that order, and neither is a model problem. Weeks one and two are the happy path, which is where a demo lives. On a London insurance engagement, Tenhaw ran a gap-and-contradiction pass over the requirement corpus that surfaced ambiguities the business had not realised were ambiguous, resolved in conversation rather than a rebuild. What surfaces later is the document that is a scan of a fax, the record with a field the specification never mentioned, the case where two experienced people give different correct answers, and the tool that returns success while doing nothing. That build is a proof of concept, not a production system." }, { "question": "Will a newer model fix our accuracy problem?", "answer": "Sometimes, at the margin, and it will not change the shape of the problem. A stronger model typically improves the average and leaves you with the same questions: what happens on the cases it still gets wrong, how you would know, and who is accountable when it does. It also resets your evaluation, because behaviour changes in both directions on an upgrade and prompts tuned to the old model frequently perform worse on the new one. The organisations that get value from upgrades are the ones with an evaluation set to run them against, which is an argument for building that first rather than an argument against upgrading." }, { "question": "Should an AI system be allowed to say it does not know?", "answer": "Yes, and it should be designed to. If the only supported output is an answer, the system will produce an answer, which is how a confident guess reaches a user. Make abstention a first-class result with a route attached: a person, a fallback, or a plainly worded decline. Then measure it and split the number in two. Correct refusals, where the system genuinely could not answer safely, and wasteful refusals, where it could have. A system that never abstains is not safe, it is unmeasured, and the ratio between those two figures is one of the more informative things you can put in front of an operations team." }, { "question": "What is AI red teaming, and how often should you do it?", "answer": "Red teaming is adversarial testing. People deliberately try to make the system say or do something it should not, so the failures are found before a user or an attacker finds them. The published work treats it as a scaled, continuing activity rather than a pre-launch workshop. One study released a dataset of 38,961 red team attacks, which is a result you only obtain by testing at volume and over time. So run it continuously. Keep a standing adversarial set that grows with every incident and near miss, refresh it on every model upgrade, and have it owned by someone who does not benefit from the release." }, { "question": "Should we buy a guardrails product or build our own controls?", "answer": "It is largely a false choice, because a rail product enforces the policy you write and nothing else. Published toolkits are programmable by design. They run at runtime, sit outside the model and stay inspectable, which is the useful property and also the catch. Somebody still has to write the rules, in your language, about your business. Tenhaw published its own, 72 engineering rules on GitHub, graded by RFC 2119 severities and enforced by an agent. Buy the runtime if it saves engineers work, and build three things yourself regardless: the accuracy contract, the evaluation set that shows the rails hold, and the standing adversarial set. The strongest controls are not rail products at all. A schema, a permission and an absent tool do more." }, { "question": "What should an AI accuracy policy actually contain?", "answer": "Three lists, agreed with the business owner and the second line before anything is built. What must never happen, which becomes a hard constraint enforced outside the model wherever possible, often in the data layer or a tool permission rather than by a model at all. What must be escalated to a person, which becomes the routing design and the resourcing question. What is merely undesirable, which becomes a metric rather than a gate. Most teams find while writing it that the second list is where all the money is. That is roughly a week of work with your risk function, it does not need a supplier, and it makes every later tooling decision obvious." }, { "question": "Can you trust a model's own confidence score?", "answer": "Not on its own, and not as the sole basis for deciding what a person reviews. A self-reported certainty tells you how sure the model is, which is not the same as how likely it is to be right. On an engagement in the London insurance market, Tenhaw scored confidence from three inputs instead: the provenance of each extracted value, the model's own certainty, and an independent search-based cross-check. Provenance is what makes the routing explainable to the person who owns the outcome, and explainable routing is what lets a business accept an accuracy figure it would otherwise reject." }, { "question": "Can you add guardrails to an AI system that is already live?", "answer": "Yes, and it is a normal place to start. Rails sit outside the model rather than inside its training, so they can be changed without retraining and read by someone who is not an engineer, which is what makes a retrofit tractable. Begin with the accuracy contract rather than the code, because the usual finding in an existing stack is a toolkit with no written policy behind it, which is a dependency rather than a control. Then design the worst failures out with a schema or a scoped permission, place rails at input, retrieval, output and action, and build the evaluation set that shows the change held." }, { "question": "What can we show a risk committee to prove our AI guardrails work?", "answer": "Give them things they can inspect rather than a paragraph asserting testing was thorough. Tenhaw hands the rails and the evaluation set to your own reviewers to test rather than reporting on its own work. Three artefacts carry most of it. The written accuracy contract, so what must never happen is stated rather than implied. The rails themselves, which run outside the model and stay readable, so a risk function can form its own view without taking an engineer's word for it. And a standing adversarial set that grows with every incident and near miss, owned by someone who does not benefit from the release, with the results of its last run." }, { "question": "Can you pick a model on its hallucination benchmark score?", "answer": "Use benchmarks to shortlist, never to settle it. A 2025 paper argues that models produce confident false statements because training and evaluation reward a guess over an admission of uncertainty, and that part of the fix is changing how existing benchmarks score uncertainty rather than adding more hallucination tests. As a procurement point, a leaderboard that scores a guess and an admission of not knowing identically is rewarding the exact behaviour you are trying to design out, so a high score can hide the failure mode you care about. What settles the choice is your own evaluation set, built from your cases, with a threshold per case type rather than one headline figure." }, { "question": "Do we need a human in the loop for every AI output?", "answer": "No, and a review step applied to everything tends to become a formality rather than a control. Route by consequence instead, sending the cases that are expensive to get wrong to a person and letting the rest through with a check the reader can perform, such as a citation they can open. On a proof of concept for Placing Platform Limited (PPL), Tenhaw routed records to a reviewer on a confidence score rather than sending every extraction to a person. The accuracy contract decides the split rather than the technology. Where volumes are low and stakes are high, reviewing everything is a reasonable starting point, provided you know what it costs." }, { "question": "What does the first month of building AI guardrails look like?", "answer": "Week one is the accuracy contract rather than code: what must never happen, what gets escalated to a person, what you will merely measure. Then take one workflow and design the failures out before filtering them out, so a schema, a scoped permission or a tool that simply does not exist removes whole classes of error. Then put the four rails around it: input, retrieval, output and action. In parallel, build an evaluation set from your own cases with a correct answer defined by whoever owns the decision, and start the adversarial set that grows with every near miss. Tenhaw staffs a first month like that with two or three senior people led by James Rooney, and agrees the exit date at kickoff." } ], "citations": [ { "name": "Kalai, Nachum, Vempala and Zhang, Why Language Models Hallucinate (2025)", "url": "https://arxiv.org/abs/2509.04664", "note": "Argues hallucinations persist because training and evaluation reward guessing over acknowledging uncertainty, and recommends changing how existing benchmarks score uncertainty rather than adding more hallucination evaluations." }, { "name": "OWASP Top 10 for Large Language Model Applications (2025)", "url": "https://genai.owasp.org/llm-top-10/", "note": "LLM01 Prompt Injection, LLM07 System Prompt Leakage and LLM09 Misinformation are the entries behind the arguments above." }, { "name": "Rebedea et al., NeMo Guardrails: a toolkit for controllable and safe LLM applications with programmable rails (2023)", "url": "https://arxiv.org/abs/2310.10501", "note": "Source of the description of rails as runtime, programmable and independent of the underlying model. Cited as an example of the category, not as a tooling recommendation." }, { "name": "Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023)", "url": "https://arxiv.org/abs/2306.05685", "note": "Over 80% agreement with human preference, plus the position, verbosity and self-enhancement biases that make a model judge useful for triage and unsafe as a release gate." }, { "name": "Ganguli et al., Red Teaming Language Models to Reduce Harms (2022)", "url": "https://arxiv.org/abs/2209.07858", "note": "Released a dataset of 38,961 red team attacks and reported that models trained with human feedback became harder to red team as they scaled. The evidence that adversarial testing is a volume activity rather than a workshop." }, { "name": "Greshake et al., Not what you've signed up for: compromising real-world LLM-integrated applications with indirect prompt injection (2023)", "url": "https://arxiv.org/abs/2302.12173", "note": "Why a system prompt is not a control: instructions planted in retrieved content were used to steer real deployed applications." } ], "metaTitle": "AI guardrails and accuracy control" } ## RAG, fine-tuning or prompting: how to choose Source: https://tenhaw.com/guides/rag-fine-tuning-or-prompting Last updated: 2026-11-05 { "slug": "rag-fine-tuning-or-prompting", "name": "RAG, fine-tuning or prompting: how to choose", "metaDescription": "Compare prompting, RAG and fine-tuning against the same task and acceptance cases. A practical decision guide for knowledge, behaviour and operating cost.", "tagline": "Choose the experiment that answers your knowledge, behaviour or cost question.", "summary": "Start with prompting and an agreed evaluation set. Add retrieval when answers need current, attributable or permission-bound knowledge. Consider fine-tuning when measured behaviour, format or cost justifies a training dataset and its upkeep. Compare the approaches against the same task, acceptance cases and operating cost; the decision table below makes those trade-offs visible.", "theme": "engineering", "evidenceBasis": "approach", "evidenceStatement": "This is a decision framework rather than a delivery write-up. We have built prompting and retrieval systems on client engagements, including the document pipeline and semantic layer delivered on a specialty insurance engagement. The comparison this guide leans on is other people's published research, cited below. Where the right answer to your question is a specialist model team rather than us, we will say so on the call.", "evidenceNextStep": { "heading": "So what should you do about it", "body": "Run the prompting baseline before anyone builds anything, and write the number down. It costs a few days, it is the only thing that makes any later comparison meaningful, and in our experience it settles more architecture arguments than any amount of design discussion. If the baseline is close to acceptable, the answer is prompting plus retrieval and the fine-tuning conversation quietly goes away, which saves you a great deal. If it is a long way off, you now have a specific, measured gap to describe to whoever you take it to, including us. Either way you own the evaluation set at the end of it, which is the durable asset in this whole exercise." }, "demandSignal": "The comparison has been run properly and the result is unfashionable. A controlled study of knowledge injection reported that retrieval consistently outperformed unsupervised fine-tuning, both for knowledge the model had seen in training and for knowledge that was entirely new, and that models struggle to learn new factual information through unsupervised fine-tuning at all. Separately, work on long inputs found that a model's use of its context degrades significantly when the relevant passage sits in the middle of it, which is the quiet limit on the assumption that a large enough context window makes the question disappear. Neither finding is obscure, and neither tends to reach the meeting where the decision is made.", "whyItStalls": [ { "heading": "The question is asked as a technology choice when it is a knowledge-freshness question", "body": "Whoever read the most recent article proposes the approach, and the discussion becomes a preference argument nobody can settle with evidence. The question that actually decides it is duller: how often does the correct answer change, does it differ by who is asking, and does anyone have to be able to see where it came from. Answer those three and the architecture is largely determined before anyone has named a technique." }, { "heading": "Fine-tuning is chosen to teach facts, which is what it is worst at", "body": "The intuition is that a model trained on your documents will know your business, and the published comparison says otherwise: retrieval outperformed unsupervised fine-tuning for knowledge injection across the board, and models struggled to learn new factual information that way at all. The cost of learning this yourself is a training run, an evaluation, and a quarter. It is the most common expensive mistake in this decision and it is entirely avoidable by reading one paper." }, { "heading": "Nobody costed the second training run, or the fifth", "body": "Fine-tuning is not a project with an end, it is a standing obligation: curating and maintaining the training data, retraining when the base model is deprecated, re-running the evaluation each time, and holding a version history you can explain to whoever asks how a particular decision was reached. Programmes that costed the first run and not the lifecycle discover the difference when a provider retires the base model on a timetable that is not theirs." }, { "heading": "Prompting is dismissed as unserious and then quietly does most of the work", "body": "Careful prompting with a good retrieval design covers a surprising share of enterprise use cases, and it is dismissed because it does not sound like engineering. The consequence is that nobody establishes the prompting baseline, so no later comparison has anything to be measured against, and the more expensive option cannot be shown to be worth its cost even when it is." }, { "heading": "Nobody wrote down what would make them change their mind", "body": "A decision taken with no stated trigger for revisiting it becomes an identity, and defending it becomes somebody's job. Write down the conditions that would flip the choice, the measurement that would detect them, and the date the decision gets re-examined, at the moment it is taken. In a field where the base models change quarterly, a decision with no expiry is a decision you will be living with long after its reasoning expired." } ], "approach": [ { "heading": "Sort the requirement into knowledge, behaviour, format and cost", "body": "Four buckets, and most requirements split across them rather than sitting in one. Knowledge is what the system needs to know that the model does not: retrieval. Behaviour is how it should reason, in what style, following which house conventions: prompting first, fine-tuning if prompting cannot hold it. Format is the shape of the output: schemas and structured output before anything else. Cost is whether a smaller model can be made to do the narrow job: the one place fine-tuning has a clear economic argument. Doing this sort in a room with the business owner takes an afternoon and prevents a quarter of drift." }, { "heading": "Establish the prompting baseline first, and make everything answer to it", "body": "Before any architecture, run the evaluation set against a strong model with careful prompting and no other machinery, and record the number. That baseline is the thing every later option has to beat by enough to justify its cost and its lifecycle. Without it, teams compare a new approach against an impression, and an impression always loses to something that was built with more effort." }, { "heading": "Use retrieval where the answer must be current, attributable or permission-bound", "body": "Three tests, and any one of them points at retrieval. If the correct answer changes on a timescale shorter than your retraining cycle, it must be retrieved. If a reader has to be able to open the source, it must be retrieved. If two people asking the same question should get different answers because they are entitled to see different things, it must be retrieved, because a model's parameters cannot be permission-checked.", "links": [ { "label": "Retrieval and permission-aware access, in full", "href": "/guides/retrieval-rag-and-permission-aware-knowledge-access" } ] }, { "heading": "Reserve fine-tuning for form and cost, and say exactly what it buys", "body": "Legitimate reasons to fine-tune: a consistent output shape prompting cannot hold reliably, a house style or tone, a narrow classification task with plentiful labelled examples, or the same job done acceptably by a smaller and cheaper model at volume. Each of those is measurable before and after. If the case for a fine-tune is that the model will know more about the company, the case is wrong and the research says so." }, { "heading": "Keep one evaluation set across every option", "body": "The same cases, the same scoring, the same thresholds, whichever approach is being tested. Teams that build a new evaluation for each candidate are comparing three systems on three different exams and then choosing a winner. The evaluation set is also the asset that outlives the decision: it survives model upgrades, supplier changes and the approach itself.", "links": [ { "label": "How to build the evaluation set", "href": "/guides/agent-evaluation-and-assurance" } ] }, { "heading": "Cost the whole life, not the experiment", "body": "For each option: the build, the per-run inference, retrieval and index refresh where it applies, training and retraining where it applies, human review of exceptions, monitoring, and a full re-evaluation on every model upgrade. Our audits produce an estimated run cost per candidate workflow before anything is committed, because this is the number that decides between designs and it is almost always the one missing from the comparison.", "links": [ { "label": "Building the case a finance function will accept", "href": "/guides/business-case-for-an-agentic-programme" } ] }, { "heading": "Write the decision down with the conditions that would reverse it", "body": "One page: what was chosen, what it was measured against, what it costs to run, what would change the answer, and when it gets re-examined. That page is what stops the same argument being had every quarter by different people, and it is what lets a new engineer understand why the system is shaped as it is without inferring it from the code." } ], "providerNotes": { "heading": "Where the model provider genuinely matters here", "body": "This is one of the places where provider choice is not cosmetic. Whether fine-tuning is offered at all, on which models, at what price, with what hosting and residency position, and how long a fine-tuned model stays supported once the base model moves on, differ substantially and change the arithmetic rather than the aesthetics. So does cost per token, which decides whether a retrieval design that passes a large context is affordable at your volume. We would test a shortlist against your own data rather than against a public benchmark, and we would get the deprecation and support position in writing before anyone commits to a fine-tune." }, "engageVia": { "slug": "agentic-proof-of-concept", "label": "Agentic Proof of Concept" }, "faq": [ { "question": "Do we need to fine-tune or use RAG?", "answer": "For getting your own knowledge into the system, retrieval. Tenhaw builds these systems as prompting plus retrieval, including the document pipeline and semantic layer delivered for Placing Platform Limited (PPL). A controlled comparison of knowledge injection found that retrieval consistently outperformed unsupervised fine-tuning, both for knowledge the model had seen in training and for knowledge that was entirely new, and that models struggle to learn new facts through unsupervised fine-tuning at all. Retrieval also gives you three things fine-tuning cannot: content that is current without retraining, answers that cite a source the reader can open, and access control applied per user at query time. Fine-tuning earns its place for behaviour, format and cost, not for facts." }, { "question": "When is fine-tuning actually the right answer?", "answer": "Four cases, and they are all about form or money rather than knowledge. When you need an output shape or house style that prompting cannot hold reliably across the long tail. When you have a narrow classification or extraction task with plenty of labelled examples and a stable definition of correct. When volume is high enough that running a smaller, cheaper, fine-tuned model beats a large general one on cost at the same measured quality. And when latency matters enough that a smaller model is the only way to meet the budget. In all four the case is measurable before you commit, and that is the test. If you cannot state the number that would prove it worked, it is not the right answer yet." }, { "question": "Is prompting on its own enough?", "answer": "More often than the market implies, and you should find out before spending anything else. Careful prompting against a strong model, with structured output and a well-designed retrieval step, covers a large share of enterprise use cases, and it is the only option with no training lifecycle attached. Tenhaw took Placing Platform Limited (PPL) from PDFs to business intelligence on Azure in a two-week proof of concept, on ground it had circled for roughly a year, with no training run anywhere in it. You establish it first not to save money but because without a baseline you have nothing to measure the expensive options against, so you cannot show they were worth their cost even when they were." }, { "question": "Do million-token context windows make RAG obsolete?", "answer": "No. Published work on long inputs found performance is highest when the relevant information sits at the beginning or the end of the context and degrades significantly when a model has to use something in the middle, including in models built for long contexts, so a larger window is not the same as reliable attention across it. Cost and latency also scale with what you put in the window, on every call. And a context window solves none of the reasons an enterprise needs retrieval in the first place: permissions per user, freshness without a rebuild, and an answer that cites a source. Retrieval is what keeps the context small, current and defensible." }, { "question": "What does fine-tuning commit us to after the first run?", "answer": "A lifecycle rather than a deliverable. Training data has to be curated, kept current and governed, because it is now part of how your system behaves. Every base model deprecation forces a retrain on the provider's timetable rather than yours. Every retrain forces a full re-evaluation, so you need the evaluation set anyway. And you carry a version history, because when someone asks why a decision came out as it did in March, the answer involves which model version was live. None of that is a reason not to fine-tune. It is a reason to make sure the case is about form or cost, where the benefit is durable, rather than about facts, where it is not." }, { "question": "How do we choose without running a three-way bake-off?", "answer": "By sorting the requirement first, which is an afternoon rather than a quarter. Split what you need into knowledge, behaviour, format and cost. Anything in the knowledge bucket that has to be current, attributable or permission-bound is retrieval, and that is settled. Everything else starts at prompting with structured output, because that is the cheapest thing that can work and it establishes the baseline. Only what is left after that, and only where the measured gap is large enough to justify a training lifecycle, is a fine-tuning candidate. A full bake-off is worth running for one decision at most, and usually the sorting has already made it unnecessary." }, { "question": "Can we combine them?", "answer": "Yes, and most systems that work in production do. A typical shape is careful prompting for the reasoning and the house conventions, retrieval for anything that has to be current or permission-bound, structured output for the shape, and possibly a smaller fine-tuned model handling one high-volume narrow step inside the workflow where the economics justify it. The discipline that makes combining safe is holding the evaluation set constant across every configuration, so you can attribute a change in the score to the thing you changed rather than to the general direction of travel." }, { "question": "Do we need to fine-tune to get consistent JSON output?", "answer": "Almost never, and the cheaper route is worth exhausting first. Output shape belongs in the format bucket, where schemas and structured output come before anything else, because they are the direct way to make a model return the fields you asked for and they cost you an afternoon rather than a training lifecycle. Fine-tuning becomes reasonable only when careful prompting and a schema still cannot hold the shape reliably across the long tail, and that is a measurable claim rather than an impression. Run your evaluation set against the prompted version first and record the number. If a fine-tune then closes the gap, you can show exactly what it bought and what it commits you to afterwards." }, { "question": "Is RAG cheaper than fine-tuning?", "answer": "Over the life of the system usually yes, though they spend money in different places, so compare lifecycles rather than build quotes. Retrieval spends its money in the running, on per-run inference and retrieval plus the index refresh that keeps content current. Fine-tuning looks like a one-off and is not. You curate and maintain the training data, retrain when the base model is deprecated, and re-run the full evaluation every time. Prompting is cheaper than both, which is why the prompting baseline is worth establishing before either gets funded. Tenhaw publishes the method behind that lifecycle comparison in full, free to adopt without hiring anyone." }, { "question": "What should we ask a model provider before fine-tuning?", "answer": "Four things, and get the last one in writing. Tenhaw runs that comparison against a client's own data rather than against a public benchmark, and treats the support commitment as part of the price. Whether fine-tuning is offered at all, on which models and at what price, because that differs substantially between providers. Where a fine-tuned model is hosted and what the data residency position is. What cost per token looks like at your volume, since that decides whether a retrieval design passing a large context is affordable anyway. And how long a fine-tuned model stays supported, because once the base model is retired the retrain happens on the provider's timetable, not yours." }, { "question": "Can a readiness audit settle whether we need RAG or fine-tuning?", "answer": "It can, and the part it settles is the cost. A Tenhaw audit ends in working prototypes and an estimated run cost per candidate workflow before anything is committed. That number is what decides between designs, and it is almost always the one missing when a team argues retrieval against fine-tuning. The audit also forces the sorting into knowledge, behaviour, format and cost, with the business owner in the room rather than only the engineers. That sort takes an afternoon and the method behind it is published free to copy, so run it yourself first. Buy the audit when the decision has to survive a finance review." }, { "question": "We fine-tuned a model and it still gets facts wrong. Why?", "answer": "Most likely because it was asked to do the one thing it is worst at. The repair is retrieval work of the kind Tenhaw did on an engagement for Placing Platform Limited (PPL), scoring each extracted value's confidence from its source provenance, model certainty and a search cross-check, none of which a fine-tune can give you. A controlled comparison of knowledge injection found retrieval consistently outperformed unsupervised fine-tuning, for knowledge seen in training and for knowledge that was entirely new. Training on your documents does not reliably put those facts inside the model, and it gives you no citation and no per-user permission check either. Keep the fine-tune, if it is worth keeping, for output shape, house style or cost at volume." }, { "question": "A supplier wants to fine-tune on our documents. Is that a red flag?", "answer": "Not on its own, but three questions will tell you what it is worth. What is the prompting baseline, measured on your own evaluation set, that this is meant to beat, and by how much? If nobody has run one, the comparison is against an impression. Is the case about knowledge, or about form and cost, given that a controlled comparison found retrieval consistently outperformed unsupervised fine-tuning for knowledge injection? And how often does the correct answer change, because anything moving faster than your retraining cycle has to be retrieved rather than trained in. Tenhaw builds prompting and retrieval systems, and where the honest answer is a specialist model team we will point you at one." }, { "question": "How do we run a prompting baseline, and how long does it take?", "answer": "A few days, and it is the highest-value few days in the whole decision. You need an evaluation set of real cases with agreed scoring, then you run them against a strong model with careful prompting and no other machinery, and you write the result down. That number is what every later option has to beat by enough to justify its cost and its lifecycle. Tenhaw pairs on this rather than running it offstage, which is why, on a two-week proof of concept for Placing Platform Limited (PPL), the client's own engineer reported 70% confidence that they could run the process unaided. You also come out of it owning the evaluation set, which is the durable asset in the whole exercise." }, { "question": "Who should be in the room for a RAG versus fine-tuning decision?", "answer": "The business owner above all, and whoever can answer for the budget. The sort is not really technical. You are splitting the requirement into knowledge, behaviour, format and cost, and the questions that settle it are how often the correct answer changes, whether it differs depending on who is asking, and whether anyone has to be able to see where an answer came from. Only the business side can answer those, which is why an engineering-only room turns the decision into a preference argument nobody can settle with evidence. Done together it takes an afternoon and prevents a quarter of drift, because the architecture is largely determined once those three answers are on the table." }, { "question": "Does being in a regulated industry change the RAG or fine-tuning answer?", "answer": "It sharpens it rather than changing it, and it usually pushes further towards retrieval. Tenhaw's document work for Placing Platform Limited (PPL) was retrieval and semantic-layer engineering, not a fine-tune. Two of the three tests that point at retrieval are regulatory. If a reader has to be able to open the source behind an answer it has to be retrieved, and if two people asking the same question are entitled to see different things it has to be retrieved, because a model's parameters cannot be permission-checked. Fine-tuning also brings an evidence obligation, because you carry a version history for the day someone asks why a decision came out as it did in March, and the answer involves which model version was live." }, { "question": "If we change model provider later, does any of this work carry over?", "answer": "The evaluation set carries over, and it is the most durable thing you will build. Held constant across every option rather than rebuilt per candidate, an evaluation set survives model upgrades, supplier changes and the approach itself. Prompts and retrieval design are your work product rather than the provider's, though you re-run the full evaluation on any model change regardless. A fine-tune is the piece that does not travel. When the base model is retired you retrain on the provider's timetable, which is why the support position is worth getting in writing first. Tenhaw writes engagements the same way, with the client owning all work product and the exit date agreed at kickoff." }, { "question": "How long does a RAG or fine-tuning decision stay valid?", "answer": "Only as long as the conditions it was taken under, which is why you write those down at the time. One page: what was chosen, what it was measured against, what it costs to run, what would change the answer, and the date it gets re-examined. In a field where the base models change quarterly, a decision with no expiry is one you will be living with long after its reasoning expired, and with no stated trigger for revisiting it, it becomes an identity somebody has to defend. That page is also what lets a new engineer see why the system is shaped as it is without inferring it from the code." } ], "citations": [ { "name": "Ovadia, Brief, Mishaeli and Elisha, Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs (2023)", "url": "https://arxiv.org/abs/2312.05934", "note": "Source of the headline finding: retrieval consistently outperformed unsupervised fine-tuning for knowledge injection, for knowledge seen in training and for entirely new knowledge, and models struggle to learn new factual information through unsupervised fine-tuning." }, { "name": "Liu et al., Lost in the Middle: How Language Models Use Long Contexts (2023)", "url": "https://arxiv.org/abs/2307.03172", "note": "The limit on the argument that a large context window replaces retrieval: performance degrades significantly when the relevant information is in the middle of a long input." }, { "name": "Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)", "url": "https://arxiv.org/abs/2005.11401", "note": "The original statement of why an explicit, non-parametric memory is used alongside a model's parametric one." } ] } ## The business case for an agentic programme: ROI, payback and what to measure Source: https://tenhaw.com/guides/business-case-for-an-agentic-programme Last updated: 2026-11-05 { "slug": "business-case-for-an-agentic-programme", "name": "The business case for an agentic programme: ROI, payback and what to measure", "metaTitle": "The business case for an agentic programme", "metaDescription": "Build an AI business case with a baseline, benefit owner, operating cost and sensitivity cases. Separate released capacity from cash savings.", "tagline": "Show what the investment buys and which assumptions could change the decision.", "summary": "An AI business case separates cost avoided, capacity released and revenue or loss changed. State the baseline, benefit owner and calculation for each, then price the build, operation, review and support. A useful case shows what evidence would change the decision. The worksheet below keeps capacity estimates separate from cash savings.", "theme": "case", "evidenceBasis": "approach", "evidenceStatement": "We produce an investment case as part of every AI Readiness Audit, with the value stated in currency, a confidence attached to each figure, and the arithmetic laid out so your finance function can rework it with their own numbers. The prices on this page are our own published ones, which you can check against the rate card, and the research figures are third-party findings with their sources attached. If a supplier shows you a return figure, ask whose process it came from, who verified it, and what the denominator was.", "evidenceNextStep": { "heading": "So what should you do about it", "body": "Build the cost side before the benefit side, because it is the half you can actually know. The build price is quotable from published rates, and ours are on the pricing page. The run cost is estimable per workflow before anything is committed, and it is the line most cases are missing entirely. Only then argue about the benefit, and argue about it with the process owner rather than with a supplier, because the supplier does not know whether your headcount will actually change. If you want the whole thing done properly that is what the audit is: four weeks, a fixed fee of £44,000, a costed plan in three-month increments capped at twelve months, a do-not-do list, and an investment case your board can act on. If you already have a case and want it stress-tested rather than written, say so on the call, because that is a shorter and considerably cheaper conversation." }, "demandSignal": "The UK government's own appraisal guidance is the most useful public document a finance director can read on this, and it is free. The Green Book requires appraisers to account for optimism bias, which it defines as the proven tendency for appraisals to be over-optimistic about key assumptions, by increasing estimates of cost and duration and decreasing estimates of benefit. It sets a business case out as five interconnected perspectives, strategic, economic, commercial, financial and management, and it is explicit that these are not five separate documents. And it puts benefits realisation, the plan for monitoring costs and actually realising the stated benefits, inside the management case rather than leaving it as something that happens afterwards. Almost no AI business case we have been shown does any of those three things.", "whyItStalls": [ { "heading": "The case is built on hours saved, and hours saved are not money", "body": "An hours figure is a statement about effort, not about cash, and it becomes money only when a headcount changes, a contract is renegotiated, or a capacity constraint that was costing you revenue is released. From our own history, the AI Voice Insights proof of concept our founder led at HSBC was projected to remove more than 1.5 million hours of manual administration a year. That figure is a projection, it was never realised, and the work was a proof of concept rather than a rollout. It is exactly the shape of number a finance function should refuse to accept on its own, including from us. The follow-up question is the whole job: whose budget line moves, by how much, and in which quarter." }, { "heading": "Optimism bias is in the numbers and not in the arithmetic", "body": "Every estimate in an AI case is made by people who want the programme to happen, which is normal and is precisely why public-sector appraisal guidance requires an explicit adjustment for it. The Green Book defines optimism bias as the proven tendency for appraisals to be over-optimistic about key assumptions, and requires appraisers to increase their estimates of cost and duration and reduce their estimates of benefit. A case with no such adjustment is not neutral, it is optimistic by default, and the reviewers who eventually find that out will discount everything else in the document too." }, { "heading": "The run cost was never estimated, so the payback is measured against half the cost", "body": "The build is the visible number and the run is the one that decides whether the thing survives its second year: per-run inference and retrieval, human review of everything the system routes to a person, monitoring and evaluation, and a full re-evaluation each time a model version changes underneath you. Cases that price only the build produce a payback period that is wrong in the direction the sponsor wanted, and the correction arrives during the year when the pilot is supposed to be scaling." }, { "heading": "It was priced as a build and it lives as an operation", "body": "The sponsor who funded an experiment is rarely the person who will carry a live system, its incidents and its audit trail, and in most organisations nobody has been asked to, so a successful pilot with no named owner for its run budget becomes an orphan. That is a finance problem before it is a technology problem, because the fix is a line in next year's plan with a name against it, agreed before the build rather than negotiated after the demo." }, { "heading": "Duration is the largest risk in the case and it is not priced", "body": "Analysis of 1,355 public-sector IT projects, averaging $130m and 35 months, found that every additional year of duration added about 4.2 percentage points to expected cost overrun, and that 18% of projects were outliers with cost overruns above 25%. A two-year agentic programme carries that risk whoever delivers it, and that includes us. The implication for the case is not that long programmes are forbidden, it is that duration belongs in the risk line with a number attached rather than in the plan as a neutral fact.", "links": [ { "label": "The published evidence on programme duration and team size", "href": "/compare/big-4-consultancies" } ] }, { "heading": "Nobody agreed who books the benefit", "body": "A benefit with no owner does not appear in anybody's budget, so it is never checked, and the programme's actual result is decided by whoever tells the story most confidently at the end. Name the person whose numbers move, get them to agree the measurement and the date before the build starts, and accept that a benefit nobody is willing to own is usually a benefit nobody believes in." } ], "approach": [ { "heading": "Start with one process and its cost in currency, before any technology", "body": "Volume, elapsed time, the people involved, the error and rework rate, and what the whole thing costs to run today. That is a fortnight of work and it is the foundation of everything else, because a benefit is a difference between two numbers and most cases only ever produce the second one. It also frequently changes which process you would attack, which is worth knowing before you fund the wrong one.", "links": [ { "label": "Why enterprise AI is a process question before it is a model one", "href": "/guides/end-to-end-agentic-workflow-implementation#enterprise-ai" } ] }, { "heading": "Split the benefit into three kinds and treat only one of them as money", "body": "Cost avoided is money when a headcount, a contract or a licence actually changes, and not before. Capacity released is money only if the released capacity is redeployed to something with a value attached, which is a decision somebody has to take and record. Revenue or loss changed, better conversion, fewer claims leaking, lower fraud losses, is the strongest kind and the hardest to attribute. Label every line as one of the three and the case becomes readable by a sceptic, which is the only kind of reader worth writing for." }, { "heading": "Attach a confidence to every line, and hand the model to finance", "body": "Each figure carries the assumption it rests on and how confident we are in it, and the whole thing is handed over in a form your finance function can rework with their own numbers rather than as a PDF. This is what the investment case in our audit deliverable is: the value in currency, the assumptions exposed, the confidence stated, the arithmetic open. A case nobody can rework is a case nobody can check." }, { "heading": "Cost the run before the build is approved", "body": "An estimated run cost per candidate workflow, produced during the audit rather than discovered in year two, covering inference and retrieval per run, the human review the routing design actually implies, monitoring, and re-evaluation on model change. Where that number is uncomfortable it usually redirects the design rather than killing the case, which is exactly what it is for." }, { "heading": "Adjust for optimism bias on the record, and say by how much", "body": "Increase the cost and duration estimates, reduce the benefit estimates, and state the adjustment as a visible line rather than quietly baking it in. Two reasons. It is what serious appraisal guidance requires, so your reviewers recognise it. And a case that has already discounted itself is far harder to argue with than one that has not, which is a negotiating advantage rather than a concession." }, { "heading": "Keep the unit of work small and the clock short, because duration is priced risk", "body": "Sequence the programme as short fixed-price stages with real decision points between them, rather than as a two-year commitment with milestones. Ours are built that way for this reason: an audit fixed over four weeks, a proof of concept fixed over two to four, and productionisation scoped at four to six weeks with a dedicated team. Whoever you buy from, a case built on stages you can stop is worth more than a case built on a plan you have to believe.", "links": [ { "label": "What the research says about long programmes", "href": "/compare/big-4-consultancies" } ] }, { "heading": "Price the first year against a published rate card rather than a range", "body": "One worked sum, on our own published prices, because a finance director reading this deserves at least one. An AI Readiness Audit is a fixed £44,000 over four weeks. An Agentic Proof of Concept is a fixed £20,000 to £55,000 over two to four weeks. Buying both, a diagnosis and one working thing, is therefore £64,000 to £99,000 before any production commitment is made. If the answer is then a build team of three at £70,000 to £85,000 a month, that is £840,000 to £1.02m across a full year, and productionising a single proof of concept is scoped at four to six weeks rather than a year of that. Those are published numbers and the arithmetic is checkable. What none of them tells you is the benefit. The audit goes first for exactly that reason, to put a defensible figure on the other side of the sum before the larger commitment is made.", "links": [ { "label": "The published rate card and every price on it", "href": "/pricing" }, { "label": "What the audit deliverable contains", "href": "/services/ai-readiness-audit" } ] }, { "heading": "Name the benefit owner and the measurement date before the build starts", "body": "One named person whose numbers move, one agreed measurement, one date in the diary, all fixed before anyone writes code. This is the same discipline as putting a named owner on the release gate, applied to the money instead of the risk, and it is what separates a programme that can prove its result from one that has to assert it.", "links": [ { "label": "Who receives a pilot once it works", "href": "/guides/agent-evaluation-and-assurance" } ] }, { "heading": "Write the do-not-do list and count it as return", "body": "The things not worth doing here, and why, are usually the most valuable page in an audit readout, and they belong in the business case rather than in an appendix. A programme that avoids a seven-figure commitment to the wrong workflow has produced a return that is real, immediate and considerably more certain than any of the benefits above. Finance functions understand avoided spend better than anyone else in the building, so write it in their language." } ], "engageVia": { "slug": "ai-readiness-audit", "label": "AI Readiness Audit" }, "faq": [ { "question": "What is the ROI of an agentic AI programme?", "answer": "Nobody can tell you without looking at your process, and a supplier quoting a figure beforehand is quoting somebody else's business. Cost avoided becomes money only when a headcount, contract or licence changes. Capacity released becomes money only once someone decides where it goes. Revenue or loss changed is strongest and hardest to attribute. Against those sit a build cost, quotable from published rates, and a run cost: inference, retrieval, human review, monitoring and re-evaluation on every model change. A case naming all three benefit types, pricing both and adjusting for optimism bias is defensible; one built on hours saved is not." }, { "question": "How do you build a business case for AI agents?", "answer": "Six steps and none of them starts with technology. Measure the current process in currency: volume, elapsed time, people, error and rework, total cost to run today. Estimate the future state and split the difference into cost avoided, capacity released, and revenue or loss changed, labelling each. Attach a confidence and the underlying assumption to every line. Price the build from a published rate card and the run per workflow, before approval rather than after. Adjust explicitly for optimism bias by increasing costs and duration and reducing benefits, and show the adjustment. Then name the benefit's owner and the date it gets measured. Tenhaw publishes the method behind these steps in full, free to adopt, so a team can work through them without hiring anyone." }, { "question": "What is the payback period on an agentic proof of concept?", "answer": "A proof of concept is bought to remove uncertainty rather than to pay back; treating it as an investment with a return is how organisations end up defending a two-week build as though it were a product. It buys a measured answer on whether the workflow can be done, an accuracy read on real data, and an estimated run cost, which together decide whether the larger production commitment is worth making. If the answer is no, it has paid for itself several times by preventing that commitment. Tenhaw's proof of concept for Placing Platform Limited (PPL) took PDFs to business intelligence on Azure in two weeks, ground the business had circled for roughly a year, and the client owned the work product either way." }, { "question": "Which running costs belong in an agentic business case?", "answer": "Any invented number would describe Tenhaw's workloads rather than yours, so it estimates a run cost per candidate workflow before anything is committed to. The lines are consistent and you can build the figure yourself. Per-run model cost, driven by the number of turns the agent takes and the tokens in its context each time. Retrieval and index maintenance, including re-embedding when the corpus or embedding model changes. Human review, set by your routing design rather than the technology, and usually the largest line in a regulated process. Monitoring and observability. And re-evaluation on every model version change, a recurring cost most plans treat as a one-off, and the line that decides between designs." }, { "question": "Should we count hours saved as savings?", "answer": "Not as savings, no. Count them as a measurement of the change, then do the second piece of work and establish what happens to those hours. If a fixed-term contract ends or a vacancy goes unfilled, that is cash and it belongs in the case with the date it lands. If the time returns to people who then do something else, it is capacity released, and it is worth money only once someone decides what that something else is and attaches a value. If nothing changes, the hours are real and the money is not. Writing that distinction into the case protects you twice, since it survives scrutiny and stops the programme being judged in a year against a saving that was never going to appear in a ledger." }, { "question": "How do you stop the business case being over-optimistic?", "answer": "By adjusting for it deliberately and visibly, which is what public-sector appraisal guidance has required for years. The Green Book defines optimism bias as the proven tendency for appraisals to be over-optimistic about key assumptions and instructs appraisers to increase their estimates of cost and duration and decrease their estimates of benefit. Three practical habits follow. Show the adjustment as its own line, so reviewers can see it rather than hunt for it. Have someone outside the programme write the pessimistic case, not the sponsor. And treat duration as a priced risk rather than a scheduling detail, because the published evidence on IT programmes is that cost risk rises with every additional year of runtime." }, { "question": "What should finance measure after go-live?", "answer": "Four things, monthly, and only the first is about the technology. Run cost per unit of work, against the estimate made before approval, because that is where an unpleasant surprise shows up first. Volume actually processed by the system rather than volume it is capable of processing, since adoption is usually the constraint. The exception rate and therefore the human review hours the system is really consuming, which is the line that quietly eats the benefit. And the benefit itself, measured on the agreed date by the named owner, against the figure in the approved case rather than against a revised one. If the fourth measurement has no owner and no date, the first three will be reported and the programme's result will never actually be established." }, { "question": "Who should own benefits realisation?", "answer": "The person whose budget or performance numbers move, which is almost never the person who sponsored the technology. Get them to agree the measurement and the date before the build starts, because agreeing it afterwards is a negotiation and agreeing it beforehand is a specification. The same person should also own the run budget, since a benefit owner with no cost exposure has an incentive to be generous. Where no such person exists, surface that immediately. A process nobody owns end to end is a programme risk long before it is a measurement problem, and it usually means the first piece of work is establishing the ownership rather than building anything." }, { "question": "Should we fund an AI programme in stages or as one multi-year commitment?", "answer": "In stages, with a real decision point between them, because duration is the largest unpriced risk in most AI business cases. Short fixed-price stages turn the commitment into a sequence of small ones. Tenhaw's founder co-designed a target operating model at HSBC Global Payment Solutions, 500 teams against a $450M portfolio, piloted first, with James’s engagement complete and the wider rollout in progress. Analysis of 1,355 public-sector IT projects, averaging $130m over 35 months, found that every additional year of duration added about 4.2 percentage points to expected cost overrun, and that 18% overran by more than 25%. That risk applies whoever delivers, Tenhaw included. A case built on stages you can stop is worth more than a plan you have to believe." }, { "question": "What does the cost side of an agentic business case look like?", "answer": "Build the cost side first, because it is the half you can actually know. It is made of three lines you can know before approval: a fixed-price diagnosis, a fixed-price proof of concept if the diagnosis warrants one, and a build team charged by the month if the proof of concept earns it. Tenhaw fixes each stage separately, agrees the exit date at kickoff and works on 30 days' notice either way, so the commitment can be stopped between stages rather than only forecast. Estimate the run cost per workflow separately, then argue about the benefit." }, { "question": "What should we ask a supplier who quotes an AI ROI figure?", "answer": "Three questions, in this order: whose process the figure came from, who verified it, and what the denominator was. A return quoted before anyone has looked at your workflow is describing somebody else's business, because it is the shape of your process, its volume, its error rate and the review it needs, that decides whether any of it transfers. Then ask whether the figure was realised or only projected, and if it was realised, what baseline it was measured against and who agreed that baseline before the work started. Arithmetic you can rework beats a headline you cannot, whoever is presenting it." }, { "question": "Can you stress-test a business case we have already written?", "answer": "Yes. A stress test asks what a sceptical reviewer will ask. Is every benefit labelled as cost avoided, capacity released, or revenue or loss changed? Does anything rest on hours saved without a headcount, a contract or a capacity constraint actually moving? Is the run cost priced beside the build, or is the payback measured against half the cost? James Rooney runs that review personally, as he does every Tenhaw engagement, and reviewing an existing case is shorter work than writing one from scratch. What usually changes is not the conclusion but the size of the number and the confidence attached to it, which is what makes a case survive its second reading." }, { "question": "Does the Green Book five case model apply to an AI business case?", "answer": "The discipline transfers, and the Green Book is the most useful free document a finance director can read on this even though it was written for public-sector appraisal. It sets a business case out as five interconnected perspectives, strategic, economic, commercial, financial and management, and is explicit that these are not five separate documents. It also puts benefits realisation, the plan for monitoring costs and actually realising the stated benefits, inside the management case rather than treating it as something that happens afterwards. For an agentic programme that is exactly where the run cost owner and the measurement date belong, and it is the section Tenhaw most often finds empty." }, { "question": "How long does it take to put an AI business case together?", "answer": "Measuring one process properly, in currency and as it actually runs today, is about a fortnight of work. That fortnight is the foundation of everything else, because a benefit is the difference between two numbers and most cases only ever produce the second one. Tenhaw's own version of that job runs four weeks and covers several candidate processes with an estimated run cost each, a costed plan in three-month increments capped at twelve months and a do-not-do list. What stretches the timeline is rarely the arithmetic. It is getting the person whose numbers move to agree the measurement." }, { "question": "Can our finance team rework the numbers in an AI business case?", "answer": "They should be able to, because a case nobody can rework is a case nobody can check. Every figure should carry the assumption it rests on and a stated confidence, and the whole thing should arrive in a form your finance function can put its own numbers into rather than as a PDF. Tenhaw hands that model over as work product the client owns outright, so your team can take it apart without asking anyone's permission. That matters because the assumptions most worth changing are yours: the fully loaded cost of an hour, the discount rate you apply, what the rework rate really is once somebody measures it. Finance revising the figures downwards is a good outcome rather than an argument." }, { "question": "Does deciding not to build something count as a return?", "answer": "Yes, and it is usually the most certain return in the document. A programme that avoids a seven-figure commitment to the wrong workflow has produced something real and immediate, which no projected benefit sitting two years out can match. So write the do-not-do list, the things not worth doing here and why, into the business case itself rather than into an appendix, and put the avoided commitment in currency beside the benefits you are claiming. Finance functions understand avoided spend better than anyone else in the building, so it is worth writing in their language. In audit readouts that list is often the most valuable page." }, { "question": "How do we choose which process to build the first business case around?", "answer": "Measure the candidates before you choose, rather than choosing and then justifying. For each one: volume, elapsed time, the people involved, the error and rework rate, and what it costs to run today. That exercise frequently changes which process you would attack, which is worth knowing before you fund the wrong one, and it produces the first of the two numbers any benefit is made from. Favour a process where you can name the person whose budget or performance numbers move if it improves, because a benefit nobody is willing to own is usually a benefit nobody believes in. A process you cannot measure today is not a first candidate, it is a measurement job." }, { "question": "Should the firm building the system also write the business case?", "answer": "They can build the cost side, because that half is quotable from a published rate card and estimable per workflow before anything is committed. The benefit side is different. A supplier does not know whether your headcount will actually change, which contracts can be renegotiated, or where released capacity would be redeployed, so those figures belong with the process owner rather than whoever is selling the build. Tenhaw reports any month that delivers no measurable value as a failed month, which is accountability a supplier can carry; a forecast of your budget is not. The practical split is the build price and run cost estimate from the firm that would deliver it, and the benefit numbers from the person inside your organisation whose budget moves if the workflow improves." } ], "citations": [ { "name": "HM Treasury, The Green Book (2026): appraisal and evaluation in central government", "url": "https://www.gov.uk/government/publications/the-green-book-appraisal-and-evaluation-in-central-government/the-green-book-2026", "note": "Source of the optimism bias definition and the instruction to increase cost and duration estimates and decrease benefit estimates, of the five case model, and of the requirement that the management case sets out plans for monitoring costs and realising benefits. Written for public-sector appraisal, and the discipline transfers." }, { "name": "Budzier and Flyvbjerg, Overspend? Late? Failure? What the Data Say About IT Project Risk in the Public Sector (2013)", "url": "https://arxiv.org/abs/1304.4525", "note": "1,355 public-sector IT projects, average budget $130m over 35 months. Each additional year of duration added about 4.2 percentage points to average cost risk, and 18% were outliers with cost overruns above 25%." } ] } ## How to define and price a business outcome Source: https://tenhaw.com/the-tenhaw-way/how-to/set-an-outcome Last updated: 2026-11-05 { "slug": "set-an-outcome", "group": "Setting up the work", "title": "How to define and price a business outcome", "summary": "Give the business result a baseline, a value and an owner, then show how you will check it.", "intro": "An outcome is the top of the work breakdown: a business result carrying a target in currency, a baseline you can measure today, arithmetic anyone can re-run, and one named person accountable for the number. Good looks like a single sentence a finance director and an engineer would both recognise as true, priced in one visible calculation, with the validation query written and tested before anything is committed. Outcomes can span quarters. Every epic, story and chapter underneath exists only because it traces back to one.", "whenToUse": "Whenever the business wants something new started, before anyone writes an epic or a ticket. Also when you inherit a roadmap nobody can trace to a number, because reconstructing the outcome behind work already in flight is the fastest way to find out what should be cancelled.", "totalTime": "PT6H", "timeLabel": "About six hours, across a few sittings", "tools": [ "The reporting system the baseline comes from", "The tracker the outcome and its epics live in" ], "steps": [ { "name": "Write the result, not the feature", "detail": "Start from what changes for the business, not what gets built. \"Rebuild the checkout\" is a feature. \"More of the customers who reach checkout complete their order\" is a result. Write one sentence naming who benefits, what changes for them, and which business metric moves in which direction. Then apply two tests. Strike out every technology, vendor and screen name: if the sentence stops making sense, you have written a solution. Ask whether it would still be true if you delivered it a completely different way. If not, you have priced a plan rather than a result, and the plan will change." }, { "name": "Pull the baseline before you name a target", "detail": "Get the current number yourself, with whoever owns the report sitting there. Record four things: the figure, the exact date range, the filters applied (internal traffic, bots, refunds, test orders), and the query or report anyone can re-run. Then re-run it a week later. If it returns a different number, you do not have a baseline, you have a screenshot. Where no baseline exists, your first epic is instrumenting it and the outcome stays in Exploring until the figure arrives. A commitment built on a baseline nobody can reproduce fails validation loudly, in front of the people whose trust you were trying to earn." }, { "name": "Price it in one line of checkable arithmetic", "detail": "Convert the movement into money in one visible line: baseline volume, expected change, value per unit, margin, period. Write the calculation into the outcome rather than presenting a figure that arrived from nowhere. Take margin and lifetime value from finance's numbers, not your own, and name whose they are. State the period explicitly, because a million annualised and a million in-quarter are different commitments. Then flex the two assumptions you are least sure of by a third each. If the target no longer justifies the quarter, it is a coin toss and the owner needs to know that before signing." }, { "name": "Name one owner who will carry the number", "detail": "One name, on the business side, in whose forecast the number lands. Not a committee, not a function, not the delivery team. The owner supplies the assumptions, agrees the baseline, and is the person who says the value did not land if it did not. Test it with one question: will you carry this number in your own plan for the year? A yes with caveats means write the caveats down and price them. A no means the business does not believe the number, which is worth finding out a quarter before you spend one. Delivery owns whether the epics ship; the owner owns whether they were the right epics." }, { "name": "Write and run the validation query before committing", "detail": "State which report, which segment, which comparison window, and what size of movement counts as signal rather than noise. Then run it today against the pre-change period. If it will not execute now, it will not execute at the monthly outcome validation either, and validation runs every month on every live outcome. Say how long the signal takes to accumulate: an epic returning forty thousand a month cannot be judged in three weeks. Finally, write down in advance the condition that would make you call this not realised. Agreeing the failure condition while everyone is optimistic is far easier than agreeing it afterwards." }, { "name": "Set the horizon, then plan headroom", "detail": "Name the quarters the outcome spans, and the single quarter each epic sits in, because epics cannot cross a roadmap boundary. Then plan more epic value than the target. Derive the ratio from your own history: value realised divided by value planned across the last four closed quarters. At 70% realisation, a £1m target needs roughly £1.43m of planned epic value behind it. Re-derive that ratio every quarter, because it moves as the team and the domain change. Headroom is not sandbagging. It is the difference between a target that survives one epic underperforming and one that collapses when it does. In a regulated firm there is one addition. Where an outcome touches pricing, reserving or a customer outcome, the validation query has to be reproducible by someone who was not in the room, and the actuarial or compliance owner is named on the outcome alongside the business owner. Not as a reviewer at the end, as a named owner from the point the horizon is set." }, { "name": "Break it into epics and check the sum", "detail": "Each epic links to this outcome and no other, carries its planned share, and belongs to one quarter. Add the shares up: a £1m outcome with four £200k epics is a £200k hole, and it belongs in planning rather than week eleven. Then check for double counting, which the sum will not catch. Two epics both claiming the same conversion lift on the same traffic total £400k on paper and deliver £200k. Where epics move the same units, fix the landing order and price each on what the previous leaves behind. Close any gap by adding an epic, raising a share with a reason, or lowering the target today." }, { "name": "Commit at the gate, close on a reason", "detail": "Committed is not a label applied because the outcome appeared in a board pack. It means five things exist: a reproducible baseline, visible arithmetic, a named owner, a validation query that runs, and an epic breakdown that sums with headroom. Until all five exist it sits in Exploring. Review live outcomes monthly, and close each on a stated reason: all linked epics done, value realised, or accepted as not realised. Use the third whenever it is the true one. An operating model that has never closed an outcome as not realised is not being measured honestly, and everyone downstream of it works that out eventually." } ], "pitfalls": [ "Pricing the solution rather than the result. \"A new mobile app, worth £2m\" welds the value to one way of getting it, so when a cheaper route appears in week four the number cannot survive the change and the whole outcome has to be reopened. Price the result and let epics compete to deliver it.", "Reverse-engineering the number from the budget, or lifting it from a vendor deck or an industry benchmark. A figure built to justify a spend already decided, or borrowed from someone else's business, cannot be reconciled to your reporting, so the first monthly validation becomes an argument about the source instead of the result. If it did not come from your own data and your own finance team, it will not survive contact with validation.", "Committing before a baseline exists. Teams commit on the strength of the story and plan to sort measurement out later. Later arrives without the historical data you needed, and the outcome closes as unvalidated, which is worse than closing as not realised because you learn nothing you can calibrate against.", "Counting a cost saving twice. A saving is only currency if the cost leaves the business. Headcount redeployed onto other work is recovered capacity, not money, and booking it as money in the outcome and again in the budget is the fastest way to lose finance's trust in the whole roadmap. Say plainly which one it is.", "One outcome broad enough to absorb the entire quarter. If every team can claim to contribute, no epic's planned share means anything and the arithmetic stops working as a check. Several concurrent outcomes with tight targets and real owners beat one that everything hangs off." ], "definitionOfDone": [ "The outcome reads as a business result and still makes sense after striking every technology, vendor and screen name out of the sentence.", "The baseline has a figure, a date range, its filters and a re-runnable query, and someone other than you has re-run it and got the same number.", "The price is one visible line of arithmetic using finance's margin figures, the period is stated as in-quarter or annualised, and the two weakest assumptions have been flexed to see what breaks.", "One named person on the business side has said they will carry the number in their own plan, and their caveats are written down.", "The validation query has been run today against the pre-change period, and the condition for calling the outcome not realised is recorded.", "The linked epics sum to the target plus headroom derived from your own realisation ratio, no two epics count the same units twice, and any remaining gap is named and owned rather than left open." ], "example": { "heading": "Worked example: pricing a checkout outcome at an invented retailer", "body": "Northgate Supply is a made-up online retailer with roughly £11m of annual online revenue. Every number below is invented, and the point is the shape of the arithmetic rather than the figures. The business says the checkout is bad. Here is that turned into an outcome.\nResult: more of the customers who reach checkout complete their order.\nBaseline: 2.4% of sessions convert, measured over the twelve weeks to 30 June from the analytics property the growth analyst owns, excluding internal IP ranges and known bot traffic. 9.1m sessions a year, £52 average order value, 41% gross margin supplied by finance. It reconciles: 9.1m at 2.4% at £52 is £11.4m, which matches the online revenue line, so the baseline is not measuring a different population from the P&L.\nTarget movement: 2.4% to 3.1%, a 0.7 percentage point lift.\nArithmetic: 9.1m multiplied by 0.007 is 63,700 additional orders a year. At £52 that is £3.31m of revenue, and at 41% margin it is £1.36m of gross profit. The outcome is priced at £1.36m annualised. The in-quarter share is stated separately: the change goes live in week six of a thirteen-week quarter, so seven weeks of benefit, £1.36m divided by 52 multiplied by 7 is £183k.\nSensitivity: if margin is really 35% and the lift is 0.5 points rather than 0.7, the number falls to £830k. Still worth the quarter, so the target stands, and both figures go in front of the owner.\nOwner: the ecommerce director, named, who confirms she will carry the £1.36m in next year's plan.\nValidation: conversion rate by device, weekly, against the twelve-week pre-change period, with orders reconciled to the finance ledger rather than to analytics. Anything under a 0.3 point move is noise. Six weeks of post-live data before anyone calls it.\nNot realised if: there is no sustained movement above 0.3 points eight weeks after the last epic ships.\nEpics, all in Q3: guest checkout £620k, payment failure retry £510k, address autocomplete £340k. All three move the same conversion number, so they are priced in landing order and each carries only what it adds on top of the one before. The shares total £1.47m against a £1.36m target. The last four quarters realised 70% of planned value, so the headroom rule wants £1.94m behind this target. That is £470k short, and it is visible in planning: either a fourth epic goes in, or the owner is told today that the honest target is £1.03m." }, "secondExample": { "heading": "Worked example: pricing a claims document outcome at an invented insurer", "body": "Marlow Mutual is a made-up UK general insurer. Every number below is invented, and the point is the shape of the arithmetic rather than the figures. A checkout conversion rate does not transfer to an insurer, so this is the same method run on an operational cost base rather than a revenue line, and it works the harder half out loud: how much of a saving is actually money.\nResult: fewer claims documents need a human to read, key and file them before the claim can move.\nBaseline: 260,000 claims documents handled in the twelve months to 30 June, counted from the claims workflow system, excluding duplicates and internal re-scans. Average handling time of 11.0 minutes per document, taken from the same system's start and finish timestamps and reconciled against a two-week sample the claims operations manager sat through herself. The loaded hourly cost is £34, supplied by finance and covering salary, on-costs and a share of supervision. It reconciles: 260,000 at 11 minutes is 47,700 hours a year, which at £34 is £1.62m, and that sits inside the claims operating cost line rather than beside it.\nTarget movement: 55% of documents handled end to end with no human keying, with the remaining 45% unchanged.\nArithmetic: 260,000 multiplied by 0.55 is 143,000 documents. At 11 minutes each that is 26,200 hours a year, and at £34 an hour it is £891k of handling effort released. That figure is the honest measure of the work removed, and it is not the price of the outcome.\nRecovered capacity, not money: 26,200 hours is 15.4 full-time equivalents at 1,700 productive hours each. Nothing has left the business until a cost line actually falls, so the outcome splits the number in two. The claims operation currently buys 6 FTE of agency cover to hold the backlog at £38 an hour loaded, which is 6 multiplied by 1,700 multiplied by £38, or £388k a year, and that cover is not renewed once the straight-through rate holds. That £388k is money. The remaining 9.4 FTE are permanent handlers who stay on the payroll, so it is recovered capacity, it is recorded as 9.4 FTE with the work it is being redeployed onto named, and it is booked in the outcome at zero. The outcome is priced at £388k annualised. The in-quarter share is stated separately: live in week five of a thirteen-week quarter, so eight weeks of benefit, £388k divided by 52 multiplied by 8 is £60k.\nSensitivity: if straight-through lands at 40% rather than 55%, and only 4 FTE of agency cover can be released, the money falls to 4 multiplied by 1,700 multiplied by £38, or £258k. Still worth the quarter, so the target stands, and both figures go in front of the owner.\nOwner: the claims director, named, who confirms she will carry the £388k as a reduction in the agency cover line in next year's plan, and who states plainly that she will not carry the 9.4 FTE as a saving.\nValidation: the agency cover spend on the purchase ledger, monthly, against the same twelve-month pre-change period, alongside straight-through rate by document type from the claims workflow system. The ledger is the primary source, because that is where money leaving the business shows up; the workflow dashboard is the explanation, not the evidence. Anything under a 5 percentage point move in straight-through rate is noise. Three months of post-live data before anyone calls it.\nNot realised if: the agency cover line has not fallen by at least £250k annualised six months after the last epic ships, whatever the straight-through rate says.\nEpics, all in Q4: document classification and routing £180k, structured extraction with confidence-routed human review £150k, straight-through handling for the two simplest document types £120k. All three move the same document population, so they are priced in landing order and each carries only what it adds on top of the one before. The shares total £450k against a £388k target. The last four quarters realised 70% of planned value, so the headroom rule wants £554k behind this target. That is £104k short, and it is visible in planning: either a fourth epic goes in, or the owner is told today that the honest target is £315k." }, "related": [ { "label": "What a defensible AI business case holds", "href": "/guides/business-case-for-an-agentic-programme" }, { "label": "Designing roles, decisions and workflows for an AI-native organisation", "href": "/guides/target-operating-model-for-an-ai-native-organisation" } ], "faq": [ { "question": "How do you define a business outcome that can actually be priced?", "answer": "Write the business result rather than the feature, then attach a number to it in one line of checkable arithmetic. Volume times unit cost, or volume times conversion times margin. If you cannot write that line, you do not have an outcome yet, you have an intention. Tenhaw publishes the whole method for doing this, worked examples and arithmetic included, free for a team to adopt without hiring anyone. The test is whether someone outside the team could recompute your figure from published or internal numbers and land in the same place." }, { "question": "How do you baseline a process before setting an improvement target?", "answer": "Measure it as it runs today, not as it is documented. Volume, elapsed time, people involved, and rework. Pull it from the system that already records it rather than from an estimate in a workshop, and pull it before you name a target, because a target set before a baseline is a number chosen to sound achievable. The baseline is also what makes the eventual claim of improvement checkable." }, { "question": "Who should own a business outcome, and what does owning it mean?", "answer": "One named person in the business, not the delivery team, and owning it means carrying the number in their own plan. If the outcome is a cost reduction, the owner is whoever holds that cost line and will show it falling. If nobody will put their name against the figure, the value was never believed in the first place, and that is the most useful signal the exercise produces." }, { "question": "How do you validate that a business outcome actually landed?", "answer": "Write the query before you commit, not after you ship. Decide now what you will run in six months, against which system, and what result would count as the outcome having landed. Running it is then arithmetic rather than argument. Writing it up front also tends to expose outcomes that cannot be measured at all, which is cheaper to discover before the work than after it. Tenhaw's Globelynx write-up reports delivery lead times falling 60% within six months, a figure that is quotable only because comparable delivery data existed before any improvement was claimed." }, { "question": "How do you set the horizon for an outcome without over-committing?", "answer": "Pick the point at which the result would be visible in the business, then plan headroom inside it rather than filling it. An outcome with no horizon never closes, and one with a horizon packed to the edge closes by being quietly redefined. The horizon also forces the split into epics, and if those epics do not sum to the number, the outcome is not funded, whatever the plan says." }, { "question": "How do you break a priced outcome into epics without losing the number?", "answer": "Split it so every epic carries a share of the figure, then add them up and check the sum against the outcome. Anything that cannot be given a share is not part of this outcome and belongs somewhere else. That check is the whole point of the split, because it is the moment you find out whether the plan actually delivers the number, or whether it delivers most of it and hopes." }, { "question": "What happens at an approval gate for a priced outcome?", "answer": "Someone commits to the number or the outcome does not start. The gate is where the owner accepts the figure into their plan, the arithmetic is checked in the open, and the validation query is agreed. Closing works the same way, on a stated reason. Either the query returned the result or it did not, and both are recorded. Outcomes that drift without either are how a roadmap fills with work nobody can defend." }, { "question": "How do you price an outcome when the benefit is avoided cost rather than revenue?", "answer": "The same arithmetic, on the cost base instead of the revenue line. Volume of the thing you no longer do, times what it costs to do it once. That volume and unit cost have to survive the real exceptions first, which is what a proof of concept is for. Tenhaw built one in two weeks inside Placing Platform Limited (PPL), PDFs in and business intelligence out on Azure, over ground the firm had circled for roughly a year. Avoided cost is harder to defend than revenue because nobody sees it arrive, which is why the baseline and the validation query matter more here rather than less. The query has to show the cost line actually falling, not that the process changed." }, { "question": "What is the difference between an outcome, an epic and a story?", "answer": "An outcome is a business result with a price and an owner. An epic is a share of that price with a delivery team behind it. A story is a piece of an epic. Nothing exists that does not trace up to a priced outcome, which is what stops a roadmap filling with work that is justified by the fact somebody asked for it." }, { "question": "How do you stop a roadmap filling with work that has no business case?", "answer": "Make the trace mandatory in both directions. Every epic points up to an outcome with a number and an owner, and every outcome divides down into epics whose shares add up to it. Tenhaw runs its own engagements on that rule, with a value target on every month and a month that delivers none reported to the client as a failed month rather than absorbed into the plan. Work that cannot make the trip in either direction does not get planned. It is a mechanical test rather than a judgement call, which is what makes it survive contact with a busy quarter." }, { "question": "How do you price work with no revenue line, like compliance, security or platform?", "answer": "Price the loss you are avoiding, not the feeling of safety. Regulatory work carries an exposure, a remediation cost and a probability of landing inside the horizon. £4m of exposure at a 30% chance is £1.2m, and legal or finance owns both of those inputs, not you. Platform work is priced through what it unblocks. If a migration is the precondition for £900k of epics that cannot start without it, that is the number, and it validates when those epics validate. What does not work is pricing effort, or pricing away a problem you were never going to have. Tenhaw prices its own platform and guardrail work that way, against a public engineering standard of 72 rules with RFC 2119 severities." }, { "question": "How precise does the price need to be?", "answer": "Precise enough that two competent people re-running the arithmetic land within about 20% of each other, and no more precise than that. The number earns its place by forcing assumptions into the open and letting you compare one outcome against another, not by predicting the P&L to the pound. If flexing a single assumption moves the answer by an order of magnitude, that assumption is the real work, so go and reduce the uncertainty before committing rather than averaging it away and hoping." }, { "question": "Can one epic contribute to two outcomes?", "answer": "No. An epic links to exactly one outcome. The moment its value is split across two, neither outcome's arithmetic can be checked and neither owner can be held to a number. If an epic serves two outcomes, decide which one it primarily belongs to, price its full share there, and note the second-order benefit on the other without booking currency against it. If that decision feels impossible, the two outcomes are probably one outcome that has been split for organisational reasons." }, { "question": "What happens when the value does not land?", "answer": "Close the outcome as accepted as not realised, with the validation figures attached and a note on which assumption broke: the baseline was wrong, the movement was smaller than expected, or the value per unit did not hold. Tenhaw ends engagements on the same principle, with an exit date agreed at kickoff and 30 days' notice either side. That closure is worth more than a quiet success, because it recalibrates the realisation ratio you use to size headroom on the next outcome. The failure mode to avoid is closing it as unvalidated. That teaches nothing and slowly inflates your calibration until the forecasts stop meaning anything." }, { "question": "How long does it take to define and price a business outcome?", "answer": "About six hours of working time, spread across a few sittings rather than one workshop. Elapsed time is longer on purpose, since you re-run the baseline query a week later and a different number means you had a screenshot rather than a baseline. The sittings are short because each needs different people. Whoever owns the report sits with you for the baseline, finance supplies the margin and lifetime value figures, and the named business owner confirms they will carry the number in their own plan for the year. Tenhaw is founder-led and James Rooney leads every engagement personally, so he sits in those conversations himself. Where no baseline exists at all, instrumenting it becomes the first epic and the outcome waits in Exploring." }, { "question": "We inherited a roadmap nobody can trace to a number. What do we do first?", "answer": "Reconstruct the outcome behind the work already in flight, one item at a time, because that is the fastest way to find out what should be cancelled. For each item, write the business result it should produce, then strike out every technology, vendor and screen name and see whether the sentence still means anything. If it does not, somebody priced a solution rather than a result. Then pull the baseline it claims to move and price it in one line of arithmetic. Anything that cannot reach a priced result with a named owner is a candidate to stop, and planning is a cheaper place to stop it than week eleven. Tenhaw starts inherited roadmaps with that reconstruction before proposing new work." }, { "question": "Two epics improve the same conversion rate. How do we price them?", "answer": "In landing order, with each epic priced on what the one before leaves behind. Two epics both claiming the same lift on the same traffic total £400k on paper and deliver £200k, and adding up the shares will not catch it, because the arithmetic sums perfectly. In the worked checkout example the three epics, guest checkout at £620k, payment failure retry at £510k and address autocomplete at £340k, all move one conversion number, so each carries only the movement it adds on top of its predecessor. Fix the order during planning, because the second epic's share depends on which one lands first." }, { "question": "Can we use an industry benchmark to price an outcome?", "answer": "No. A figure lifted from a vendor deck or an industry benchmark comes out of somebody else's business, so it cannot be reconciled to your reporting, and the first monthly validation becomes an argument about the source instead of the result. A number reverse-engineered from a budget already approved fails the same way, because it was built to justify a spend rather than to describe one. Price it from your own data: baseline volume from the system that records it, margin and lifetime value from finance with whose figures they are, and one visible line of arithmetic. Then flex the two assumptions you trust least by a third each and see whether the target still justifies the quarter." } ], "aiNote": "Outcome setting is identical in both delivery modes: the baseline, the price, the owner and the validation plan do not change. What changes is what sits underneath. AI-augmented teams break the outcome into epics and then stories. AI-native teams stop at the epic, so the outcome ticket has to carry the currency share, the key user journeys and the test requirements itself, because there is no story layer below it to hold that detail. Use a model to pressure-test the arithmetic, generate the sensitivity cases and draft the first epic breakdown for a human to cut. Do not let it produce the baseline: that figure comes out of your own reporting, run by a named person who can run it again next month." } ## How to build a quarterly product roadmap Source: https://tenhaw.com/the-tenhaw-way/how-to/quarterly-roadmap Last updated: 2026-11-05 { "slug": "quarterly-roadmap", "group": "Setting up the work", "title": "How to build a quarterly product roadmap", "summary": "Choose a quarter of work against real capacity, an outcome target and a forecast you can explain.", "intro": "A quarterly roadmap is one quarter of epics, each linked to exactly one outcome and each carrying a stated share of that outcome's currency target. Outcomes span quarters; epics never do. Every roadmap opens with a Tech Debt epic and a Bug Budget epic so neither can hide. Good looks like this: capacity counted in person-weeks before any work is chosen, planned value covering the remaining target with calibration headroom on top, a p85 forecast of the whole set landing inside the quarter, and the first six weeks already gated.", "whenToUse": "Two to three weeks before the quarter starts, once the outcomes you intend to work on are committed and priced. Also mid-quarter, the moment enough has changed that the published forecast is no longer honest.", "totalTime": "P2D", "timeLabel": "About two days, two to three weeks before the quarter", "tools": [ "The delivery tracker holding the quarter's epics", "An export of the last eight to twelve weeks of throughput", "A spreadsheet for the whole-set forecast" ], "steps": [ { "name": "Count delivery weeks before opening the backlog", "detail": "Write down the first and last working day. Thirteen calendar weeks is sixty-five working days per person. Subtract public holidays, booked leave, on-call rotations, the mandatory training week, any release freeze, and then the final week of the quarter, because a roadmap that plans work into its last days has no room to close. Multiply what remains by headcount. Nine and a half weeks across six people is fifty-seven person-weeks, not the seventy-eight the calendar implies, and that gap is where most over-committed quarters begin. Do this first, or the calendar becomes something you negotiate with after you have promised." }, { "name": "Create Tech Debt and Bug Budget first", "detail": "Both exist before you write a single feature epic, each with a named owner and a share of the person-weeks you just counted. Ten per cent each is a workable opening split. Then correct it with last quarter's actuals rather than last quarter's intentions: if bug work burned eighteen per cent, budget eighteen. Every bug raised and every debt item found this quarter links to one of these two epics. Nothing floats unbucketed. By week six the burn rate against each budget tells you whether quality is improving or rotting, and it is the only quality trend you get without building anything." }, { "name": "Bring forward committed outcomes and cap them", "detail": "List every outcome in Committed or Working On with its currency target and the value already claimed by epics that closed in earlier quarters. Target minus claimed is the remainder, and write that number beside each outcome, because the remainder is what this quarter's epics have to cover, not the headline target. Anything in Idea or Exploring gets no roadmap space until it has been shaped, priced and committed. Then cap it at three or four live outcomes per team. Past that the epics compete for the same people and the quarter ends with four things at eighty per cent and nothing validated." }, { "name": "Write the epics and show the pricing arithmetic", "detail": "For each outcome, write the epics that move it this quarter. One outcome each, one roadmap each, and a planned contribution in pounds with the sum written out on the ticket. Not \"improves retention\" but \"1,200 failed renewals a year, 18 per cent recoverable, £1,900 average contract, so £410k\". Name each input's source and who owns that number. If an epic will not finish inside the quarter, split it into a piece that ships now and a piece that goes on the next roadmap, each carrying its own share of the value. An epic that resists that split usually means the outcome under it was never shaped." }, { "name": "Sum against the remainder, then add calibration headroom", "detail": "Add each outcome's epic values and compare the total to the remainder from step three. Then divide the remainder by your realised-versus-planned ratio from the last four quarters of outcome validation. At 70 per cent realisation a £750k remainder needs roughly £1.07m of planned epics behind it. If your epics total £910k, the finding is a £160k gap that someone names before the quarter opens, or the team says out loud that the outcome will not close this quarter. Use your own history, not an industry figure. With no history, plan uncalibrated and record realised against planned so the next quarter has something to work from." }, { "name": "Sequence for dependencies, not for enthusiasm", "detail": "Put the epics on weeks. For each one, write down what it needs that the team does not control: another team's API, a vendor contract, a security or legal review, a data migration, one named specialist. Every external dependency gets raised as an issue with an owner and a date in the previous quarter, not discovered in week two of this one. Then check that nothing critical lands in the freeze week and that no two epics need the same specialist in the same fortnight. Sequencing is what turns a list that adds up on paper into a plan that runs." }, { "name": "Forecast the whole set, not the loudest epic", "detail": "Size the epics, then simulate the entire roadmap against your own throughput, sampling completed items weekly from the last eight to twelve weeks, and take p50 and p85 completion dates for the set. Do not forecast epics one at a time and add the dates together: four confident epics all landing in the same fortnight is where plans die, and only a whole-set simulation shows it. If your history is shorter than eight weeks, or the team changed size, say the forecast is unreliable and re-run it at the end of week three rather than publishing a number you do not believe." }, { "name": "Cut whole epics until p85 fits, then publish", "detail": "Reduce until the whole set lands inside the quarter at p85, not p50. Cut whole epics rather than shaving scope off all of them: a half-built epic delivers none of its planned value, a deferred one delivers all of it next quarter. Do not cut tech debt or the bug budget, that reflex is what the fixtures exist to stop. Publish where the business reads it, not only in the delivery tool: outcomes, the epics under each, currency planned, gate status, and the cut list with the pounds each cut epic carried. The cut list is the part teams skip and the part that protects them in month three." }, { "name": "Gate the first six weeks, book the close", "detail": "An epic does not open the quarter until it has product approval, engineering approval and, on an AI-augmented team, at least one product-approved story attached. On an AI-native team the same two approvals apply, with the key user journeys and the test requirements written on the ticket in place of the child story. Work backwards: research and design for those epics happens in the previous quarter. Expect four to six weeks gated on day one and the rest still in research; a fully gated roadmap usually means the later epics were written thin. Last, put the monthly outcome validation dates and the close date in the calendar with owners." } ], "example": { "heading": "Worked example: a subscription software business, one team of six", "body": "A made-up subscription software business, one team of six, and every number below is invented with it. Capacity. Thirteen calendar weeks, minus one week of public holidays and booked leave, minus half a week of training, minus one week of year-end freeze, minus the final week for close, leaves 9.5 delivery weeks. Across six people that is 57 person-weeks. Ten per cent to Tech Debt and ten per cent to Bug Budget takes 11.4, leaving 45.6 person-weeks for feature epics. Outcome. \"Reduce involuntary churn\", target £900k of retained annual revenue, of which £150k was already claimed by epics that closed last quarter. The remainder is £750k. The team realises 70 per cent of what it plans, so the roadmap needs about £1.07m of planned epic value behind that £750k. Epics and their arithmetic. Smart card retry: 1,200 failed renewals a year, 18 per cent recoverable, £1,900 average contract, so £410k. Dunning sequence: a further 9 per cent of the same 1,200 recovered, so £205k. Pre-expiry card update prompt: 700 cards expire a year, 40 per cent currently lapse, the prompt saves 30 per cent of those, so £160k. Self-serve downgrade instead of cancel: 900 cancellations a year, 15 per cent downgrade instead at £1,000 retained, so £135k. Total £910k against £1.07m needed, a £160k gap named in planning. Forecast and cut. The p85 for all four epics lands three weeks past quarter end, so the £135k downgrade epic is cut to the deferred list. The remaining £775k at 70 per cent realisation forecasts about £540k landing this quarter, which leaves roughly £210k of the outcome for the next roadmap. That sentence goes on the published roadmap, because it is far cheaper to say it in planning than to discover it at the close." }, "pitfalls": [ "Letting an epic straddle two quarters because splitting it feels wasteful. The moment an epic has no single quarter it must ship in, nobody owns the date, the value cannot be attributed to a roadmap, and the forecast loses the unit it counts in. Split it, price both halves, and put the second half on the next roadmap.", "Planning to p50 and calling it a commitment. A p50 plan is a coin toss dressed as a date, and half of everything on it lands late by definition. Teams do it because p85 forces them to say no to something in planning, which is the whole point of running the forecast before the quarter rather than during it.", "Creating the Tech Debt and Bug Budget epics and then raiding their capacity in week three. The tell is bug work logged against feature epics, or debt items sitting with no epic at all. Once that starts the burn rate means nothing, and the one quality trend you had is gone for the rest of the quarter.", "Reverse-engineering the epic prices so the sum lands neatly on the target. The totals reconcile, the business quietly stops believing the numbers, and your calibration data is worthless next quarter. If the epics do not add up, the gap is the finding, not an error to be corrected in the spreadsheet.", "Holding roadmap space for outcomes that are still ideas. Whoever asks loudest in week five fills that space with unpriced work, no cut list records what it displaced, and by the close there is no way to tell whether the quarter under-delivered or was quietly reloaded." ], "definitionOfDone": [ "Every epic links to exactly one outcome, carries a planned value in pounds with the arithmetic visible on the ticket, and belongs to this quarter only.", "A Tech Debt epic and a Bug Budget epic exist with named owners and a share of the person-weeks set from last quarter's actuals, not from a default.", "For every outcome, the epic values cover the remaining target divided by your realised-versus-planned ratio, or the shortfall is written on the roadmap with a name against it.", "A p85 forecast of the full set lands inside the quarter, and the cut list is published with the pounds each deferred epic carried.", "Every epic starting in the first six weeks has product approval, engineering approval and the third condition its delivery mode requires, and every cross-team dependency has an owner and a date.", "The monthly outcome validation dates and the roadmap close date are in the calendar with named owners." ], "related": [ { "label": "Designing roles, decisions and workflows for an AI-native organisation", "href": "/guides/target-operating-model-for-an-ai-native-organisation" }, { "label": "What a defensible AI business case holds", "href": "/guides/business-case-for-an-agentic-programme" } ], "faq": [ { "question": "What if an outcome needs longer than a quarter?", "answer": "That is normal and the model expects it. Outcomes span quarters; epics do not. Tenhaw plans its own engagements the same way, retainer-shaped and monthly with an exit date agreed at kickoff. Put the epics that move the outcome this quarter on this roadmap with their share of the value, leave the rest of the target unallocated until the next planning round, and carry the remainder forward. The outcome stays open in Working On or Value Monitoring across both quarters and only closes when every epic under it has closed. What you must not do is write one epic spanning both quarters to avoid the split, because that is the single change that makes the whole portfolio unforecastable." }, { "question": "How do I set a calibration factor before I have any history?", "answer": "Plan the first quarter at one to one and write on the roadmap that it is uncalibrated. Then record planned value against realised value for every epic, at each monthly outcome validation and again at the close. After two quarters you have a ratio worth using and after four it is stable enough to plan against. The first honest number is usually lower than the team expected, and that conversation is worth more than the arithmetic it changes." }, { "question": "Something urgent lands in week five. What do I do with it?", "answer": "Urgent work takes the same route as everything else. Shape it, price it and commit it as an outcome, or attach it to an existing epic if it belongs under one. Then it has to displace something. Find the epic with the lowest planned value per delivery week, move it to the cut list with its pounds attached, re-run the p85 forecast for the reduced set, and publish both changes together. Tenhaw takes that call as delivery lead inside Programme and Delivery Management, which is bought on its own even where another supplier is building. The failure mode is adding the new work without naming what it pushed out, because by the close nobody can separate under-delivery from a quarter that was quietly reloaded." }, { "question": "Do the Tech Debt and Bug Budget epics carry a currency value?", "answer": "No. They carry a share of the person-weeks, not a planned contribution to any outcome target. Their return is avoided rework and avoided incidents, which cannot be attributed honestly at planning time without inventing a number, and inventing one there corrupts the calibration data everywhere else. Count them in capacity, report their burn rate monthly, and never let their absence from the value column become the argument for cutting them, which is the exact argument the two fixtures exist to defeat." }, { "question": "How do you calculate team capacity for a quarterly plan?", "answer": "Start from the calendar and subtract before you count anything else. Write down the first and last working day, and thirteen calendar weeks gives you sixty-five working days per person. Then subtract public holidays, booked leave, on-call rotations, mandatory training, any release freeze, and the final week of the quarter, because a plan that runs into its last days has no room to close. Multiply what remains by headcount. Nine and a half delivery weeks across six people is 57 person-weeks, not the 78 the calendar implies, and that gap is where most over-committed quarters begin. Do this before opening the backlog, not after you have promised." }, { "question": "How much capacity should go to tech debt and bug fixing each quarter?", "answer": "Open with ten per cent each for tech debt and bug fixing, then correct it with last quarter's actuals rather than intentions, so if bug work burned eighteen per cent of capacity, budget eighteen. Tenhaw won that argument at Greggs with delivery data showing tech debt work accelerated timelines, across two squads over six months. Both live as epics created before any feature epic, each with a named owner and a share of the person-weeks. Every bug and debt item found in the quarter links to one of the two, so nothing floats unbucketed, and by week six the burn rate against each budget tells you whether quality is improving or rotting. Resist raiding them mid-quarter; that reflex is what the two fixtures exist to stop." }, { "question": "How many outcomes should one team take on in a quarter?", "answer": "Cap it at three or four live outcomes per team. Past that the epics compete for the same people and the quarter ends with four things at eighty per cent done and nothing validated. The cap also forces the right planning conversation. Only outcomes that have been shaped, priced and committed get roadmap space at all, and ideas still being explored wait. For each outcome you do take on, plan against the remaining target, the headline number minus the value already claimed by epics that closed in earlier quarters, because that remainder is what this quarter's epics actually have to cover." }, { "question": "Should I forecast each epic separately or the roadmap as a whole?", "answer": "Forecast the whole roadmap as one set. Size the epics, then simulate the entire quarter against your own throughput, sampling completed items weekly from the last eight to twelve weeks, and take p50 and p85 completion dates for the set. Four individually confident epics all landing in the same fortnight is the failure that kills most plans, and only a whole-set simulation exposes it. Forecast them one at a time, add the dates together, and that collision stays hidden. If your history is shorter than eight weeks or the team has changed size, say the forecast is unreliable and re-run it at the end of week three rather than publishing a number you do not believe." }, { "question": "The plan does not fit the quarter. Do we trim scope everywhere or drop an epic?", "answer": "Drop whole epics until the p85 forecast for the remaining set lands inside the quarter. A half-built epic delivers none of its planned value, while a deferred one delivers all of it next quarter, so cutting whole epics keeps the arithmetic honest and shaving scope off everything does not. Choose cuts by planned value per delivery week, protect the Tech Debt and Bug Budget epics from the cutting reflex, and publish the cut list with the pounds each deferred epic carried. That list is the part teams skip, and the part that protects them in month three when someone asks why something is missing." }, { "question": "What should a published quarterly roadmap show the business?", "answer": "Publish the roadmap where the business actually reads, not only inside the delivery tool. Tenhaw writes and maintains that published version inside Programme and Delivery Management. Show the outcomes with their currency targets, the epics under each with the planned value and the arithmetic visible, gate status for every epic, and the cut list with the pounds each deferred epic carried. Expect four to six weeks of work gated on day one and the rest still in research; a fully gated roadmap usually means the later epics were written thin. Finish by putting the monthly outcome validation dates and the close date in the calendar with named owners, so the quarter has a booked ending as well as a plan." }, { "question": "When should we start planning next quarter, and how long does it take?", "answer": "Start two to three weeks before the quarter begins, and budget about two days of focused work for one pass. Do it only once the outcomes you intend to work on are shaped, priced and committed, because a session that also has to invent the outcomes will produce neither them nor a plan. Three things need to be to hand: the delivery tracker holding the quarter's epics, an export of the last eight to twelve weeks of throughput, and a spreadsheet for the whole-set forecast. Plan again mid-quarter, the moment enough has changed that the published forecast is no longer honest." }, { "question": "Who needs to be involved in quarterly roadmap planning?", "answer": "Fewer people than most planning events assume. Product and engineering both have to approve every epic that opens the quarter, so both belong in the room rather than being consulted after it. Tenhaw staffs planning with senior people only, no junior rung, and its founder co-designed HSBC Global Payment Solutions' target operating model with selected teams there. Each epic's value arithmetic needs a named person who owns each input, usually whoever runs the system that number is read from. The Tech Debt and Bug Budget epics each need a named owner, set before any feature epic is written. Every cross-team dependency needs an owner and a date. The monthly outcome validation dates and the close date go in the calendar with names against them." }, { "question": "What does a quarterly roadmap look like with the numbers filled in?", "answer": "Take a made-up subscription business with one team of six. Public holidays, booked leave, half a week of training, a year-end freeze and the final week for close cut thirteen calendar weeks to 9.5 delivery weeks, so 57 person-weeks, of which 11.4 go to Tech Debt and Bug Budget. The outcome is reducing involuntary churn against a £900k target, with £150k already claimed by epics that closed, leaving a £750k remainder that at 70 per cent realisation needs about £1.07m of planned epic value behind it. Four priced epics total £910k, so a £160k gap gets named in planning, and the p85 lands three weeks late, which puts the smallest epic on the cut list." }, { "question": "What should happen when the quarter closes?", "answer": "The close is booked in the calendar during planning, with an owner against it, and the final week of the quarter is kept clear of planned work so there is room to do it properly. Tenhaw applies the same test monthly, and a month that delivers no measurable value is reported to the client as a failed month. At the close you read realised value against planned for every epic, and that reading is what sets the realisation ratio the next roadmap plans against. Outcomes that are not finished stay open and carry forward the remainder, the target minus the value claimed by epics that have closed. The cut list goes into the next planning session with the pounds each deferred epic carried attached." }, { "question": "How much of the quarter should be ready to start on day one?", "answer": "Expect four to six weeks of it gated and the rest still in research. An epic does not open the quarter until it has product approval, engineering approval and the gate content its delivery mode requires, which means the research and design behind those first epics happened in the previous quarter. A fully gated roadmap is not the sign of a well-prepared team. It usually means the later epics were written thin, because nobody can honestly specify week eleven's work two weeks before the quarter opens. Gate the front, keep the research running behind it, and gate the rest as it becomes real." }, { "question": "How do we tell by week six whether the quarter is off track?", "answer": "Three readings, none of which needs a new report. Tenhaw built that reporting at Globelynx across 16 client deliveries, where lead times fell 60 per cent inside six months. Re-run the whole-set forecast at the end of week three, especially if the throughput history you planned from was short or the team changed size, then compare that p85 date with the one you published. Read the burn rate against the Tech Debt and Bug Budget epics, which is the only quality trend you get without building anything, and which shows whether those budgets are quietly being raided for feature work. And one monthly outcome validation will have run by then, so you hold a real reading of realised against planned rather than a status colour." }, { "question": "Can we give the business a twelve-month roadmap?", "answer": "You can show a year of intent, but only the current quarter gets dated epics. Tenhaw's own engagements run monthly on the same footing, stopping on 30 days' notice either side rather than a twelve-month commitment. Outcomes span quarters and carry a currency target; an epic never spans one, and an epic with no single quarter to ship in makes a whole portfolio unforecastable. Beyond the quarter, publish the outcomes with the remainder still to cover, the target minus the value already claimed by epics that closed, alongside the cut list and the pounds each deferred epic carried. That is honest and useful. Dated epics twelve months out are neither, because the forecast behind them samples throughput the team has not generated yet." }, { "question": "Does quarterly planning work differently on an AI-native team?", "answer": "Less than people expect. The capacity count, the value arithmetic and the p85 forecast are identical whether a team is AI-augmented or AI-native. Two things change. Gating is the first. An AI-native epic needs product and engineering approval with the key user journeys and the test requirements written on the ticket, in place of the product-approved child story an AI-augmented epic carries, so the research behind those journeys happens in the previous quarter. The second is forecasting. AI-native throughput moves fast enough that twelve weeks of history can flatter or punish you, so use the shortest window covering at least eight completed items and re-run at the end of week three. Tenhaw's own AI-native work runs against an open-source handbook of 72 rules, enforced by an agent rather than remembered." } ], "aiNote": "The capacity count, the value arithmetic and the p85 forecast are identical in both modes. Two things change. Gating: an AI-augmented epic needs at least one product-approved story attached before it can open the quarter, while an AI-native epic needs the key user journeys and the test requirements written on the ticket instead, so the research that produces those journeys still has to happen in the previous quarter. Forecasting: AI-native throughput moves fast enough that twelve weeks of history can flatter or punish you, so use the shortest window that covers at least eight completed items and re-run the forecast at the end of week three rather than trusting a planning number for thirteen weeks." } ## How to write an epic that carries a value target Source: https://tenhaw.com/the-tenhaw-way/how-to/write-a-value-focused-epic Last updated: 2026-11-05 { "slug": "write-a-value-focused-epic", "group": "Setting up the work", "title": "How to write an epic that carries a value target", "summary": "Connect one quarter of work to its share of a business result, with the calculation visible.", "intro": "An epic is one quarter of work carrying a named share of exactly one outcome's currency target. Good looks like this: a reader opens the ticket and sees which outcome it ladders to, how much money it is planned to return, the arithmetic behind that number, the baseline it was measured from, the date that baseline was read, and how anyone will know afterwards whether it landed. No second document, no follow-up conversation. If the epic carries no number, it is a feature wishlist with a due date attached.", "metaDescription": "Price the epic before anyone builds: the population, the change you expect in it, what one unit is worth, the baseline reading and the date it was taken.", "whenToUse": "Write one whenever an outcome has moved to Committed and you are breaking it down for the coming quarter. Also use it to repair an existing epic that nobody in the room can put a number against.", "totalTime": "PT4H", "timeLabel": "About half a day, including both approvals", "tools": [ "The delivery tracker", "The parent outcome's baseline reading" ], "steps": [ { "name": "Start from a committed outcome", "detail": "Open the parent outcome before you open a blank epic. It should sit in Committed, carry a currency target and have a named owner; if it does not, fix that first, because an epic priced against an unpriced outcome is guesswork with decimal places. An epic links to exactly one outcome. If the work serves two, pick the one it moves most, claim value there only, and record the secondary benefit as a note so nobody banks it twice. If you cannot name a parent outcome at all, the work is not an epic: route it to this quarter's Tech Debt or Bug Budget epic and move on." }, { "name": "Name it after the change, not the build", "detail": "Name the epic after the change in the world, not the thing you plan to build. \"Remove the forced account creation blocking guest checkout\" survives a change of solution. \"Apple Pay integration\" does not, and it settles the design before research has run. Apply one test: could a different implementation satisfy this title? If not, you have written a solution, and at validation the only question you can answer is whether you shipped it. Keep the title under about twelve words so it reads whole in a roadmap view, and put the solution you currently favour in the body, where it can change without a rename." }, { "name": "Show the arithmetic a sceptic could re-run", "detail": "Write the money as three numbers and one multiplication: the population, the change you expect in it, and what one unit of that change is worth. 480,000 checkout starts a year at 62% completion with an £84 average order value makes one percentage point worth 4,800 orders, about £403,000. State the headroom as well: 38% abandon, so there is a ceiling, and a claimed 5pp lift needs evidence rather than enthusiasm. If the outcome is priced on profit, convert at margin before writing the number. Put the chain in the ticket, and state the window, in-quarter or annualised, the same window on every epic." }, { "name": "Price the siblings together, then check the total", "detail": "Price every epic under an outcome in one sitting, off one baseline reading, with the same person holding the pen. Priced separately, two epics will both claim the payment screen and their combined lift will exceed anything that screen can produce. Then sum them against the outcome target: four epics at £200,000 under a £1m outcome is a £200,000 hole, and week one is the time to say so, not week eleven. Apply your own realisation rate on top: if you historically bank 70% of what you plan, a £1m target needs about £1.43m planned. Take the rate from your delivery history, and give the residual gap an owner's name." }, { "name": "Name the measurement before anyone builds", "detail": "Write down the metric, the system it is read from, the baseline reading, the date it was taken, who reads it each month, and how long the epic must run before a real change is distinguishable from noise. At 480,000 starts a year, a 0.5pp move needs weeks of data before you can call it, so say that in the ticket and nobody demands a verdict in week two. If the metric does not exist yet, the instrumentation is inside this epic's scope, not a follow-up. This section is what monthly outcome validation judges the epic against once it sits in value monitoring." }, { "name": "Fit it to one quarter, then size it", "detail": "An epic belongs to one roadmap: the quarter it ships in. Size it against the weeks that quarter contains, minus holidays, minus the capacity already standing behind the Tech Debt and Bug Budget epics, and forecast from your p85 throughput rather than a single-point estimate. If it will not fit, split it into two epics in consecutive quarters and give each its own share of the value. Refuse the phase-one pattern where the first epic returns nothing and all the money hides in phase two: if a slice cannot carry a number of its own, it is not an epic, it is part of the next one." }, { "name": "Write the gate content your mode requires", "detail": "Both modes need the outcome link, the currency share and the scope boundaries. Write what is out of scope as plainly as what is in it, because that is the line a builder will otherwise redraw alone. AI-augmented teams then attach at least one product-approved story, small enough to build in a sprint and specific enough to test. AI-native teams stop at the epic, so the ticket itself carries the key user journeys end to end and the test requirements in full, written to be handed over whole to a model with no follow-up conversation available. If you would need a conversation to explain it, it is not finished." }, { "name": "Take both approvals and log the objections", "detail": "Product approval and engineering approval ask different questions. Product asks whether the value chain is credible and the measurement is real. Engineering asks whether it fits the quarter, what it depends on and what debt it creates. An epic cannot leave Ready for Dev without both approvals plus the mode's gate content, and no story beneath it can leave the backlog until product has approved the epic. Record every objection raised and how it was resolved, in the ticket, including the ones you overruled. Three months later, when the number has not moved, that record is the only honest account of what you knew at the time." } ], "example": { "heading": "Worked example: a £1.2m outcome broken into three value-carrying epics", "body": "Take a fictional mid-market UK retailer. The outcome is \"Recover revenue lost at checkout\", target £1.2m annualised, owned by the commerce director. Baseline read from the analytics platform on 3 January: 480,000 checkout starts a year, 62% completion, £84 average order value. One percentage point of completion is 4,800 orders, about £403,000, and the theoretical ceiling is the 38% who abandon. Three epics are priced in one sitting against that single baseline, alongside the roadmap's standing Tech Debt and Bug Budget epics. One-tap wallet payment on mobile: mobile is 61% of starts, 292,800 a year, and 2pp there is 5,856 orders, £492,000. Remove forced account creation for guests: research supported 1.4pp, but mobile guests are already counted inside the wallet epic, so this one is priced at 0.9pp, 4,320 orders, £363,000, with the 0.5pp difference noted in the ticket as claimed next door. Address lookup plus readable card-decline messaging: 0.5pp, 2,400 orders, £202,000. Planned total £1.06m against a £1.2m target, a visible £144,000 shortfall before a line of code is written. Apply the retailer's own 70% realisation rate and it is worse: £1.06m planned lands nearer £739,000, and covering £1.2m credibly needs about £1.71m of planned epic value, so roughly £658,000 is missing. Two honest choices follow. Find another epic worth that much, or restate the outcome target as what this quarter can carry. What you cannot do is leave the sum unexamined and discover it in April." }, "pitfalls": [ "Naming the epic after the build. \"Apple Pay integration\" gets written because the item arrived from a backlog that had already picked the solution. The cost lands at validation, where the only question you can answer is whether you shipped it, not whether the number moved.", "Dividing the outcome target by the number of epics. The tell is a set of suspiciously round values that sum to exactly the target with nothing left over. It means the epics were priced to satisfy the arithmetic rather than measured, and the first honest baseline reading collapses the roadmap.", "Double counting the same funnel step. Two epics both claim a lift on the payment screen, and their combined value exceeds anything that screen can produce, because each was priced in isolation by a different person on a different day.", "Phase-one epics that carry no value. All the return is deferred to a phase two next quarter, so the quarter closes with an epic that cannot enter value monitoring and cannot be validated. Splitting work is fine. Splitting it so that only the second half is worth anything is not.", "Deciding the measurement after release. Nobody captured the baseline, so validation becomes an argument about what the number was beforehand and the epic gets closed as \"probably worked\". A baseline reading costs an hour before the build and cannot be reconstructed after it." ], "definitionOfDone": [ "The epic links to exactly one outcome and sits in exactly one quarter's roadmap.", "It carries a planned currency value with the arithmetic shown in the ticket, plus the baseline reading and the date it was taken.", "Every epic under the outcome has been summed against its target, the realisation rate applied, and any remaining gap written down with an owner's name against it.", "The measurement plan names the metric, its source, who reads it monthly, and how long it must run before a change is distinguishable from noise.", "The mode's gate content is present: at least one product-approved story for AI-augmented teams, or the key user journeys plus the test requirements for AI-native teams.", "Product and engineering have both approved, and the objections raised along the way are recorded in the ticket with how each was resolved." ], "related": [ { "label": "What a defensible AI business case holds", "href": "/guides/business-case-for-an-agentic-programme" }, { "label": "What an AI-native SDLC changes", "href": "/guides/ai-native-sdlc-and-product-delivery" } ], "faq": [ { "question": "What if the epic has no revenue attached, like a compliance deadline or a platform migration?", "answer": "Price the avoided loss instead of the gain: the fine, the contract at risk, the run cost you stop paying, the hours the migration hands back to the team each month. Use the same currency and the same window as every other epic on the roadmap. If nobody will put a number on it after that conversation, you have learned something about its priority rather than found an exemption. Standing engineering work is different, sized as capacity in the quarter's Tech Debt epic rather than priced as value." }, { "question": "How precise does the value estimate need to be?", "answer": "Precise enough that someone else could re-run it and land in the same order of magnitude, and no more. The point is not accuracy to the pound, it is exposing the assumption: which population, what lift, what one unit is worth. A wrong number with visible arithmetic gets corrected in five minutes at validation. A confident number with nothing behind it cannot be corrected at all, only argued about." }, { "question": "Who writes the value, product or finance?", "answer": "Product writes it and finance checks the conversion. The product manager owns the population and the expected lift, because those come from research and funnel data. Finance owns whether you are converting at revenue or at margin, and whether the window matches how the business reports. Getting that second signature once, at the start of the quarter, avoids the meeting in April where a £1m roadmap turns out to be £300,000 of gross profit. Tenhaw's founder, James Rooney, co-led the design and piloting of HSBC Global Payment Solutions' target operating model for 500 teams and a $450M portfolio, where who holds the pen on the money is exactly the sort of line such a model settles." }, { "question": "Can two epics share one outcome's value if they only work together?", "answer": "No. If neither delivers anything alone, they are one epic split for scheduling convenience, so merge them or move the combined work into a single quarter. If one delivers something alone and the other adds to it, price the first at its standalone value and the second at the increment only. Never write the same pound into two tickets." }, { "question": "What makes a good epic title?", "answer": "Name it after the change in the world, not the thing you intend to build, so the title survives a change of solution. \"Remove the forced account creation blocking guest checkout\" passes that test. \"Apple Pay integration\" fails it, because it settles the design before research has run, and at validation the only question you can answer is whether you shipped it, not whether the number moved. Could a different implementation satisfy this title? If not, you have named a solution, not a change. Keep the title under about twelve words so it reads whole in a roadmap view, and put your current favourite solution in the body, where it can change without a rename." }, { "question": "How do you put a financial value on an epic?", "answer": "Write it as three numbers and one multiplication: the population, the change you expect in it, and what one unit of that change is worth. For example, 480,000 checkout starts a year at 62% completion and an £84 average order value make one percentage point of completion worth 4,800 orders, about £403,000. State the headroom too, because the 38% who abandon are the ceiling, and a claimed 5pp lift needs evidence rather than enthusiasm. If the outcome is priced on profit, convert at margin. Tenhaw reports any month that returns no measurable value as a failed month, and that test needs this arithmetic. Put the whole chain in the ticket and use the same window, in-quarter or annualised, on every epic so the sums compare." }, { "question": "What if the epics under an outcome don't add up to its target?", "answer": "Then the gap is the finding, and week one is the time to say so, not week eleven. Price every epic under the outcome in one sitting, off one baseline reading, then sum them. Four epics at £200,000 under a £1m outcome is a £200,000 hole. Tenhaw runs that session with James Rooney in the room, because he leads every engagement personally. Apply your realisation rate as well, since a £1m target needs about £1.43m planned if you historically bank 70% of what you plan. From there, either find another epic worth the shortfall or restate the target as what the quarter can carry. Never price epics backwards so they sum neatly to the target, and give any residual gap an owner's name." }, { "question": "What if the metric an epic needs doesn't exist yet?", "answer": "Then building the instrumentation is inside this epic's scope, not a follow-up. The measurement plan is written before anyone builds: the metric, the system it is read from, the baseline reading and the date it was taken, who reads it each month, and how long the epic must run before a real change is distinguishable from noise. A baseline reading costs an hour before the build and cannot be reconstructed after it; skip it and validation becomes an argument about what the number was beforehand, and the epic gets closed as \"probably worked\". That plan is what monthly outcome validation judges the epic against once it is live." }, { "question": "What if an epic is too big to fit in one quarter?", "answer": "Split it into two epics in consecutive quarters and give each its own share of the value. An epic belongs to exactly one roadmap, the quarter it ships in, and it is sized against the weeks that quarter really contains: minus holidays, minus the capacity already standing behind the Tech Debt and Bug Budget epics, forecast from your p85 throughput rather than a single-point estimate. What you must refuse is the phase-one pattern, where the first epic returns nothing and all the money hides in phase two. If a slice cannot carry a number of its own, it is not an epic, it is part of the next one." }, { "question": "What extra detail does an epic need when AI writes the code?", "answer": "Everything a builder would otherwise ask you in conversation, because no later conversation is available. When a model does the build there are no stories beneath the epic, so the ticket itself carries the key user journeys end to end and the test requirements in full. The scope boundary becomes the load-bearing section, so write what is out of scope as plainly as what is in, because a vague boundary gets a model's interpretation of the gap, at speed, across the whole codebase. Tenhaw applies that standard in its open-source engineering handbook of 72 rules on GitHub, enforced by an agent. Teams where developers write the code with AI assisting pass the gate differently, with at least one product-approved story attached, so some detail can wait." }, { "question": "How long does it take to write an epic that carries a value target?", "answer": "About half a day, and that includes both approvals. Most of that time goes on the arithmetic and the measurement plan rather than the prose, because the writing is short once the numbers exist: the population, the change you expect in it, what one unit of that change is worth, plus the baseline reading and the date it was taken. The baseline costs about an hour, and it cannot be reconstructed once the build has shipped, which is the whole argument for reading it now. Book the half day while you are breaking the outcome down for the coming quarter, not after the work has been scheduled. Tenhaw publishes the method in full, free to adopt without hiring anyone." }, { "question": "Do we need a new ticket type for a value-focused epic?", "answer": "Same ticket type, plus a few more requirements that all point at a number someone will check. A value-focused epic links to exactly one outcome, is named after the change in the world rather than the build, and shows the money as visible arithmetic with the baseline reading and the date it was taken. It names the measurement before anyone builds, including who reads the metric each month and how long the epic must run before a real change is distinguishable from noise. It sits in exactly one quarter's roadmap. And it cannot leave Ready for Dev without both product and engineering approval." }, { "question": "How do we fix an existing epic nobody can put a number against?", "answer": "Work backwards from its parent outcome before you touch the epic itself. If that outcome is not sitting in Committed with a currency target and a named owner, fix that first, because an epic priced against an unpriced outcome is guesswork with decimal places. At Tenhaw this is a roadmap-level repair, because an unpriced epic rarely travels alone. Then re-price every epic under the outcome in one sitting, off one baseline reading, with the same person holding the pen, so nothing gets claimed twice. Rename anything titled after a build. If the work still has no parent outcome after that conversation, it is not an epic. Route it to the quarter's Tech Debt or Bug Budget epic and move on." }, { "question": "What does engineering check before approving an epic?", "answer": "Whether it fits the quarter, what it depends on, and what debt it creates. That is deliberately a different question from the one product asks, which is whether the value chain is credible and the measurement is real. Fit is the part that takes work. Size the epic against the weeks the quarter really contains, minus holidays, minus the capacity already standing behind the Tech Debt and Bug Budget epics, and forecast it from p85 throughput rather than a single-point estimate. An epic needs both approvals plus its mode's gate content before it can leave Ready for Dev, and no story beneath it can leave the backlog until product has approved the epic." }, { "question": "Why record objections in the ticket after the decision is made?", "answer": "Because three months later, when the number has not moved, that record is the only honest account of what you knew at the time. Log every objection raised at either approval and how it was resolved, including the ones you overruled. It costs a paragraph, and it turns validation from an argument about who said what into a review of a decision that had reasons attached. It also protects both people involved, the one who raised the concern and the one who overruled it, because both positions were written down at the time rather than reconstructed afterwards." }, { "question": "How can you tell an epic's value was made up?", "answer": "Look for round numbers that sum to exactly the outcome target with nothing left over. That is the tell for epics priced backwards to satisfy the arithmetic rather than measured, and the first honest baseline reading collapses the roadmap. Two more tells are a missing baseline reading and date, and a value with no visible chain from population to expected change to unit value, so nobody else can re-run the sum. A fourth is two epics both claiming a lift on the same funnel step, which happens when siblings are priced in isolation, by a different person on a different day. Tenhaw reads a roadmap for these four tells first, because each is visible on the page without asking anyone." }, { "question": "Should an epic's value be annualised or just the quarter?", "answer": "Either, provided every epic on the roadmap uses the same window and the ticket says which one. A roadmap where one epic is annualised while its sibling is priced in-quarter will not add up against the outcome target, because the sums only mean anything if they compare. Pick the window that matches how the business reports, and settle it once at the start of the quarter rather than epic by epic. Whichever you choose, say in the ticket how long the epic must run before a real change is distinguishable from noise, because an annualised number is still judged on weeks of live data, not a year of it." }, { "question": "What does an outcome split into priced epics actually look like?", "answer": "Take a fictional mid-market UK retailer targeting £1.2m annualised on revenue lost at checkout, baselined on 3 January at 480,000 checkout starts a year, 62% completion and an £84 average order value. Three epics are priced together off that one baseline. One-tap wallet payment on mobile, 61% of starts, at 2pp is 5,856 orders, £492,000. Removing forced account creation at 0.9pp rather than the 1.4pp research supported, since mobile guests are already counted next door, is 4,320 orders, £363,000. Address lookup plus readable card-decline messaging at 0.5pp adds 2,400 orders, £202,000. Planned total £1.06m against £1.2m, so the £144,000 gap is visible in week one." } ], "aiNote": "The arithmetic is identical in both modes; the gate content is not. An AI-augmented epic passes with at least one product-approved story attached, so some detail can wait for story writing. An AI-native epic is the unit of work, so the ticket carries the key user journeys and the test requirements in full and there is no later conversation to fill the gaps. That makes the scope boundary the load-bearing section: an AI-native epic that is vague about what is out of scope gets a model's interpretation of the gap, at speed, across the whole codebase." } ## How to break an epic into stories Source: https://tenhaw.com/the-tenhaw-way/how-to/break-an-epic-into-stories Last updated: 2026-11-05 { "slug": "break-an-epic-into-stories", "group": "Setting up the work", "title": "How to break an epic into stories", "summary": "Split the journey into small user-visible changes that can each be built, tested and released.", "intro": "Breaking an epic into stories is where a priced piece of a business outcome becomes work a team can start on Monday. A story is the smallest piece of user-visible value you can ship: small enough to build inside a sprint, specific enough to test, traceable to one epic and one outcome. Good looks like four to eight stories that between them cover every journey step the epic changes, each releasable on its own, each sized against your own flow data, with at least one product-approved so the epic can leave Ready for Dev.", "whenToUse": "Do this once an epic has a parent outcome, a planned currency contribution and a quarter on the roadmap, and before it can leave Ready for Dev. If your team runs AI-native, skip it: the epic-level outcome ticket is the unit of work.", "totalTime": "PT4H", "timeLabel": "About half a day, plus a refinement session", "tools": [ "A whiteboard or shared document for the journey map", "Your team's flow data, for the size ceiling", "The delivery tracker" ], "steps": [ { "name": "Check the epic is fit to split", "detail": "Open the epic and confirm four things before you write a single story: it links to exactly one outcome, it carries a planned currency contribution to that outcome's target, it belongs to one quarter's roadmap, and it has been through research and design rather than sitting in idea. Fix anything missing now. Splitting an epic with no number attached produces stories nobody can prioritise, and splitting one with no quarter produces work that quietly rolls forward. Write the epic's planned contribution at the top of whatever you are splitting on. Everything you produce gets checked back against that figure at the gate." }, { "name": "Map the journey before you slice", "detail": "Write the user journey the epic touches, from the trigger to the moment value is realised, as a numbered line of plain-language steps. Six to twelve steps is normal. Do it with the designer and one engineer, on a whiteboard or in a shared doc, in under an hour. Mark which steps exist today and which this epic introduces or changes. Keep the numbers, because every story should name the steps it covers, and that is what turns coverage into something you can check rather than something you feel. A step nobody can name is a step nobody has designed." }, { "name": "Draft with AI, then cut it hard", "detail": "Paste the outcome, the epic, its currency contribution and the numbered journey into a model, and ask for candidate stories with acceptance criteria and the journey steps each one covers. Ask for more than you need, ten or so, because editing a draft beats staring at an empty backlog. Then cut. Models duplicate slices, write one story per screen rather than per journey step, and produce criteria that restate the title. Delete anything that does not change what a user can do, merge anything that cannot ship alone, and rewrite every criterion in your own words. Expect to keep about half." }, { "name": "Slice vertically, never by layer", "detail": "Every story crosses the whole stack for a narrow piece of journey: data, logic, interface, and whatever the user sees. Refuse stories called build the API, add the database table or wire up the front end. Those are chapters, and they belong to a developer inside one story rather than on your board. The test is blunt: could you release this alone, and describe the difference to a customer in one sentence without mentioning another story? If not, you have sliced by layer or by team, and the epic will sit at ninety per cent complete for weeks with nothing shippable." }, { "name": "Size against your flow data, split above ceiling", "detail": "Take each candidate to the two-weekly refinement session and size it. Set the ceiling from your flow data rather than from opinion: look at the stories your team finished last quarter, find the size at which cycle times start to scatter, and split anything at or above it. Five splits are worth knowing: by journey step, by business rule, by data variation, by interface, and happy path first with the exceptions following. Prefer happy path first. Record the sizes rather than arguing about them, because that record feeds your p50 and p85 forecasts. A story nobody can size is discovery, and it goes back to research." }, { "name": "Sequence so the first slice ships alone", "detail": "Order the set so the first story is a thin path through the whole journey that a real user could complete, even if it handles one case for one segment. It has to build without any other story in the set, and it should touch the measure the outcome is judged on, so live monitoring gets a signal early. Everything after it thickens the path. While you sequence, write a rough value share against each story, marked as not tracked: the model prices epics, not stories, and these numbers exist to say what gets built first and what a slip would cost." }, { "name": "Write acceptance criteria you can fail", "detail": "Each story needs criteria specific enough that a tester, or a model, could disprove them without asking you a question. Write observable behaviour: given this state, when the user does this, then this happens. Three to seven per story is normal. If you need more than ten, you are describing two stories, so go back and split. Include the negative cases you care about and name the ones you have decided to ignore, because an unnamed exclusion turns into a defect argument later. State the assumptions the story carries. If a criterion contains intuitive, robust or seamless, it is a hope, not a criterion." }, { "name": "Clear the approval gate deliberately", "detail": "The epic cannot leave Ready for Dev without product approval, engineering approval and at least one product-approved story attached. Run it as a forty-five minute working session with the whole set on screen, not a signature. Product confirms the stories cover every journey step the epic changes, sums the rough value shares against the epic's planned contribution, and agrees the first story is the right first one. Engineering confirms each slice is buildable as written and flags any that hides a dependency on another team. Stage the rest of the set if you like: until the epic is product-approved, none of its stories can start." }, { "name": "Route everything that is not a story", "detail": "A split always throws off work that is not user-visible value, and each kind has a home. Work a developer discovers mid-build becomes a chapter under the story it belongs to. Defects in something already shipped go to the quarter's Bug Budget epic and never into this epic's list, or the epic's burn-down quietly swallows your quality signal. Refactoring and platform work this epic depends on goes to the Tech Debt epic and gets sequenced against it. If a piece of work fits none of those, it belongs to a different outcome, and the honest move is to raise it there." } ], "example": { "heading": "Worked example: a £400k epic under a £1.2m outcome", "body": "Northgate Home is an invented mid-market online retailer. Its outcome is reduce checkout abandonment on mobile, target £1.2m of recovered annual revenue, spanning two quarters. One epic under it is guest checkout without account creation, planned contribution £400k, sitting in Q3. The journey line runs seven steps: land on basket, choose checkout, enter delivery address, choose delivery option, pay, confirm, receive confirmation email. Steps three to five are what this epic changes. The split produces five stories. One, a guest completes a card payment on standard delivery to a UK address, sized 8. Two, a guest chooses express delivery, sized 3. Three, a guest pays with either of the two wallet methods, sized 5. Four, a guest receives an order confirmation and tracking email without an account, sized 3. Five, a guest is offered account creation after payment rather than before, sized 5. Story one is the thin path: it goes live in week two and produces a real abandonment signal three weeks before the epic finishes. The rough value shares are £150k on story one, £30k on two, £110k on three, £60k on four and £50k on five, summing to the epic's £400k. They are a sequencing tool rather than commitments, because the model prices epics and not stories. They say story one carries most of the money and gets built first, and that if stories three and five slip out of Q3 the visible gap is £160k against a £400k epic. That is a conversation at the next monthly outcome validation rather than a surprise at roadmap close." }, "pitfalls": [ "Stories sliced by layer or by team. It happens because the slices mirror the org chart and because layer-sized estimates feel easier to give. The result is an epic stuck at ninety per cent for a month with nothing released, because the value only appears when the last layer lands.", "Scope leaking out during the split. The expensive slice gets deferred to next quarter, nobody restates the epic's planned contribution, and the epic ships at a third of its number while the outcome still shows the full target. Sum the shares at the approval gate, not at value monitoring.", "One giant first story with a queue of tidy-up behind it. Usually a sign the team wanted to start rather than to slice. It defeats forecasting, because most of the epic's risk sits in one lump nobody could size honestly.", "Acceptance criteria written after the code. They describe what was built rather than what was needed, and product approval at release becomes a formality. Write them before the story leaves the backlog.", "Splitting an AI-native team's epic into stories out of habit. It strips out the surrounding context the model needed and pre-empts judgement the model can exercise itself, so you pay the decomposition cost and get slower delivery for it. Put the journeys and the test requirements in the outcome ticket instead." ], "definitionOfDone": [ "Every numbered journey step the epic changes is covered by at least one story, and every story names the steps it covers.", "Every story is a vertical slice you could release on its own and describe to a customer in one sentence.", "Every story is sized, and nothing sits at or above the ceiling your flow data sets.", "Every story has acceptance criteria written as observable behaviour, with the negative cases you care about named and the ones you are ignoring stated.", "The rough value shares sum to the epic's planned contribution, or the difference is written down and the epic's number restated.", "The epic has product approval, engineering approval and at least one product-approved story, and the team agrees which story is built first." ], "related": [ { "label": "What an AI-native SDLC changes", "href": "/guides/ai-native-sdlc-and-product-delivery" }, { "label": "An outcome ticket against a story on the same epic: 54 front-end changes to 1", "href": "/case-studies/agentic-ai-in-insurance-operating-model" } ], "faq": [ { "question": "How many stories should an epic have?", "answer": "There is no fixed number, but four to eight is the usual landing point for an epic that fits one quarter. One story means either a small epic, which is fine, or that you have not sliced. More than about twelve usually means the epic was two epics wearing one number, and the honest fix is to split the epic and divide its planned contribution rather than carry a backlog nobody can hold in their head." }, { "question": "Do stories carry a currency value?", "answer": "No. Epics carry the planned contribution to the outcome's target, and that is the number reported and validated. Tenhaw's founder co-led the design of the governance and reporting layer of a target operating model for 500 teams at HSBC Global Payment Solutions, piloted, with James’s engagement complete and the wider rollout in progress. Rough per-story shares are still worth writing during sequencing, because they tell you what to build first and what a slip costs, but they are not tracked, not reported and not something anyone is held to. If per-story figures start appearing in status reports, you have created a second set of numbers that will disagree with the first." }, { "question": "What do I do with a story nobody can size?", "answer": "Treat it as discovery, not as a story. If refinement cannot size it because the team does not understand the problem, the shape of the data or what the user needs, send it back to the research phase with a specific question attached and a date you need the answer by. Sizing it anyway produces a number with no information in it, and that number then pollutes the p50 and p85 forecasts everyone is planning against." }, { "question": "Can I write stories before the epic is approved?", "answer": "Yes, and staging the whole set in advance is often the point of the working session. What you cannot do is start them. A story cannot leave the backlog until its parent epic is product-approved, and the epic cannot leave Ready for Dev without product approval, engineering approval and at least one product-approved story attached. So the set can exist, sized and sequenced, waiting on the gate." }, { "question": "What is a vertical slice, and why not split stories by layer?", "answer": "A vertical slice crosses the whole stack for one narrow piece of the user journey: data, logic, interface and whatever the user sees, releasable on its own. Stories called build the API or wire up the front end are really chapters, and a chapter belongs to a developer inside one story rather than on the board. The blunt test is whether you could release this alone and describe the difference to a customer in one sentence without mentioning another story. Split by layer and the value only appears when the last layer lands, which is why layer-sliced epics sit at ninety per cent complete for weeks with nothing shippable." }, { "question": "How do you know when a user story is too big?", "answer": "Set the ceiling from your own flow data rather than from opinion. Look at the stories your team finished last quarter, find the size at which cycle times start to scatter, and split anything at or above that size at the refinement session. A story should be small enough to build inside a sprint and specific enough to test, and a giant story defeats forecasting because most of its risk sits in one lump nobody can size honestly. Five splits are worth knowing: by journey step, by business rule, by data variation, by interface, and happy path first with the exceptions following. Prefer happy path first, and record the sizes, because that history feeds your p50 and p85 forecasts." }, { "question": "Can AI write the story breakdown for us?", "answer": "It writes a good first draft and a poor final one. Paste the outcome, the epic, its currency contribution and the numbered journey into a model, and ask for candidate stories with acceptance criteria and the journey steps each one covers. Ask for ten or so, because editing a draft beats an empty backlog. Then cut hard. Models duplicate slices, write one story per screen rather than per journey step, and produce criteria that restate the title. Delete anything that does not change what a user can do, merge anything that cannot ship alone, rewrite every criterion yourself, and expect to keep about half. Tenhaw checks model-written code the same way, against 72 handbook rules published on GitHub and enforced by an agent." }, { "question": "Which story should the team build first?", "answer": "The thinnest path through the whole journey that a real user could complete, even if it handles one case for one segment. It has to build without any other story in the set, and it should touch the measure the outcome is judged on, so live monitoring gets a signal early. Everything after it thickens the path. Rough value shares, written during sequencing and marked as not tracked, keep the choice honest, because they exist to say what gets built first and what a slip would cost. In the guide's worked example the thin first slice goes live in week two and produces a real signal three weeks before the epic finishes." }, { "question": "What makes good acceptance criteria for a user story?", "answer": "Criteria a tester, or a model, could disprove without asking you a question. Write observable behaviour. Given this state, when the user does this, then this happens. Three to seven per story is normal, and more than ten means you are describing two stories, so go back and split. Include the negative cases you care about and name the ones you have decided to ignore, because an unnamed exclusion turns into a defect argument later, and state the assumptions the story carries. If a criterion contains intuitive, robust or seamless, it is a hope rather than a criterion, and nobody can fail it." }, { "question": "Do AI-native teams still need user stories?", "answer": "No. In an AI-native team the epic-level outcome ticket is the unit of work, and splitting it into stories out of habit strips out context the model needed and pre-empts judgement it can exercise itself, so you pay the decomposition cost and get slower delivery. The journey line becomes the key user journeys section of the outcome ticket, and the acceptance criteria become its test requirements, which must pass before the ticket is done. The thinking moves up a level rather than vanishing. Tenhaw's own two-week proof of concept for Placing Platform Limited (PPL) ran on that footing, paired throughout with one of the client's engineers. Story splitting is for AI-augmented teams, where a developer writes the code and the model assists." }, { "question": "How long does it take to break an epic into stories?", "answer": "About half a day, plus the next two-weekly refinement session where the stories actually get sized. Inside that half day the journey map takes under an hour with the designer and one engineer, the rest goes on the AI-assisted first draft, the cutting, the sequencing and the acceptance criteria, and the approval gate runs as a forty-five minute working session with the whole set on screen. Sizing is the part you cannot do from a desk, because the ceiling comes from the team's own flow data rather than from opinion. Tenhaw publishes the whole method in full, so a team can run that half day itself without hiring anyone." }, { "question": "What should we check before splitting an epic into stories?", "answer": "Four things, and fix any that are missing before you write a single story: the epic links to exactly one outcome, it carries a planned currency contribution to that outcome's target, it belongs to one quarter's roadmap, and it has been through research and design rather than sitting in idea. Splitting an epic with no number attached produces stories nobody can prioritise, and splitting one with no quarter produces work that quietly rolls forward. Write the planned contribution at the top of whatever you are splitting on, because everything you produce gets checked back against that figure at the gate." }, { "question": "Do we need a user journey map before slicing an epic?", "answer": "Yes, and it should take under an hour. Write the journey the epic touches, from the trigger to the moment value is realised, as a numbered line of plain-language steps, and six to twelve steps is normal. Do it with the designer and one engineer, on a whiteboard or in a shared doc, and mark which steps exist today and which this epic introduces or changes. That marking is what tells you where the work really is, because the steps the epic changes are the ones your stories have to account for. A step nobody can name is a step nobody has designed." }, { "question": "Who needs to be in the room to approve a story breakdown?", "answer": "Product and engineering together, for forty-five minutes, with the whole set on screen. Run it as a working session rather than a signature. James Rooney chairs the first few of these gates on a Tenhaw engagement, then steps out once the team runs them itself. Product sums the rough value shares against the epic's planned contribution and agrees the first story is the right first one. Engineering confirms each slice is buildable as written and flags any that hides a dependency on another team. The epic cannot leave Ready for Dev without both approvals and at least one product-approved story attached, and until it is approved none of its stories can start." }, { "question": "What does a real epic-to-story breakdown look like?", "answer": "Tenhaw publishes one worked end to end. Northgate Home is an invented mid-market online retailer whose outcome is reducing mobile checkout abandonment, targeting £1.2m of recovered annual revenue across two quarters. One epic under it, guest checkout without account creation, carries a planned £400k and sits in Q3. The journey runs seven numbered steps and the epic changes steps three to five, which yields five stories: a guest pays by card on standard delivery, a guest chooses express delivery, a guest pays with either wallet method, a guest gets a confirmation and tracking email, and a guest is offered an account after payment rather than before." }, { "question": "What does it cost when a story slips out of the quarter?", "answer": "You can say the number on the day it happens, which is why rough value shares get written at sequencing. In a worked £400k guest checkout epic the shares are £150k, £30k, £110k, £60k and £50k, so if the wallet story and the account-creation story slip out of Q3, the visible gap is £160k against a £400k epic. It surfaces at the next monthly outcome validation rather than as a surprise at roadmap close. Tenhaw holds itself to the same test of measurable value every month, and a month that delivers none is reported as a failed month. The trap is deferring the expensive slice and leaving the epic's planned contribution untouched, so it ships at a third of its number." }, { "question": "How do we make sure the stories cover the whole epic?", "answer": "Give every story the numbers of the journey steps it covers, then read the set back against the numbered line rather than against your memory. The split is not finished until every step the epic changes has at least one story standing against it, and product runs exactly that check at the approval gate with the whole set on screen. It works only when the journey line is numbered and each story names its steps in writing. Without both, coverage becomes a show of hands. It is also the first line of the definition of done for a split, ahead of sizes and acceptance criteria." }, { "question": "Where do bugs and refactoring found during the split go?", "answer": "Anywhere but this epic. A split always throws off work that is not user-visible value, and each kind has a home. Work a developer discovers mid-build becomes a chapter under the story it belongs to. Defects in something already shipped go to the quarter's Bug Budget epic and never into this epic's list, or the epic's burn-down quietly swallows your quality signal. Refactoring and platform work this epic depends on goes to the Tech Debt epic and gets sequenced against it. If a piece of work fits none of those, it belongs to a different outcome, and the honest move is to raise it there." } ], "aiNote": "For an AI-augmented team this guide runs as written, with the model producing the first-draft split and a human cutting it back. An AI-native team does not do it at all: the epic-level outcome ticket is the unit of work, and decomposing below it strips out the context the model needed. The thinking does not disappear though, it moves up a level. The journey line becomes the key user journeys section of the outcome ticket, and the acceptance criteria become its test requirements, which is what has to pass before the ticket is done." } ## How to run product discovery research Source: https://tenhaw.com/the-tenhaw-way/how-to/discovery-research Last updated: 2026-11-05 { "slug": "discovery-research", "group": "Setting up the work", "title": "How to run product discovery research", "summary": "Test the assumption behind the investment and let the evidence change the plan.", "intro": "Discovery research turns an assumption you are pricing into evidence that could have proved you wrong. In The Tenhaw Way it runs in an epic's research phase, before design and before the ready-for-dev gate, and its only job is to test whether the epic's planned share of the outcome's currency target survives contact with reality. Good discovery is timeboxed against the quarter it has to ship in, ends in a revised number rather than a report, and leaves its evidence attached to the epic where the gate and outcome validation will both look for it.", "whenToUse": "Run discovery when an epic's planned currency share rests on something you cannot yet evidence: that the behaviour you are pricing exists, that enough people have the problem, or that the value is the size you claimed. If no finding would change the number, the scope or the decision to build, skip discovery and start building.", "totalTime": "P14D", "timeLabel": "Two weeks, capped, and one week under about £100k of value", "tools": [ "A markdown repository for the evidence corpus", "The epic the research is attached to" ], "steps": [ { "name": "Name the number the research could move", "detail": "Discovery is not a standing activity. Open the epic and write three figures at the top of the research file: the outcome's currency target, the epic's planned share of it, and the threshold at which your decision changes. Something like: below £120k this epic leaves Q3. Without that threshold you will produce findings nobody can act on, because no result is ever obviously bad enough. If the epic does not exist yet, this is outcome shaping rather than discovery, and the question is whether the outcome can carry a credible number at all." }, { "name": "Write the hypothesis so it can fail", "detail": "State the belief in one sentence with the number attached: trade customers reorder near-identical baskets often enough that one-tap templates lift repeat order rate by two points, worth £300k of gross profit this quarter. Underneath it write the kill condition as a measurable result, not a mood: fewer than four of eight interviewees rebuild orders from history, or under a quarter of accounts show repeat baskets in the data. Have product and engineering sign off both before fieldwork starts. Arguing about what would change your mind is cheap beforehand and impossible afterwards." }, { "name": "Timebox discovery against the quarter", "detail": "The roadmap is one quarter, twelve to thirteen weeks, and the epic has to ship inside it. Cap discovery at two weeks, one week for anything under roughly £100k of planned value, and put the end date on the epic before you start. If two weeks cannot answer the question, that is itself a finding: either the epic is too speculative for this quarter and moves out, or you cut the question down to the single assumption that most threatens the number and answer only that. Open-ended research is how epics quietly leave the roadmap without anyone deciding they should." }, { "name": "Pick methods that can return a no", "detail": "Match the method to the risk. For desirability, six to eight interviews with people who have the problem and are not your fans, recruited on day one from support tickets, churned accounts and lapsed users rather than the customer advisory board. For the size of the prize, query behaviour you already hold: order history, funnel drop-off, ticket volumes, cancellation reasons, and pull the denominator first so you know how many people the epic can reach. For feasibility, put an engineer in the room for an afternoon. Run at least one method that returns a number; interviews alone will not defend a currency figure at the gate." }, { "name": "Build the evidence corpus in markdown", "detail": "Convert every source into markdown in a repository the whole team and your models can read: interview transcripts in full, the query you ran alongside its output, ticket exports, competitor behaviour described in prose. One source per file, each opening with a date, who or what it came from, and how it was collected. Do not summarise on the way in. A tidy summary written during collection is where the inconvenient quote disappears, and it strips out the raw material the next step works on. Name files so a person and a model can both tell what is inside without opening them." }, { "name": "Run a contradiction pass, then check it", "detail": "Ask a model to read the whole corpus and return, with file and line references, what the evidence supports, what it contradicts, and where it is silent. Silence is the useful category: it tells you which part of your number nothing in the corpus touches. Then open every reference and confirm the quote exists and says what the summary claims. Expect to reject some of it, and record which claims you dropped and why, because that record is what stops the same claim reappearing at the gate. Synthesis across forty documents is what AI is good at. Being the only reader of the evidence is what it is not." }, { "name": "Re-price the epic in the open", "detail": "Discovery ends with a number, not a narrative. Take the epic's planned share and confirm it, change it, or set it to zero and close the epic. Show the arithmetic in four lines: how many people the behaviour applies to, the adoption or conversion rate the evidence supports, value per event, and the realisation factor your own delivery history justifies. If you land seventy per cent of what you plan, apply seventy per cent now rather than discovering it in month three. Record the baseline the epic starts from as well, because validation cannot prove a lift without one. A smaller number is a successful discovery." }, { "name": "Write the decision and clear the gate", "detail": "Discovery is finished when a written decision sits on the epic: proceed at this number, reshape to this smaller scope, or stop. Underneath it, list the questions you knowingly proceeded without and name who owns each one. Proceeding on an unanswered question is a decision someone owns, not an omission nobody noticed. Then take it through the gate your team's mode requires. In an AI-augmented team the discovery output becomes the first product-approved story attached to the epic, which ready-for-dev demands alongside product and engineering approval. If the epic cannot clear the gate, say so in the decision rather than letting it drift." }, { "name": "Book the rematch at outcome validation", "detail": "Write the discovery estimate, in currency, into the epic itself, so outcome validation reads it without archaeology. Every live outcome is validated monthly, and the epic sits in value monitoring until its value is confirmed or deliberately written off. When that happens, compare what landed against what discovery predicted and record the ratio as a single number. After three or four quarters you have a calibration curve built from your own work, and discovery stops being an argument about optimism and becomes an argument about your own history. This is the loop almost nobody closes, and it is what makes the next discovery cheaper." } ], "pitfalls": [ "Research designed to confirm. If the sample is customers who already love the product and the questions ask whether they would like a new feature, everyone says yes and nothing is learned. Recruit for the problem rather than for the affection, and write the kill condition before the first conversation rather than after it.", "No baseline captured. Discovery predicts a two-point lift and nobody records what the rate was the week before the epic shipped. Four months later, outcome validation has a number with nothing to compare it against, and the epic gets closed on an argument instead of evidence. Capture the baseline while you are already in the data.", "Discovery as a parking bay. An epic nobody wants to kill can sit in the research phase indefinitely and look busy. If an epic has been in research for more than two weeks with no decision, that is not research, it is an unmade prioritisation call. Move it out of the quarter and say so out loud.", "Treating AI synthesis as the evidence. A model summarising forty transcripts produces something coherent and compresses away the outlier that changes the answer. Ask for file and line references, open them, and treat any claim without a traceable source as not yet true.", "Leaving the planned value untouched. Teams run discovery, learn something material, then ship the epic carrying the number it was given in planning. If the evidence did not move the currency figure or explicitly confirm it, discovery has not finished." ], "definitionOfDone": [ "The hypothesis is written as a statement that could have been falsified, with a measurable kill condition, and the evidence says which way it went.", "Every claim traces to a named source in the corpus, a transcript, a query, a ticket, that someone else can open without asking you.", "The epic's planned currency share is confirmed or changed, with the arithmetic, the baseline it starts from and the realisation factor shown.", "A written decision sits on the epic: proceed, reshape or stop, plus the questions you knowingly proceeded without and who owns them.", "The evidence and the decision are attached to the epic and its outcome where the work is tracked, not in a shared drive.", "The epic can clear its mode's ready-for-dev gate, or discovery has stated plainly that it cannot yet and why." ], "example": { "heading": "Worked example: saved order templates at a trade distributor", "body": "Kesteven Supply is an invented UK trade distributor running a trade account portal, and every number below is made up to show the shape of the work. The outcome: lift repeat order rate on the portal, target £1.4m of additional gross profit across four quarters. One Q3 epic, saved order templates, carries a planned share of £300k. Hypothesis: trade customers reorder near-identical baskets often enough that one-tap templates lift repeat order rate by two points. Kill condition: fewer than four of eight interviewees rebuild orders from history, or under a quarter of accounts show repeat baskets in twelve months of order data. Threshold for action: below £120k the epic leaves Q3. Nine working days of a two-week box. The order history query returns 4,200 active trade accounts, of which 1,600, or 38 per cent, place orders where at least 80 per cent of lines match a previous order. Eight interviews recruited from lapsed and mid-tier accounts rather than the top twenty: six rebuild baskets by scrolling order history, two paste product codes from their own spreadsheet. Both spreadsheet users said templates would save them time, then said they order weekly whatever happens. A fake-door save-as-template button shown to 900 accounts for two weeks was clicked by 31 per cent. The kill condition was not met, so the epic survived. The number did not. Re-priced: 1,600 eligible accounts, 20 per cent sustained use rather than the 31 per cent click, giving 320 accounts. Of those, the evidence supports extra orders only from the group citing hassle on small top-ups, about 40 per cent, so 128 accounts, one extra order a month, £340 average gross profit per order, over three months: £130,560. Apply the 70 per cent realisation factor from the last four quarters and it is £91k. Decision: proceed at £91k, sequenced behind two larger epics. Baseline recorded: 21 per cent repeat order rate over the prior 90 days. Proceeding without an answer on whether templates cannibalise higher-margin phone orders through the sales desk, owned by the commercial lead. The visible consequence is £209k of the outcome's target with no epic behind it, argued in week three of the quarter rather than week thirteen." }, "related": [ { "label": "Taking the whole process agentic rather than the step", "href": "/guides/end-to-end-agentic-workflow-implementation" }, { "label": "What a defensible AI business case holds", "href": "/guides/business-case-for-an-agentic-programme" } ], "faq": [ { "question": "How long should discovery take?", "answer": "Two weeks at most, and one week for anything under roughly £100k of planned value. The constraint is not research quality but the quarter. The epic has to ship inside the same twelve to thirteen weeks, so every day in research is a day off the build. Tenhaw's own evidence work runs in that box, and a proof of concept for Placing Platform Limited (PPL) took PDFs through to business intelligence on Azure in a fortnight, over ground the client had circled for roughly a year. If two weeks cannot answer the question, reduce the question to the single assumption that most threatens the number, or move the epic out of the quarter and say why." }, { "question": "Does every epic need discovery?", "answer": "No. Discovery is for epics whose planned currency share rests on an assumption you cannot evidence. If the behaviour is already visible in your data, the value is a rate change you can calculate, and nothing you could learn would change the scope or the number, skip it and build. Running discovery on an epic whose answer is already known is the most common way a research phase turns into a parking bay." }, { "question": "Who should run discovery?", "answer": "The product manager who owns the epic, with an engineer present for at least one session and a second person reading the raw evidence. Whoever wants the epic to succeed should not be the only one reading the transcripts. Tenhaw staffs its own engagements the same way, with teams of two or three senior people and partner oversight on every one, so the second reader is a named person rather than a spare pair of hands. It is the same reason the kill condition is written and signed off before fieldwork rather than after the results arrive." }, { "question": "Can AI do the discovery for us?", "answer": "It can do the synthesis, which is the slow part, reading forty documents and returning what they support, contradict and leave silent, with references. Tenhaw runs the same split on paid work, a model doing the corpus pass and a named person checking every reference, with partner oversight on every engagement. It cannot be the only reader. Models compress away the outlier, and the outlier is often the finding. Ask for file and line references, open them, and treat anything without a traceable source as not yet true." }, { "question": "What is a kill condition in product discovery?", "answer": "A kill condition is the measurable result that would make you stop or reshape the work, written down before the research starts. It has to be a number, not a mood. Fewer than four of eight interviewees rebuild orders from history, say, or under a quarter of accounts show the behaviour in the data. Product and engineering sign it off before fieldwork begins, because agreeing what would change your mind is cheap beforehand and nearly impossible once the results are in. Without one, no finding is ever obviously bad enough to act on, and discovery drifts into confirming what the team already wanted to build." }, { "question": "What happens if discovery shows the epic is worth less than planned?", "answer": "You change the number in the open and treat that as a success, because a smaller honest figure beats a large fiction discovered in month three. Tenhaw reports a month that delivers no measurable value as a failed month. Show the arithmetic in four lines: how many people the behaviour reaches, the adoption rate the evidence supports, value per event, and the realisation factor your delivery history justifies. If the revised figure still clears the threshold you set at the start, proceed at the new number and re-sequence it against the other epics. If it falls below that threshold, the epic leaves the quarter, and the gap it leaves in the outcome's target gets argued in week three rather than week thirteen." }, { "question": "Where should discovery evidence be kept?", "answer": "Attached to the epic it belongs to, in a markdown repository the whole team and your models can read, not in a shared drive nobody revisits. One source per file: full interview transcripts, the query you ran alongside its output, ticket exports, each opening with a date, where it came from and how it was collected. Do not summarise on the way in, because a tidy summary written during collection is where the inconvenient quote disappears. The approval gate and monthly outcome validation both look for evidence on the epic, so keeping it anywhere else guarantees archaeology later." }, { "question": "Can you run discovery with interviews alone?", "answer": "Not if the output has to defend a currency figure. Interviews are the right tool for desirability, six to eight with people who actually have the problem, recruited from support tickets, churned accounts and lapsed users rather than your fans. But at least one method has to return a number, which means querying the behaviour you already hold, order history, funnel drop-off, ticket volumes, and pulling the denominator first so you know how many people the work can reach. People also say things their behaviour contradicts; in one worked example, two interviewees praised a proposed template feature and then explained they would order weekly whatever happened. The data settles it." }, { "question": "What is a realisation factor and how do you set one?", "answer": "A realisation factor is the share of planned value your team historically delivers, applied to every new estimate before anyone commits to it. If your last four quarters landed seventy per cent of what was planned, apply seventy per cent now rather than discovering the shortfall in month three. You build it by closing the loop: write the discovery estimate into the epic, compare what landed at outcome validation, and record the ratio. Tenhaw publishes the method in full, free to adopt, so a team can run that loop without hiring anyone. After three or four quarters you have a calibration curve built from your own work, and estimating stops being an argument about optimism and becomes an argument about your own history." }, { "question": "Should discovery end in a report?", "answer": "No. Discovery ends in a revised number and a written decision on the epic: proceed at this figure, reshape to a smaller scope, or stop. Underneath the decision, list the questions you knowingly proceeded without and name who owns each one, because proceeding on an unanswered question is a decision someone owns, not an omission nobody noticed. The 500-team target operating model James Rooney co-led at HSBC Global Payment Solutions was piloted rather than presented. A report is where findings go to be admired; a decision is something the approval gate can act on. If the research cannot yet produce one, say that plainly and move the epic out of the quarter rather than letting it sit in research looking busy." }, { "question": "Continuous discovery vs a two-week timebox: which is better?", "answer": "They do different jobs, and the timeboxed pass is the one an approval gate can act on. A continuous habit keeps a research cadence running whatever the roadmap is doing, which is useful for building a picture of your users over time. The timeboxed pass runs inside one epic's research phase, is capped at two weeks and at one week for anything under roughly £100k of planned value, and ends by confirming, changing or zeroing that epic's planned share of the outcome's currency target. Its trigger is narrower too. If no finding would change the number, the scope or the decision to build, you skip it and start building." }, { "question": "Is a fake door test enough evidence to price a feature?", "answer": "It is good evidence of interest and weak evidence of value, so discount it hard before it reaches the number. In one worked example a save-as-template button shown to 900 accounts for two weeks was clicked by 31 per cent, and the re-price used 20 per cent sustained use instead, because a click costs nothing and changing how you order every week costs something. Pair it with behaviour you already hold, such as order history or funnel drop-off, then apply the realisation factor your own delivery history justifies. A click-through rate carried straight into a currency figure is the quickest route to arriving short in month three." }, { "question": "What are the steps in a two-week discovery sprint?", "answer": "Eight steps inside the box, plus a rematch at outcome validation later. Write three figures at the top of the research file: the outcome's currency target, the epic's planned share of it, and the threshold at which your decision changes. State the hypothesis with a measurable kill condition and have product and engineering sign both off before fieldwork starts. Put the end date on the epic, recruit interviewees on day one, and run at least one method that returns a number rather than only conversations. Convert every source into markdown, run the contradiction pass and check its references yourself, re-price the epic, then write the decision on it. The worked example lands in nine working days of the two-week box." }, { "question": "We only have an idea, not an epic yet. Is that discovery?", "answer": "No, that is outcome shaping, and it is a different question. Discovery tests whether an epic's planned share of a priced outcome survives contact with reality, so it cannot start until you can write three figures at the top of the research file: the outcome's currency target, the epic's planned share of it, and the threshold at which your decision changes. If nothing has been priced, the question in front of you is whether the outcome can carry a credible number at all. Set the outcome first, break it into epics, then run discovery on the ones whose share rests on something you cannot yet evidence. Fieldwork first produces interesting findings and no decision." }, { "question": "Does discovery change if AI is writing the code?", "answer": "Yes, and it changes the fieldwork rather than just the write-up. In an AI-augmented team, where a developer writes the code and the model assists, this guide runs as written. In an AI-native team there is no child story to hold the detail, so discovery has to produce the key user journeys and the test requirements that go into the outcome ticket itself. That means capturing the sequence people actually follow, the edge cases and the failure states, not sentiment about whether they would like the feature. It raises the bar on the evidence corpus, because the same markdown files are what the model builds from. Tenhaw's engineering handbook targets that reader, 72 rules on GitHub written to be enforced by an agent." }, { "question": "Why capture the baseline during discovery rather than at launch?", "answer": "Because you are already in the data, and nobody goes back for it later. Discovery predicts a two-point lift, the epic ships, and if nobody recorded what the rate was the week before, validation four months on has a number with nothing to compare it against and the epic gets closed on an argument instead of evidence. Record it in the same pass that produces the denominator and the adoption rate, and write it into the epic beside the revised currency figure. The worked example on this page notes a 21 per cent repeat order rate over the prior 90 days, because a lift needs a starting point." }, { "question": "How do you work out how many customers a feature would reach?", "answer": "Pull the denominator before you count anything else. Query behaviour you already hold, order history, funnel drop-off, ticket volumes or cancellation reasons, and start from the total population the epic could ever touch, so every percentage after it has a ceiling. In one worked example that is 4,200 active trade accounts, of which 1,600, or 38 per cent, place orders where at least 80 per cent of lines match a previous order. The 1,600 is the reachable group, not the value. Adoption, the share of that group whose behaviour actually changes, and value per event come after it. Skip the denominator and you finish with a rate and no idea what it applies to." }, { "question": "What do you do when the research says nothing either way?", "answer": "Name the silence and then decide what to do about it in the open. A corpus that neither supports nor contradicts part of your number is telling you which assumption you never actually tested, which is more useful than a weak signal you might have leaned on. Two honest options fit inside the timebox. Spend what is left of it on that single gap, or re-price on what you do have and carry the question forward with an owner beside it. In the worked example the team proceeded without knowing whether templates cannibalise higher-margin phone orders through the sales desk, and named the commercial lead as the person who owns it." } ], "aiNote": "In an AI-augmented team, discovery output becomes the first product-approved story attached to the epic, and that story is one of the three things ready-for-dev checks. In an AI-native team there is no child story, so discovery has to produce the key user journeys and the test requirements that go into the outcome ticket itself. That changes the fieldwork, not just the write-up: you need the sequence people actually follow, the edge cases and the failure states, rather than sentiment about whether they would like the feature. It also raises the bar on the corpus, because the same markdown files are what the model builds from." } ## How to write a user story Source: https://tenhaw.com/the-tenhaw-way/how-to/write-a-story Last updated: 2026-11-05 { "slug": "write-a-story", "group": "Writing the work", "title": "How to write a user story", "summary": "Give the team a user-visible change and acceptance examples they can build and check from the ticket.", "intro": "A story is the smallest slice of user-visible value your team can ship: one parent epic, one sprint, testable by someone who did not write the code. Good means a developer who missed refinement can build it from the ticket alone, and a tester can pass or fail every criterion without asking your opinion. It names the user, the change to their experience, the acceptance criteria, the signal that will prove the epic is moving, and the assumptions it carries. If nothing a user can see or do changes, it is a chapter, not a story.", "whenToUse": "Write stories while the parent epic is still moving through its product phases, because that epic cannot leave Ready for Dev without at least one product-approved story attached. This is an AI-augmented team practice: AI-native teams stop at the epic and write an outcome ticket instead.", "totalTime": "PT2H", "timeLabel": "About two hours, plus sizing at refinement", "tools": [ "The delivery tracker", "The approved parent epic" ], "steps": [ { "name": "Anchor the story to an approved epic", "detail": "Open the parent epic before you open the blank ticket. Three checks: it links to exactly one outcome, it carries a stated share of that outcome's currency target, and it sits in this quarter's roadmap. A missing link or a blank value means fix the epic first, because a story hanging off a broken chain is work nobody can defend at the end of the quarter. If the epic is not product-approved yet, write the story anyway and stage it in the backlog. It cannot start until approval lands, and the epic cannot leave Ready for Dev until a story under it is approved. Someone goes first." }, { "name": "Prove the slice is vertical", "detail": "Check the slice crosses every layer it touches: data, service, interface. \"Build the payments API\" and \"Build the payments screen\" are two halves of nothing shippable. Slice instead on one step in the workflow, one type of user, one business-rule variation, one data source out of several, or the happy path with the edge cases as siblings behind it. Then run one test on what you are about to write. If we shipped only this, could a real user do something they could not do yesterday? If the answer is no, you are holding a chapter, and a chapter belongs to a story, not to an epic." }, { "name": "Title it as the change to the user", "detail": "Write the title as a person doing a thing: \"Returning customer pays with a saved card\", not \"Implement saved-card retrieval\". The subject is the user, the verb is observable from outside the codebase, and the whole thing runs under about ten words. If the only honest subject is a service or a component, you sliced by layer, so go back a step. Skip the \"As a user, I want, so that\" template unless your team still reads it closely. On most boards it has become a formatting ritual that pads the title and buries the specifics in a clause nobody finishes." }, { "name": "Write context that reads cold", "detail": "Assume the reader missed refinement and gets no follow-up conversation. Four sentences, in this order: which user, where in the journey, what happens today, what should happen once this ships. Link the discovery evidence rather than summarising it, so the claim stays attached to its source, and link the epic rather than restating it. Then add a line headed \"not in this story\", naming the sibling tickets that cover the rest. Leaving the boundary implicit is the fastest route to gold-plating: a developer with a gap in front of them fills it generously, and the sprint pays for it." }, { "name": "Write acceptance criteria a tester can run", "detail": "Three to seven criteria, each one passable or failable by someone who did not write the code. Given, when, then works. A plain checklist works. Opinions do not: replace \"the page should feel fast\" with \"the saved-card list renders within 500ms at p95 on a 4G profile\", and \"the flow should be intuitive\" with the click count or the exact error copy. Write the negative paths as deliberately as the happy one: empty state, expired data, the dependency timing out, the user who has none of the thing. Put the non-functional rules that matter here as numbers or named standards, not in a document nobody opens." }, { "name": "Name the signal that proves it worked", "detail": "The epic carries the currency. The story carries the signal that tells you the epic is moving. Write four things down: the metric, today's baseline, the dashboard or query the number is read from, and the event or property the build has to emit for that number to exist at all. If the instrumentation does not exist yet, make it an acceptance criterion in this story rather than a follow-up nobody schedules. This is what makes monthly outcome validation possible later. A story that ships with no way to read its effect becomes an argument in three months' time." }, { "name": "Write down the assumptions and open questions", "detail": "Every story carries assumptions: traffic volumes, what a third party returns, how many users sit in the segment. List each one with what you will do if it turns out wrong, in a single line: assume under 5% of sessions hit this, if it is higher we add pagination. Then list the open questions with a named owner and a date, not \"the team\" and not \"soon\". Two rules keep this honest. A question that blocks the build blocks the story leaving the backlog. A question you have chosen to proceed without is a decision, recorded as one, not an omission somebody finds mid-sprint." }, { "name": "Size it with the team, split anything oversized", "detail": "Take it to the fortnightly refinement and size it with the people who will build it, not at your desk. If they cannot build and review it inside one sprint, split it now, while splitting is cheap, along the same seams you used to slice: rule variations, user types, data sources, happy path first. Record the size so it feeds your throughput data and the forecasts. Do not pre-split an oversized story into chapters to make it feel manageable. Chapters are what a developer creates mid-build when a story turns out bigger than expected, and writing them upfront hides the size problem from the people sizing it." }, { "name": "Take it through product approval", "detail": "Approval is a check, not a signature. The approver confirms three things: the slice is user-visible, every acceptance criterion is testable by someone else, and the story belongs under this epic rather than a neighbouring one. Do this early for at least one story per epic, because the epic cannot leave Ready for Dev without product approval, engineering approval and one approved story attached, and that story is usually what everyone is waiting on. One rule for the approver: if you would have to explain the ticket in person for it to make sense, send it back rather than explaining it." } ], "pitfalls": [ "Slicing by layer because that is how the team is organised. \"API work\" and \"front-end work\" each look like a story, and neither ships anything a user can use, so neither can be credited with any part of the epic's planned value at validation. It happens when the org chart rather than the user's journey is doing the splitting.", "Acceptance criteria written for the people who were in the room. Fast, clean and intuitive pass review because everyone present knows what was meant, then fail in testing three weeks later when the ticket is the only surviving record. The cost lands on whoever picks the work up, not on whoever wrote it.", "The thirty-criterion story. Length is the tell that it is an epic wearing a story's ticket type. It happens because splitting feels like losing the thread of the feature, so the author keeps adding criteria instead of siblings, and the team sizes something nobody can finish in a sprint.", "Starting work on a story whose epic is not product-approved. The gate exists because an unapproved epic is still moving: its scope, its share of the outcome, sometimes the outcome itself. Work started against it gets rebuilt, and the rebuild eats capacity the quarter had already promised elsewhere.", "Writing stories in an AI-native team out of habit. Decomposing an outcome ticket into stories strips out the surrounding context the model needed and pre-empts judgement the model can exercise itself. The habit outlives the mode change because the ticket templates and the refinement calendar did not change with it." ], "definitionOfDone": [ "The story links to exactly one epic, and that epic links to one outcome with a currency target and sits in this quarter's roadmap.", "Someone who was not in refinement could build it from the ticket alone, with no follow-up conversation.", "Every acceptance criterion can be passed or failed by a person who did not write the code, and at least one covers a negative path.", "The metric, its baseline, where the number is read from, and any instrumentation the build must add are all named in the ticket.", "The team has sized it and agrees it fits inside one sprint, or it has already been split into stories that do.", "Product has approved it, so it is either free to start or deliberately staged behind an epic still waiting for approval." ], "example": { "heading": "Worked example: a saved-card story at a fictional retailer", "body": "Northbay Outdoors is an invented online retailer and every number below is invented with it.\n\nOutcome (L1): cut abandonment at the mobile checkout. Target £1.2m of additional annual revenue, currently in Working On, spanning two quarters.\n\nEpic (L2): saved payment methods for returning customers. Planned share of the outcome £380k, sitting in the Q3 roadmap, product and engineering approved.\n\nStory title: Returning customer pays with a saved card.\n\nContext: a customer who has bought at least once in the past twelve months reaches the payment step on mobile. Today they retype the full card number every time, and the session recordings from the March discovery (linked on the ticket) put 38% of mobile drop-off on that screen. Once this ships, cards stored on a previous order appear as selectable options showing brand, last four digits and expiry, and the customer pays with CVV only. Not in this story: adding a new card during checkout, editing or deleting stored cards, and desktop. Those are three sibling tickets under the same epic.\n\nAcceptance criteria. One: a customer with one or more stored cards sees them listed at the payment step, most recently used first, showing brand, last four digits and expiry. Two: selecting a stored card and entering the correct CVV completes the payment. Three: a customer with no stored cards sees the existing card form, unchanged. Four: an expired stored card is listed but disabled, with the message This card has expired. Five: if the payment provider does not respond within three seconds, the existing card form is shown and no error page appears. Six: the list renders within 500ms at p95 on a 4G profile. Seven: the build emits checkout_payment_method_selected with a method property of saved_card or new_card.\n\nSignal: share of mobile checkouts completed with a stored card. Baseline 0%, because the feature does not exist yet. Read from the checkout funnel dashboard, using the event added in criterion seven. The epic's own measure stays mobile checkout completion rate, baseline 61.4%.\n\nAssumptions: roughly 45% of mobile sessions are returning customers with a stored token, and if it turns out to be under 20% the epic's £380k share goes back to the outcome owner before more stories are written. The provider returns brand and expiry on the stored token, and if it does not, criterion one drops to last four digits only.\n\nOpen question: does risk require step-up authentication on stored cards above £250? Owner: the risk lead, answer due 14 August. This one blocks the build, so the story stays in the backlog until it is answered.\n\nSize: five points at refinement, comfortably inside a two-week sprint. Product-approved the same day, and it is the approved story that lets the epic leave Ready for Dev." }, "related": [ { "label": "What an AI-native SDLC changes", "href": "/guides/ai-native-sdlc-and-product-delivery" } ], "faq": [ { "question": "Who writes the story, the product manager or the engineer?", "answer": "One named person owns the ticket. In an AI-augmented team that is usually the product manager, who owns the context, the acceptance criteria and the signal. Engineers shape the slice and set the size in refinement, and they are the ones who catch a slice that is secretly two. Approval is a separate pair of eyes from whoever wrote it, otherwise the check is not a check. On Tenhaw engagements that ownership sits with two or three senior people under partner oversight from James Rooney, and that is the whole delivery team." }, { "question": "How small is too small?", "answer": "There is no minimum size, only a floor on visibility, so anything that changes what a user can see or do can be a story however small. If it changes nothing user-visible, it is a chapter under a story rather than a story of its own. The signal you have gone too small is three tickets that always ship together and none of which makes sense alone. That is one story that got filleted." }, { "question": "The epic is not product-approved yet. Can I write stories against it?", "answer": "Yes, and you should. Write them and leave them staged in the backlog. They cannot start until the epic is product-approved, and the epic cannot leave Ready for Dev without at least one product-approved story attached, so the writing has to happen first. What you should not do is start building against an epic whose scope or currency share is still moving." }, { "question": "Do we still need the As a user, I want, so that template?", "answer": "Only if your team still reads it. The template's job was to force the user and the benefit into the ticket. If the title names the user and the change, and the context links the discovery evidence behind it, the template adds a sentence and no information. Keep it if it earns its place in your refinement conversation, drop it if people's eyes slide past it, but do not keep it and then write vague criteria underneath." }, { "question": "What does it mean to slice a user story vertically?", "answer": "A vertical slice crosses every layer it touches, data, service and interface, so shipping it alone changes something a real user can do. \"Build the payments API\" and \"Build the payments screen\" are two halves of nothing shippable. Slice instead on one step in the workflow, one type of user, one business-rule variation, one data source out of several, or the happy path with the edge cases as siblings behind it. Then run one test on the slice. If we shipped only this, could a user do something they could not do yesterday? If not, it is a sub-task under a story, not a story of its own." }, { "question": "How many acceptance criteria should a user story have?", "answer": "Three to seven, each one passable or failable by someone who did not write the code. Given, when, then works, and so does a plain checklist; opinions do not. Replace \"the page should feel fast\" with a number, such as \"renders within 500ms at p95 on a 4G profile\", and write the negative paths as deliberately as the happy one: empty state, expired data, the dependency timing out. Treat length as a diagnostic too. A story with thirty criteria is an epic wearing a story's ticket type, and the fix is sibling stories, not more criteria." }, { "question": "Should every user story have a metric?", "answer": "Yes, but not a financial one. The epic carries the value; the story carries the signal that shows the epic is moving. Write four things on the ticket: the metric, today's baseline, the dashboard or query the number is read from, and the event the build must emit for that number to exist at all. If the instrumentation does not exist yet, make it an acceptance criterion in this story rather than a follow-up nobody schedules. Tenhaw reads its own work the same way, reporting measurable value monthly and recording a month that delivers none as a failed month. A story that ships with no way to read its effect becomes an argument in three months' time." }, { "question": "How do you handle assumptions and open questions in a user story?", "answer": "List every assumption with what you will do if it turns out wrong, one line each. Assume under 5% of sessions hit this, and add pagination if it is higher. Give every open question a named owner and a date, not \"the team\" and not \"soon\". Then apply two rules. A question that blocks the build blocks the story leaving the backlog. A question you have chosen to proceed without is a decision, recorded as one, so nobody discovers the gap mid-sprint." }, { "question": "How much detail does a user story need?", "answer": "Enough that a developer who missed refinement can build it from the ticket alone, with no follow-up conversation. That means four sentences of context in a fixed order: which user, where in their journey, what happens today, what should happen once this ships. Link the discovery evidence and the parent epic rather than restating them. Then add a line headed \"not in this story\", naming the sibling tickets that cover the rest. A developer with a gap in front of them fills it generously and the sprint pays for it, which is why an implicit boundary is the fastest route to gold-plating." }, { "question": "Should an AI-native team split an epic into stories?", "answer": "No. In The Tenhaw Way, stories are an AI-augmented practice, for teams where a developer writes the code and AI assists. An AI-native team, where the model builds and the developer directs and verifies, stops at the epic and writes an outcome ticket instead: the value share, the key user journeys and the test requirements, handed over whole. Tenhaw publishes both modes in full and free to adopt without hiring anyone, and the line between them falls at exactly this ticket type. Splitting that into stories strips out the context the model needed and pre-empts judgement it can exercise itself. The habit tends to outlive the mode change because the ticket templates and the refinement calendar did not change with it." }, { "question": "How long does it take to write a user story?", "answer": "About two hours for the ticket itself, plus sizing with the team at refinement. That two hours covers anchoring the story to an approved epic, checking the slice is vertical, four sentences of context, three to seven acceptance criteria, the signal with its baseline, and the assumptions with what you will do if each one is wrong. Tenhaw writes stories to that same two hours inside client teams. If it is taking a day, the slice is too big or the parent epic is still moving and the story keeps chasing it, so the problem is upstream rather than in the writing. Fix that first. Two hours is cheap against a vague ticket that passes review and then fails in testing three weeks later." }, { "question": "What should you leave out of a user story?", "answer": "Everything this ticket will not deliver, named rather than assumed. Add a line headed \"not in this story\" listing the sibling tickets that cover the rest. An implicit boundary is the fastest route to gold-plating, because a developer with a gap in front of them fills it generously and the sprint pays for it. Edge cases can sit behind the happy path as siblings when that keeps the slice shippable. Leave out anything with no user-visible change, since that is a chapter under the story rather than a story of its own. Leave the discovery evidence and the epic detail out too, and link both instead, so each claim stays attached to its source." }, { "question": "How should a user story be titled?", "answer": "Title it as a person doing a thing, so \"Returning customer pays with a saved card\", not \"Implement saved-card retrieval\". The subject is the user, the verb is observable from outside the codebase, and the whole title runs under about ten words. If the only honest subject you can find is a service or a component, you sliced by layer rather than by the user's journey, so reslice before writing anything else. That makes the title the cheapest diagnostic on the board. A ticket named after a component is usually layer-sliced work, and layer-sliced work ships nothing a user can use, so it cannot be credited with any part of the epic's planned value at validation." }, { "question": "What does product check before approving a story?", "answer": "Three things, and it is a check rather than a signature. Is the slice user-visible, can every acceptance criterion be passed or failed by someone else, and does the story belong under this epic rather than a neighbouring one. The approver has to be someone other than the author, or nothing is actually being checked. If you would have to explain the ticket in person for it to make sense, send it back instead of explaining it. Do this early for at least one story per epic, because the epic cannot leave Ready for Dev without product approval, engineering approval and an approved story attached." }, { "question": "Where do non-functional requirements go in a user story?", "answer": "In the acceptance criteria, as numbers or named standards, not in a document nobody opens. Performance, resilience and accessibility rules only get tested if a tester can pass or fail them, so write \"the saved-card list renders within 500ms at p95 on a 4G profile\" instead of \"the page should feel fast\", and \"if the payment provider does not respond within three seconds, the existing card form is shown and no error page appears\" instead of a note about graceful degradation. Keep them beside the functional criteria for the same slice. A non-functional rule that lives one layer up in a standards page is a rule nobody was asked to meet on this ticket." }, { "question": "Can a story sit under more than one epic?", "answer": "No. A story links to exactly one epic, and that epic links to one outcome that carries a currency target and sits in this quarter's roadmap. If a story looks like it serves two epics, either you sliced across the seam between them and are holding two stories, or the epics themselves overlap and need fixing before more work hangs off them. Product approval checks that link deliberately, because the chain is what lets you say at the end of the quarter which work moved which number. A story hanging off a broken chain is not a filing problem, it is work nobody can defend." }, { "question": "What does AI get wrong when it drafts a user story?", "answer": "The parts it cannot know. Given a committed epic, a model produces a plausible ticket structure fast, which beats an empty backlog, but it writes criteria that sound testable and that nobody can actually pass or fail, and assumptions stated with a confidence it has no basis for. It does not know today's baseline, the dashboard or query the number is read from, or whether the event the build must emit exists yet. Tenhaw's engineering handbook, open on GitHub, holds 72 rules an agent enforces rather than a human remembers. Treat the draft as a first pass and edit three things by hand before approval: the acceptance criteria, the signal, and each assumption with what you will do if it is wrong." }, { "question": "What does a well-written user story look like?", "answer": "Tenhaw's story guide works one through end to end at Northbay Outdoors, an invented retailer with invented numbers. It is titled \"Returning customer pays with a saved card\", and the context says a customer who has bought in the past twelve months reaches the mobile payment step, retypes the full card number today, and afterwards selects a stored card showing brand, last four digits and expiry and pays with CVV only. Seven criteria cover the empty state, an expired card and a three-second provider timeout. The signal is the share of mobile checkouts paid with a stored card, baseline zero. It was five points at refinement and product-approved the same day." } ], "aiNote": "In an AI-augmented team the story is the unit of work and this guide applies as written. AI can draft a first-pass story structure from a committed epic, which beats starting from an empty backlog, but the criteria, the signal and the assumptions need a human edit before approval: a model drafting from epic text will produce plausible criteria that nobody can pass or fail, and confident assumptions it has no basis for. In an AI-native team, do not write stories at all. The unit of work is an outcome ticket at epic level carrying the currency share, the key user journeys and the test requirements, handed over whole. Splitting it into stories strips out the context the model needed and pre-empts judgement it can exercise itself." } ## How to write a developer subtask Source: https://tenhaw.com/the-tenhaw-way/how-to/write-a-chapter Last updated: 2026-11-05 { "slug": "write-a-chapter", "group": "Writing the work", "title": "How to write a developer subtask", "summary": "Use a small technical subtask, called a chapter in this method, to finish a story without changing its acceptance boundary.", "intro": "A chapter is the fourth and optional level of the breakdown: a sub-task a developer creates mid-build, once a story turns out to be bigger than refinement thought. It carries no currency value, no product approval and no user-visible slice, because the story above it holds all three. A good chapter names one technical deliverable, is reviewable on its own, takes under two days, and leaves the story's acceptance criteria untouched. Chapters exist so a big story stays visible and reviewable, not so a big story stays hidden.", "metaDescription": "Write a developer subtask with one technical deliverable, review evidence and a clear parent story. A practical guide with an insurer renewal example.", "whenToUse": "Use this the moment you are mid-build on a story and realise it will not finish in one clean pass. If you are reaching for chapters before development has started, you have a sizing problem in refinement, not a chapter.", "totalTime": "PT1H", "timeLabel": "About an hour, mid-build", "tools": [ "The delivery tracker", "The parent story's feature flag" ], "steps": [ { "name": "Run the two-question test before writing it", "detail": "Question one: if this shipped alone, could a user do something new? If yes it is a story, product owns it, take it back. Question two: do the parent story's acceptance criteria fail without it? If they pass regardless, it is not a chapter, it is tech debt or a bug, and it links to the quarter's Tech Debt or Bug Budget epic. Only work that is invisible to the user and required for the story to pass is a chapter. A migration, an idempotency key, a flag wiring change: chapters. A new screen, a new field, a new email: stories, however small." }, { "name": "Split at the keyboard, not in refinement", "detail": "Create chapters when the code has told you something refinement could not: a service returns stale data, a migration has to run before the endpoint changes, a dependency needs versioning first. If you can list the chapters before anyone opens the editor, you have not found chapters, you have found a story that is too big. Take it back, split it into stories that each change something a user can see, and get product approval on each one. Chapters written in advance become a private backlog nobody outside the team ever reads." }, { "name": "Cut along seams, cap the count at four", "detail": "Split where the system already has a joint: a schema change, a service boundary, a contract between two components, a feature flag. Each chapter should merge on its own behind the parent's flag, so one pull request is one chapter and a colleague can judge it without waiting for the next. Half a day to two days each. If a chapter will not fit in two days, do not nest another level, because there is no level below chapter. Move the seam instead. If you need more than about four, the story was several stories and belongs back with product." }, { "name": "Write five fields and stop", "detail": "Title: a verb plus the thing, for example \"Make the renewal accept endpoint idempotent\". Parent story: one link, nothing else. What changes: the files, services or tables affected. How you will know: the test or check that proves it, an engineering check rather than a product criterion. Out of scope: the neighbouring work you are deliberately not doing here. Under two hundred words in total. Never name a chapter \"Part 1 of 3\", and never split by backend and frontend out of reflex: neither tells a reader what is left, and neither half can be reviewed alone." }, { "name": "Sequence by dependency, keep one in progress", "detail": "Order chapters by what unblocks what, and put schema and contract changes first so the later ones build on a settled shape. Keep exactly one in progress. Three chapters running in parallel across two developers finishes the story later than one developer taking them in order, because the merges fight each other and the review queue backs up. Chapters run the same phases as everything else: todo, in progress, in review, done. If a later chapter turns out to be unnecessary once the earlier ones land, close it with the reason rather than deleting it." }, { "name": "Close chapters on engineering, the story on product", "detail": "A chapter is done when its own check passes and a human has reviewed the change. That is the whole bar: no currency figure, no product approval, no acceptance criteria of its own, because the epic already carries the value and the story already carries the criteria. Copying either downwards double-counts the value and gives you two competing versions of done. The story moves to in review only once every chapter is closed and its acceptance criteria are demonstrated end to end, not inferred from three green chapters. Chapters do not release, and value monitoring is for epics." }, { "name": "Take the split to refinement, not the points", "detail": "Refinement re-points work whose understanding has shifted, but not a story already in flight: that story keeps the size it started with, so cycle time stays honest and your p50 and p85 forecasts keep meaning something. Bring the split to the next fortnightly refinement instead and say two things: what the story looked like going in, and what it turned out to contain. Two or three of these a quarter is normal. The same shape every fortnight, always around the same service or the same kind of change, is a sizing signal to act on." } ], "pitfalls": [ "Writing the chapters at refinement, before anyone has opened the code. It reads as diligence and it is a sizing failure in disguise: work that is knowable in advance belongs in stories product can see and approve, not in sub-tasks only the team reads.", "Burying a refactor or a defect as a chapter because raising it properly takes longer. The Tech Debt and Bug Budget epics exist so the quarter's burn rate is visible, and every item hidden under a story makes that number a lie.", "Giving a chapter a currency value or its own product acceptance. Both double-count: the epic already holds the planned share of the outcome, the story already holds the criteria, and a chapter carrying a number appears twice when somebody sums the quarter.", "Re-pointing the parent story once the chapters appear, so the burn-down looks tidier. It corrupts the only throughput data you have, and the p50 and p85 forecasts built on it get quietly worse while the board looks better.", "Naming chapters \"Part 1 of 3\", or splitting by backend and frontend out of habit. Nobody reading the board can tell what is left, and the reviewer cannot judge one half without the other, which removes the reason for splitting at all." ], "definitionOfDone": [ "Every chapter names one technical deliverable, links to exactly one story, and merges on its own behind the parent's flag.", "No chapter carries a currency value, a product approval or an acceptance criterion of its own.", "The chapters together cover the parent story's acceptance criteria, and anything outside them was raised against the quarter's Tech Debt or Bug Budget epic.", "Each chapter is under two days, only one is in progress at a time, and there are no more than about four.", "The story moved to in review only after every chapter closed and its criteria were demonstrated end to end.", "The split was raised at the next refinement and the parent story's original size was left unchanged." ], "example": { "heading": "Splitting a renewal story at Coastal Mutual, an invented insurer", "body": "Coastal Mutual and its numbers are invented. The outcome is \"Cut renewal churn\", targeted at £1.4m of retained annual premium across two quarters. Under it sits this quarter's epic, \"One-click renewal in the customer portal\", carrying a planned £420k of that target. Under the epic sits a story: \"A renewing customer can accept their quote in one click from the renewal email.\" The epic has product and engineering approval, the story is product-approved, and it was sized at five points and expected to take two days. On day two the developer finds two things refinement could not have known. The quote service returns a stale premium when the policy had a mid-term adjustment in the previous 24 hours, and the accept endpoint has no idempotency, so a double-click creates two policies. Neither changes what the user sees, so this stays one story and gets three chapters. One: recalculate the premium at accept time when a mid-term adjustment exists in the last 24 hours. Check: an integration test with an adjustment timestamped three hours ago returns the adjusted figure. Half a day. Two: make the accept endpoint idempotent on a request key. Check: two identical accepts 200 milliseconds apart create one policy. One day. Three: build the accept flow in the portal and the link in the email. Check: the journey works end to end behind the renewal-v2 flag. One day. They run in that order, one at a time, one pull request each. Two things did not become chapters. An SMS renewal reminder is user-visible, so it becomes a new story under the same epic and product decides whether it belongs this quarter. Moving the quote service off its deprecated pricing provider is not needed for the story's criteria to pass, so it is raised against the quarter's Tech Debt epic. The story keeps its five points. At the next refinement the developer reports it in one line: sized for two days, contained a stale-premium path and a missing idempotency key, took four." }, "related": [ { "label": "What an AI-native SDLC changes", "href": "/guides/ai-native-sdlc-and-product-delivery" } ], "faq": [ { "question": "Can a chapter have chapters of its own?", "answer": "No. Chapters are the last level. A chapter you cannot finish in two days is telling you the seam is in the wrong place, or that the parent should have been more than one story. Move the seam first, and if that does not work, stop and take the story back to product rather than inventing a level below." }, { "question": "Do chapters get story points?", "answer": "No. Points stay on the parent story, and that story keeps the size it was given before the build started. Size chapters in days, for sequencing only, and never sum those days back onto the story. The moment chapter sizes feed your velocity, the throughput data that produces your p50 and p85 stops being comparable across quarters." }, { "question": "Does product need to approve chapters?", "answer": "No. The developer creates and closes them, and they should still be visible on the board so the chain from chapter to story to epic to outcome holds. If product is being asked to make a call on a chapter, whatever is under discussion is user-visible and should have been raised as a story." }, { "question": "What if a chapter turns out to be user-visible after all?", "answer": "Convert it. Raise it as a story under the same epic, get product approval, and let product decide whether it belongs in this quarter or the next. Do not ship a user-visible change under a chapter because the branch and the flag happen to be there already, which is exactly how work disappears from the board product reads." }, { "question": "What is a chapter in The Tenhaw Way?", "answer": "A chapter is The Tenhaw Way's name for a sub-task, the fourth and optional level of the breakdown, below the outcome, the epic and the story. A developer creates chapters mid-build, once a story turns out to be bigger than refinement thought. A chapter carries no currency value, no product approval and no user-visible slice, because the story above it holds all three. A good one names one technical deliverable, merges on its own behind the parent story's flag, and takes half a day to two days, about four per story at most. Chapters exist so a big story stays visible and reviewable, not so it stays hidden. Tenhaw publishes the whole method, these rules included, free to adopt without hiring us." }, { "question": "How do you split a story that turns out to be bigger than estimated?", "answer": "Split it mid-build along the system's existing seams: a schema change, a service boundary, a contract between two components, a feature flag. Each sub-task should name one technical deliverable, merge on its own behind the parent story's flag, and take between half a day and two days, with no more than about four in total. Do not nest a level below; if a piece will not fit in two days, move the seam instead. And keep the split invisible to the user. Anything a user could see when shipped alone is a new story that needs product approval, not a sub-task." }, { "question": "What is the difference between a sub-task and a story?", "answer": "A story changes something a user can see and needs product approval; a sub-task is invisible to the user and exists only so the story's acceptance criteria can pass. Two questions settle it. If the work shipped alone, could a user do something new? If yes it is a story. A new screen, a new field or a new email all qualify, however small. Do the parent story's acceptance criteria fail without it? If they pass regardless, it is not a sub-task either, it is tech debt or a bug and belongs against those budget epics. A genuine sub-task, like a migration or an idempotency key, is invisible to the user and something the criteria cannot pass without." }, { "question": "Should sub-tasks be created during planning or during development?", "answer": "During development. Create sub-tasks when the code has told you something planning could not: a service returns stale data, a migration has to run before an endpoint changes, a dependency needs versioning first. If you can list the sub-tasks before anyone opens the editor, you have not found sub-tasks, you have found a story that is too big. Take it back, split it into stories that each change something a user can see, and get product approval on each one. Sub-tasks written in advance read as diligence, but they are a sizing failure in disguise, and they become a private backlog nobody outside the team ever reads." }, { "question": "Should developers work on sub-tasks in parallel or in sequence?", "answer": "In sequence, with exactly one in progress at a time. Order them by what unblocks what, putting schema and contract changes first so later work builds on a settled shape. Three sub-tasks running in parallel across two developers finishes the story later than one developer taking them in order, because the merges fight each other and the review queue backs up. Each sub-task should be one pull request that merges on its own behind the parent story's flag, so a colleague can review it without waiting for the next. If a later one becomes unnecessary once earlier work lands, close it with the reason rather than deleting it." }, { "question": "Should a refactor found mid-build be a sub-task of the story?", "answer": "Only if the story's acceptance criteria fail without it. A genuine sub-task is work the story cannot pass without, and that is the whole test. A refactor the criteria do not depend on belongs against the quarter's Tech Debt epic instead, raised properly rather than buried under a story because raising it takes longer. Those budget epics exist so the quarter's burn rate stays visible, and every item hidden under a story makes that number a lie. Defects you trip over mid-build follow the same rule. If the story's criteria pass regardless, they are raised against the Bug Budget epic, not slipped in as a sub-task." }, { "question": "Is it OK to split a ticket into backend and frontend sub-tasks?", "answer": "No, and it is the split that most often defeats the point of splitting. Backend and frontend are two halves of one change. The reviewer cannot judge either half without the other, and nobody reading the board can tell what is actually left. A sub-task has to stand up on its own, one pull request that merges behind the parent story's flag, so cut at a joint the system already has instead. Names like Part 1 of 3 fail the same test, because they describe the order somebody happened to work in rather than what the work is." }, { "question": "What should a sub-task ticket actually contain?", "answer": "Five fields, and stop, in under two hundred words in total. A title that is a verb plus the thing, such as Make the renewal accept endpoint idempotent. One link to the parent story, nothing else. What changes: the files, services or tables affected. How you will know: the test or check that proves it, an engineering check rather than a product criterion. And out of scope: the neighbouring work you are deliberately not doing here. Anything beyond that is either restating the story above it or hiding a second sub-task inside this one." }, { "question": "Is splitting a story mid-sprint just extra admin?", "answer": "It costs about an hour, once, on the story that surprised you. Four sub-tasks at most, five short fields each, under two hundred words in total, written at the keyboard by the developer who found the problem. Tenhaw's engineers work that way on client delivery, against a published standard of 72 rules written to be enforced by an agent. Set that against the alternative, where a story sized at two days quietly takes four, one large pull request lands at the end, and until it does nobody outside the team can see what is left or review any of it in pieces. The admin is not the split. It is what you pay when a big story stays hidden instead of staying visible and reviewable." }, { "question": "Is a story finished once all its sub-tasks are closed?", "answer": "No. A sub-task closes on engineering when its own check passes and a human has reviewed the change. The story moves to in review only once every sub-task is closed and its acceptance criteria are demonstrated end to end, not inferred from three green sub-tasks in a row. That gap is where defects live, because each piece was verified against its own technical check and never against what the user was actually promised. Sub-tasks do not release either, and value monitoring stays at epic level, where the currency target sits." }, { "question": "How often is it normal for a story to split mid-build?", "answer": "Two or three times a quarter is normal, and it deserves one line at the next fortnightly refinement rather than a post-mortem, covering what the story looked like going in and what it turned out to contain. The pattern is what matters. The same shape every fortnight, always around the same service or the same kind of change, is a sizing signal to act on. Tenhaw watches that rate on client delivery, the same attention to flow that cut delivery lead times by 60% within six months at Globelynx. At that refinement, resist re-pointing a story already in flight to tidy the burn-down, because it corrupts the throughput history your p50 and p85 forecasts are built on." }, { "question": "Why can't a sub-task carry its own value target?", "answer": "Because the value is already held above it, and putting a number on the sub-task counts it twice. Tenhaw reports the money at outcome and epic level for the same reason. The epic carries the planned share of the outcome in currency and the story carries the acceptance criteria, so a sub-task with its own figure appears a second time when somebody sums the quarter, and the roadmap starts claiming more value than the outcome was ever priced at. Giving one its own product acceptance does the same damage in a different direction. You end up with two competing versions of done, and a developer choosing between them mid-build." }, { "question": "How do we introduce sub-tasks to a team that doesn't use them?", "answer": "Do not roll them out as a process. A sub-task is what a developer writes at the keyboard, once the code has told them something refinement could not, so the habits matter more than the tooling: create one only mid-build; cap them at about four per story, half a day to two days each; keep exactly one in progress; give each five fields and no currency value; and report the split in a line at the next refinement without re-pointing a story already in flight. Tenhaw calls them chapters and passes habits like these on by pairing with a client's own engineers. The name matters less than the rules. Run it for a quarter and your real sizing problems surface on their own." }, { "question": "Should you split a ticket before handing it to an AI coding agent?", "answer": "No. An AI-native team, where the model builds and the developer directs and verifies, runs two levels, so there is nothing to split. The outcome ticket is the unit of work and it goes over whole. If you feel the urge to decompose it, read that as a signal about the ticket rather than about the size of the work. It is usually thin, missing a key user journey or a test requirement, and the fix is upstream. Add what is missing, hand the ticket over again, and do not invent a sub-level the mode does not have. Tenhaw staffs that mode as an Agentic Build Team of three practitioners under partner oversight, with James Rooney on every engagement." } ], "aiNote": "This is an AI-augmented practice. AI-native teams run two levels, so there is no story to split and no chapter to write: the outcome ticket is the unit of work. If you are directing a model and feel the urge to decompose, the ticket is thin rather than the work being big, and the fix is upstream. Something is missing from it, usually a key user journey or a test requirement. Add that, hand the ticket over whole again, and do not invent a sub-level the mode does not have." } ## How to write a bug report Source: https://tenhaw.com/the-tenhaw-way/how-to/raise-a-bug Last updated: 2026-11-05 { "slug": "raise-a-bug", "group": "Writing the work", "title": "How to write a bug report", "summary": "Record the approved behaviour that failed, the evidence to reproduce it and the test that will close it.", "intro": "A bug is a defect in something already shipped: live behaviour that contradicts what was approved. It is not a missed requirement, and it is not code you dislike. Every bug hangs off the quarter's Bug Budget epic, so opening one still traces up to a priced outcome. A good bug ticket reproduces on someone else's machine, states expected against actual with the approved source linked, carries a severity set against a published rubric and priced per week, and closes on a regression test that failed before the fix and passes after it.", "metaDescription": "A missed requirement is not a bug. A bug contradicts behaviour that was approved and shipped, and closes on a regression test that failed before the fix.", "whenToUse": "Raise a bug when live behaviour contradicts something that was approved and shipped. If nobody ever agreed the behaviour being demanded, you are writing a story or an epic. If no user can see the problem, you are writing tech debt.", "totalTime": "PT1H30M", "timeLabel": "About ninety minutes, to raise and triage it", "tools": [ "The delivery tracker, and the quarter's Bug Budget epic", "The environment the defect was seen in", "Logs, a trace ID and a screen recording of the failure" ], "steps": [ { "name": "Classify it before you write a word", "detail": "Three destinations, one question each. Did someone approve this behaviour and did we ship something different? Bug, under the quarter's Bug Budget epic. Did nobody ever agree it? New scope: a story under an existing epic, or a new epic if it carries its own value. Is the defect invisible to users and painful only to engineers? Tech debt, under the quarter's Tech Debt epic. Read the source before you decide: the acceptance criteria on the story, or the user journeys and test requirements on the outcome ticket in an AI-native team. Where the source is silent, it is scope, not a bug." }, { "name": "Reproduce it three times, then record the path", "detail": "Write steps a stranger can follow without asking you anything: environment, build or release version, account type and permissions, data state, the exact input, and the timestamp of an occurrence you watched happen. Run it three times and record the hit rate, three of three or one of five, because intermittency changes both the fix and the test. Attach a request or trace ID, the log lines either side of the failure, and a short screen recording. If you cannot reproduce it, say so in the first line rather than leaving it implied, and list what you tried, how many users hit it per day, and where you stopped." }, { "name": "State expected against actual, cite the source", "detail": "Two sentences, then a title. Expected: the behaviour that was approved, quoting the acceptance criterion, user journey or test requirement it comes from, with a link. Actual: what happens instead, in the same terms. Citing the source is what stops triage becoming an argument about whether anyone agreed the behaviour. Title in the shape area, what breaks, for whom, under what condition. \"Checkout: card payments over £500 fail for guest users with a generic error\" beats \"payments broken\". Keep your theory of the cause out of the title and put it in the body, labelled as a theory." }, { "name": "Set severity by impact, then price it", "detail": "Publish a rubric and hold to it. S1: a key journey cannot be completed, no workaround exists, or there is revenue, safety or regulatory exposure. S2: a key journey is degraded and support can talk a user through a workaround. S3: everything else, including cosmetic defects and edge cases with negligible reach. Then price it: affected users per week times the value lost per affected user, using the conversion and basket numbers the parent outcome was priced with. A weekly cost in currency makes the bug rankable against features. One named person may downgrade a severity, and the reason goes in the ticket." }, { "name": "Link it to the budget and its origin", "detail": "Attach every bug to the current roadmap's Bug Budget epic, so the chain from defect to priced outcome stays intact. Then add the link most teams skip: the epic, story or release that introduced it, traced from the release that first shipped the broken behaviour. Keep an \"introduced by\" field and leave it empty rather than guessing. That backlink turns the budget from a bucket into three numbers at quarter close: burn against forecast, defects per epic, and the severity mix quarter on quarter. Without it you have a count of bugs, which measures how busy you were." }, { "name": "Make the rollback call explicitly", "detail": "Anything found inside the seven-day live monitoring window gets a continue, watch or roll back call from a named person, on the record, rather than by drift. Compare two numbers you can write down: exposure per day in currency, and hours to a fix you would trust. Roll back when exposure per day exceeds what the epic earns per day and the trusted fix is more than a day out, or when you cannot yet bound the blast radius. Record the call, both numbers and the reasoning in the bug, including a decision not to roll back, because the root cause analysis will ask." }, { "name": "Size it and schedule it against the budget", "detail": "Bugs go through refinement and sizing like everything else, and through the same phases: todo, in progress, in review, done. Set the budget from your own data, taking the capacity defects consumed over the last three quarters as a percentage of throughput and reserving that much. It is a forecast, not a permission slip. S1s interrupt the sprint. S2s are scheduled inside the quarter. S3s compete on value with everything else, and some should be closed unfixed with a note saying so. Past half the budget before half the quarter, raise it at the next monthly health check while there is still quarter left to react in." }, { "name": "Fix with a failing test, close with evidence", "detail": "Write the regression test first and watch it fail on the affected build. A fix with no test that failed beforehand is a claim, and it is why the same defect returns two releases later. Name the test and the failing build in the ticket so the next person can rerun it. Close when the test passes, a human has reviewed the change, product has approved it, and the behaviour has been demonstrated in the environment where it was reported, not only in staging. Book the root cause analysis for any S1 before you close, and investigate the system that let it through, not the person who wrote the line." } ], "pitfalls": [ "Reclassifying bugs as stories so the budget looks healthy. It happens quietly, usually under pressure near quarter close, and it makes the burn rate worthless: the number stops describing quality and starts describing how motivated the team was to protect it.", "Severity inflation. When everything is an S1, nothing is, and the on-call rota becomes a lottery. It happens because severity gets set by whoever is most annoyed rather than against a published rubric, so the fix is the rubric plus one named person allowed to downgrade.", "The unreproducible bug that ages for a quarter. Nobody closes it because nobody can prove it has gone, so it sits radiating uncertainty. Give it a decision date fourteen days out: reproduce it, ship instrumentation to catch it next time, or close it with the reasoning written down.", "Prompting the fix straight into the code in an AI-native team. The symptom disappears and the requirement corpus stays untouched, so the next full re-evaluation rebuilds the defect. The corrected journey and the failing test requirement belong in the outcome ticket first.", "Closing on the model's word. A model reporting that it has fixed something is not evidence that it has, any more than it is evidence when it reports a feature is finished. The passing regression test and the demonstrated journey are the evidence." ], "definitionOfDone": [ "Reproduction steps are written with a hit rate recorded, and someone other than the reporter has followed them successfully, or the ticket states plainly that it is not reproducible and lists what was tried.", "Expected and actual behaviour are both stated, with a link to the acceptance criterion, journey or test requirement that made the expected behaviour agreed.", "Severity is set against the published rubric, the user impact is named, and a weekly cost in currency is estimated for anything above S3.", "The bug is linked to the quarter's Bug Budget epic and, where identifiable, to the epic, story or release that introduced it.", "A named regression test failed on the affected build and passes after the fix, and the behaviour has been demonstrated in the environment where it was reported.", "The continue, watch or roll back call is recorded against a named person with the two numbers behind it, and any S1 has a root cause analysis booked before the bug closes." ], "example": { "heading": "A checkout defect, priced and routed", "body": "Northbound, an invented online retailer, runs a Q3 roadmap against the outcome \"Lift checkout conversion from 2.1% to 2.4%\", target £1.2m. One epic under it, \"Guest express payments\", carries a planned £280k and released on day 31 of the quarter. Three days into live monitoring, support flags that guest users paying by card on baskets over £500 get a generic error. It reproduces three times out of three: guest session, card payment, basket £512, build 26.7.3. The ticket reads \"Checkout: card payments over £500 fail for guest users with a generic error\", links the acceptance criterion that approved the £500 path, and prices the exposure: about 30 failed attempts a day at an average £610 basket, of which roughly half recover by signing in or retrying, so £9k a day. The rubric says S2 because a workaround exists, but guests hit it silently and never contact support, so it goes S1 on revenue exposure with the reason recorded. Trusted fix is two days against £9k a day, so the release is rolled back the same afternoon, named and reasoned in the ticket. The fix ships with a regression test that failed on 26.7.3, and the bug carries an \"introduced by\" link to the guest payments epic, which is what the quarter close report will read." }, "related": [ { "label": "Bounding what a wrong answer costs", "href": "/guides/guardrails-hallucination-and-accuracy-control" }, { "label": "Evaluation, ground truth and the release gate", "href": "/guides/agent-evaluation-and-assurance" } ], "faq": [ { "question": "Is a missed requirement a bug?", "answer": "No. If nobody approved the behaviour being asked for, nothing is defective, the product is doing what was agreed. That is new scope, either a story under an existing epic or a new epic carrying its own share of an outcome's value. The test is whether you can link to an acceptance criterion, user journey or test requirement that the live behaviour contradicts. If you cannot produce that link, you are raising a change request wearing a bug's clothes, and the bug budget will lie about quality all quarter." }, { "question": "Do defects found before release count against the bug budget?", "answer": "No. A story in progress or in review that does not meet its acceptance criteria is not done, so send it back rather than opening a bug. The same applies in an AI-native team when a test requirement in the outcome ticket fails before release. The bug budget forecasts what escapes into production, and polluting it with in-flight rework destroys the one signal it carries. Track pre-release rework, if you want it, as its own measure of how well work is being written and reviewed." }, { "question": "Who sets severity, and can it be changed?", "answer": "The reporter sets an opening severity against the published rubric, because they have seen the impact. One named person, usually the product manager who owns the parent outcome, is allowed to change it, and the reason goes in the ticket. Set the rubric on user and revenue impact, never on how loudly the request arrived. Severity decides three things: whether the bug interrupts the current sprint, whether it triggers a rollback conversation, and whether it earns a root cause analysis, so an inflated S1 costs real capacity." }, { "question": "What do we do when the bug budget runs out mid-quarter?", "answer": "Raise it at the next monthly health check and treat it as a forecast that has broken, not a cap you have breached. Tenhaw reports its own months on those terms, and a month that moves no number the client agreed is written up as a failed month. S1s still interrupt the sprint, because a budget does not make revenue exposure acceptable. What changes is the planned work. Something in the quarter gives way, and product decides which epic slips rather than letting the team absorb it silently. Then look at where the defects came from using the introduced-by links, because an overrun concentrated in one epic is a different problem from one spread evenly." }, { "question": "What makes a good bug report?", "answer": "One a stranger can act on without finding you. It reproduces on someone else's machine, with the environment, build version, account type, data state and exact input written down. It states expected against actual in two sentences, linking the acceptance criterion or approved journey the live behaviour contradicts, which is what stops triage becoming an argument about whether anyone agreed the behaviour. Severity is set against a published rubric, and anything serious carries a weekly cost in currency. The title names the area, what breaks, for whom and under what condition, so \"Checkout: card payments over £500 fail for guest users\" beats \"payments broken\". Keep your theory of the cause in the body, labelled as a theory." }, { "question": "How do you decide between rolling back and fixing forward?", "answer": "Compare two numbers you can write down, the exposure per day in currency and the hours to a fix you would actually trust. Roll back when the daily exposure exceeds what the change earns per day and the trusted fix is more than a day out, or when you cannot yet bound the blast radius. The call is made explicitly by a named person, on the record, not by drift. On Tenhaw engagements that person is James Rooney, who leads every engagement personally and stays the named escalation route throughout. Record the decision in the ticket with both numbers and the reasoning, including a decision not to roll back, because the root cause analysis will ask for it." }, { "question": "What should we do with a bug nobody can reproduce?", "answer": "Say so in the first line rather than leaving it implied, then list what you tried, how many users hit it each day, and where you stopped. Attach whatever evidence exists: a trace ID, the log lines either side of the failure, a screen recording. Then give it a decision date fourteen days out with three ways to resolve: reproduce it, ship instrumentation that will catch it next time, or close it with the reasoning written down. The failure mode to avoid is the unreproducible bug that ages for a quarter because nobody can prove it has gone, sitting in the backlog radiating uncertainty." }, { "question": "How do you put a cost on a bug?", "answer": "Multiply the affected users per week by the value lost per affected user, using the same conversion and basket numbers the parent work was priced with, so both sides of the ledger match. A checkout defect hitting 30 guest payments a day at an average £610 basket, where roughly half recover by retrying, costs about £9k a day. A weekly figure in currency does two jobs. It makes the bug rankable against feature work in planning, and it gives the rollback decision a number to weigh against the hours to a trusted fix. Without it, priority gets set by whoever is most annoyed. Tenhaw reports its own engagements as a measured movement, with delivery lead times at Globelynx falling 60% within six months." }, { "question": "Should you fix every bug?", "answer": "No. Fix by severity and by value. A defect that blocks a key journey with no workaround, or carries revenue, safety or regulatory exposure, interrupts the sprint immediately. A degraded journey with a workaround gets scheduled inside the quarter. Everything else, cosmetic defects and edge cases with negligible reach, competes on value with the rest of the backlog, and some of it should be closed unfixed with a note saying so. Closing a low-value bug deliberately, with the reasoning recorded, is healthier than letting it age in the backlog pretending it will ever be scheduled." }, { "question": "How do you stop the same bug coming back?", "answer": "Write the regression test before the fix and watch it fail on the affected build. Tenhaw closes defects on that failing-then-passing test, against an open-source standard of 72 rules with RFC 2119 severities. A fix with no test that failed beforehand is a claim, and it is why the same defect returns two releases later. Name the test and the failing build so the next person can rerun it, and close only when the behaviour is demonstrated in the environment where it was reported, not staging. When a model wrote it, correct the requirement first, the user journey or test requirement on the ticket, then have the model implement against it; patching the symptom leaves the requirement wrong, and the next rebuild reintroduces the defect." }, { "question": "How long should it take to raise and triage a bug?", "answer": "About ninety minutes, covering both the writing and the triage. That pass classifies it, reproduces it three times, states expected against actual with the approved source linked, sets a severity and a weekly cost, links it to the quarter's bug budget and to the release that introduced it, and records the continue, watch or roll back call. Against dropping a message into a channel that sounds heavy, and it is meant to. The time comes back twice. Nobody re-argues at triage whether the behaviour was ever agreed, and the tickets never bounce back to you for the detail you left out." }, { "question": "How many times should you reproduce a bug before raising it?", "answer": "Three, and record the hit rate rather than just the fact that it happened. Intermittency changes both the fix and the test that has to prove it, so three of three and one of five are different defects, and an unstated hit rate hides the hardest part of the work. Note the timestamp of an occurrence you watched happen, and attach a request or trace ID, the log lines either side of the failure, and a short screen recording. Then have someone other than the reporter follow the steps successfully before the ticket moves, which is the only real test of whether they were written for a stranger." }, { "question": "When is a defect tech debt rather than a bug?", "answer": "When no user can see it and it is painful only to engineers. Tenhaw made that split pay at Greggs, where throughput data proved that paying down tech debt made delivery faster. A defect users experience goes under the quarter's Bug Budget epic, and one that only slows the team down goes under the Tech Debt epic. Keeping the line clean matters because the bug budget is meant to describe what customers actually met in production, so filing debt against it makes the burn rate say something it does not mean, and filing a customer-visible defect as debt hides it from the quality signal entirely. Decide before you write a word, because everything downstream trusts the classification." }, { "question": "How do you track which release introduced a bug?", "answer": "Keep an \"introduced by\" field on every bug, traced from the release that first shipped the broken behaviour, and leave it empty rather than guessing at it. It is the link most teams skip, and it is what turns the bug budget from a bucket into three numbers at quarter close: burn against forecast, defects per epic, and the severity mix quarter on quarter. Without it you have a count of bugs, which measures how busy the team was rather than where quality is leaking. The quarter close report reads that field, so a bug raised without it goes missing exactly when the pattern matters most." }, { "question": "Who has to sign off before a bug can be closed?", "answer": "Four conditions, rather than one person's opinion. Tenhaw holds the same four inside client repositories, where the code and documentation are the client's from day one. The named regression test passes, having failed first on the affected build. A human has reviewed the change. Product has approved it. And the behaviour has been demonstrated in the environment where the defect was reported, not only in staging, because staging is not where the report came from. Anything set as S1 also has its root cause analysis booked before the bug closes, and that session investigates the system that let the defect through rather than the person who wrote the line. A model reporting that it has fixed something is not evidence that it has." }, { "question": "How do you stop teams relabelling bugs to protect the numbers?", "answer": "Make the classification a test anyone can apply, and apply it at triage rather than at quarter close. Tenhaw publishes that test in full and free to adopt, so a delivery team can be held to a public document rather than a private convention. A bug has to cite the acceptance criterion, journey or test requirement that the live behaviour contradicts, so moving one to a story is a claim that no such citation exists, which is checkable, rather than a quiet edit that makes a number look better. It matters because relabelling happens under pressure near the end of a quarter, and it leaves the burn rate worthless. The number stops describing quality and starts describing how motivated the team was to protect it." }, { "question": "What do you check a bug against when there are no acceptance criteria?", "answer": "The user journeys and test requirements written on the outcome ticket. An AI-native team writes no stories and no acceptance criteria, so those are the approved source, and a bug is live behaviour that contradicts one of them. Cite it by link on the expected line exactly as you would an acceptance criterion, because the citation is what stops triage becoming an argument about whether anyone ever agreed the behaviour. Where the ticket is silent on the behaviour being demanded, you are not looking at a defect at all, you are looking at scope, and it belongs in the ticket as a new requirement." }, { "question": "Do bug tickets need refining and sizing like user stories?", "answer": "Yes. Bugs go through refinement and sizing like everything else, and through the same phases: todo, in progress, in review, done. The reason is capacity. Unsized bug work still consumes the team's week, it just consumes it somewhere the plan cannot see, and a budget tracked in ticket counts rather than in sized work tells you how busy people were rather than how much of the quarter has gone. Sizing also puts a defect and a feature in the same units, which makes the choice between them a planning decision instead of an improvisation. Past half the budget before half the quarter, raise it at the next health check." } ], "aiNote": "In an AI-native team the bug is measured against the user journeys and test requirements written on the outcome ticket, since there are no stories or acceptance criteria to cite. That makes the fix a two-part job: correct the journey or add the missing test requirement in the ticket first, then have the model implement against it. Prompting a patch straight into the code leaves the requirement corpus wrong, so the next re-evaluation reintroduces the defect. Closing evidence is the same in both modes and unusually load-bearing here: the regression test passing and the journey demonstrated, never the model's report that it is done." } ## How to write a risk or issue in a RAID log Source: https://tenhaw.com/the-tenhaw-way/how-to/write-a-risk-or-issue Last updated: 2026-11-05 { "slug": "write-a-risk-or-issue", "group": "Writing the work", "title": "How to write a risk or issue in a RAID log", "summary": "Name the event, its effect on delivery and the person who can make the next decision.", "intro": "A risk is something that might reduce an outcome's value or move its date. An issue is something already doing it. Both are written the same way: one sentence of cause, event and consequence, attached to one epic, priced in currency, owned by one named person, with a date by which a decision has to be taken. Written well, a stranger can read it and act. If you cannot say what value is exposed and by how much, you have written a worry, not a risk.", "whenToUse": "The moment you spot something that could move a date, a number or a scope and you cannot resolve it inside the day. Also the moment an assumption you were relying on turns out to be false, whether or not it has cost you anything yet.", "totalTime": "PT45M", "timeLabel": "About forty-five minutes, per entry", "tools": [ "The RAID log", "The epic the item attaches to" ], "steps": [ { "name": "Classify it: risk, issue, bug, debt or dependency", "detail": "Classify it first. Ask whether the damaging event has happened yet: if it is ahead of you it is a risk, if it is costing time, money or scope now it is an issue. A defect in something already shipped is a bug and links to the quarter's Bug Budget epic. A shortcut that will slow you later is tech debt and links to the Tech Debt epic. Something another team owes you on a date is a dependency. Something you are relying on and cannot prove is an assumption. If it is work with an obvious owner and takes under a day, do it instead of logging it." }, { "name": "Write it as cause, event, consequence", "detail": "Write one sentence in three parts: because [something verifiable that is true today], [event] may happen, which would [effect on the epic's value or its date]. For an issue, change the tense: because X, Y has happened, and it is costing Z. The cause has to be a fact someone could check, not a feeling. Test it by reading it to a person outside the team. If they cannot tell whether the event has already happened, or what it costs, rewrite it. Noun titles like vendor risk or resourcing risk give a reader nothing to decide on." }, { "name": "Attach it to exactly one epic", "detail": "Attach every item to exactly one epic, or to the outcome above them when the threat spans several epics. Do not clone one risk across three epics: raise it once at outcome level with the aggregate exposure, or you will count the same money three times when you rank the log. The link is what lets a reader trace from the item to a currency target in one step, and it decides who reviews it and which quarter it sits in. If there is no live work to attach it to, it belongs on the company-level log, or it is an assumption rather than a risk." }, { "name": "Price the exposure in currency", "detail": "Two numbers, then multiply. Value at risk is the share of the epic's planned contribution that is threatened, which is rarely all of it: a one-quarter delay to a £400k epic exposes roughly a quarter of the annual benefit, so write £100k and record the assumption behind it so someone can argue with it. Probability is your honest estimate in tens, because 65% persuades nobody. Exposure is the two multiplied. Sort the log by exposure and compare the top three against the epics they threaten. Anything worth less than a couple of percent of the outcome target gets logged and then left alone." }, { "name": "Name one owner and a decide-by date", "detail": "Name one person, never a team and never a role. The owner is whoever can make the call, which is usually not the person who spotted it: if the response needs budget or a supplier conversation, the owner is the person holding the budget or the relationship. Then set the decide-by date, the last day on which a decision still changes the result. Work it back from the release window and from roadmap close, because an epic ships in one quarter and a decision taken after close has made itself. If the decide-by date passes, record which default you took and why." }, { "name": "Choose a response and one next action", "detail": "For a risk, choose one of avoid, reduce, transfer or accept, and write the choice down. For an issue, choose contain, fix or replan. Then write exactly one next action with a name and a date on it, not three. Two rules keep this honest. A mitigation that costs more than the exposure is not a mitigation, so price the response before you commit to it. And accept is a real answer: a log where nothing is ever accepted is a log where everything is being quietly mitigated by nobody, which is how twenty amber items survive a whole quarter." }, { "name": "Book the review point when you write it", "detail": "Give the item a review point at the moment you write it, or it gets reviewed when somebody happens to remember. Decide-by date inside two weeks: it joins the daily lookahead. Threatens the value of a live outcome: it goes into monthly outcome validation, where that number is being tested anyway. Everything else waits for roadmap close, where each open item is either carried into the next roadmap with a fresh decide-by date or closed. Patterns across items, such as the same supplier appearing four times, belong in the retro rather than in the log." }, { "name": "Escalate by moving a number, not a colour", "detail": "Escalation is a number moving where people outside the team can see it, not an email with a colour on it. If the response is to descope, cut the epic's planned contribution and let the hole show against the outcome's target so somebody owns it. If the response is to move the work, the epic leaves this quarter's roadmap, because an epic lives in exactly one quarter. If the response is more people, name what leaves the roadmap to pay for them. Apply the test before you send anything: which number moved, and on whose screen?" }, { "name": "Close it with a reason", "detail": "Close every item with one of four reasons: the event happened, the event can no longer happen, we accepted it, or it was never real. Then record what happened to the exposed value: landed, partly landed, or lost. That second note is why the log is worth reading next quarter, because the pattern across closed items tells you which risks your organisation habitually underprices. Record it where the answer is embarrassing, especially there. Count closures at roadmap close: a log that closed fewer items than it opened all quarter is an archive, not a management tool." } ], "example": { "heading": "Worked example: a payment SDK deprecation", "body": "Ravensworth Home is an invented online homeware retailer, and every number below is invented with it. Its outcome for the year is Reduce checkout abandonment, with a target of £1.2m in recovered revenue. One epic on the Q3 roadmap, One-page checkout, carries a planned contribution of £400k. In week two of the quarter the tech lead sees that the payment provider has announced the current SDK will stop accepting new integrations from 1 September, and the replacement sandbox has not been granted. Written up, it reads: because the payment provider closes the current SDK to new integrations on 1 September and we still have no sandbox access, the one-page checkout epic may not be releasable this quarter, which would push £400k of planned contribution into Q4. Value at risk is not £400k. A one-quarter delay costs a quarter of the annual benefit, so the team writes £100k and notes the assumption next to it. Probability, honestly, 60%. Exposure is £60k, which puts it second on a log of six items and above the three the team had been talking about most. The owner is the named engineering manager who can call the provider's account team, not the checkout squad. Decide-by is 8 August, because integrating and testing against a new sandbox takes six weeks and the last release window of the quarter is mid-September. The response is reduce, with one action: obtain sandbox credentials or written confirmation of an extension by 8 August, fallback of descoping to the existing SDK. On 8 August there is no answer, so the owner records the default taken and starts the fallback. On 12 August the provider confirms no sandbox until October. The risk becomes an issue on the same record, keeping its history, and probability stops mattering: the £100k is exposed. The team replans, cuts the epic's planned contribution from £400k to £300k, and the roadmap now shows a £100k gap against the £1.2m target in week seven of the quarter rather than in the last week. At roadmap close it is closed with reason: the event happened, value partly landed. The return on writing it properly in week two is nine weeks of warning." }, "pitfalls": [ "Tasks wearing a risk label. \"Risk: we have not hired a second tester\" is a piece of work with an owner and a date, not a risk. It gets logged because nothing gates the log, and it crowds out the two items that needed a decision from someone senior.", "Red, amber and green instead of currency. Colours cannot be summed, ranked against an epic's planned value or compared with last quarter, so every review turns into an argument about whether something is amber. An exposure figure ends that argument in one line.", "Value at risk set to the whole epic. Copying the full epic value into every item makes the log read as catastrophic and makes ranking impossible, so nothing gets prioritised. Most threats cost a share of the benefit, usually a delay's worth, and the arithmetic takes a minute.", "Ownership handed to a team. A risk owned by \"platform\" is owned by nobody. It gets discussed for six weeks without resolution, because no individual has been asked to decide and no individual is uncomfortable that it is still open.", "An issue that changed nothing. If raising it did not move a date, a scope or a planned contribution, it was a status update. Either the response was never chosen or the escalation stopped at a slide, and the quarter ends with the same number it started with." ], "definitionOfDone": [ "One sentence in cause, event, consequence form that a reader outside the team can act on without asking whether it has happened yet", "Linked to exactly one epic or outcome, with no duplicate copies of the same threat across sibling epics", "Value at risk, probability in tens and exposure all recorded, plus the assumption behind the value at risk", "One named person owns it, and the decide-by date falls before the release window and before roadmap close", "A response chosen from the four (or contain, fix, replan for an issue) and exactly one next action carrying a name and a date", "A review point set: daily lookahead, monthly outcome validation, or roadmap close" ], "related": [ { "label": "Governance, accountability and the evidence the work produces", "href": "/guides/ai-governance-and-regulatory-evidence" }, { "label": "Evaluation, ground truth and the release gate", "href": "/guides/agent-evaluation-and-assurance" } ], "faq": [ { "question": "When a risk becomes an issue, do I raise a new item?", "answer": "No. Change the type on the same record so the history stays attached: the original cause, the decide-by date, the response you chose and the date it converted. Probability stops applying, because the event has happened, so the value at risk becomes the exposure. Raising a fresh item hides how long you knew, which is the part worth learning from at roadmap close." }, { "question": "How do I set a probability when I have no data?", "answer": "Ask how often this has happened to you before in similar circumstances, and start there. Estimate in tens, get a second person to write a number down independently before either of you speaks, and take the higher one if they disagree by more than twenty points. You are not trying to be right to the percentage point. You are trying to rank this item honestly against the other things on the log." }, { "question": "Our board wants a RAG status. Do we abandon that?", "answer": "Keep the colour as a presentation layer and derive it from the exposure, for example red above a set share of the outcome's target, amber above a lower one. Nobody argues about a band that a formula produced. Reporting rules of this kind sit inside the target operating model Tenhaw's founder co-led at HSBC Global Payment Solutions, designed and piloted for 500 teams, with James’s engagement complete and the wider rollout in progress. What you should not do is store the colour as the underlying record, because then the number that lets you rank and compare no longer exists." }, { "question": "How big should a RAID log be?", "answer": "Small enough to review every open item at roadmap close in an hour, and if it is bigger than that, the entry bar is too low. Items with an exposure under a couple of percent of the outcome target are recorded and left alone rather than managed, and anything that is work with an owner and a date belongs in the backlog instead." }, { "question": "What is the difference between a risk and an issue?", "answer": "The test is whether the damaging event has happened yet. A risk is something that might reduce an outcome's value or move its date; an issue is something already costing you time, money or scope. Both are written the same way, as one sentence of cause, event and consequence, attached to one epic, priced in currency and owned by one named person. The distinction matters because probability only applies to a risk. Once the event happens, the value at risk simply becomes the exposure. Check it is neither before you log it: a defect in shipped work is a bug, a shortcut that slows you later is tech debt, and something another team owes you on a date is a dependency." }, { "question": "How do you write a good risk statement?", "answer": "Write one sentence in three parts: because of a cause someone could verify today, an event may happen, which would have a stated effect on the epic's value or its date. For an issue the tense changes, so the sentence becomes because X, Y has happened, and it is costing Z. The cause must be a checkable fact, not a feeling, and noun titles like vendor risk or resourcing risk give a reader nothing to decide on. Test the sentence by reading it to someone outside the team, and rewrite it if they cannot tell whether the event has already happened or what it costs. A well-written item lets a stranger read it and act." }, { "question": "How do you put a monetary value on a project risk?", "answer": "Multiply two numbers. Value at risk is the share of the epic's planned contribution actually threatened, which is rarely all of it. A one-quarter delay to a £400k epic exposes roughly a quarter of the annual benefit, so write £100k and record the assumption behind it so someone can argue with it. Probability is your honest estimate in tens, because 65% persuades nobody. The product of the two is the exposure, and it is what you sort the log by. Anything under a couple of percent of the outcome target gets logged and left alone, which keeps attention on the handful of items that genuinely threaten the number. Tenhaw runs this arithmetic on its own engagements and publishes the method in full." }, { "question": "Who should own a risk, and can it be the person who raised it?", "answer": "One named person, never a team and never a role, and usually not the person who spotted it. The owner is whoever can actually make the call, so if the response needs budget or a supplier conversation, it is the person holding the budget or the relationship. A risk owned by \"platform\" is owned by nobody, and it will be discussed for six weeks without resolution because no individual has been asked to decide. Give the owner a decide-by date as well, the last day on which a decision still changes the result, and if that date passes, record which default was taken and why." }, { "question": "How often should a RAID log be reviewed?", "answer": "Item by item rather than on a fixed cadence. Book each entry's review point at the moment you write it, or it gets reviewed when somebody happens to remember. Three tiers cover it: anything with a decide-by date inside two weeks joins the daily lookahead, anything threatening the value of a live outcome is tested at monthly outcome validation, where that number is being examined anyway, and everything else waits for roadmap close, where each open item is carried forward with a fresh decide-by date or closed with a reason. Count closures at roadmap close as well, since a log that closed fewer items than it opened all quarter is an archive, not a management tool." }, { "question": "Is it ever right to just accept a risk?", "answer": "Yes, and a healthy log uses it regularly. Accept is one of the four legitimate responses, alongside avoid, reduce and transfer. A mitigation that costs more than the exposure is not a mitigation, so price the response before you commit to it, and where the arithmetic does not work, accepting the risk is the honest call. Write the acceptance down and set a review point so the decision is revisited if the numbers move. A log where nothing is ever accepted is a log where everything is being quietly mitigated by nobody, which is how twenty amber items survive a whole quarter." }, { "question": "What should you do first when an issue is already costing money?", "answer": "Pick the response and write it down: contain it, fix it, or replan around it. Then one next action carrying a name and a date, not three, because a list of five actions is a list nobody owns. Price the response before you commit, since a mitigation that costs more than the exposure is not a mitigation. At Tenhaw that call sits with James Rooney, who leads every engagement personally, because it usually needs budget authority. And make the escalation move a number rather than a colour. If the answer is to descope, cut the epic's planned contribution so the hole shows against the outcome's target and somebody has to own it. If nothing moved, what you sent was a status update." }, { "question": "How do you close a risk in a RAID log?", "answer": "Give it one of four closing reasons: the event happened, the event can no longer happen, you accepted it, or it was never real. Then add the line most logs skip, what became of the money that was exposed, recorded as landed, partly landed or lost. That second note is the only reason anyone reads the log next quarter, because the pattern across closed items shows which threats your organisation habitually underprices, and it earns its keep in exactly the cases where the answer is embarrassing. If you cannot name which of the four reasons applies, the item is not closed. It has just gone quiet." }, { "question": "Do assumptions and dependencies belong in the same log as risks?", "answer": "Yes, that is what the A and the D of RAID are for, but classify each entry before you write it. Work another team owes you by a date is a dependency. A belief you are relying on and cannot prove is an assumption, and the day it turns out to be false you have a risk or an issue, whether or not it has cost you anything yet. Keep the neighbours out: a defect in shipped work is a bug and links to the quarter's Bug Budget epic, a shortcut that will slow you later links to the Tech Debt epic, and anything with an obvious owner that takes under a day is simply work." }, { "question": "Should we log the same risk against every epic it affects?", "answer": "No, log it once. Every item attaches to exactly one epic, and when a threat genuinely spans several it belongs on the outcome above them, raised once with the aggregate exposure. Cloning one risk across three epics counts the same money three times, so when you sort the log by exposure the top of the list stops meaning anything and the items that deserve attention get buried. That single link does other work too, putting the entry one step from a currency target and settling who reviews it and which quarter it sits in. With no live work to attach it to, it goes on the company-level log." }, { "question": "How do you set a deadline for a decision on a risk?", "answer": "Work backwards from the release window and from roadmap close, rather than forwards from today. The decide-by date is the last day on which a decision still changes the result, so it depends on how long your chosen response takes to execute. In this guide's worked example a replacement payment sandbox needs six weeks to integrate and test, and the quarter's final release window falls mid-September, which puts decide-by at 8 August. Roadmap close is the second boundary, because an epic lives in one quarter and a decision taken after close has made itself. If the date passes with no answer, record the default you took and start the fallback." }, { "question": "Why does our risk log get updated but nothing ever changes?", "answer": "Usually three habits have crept in together. Tasks are wearing risk labels. A line like \"we have not hired a second tester\" is a piece of work with an owner and a date, and it crowds out the two items that needed a decision from someone senior. Value at risk has been copied from the whole epic onto every item, which makes the log read as catastrophic and ranking impossible, so nothing gets prioritised. The response was never chosen, so the escalation stopped at a slide. Tenhaw runs the same test on itself, reporting a month that delivered no measurable value as a failed month. Check each open item. If raising it moved no date, scope or planned contribution, it was a status update." }, { "question": "How long should it take to write up a risk properly?", "answer": "About forty-five minutes for an entry that matters, and a real entry bar is what keeps that to a handful of items a quarter. Most of the time goes on the arithmetic behind value at risk, and on working out the one person who can genuinely make the call. Those are the two parts that pay. At Tenhaw this is work for the two or three senior people on the engagement. In this guide's worked example, a payment SDK deprecation spotted in week two is priced at £100k of value at risk at 60% probability, so £60k of exposure, which lands it second on a log of six and above the three items the team had talked about most. The return was nine weeks of warning." }, { "question": "What should a risk log cover when a model writes the code?", "answer": "The same arithmetic, plus three classes of threat you would not have logged before. First, test requirements never defined precisely enough for anyone to prove the outcome is met, so nobody can say whether it is done. Second, completion reported by a model with no demonstrated user journey behind it. Third, supply risks carrying real dates: model deprecations, context and rate limits, provider pricing changes. Tenhaw publishes on GitHub the 72 rules it holds model-written code to. Items attach to the outcome ticket at epic level, because an AI-native team has no stories to hang them on. Agents will also draft candidate risks faster than anyone can read them, so gate the log on exposure and keep the owner human." } ], "aiNote": "AI-augmented teams tend to surface risks during refinement and mid-build, when a story turns out to be bigger than it looked. AI-native teams have no stories, so items attach to the outcome ticket at epic level, and the risk classes shift. Watch for three in particular: test requirements that were never defined precisely enough for anyone to prove the outcome is met, completion reported by a model with no demonstrated user journey behind it, and supply risks with real dates on them such as model deprecations, context or rate limits, and provider pricing changes. Agents also generate candidate risks faster than a team can read them, so gate the log on exposure and keep the owner human. A model can draft the cause, event, consequence sentence and do the delay arithmetic; it cannot hold the decide-by date." } ## How to write release notes Source: https://tenhaw.com/the-tenhaw-way/how-to/write-release-notes Last updated: 2026-11-05 { "slug": "write-release-notes", "group": "Shipping and supporting", "title": "How to write release notes", "summary": "Give customers, support, executives and the on-call developer the handover each needs from one release.", "intro": "Release notes are four artefacts produced from one shipped change: a customer note, a support pack, an executive one-pager and an on-call entry, plus the rollout plan and comms that schedule them. Together they are the ship kit, and they are written before the release date is set rather than after it lands. Good ones trace back to the epic and its share of the outcome's currency target, and claim nothing the passing tests and demonstrated journeys support. Bad ones are a changelog dump forwarded to four inboxes and read carefully by none of them.", "whenToUse": "Every time a done story, or in an AI-native team a done outcome ticket, has been reviewed and product-approved and is about to be given a release date. Write the ship kit as part of scheduling the release, not as a tidy-up after it.", "totalTime": "PT3H", "timeLabel": "About three hours, for all four artefacts", "tools": [ "An evidence folder holding the commit range, the passing test run and screenshots", "The delivery tracker" ], "steps": [ { "name": "Build the evidence folder before you write", "detail": "Twenty minutes, one folder, no prose yet. Walk up the chain from the ticket: story, epic, outcome, currency target, and copy out the epic's planned share, which the executive note needs exactly. Then collect the commit range, the passing test run, the acceptance criteria or key user journeys with evidence they were demonstrated, screenshots of the new state, the flag name and its default, any migration and whether it reverses, and the rollback procedure with the date it was last run against a real environment. A missing piece is a readiness problem, not a writing problem, and cheaper to find now than at 3am." }, { "name": "Write the customer note in the customer's words", "detail": "Eighty words, three parts: one sentence on what changed, two or three lines on what to do differently, and where to find it. First read the last five support tickets touching this workflow and steal the words customers use, because they will not be the feature name your team argued over. Strip ticket IDs, service names, internal team names and anything about the pipeline that built it. If the change sits near a workflow people are protective of, name what has not changed, because that is the fear the note has to answer. If the change is invisible to users, write no note and record that decision on the ticket." }, { "name": "Write the support pack as answers, not narrative", "detail": "Build a four-column table: the symptom in the customer's words, the one-line answer, the ready-to-send macro, and when it escalates. Three rows minimum, covering the questions this change will generate, not the ones you wish it would. Add who is affected and who is not, by plan or cohort, and the limitations you shipped on purpose. Then the two lines that save the most time: what a ticket must contain before it returns to the team, and the signal that means stop widening the rollout. Support gets this 48 hours before the first cohort, not on the morning. Hearing about a release from a customer costs more goodwill than the release earned." }, { "name": "Write the executive one-pager against the number", "detail": "One page, four lines: the epic, the outcome it rolls up to, the epic's planned currency share, and what shipped in plain English. Then the honest part, in three more: the evidence that exists today, the value expected and through what mechanism, and the date of the next monthly outcome validation, when this number first gets tested. Add a line reading realised value to date, nil. Do not claim delivered value. The epic sits in value monitoring until validation confirms the number or someone closes it saying the value did not land. Reporting a shipped feature as banked value is the one sentence that breaks the loop." }, { "name": "Write the on-call entry for a tired stranger", "detail": "This is the developer changelog in the ship kit. Its reader was not involved, is half asleep, and has a pager going off. Name the commit range, the flag and its current default, any migration and whether it reverses, new configuration and secrets, changed dependencies, and the two dashboards or alerts most likely to move. Then the line that matters: the exact rollback step, how long it takes to bite, and whether it has ever been run against a real environment. If rollback means a forward fix because a migration is destructive, put that in the first two lines. Finish with blast radius: which cohorts, which regions, which downstream services." }, { "name": "Attach the rollout plan and comms schedule", "detail": "The notes are useless until someone has agreed when each lands. Write the cohorts as percentages with dates: who gets it first, how long you hold before widening, and the named metric and threshold that must hold at each step, so widening is a check rather than a mood. Then the comms: who sends the customer note and when, who briefs support, who posts internally, and whether anything goes to a public changelog. A person's name against every line, never a team name. Hold one cohort at the old behaviour until the seven-day live monitoring check closes, so you keep a clean comparison group for when the number is questioned." }, { "name": "Draft with AI, then verify claim by claim", "detail": "This is one of the places AI earns its keep. Hand a model the diff, the ticket, the test output and the demonstrated journeys, and ask for all four artefacts in one pass, each written for its own reader, marking anything it inferred rather than read. Then go line by line and strike every factual claim you cannot point at a specific file in the evidence folder. Models write confident notes about behaviour nobody implemented, and a model reporting that it has finished is not evidence that it has. Product approves the customer note and the one-pager; whoever presses release signs the on-call entry." }, { "name": "Publish, link back, hand into live monitoring", "detail": "Attach all four artefacts to the ticket so the chain holds: anyone opening it reaches the notes, the epic, the outcome and the number behind it. Publish on the schedule, in the order you wrote. Then hand the customer note and the support pack into live monitoring, because for seven days they are what you triage against: the FAQ gets extended as real questions arrive, the macro gets corrected where it was wrong, and the continue, watch or rollback call is made against thresholds you already agreed rather than invented under pressure. When live monitoring closes, the epic stays in value monitoring until outcome validation reports." } ], "pitfalls": [ "One note relabelled four times. The customer note gets a technical paragraph pasted on the end and goes to everyone. It happens because writing one thing is quicker, and it fails because a note aimed at four readers is read carefully by none of them.", "The changelog dump. Ticket titles and commit messages pasted straight into a customer note. Ticket titles are written for the team that raised the work and describe the work, not the change in the customer's day.", "Reporting shipped as delivered. An executive note that presents the epic's planned share as money earned kills outcome validation before it runs, because nobody re-examines a number already reported as banked.", "Writing the notes after the release. By then the person who knows the blast radius is on something else, and the on-call entry gets reconstructed from the diff by whoever is free, at the moment it is least useful.", "A rollback line nobody has ever run. Revert the deploy is not a procedure when the migration is destructive or the flag was deleted in the same release. If the step has never been executed against a real environment, the note says so in those words." ], "definitionOfDone": [ "All four artefacts exist, each names its audience, and all four are attached to the ticket before the release date is agreed.", "Someone outside the team read the customer note and could say what changed and what to do next, and it contains no ticket IDs, service names or internal team names.", "Support has the pack at least 48 hours before the first cohort, with one answer, one macro and one escalation rule per user-visible change.", "The on-call entry names the flag and its default, the rollback step, how long it takes to take effect, whether it has been run for real, the blast radius, and the two dashboards to watch.", "The executive one-pager states the epic's planned currency share and the date of the next outcome validation, and records realised value to date as nil.", "Every line of the rollout plan and comms schedule carries a person's name and a date, with one cohort held back until the seven-day live monitoring check closes." ], "example": { "heading": "A worked example: Ledgerly, an invented invoicing product", "body": "The outcome is cut days sales outstanding from 47 to 38, with a currency target of £1.2m in recovered working capital over two quarters. The Q3 roadmap carries an epic, automated payment reminders, with a planned share of £400k. The story shipping on Tuesday is the reminder schedule editor, behind the flag reminders_v2, default off. Customer note, 71 words: You can now set your own reminder schedule. Until today, Ledgerly chased an unpaid invoice once, seven days after the due date. Go to Settings, then Reminders, to choose up to three chase points and edit the wording of each. Your existing invoices keep the single reminder until you change it, so nothing new goes to a customer without you deciding it should. Support pack: three symptoms with an answer and a macro each (why did my customer get two reminders, can I turn this off, why can I not edit the template), a line that legacy-plan accounts do not see the setting and must not be promised it, and one escalation rule, that any report of a reminder sent against a paid invoice comes straight back to the team as a P1 and stops the rollout. Executive one-pager: epic automated payment reminders, outcome DSO 47 to 38, planned share £400k, shipped so that customers can set their own chase schedule. Evidence today, 14 acceptance tests passing and three key journeys demonstrated. Expected value comes from invoices being chased earlier and more than once, and is first tested at outcome validation on 3 October. Realised value to date, nil. On-call entry: commits a41f2c to 9be013, flag reminders_v2 default off, one additive migration creating reminder_schedule, reversible, no new secrets, new dependency on the scheduler queue. Watch reminder_send_rate and invoice_webhook_errors. Rollback, set reminders_v2 to off in the flag console, effective within 60 seconds, no deploy needed, migration can stay in place, last run for real in staging on 22 September. Rollout and comms: 5% internal accounts on Tuesday (Priya), 25% on Thursday if reminder_send_rate stays within 10% of baseline and no P1 has been raised (Priya), 60% the following Tuesday (Priya), with a 10% cohort held at the old behaviour until live monitoring closes on 6 October. Customer note in-app on Thursday (Dan), support pack to the team on Monday (Dan), public changelog at the 60% step (Dan)." }, "aiNote": "An AI-augmented team writes the ship kit per story. An AI-native team has no stories, so it comes off the outcome ticket, and the evidence is what that ticket already demanded: the test requirements passing and the key user journeys demonstrated working. That makes drafting easier and verification harder. Nobody typed the code, so no one carries the blast radius in their head, and the on-call entry has to be derived from the diff and the migration files, then read line by line by whoever will be paged. Ask the model to mark every claim it inferred rather than read, and delete those first.", "related": [ { "label": "What an AI-native SDLC changes", "href": "/guides/ai-native-sdlc-and-product-delivery" } ], "faq": [ { "question": "Who writes the release notes, product or engineering?", "answer": "Whoever shipped the change drafts all four, with a model doing the first pass. Product approves the customer note and the executive one-pager, and whoever presses release signs the on-call entry. One drafter, two approvers, no committee. Tenhaw runs the same split on client engagements, where the agentic lead owns delivery and decision rights inside the client's own management structure and the client owns all code, documentation and work product from day one. If the drafter is not on the rota, someone who is reads and countersigns the on-call entry before the release date is confirmed, because that is the artefact they will be woken up by." }, { "question": "What if the change is invisible to customers?", "answer": "Then there is no customer note, and saying so is the correct output. Write no customer note required on the ticket so the absence is a recorded decision rather than an oversight. You still owe the on-call entry, because invisible changes are the ones that page people, and support still gets two lines if anything they can see in an admin tool, an export or a log has moved." }, { "question": "Can we auto-generate the notes from commit messages?", "answer": "You can generate a draft, and you should. What you cannot do is publish it unread. Tenhaw drafts the same way on client work and then verifies every claim by hand against an engineering standard it publishes as open source, 72 rules with RFC 2119 severities on GitHub. Commit messages describe the work, not the change in the customer's day, and they carry service names and ticket IDs that have no business in a customer note. Treat generated text as a first pass to be verified against the evidence folder, and expect to rewrite the customer note almost entirely, because that reader sits furthest from the diff." }, { "question": "Is this proportionate for a hotfix at 2am?", "answer": "Write the on-call entry first and four lines is enough: what changed, the flag, the rollback, the blast radius. The rest follows within one working day. Do not skip the executive line if the fix changes what the epic is expected to be worth, and do not skip the support pack if a customer might notice, because a hotfix support has not been briefed on generates the same tickets a planned release does, at a worse moment." }, { "question": "Should release notes be written before or after the release goes out?", "answer": "Before, as part of scheduling the release, not as a tidy-up after it lands. Write the ship kit when a change has been reviewed and product-approved and is about to get a release date, because that is when the person who knows the blast radius is still on the work. Written afterwards, the on-call entry gets reconstructed from the diff by whoever is free, at exactly the moment it is least useful. Budget about three hours for all four artefacts, starting with twenty minutes building an evidence folder: the commit range, the passing test run, screenshots, the flag and the rollback procedure. A missing piece is a readiness problem, and far cheaper to find now than at 3am." }, { "question": "What should customer-facing release notes say?", "answer": "About eighty words in three parts: one sentence on what changed, two or three lines on what to do differently, and where to find it. Write it in the customer's words, and the quickest way to find those is to read the last five support tickets touching this workflow and borrow the language customers actually use, which is rarely the feature name your team argued over. Strip ticket IDs, service names, internal team names and anything about the pipeline that built it. And if the change sits near a workflow people are protective of, name what has not changed, because that is the fear the note has to answer." }, { "question": "What does the support team need before a new feature is released?", "answer": "A support pack, at least 48 hours before the first cohort, never on the morning of the release. The useful shape is a table rather than a narrative: the symptom in the customer's words, the one-line answer, a ready-to-send macro, and when it escalates, with three rows minimum covering the questions the change will actually generate. Add who is affected and who is not, by plan or cohort, the limitations you shipped on purpose, what a ticket must contain before it comes back to the team, and the signal that means stop widening the rollout. Hearing about a release from a customer costs support more goodwill than the release earned." }, { "question": "What goes in the developer changelog for a release?", "answer": "The commit range, the feature flag and its current default, any migration and whether it reverses, new configuration and secrets, changed dependencies, and the two dashboards or alerts most likely to move. Write it for a tired stranger. The reader is a developer who was not involved in the change, is half asleep, and has a pager going off. Then the line that matters most: the exact rollback step, how long it takes to bite, and whether it has ever been run against a real environment. If rollback really means a forward fix because a migration is destructive, that fact belongs in the first two lines, not the last. Finish with the blast radius: which cohorts, regions and downstream services." }, { "question": "Should we report a shipped feature as delivered value?", "answer": "No. A shipped feature is a claim about work, not about money, and reporting the epic's planned share as banked value kills outcome validation before it runs, because nobody re-examines a number already reported as earned. Tenhaw reports value monthly on its own engagements against the numbers agreed with the client, and a month that moves none of them is reported as a failed month. So the executive one-pager states the epic, the outcome, the planned currency share and what shipped, then the evidence that exists today, the mechanism the value should arrive through, the date of the next monthly outcome validation, and a line reading realised value to date, nil. The epic stays in value monitoring until validation settles it." }, { "question": "What is a ship kit in The Tenhaw Way?", "answer": "The ship kit is everything The Tenhaw Way requires before a release date is agreed: a customer note, a support pack, an executive one-pager and an on-call entry, plus the rollout plan and comms schedule naming who sends each and when, a person's name against every line. Tenhaw publishes The Tenhaw Way in full, seventeen how-to guides included, free to adopt without hiring anyone. The kit exists because one note relabelled four times gets read carefully by nobody, so each artefact is written for its own reader. All four attach to the ticket so the chain holds back to the epic, the outcome and its currency target, and the customer note and support pack then hand into the seven-day live monitoring window." }, { "question": "What do release notes say when only some customers get the feature?", "answer": "Say who has it and who does not, in the customer note and again in the support pack. The pack carries a line naming who is affected by plan or cohort, and support needs the negative case in writing, which in the worked example is that legacy-plan accounts cannot see the new setting and must not be promised it. Timing follows the rollout plan rather than the deploy, so the in-app note lands with the cohort that has the change and the public changelog waits until the rollout is wide enough for it to be true. The on-call entry then names the blast radius: which cohorts, which regions, which downstream services." }, { "question": "Our release notes are a changelog dump. What do we fix first?", "answer": "Split them by reader, then run the test that settles it. Give your last note to someone outside the team and see whether they can say what changed and what to do next. If they cannot, the customer note is the first rewrite, in the words customers use rather than the ones the ticket used. The on-call entry comes next, because written after the event it gets reconstructed from the diff by whoever is free, at the moment it is least useful. Then the support pack, 48 hours before the first cohort. The executive one-pager can wait until the first three are routine. Tenhaw does this repair inside client teams, where a client engineer finished a two-week build 70% confident of running it unaided." }, { "question": "What should you gather before writing release notes?", "answer": "Twenty minutes of collecting, before any prose. Walk up the chain from the ticket to the story, the epic, the outcome and its currency target, and copy out the epic's planned share, because the executive note needs that figure exactly. Then the commit range, the passing test run, the acceptance criteria or key user journeys with evidence they were demonstrated, screenshots of the new state, the flag name and its default, any migration and whether it reverses, and the rollback procedure with the date it was last run against a real environment. The folder is also what you verify the draft against, so strike any claim you cannot point at a file inside it." }, { "question": "How do you decide when to widen a release to more customers?", "answer": "Decide before the release. The rollout plan carries the cohorts as percentages with dates, and the named metric and threshold that must hold at each step, so widening is a check rather than a mood. Tenhaw runs client rollouts this way on its engagements, under partner oversight from James Rooney. A plan might start at 5% internal accounts on Tuesday, widen to 25% on Thursday if the send rate the change moves stays within 10% of baseline and no P1 has been raised, then 60% the following week. Every line carries one person's name and a date, never a team name. Support also gets a written signal that means stop widening." }, { "question": "Why keep some customers on the old version during a rollout?", "answer": "To keep a clean comparison group for when the number is questioned. Hold one cohort, say 10%, at the old behaviour until the seven-day live monitoring check closes. Without it, the only comparison you have is the same population before and after, which moves for a dozen reasons that have nothing to do with your change, and that is exactly what gets picked apart at outcome validation. The held cohort is also a cheap sanity check while the window is open. If a metric looks alarming in both groups, the cause is the week rather than the release." }, { "question": "What if our rollback procedure has never been run for real?", "answer": "Then the on-call entry says so, in those words, and the conversation happens before the release date is agreed rather than at 3am. Revert the deploy is not a procedure when the migration is destructive or the flag was deleted in the same release. The entry names the exact rollback step, how long it takes to bite, and the date it was last run against a real environment. If rollback really means a forward fix, that belongs in the first two lines, not the last. In practice, having to write that date is what sends someone to run the step in staging first, which is the point of asking." }, { "question": "How do release notes change when a model wrote the code?", "answer": "Drafting gets easier and verification gets harder. On Tenhaw's own AI-native builds the notes come off the outcome ticket rather than off stories, and the team reading them is two or three senior people, founder-led. An AI-native team has no stories, and the evidence is whatever that ticket already demanded, the test requirements passing and the key user journeys demonstrated working. The catch is that nobody typed the code, so no one carries the blast radius in their head. The on-call entry has to be derived from the diff and the migration files, then read line by line by whoever will be paged. Ask the model to mark every claim it inferred rather than read, and delete those first." }, { "question": "Are release notes finished once they have been published?", "answer": "No. For seven days the customer note and the support pack are what you triage against. The FAQ gets extended as real questions arrive, the macro gets corrected where it was wrong, and the continue, watch or rollback call is made against thresholds agreed in advance rather than invented under pressure. All four artefacts stay attached to the ticket, so anyone opening it reaches the notes, the epic, the outcome and the number behind it. Tenhaw runs the same window on client engagements and deliberately leaves the FAQ, the macros and the thresholds with the client's own team, because the exit date is agreed at kickoff. The epic then sits in value monitoring until outcome validation tests the planned share." } ] } ## How to monitor the first week after a release Source: https://tenhaw.com/the-tenhaw-way/how-to/live-monitoring-after-release Last updated: 2026-11-05 { "slug": "live-monitoring-after-release", "group": "Shipping and supporting", "title": "How to monitor the first week after a release", "summary": "Watch the first seven days against agreed thresholds and record when to continue, investigate or roll back.", "intro": "Live monitoring is the structured watch you keep on a shipped change for its first days in production, and it ends in a recorded decision rather than a feeling. It opens when the release goes out, runs against the journeys the ticket named, and produces four things: a customer impact assessment, a published FAQ entry, a support macro, and a continue, watch or rollback call. Done well it catches the ways production contradicts staging while a rollback is still cheap, and it hands a clean epic to value monitoring.", "metaDescription": "Live monitoring runs seven days by default: checks at one, four and 24 hours, key journeys walked on production, and a continue, watch or rollback call.", "whenToUse": "Every time a change reaches real users, including the small ones nobody is worried about. It opens at the moment of release and runs until the window closes, and the parent epic cannot enter value monitoring until it does.", "totalTime": "P7D", "timeLabel": "Seven days, the default window", "tools": [ "The dashboards and saved queries named in the ship kit", "A real production account for walking the journeys", "The devices your users actually have" ], "steps": [ { "name": "Name the owner and window before you ship", "detail": "Plan the watch in the ship kit, not after the deploy. When you schedule the release, write into the ticket who owns it, who deputises on that person's days off, when the window opens and when it closes. Seven days is the default: long enough to cover a full weekly cycle including the weekend, short enough that nobody forgets it is open. Extend it only for a stated reason, such as a change touching monthly billing that needs the window to reach the next run. If the release lands on a Friday, move it or accept that you have chosen to staff the weekend." }, { "name": "Write the baseline and the query before release", "detail": "Before the deploy, record the numbers you expect afterwards with today's baseline beside each: error rate on the affected endpoints, p95 latency, completion rate for every journey the ticket names, support contacts on the tag, and any business counter the epic is meant to move. Take the baseline from the same weekday and hour you will compare against, so Tuesday lunchtime is judged against Tuesday lunchtime. Paste the saved query or dashboard link beside each number, so the person on the four-hour check is not inventing one. Without a written prediction you will talk yourself into calling a regression seasonal variation." }, { "name": "Agree the thresholds while nothing is on fire", "detail": "Write the number that triggers each of the three calls into the ticket before release. For example: roll back if the checkout error rate stays above 1% for two hours, or if any journey named in the ticket fails outright for more than 10% of sessions. Watch if a metric moves outside the normal range but impact is bounded and a fix is in reach the same day. Continue otherwise. Name the one person who makes the call, name who can wake them, and state plainly that they can roll back without convening anyone. A threshold agreed at 11am on a calm Tuesday beats a debate at 2am." }, { "name": "Walk the journeys yourself, on production", "detail": "Dashboards tell you what broke, not that the thing works. Within the first hour, walk every key user journey the ticket names, on production, on the devices your users have, with a real account rather than a seeded one. Walk the failure paths too: declined card, expired session, empty state. Use a fresh account and a cold cache, because your browser is holding the flags and cookies that hide the fault. Screenshot each pass onto the ticket. This catches the misconfigured flag, the missing environment variable and the CDN still serving last week. A green deploy and a model reporting success are evidence about the build, not the live system." }, { "name": "Check on a fixed rhythm, not on anxiety", "detail": "Set the cadence and hold it: one hour, four hours, 24 hours, then once a day until the window closes. Put the checklist in the ticket so the deputy runs the same one: the numbers against your written baseline, new error signatures rather than error volume alone, support contacts on the tag, and anything monitoring surfaced automatically. One new signature at low volume matters more than a familiar one at high volume. Post the result in the same channel every time, including when everything is flat, because a flat check posted publicly is what tells everyone else the change is holding. Each check should take minutes." }, { "name": "Publish the FAQ entry and macro early", "detail": "The first time a customer hits the problem, someone in support writes an answer under time pressure. Write that answer once, correctly, and early. For each issue found, record who is affected and how many in numbers not adjectives: roughly 3% of sessions, about 40 orders a day. Publish the FAQ entry in the words a customer would search for, not the words the ticket uses. Give support a macro that names the workaround and says whether a fix is coming, then have them send it live once and confirm it closed the contact without escalating. The cost of an unsupported release is paid by second line." }, { "name": "Make the call and record it", "detail": "At the 24-hour check, make the call explicitly, even when the answer looks obvious, and record the decision, the time, the person and the numbers it rested on. Continue means the window keeps running with no action. Watch means a named fix, an owner, a date and a re-check written into the ticket. Rollback means the change comes out now and the reasoning is written down before anyone starts on the fix. Weigh it in currency: a day of bounded impact costed against the planned value of the epic you would pull. Never let a release drift through the window undecided. An undecided release is one nobody owns." }, { "name": "Bucket what you found, then close the window", "detail": "Nothing found in live monitoring is allowed to float. A defect in shipped behaviour is a bug: raise it and link it to the quarter's Bug Budget epic, including the one you already hot-fixed. A shortcut taken to stabilise the release is tech debt, linked to the quarter's Tech Debt epic. Something the change was meant to do and never did is a missed requirement, so it is a story or an outcome ticket, not a bug. Close the window with a short note covering what happened and what you changed, then move the parent epic into value monitoring, where the currency gets tracked month by month." } ], "pitfalls": [ "Treating silence as success. No alerts often means no instrumentation on the new path, and a journey that fails before it emits anything looks identical to a journey nobody used. Confirm the new code is producing signal at all before concluding it is producing good signal.", "Watching the aggregate and missing the segment. A fault confined to one browser, one region or one enterprise tenant disappears into a 99.6% overall success rate. Split every headline number by platform and by customer segment at least once in the first four hours.", "Leaving the rollback threshold to be agreed during the incident. Under pressure the person with the most to lose from a rollback is usually the loudest voice in the room, and the number quietly moves to fit the argument. Write it down while nothing is on fire.", "Fixing forward off the ticket. A hot fix pushed without raising the bug keeps the board tidy and destroys the Bug Budget burn rate, which is the only signal you have about whether quality is improving or rotting.", "Running live monitoring and value monitoring as one activity. Live monitoring asks whether the change is safe and does what it said, over days, at the level of what shipped. Value monitoring asks whether the epic produced its currency share, over months. Merging them gets an epic called delivered on the strength of a quiet week." ], "definitionOfDone": [ "Every key user journey named in the shipped ticket has been walked on production by a person, with the screenshots stored against the ticket.", "A continue, watch or rollback decision is recorded with its timestamp, its owner and the numbers it rested on.", "Every defect found in the window is raised and linked to the quarter's Bug Budget epic, and every stabilising shortcut is linked to the Tech Debt epic.", "The customer FAQ entry and the support macro are published, and support has answered a real contact with them without escalating.", "The window is closed with a written note, and the parent epic has either entered value monitoring or been held with a stated reason and a date." ], "related": [ { "label": "Evaluation, ground truth and the release gate", "href": "/guides/agent-evaluation-and-assurance" }, { "label": "Bounding what a wrong answer costs", "href": "/guides/guardrails-hallucination-and-accuracy-control" } ], "faq": [ { "question": "How long should the window be?", "answer": "Seven days by default, because it covers a full weekly cycle including the weekend and is short enough that people remember it is open. Shorten it only for changes with no customer-visible surface. A billing change needs the window to reach the next run, and a seasonal feature needs it to reach the first real peak, so extend it only with a reason and a new closing date written in the ticket. A window with no end date is not monitoring, it is a browser tab left open." }, { "question": "Is this the same as being on call?", "answer": "No, and running them as one job is why post-release problems get missed. On call reacts to things that alert. Live monitoring goes looking for the things that do not: a journey that quietly completes at 54% instead of 61%, a support tag creeping up, a new error signature at ten a day. On call can hold the pager for the release, but somebody has to own the fixed-rhythm checks, the FAQ entry and the recorded decision, and that is a different piece of work. Tenhaw keeps the two apart, with the watch named on the ticket and one senior lead holding it for the window rather than a rota, under partner oversight from James Rooney." }, { "question": "What if the change is behind a flag or a percentage rollout?", "answer": "The window opens at first real user exposure, not at deploy, and your thresholds apply to the exposed cohort rather than to total traffic. A 2% error rate inside a 5% rollout is invisible in the overall number and is still the reason to stop. Write the ramp steps into the ticket with the check you run before each one, and treat turning the flag off as the cheap rollback it is, rather than waiting to pull the whole release." }, { "question": "Who owns it, product or engineering?", "answer": "One named person on the ticket, whichever function they sit in, with a named deputy. Where Tenhaw carries that name it is on a Programme and Delivery Management engagement, and one person holds it rather than a rota. In practice the product manager tends to own the customer impact numbers, the FAQ entry and the macro, and the engineer who shipped the change tends to own the error and latency checks. Split the checklist between them if you like, but only one name carries the decision, and that person needs standing authority to roll back without convening a meeting." }, { "question": "Why does a release work in staging but break in production?", "answer": "Because production carries the things staging cannot rehearse: real accounts, real devices, a misconfigured flag, a missing environment variable, a CDN still serving last week's assets. That is why the first check is a person walking every key journey on production within the first hour, on the devices your users actually have, with a fresh account and a cold cache, because your own browser holds the flags and cookies that hide the fault. Watch the segments too. A fault confined to one browser, one region or one enterprise tenant disappears into a 99.6% overall success rate, so split every headline number by platform and by customer segment in the first four hours." }, { "question": "How often should you check on a release after it goes live?", "answer": "On a fixed rhythm, not on anxiety: one hour, four hours, 24 hours, then once a day until the monitoring window closes. Put the checklist in the ticket so a deputy runs exactly the same one: the numbers against the written baseline, new error signatures rather than raw volume, and support contacts on the tag. One new signature at low volume matters more than a familiar one at high volume. Post every result in the same channel, including when everything is flat, because a flat check posted publicly is what tells everyone the change is holding. Each check takes minutes, and if the release lands on a Friday, move it or accept that you have chosen to staff the weekend." }, { "question": "What should you baseline before a release goes out?", "answer": "Record, before the deploy, the numbers you expect afterwards with today's value beside each: error rate on the affected endpoints, p95 latency, completion rate for every journey the ticket names, support contacts on the tag, and any business counter the epic is meant to move. Take the baseline from the same weekday and hour you will compare against, so Tuesday lunchtime is judged against Tuesday lunchtime. Paste the saved query or dashboard link beside each number, so the person running the four-hour check is not inventing one under pressure. Without a written prediction you will talk yourself into calling a regression seasonal variation." }, { "question": "When should you roll back a release instead of fixing forward?", "answer": "When it crosses a threshold you wrote down before the release, not one negotiated during the incident. Roll back, for example, if the checkout error rate stays above 1% for two hours, or if any named journey fails outright for more than 10% of sessions. Watch, with a named fix, an owner and a date, when the movement is bounded and a fix is in reach the same day. One named person makes the call and can roll back without convening anyone. Weigh a day of bounded impact, in currency, against the planned value of the epic you would pull. If you fix forward, raise the bug anyway, because a hot fix off the ticket hides your quality signal." }, { "question": "How do you prepare customer support for a release?", "answer": "Write the answer once, correctly, and early, before the first customer hits the problem and someone in support has to improvise under time pressure. For each issue found in the first week, record who is affected and how many in numbers not adjectives, such as roughly 3% of sessions or about 40 orders a day. Publish the FAQ entry in the words a customer would search for, not the words the ticket uses. Give support a macro that names the workaround and says whether a fix is coming, then have them send it live once and confirm it closed the contact without escalating. The cost of an unsupported release is paid by second line." }, { "question": "Can an AI agent run post-release monitoring?", "answer": "It can run most of it, and it should. Let an agent run the fixed-rhythm checks, diff the numbers against the written baseline, cluster new error signatures and draft the FAQ entry and the support macro. Two things stay with a person, the journey walk on production with a real account and the continue, watch or rollback decision, which needs a named owner with standing authority to act. Tenhaw builds these agents for clients against an open-source standard of 72 rules. A model reporting that the release is healthy is evidence about the build, not the live system, and a journey that fails before it emits anything looks identical to a journey nobody used. Automate the checking, keep the judgement." }, { "question": "We had no alerts after the release. Is that a good sign?", "answer": "Not on its own. Silence often means the new path was never instrumented, and a journey that breaks before it emits anything looks exactly like a journey nobody used. Confirm the new code is producing signal before you decide it is producing good signal, by finding the first real event from the new path and comparing its volume with the completion rate you recorded before the deploy. Dashboards tell you what broke, not that the thing works, so the check that settles it is a person walking the journeys the ticket named on production, with a real account and the devices your users actually have. Treating silence as success is how a bad release survives its first day." }, { "question": "What do you do with the problems you find in the first week?", "answer": "Bucket every one of them before the window closes, because nothing found in live monitoring is allowed to float. A defect in shipped behaviour is a bug. Raise it and link it to the quarter's Bug Budget epic, including the one you already hot-fixed, since a fix pushed without a ticket flatters the board and wrecks the burn rate that tells you whether quality is improving or rotting. A shortcut taken to stabilise the release is tech debt, linked to the quarter's Tech Debt epic. Something the change was meant to do and never did is a missed requirement, so it becomes a story or an outcome ticket rather than a bug." }, { "question": "How much extra work does watching a release actually add?", "answer": "Less than teams expect, and most of it lands before the deploy. Naming the owner and the deputy, writing the baseline numbers with a saved query beside each, and agreeing the rollback thresholds all happen while nothing is on fire, and that preparation is what makes the rest cheap. Tenhaw runs the watch with the same two or three senior people who shipped the change. The checks then run on the fixed one, four and 24-hour rhythm and should take minutes each: the numbers against the written baseline, new error signatures, support contacts on the tag. The real effort sits in the journey walk on production in the first hour, and in writing the customer FAQ entry and the support macro properly." }, { "question": "What does a release watch look like when something goes wrong?", "answer": "It looks like a rule firing rather than an argument. The worked example here, an invented retailer with invented numbers, ships a guest checkout epic on a Tuesday at 13:00 against a written baseline of 0.4% checkout error rate, 910ms p95 latency and 61.5% completion. The one-hour check finds all four named journeys passing on real devices, but a 1.9% error rate concentrated in one browser's address-lookup fallback. The agreed threshold needs two hours above 1%, so 14:20 is recorded as watch with a re-check booked for 16:20. Still 1.7%, so at 16:35 the flag goes back to the previous provider, a partial rollback rather than pulling the epic, and it is 0.6% by 17:00." }, { "question": "We ship without any of this today. What is the first thing to add?", "answer": "Three lines in the ticket of your next release, all written before the deploy. Tenhaw publishes all seventeen guides free to adopt. One: a named owner, a named deputy and the date the window closes, seven days out by default. Two: the numbers you expect afterwards with today's baseline and the saved query beside each, taken from the same weekday and hour you will compare against. Three: the threshold that would trigger a rollback, and who can call it without convening anyone. That alone catches most of what teams miss. Add the fixed-rhythm checks on the release after that, then the customer FAQ entry and the support macro. One release watched properly teaches a team more than a process document ever will." }, { "question": "Do we need new monitoring tools to watch a release?", "answer": "Almost certainly not. Tenhaw runs this watch on a client's existing dashboards. The kit is the dashboards and saved queries you already have, named and linked in the ship kit so the person running the four-hour check is not inventing a query under pressure. What teams actually lack is simpler, a real production account rather than a seeded one for walking the journeys, and the devices your users have rather than the laptop the change was built on. Log in clean, with nothing cached, because the browser you have been developing in will hide the fault. If a number genuinely is not visible, the gap is usually instrumentation on the new code path, not your tooling budget." }, { "question": "What has to be true before you can call a release done?", "answer": "In The Tenhaw Way, five things, all evidenced on the ticket. Every key journey the shipped ticket named has been walked on production by a person, with the screenshots stored against the ticket. A continue, watch or rollback decision is recorded with its timestamp, its owner and the numbers it rested on. Every defect found is linked to the quarter's Bug Budget epic and every stabilising shortcut to the Tech Debt epic. The customer FAQ entry and the support macro are published and have answered a real contact without escalating. Then the window closes with a written note, and the parent epic moves into value monitoring or is held with a stated reason and a date." }, { "question": "Who covers the release watch at the weekend or on holiday?", "answer": "A deputy named in the ticket when the release is scheduled, not found on the day. Tenhaw names the deputy alongside the owner. Seven days is the default window precisely because it covers a full weekly cycle including the weekend, so cover is a planning question rather than an emergency. The deputy works from the same written baseline, the same thresholds and the same channel as the owner, which is why all of that goes into the ticket before the deploy rather than into someone's head. Hand over the open items too: what sits on watch, when the next re-check falls, and who can be woken. If nobody can cover the window, move the release rather than hoping the days off stay quiet." } ], "example": { "heading": "Seven days on a guest checkout release", "body": "Northbank Home is an invented retailer and every number below is invented with it. The outcome: lift checkout completion by 1.2%, worth £1.4m a year. Under it sits a guest checkout epic carrying a planned £420k share. Baseline written before release: checkout error rate 0.4%, p95 checkout latency 910ms, completion 61.5%, twelve support contacts a day on the checkout tag. Threshold agreed the previous week: roll back if the error rate stays above 1% for two hours, or if any named journey fails for more than 10% of sessions. The epic ships Tuesday at 13:00. One-hour check: all four named journeys walked on production on an iPhone and an Android handset with a real account, and both guest and saved-card checkout complete. Error rate is 1.9%, concentrated in one browser on the address-lookup fallback, about 3% of sessions. The rule needs two hours, so the 14:20 entry records watch with a re-check booked for 16:20. At 16:20 it is still 1.7%, so the rule has fired. Rather than pull the epic, they flip the address-lookup flag back to the previous provider at 16:35 and record it as a partial rollback with the numbers behind it. Error rate is 0.6% by 17:00. The impact was costed at roughly 40 orders a day at £62 average order value, about £2,500 a day, which framed how fast to fix it rather than whether the rule applied. FAQ entry and support macro published Tuesday afternoon; support sent the macro nine times that week with no escalations. Fix ships Wednesday, the flag goes back on Thursday morning, error rate settles at 0.5% and completion reaches 62.4% by Friday. Two bugs raised against the quarter's Bug Budget epic, one tech debt item raised for the fallback. The following Tuesday the window closes with a four-line note, and the epic moves into value monitoring, where its £420k share is tracked month by month against the outcome." }, "aiNote": "In an AI-native team the outcome ticket already names the key user journeys and the test requirements, so live monitoring is a re-run of the same done bar on production: tests pass and journeys demonstrated working. Let an agent run the fixed-rhythm checks, diff the numbers against the written baseline, cluster new error signatures and draft the FAQ entry and macro. Keep two things with a person: the journey walk on production with a real account, and the continue, watch or rollback call. A model reporting that the release is healthy is evidence about the build, not about the live system." } ## How to run a root cause analysis Source: https://tenhaw.com/the-tenhaw-way/how-to/root-cause-analysis Last updated: 2026-11-05 { "slug": "root-cause-analysis", "group": "Shipping and supporting", "title": "How to run a root cause analysis", "summary": "Build a shared incident timeline and turn the contributing control gaps into owned, testable actions.", "intro": "A root cause analysis is the structured investigation you run after a failure that reached users, or came close enough that luck was the control. It asks what in the system allowed the failure, not who typed the wrong thing. A good one produces a timeline nobody disputes, two to four contributing causes that each name a control you own, a measurable cut in how fast you would detect it next time, and actions that exist as tickets on the quarter's Bug Budget or Tech Debt epic. If it ends as a document, it did not happen.", "whenToUse": "Run one when a defect reached production and affected users, when live monitoring produced a rollback call, or when a near miss was caught by a person noticing rather than by a control firing. A failure shape that has now happened three times earns one even when each occurrence looked small.", "totalTime": "P5D", "timeLabel": "Five working days, evidence frozen inside the first 24 hours", "tools": [ "A frozen evidence record: logs, alerts, deploy records and the incident chat", "The RAID log", "The delivery tracker" ], "steps": [ { "name": "Set the triggers before you need them", "detail": "Agree the trigger list in advance and apply it without debate: anything that reached users, any rollback called during live monitoring, any severity-one defect, any near miss caught by a person rather than by a control, and the third occurrence of the same failure shape. The delivery lead for that team makes the call within one working day and records which trigger fired. Everything else gets a bug ticket on the Bug Budget epic and no ceremony. Deciding case by case, in the mood immediately after an outage, is how the embarrassing incidents get investigated and the boring recurring ones never do." }, { "name": "Freeze the evidence within 24 hours", "detail": "Name one owner for the incident record and give them a day. Copy logs, dashboard screenshots, alert payloads, deploy records, feature-flag changes, support tickets and the incident chat into the record itself, rather than linking out to systems that roll at seven or thirty days. Then ask everyone involved to write down separately, before any group meeting, what they saw and when they saw it, in their own words. Separate accounts captured first are your only defence against the room converging on the loudest person's version inside the first ten minutes, and once that has happened you cannot recover the original recollections." }, { "name": "Build one agreed timeline in UTC", "detail": "Book ninety minutes within five working days. Invite the people who were there, plus a facilitator who was not. Open with the timeline and nothing else. Mark four moments: when the condition entered the system, when the failure began, when a human or a control first noticed, and when customers stopped being affected. Timestamp to the minute in UTC and name the source of every entry. Failure to notice is your time to detect. Failure to recovery is your time to restore. Circulate the timeline for correction before anyone discusses causes, so the argument about causes is not a disguised argument about the times." }, { "name": "Separate the trigger from the conditions", "detail": "Draw two columns. The trigger is the change that lit the fuse: a deploy, a traffic spike, an expiring certificate, a third party altering a response shape. The conditions are what made the system flammable: no timeout, no fallback, an alert routed to a channel with no rota, a test suite that never covered that journey. For each condition, ask what should have contained this and write the answer next to it. A rollback removes the trigger and leaves every condition standing, which is why the same incident returns wearing different clothes. Nearly every action you raise should come out of the conditions column, not the trigger." }, { "name": "Ask why until you reach a control", "detail": "Take each condition and ask why it was there, stopping when you reach something the team can change: a standard, a test, an alert, a gate, a default, a runbook. If an answer names a person, you are not finished, ask why the system let one person's attention be the control. If an answer names something outside your control, a supplier's API or a cloud region, then the control you own is whatever should have contained it: a timeout, a circuit breaker, a cached fallback, a contract test. Three or four whys usually gets there. Chains of seven tend to be creative writing." }, { "name": "Test every cause against a counterfactual", "detail": "For each candidate cause ask one question: if this alone had been different and nothing else had changed, would the incident still have happened? If yes, it is context and belongs in a background paragraph rather than in the causes list. If no, it is a contributing cause and it earns exactly one action. Run the test out loud so the room hears which items fail it. This is the step that stops an RCA becoming a list of everything anyone dislikes about the codebase. Expect two to four contributing causes. If you have eleven, you have written a wish list, and nobody funds a wish list." }, { "name": "Produce at least one detection action", "detail": "Every RCA raises a detection action, no exceptions. Write three things down: the current time to detect taken from the timeline, the target you are committing to, and the specific signal that would have fired. Name it precisely, an error-rate threshold on a stated endpoint, a synthetic journey run every five minutes, a support-contact spike, a queue depth. Then say who is rostered to answer it, because an alert routed to a channel nobody owns is not detection. If a customer told you before your systems did, state that plainly. Fixing this defect without shortening detection leaves you equally blind to the next one." }, { "name": "Convert findings into linked, budgeted tickets", "detail": "An action that is not a ticket is a sentence. Raise every one during the session, on screen, before people leave. Defects go to this quarter's Bug Budget epic. Missing tests, alerting and hardening go to the Tech Debt epic. Anything large enough to need product prioritisation becomes an epic under an outcome, with a quarter and a planned currency share, competing for capacity like any other epic. Give each ticket a named owner, not a team. If an action cannot get an owner and a quarter in the room, record it as deliberately declined, with the reason and who declined it, instead of listing it and quietly dropping it." }, { "name": "Publish it, then close actions in the open", "detail": "Publish within five working days, in the same place every time, readable by anyone in the organisation rather than only the people on the call. Put the actions on the agendas of the next two retrospectives and close them out loud there. Where the failure exposed a standing risk you are choosing to live with, add it to the RAID log as an accepted risk with a named owner and a review date, so it stays visible rather than being resolved by silence. Before you open the next quarter's roadmap, reread the quarter's RCAs together: a failure shape that appears twice is the tech debt you should be funding." } ], "pitfalls": [ "Stopping at human error. \"The engineer skipped the check\" describes the last thing that happened, not why the system depended on someone remembering. It feels conclusive, which is exactly why teams stop there, and it produces the one action nobody can implement: be more careful.", "Running it two weeks late. Logs have rotated, dashboard retention has closed, and everyone has rehearsed a version of events that makes sense to them. What you get is a coherent document that is partly fiction, and it reads well enough that nobody challenges it.", "Actions with no ticket, no owner and no quarter. They read like commitments inside the document and are invisible by week three, because they never entered the roadmap and so never competed for capacity against anything else.", "Treating the rollback as the fix. Reverting restores service and removes the trigger while every condition that let the trigger matter survives. The incident returns with a different trigger, and the team concludes that RCAs do not work.", "Only investigating the incidents that were embarrassing. Small recurring failures cost more in aggregate and never clear the bar for attention, which is why a written trigger threshold beats judgement in the moment." ], "definitionOfDone": [ "The timeline is agreed by everyone who was in the incident, every entry names its source, and it states both time to detect and time to restore.", "Every contributing cause survived the counterfactual test and names a control the team can change, not a person.", "At least one action shortens time to detect, with the current number, the target and the specific signal all written down.", "Every action is a ticket with a named owner, linked to the quarter's Bug Budget epic, the Tech Debt epic or a new epic carrying a quarter and a currency share, or is recorded as deliberately declined with a reason.", "The document is published within five working days, where anyone in the organisation can read it without asking.", "The actions are on the agenda of the next two retrospectives and closed there in front of the team, rather than in private." ], "example": { "heading": "Worked example: forty-one minutes blind on checkout", "body": "The incident below is invented and every figure in it with it. An online retailer ships a payment-provider config change at 09:12 UTC. Card declines rise from 2% to 31%. The first signal is a customer email at 09:53, so time to detect is 41 minutes. Rollback is called at 10:04 and customers are clear at 10:11, so time to restore is 59 minutes. Trigger: the config change. Conditions: no alert on decline rate, and no contract test covering the provider's new error code. Counterfactual on \"the change went out on a Friday\": the incident happens on any day of the week, so that is context, not a cause. Two actions leave the session as tickets. First, a decline-rate alert at 5% sustained over five minutes, target time to detect under five minutes, owned by a named engineer, on the quarter's Tech Debt epic. Second, a contract test on the provider's error codes, same epic, same quarter. The Friday deploy question goes in the background paragraph and gets no ticket." }, "aiNote": "For AI-augmented teams the failing control is usually a story's acceptance criteria, or the human review that waved the change through, so the actions land in the definition of done and the test suite. For AI-native teams the unit of work is the outcome ticket, and two conditions recur: a user journey that was never listed in the ticket, and verification that rested on the model reporting it had finished. Fix both in the ticket itself, by adding the journey to the standing list and by requiring a demonstrated run rather than a claimed one. Ask one extra question in every AI-native RCA: what did the ticket fail to say?", "related": [ { "label": "Evaluation, ground truth and the release gate", "href": "/guides/agent-evaluation-and-assurance" }, { "label": "Bounding what a wrong answer costs", "href": "/guides/guardrails-hallucination-and-accuracy-control" } ], "faq": [ { "question": "How is an RCA different from live monitoring?", "answer": "Live monitoring is the scheduled watch over a change after it ships: customer impact, support volume, the FAQ and the support macro, and the continue, watch or rollback call. It runs on every release, whether or not anything is wrong. An RCA is triggered by what live monitoring finds, or by an incident that arrives with no warning at all. Live monitoring asks whether this release is behaving. An RCA asks why the system allowed it not to, and it ends in funded tickets rather than a call." }, { "question": "Who should run the session?", "answer": "Someone who did not build the thing that failed and does not manage the people who did. Their job is the method rather than the investigation: holding the timeline until it is agreed, applying the counterfactual test to every candidate cause, and stopping the room whenever an answer names a person instead of a control. Tenhaw does this as an outside facilitator, brought in because it did not build the system under discussion. Keep the room to the people who were there plus that facilitator. Once it becomes a stakeholder audience, people start performing rather than remembering, and you lose the detail you came for." }, { "question": "What if the cause sits with a supplier we do not control?", "answer": "Then you have found the trigger and you still owe the conditions. You cannot action a third party's deploy schedule, but you can action the timeout you did not set, the fallback you did not build, the contract test you did not write, and the alert that would have told you their response had changed shape. Raise the commercial conversation separately, and log the dependency in the RAID log as an accepted risk with a review date, but do not let a supplier's name become the reason no ticket was raised." }, { "question": "Is this supposed to be blameless?", "answer": "Blameless means the cause is never a person, not that names vanish from the timeline. Write plainly that an engineer ran the deploy at 09:12, because the timeline is worthless without it. What you never write is that the engineer was the cause. If the honest finding is that one person's memory was the only thing standing between a change and production, then the missing control is a gate, and the action is to build it." }, { "question": "Which incidents actually need a root cause analysis?", "answer": "Agree the trigger list in advance and apply it without debate: anything that reached users, any rollback called during live monitoring, any severity-one defect, any near miss caught by a person rather than by a control firing, and the third occurrence of the same failure shape. The delivery lead makes the call within one working day and records which trigger fired; everything else becomes a bug ticket with no ceremony. Deciding case by case, in the mood straight after an outage, means the embarrassing incidents get investigated and the boring recurring ones, which cost more in aggregate, never do." }, { "question": "How soon after an incident should you run a root cause analysis?", "answer": "Freeze the evidence within 24 hours and hold the session within five working days. One named owner copies logs, alert payloads, deploy records and the incident chat into a record that will not rotate away, and everyone involved writes down what they saw before any group meeting. The ninety-minute session then builds one agreed timeline before causes are discussed, and the write-up is published within five working days. Wait two weeks and logs have rotated and everyone has rehearsed a version of events, so what you get is a coherent document that is partly fiction." }, { "question": "What is the difference between a trigger and a root cause?", "answer": "The trigger is the change that lit the fuse: a deploy, a traffic spike, an expiring certificate, a third party altering a response shape. The root causes almost always sit in the conditions that made the system flammable: no timeout, no fallback, an alert routed to a channel with no rota, a test suite that never covered that journey. A rollback removes the trigger and leaves every condition standing, which is why the same incident returns wearing different clothes. Nearly every action you raise should come out of the conditions column, not the trigger." }, { "question": "How do you decide if something is a real cause or just context?", "answer": "Run a counterfactual test on every candidate. If this alone had been different and nothing else had changed, would the incident still have happened? If yes, it is context and belongs in a background paragraph. If no, it is a contributing cause and it earns exactly one action. Run the test out loud so the room hears which items fail it. Expect two to four contributing causes from a typical incident; a list of eleven is a wish list of everything anyone dislikes about the codebase, and nobody funds a wish list." }, { "question": "How many whys should a five whys analysis actually take?", "answer": "Usually three or four. Stop when you reach a control the team can change: a standard, a test, an alert, a gate, a default, a runbook. If an answer names a person, you are not finished; ask why the system let one person's attention be the control. If it names something outside your control, the control you own is whatever should have contained it, such as a timeout, a circuit breaker or a contract test. Chains of seven whys tend to be creative writing rather than analysis." }, { "question": "How do you stop RCA actions being forgotten?", "answer": "Raise every action as a ticket during the session, on screen, before anyone leaves; an action that is not a ticket is a sentence. Each one gets a named owner, not a team, and a home where it competes for capacity, which in The Tenhaw Way means the quarter's Bug Budget epic for defects, the Tech Debt epic for missing tests and alerting, or a new epic under an outcome for anything larger. If an action cannot get an owner and a quarter in the room, record it as deliberately declined, with the reason. Then put the actions on the agendas of the next two retrospectives and close them out loud." }, { "question": "Is a root cause analysis the same as a post-mortem?", "answer": "The names get used interchangeably, and the label matters far less than the bar you hold the session to. Four things decide whether it was worth running, whatever you call it: evidence frozen inside the first 24 hours, one timeline in UTC that names the source of every entry and states both time to detect and time to restore, every candidate cause put through a counterfactual so you finish with two to four rather than eleven, and at least one action that shortens detection. Every action leaves the room as a ticket with a named owner. If it ends as a document, it did not happen." }, { "question": "How do we introduce root cause analysis to a team that has never done one?", "answer": "Set the mechanics up before the next incident rather than during it. Tenhaw publishes this method in full, and a team can adopt every step of it without hiring anyone. Write the trigger list down and get it agreed, and name the delivery lead for that team as the person who applies it within one working day. Decide where the incident record lives and who owns freezing it inside 24 hours, because logs and dashboards roll at seven or thirty days. Agree who facilitates, which has to be someone who did not build the thing that failed. Then book ninety minutes within five working days, open with the timeline and nothing else, and raise every action as a ticket on screen before anyone leaves." }, { "question": "How do you build an incident timeline everyone agrees on?", "answer": "Collect separate accounts before the room ever meets. Ask everyone involved to write down what they saw and when they saw it, in their own words, ahead of any group discussion, because ten minutes into a meeting the room has converged on the loudest person's version and the original recollections cannot be recovered. Then open the session with the timeline and nothing else. Timestamp to the minute in UTC, name the source of every entry, and mark four moments: when the condition entered the system, when the failure began, when a person or a control first noticed, and when customers stopped being affected. Circulate it for correction before anyone discusses causes." }, { "question": "Do we still need an alert once the bug itself is fixed?", "answer": "Yes. Every RCA raises at least one detection action, because fixing the defect closes one hole and leaves you exactly as blind to the next one. Write three things down: the current time to detect taken from the timeline, the target you are committing to, and the specific signal that would have fired. Then name who is rostered to answer it, because an alert routed to a channel nobody owns is not detection. In the invented worked example on this page, card declines jump from 2% to 31% and a customer email is the first signal 41 minutes later, so the action is a decline-rate alert at 5% sustained over five minutes." }, { "question": "What should a root cause analysis report include?", "answer": "A timeline agreed by everyone who was there, with a named source on every entry and both time to detect and time to restore stated. Then two to four contributing causes, each having survived the counterfactual test and each naming a control the team can change rather than a person. Then the detection action, written as the current number, the target and the signal that would have fired. Then every other action as a linked ticket with a named owner and a quarter. Anything that failed the counterfactual sits in a background paragraph and gets no ticket. Publish within five working days, where anyone in the organisation can read it without asking." }, { "question": "What changes in a root cause analysis when AI wrote the code?", "answer": "On AI-native teams two conditions recur, and both are fixed in the outcome ticket rather than in the codebase. The first is a user journey the ticket never listed, so nothing tested it, and the fix is to add it to the standing list. The second is verification that rested on the model reporting it had finished, and the fix is to require a demonstrated run instead of a claimed one. On AI-augmented teams the failing control is usually a story's acceptance criteria or the human review that waved the change through, so the actions land in the definition of done and the test suite. Tenhaw's engineers build to a 72-rule engineering handbook, open source and enforced by an agent rather than by memory." }, { "question": "How should past incidents shape the next quarter's plan?", "answer": "Reread the quarter's RCAs together before you open the next quarter's roadmap. A failure shape that appears twice is not bad luck, it is the tech debt you should be funding, and it will keep recurring until it is carried by something that competes for capacity. Missing tests and alerting go to the Tech Debt epic; anything large enough to need product prioritisation becomes an epic under an outcome, with a quarter and a planned currency share. Those epic types come from The Tenhaw Way, the operating model behind every Tenhaw engagement. Where an incident exposed a risk you are choosing to live with, put it in the RAID log as an accepted risk with a named owner and a review date." }, { "question": "Is a formal root cause analysis overkill for a small team?", "answer": "Not at the size it actually runs, one person spending a day freezing evidence and then a ninety-minute session. The written trigger list is what keeps it proportionate, because most incidents never clear the bar and get a bug ticket with no ceremony at all. In small teams the real failure mode is the opposite of overkill. Only the embarrassing outages get investigated, while the small recurring failure that costs more in aggregate never earns anyone's attention, because the call is made in the mood straight after an outage rather than against a threshold agreed in advance. Ninety minutes beats paying for the same incident a third time." } ] } ## How to measure the value a project delivered Source: https://tenhaw.com/the-tenhaw-way/how-to/measure-value Last updated: 2026-11-05 { "slug": "measure-value", "group": "Measuring and managing", "title": "How to measure the value a project delivered", "summary": "Compare the live result with its baseline and forecast, then record what value has been realised.", "intro": "Outcome validation is the monthly check on whether the money you planned has arrived. Every live outcome, every month. Value is measured at the epic, because the epic is what carries a priced share of the outcome's target. Good looks like this: the metric and the attribution method were fixed in the epic before release, the baseline was exported before the change shipped, and every epic in value monitoring has a realised-to-date number and a dated decision from the last calendar month. A shipped epic is not a delivered epic.", "whenToUse": "Run it on a fixed date every month, on every outcome with at least one epic in value monitoring, from the first release under that outcome until the last epic closes. Set the measurement up far earlier, when the epic is written, because a baseline cannot be captured retrospectively.", "totalTime": "PT3H", "timeLabel": "About three hours a month, of which the session is forty-five minutes", "tools": [ "The reporting system the measure is pulled from", "The outcome's recorded baseline", "The delivery tracker" ], "steps": [ { "name": "Define the measure before you build", "detail": "Measurement belongs in the epic, not a follow-up ticket. Before it leaves Ready for Dev, write five things in: the metric, the exact source of the number (system, table or saved report, plus the query), the arithmetic that converts that metric into currency, the window the value needs to accumulate over, and one person who can pull it. Written out it reads like this: weekly completed checkouts from the orders table, saved query val_checkout_v1, times £62 average order value at 41% gross margin, read monthly for six months, pulled by the team's analyst. If nobody can name the query, the epic is not ready for dev." }, { "name": "Capture the baseline before you release", "detail": "Pull the metric for the four quarters before the change ships and paste the raw weekly numbers into the epic, not a dashboard link whose definition will drift. You want three things from it: the level, the week-to-week spread, and the seasonal shape, because a 2% lift in November proves nothing if November is always up 2%. Note anything else landing in the same window: a price change, a campaign, another epic touching the same journey. If the source system only retains ninety days, start the export today and set a weekly snapshot. An epic without a baseline produces an argument, not a number." }, { "name": "Choose an attribution method and write it down", "detail": "Record the strongest method you can afford in the epic before release. A holdout or A/B split gives you a counterfactual and is the default where traffic allows, but check the volume can detect the effect you priced: a 1% conversion lift needs tens of thousands of sessions per arm. Next best is a staged rollout by region or cohort, released against not-yet-released. Below that, interrupted time series against the baseline trend, credible only when you can name what else moved in the window. Last is a declared assumption, signed off by someone commercial and labelled an assumption for good. Never change the method after seeing the result." }, { "name": "Move the epic into value monitoring", "detail": "Live monitoring and value monitoring are different jobs. For seven days after release you are triaging stability: customer impact, FAQ, support macro, and the continue, watch or rollback call. That tells you nothing about value. On day eight, do four things. Move the epic, not the story, into value monitoring, because value sits on the thing that was priced. Name an owner, usually the product manager who priced it. Record the first read date. Move the outcome into value monitoring too, so it stops being reported as delivered. An epic in value monitoring is not finished work, and a roadmap review that counts it as delivered is reporting fiction." }, { "name": "Run the validation monthly, every live outcome", "detail": "Same date each month, forty-five minutes, one session across every live outcome. In the room: the product manager for each outcome, an engineering lead, and someone commercial who can challenge the conversion. Numbers are pulled and circulated the day before, never queried live: the session that waits for a dashboard is the one that gets cancelled. Epic by epic, state the measured number, realised value to date and forecast at close, then take one of four decisions: on track, needs longer with a next read date, partially realised so re-forecast, or not realised so close it. Record the number, the decision and the date, even in a month where nothing moved." }, { "name": "Convert to currency the same way every time", "detail": "Keep one conversion sheet per outcome and show the arithmetic on it. Write down the unit economics in use, average order value, gross margin, cost per support contact, fully loaded hourly rate, with the source and date of each figure. Use margin, not revenue, whenever the outcome is stated in profit. For cost savings, count only money that leaves the profit and loss or hours redeployed to something named; hours saved in the abstract are not money. Refresh the figures once a quarter and re-run last month's numbers when one changes. Check across epics that no pound is claimed twice, which happens whenever two epics touch the same journey." }, { "name": "Roll epic actuals up and re-forecast", "detail": "After the epic pass, sum realised value to date and forecast at close across every epic under the outcome, and set both against the outcome's target. Three numbers, and they are not meant to agree. Apply your calibration factor to anything still unmeasured: if the organisation has realised 70% of planned value across the last four quarters, forecast unmeasured epics at 70% of plan. Measured epics use their measurement, never the factor. When the roll-up falls short, the decision is explicit and taken in the room: add an epic to the next quarter's roadmap, or restate the target with a written reason and the name of whoever agreed it." }, { "name": "Close honestly, including the epics that missed", "detail": "An epic leaves value monitoring one of two ways: value confirmed against the method recorded before release, or deliberately closed with a note that the expected value did not land. Name the assumption that broke: demand, adoption, unit economics or attribution. Closing at 40% of plan with the reason written beats an epic left open for a year. When every epic under an outcome is closed, close the outcome with its reason: all linked epics done, value realised, or accepted as not realised. Then do the arithmetic that pays for all of this, realised divided by planned across the last four quarters, and plan next quarter with that factor." } ], "pitfalls": [ "Reporting activity instead of value. \"Fourteen thousand people used the new flow\" is a usage number, and it gets offered up because usage data exists on day one while margin data lags by weeks. Write the currency conversion into the epic before release and the usage number has somewhere to go.", "Changing the measure after seeing the result. The primary metric is flat, so a friendlier secondary metric appears in the pack. It happens whenever the measure was chosen after release rather than fixed in the epic before it.", "No baseline, so validation becomes anecdote. Nobody exported the pre-change numbers, the source system retains ninety days, and the meeting turns into who remembers what conversion used to be. It is the least recoverable failure on this list.", "Counting the same pound twice. Two epics touching the same checkout claim the same uplift, the outcome reports 140% of target, and finance can see revenue is flat. It comes from two people pricing epics against one metric without a shared conversion sheet.", "Cancelling the month because it is too early to tell. Skip validation twice and it stops existing, and six months later nobody can say what the quarter produced. Run it anyway and record \"no movement, next read 3 September\"." ], "definitionOfDone": [ "Every live outcome has a validation record dated within the last calendar month, containing a number rather than a comment.", "Every epic in value monitoring carries a named measure, a source query someone can run, an exported baseline, an attribution method recorded before release, and a realised-to-date figure.", "Each outcome shows three numbers side by side: planned value, realised to date, and forecast at close, with unmeasured epics forecast at the calibration factor.", "No pound of value appears under two epics, checked against the single conversion sheet for that outcome.", "Every epic that has left value monitoring left with either confirmed value or a written reason the value did not land, naming the assumption that broke.", "The organisation has a calibration factor from the last four quarters of realised against planned, and next quarter's plan is built with it." ], "example": { "heading": "A worked example: a £200k checkout epic", "body": "An invented online retailer prices an epic at £200k: remove the forced account-creation step in checkout. Every number below is invented with it. The measure is weekly completed checkouts from the orders table, saved query val_checkout_v1, converted at £62 average order value and 41% gross margin. The baseline for the four quarters before release is 41,000 sessions a week at 2.9% completion, with November running two points above the rest of the year. The method is a 50/50 holdout, chosen because 41,000 sessions a week can detect the priced effect inside three weeks. Month one: holdout 2.9%, treatment 3.3%, which is 164 incremental orders a week across full traffic and £4,170 of margin a week, so £18k realised. Forecast at close over twelve months, £216k. Decision: on track, next read 14 March. Month five: the lift settles at 0.3 points once the launch novelty fades. Realised to date £71k, forecast at close £165k. Decision: partially realised, re-forecast. The outcome target is £500k across three epics, this one at £200k and two others at £150k each. The roll-up now reads £500k planned, £71k realised, £375k forecast, because the two unmeasured epics are forecast at the organisation's calibration factor of 70%. The £125k gap goes on next quarter's roadmap as a named epic rather than into the following quarter's optimism." }, "aiNote": "Nothing about the measurement changes; where it is written does. An AI-native team has no stories, so the measure, the source query, the attribution method and the baseline go into the outcome ticket alongside the key user journeys and the test requirements, and the ticket is not approved without them. That is an advantage, because the unit of work and the unit of value are the same object and nobody has to reconstruct which epic a release belonged to. The risk runs the other way. AI-native teams ship more per quarter, so the number of epics in value monitoring grows faster than the ritual scales and forty-five minutes stops being enough. Split the session by outcome before you start skipping epics. Agents can pull and format the monthly numbers, and should, but the four decisions stay with the people in the room.", "related": [ { "label": "What a defensible AI business case holds", "href": "/guides/business-case-for-an-agentic-programme" }, { "label": "Designing roles, decisions and workflows for an AI-native organisation", "href": "/guides/target-operating-model-for-an-ai-native-organisation" } ], "faq": [ { "question": "How long should an epic sit in value monitoring?", "answer": "As long as the money takes, and you decide that when you write the epic rather than when someone asks. Tenhaw agrees an exit date at kickoff, so epics routinely sit in value monitoring long after the team that built them has left, read by the client's own named owner. Divide planned value by a realistic monthly run rate. A £200k epic earning £40k a month needs five months at minimum, plus the adoption ramp. If an epic needs more than about two quarters to prove, schedule interim reads at thirty, sixty and ninety days and record a forecast at each rather than going quiet until the end. Epics belong to one roadmap, but the money does not stop at the quarter boundary." }, { "question": "What if we cannot attribute the value cleanly?", "answer": "Say so in the epic, take the strongest method you can afford, and label the number with the method that produced it. Holdout first, then staged rollout by cohort or region, then interrupted time series against the baseline trend, then a declared assumption signed off by someone commercial. A weak method that is disclosed is workable, because everyone reading it knows what it is worth. An unlabelled claim built on an assumption is worse than no number, because it gets planned against. Tenhaw reports the two-week proof of concept it built inside Placing Platform Limited (PPL) as exactly that, never as production. If attribution is impossible in principle, say so before the epic is approved." }, { "question": "Does every epic need this, including tech debt and bugs?", "answer": "No. The tech debt epic and the bug budget epic that open every roadmap are fixtures rather than priced contributions to an outcome, so validating them in currency invents numbers nobody believes. Track those two on burn rate instead: how much debt was linked and cleared, how many bugs were raised and closed, and whether the trend across quarters is improving or rotting. Everything else in the roadmap carries a currency share of an outcome and goes through the full validation, including epics that are enablers for later work. Price those against the value they unlock, not at zero." }, { "question": "How do we stop this becoming a blame exercise?", "answer": "The number, not the person, and two habits keep it that way. The decision options include not realised so close it, which makes closing an epic short a normal result of the ritual rather than an escalation. And the calibration factor is an organisational figure rather than a scorecard. Realising 70% of plan is common, and knowing it lets you plan headroom instead of pretending. The failure mode to watch for is the product manager who quietly stops bringing epics to the session. If attendance starts slipping, the honesty has already gone, and no amount of template will bring it back." }, { "question": "What is the difference between monitoring a release and measuring its value?", "answer": "They are different jobs on different clocks. For seven days after release you are triaging stability: customer impact, support noise, and the continue, watch or rollback call. None of it says whether the money arrived. On day eight the epic moves into value monitoring with a named owner, usually the product manager who priced it, and a first read date. The question changes from stable to earning, read monthly against the baseline until the planned value is confirmed or closed short. Tenhaw reports on that second clock, and what it carries out of Globelynx is a 60% lead-time reduction inside six months, not a quiet launch week. A team that stops at the stability window has checked the plumbing and never read the meter." }, { "question": "How big does a holdout group need to be to prove a conversion lift?", "answer": "Big enough to detect the effect you priced, and you check that before committing to the method. A 1% conversion lift needs tens of thousands of sessions per arm; a checkout seeing 41,000 sessions a week can detect a priced effect inside three weeks, while a journey with a few hundred sessions a month may never separate signal from noise. If the volume is not there, do not run an underpowered test and hope. Step down to the next strongest method, a staged rollout by region or cohort, or an interrupted time series against the baseline trend, and record the choice in the epic before release, because the method never changes after seeing the result." }, { "question": "Our reporting system only keeps ninety days of history. Can we still baseline?", "answer": "Yes, if you act the day the epic is written rather than the day it ships. Start the export today and set a weekly snapshot, so history accumulates while the epic moves through design and build. You want four quarters of raw weekly numbers pasted into the epic itself, not a dashboard link whose definition will drift. That gives you the level, the week-to-week spread and the seasonal shape, because a 2% lift in November proves nothing if November is always up 2%. A baseline cannot be captured retrospectively, and an epic without one produces an argument rather than a number, so treat the missing export as a blocker." }, { "question": "How do you stop two projects claiming the same benefit?", "answer": "Keep one conversion sheet per outcome and check across epics that no pound is claimed twice. Double counting happens whenever two pieces of work touch the same journey and two people price them against the same metric without a shared sheet; the symptom is an outcome reporting 140% of target while finance can see revenue is flat. The sheet holds the unit economics in use, average order value, gross margin, cost per support contact, with the source and date of each figure, so every epic converts its metric to currency the same way. Run the cross-check as part of the monthly validation, before the roll-up, so an inflated claim never reaches the outcome's forecast." }, { "question": "Are usage numbers proof that a feature delivered value?", "answer": "No. Tenhaw contracts against that distinction, and on its monthly engagements a month that delivers no measurable value is reported as a failed month. \"Fourteen thousand people used the new flow\" is an activity number, offered up because usage data exists on day one while margin data lags by weeks. Usage says the change was found and adopted, which matters, but value is the metric you priced converted into currency: orders completed times margin, contacts avoided times cost per contact, hours redeployed to something named. Write the currency conversion into the epic before release, with the source query and the arithmetic, and the usage number has somewhere to go. Report adoption alongside value if it helps the story, never instead of it." }, { "question": "Should delivered benefits be measured in revenue or margin?", "answer": "Margin, whenever the outcome is stated in profit. A checkout change that adds £1m of revenue at 41% gross margin has delivered £410k, and quoting the revenue figure overstates the result by more than double. Keep the unit economics on the outcome's single conversion sheet, each figure with its source and date, and refresh them quarterly, re-running last month's numbers when a figure changes. For savings, count only money that leaves the profit and loss, or hours redeployed to something named. Hours saved in the abstract are not money." }, { "question": "What if it is too early to tell whether the value has landed?", "answer": "Run the session anyway and record a number rather than a comment. The month where nothing has moved yet is exactly the one teams cancel, and skipping it twice is how the ritual quietly stops existing, after which nobody can say six months later what the quarter produced. So take the read, write down what the metric did, and take the decision that fits. Needs longer, with the next read date named, is a perfectly good result, recorded as no movement, next read 3 September. Every live outcome should carry a validation record dated inside the last calendar month, and an early read at least proves the measurement plumbing works." }, { "question": "Do we need someone from finance in the monthly value review?", "answer": "You need someone commercial who can challenge the conversion, and in most organisations that is finance. On programmes Tenhaw governs, its own workstreams appear in the same pack, to the same standard, as every other supplier's. The session is forty-five minutes on a fixed date each month, one pass across every live outcome, with each outcome's product manager, an engineering lead and that commercial voice. Their job is the arithmetic rather than the delivery: whether the unit economics on the conversion sheet, average order value, gross margin, cost per support contact, are still current, whether a claimed saving is money that genuinely leaves the profit and loss, and whether two epics touching the same journey are counting the same pound twice." }, { "question": "Is a monthly value review worth the overhead on a small team?", "answer": "Yes. The arithmetic is about three hours a month, of which forty-five minutes is the session itself, one pass covering every live outcome rather than one per team. The rest is pulling and circulating the numbers the day before, a saved query someone runs rather than an analysis project. Without it nobody can say what the quarter actually produced, and the calibration factor that makes next quarter's plan realistic never comes into existence. Tenhaw publishes the method in full and it is free to adopt, so nothing here requires hiring anyone. The overhead scales with what you have shipped, not with headcount, which leaves a small team fewer epics and a shorter session rather than a heavier one." }, { "question": "Why not just do a post-implementation review at the end?", "answer": "Because by the end, the two things that make the number defensible are already gone. Tenhaw's monthly engagements are cancellable on 30 days' notice either way, which only means something if the value read arrives monthly too. A post-implementation review hunts for a baseline nobody exported and picks its attribution method after the result is visible, which is how a friendlier secondary metric reaches the pack. Monthly validation fixes the metric, the arithmetic and the method before release, then produces a dated decision every month: on track, needs longer with a read date, partially realised so re-forecast, or not realised so close it. You also get to act while the plan can still respond, putting a shortfall on the next quarter's roadmap." }, { "question": "What do you do when a launch lift fades after a few months?", "answer": "Re-forecast it, record the new number, and keep the epic in value monitoring instead of quietly banking the month one figure. That shape is normal. In this guide's worked example, an invented retailer's checkout epic reads a 0.4 point conversion lift in month one, worth £18k realised and a £216k forecast at close, then settles at 0.3 points by month five as the launch novelty fades, giving £71k realised and a £165k forecast. The decision is partially realised, re-forecast, and the resulting gap against the outcome's target goes on the next quarter's roadmap as a named epic. What you never do is reopen the method because the number softened." }, { "question": "How should unmeasured epics be counted in an outcome's forecast?", "answer": "At the organisation's calibration factor, never at full plan. Sum realised value to date and forecast at close across every epic under the outcome, and set both against the target, so the roll-up carries three numbers that are not meant to agree. Epics with a measurement use their measurement and never the factor. Epics still unmeasured are forecast at the share of planned value the organisation has actually realised over the last four quarters, so at a factor of 70% a £150k epic carries £105k. That is how an outcome with £500k planned and one epic read can honestly report £71k realised and £375k forecast." }, { "question": "Can we change the metric mid-build if we learn something?", "answer": "Before release, yes, provided you rewrite the epic properly: the new metric, its source and query, the currency arithmetic, the window, the named puller, and a baseline that fits the new metric rather than the old one. After release, once a result is visible, no. Changing the measure after seeing the result is the pitfall everybody recognises, where the primary metric is flat and a friendlier secondary metric appears in the pack. The same holds for the attribution method, recorded before release and never changed after. If the measure genuinely turns out to be the wrong one, close the epic with the reason written and name attribution as the assumption that broke." }, { "question": "Which parts of the monthly value review can agents take over?", "answer": "The preparation, and they should. Running the saved queries, pulling the numbers and formatting the pack the day before is exactly the work an agent handles well, and it removes the usual reason a session gets cancelled, which is a room waiting for a dashboard. The four decisions stay with the people in the room. Tenhaw draws the same line in its own work, where the 72 rules in its open-source engineering handbook are enforced by an agent rather than remembered by a human. AI-native teams also ship more per quarter, so epics accumulate in value monitoring faster than forty-five minutes can absorb. Split the session by outcome before you start skipping epics." } ] } ## How to run product delivery day to day Source: https://tenhaw.com/the-tenhaw-way/how-to/manage-product-delivery Last updated: 2026-11-05 { "slug": "manage-product-delivery", "group": "Measuring and managing", "title": "How to run product delivery day to day", "summary": "Use a short daily pass to spot blocked decisions, changing scope and the next useful sequencing choice.", "intro": "Day-to-day product delivery is the work of keeping a quarter's committed epics moving through their gates in a sensible order, and making the small decisions that unblock people within hours rather than days. It is not status collection. Done well, your day is a short pass over the board, a pass over the approval queue, and then the two or three sequencing calls nobody else can make. If you are in chat all day, you are being used as a lookup table for information the board should already carry.", "whenToUse": "Every working day of a live quarter, from the day the roadmap opens to the day it closes. The trigger is epics being in flight, not a problem appearing: if you only run this when something is wrong, you are running recovery rather than delivery.", "totalTime": "PT2H", "timeLabel": "About two hours a day, spread across the working day", "tools": [ "The delivery board", "The quarter's p50 and p85 forecast range" ], "steps": [ { "name": "Open the board before you open chat", "detail": "Five to ten minutes, first thing, in this order. Work in progress per team against the limit you set: over it, you stop starting before you start chasing. Anything with no state change and no comment for two working days: name it and act on it today. The Ready for Dev queue: is anything sitting one approval away from starting. Bug budget burn against the pace you planned for this week of the quarter. Write down the two or three items you will act on. If the pass takes forty minutes because you are reconstructing the truth from memory, the board is stale, and fixing it is item one." }, { "name": "Clear the approval queue before anything else", "detail": "You are the constraint on this queue, so work it daily, in sequence order rather than arrival order. In AI-augmented teams an epic cannot leave Ready for Dev without product approval, engineering approval and at least one product-approved story attached, and a story cannot leave the backlog until its parent epic is product-approved. In AI-native teams the outcome ticket needs its currency share, its key user journeys and its test requirements before either approval lands. Approve or reject inside one working day, and make a rejection name the missing artefact. Never approve an epic whose planned value is blank: that is the number the quarter is measured on." }, { "name": "Set one sequence per team and hold it", "detail": "Each team gets exactly one prioritised order, visible on the board, with a single item labelled next. Set it or confirm it once a day, in the same pass. When two items look equally urgent, take the one attached to the outcome furthest from its currency target, because that is where the missing value sits. Three things earn a resequence: a validation result, a slip that changes what can land, a dependency arriving early or late. A message from a loud stakeholder is not one of them. Resequencing on volume costs the switch, costs the re-plan, and costs you a team that has stopped believing the order means anything." }, { "name": "Route every new item to a parent immediately", "detail": "Nothing lives unparented, and you route in the morning pass, not at the end of the week. A bug found this quarter links to the Bug Budget epic. Debt raised during the build links to the Tech Debt epic. A missed requirement is not a bug: it is a story, or in AI-native mode an amendment to the outcome ticket, so relabel it. Risks and issues go to the RAID log with a named owner. Anything that cannot name a parent epic, and through it an outcome, does not start today. Ten minutes of routing is what keeps the burn rates meaningful for the rest of the quarter." }, { "name": "Answer blocking questions inside four hours", "detail": "Hold one fixed window in the middle of the day, forty-five minutes, and publish it. The rule: any question blocking a build gets an answer, or a written assumption on the ticket, within four working hours. Anything not blocking waits for refinement. Recording the assumption is the important half. Write what you assumed, who owns it, and what you would change if it turns out wrong. A guess is safe once it is written down and owned, and dangerous only while it lives in someone's head. Watch the pattern too: repeated questions on one epic mean the ticket was approved too thin, so fix the ticket." }, { "name": "Move the value number the day scope moves", "detail": "When scope changes, the epic's planned currency share changes the same day, by you, not by the monthly outcome validation. If a £240k epic loses the half of its scope that carried most of the value, it is a £150k epic now and its outcome is £90k short. Write the new number on the epic, let the gap show on the outcome, and add one line naming the option you would take: a new epic, a scope trade from elsewhere, or a lower target agreed openly. With eight weeks of quarter left, that gap is something you can still act on. In week thirteen it is a postmortem." }, { "name": "Hand shipped work into monitoring, not done", "detail": "A done story is reviewed, product-approved, scheduled, and released with its ship kit: user notes, developer changelog, executive one-pager, rollout plan and comms. It then enters live monitoring for the first seven days, where a named person owns the continue, watch or rollback call. Only epics go on to value monitoring, and an epic stays there until validation confirms the value or you close it with an honest note that the value did not land. In AI-native teams nothing is shippable until the ticket's test requirements pass and its named journeys are demonstrated working in front of a person." }, { "name": "Close the day by naming what slipped", "detail": "Five minutes. Compare where the work sits against your p50 and p85 forecast range rather than a single invented date. Anything that has fallen outside p85 for landing inside the quarter gets said today, to the person who has to make the choice, with the three options attached: cut scope, move the epic to next quarter, accept the risk. Do not soften it into an amber status with no ask. A slip raised in week four costs a conversation. The same slip raised in week twelve costs a commitment somebody else has already made to the business." } ], "pitfalls": [ "Deciding in chat and leaving the ticket unchanged. The decision was real, the trail is not, and three weeks later nobody can say why the scope moved or who owned the assumption. Every decision from your midday window lands on the ticket the same hour, or it did not happen.", "Answering status questions the board should answer. If people ask you rather than reading it, you have accepted a job that scales to about six people and then collapses. Answer once with a link, fix whatever made the link useless, then stop answering.", "Waving an epic through the gate because a developer is idle. That is the exact pressure the gate exists for. An epic approved with no product-approved story, or with no journeys and test requirements in AI-native mode, buys two days of activity and pays for it with a rebuild.", "Re-prioritising every morning. A sequence that changes daily is not a sequence, it is a mood. Change it on evidence: a validation result, a slip, a dependency landing early. Not because a message arrived before your morning pass.", "Accepting a model's own report of completion in an AI-native team. Done means both things: the test requirements in the ticket pass, and the key user journeys are demonstrated. A model saying it has finished is a claim, not evidence." ], "definitionOfDone": [ "Every epic in flight carries product approval, engineering approval, and either at least one product-approved story or, in AI-native mode, written key user journeys and test requirements.", "Nothing raised in the last working day is unparented: every bug, debt item and story sits under an epic, and every epic under an outcome.", "Every question that blocked a build in the last working day has an answer on the ticket, or a written assumption with a named owner.", "Each team has one visible prioritised order with exactly one item marked next, and you can say what changed it since yesterday.", "For each live outcome, the sum of its epics' planned values still covers the target, or the gap is written on the outcome with an owner, a date and a proposed option.", "Anything now outside p85 for landing this quarter has been raised on the day it became visible, with cut, defer or accept attached." ], "example": { "heading": "A Tuesday in week four of a thirteen-week quarter", "body": "Northbank Retail, an invented mid-market retailer, has one live outcome: lift checkout conversion by 1.4 points, worth £1.2m a year. The quarter's roadmap carries four value epics plus the standing Tech Debt and Bug Budget epics. Guest checkout is planned at £480k, saved payment methods at £300k, address autocomplete at £180k, error-state rework at £140k. That is £1.1m against a £1.2m target, a £100k gap that was visible before the quarter started. The morning pass takes eight minutes and surfaces three things. Guest checkout has sat in Ready for Dev for three days with engineering approval and no product-approved story, so it is blocked by the product manager rather than by engineering: two stories are written and approved by ten o'clock. The bug budget is at 14 bugs in four weeks against a planned 30 across thirteen, so burn is running hot, and the pattern behind it goes on Thursday's refinement agenda rather than into a new meeting. Address autocomplete has lost its international-address scope, so its planned value drops to £110k and the outcome gap moves from £100k to £170k. That number goes on the outcome the same day, with a line proposing a fifth epic, rather than being discovered quietly in week twelve. Everything above took under an hour, spread across three passes." }, "related": [ { "label": "What an AI-native SDLC changes", "href": "/guides/ai-native-sdlc-and-product-delivery" }, { "label": "Designing roles, decisions and workflows for an AI-native organisation", "href": "/guides/target-operating-model-for-an-ai-native-organisation" } ], "faq": [ { "question": "How long should this take each day?", "answer": "Twenty to thirty minutes of passes, plus one forty-five minute decision window. Roughly ten minutes on the board and routing, ten on the approval queue, five on the close. If it reliably takes more, diagnose which part is swelling. A long morning pass means a stale board. A long approval queue means you are batching approvals that should clear daily, or tickets are arriving too thin to approve at all." }, { "question": "What if I cover three teams?", "answer": "One sequence per team, one pass covering all three, one shared decision window. The part that does not scale is the approval queue, because you are the constraint on it. If clearing it takes more than about forty-five minutes a day, delegate product approval to a named person per team with the gate rules unchanged, rather than approving faster and looking less closely. Tenhaw's founder James Rooney worked that delegation problem at scale in a role inside HSBC Global Payment Solutions, co-leading a target operating model for 500 teams and a $450M portfolio, piloted, with James’s engagement complete and the wider rollout in progress." }, { "question": "An urgent customer request just came in. Does it beat the sequence?", "answer": "A live production incident does, and it goes into the RAID log as an issue rather than being quietly slotted into the roadmap. Everything else waits for your next sequencing pass. If it does beat the current next item, say out loud what it displaces and where that work now lands, because unnamed displacement is how a quarter goes missing." }, { "question": "How is this different from stand-up?", "answer": "Stand-up belongs to the team and covers what they are doing. This is your own loop, and most of it happens before stand-up so you arrive with decisions rather than questions. If you need stand-up to find out the state of the board, the board is the problem, and fixing that will save you more time than any change to the meeting." }, { "question": "How do I stop spending my whole day answering status questions?", "answer": "Make the board answer them. Tenhaw ran that move at Globelynx, where operational data was collated continuously into trends leadership could act on rather than assembled into a report whenever somebody asked, and delivery lead times fell 60% within six months. If people ask you instead of reading it, you have accepted a job that scales to about six people and then collapses. Answer once with a link to the item, then fix whatever made the link useless: a stale state, a missing comment, a decision that happened in chat and never landed on the ticket. Your morning pass exists to keep the board carrying the truth, so anyone can self-serve the state of any epic without you." }, { "question": "How do I spot stalled work on a delivery board?", "answer": "Look for anything with no state change and no comment for two working days, then name it and act on it the same day. That check sits in the morning pass, alongside work in progress against each team's limit, the Ready for Dev queue and bug budget burn. Silence usually traces back to a question blocking the build that never got an answer, a ticket approved too thin, or a decision made in chat that never reached the board. Find which one it is, then answer the question, fix the ticket, or bring the record back onto the board. And if the pass takes forty minutes, the board is stale, and fixing it is item one." }, { "question": "How fast should a product manager answer questions that block a build?", "answer": "Within four working hours, either with an answer or with a written assumption on the ticket. Hold one fixed forty-five minute window in the middle of the day, publish it, and let anything that is not blocking wait for refinement. The written assumption is the important half. Record what you assumed, who owns it, and what you would change if it turns out wrong, because a guess is safe once it is written down and owned. Watch the pattern too. Repeated blocking questions on one epic mean the ticket was approved too thin, so the durable fix is the ticket, not faster answers." }, { "question": "How often should a team's priority order actually change?", "answer": "Confirm it daily, change it rarely. Each team gets exactly one visible prioritised order with a single item marked next, set or confirmed once a day in the same pass. Only three things earn a resequence: a validation result, a slip that changes what can land, or a dependency arriving early or late. A message from a loud stakeholder is not one of them. An order that changes every morning is not a sequence, it is a mood, and it costs the switch, the re-plan, and a team that has stopped believing the order means anything." }, { "question": "What should happen to an epic's value when it loses scope mid-quarter?", "answer": "The planned value moves the same day, and you move it, rather than leaving it for the monthly outcome validation to discover. Tenhaw holds its own retainers to that standard, reporting a month that delivers no measurable value as a failed month. If a £240k epic loses the half of its scope that carried most of the value, it is a £150k epic now and its outcome is £90k short. Write the new number on the epic, let the gap show on the outcome, and add one line naming the option you would take: a new epic, a scope trade, or a lower target agreed openly. With eight weeks of quarter left that gap is actionable. Found in week thirteen, it is a postmortem." }, { "question": "A developer is free. Should I rush the next epic through approval?", "answer": "No. An idle developer is the exact pressure the approval gate exists to resist. An epic waved through without product approval, engineering approval and at least one product-approved story buys two days of activity and pays for it with a rebuild. Tenhaw does not trust memory to hold rules under that pressure either, which is why its engineering handbook carries 72 numbered rules written to be enforced by an agent. The honest fix is queue speed, not gate removal. Work the approval queue every day, in sequence order rather than arrival order, and approve or reject inside one working day, with any rejection naming the missing artefact. Run that way, the gate rarely leaves anyone idle for long, and the short wait is cheaper than the rework." }, { "question": "Isn't checking the board every morning just micromanagement?", "answer": "No, because the pass is over the board, not over the people. Tenhaw spent six months on that job at Greggs, Scrum Master support across the Mobile App and Integration squads plus agile coaching beyond engineering. It takes five to ten minutes and checks four things: work in progress against each team's limit, anything with no state change or comment for two working days, the Ready for Dev queue, and bug budget burn. You are not asking anyone how their day is going. You are collecting the two or three items only you can act on, clearing the approvals that are yours, and getting out of the way. Micromanagement is telling people how to do the work. This is deciding what order it happens in." }, { "question": "What do you actually need to run delivery day to day?", "answer": "Two artefacts carry most of it, a delivery board that holds the truth, and the quarter's p50 and p85 forecast range. Around those, four rules agreed in advance. A work in progress limit per team. One visible prioritised order per team, with a single item marked next. A planned currency value on every epic, because that is the number the quarter is measured on and no epic should be approved with it blank. And gate rules everyone knows, so an epic cannot leave Ready for Dev without product approval, engineering approval and at least one product-approved story attached. None of it is proprietary, and Tenhaw publishes the method in full, free for anyone to adopt. Everything else in the routine is habit rather than kit." }, { "question": "What should I do first when a team is over its work in progress limit?", "answer": "Stop starting before you start chasing. Pushing on every item at once feels like management, but attention does not add capacity and the queue only shortens when work leaves it, so the first move is to start nothing new and finish something. Take the item closest to done and clear whatever it is holding: an approval sitting in your own queue, a question blocking the build, a review nobody picked up. The breach itself should be visible the same day, because work in progress against the limit is the first thing the morning pass checks, ahead of stalled items, the Ready for Dev queue and bug budget burn." }, { "question": "Two pieces of work look equally important. Which one goes first?", "answer": "Take the one attached to the outcome furthest from its currency target, because that is where the missing value sits. Urgency and volume are the tie-breakers most teams reach for, and the least informative, because two items can both be genuinely pressing while only one of them closes a gap on the number the quarter is measured against. Each team carries exactly one prioritised order with a single item marked next, so the tie gets broken somewhere visible rather than in a private judgement call. Put the choice on the board as the order, not in your head, or you will re-explain it every morning." }, { "question": "How do we stop delivery decisions getting lost in chat?", "answer": "Land every decision on the ticket in the hour it is made. The familiar failure is that something real gets decided in a thread and the ticket is never touched, so the decision was real but the trail is not, and three weeks later nobody can say why the scope moved or who owned the assumption. Hold one published decision window in the middle of the day, forty-five minutes, then write each call onto the item it affects the same hour, with any assumption named and owned. Chat is fine for reaching a decision. It is not a record, because nobody reads a thread back, and the board is what everyone else self-serves from." }, { "question": "Does every new item really need to sit under an epic?", "answer": "Yes, and routing them takes about ten minutes a day. A bug found this quarter links to the Bug Budget epic, debt raised during the build links to the Tech Debt epic, risks and issues go to the RAID log with a named owner, and every epic sits under an outcome. The Tenhaw Way is blunt about the rest. Anything that cannot name a parent does not start today. It reads like bureaucracy until you see that it buys burn rates that stay meaningful all quarter, and an honest answer when someone asks where the capacity went. Route in the morning pass, not at the end of the week when nobody remembers half the items." }, { "question": "Why does one stuck ticket hurt more when the model writes the code?", "answer": "Because the board carries fewer, larger items. In an AI-native team there are no stories, so your approval queue is epic-level outcome tickets and you approve the currency share, the key user journeys and the test requirements rather than acceptance criteria. One of those stalling is a far bigger share of the quarter than a stalled story ever was, so a two-day stall matters more, not less, and it gets named the morning it shows. Tenhaw sets that operating model up with a delivery team of two or three rather than twenty. A model reporting completion is a claim, not evidence, so done costs more attention too, with someone named watching the tests pass and the journeys demonstrated in front of a person." }, { "question": "What should a delivery lead check at the end of each day?", "answer": "Five minutes, comparing where the work sits against your p50 and p85 forecast range rather than a single invented date. Anything that has fallen outside p85 for landing inside the quarter gets said today, to the person who has to make the choice, with the three options attached: cut scope, move the epic to next quarter, or accept the risk. Do not soften it into an amber status with no ask, because that hands over the worry without handing over the decision. Tenhaw agrees the exit date at kickoff rather than negotiating it at the end. A slip raised in week four costs a conversation. The same slip raised in week twelve costs a commitment somebody else has already made to the business." } ], "aiNote": "The rhythm is identical, the artefacts are not. In an AI-augmented team the approval queue is mostly stories, and your morning pass watches story-level flow: what is in review, what has sat in progress too long, what needs breaking into chapters. In an AI-native team there are no stories, so the queue is epic-level outcome tickets, and you are approving the currency share, the key user journeys and the test requirements rather than acceptance criteria. The board carries fewer, larger items, so one stuck ticket is a much bigger share of the quarter and a two-day stall matters more, not less. The heaviest difference sits at the other end: done costs you real attention, because you or a named developer has to watch the tests pass and the journeys demonstrated before the ticket moves." } ## How to deliver a project on time Source: https://tenhaw.com/the-tenhaw-way/how-to/deliver-on-time Last updated: 2026-11-05 { "slug": "deliver-on-time", "group": "Measuring and managing", "title": "How to deliver a project on time", "summary": "See a changing forecast early enough to choose scope, sequence or an explicit risk decision.", "intro": "Managing delivery to be on time is a detection problem, not an effort problem. You cannot make a late epic early by pushing, but you can see the slip in week four instead of week eleven, while cutting scope, moving an epic or reallocating people are still real options. The method is four habits: commit to dates derived from your own throughput, watch approval gates as the early warning, re-forecast every fortnight, and turn every slip into a priced decision with a named owner and an edited roadmap rather than a status update.", "whenToUse": "Use it from the day a quarter's roadmap is committed until the day it closes. Reach for it in particular when someone asks whether a date will hold and the honest answer is that nobody in the room knows.", "totalTime": "P3M", "timeLabel": "One quarter, from the day the roadmap commits to the day it closes", "tools": [ "The delivery tracker", "An export of the last eight to twelve weeks of throughput", "A spreadsheet for the forecast simulation" ], "steps": [ { "name": "Define on time before the quarter starts", "detail": "Put every epic in the quarter on one table, including the standing Tech Debt and Bug Budget epics. Each row carries four things: the date it must clear Ready for Dev, the date it must be shipped and in value monitoring, its planned share of its outcome's currency target, and the name of the person who accepted that date. That table is the commitment, and on time is a property of the whole table rather than of whichever epic is being asked about loudest. Anything not on the table is not committed, and you say so the first day someone assumes otherwise." }, { "name": "Build a throughput baseline from your own history", "detail": "Export the last two or three quarters of finished work from your tracker. Count completed items per team per week: stories for an AI-augmented team, outcome tickets for an AI-native one. Keep the weekly samples raw, bad weeks, holidays and incident weeks included, because those recur. Aim for at least twelve weekly samples. Record a start date and a done date for every item so you get cycle time as well. Then compute the number few teams have: your calibration ratio, validated currency value over planned currency value from outcome validation. Plan four million, validate two point eight, and every forecast you publish carries seventy per cent." }, { "name": "Forecast in ranges, plan p50, commit p85", "detail": "Never hand the business a single date. Take the weekly throughput samples, draw from them at random ten thousand times against the remaining item count, and read the distribution. Plan the team's work against the p50. Give anyone outside the team the p85. Inflate the item count first by your historical scope growth: if last quarter's epics finished with twenty per cent more items than they opened with, forecast 1.2 times today's count and say that is what you did. Publish the count next to the dates, because it is the variable that moves them most. When the count changes, re-run the simulation rather than debating the old date." }, { "name": "Sequence the portfolio, not the loudest epic", "detail": "Capacity is shared even when boards are not. Lay every epic in the quarter on one timeline, mark the named people or teams each one needs, and find the weeks where two epics want the same person. Stagger start dates until every epic clears its p85 inside the quarter, and cap epics in flight per team at two, because a third converts into cycle time rather than output. Then redo the arithmetic: if the moved epics no longer cover their outcomes' currency targets, that gap gets a named owner before the quarter opens, not after it ends. One safe epic and three sitting at p50 is not a schedule." }, { "name": "Track gate dates as your earliest warning", "detail": "Ship dates slip because gates slip first, weeks earlier and in plain view. For an AI-augmented team the gate is product approval, engineering approval and at least one product-approved story attached before the epic leaves Ready for Dev. For an AI-native team it is the currency share, the key user journeys and the test requirements written down, with both approvals. Show days-to-gate on every epic and review it weekly. Treat a gate date that moves as a ship date that has already moved at least as far, and raise it the same day. Miss a gate early, react early." }, { "name": "Run a five-minute daily lookahead", "detail": "One person, five minutes, before stand-up rather than during it. Check five things: any item untouched for longer than twice its usual cycle time, any team over its work-in-progress limit, any gate due inside ten working days that is not ready, the Bug Budget epic's burn rate against the same week last quarter, and anything blocked on a team that does not know it is blocking. Write only the exceptions, in a channel the team already reads, one line each with a name attached. The day it becomes a meeting with a round-the-room, it has stopped doing its job." }, { "name": "Re-forecast fortnightly and raise slips the same day", "detail": "At refinement, re-point whatever has shifted in understanding, then re-run the simulation against updated throughput and the current item count. Compare the new p85 with the committed date on your table. If the p85 has crossed it, that is a slip, and it goes to the accountable person that day in one line: this epic was committed for 12 September, its p85 is now 3 October, and the value at risk is six hundred thousand pounds. Send it before you know what to do about it. Nobody thanks you for a slip announced in week eleven that the numbers showed in week four." }, { "name": "Turn every slip into a priced decision", "detail": "A slip is a choice, so arrive with the options costed. There are four. Cut scope inside the epic, naming which stories or user journeys come out and what the planned value drops to. Move the epic to next quarter and move its currency share with it, so the outcome gap shows in the plan. Pull people off a named lower-value epic and state what that epic loses. Or accept the later date and say what it costs in months of value monitoring before the year ends. One named person picks, the decision and its date go in the RAID log, and the roadmap is edited to match the choice." }, { "name": "Close the quarter honestly and recalibrate", "detail": "At roadmap close, record for every epic whether it shipped, moved or was dropped, and what value travelled with it. Compute two numbers: committed epics landed over committed epics, and validated value over planned value once outcome validation has caught up. Both feed the next quarter's baseline, and the second is your calibration ratio. Do this before you open the next roadmap, because a roadmap opened on last quarter's assumptions inherits last quarter's overrun. Teams that skip it forecast from optimism, and their p85 becomes decoration nobody outside the team believes twice." } ], "pitfalls": [ "Treating percentage complete as progress. It is self-reported, it converges on ninety per cent and stays there, and it is why teams discover slips in the final fortnight. A gate date that has moved tells you more in one second than a week of status percentages.", "Forecasting from capacity rather than throughput. Counting available developer days and dividing assumes nobody is interrupted, blocked, ill or on holiday. Historical throughput already has all of that baked in, which is why it is the only input worth trusting.", "Forecasting against a fixed item count. Backlogs grow as work is understood, so a forecast against today's count is optimistic by construction. Measure how much last quarter's epics grew between opening and closing, and carry that multiplier openly rather than absorbing the growth as a surprise in week nine.", "Protecting the loudest epic and quietly starving the rest. The escalated epic gets the people, three others drift unwatched, and the quarter still misses, because on time was always a property of the whole roadmap rather than of the epic with the most senior sponsor.", "Re-baselining the date instead of recording the slip. Quietly moving a committed date to match the current forecast makes every quarter look successful and destroys the calibration data the next forecast needs. Move the date if that is the decision, but record that it moved, when, and why." ], "definitionOfDone": [ "Every epic in the quarter, Tech Debt and Bug Budget included, has a committed gate date, a committed ship date, a planned currency share and a named person who accepted it, in one place the business can read without asking.", "Every date given outside the team is a p85 derived from your own throughput, published with the p50 and the item count the simulation assumed.", "No slip reaches the accountable person later than the fortnight in which the forecast first showed it.", "Every raised slip has a recorded decision, a named owner, a stated currency consequence and a roadmap edited to match.", "The quarter closed with a landed-epic hit rate and a recomputed calibration ratio, and both are inputs to the next quarter's forecast." ], "example": { "heading": "A slip found in week four instead of week eleven", "body": "Northbridge Retail is an invented mid-sized online retailer, and the numbers below are invented with it. It opens Q3 with one outcome, reduce checkout abandonment, target 2.4 million pounds, carried by four epics at 700k, 600k, 600k and 500k plus the standing Tech Debt and Bug Budget epics. Its calibration ratio last year was seventy per cent, so the plan expects about 1.68 million against a 2.4 million target. That gap sits on the table before the quarter opens rather than surfacing in October. Two teams have averaged nine completed stories a week over three quarters, with weekly samples ranging from four to fourteen. The 600k guest checkout epic holds 46 items, inflated to 55 for scope growth: p50 lands on 5 September, p85 on 12 September, and 12 September becomes the commitment. In week four the daily lookahead flags days-to-gate on that epic going negative, because engineering approval is stuck behind payment sandbox access. The gate moves two weeks. At the next refinement the item count is 58 and the p85 lands on 3 October, past the commitment and past quarter end. The delivery lead sends the sponsor one line that afternoon naming 600k as the value at risk, with three costed options. The sponsor cuts the saved-card journey, dropping the epic's planned value to 380k, and moves one engineer from the 500k epic, which slips its own p85 by nine days. Both changes go in the RAID log and onto the roadmap that week. Found in week eleven, the only surviving option would have been accepting 3 October." }, "related": [ { "label": "What an AI-native SDLC changes", "href": "/guides/ai-native-sdlc-and-product-delivery" }, { "label": "Designing roles, decisions and workflows for an AI-native organisation", "href": "/guides/target-operating-model-for-an-ai-native-organisation" } ], "faq": [ { "question": "We have no clean history to forecast from. Where do we start?", "answer": "Start counting this week and forecast anyway. Four weekly throughput samples give a crude range that beats an invented date, and you widen the gap between p50 and p85 to reflect how thin the data is. The numbers have to come from this team's own flow, so do not borrow another team's velocity or an industry benchmark. Until the samples build up, lean on gate dates as the primary signal, because they are observable from day one and need no history to mean something." }, { "question": "The business will not accept a range. They want one date.", "answer": "Give them the p85 as their one date. That is what the range is for. You plan the team against the p50, you commit externally to the p85, and you keep the p50 inside the team because outside it the earlier number is heard as the date. Say the p85 is a date you expect to beat five times in six, based on the last three quarters of this team's throughput and an item count you publish alongside it. That answer survives week nine, which a confident single date does not. Tenhaw gives its own clients p85 dates for exactly this reason, and publishes the method that produces them." }, { "question": "The date is fixed externally, by a regulator or a contract. What changes?", "answer": "The date stops being the variable and scope becomes the variable, so run the simulation backwards. Ask how many items this team finishes by the fixed date at p85, compare that with the item count in the epic, and the difference is scope you cut now rather than in the final fortnight. Take it out explicitly, restate the epic's planned currency value at the reduced scope, and have that accepted by name. Tenhaw works to hard windows of its own, and a proof of concept for Placing Platform Limited (PPL) took PDFs through to business intelligence on Azure inside two weeks, ground it had circled for roughly a year. A fixed date with unfixed scope is not a commitment, it is a countdown." }, { "question": "How large a forecast movement is worth escalating?", "answer": "Any movement that puts the p85 past the committed date on the table, however small, and any gate date that moves at all. Everything else stays inside the team. The accountable person then hears from you rarely, and when they do it always means a decision is needed, which is what keeps escalation cheap and credible. Escalating every wobble in the p50 trains people to ignore you, which is how the week-eleven surprise reaches teams that were technically reporting all along." }, { "question": "Why do delivery slips only show up in the last few weeks?", "answer": "Because most teams track percentage complete, which is self-reported, converges on ninety per cent and sits there until the final fortnight makes it unignorable. Ship dates slip because approval gates slip first, weeks ahead of the deadline, so the slip itself was there much earlier and in plain view. Tenhaw installs that early warning on the client programmes it runs. Show days-to-gate on every epic, review it weekly, and treat any gate date that moves as a ship date that has already moved at least as far. Add a fortnightly re-forecast from your own throughput and no slip needs to reach the accountable person later than the fortnight in which the numbers first showed it." }, { "question": "Can we get a late project back on time by pushing the team harder?", "answer": "No. You cannot make a late epic early by pushing, which is why delivering on time is a detection problem rather than an effort problem. The teams that land their quarters are the ones that see the slip in week four instead of week eleven, while cutting scope, moving an epic or reallocating people are still real choices. That takes four habits: commit to dates derived from your own throughput, watch approval gates as the early warning, re-forecast every fortnight, and turn every slip into a priced decision with a named owner. Tenhaw runs those four habits with teams of two or three senior people, James Rooney on every engagement, rather than a surge at the end." }, { "question": "What are our options when an epic is going to miss its date?", "answer": "Four, and the job is to arrive with all of them costed rather than with a status update. Tenhaw's delivery leads cost the same four for clients before a slip reaches a sponsor. Cut scope inside the epic, naming which stories or user journeys come out and what the planned value drops to. Move the epic to next quarter and move its currency share with it, so the outcome gap shows in the plan. Pull people off a named lower-value epic and state what that epic loses. Or accept the later date and say what it costs in months of value monitoring. One named person picks, the decision and its date go in the RAID log, and the roadmap is edited to match the choice." }, { "question": "Should we forecast delivery from team capacity or from throughput?", "answer": "From throughput, every time. Counting available developer days and dividing assumes nobody is interrupted, blocked, ill or on holiday, and historical throughput already has all of that baked in. Export the last two or three quarters of finished work from your tracker and count completed items per team per week, stories for an AI-augmented team and outcome tickets for an AI-native one. Keep the weekly samples raw, bad weeks, holidays and incident weeks included, because those recur. Aim for at least twelve samples, record a start and a done date for every item so you get cycle time as well, then simulate against the remaining item count and read the distribution rather than averaging it." }, { "question": "Is it OK to move a committed date when the forecast slips?", "answer": "Moving the date can be the right decision; moving it quietly never is. Re-baselining the commitment to match the current forecast makes every quarter look successful and destroys the calibration data the next forecast needs, so if the date moves, record that it moved, when, and why. The honest route is to raise the slip the same day the fortnightly re-forecast shows the p85 crossing the committed date, put the choice to the accountable person with the value at risk attached, and edit the roadmap to match whatever they decide. At quarter close, count how many committed epics actually landed, so the next plan inherits your record rather than your optimism." }, { "question": "Can we count work as done when the AI says it is finished?", "answer": "No. An outcome ticket counts as done only when its test requirements pass and its key user journeys are demonstrated working, never when the model reports it has finished. Count the model's word as done and your throughput inflates, the forecast built on it tightens, and the slip you should have seen in week four surfaces in value monitoring instead. Tenhaw holds that line on its own client builds, against an open-source standard of 72 rules enforced by an agent. Forecasting in outcome tickets also needs patience. An AI-native team may finish two or three a week rather than nine stories, so the samples are fewer and fatter and the gap between p50 and p85 is wider for the same confidence." }, { "question": "Who should own a delivery date, the delivery lead or the sponsor?", "answer": "Both, at different points, and the split is what makes the date mean anything. Tenhaw staffs the delivery lead's half of that split on client programmes it runs. Every committed date on the quarter's table carries the name of the person who accepted it, and that person owns the decision when the date comes under threat. The delivery lead owns detection: tracking days-to-gate, re-forecasting every fortnight, and putting the slip in front of the accountable person the day the numbers show it, with the options costed and the value at risk attached. What a delivery lead must not do is choose quietly on the sponsor's behalf, because a date moved without a named decision destroys the record the next forecast is built on." }, { "question": "How many epics should one team have in flight at once?", "answer": "Two. A third converts into cycle time rather than output, so cap epics in flight per team and stagger start dates instead. What makes the cap real is a portfolio view: lay every epic in the quarter on one timeline, mark the named people or teams each one needs, and find the weeks where two epics want the same person, because capacity is shared even when boards are not. Move start dates until every epic clears its p85 inside the quarter, then redo the arithmetic. If the moved epics no longer cover their outcomes' currency targets, that gap gets a named owner before the quarter opens. One safe epic and three sitting at p50 is not a schedule." }, { "question": "What does on time actually mean for a delivery team?", "answer": "It means the whole commitment table landed, not that one epic landed. Before the quarter starts, put every epic on one table and give each row four things: the date it must clear Ready for Dev, the date it must ship and be in value monitoring, its planned share of its outcome's currency target, and the name of the person who accepted that date. The standing Tech Debt and Bug Budget epics sit on it too, because they consume the same capacity. That table is the commitment, it lives where the business can read it without asking, and on time is a property of all of it rather than of the epic being asked about loudest." }, { "question": "Why does the quarter still miss after we rescue the escalated epic?", "answer": "Because on time was always a property of the whole roadmap rather than of the epic with the most senior sponsor. The rescue takes people from somewhere, three other epics drift unwatched, and the arithmetic that mattered was never redone. If people move, name the lower-value epic they come off, state what it loses and re-forecast it, because its p85 moves too. Keep days-to-gate visible on every epic in the quarter rather than on the escalated one alone, and let the fortnightly re-forecast cover all of them. Then reallocation is a priced trade with a named owner, and the epics nobody rescued stop being a surprise in week eleven." }, { "question": "We plan more value than we deliver every quarter. What should we do?", "answer": "Measure the ratio and carry it into the next plan instead of promising the full number again. Tenhaw reports a month that delivers no measurable value as a failed month for the same reason. At quarter close, once outcome validation has caught up, compute validated currency value over planned currency value. Plan four million, validate two point eight, and every forecast you publish afterwards carries seventy per cent. Compute the landed hit rate alongside it, committed epics that shipped over committed epics. Do both before you open the next roadmap, which otherwise inherits last quarter's overrun. That ratio puts the shortfall on the table in week one, where a sponsor can add an epic or accept a lower target, rather than in October." }, { "question": "Someone assumed their project was committed. How do we handle it?", "answer": "Anything not on the quarter's table is not committed, and you say so the first day the assumption surfaces. That is blunt, and it is far cheaper than a business planning around a date nobody ever accepted. Then treat the request as what it is, a change to the commitment rather than a favour. Putting it on the table means a named person accepts its dates, and it means something else moves, so arrive with the options costed: cut scope inside a named epic, move one to next quarter, or accept a later date and say what that costs in months of value monitoring. One person decides, the decision goes in the RAID log, and the roadmap is edited to match." }, { "question": "What does catching a slip in week four actually look like?", "answer": "Take an invented retailer and invented numbers. A 600k guest checkout epic holds 46 items, inflated to 55 for historical scope growth, so the p50 lands on 5 September and the p85 on 12 September, and 12 September becomes the commitment. In week four the daily lookahead flags days-to-gate going negative, because engineering approval is stuck behind payment sandbox access, and the gate moves two weeks. At the next refinement the count is 58 and the p85 is 3 October. The sponsor cuts the saved-card journey, dropping planned value to 380k, and moves one engineer off a 500k epic, which slips nine days. Found in week eleven, the only surviving option is 3 October." }, { "question": "Is a rising bug count an early warning that a date will slip?", "answer": "It is, and it is worth reading weekly rather than at the end. The Bug Budget epic sits on the same commitment table as everything else, so time spent over its budget is time taken from committed work. The daily lookahead compares its burn rate with the same week last quarter, which beats arguing about whether things feel noisier than usual, and a burn running ahead of that comparison is capacity quietly leaving the plan. It will surface soon enough in the fortnightly re-forecast as a p85 drifting past a committed date. Raise it the same day, exactly as you would a gate that has moved, and name the epic it is costing." } ], "aiNote": "For an AI-native team the arithmetic is the same but the samples are fewer and fatter: throughput is counted in outcome tickets, so a team may finish two or three a week rather than nine stories, and the gap between p50 and p85 will be wider for the same confidence. Collect more weeks before trusting it. The bigger risk is the definition of done. An outcome ticket counts as done only when its test requirements pass and its key user journeys are demonstrated working, never when the model reports it has finished. Count the model's word as done and throughput inflates, the forecast tightens, and the slip surfaces in value monitoring instead of week four." } ## How to forecast delivery with confidence intervals Source: https://tenhaw.com/the-tenhaw-way/how-to/forecast-with-confidence-intervals Last updated: 2026-11-05 { "slug": "forecast-with-confidence-intervals", "group": "Forecasting and team health", "title": "How to forecast delivery with confidence intervals", "summary": "Use your own throughput history to publish a planning range, its probability and the decision it supports.", "intro": "Forecasting with confidence intervals replaces the invented date with two numbers taken from your own delivery history: a p50 the team plans against and a p85 the business commits to. You get there by counting completed items per week, inflating the remaining scope by your historical split rate, and simulating ten thousand possible quarters against that history in a spreadsheet. Good looks like a forecast refreshed weekly, published as a range with the unit and window written beside it, and converted into the currency at risk the moment the range misses the quarter end.", "whenToUse": "Run it the week a quarterly roadmap opens, then again every week until the quarter closes. Reach for it the moment anyone asks whether an epic will land, or asks for a date before the work has been counted.", "totalTime": "PT2H", "timeLabel": "About two hours, then half an hour a week to refresh it", "tools": [ "A spreadsheet", "Twelve weeks of finished-per-week throughput history" ], "steps": [ { "name": "Pick one countable unit and right-size it", "detail": "Pick the thing you will count finished per week and do not change it mid-quarter. AI-augmented teams count stories closed. AI-native teams count key user journeys demonstrated, because outcome tickets are too few and too large to sample. Count items, never story points: points get re-baselined every time a team re-points, so rising velocity can mean nothing changed except the scale. Then check the items are interchangeable enough to sample. If single items routinely take more than two weeks, split them at refinement first, because the arithmetic assumes one item is much like another. Write the unit at the top of the forecast." }, { "name": "Pull twelve weeks of throughput history", "detail": "Export the finished-per-week count for the last twelve weeks. Eight is the minimum that gives a usable spread. Use the date an item met done, not the date it merged or demoed. Include everything that consumed capacity: stories, bugs linked to the Bug Budget epic, tech debt linked to the Tech Debt epic. Keep the zero weeks and the holiday weeks. They are the tail, and deleting them is the most common way a forecast turns optimistic. If the system changed materially, a team split, a two-week freeze, a switch of delivery mode, shorten the window and accept a wider interval rather than editing the numbers." }, { "name": "Count the remaining scope and inflate it", "detail": "Count the items left in the roadmap, including the Tech Debt and Bug Budget epics, which are the first things people forget and the last things that stop consuming throughput. Count anything already in progress as remaining, because half-counting it flatters the answer. Then apply your split rate: the items an epic finally shipped divided by the items it carried when it left Ready for Dev, averaged over your last five closed epics. Most teams land between 1.2 and 1.5. If you have never measured it, use 1.3 this quarter and measure it properly for the next one. Forecast against the inflated number." }, { "name": "Simulate ten thousand quarters in a spreadsheet", "detail": "A spreadsheet is enough. Put the twelve weekly counts in A1:A12. For the fixed-date question, fill B1 across to J1, one cell per remaining week, with =INDEX($A$1:$A$12, RANDBETWEEN(1,12)), total the row in K1, then fill the block down ten thousand rows. Column K is now ten thousand plausible quarters. For the when-is-it-done question, extend the draws to thirty columns, running-total across the row, and record the first column that reaches your inflated scope. Copy and paste the results as values before reading percentiles, or the sheet resamples every time you touch it. Never average the twelve weeks. The spread is the whole point." }, { "name": "Read p50 and p85, commit at p85", "detail": "Read the percentiles off the results column with PERCENTILE.INC. For a date, 0.5 and 0.85 give the week half the simulations finished by and the week 85% of them did. For an item count against a fixed date, the 85% confident number is the 15th percentile of the totals: being 85% sure of delivering at least N items means only 15% of simulated quarters came in below N. Publish one sentence carrying both, for example forty-three items at p50 and thirty-eight at p85. Plan against p50, commit against p85. The gap between them measures how variable your delivery is, and closing it beats pushing the average up." }, { "name": "Convert the shortfall into currency", "detail": "Rank the remaining epics in the order you will really deliver them, draw a line at the p85 count, and name every epic that falls below it. Each carries a planned share of its outcome's currency target, so sum those shares and the shortfall stops being a mood and becomes a number. Take it to the outcome owner with three options and no fourth: defer this value to next quarter, cut something above the line to pull it up, or accept that the outcome target moves. Going faster is not on the list, and offering it wastes the meeting and the credibility of the forecast." }, { "name": "Refresh weekly and plot the trend", "detail": "Re-run it the same morning every week with the new throughput week and a fresh scope count, and keep every version in one sheet: date, sample window, split rate, inflated scope, p50, p85. The useful artefact is not this week's p85, it is the line of p85s across the quarter. Three consecutive weeks drifting later is a signal you can act on in week five, whereas one bad week is noise. Put the current range on the daily dashboard next to the quarter end date, so divergence surfaces before stand-up rather than at a steering meeting in week eleven." }, { "name": "Check your calibration at roadmap close", "detail": "At roadmap close, write down three comparisons: the p85 count you published against what finished, the forecast completion week against the real one, and the split rate you assumed against the one you got. Four quarters of that gives you a realisation rate, and if your organisation reliably realises 70% of what it plans, 70% is the number next quarter's planning uses rather than optimism. If actuals fall below p85 more often than roughly one quarter in six, the sample window is flattering you: too short, too recent, or missing the bad weeks." } ], "pitfalls": [ "Forecasting from velocity in story points. Points get re-baselined every time a team re-points, so the series measures the scale as much as the work, and a rising line can mean nothing moved. Counted items cannot be quietly inflated.", "Ignoring scope growth. Teams forecast accurately against the backlog they counted and miss anyway, because the epics grew by a third between Ready for Dev and release. If you are not measuring your split rate, you are forecasting the wrong quantity precisely.", "Averaging the history instead of sampling it. An average assumes every remaining week is an average week. The interval exists because they are not, and the tail is where the misses live.", "Deleting the weeks that look unrepresentative. The zero week, the week of the incident, the week between Christmas and New Year: strip those out and you have removed exactly the variability the interval is meant to price, leaving a confident forecast of a quarter you have never had.", "Publishing p50 and calling it the date. Half your quarters miss it by definition. The business hears a commitment, the team hears a stretch, and when it slips the method loses credibility rather than the choice of percentile." ], "definitionOfDone": [ "The forecast is published as p50 and p85 together, and no single date appears anywhere in it.", "The unit counted, the sample window and the split rate used are written next to the numbers, so anyone can reproduce them.", "The scope count includes the quarter's Tech Debt and Bug Budget epics and everything currently in progress.", "Every epic falling below the p85 line is named with its planned currency share, so the value at risk is a number rather than a warning.", "The forecast was refreshed within the last seven days, and the trend of p85 across the quarter is visible on the daily dashboard.", "The outcome owner has made an explicit call on the shortfall, or there is no shortfall to call." ], "example": { "heading": "A worked example: one team, nine weeks left", "body": "Take a generic mid-market grocer, call it Northwind Foods. One AI-augmented team, nine weeks left in the quarter. Their last twelve weekly story counts were 5, 3, 7, 4, 6, 2, 5, 8, 4, 5, 3 and 6: fifty-eight items, an average of 4.8 a week, and a spread wide enough to matter. The roadmap has 40 items left in it, Tech Debt and Bug Budget epics included, and their last five closed epics shipped 1.2 items for every item counted at Ready for Dev, so they forecast against 48 rather than 40. Ten thousand simulated nine-week runs give a p50 of 43 items and a p85 of 38. The coin-flip number misses 48 by five. The number they would commit to misses it by ten. Asked as a date instead, 48 items lands at p50 after ten weeks and at p85 after twelve, against nine weeks of quarter remaining. So ten items leave the quarter. Ranked in delivery order, the ten below the line are the whole of one epic carrying £180k of a £1.2m outcome target. Nobody asks for overtime. With nine weeks still to run, the outcome owner chooses between deferring that £180k to next quarter and dropping something above the line to pull it up. The same conversation with two weeks left has no options in it." }, "aiNote": "For an AI-augmented team the countable unit is stories closed and twelve weeks of history is plenty. For an AI-native team the outcome ticket is the unit of work, and outcome tickets are too few and too large to sample well, so count key user journeys demonstrated instead and pull sixteen weeks rather than twelve. Expect a wider p50 to p85 gap and publish it rather than smoothing it. Two further differences bite. Throughput history goes stale faster, because a change of model, harness or verification standard changes the system as much as a team split does, so treat those as reasons to shorten the window. And measure the split rate rather than assuming it: outcome tickets tend to grow more between approval and done than stories do, because more of the decomposition happens after the ticket is written.", "related": [ { "label": "What an AI-native SDLC changes", "href": "/guides/ai-native-sdlc-and-product-delivery" }, { "label": "What a defensible AI business case holds", "href": "/guides/business-case-for-an-agentic-programme" } ], "faq": [ { "question": "How much history do I need before I can forecast?", "answer": "Eight weeks is the working minimum and twelve is comfortable. What matters more than the length is whether the window describes the system you are in now. If the team doubled, the delivery mode changed, or the quarter opened with a two-week freeze, use the weeks since that change and accept a wider interval rather than padding the sample with data from a team that no longer exists. A wide honest range beats a narrow invented one." }, { "question": "The team is brand new and has no throughput at all. What then?", "answer": "Borrow a reference class for the first six weeks by taking the weekly throughput of a comparable team in the same organisation, labelling the forecast clearly as borrowed history, and publishing a deliberately wide range. Replace one borrowed week with a real one every week until the sample is entirely yours. What you must not do is fall back on a single date because you have no data, since that is the situation in which invented dates are least defensible. Tenhaw borrows history the same way at the start of a build engagement, before the new team has weeks of its own to sample." }, { "question": "The business will not accept a range. What do I give them?", "answer": "Give them p85 as the committed number and keep p50 as the internal plan, then say the remaining 15% out loud so nobody is surprised later. The range is not there for comfort, it is there to produce the currency number. James Rooney runs every Tenhaw engagement, so the partner is the one holding that conversation with the outcome owner. Once the value at risk is on the table, the conversation stops being about whether the date is right and becomes a decision about which value gets deferred, which is the only version anyone can act on." }, { "question": "Do I need a forecasting tool to do this?", "answer": "No. Ten thousand rows and two formulas in a spreadsheet produce the same answer as any Monte Carlo tool, and doing it by hand for a quarter teaches you where the forecast is fragile. Tools earn their place later, when you want the weekly refresh and the trend of p85 maintained without someone remembering. Buying one first does not help, because the arguments are always about the inputs, the unit, the window and the split rate, not about the maths." }, { "question": "What is the difference between a p50 and a p85 forecast?", "answer": "A p50 is the point half your simulated quarters finish by, and a p85 is the point 85% of them do. Resample your own weekly throughput history into ten thousand plausible quarters and read both percentiles off the same results. This arithmetic is published in full by Tenhaw and free to adopt without hiring anyone. The team plans against the p50, the business commits to the p85, and the gap between them measures how variable your delivery is, so closing that gap beats pushing the average up. Publish both in one sentence, for example forty-three items at p50 and thirty-eight at p85, so nobody mistakes the coin-flip number for the commitment." }, { "question": "Should we forecast delivery in story points or counted items?", "answer": "Counted items, always. Story points get re-baselined every time a team re-points, so a velocity line measures the scale as much as the work, and a rising line can mean nothing moved. Counted items cannot be quietly inflated. Pick one unit and do not change it mid-quarter, whether that is stories closed for an AI-augmented team or key user journeys demonstrated for an AI-native one, because outcome tickets are too few and too large to sample well. Write the unit at the top of the forecast, check the items are interchangeable enough to sample, and split anything that routinely takes more than two weeks at refinement first." }, { "question": "How do you account for scope growth in a delivery forecast?", "answer": "Inflate the remaining scope by your split rate before you forecast against it. The split rate is the number of items an epic finally shipped divided by the items it carried when it left Ready for Dev, averaged over your last five closed epics; most teams land between 1.2 and 1.5. If you have never measured it, use 1.3 this quarter and measure it properly for the next one. Unmeasured scope growth is why teams forecast accurately against the backlog they counted and miss anyway, since the epics grew by a third between Ready for Dev and release. Count anything already in progress as remaining too, because half-counting it flatters the answer." }, { "question": "How do you run a Monte Carlo delivery forecast in a spreadsheet?", "answer": "Put your last twelve weekly finished-item counts in A1:A12, fill one cell per remaining week with =INDEX($A$1:$A$12, RANDBETWEEN(1,12)), total the row, and fill the block down ten thousand rows. The totals column is now ten thousand plausible quarters. Read it with PERCENTILE.INC, where 0.5 gives the p50 item count and 0.15 gives the 85% confident one, because being 85% sure of at least N items means only 15% of simulations fall below N. For a completion date instead, extend the draws to thirty columns, running-total across the row, and record the first column that reaches your inflated scope. Paste the results as values before reading percentiles, or the sheet resamples every time you touch it." }, { "question": "Should I exclude holiday weeks and outliers from throughput data?", "answer": "No. Keep the zero weeks, the incident weeks and the week between Christmas and New Year, because they are the tail of your distribution and the tail is where the misses live. Deleting the weeks that look unrepresentative removes exactly the variability the interval is meant to price, leaving a confident forecast of a quarter you have never actually had. It is also the most common way a forecast quietly turns optimistic. The one legitimate reason to drop history is a material change to the system, a team split, a two-week freeze or a switch of delivery mode. In that case, shorten the window to the weeks since the change and accept a wider interval rather than editing the numbers." }, { "question": "What do you do when the forecast says the quarter will not fit?", "answer": "Convert the shortfall into currency and take it to the outcome owner as a decision. Rank the remaining epics in the order you will really deliver them, draw a line at the p85 count, and name every epic below it with its planned share of its outcome's currency target. Summed, those shares turn the miss from a warning into a number. Tenhaw reports any month that delivers no measurable value as a failed month. Then offer three options and no fourth: defer that value to next quarter, cut something above the line to pull it up, or accept that the outcome target moves. Going faster is not on the list, and offering it wastes the meeting and the credibility of the forecast." }, { "question": "Is it worth re-running the forecast every week?", "answer": "Yes, and the reason is the trend rather than this week's number. Re-run it the same morning every week with the new throughput week and a fresh scope count, and keep every version as a row in one sheet: date, sample window, split rate, inflated scope, p50 and p85. The useful artefact is the line of p85s across the quarter, because one bad week is noise while three consecutive weeks drifting later is a signal you can act on in week five. Put the current range on the daily dashboard next to the quarter end date, so divergence surfaces before stand-up rather than at a steering meeting in week eleven." }, { "question": "How do we know if our delivery forecasts are actually any good?", "answer": "Mark them at roadmap close and write down three comparisons: the p85 count you published against what finished, the forecast completion week against the real one, and the split rate you assumed against the one you got. Do that for four quarters and you know whether to believe the next forecast, which is the only honest reason anyone should. There is a faster diagnostic in the meantime. If actuals fall below p85 more often than roughly one quarter in six, the sample window is flattering you: too short, too recent, or missing the bad weeks. Keep the published numbers somewhere you can find them later, or there is nothing to mark." }, { "question": "Does throughput forecasting still work when the model writes most of the code?", "answer": "It does, with three adjustments. An AI-native team counts key user journeys demonstrated rather than stories closed, because outcome tickets are too few and too large to sample well, and pulls sixteen weeks of history rather than twelve. Expect a wider p50 to p85 gap and publish it rather than smoothing it. A change of model, harness or verification standard changes the system as much as a team split does, so throughput history goes stale faster and each of those is a reason to shorten the window. And measure the split rate rather than assuming it, because outcome tickets grow more between approval and done than stories do. Tenhaw publishes its own engineering standard as 72 rules on GitHub, enforced by an agent rather than remembered." }, { "question": "How much effort is it to set up delivery forecasting from scratch?", "answer": "About two hours the first time, then roughly half an hour a week to keep it current. Most of those two hours goes on inputs rather than maths: exporting twelve weeks of finished-per-week counts, deciding the one unit you will count and not change mid-quarter, counting the remaining scope including the Tech Debt and Bug Budget epics and everything in progress, and working out your split rate from the last five closed epics. The simulation itself is two formulas and a fill-down in a spreadsheet. Tenhaw pair-programs this with a client engineer so the sheet stays theirs. You leave that first sitting with a p50 and a p85 you can publish, and every week after it is an update rather than a project." }, { "question": "Why not just average our weekly throughput to get a date?", "answer": "Because an average only answers what happens if every remaining week is an average week, which is not a question anybody asked. Take a team whose last twelve weekly counts were 5, 3, 7, 4, 6, 2, 5, 8, 4, 5, 3 and 6. That averages 4.8 a week, so nine weeks looks like 43 items, which is exactly the p50 the simulation gives. Resampling the same history puts p85 at 38, and those five items are the risk the average deleted. Worse, it deletes the tail while it is at it, the zero weeks and the incident weeks, and the tail is where the misses actually live." }, { "question": "What should a delivery forecast include besides the two numbers?", "answer": "The unit you counted, the sample window and the split rate you used, written next to the numbers so anyone can reproduce them, and no single date anywhere in the document. Add the date it was last refreshed, because a forecast older than a week is history rather than a forecast. If the p85 falls short of the quarter, name every epic below the line with its planned share of its outcome's currency target, so the value at risk is a number rather than a warning, then record what the outcome owner decided. Published that way, the inputs are on the table and the meeting argues about the decision instead of the maths." }, { "question": "Do bug fixes and tech debt count towards throughput?", "answer": "Yes, on both sides of the arithmetic. Throughput history should include everything that consumed capacity: stories, bugs linked to the Bug Budget epic, and tech debt linked to the Tech Debt epic. Use the date an item met done, not the date it merged or demoed. The remaining scope count has to include those same epics, which are the first things people forget and the last things that stop consuming throughput. Leave them out of both and you have forecast a team that only ever does feature work, which is not the team you have." }, { "question": "Does counting items work if our stories are wildly different sizes?", "answer": "Only if they are interchangeable enough to sample, because the arithmetic assumes one item is much like another. Ordinary variation is fine, since counting finished items per week already absorbs it and that spread across weeks is exactly what the interval prices. What breaks the forecast is the item that runs for a month. The weeks it sits in look empty, the week it lands looks exceptional, and the simulation then draws from a history describing nothing real. Fix that at refinement rather than in the spreadsheet, splitting anything that routinely takes more than two weeks before it reaches the board you are counting from." } ] } ## How to run a team health check Source: https://tenhaw.com/the-tenhaw-way/how-to/team-health-check Last updated: 2026-11-05 { "slug": "team-health-check", "group": "Forecasting and team health", "title": "How to run a team health check", "summary": "Give the team a consistent monthly way to score its experience, discuss the evidence and choose a small number of actions.", "intro": "A team health check is a monthly forty-five minute session where everyone on one team privately scores the same fixed statements about the work, with the tracker data on the table so predictability and quality are not scored on mood. Good looks like this: ten cards whose wording never changes, scores revealed at once, a colour and a direction arrow on each, at most two actions that exist as real tickets, the card published unedited to management the same day, and a decision rule that acts on three consecutive bad months rather than on one.", "whenToUse": "Once a month, per team, in the same week every month, alongside the fortnightly retrospective rather than instead of it. Run an extra one as a baseline in your first fortnight with a new team, and again when a team switches between AI-augmented and AI-native delivery.", "totalTime": "PT2H", "timeLabel": "About two hours a month, of which the session is forty-five minutes", "tools": [ "The ten health-check cards", "One page of the month's delivery data", "A way to score privately before the reveal" ], "steps": [ { "name": "Fix ten cards and name an owner", "detail": "A health check is ten statements the team scores, not an open conversation about how things are going. Use these ten: line of sight, value confidence, predictability, flow, work arriving ready, tech debt, defects, tooling, safety, pace. Word each so a person can agree or disagree with it: \"I know which outcome my current work rolls up to and what it is worth\", \"we landed what our p85 forecast said we would\", \"I can change this codebase without fear\". The delivery lead owns the ritual, books the same slot monthly, and freezes the wording so month three compares to month one. Ten cards, no additions." }, { "name": "Pull the delivery data before the room opens", "detail": "Half these cards have evidence in the tracker, so bring it. For the month just gone: throughput, what the p85 forecast said against what landed, the quarter's Bug Budget epic burn as raised against closed, the Tech Debt epic burn, how many epics bounced back out of Ready for Dev for a missing approval or a missing approved story, and the outcome validation result for every live outcome. One page, circulated the day before so nobody meets it for the first time in the room. The scores stay subjective and should. The data stops predictability and quality being scored on last Tuesday." }, { "name": "Score privately, then reveal at once", "detail": "Everyone scores all ten cards alone before any discussion: a form, a spreadsheet column, sticky notes face down. Reveal the lot together. If the delivery lead or the engineering manager scores out loud first, every other score drifts towards theirs and you have spent forty-five minutes measuring the most senior person in the room. Testers, designers, contractors and the product manager all score, because they hit different failure modes. Anyone on the team less than two weeks abstains and says so out loud rather than guessing, which tells you something about onboarding on its own." }, { "name": "Score two things per card: state and direction", "detail": "Every card gets a colour and an arrow. Green is good, amber is workable with friction, red is hurting us. Record 2, 1 and 0 beside the colours. Take the arrow from the team median against last month's median, then let the team overturn it out loud with a reason. The pair carries more than either half on its own: amber-improving is a team already fixing something and needs no intervention, green-worsening is the card worth an hour while it still looks fine. Same sheet every month, so twelve months fit on one chart at year end." }, { "name": "Discuss divergence and movement, nothing else", "detail": "Ten cards will not fit in forty-five minutes, so do not try. Five minutes on last month's actions, five on the reveal, thirty on six cards, five agreeing new ones. Pick the six: the three with the widest spread across the team, then the three that moved a step since last month. Spread first, because when half the team scores green and half red on the same statement, the gap is the finding, and it usually means two groups are living in different parts of the system. Ask what someone specifically saw that made them score that. Five minutes a card, facilitator cutting it. Everything else is logged, not debated." }, { "name": "Leave with two actions, each a real ticket", "detail": "Two actions that land beat ten that do not. Each needs a named owner, a date inside the next month, and a ticket in the same tracker as everything else: tech debt goes on the quarter's Tech Debt epic, anything that changes the product becomes a story or an outcome ticket under a real epic, anything the team cannot fix itself becomes a RAID entry with an owner outside the team. Nothing lives only in the health check document. Open last month's two at the start of the next session and mark each done or not done, out loud, before anyone proposes a new one." }, { "name": "Publish the card unedited the same day", "detail": "The card is read by the team and by management, and it is worth nothing if the second audience gets a softened version. Send the scores, the arrows, the two actions and the list of things the team cannot fix alone, in full, on the day. Agree the protective rule in writing before the first session and hold management to it: the card exists to remove obstacles, it never enters an individual's performance review or a league table of teams, and nobody outside the team changes a number. The first time a red quietly becomes an amber before it reaches a director, every score after it is decoration." }, { "name": "Read the trend, act on three in a row", "detail": "One month is noise: somebody had a bad sprint, a release went sideways, half the team was on leave. At quarterly roadmap close, put the quarter's three cards side by side with the Tech Debt and Bug Budget epics you are closing out and read them together. Hold this rule: a card red or worsening three months running is not a retro item, it is structural, and it gets an owner at management level plus a named change in next quarter's roadmap. Re-word cards once a year at most, or when a team changes delivery mode, and mark the break on the chart so nobody compares across it." } ], "pitfalls": [ "The manager scores in the room, or scores at all on a team of six. Anchoring is fast and quiet: two people watch the boss put green on predictability and their own amber starts to feel unfair. Three months later every card is green and the instrument confirms what you already believed.", "Scoring predictability and quality with no numbers in front of you. Feelings track the last two days, so a team that missed its p85 forecast by three epics scores green because this week went well. Put throughput, forecast against actual, and bug budget burn on the table first, then let people score against reality.", "Reacting to one red month with a reorg, a new process and a working group, all landing the month the score would have recovered on its own. The rule is three consecutive months red or worsening, and holding that line when a director wants visible action is most of the discipline.", "Actions that live only in the health check document. If it is not a ticket with an owner and a date in the normal tracker, it competes with committed epic work and loses every time, the same card scores the same next month, and the team learns the ritual changes nothing.", "Averaging the cards into an organisation-wide health percentage for a board slide. It destroys the two properties that make the instrument useful, per team and per card, and it rewards generous scoring. Report the cards as cards, and report which ones have been red three months running." ], "definitionOfDone": [ "Every person on the team scored all ten cards privately, and the scores were revealed simultaneously.", "Each card carries a colour and a direction arrow, recorded as 2, 1 or 0 beside the previous two months.", "Last month's actions were opened and marked done or not done before any new action was agreed.", "At most two new actions exist, each with a named owner, a date inside the month and a ticket in the normal tracker.", "The unedited card reached management the same day, including everything the team cannot fix alone.", "Any card red or worsening for three consecutive months has a named owner outside the team and a named change in the next roadmap." ], "example": { "heading": "Three months of cards at an invented retailer", "body": "Take an invented mid-market retailer, call it Retailer A, and its nine-person payments team running AI-augmented. The live outcome is reducing failed-payment churn, target £1.4m, with five epics planned at £1.2m combined, so there is a visible £200k gap. Month one: line of sight comes back red, four of nine people scoring zero, and all four are developers. Predictability is amber-worsening, because the p85 forecast said six epics would land by this point in the quarter and four did. Quality is amber-worsening: 61 bugs raised against a 40-bug budget. Safety is green-flat. The widest spread is on work arriving ready, product green and engineering red, and the discussion surfaces that seven of eleven epics left Ready for Dev without an approved story attached, so developers were starting on faith. Two actions: the product manager puts the outcome name and its currency share at the top of every epic, and the engineering manager blocks the Ready for Dev transition until both approvals and one approved story exist. Both become tickets with dates inside the month. Month two: line of sight amber-improving, one developer still scoring zero. Quality still amber-worsening, 78 raised against 41 closed. Predictability flat. Month three: line of sight green. Quality red, and worsening for the third month running. That trips the rule, so it stops being a retro item: the engineering director owns it, and next quarter's roadmap carries a named change, the bug budget cut to 25 with a defect triage owner, rather than a fourth conversation about testing more." }, "related": [ { "label": "Designing roles, decisions and workflows for an AI-native organisation", "href": "/guides/target-operating-model-for-an-ai-native-organisation" }, { "label": "What an AI-native SDLC changes", "href": "/guides/ai-native-sdlc-and-product-delivery" } ], "faq": [ { "question": "How is this different from a retrospective?", "answer": "Different scope and different audience. Tenhaw installs both the retrospective and the monthly health check alongside client teams, as part of the operating model an Agentic Design Team designs at £35k to £55k a month. The retrospective runs fortnightly, belongs to the team, and works on the last two weeks, asking what happened and what to try next. The health check is monthly, scored, and read by management as well as the team, and it works on the system the team sits inside: line of sight, tech debt, defects, safety, pace. Run both. Fold the health check into the retro and the structural problems get traded away for the nearest process tweak, while management never sees the card." }, { "question": "Should the scores be anonymous?", "answer": "Private until the reveal, not anonymous after it. Anonymous scores kill the only question worth asking, which is what did you specifically see that made you score it that way. Collect scores individually so nobody anchors, reveal them together, then discuss them attributed. If people will not put a red on the board with their name against it, that is your safety card answering itself, and it is a bigger finding than anything else in the session." }, { "question": "What if every card comes back green?", "answer": "Assume a measurement problem before you assume a healthy team. Check three things: whether a manager scored or spoke first, whether the data agrees (forecast against actual, bug budget burn, epics bouncing out of Ready for Dev), and whether the wording is soft enough that agreeing costs nothing. A card everyone can agree with in a bad month is a badly worded card. Fix it at the annual re-word, not mid-year, and mark the break on the chart." }, { "question": "Who runs it and who attends?", "answer": "The delivery lead owns and facilitates it, one team at a time: engineers, testers, designers, the product manager, and any contractor who has been there more than two weeks. Tenhaw supplies that facilitator on client teams at its published senior practitioner day rate of £1,250 excluding VAT, and hands the ritual to a client delivery lead before the engagement ends. Line managers of the people in the room do not attend, and no score is taken from anyone outside the team. If the delivery lead also line-manages half the room, borrow a facilitator from another team and have the lead abstain from scoring, because a score from the person who writes your review is not a score." }, { "question": "What questions should a team health check include?", "answer": "Ten fixed statements covering line of sight to the outcome, value confidence, predictability, flow, work arriving ready, tech debt, defects, tooling, safety and pace. Word each as a statement a person can agree or disagree with, like \"I know which outcome my current work rolls up to and what it is worth\" or \"I can change this codebase without fear\". Then freeze the wording, because month three only means something if it is answering exactly the same question as month one. Ten cards, no additions, re-worded at most once a year with the break marked on the chart so nobody compares across it." }, { "question": "How often should you run a team health check?", "answer": "Monthly, per team, in the same week every month, and alongside the fortnightly retrospective rather than instead of it. Tenhaw keeps that monthly beat with the client teams it works alongside. The session itself is forty-five minutes; the whole ritual costs about two hours a month once you include pulling the delivery data and publishing the card. Run an extra one as a baseline in your first fortnight with a new team, and again when a team changes how it delivers, treating the month after as a fresh start. Any single month is noise. Monthly matters because the instrument reads trends, and the signal only appears when the same cards line up across a quarter." }, { "question": "How should a team health check be scored?", "answer": "Two marks per card, a colour for the current state and an arrow for direction. Green, amber and red are recorded as 2, 1 and 0, and the arrow comes from the team median against last month's median, which the team can overturn out loud with a reason. Everyone scores privately before any discussion, then everything is revealed at once, because if a senior person scores first every other number drifts towards theirs. The pair tells you more than either half alone, since amber-improving is a team already fixing something and green-worsening is the card worth an hour while it still looks fine." }, { "question": "What data should we bring to a team health check?", "answer": "One page of the month's delivery data, circulated the day before so nobody meets it for the first time in the room: throughput, what the forecast said against what actually landed, bugs raised against the bug budget and closed, tech debt burn, how many pieces of work bounced back for missing approvals, and the validation result for every live outcome. The scores stay subjective and should, but half the cards have evidence in the tracker, and without it predictability and quality get scored on how last Tuesday felt. A team that missed its forecast by three epics can still score itself green in a week that happened to go well." }, { "question": "When should leadership act on a bad health check score?", "answer": "On the trend, not the month. One red month is noise: somebody had a bad sprint, a release went sideways, half the team was on leave. The rule worth writing down is three consecutive months red or worsening, at which point the card stops being a retro item and becomes structural, with an owner at management level and a named change in the next quarter's roadmap. Not reacting early takes the same discipline. A reorg, a new process and a working group landing the month a score would have recovered on its own is the overreaction to resist, and holding that line when a director wants visible action is most of the work." }, { "question": "Should team health check results be shared with managers?", "answer": "Yes, unedited and the same day. Management is half the card's audience, and the card exists to surface the obstacles the team cannot remove alone. Tenhaw does the same on client engagements, and reports a month that delivers no measurable value as a failed month. What makes that survivable is a protective rule, agreed in writing before the first session, that keeps the card out of an individual's performance review and out of a league table of teams, and stops anyone outside the team changing a number. Line managers do not sit in the session, but they see everything it produces, in full. The first time a red is quietly softened to amber on its way to a director, every score after it is decoration." }, { "question": "How do you keep a team health check to 45 minutes?", "answer": "Stop trying to discuss all ten cards. Split the forty-five minutes: five on last month's actions, five on the reveal, thirty on six cards, five agreeing the new ones. Pick the six deliberately, taking the three with the widest spread across the team first, then the three that moved a step since last month. Five minutes a card, with the facilitator cutting it off, and everything else logged rather than debated. The cards you do not discuss still carry their colour and their arrow, so the chart stays complete and nothing is lost by leaving them alone this month." }, { "question": "Half the team scored green and half red. What does that mean?", "answer": "The gap is the finding, and it usually means two groups are living in different parts of the system. That is why spread is the first thing you pick for discussion, ahead of the cards that simply look bad, because a card the whole team scores the same way is a shared view, while a split card is something one group can see and the other cannot. Ask what a person specifically saw that made them score it that way, rather than debating the number. In the worked example, product scored work arriving ready green and engineering scored it red, and the discussion surfaced that seven of eleven epics had left Ready for Dev with no approved story attached." }, { "question": "Why leave a health check with only two actions?", "answer": "Because two actions that land beat ten that do not. Each of the two gets a named owner, a date inside the next month and a ticket in the normal tracker: tech debt goes on the quarter's Tech Debt epic, anything that changes the product becomes a story or an outcome ticket under a real epic, and anything the team cannot fix itself becomes a RAID entry owned by somebody outside the team. Nothing lives only in the health check document, because an action that is not in the tracker competes with committed epic work and loses. Open both at the start of the next session and mark them done or not done out loud." }, { "question": "Should new joiners score in a team health check?", "answer": "Anyone who has been on the team less than two weeks abstains, and says so out loud rather than guessing at a system they have not seen yet. Several abstentions after a fortnight tell you something about onboarding before a single card is scored, so treat them as a finding in their own right. Everyone else doing the work scores, engineers, testers, designers and the product manager included, because they each hit different failure modes. With a genuinely new team, run a session in the first fortnight anyway and treat it as the baseline rather than a verdict, then read the real signal from the months that follow." }, { "question": "Can we roll team health scores up into one number for the board?", "answer": "Better not to. Tenhaw reports the cards to client boards, not an index, inside Programme and Delivery Management at £18k to £35k a month. Averaging them into an organisation-wide health percentage destroys the two properties that make the instrument useful, per team and per card, and it quietly rewards generous scoring, so the team marking itself most honestly looks worst. Report the cards as cards. If the board wants one line, give it the list of cards that have been red or worsening for three consecutive months, because those are the ones that already carry an owner at management level and a named change in the next quarter's roadmap. It is a shorter list than a percentage and somebody can actually act on it." }, { "question": "What do you do when the team cannot say what its work is worth?", "answer": "Treat it as a line of sight failure and fix it at the epic rather than with a briefing. That card asks people to agree with a statement like \"I know which outcome my current work rolls up to and what it is worth\", and a red on it is common in a first month. In the worked example, four of nine people scored it zero and all four were developers. The product manager put the outcome name and its share of the currency target at the top of every epic, a small and structural action. The card came back amber and improving the next month, and green in month three." }, { "question": "Which health check cards change for an AI-native team?", "answer": "The instrument stays the same and three cards change wording. Work arriving ready is scored on the outcome ticket instead of an attached story, because stories and chapters do not exist in that mode. Ask whether the ticket carries the outcome's currency share, the key user journeys and the test requirements, or whether you are filling the gaps by guessing. Swap the codebase-fear card for a verification card, \"I can tell whether what the model produced works, not just that it ran\", since that is where AI-native teams fail quietly. Predictability and value confidence stay exactly as written. When a team changes mode, re-word once, mark the break on the chart and treat the next month as a fresh baseline." }, { "question": "How do health check scores feed into quarterly planning?", "answer": "At quarterly roadmap close, put the quarter's three cards side by side and read them next to the Tech Debt and Bug Budget epics you are closing out, because the scores and the burn explain each other. A card that has been red or worsening all three months earns a line in next quarter's plan, a named change owned above the team rather than another retro action. In the worked example, quality was red for the third month running by the end of the quarter, so the plan carried the bug budget cut from 40 to 25 with a named defect triage owner, instead of a fourth conversation about testing more." } ], "aiNote": "The instrument is the same and three cards change wording. On an AI-native team stories and chapters do not exist, so work arriving ready is scored on the outcome ticket instead: does it carry the outcome's currency share, the key user journeys and the test requirements, or are you filling the gaps by guessing? Replace the codebase-fear card with a verification card, \"I can tell whether what the model produced works, not just that it ran\", because that is where AI-native teams fail quietly. Predictability and value confidence stay as they are, since outcomes, epics and the quarterly roadmap do not change with the mode. When a team moves between modes, re-word once, mark the break on the chart, and treat the next month as a fresh baseline rather than a drop." }