How to run live monitoring after a release
The first seven days post-release are when reality contradicts the staging environment. Live monitoring is the structured check that catches it.
- steps
- 8
- named failure modes
- 5
- definition-of-done criteria
- 5
How to run live monitoring after a release, in one paragraph
Live monitoring is the structured watch you keep on a shipped change for its first days in production, and it ends in a recorded decision rather than a feeling. It opens when the release goes out, runs against the journeys the ticket named, and produces four things: a customer impact assessment, a published FAQ entry, a support macro, and a continue, watch or rollback call. Done well it catches the ways production contradicts staging while a rollback is still cheap, and it hands a clean epic to value monitoring.
What these are. The delivery operating model our engagements install alongside client teams: the method underneath the agentic work rather than the agentic work itself, published in full and free to use. It is written for the person running a quarter, not for a buyer, so if you are evaluating us, read the five priced engagements or the case studies instead.
Every time a change reaches real users, including the small ones nobody is worried about. It opens at the moment of release and runs until the window closes, and the parent epic cannot enter value monitoring until it does.
- Time to run it once
- Seven days, the default window
- What you need
- The dashboards and saved queries named in the ship kit
- A real production account for walking the journeys
- The devices your users actually have
Step by step
Each step is deep-linkable, so you can send a colleague the one that is in dispute.
- 1
Name the owner and window before you ship
Plan the watch in the ship kit, not after the deploy.
When you schedule the release, write into the ticket who owns it, who deputises on that person's days off, when the window opens and when it closes. Seven days is the default: long enough to cover a full weekly cycle including the weekend, short enough that nobody forgets it is open. Extend it only for a stated reason, such as a change touching monthly billing that needs the window to reach the next run. If the release lands on a Friday, move it or accept that you have chosen to staff the weekend.
- 2
Write the baseline and the query before release
Before the deploy, record the numbers you expect afterwards with today's baseline beside each: error rate on the affected endpoints, p95 latency, completion rate for every journey the ticket names, support contacts on the tag, and any business counter the epic is meant to move. Take the baseline from the same weekday and hour you will compare against, so Tuesday lunchtime is judged against Tuesday lunchtime. Paste the saved query or dashboard link beside each number, so the person on the four-hour check is not inventing one. Without a written prediction you will talk yourself into calling a regression seasonal variation.
- 3
Agree the thresholds while nothing is on fire
Write the number that triggers each of the three calls into the ticket before release.
For example: roll back if the checkout error rate stays above 1% for two hours, or if any journey named in the ticket fails outright for more than 10% of sessions. Watch if a metric moves outside the normal range but impact is bounded and a fix is in reach the same day. Continue otherwise. Name the one person who makes the call, name who can wake them, and state plainly that they can roll back without convening anyone. A threshold agreed at 11am on a calm Tuesday beats a debate at 2am.
- 4
Walk the journeys yourself, on production
Dashboards tell you what broke, not that the thing works.
Within the first hour, walk every key user journey the ticket names, on production, on the devices your users have, with a real account rather than a seeded one. Walk the failure paths too: declined card, expired session, empty state. Use a fresh account and a cold cache, because your browser is holding the flags and cookies that hide the fault. Screenshot each pass onto the ticket. This catches the misconfigured flag, the missing environment variable and the CDN still serving last week. A green deploy and a model reporting success are evidence about the build, not the live system.
- 5
Check on a fixed rhythm, not on anxiety
Set the cadence and hold it: one hour, four hours, 24 hours, then once a day until the window closes.
Put the checklist in the ticket so the deputy runs the same one: the numbers against your written baseline, new error signatures rather than error volume alone, support contacts on the tag, and anything monitoring surfaced automatically. One new signature at low volume matters more than a familiar one at high volume. Post the result in the same channel every time, including when everything is flat, because a flat check posted publicly is what tells everyone else the change is holding. Each check should take minutes.
- 6
Publish the FAQ entry and macro early
The first time a customer hits the problem, someone in support writes an answer under time pressure.
Write that answer once, correctly, and early. For each issue found, record who is affected and how many in numbers not adjectives: roughly 3% of sessions, about 40 orders a day. Publish the FAQ entry in the words a customer would search for, not the words the ticket uses. Give support a macro that names the workaround and says whether a fix is coming, then have them send it live once and confirm it closed the contact without escalating. The cost of an unsupported release is paid by second line.
- 7
Make the call and record it
At the 24-hour check, make the call explicitly, even when the answer looks obvious, and record the decision, the time, the person and the numbers it rested on. Continue means the window keeps running with no action. Watch means a named fix, an owner, a date and a re-check written into the ticket. Rollback means the change comes out now and the reasoning is written down before anyone starts on the fix. Weigh it in currency: a day of bounded impact costed against the planned value of the epic you would pull. Never let a release drift through the window undecided. An undecided release is one nobody owns.
- 8
Bucket what you found, then close the window
Nothing found in live monitoring is allowed to float.
A defect in shipped behaviour is a bug: raise it and link it to the quarter's Bug Budget epic, including the one you already hot-fixed. A shortcut taken to stabilise the release is tech debt, linked to the quarter's Tech Debt epic. Something the change was meant to do and never did is a missed requirement, so it is a story or an outcome ticket, not a bug. Close the window with a short note covering what happened and what you changed, then move the parent epic into value monitoring, where the currency gets tracked month by month.
Seven days on a guest checkout release
Northbank Home is an invented retailer and every number below is invented with it. The outcome: lift checkout completion by 1.2%, worth £1.4m a year. Under it sits a guest checkout epic carrying a planned £420k share. Baseline written before release: checkout error rate 0.4%, p95 checkout latency 910ms, completion 61.5%, twelve support contacts a day on the checkout tag. Threshold agreed the previous week: roll back if the error rate stays above 1% for two hours, or if any named journey fails for more than 10% of sessions. The epic ships Tuesday at 13:00. One-hour check: all four named journeys walked on production on an iPhone and an Android handset with a real account, and both guest and saved-card checkout complete. Error rate is 1.9%, concentrated in one browser on the address-lookup fallback, about 3% of sessions. The rule needs two hours, so the 14:20 entry records watch with a re-check booked for 16:20. At 16:20 it is still 1.7%, so the rule has fired. Rather than pull the epic, they flip the address-lookup flag back to the previous provider at 16:35 and record it as a partial rollback with the numbers behind it. Error rate is 0.6% by 17:00. The impact was costed at roughly 40 orders a day at £62 average order value, about £2,500 a day, which framed how fast to fix it rather than whether the rule applied. FAQ entry and support macro published Tuesday afternoon; support sent the macro nine times that week with no escalations. Fix ships Wednesday, the flag goes back on Thursday morning, error rate settles at 0.5% and completion reaches 62.4% by Friday. Two bugs raised against the quarter's Bug Budget epic, one tech debt item raised for the fallback. The following Tuesday the window closes with a four-line note, and the epic moves into value monitoring, where its £420k share is tracked month by month against the outcome.
Where this goes wrong
- 01
Treating silence as success.
No alerts often means no instrumentation on the new path, and a journey that fails before it emits anything looks identical to a journey nobody used. Confirm the new code is producing signal at all before concluding it is producing good signal.
- 02
Watching the aggregate and missing the segment.
A fault confined to one browser, one region or one enterprise tenant disappears into a 99.6% overall success rate. Split every headline number by platform and by customer segment at least once in the first four hours.
- 03
Leaving the rollback threshold to be agreed during the incident.
Under pressure the person with the most to lose from a rollback is usually the loudest voice in the room, and the number quietly moves to fit the argument. Write it down while nothing is on fire.
- 04
Fixing forward off the ticket.
A hot fix pushed without raising the bug keeps the board tidy and destroys the Bug Budget burn rate, which is the only signal you have about whether quality is improving or rotting.
- 05
Running live monitoring and value monitoring as one activity.
Live monitoring asks whether the change is safe and does what it said, over days, at the level of what shipped. Value monitoring asks whether the epic produced its currency share, over months. Merging them gets an epic called delivered on the strength of a quiet week.
Done means
- Every key user journey named in the shipped ticket has been walked on production by a person, with the screenshots stored against the ticket.
- A continue, watch or rollback decision is recorded with its timestamp, its owner and the numbers it rested on.
- Every defect found in the window is raised and linked to the quarter's Bug Budget epic, and every stabilising shortcut is linked to the Tech Debt epic.
- The customer FAQ entry and the support macro are published, and support has answered a real contact with them without escalating.
- The window is closed with a written note, and the parent epic has either entered value monitoring or been held with a stated reason and a date.
In an AI-native team the outcome ticket already names the key user journeys and the test requirements, so live monitoring is a re-run of the same done bar on production: tests pass and journeys demonstrated working. Let an agent run the fixed-rhythm checks, diff the numbers against the written baseline, cluster new error signatures and draft the FAQ entry and macro. Keep two things with a person: the journey walk on production with a real account, and the continue, watch or rollback call. A model reporting that the release is healthy is evidence about the build, not about the live system.
The two delivery modes, side by side →Questions
How long should the window be?
Seven days by default, because it covers a full weekly cycle including the weekend and is short enough that people remember it is open. Shorten it only for changes with no customer-visible surface, and extend it only with a reason and a new closing date written in the ticket: a billing change needs the window to reach the next run, a seasonal feature needs it to reach the first real peak. A window with no end date is not monitoring, it is a browser tab left open.
Is this the same as being on call?
No, and running them as one job is why post-release problems get missed. On call reacts to things that alert. Live monitoring goes looking for the things that do not: a journey that quietly completes at 54% instead of 61%, a support tag creeping up, a new error signature at ten a day. On call can hold the pager for the release, but somebody has to own the fixed-rhythm checks, the FAQ entry and the recorded decision, and that is a different piece of work.
What if the change is behind a flag or a percentage rollout?
The window opens at first real user exposure, not at deploy, and your thresholds apply to the exposed cohort rather than to total traffic. A 2% error rate inside a 5% rollout is invisible in the overall number and is still the reason to stop. Write the ramp steps into the ticket with the check you run before each one, and treat turning the flag off as the cheap rollback it is, rather than waiting to pull the whole release.
Who owns it, product or engineering?
One named person on the ticket, whichever function they sit in, with a named deputy. In practice the product manager tends to own the customer impact numbers, the FAQ entry and the macro, and the engineer who shipped the change tends to own the error and latency checks. Split the checklist between them if you like, but only one name carries the decision, and that person needs standing authority to roll back without convening a meeting.
More on shipping and supporting
These guides are written to be read in order.
03How to run a root cause analysis
An RCA is not a blame exercise. It is an investigation into the system that allowed the failure, so the next one does not happen the same way.
← All seventeen guidesWant help installing this?
These guides are free and you owe us nothing for using them. If you would rather have operators install the operating model alongside your teams and stay until it sticks, that is what our engagements do.
Most organisations start with a fixed-price Agent-Readiness Audit · £30k–£90k · 6–8 weeks