Improvement appears in the next result
After a difficult release it is easy to list things that should be done better. It is harder to choose a change that the team will actually make and whose effect it can evaluate. I regard a quality retrospective as the transition from experience to an experiment. It is neither a competition to produce the longest action list nor a search for one person to blame for a complex outcome.
I would start by naming the problem we want to reduce. A late report, repeatedly ambiguous requirements and slow service recovery require different changes. If the team combines them into “improve communication”, it has no testable conclusion. A useful proposal describes a different behaviour, the circumstances in which it applies, its owner and the point at which we will evaluate whether it helped.
A documented case: Cloudflare, global service, July 2019
In its account of the 2 July 2019 outage, Cloudflare described processor exhaustion following a web application firewall rule change. The explanation went beyond the expression itself to consider testing and deployment practices. This is the operator's primary account; the retrospective proposals below are my analysis, not claims about its subsequent internal results. [1]
What interests me is how a seemingly small change can have a large operational reach. I would therefore examine more than the correctness of one rule. I would ask why the effect could not first appear in a smaller part of production and why existing verification did not distinguish dangerous resource consumption. An improvement needs to address the failure mechanism rather than one particular file.
A hypothetical reporting team
Imagine a team that corrected twenty banking export defects during a quarter. Each correction had a separate ticket. Grouping the evidence reveals that eleven involved different interpretations of the reference date. That does not prove a common technical cause. It does provide a reason to investigate a shared ambiguity before accepting another sequence of isolated corrections.
I would propose an experiment: for the next four export changes, agree an example covering the boundary day before development begins. If one process uses trade date and another uses booking date, that distinction should appear in the requirement. I would assess success through fewer late surprises and actual use of the agreed definition, rather than the number of completed template fields.
The experiment would also record its cost. If preparing examples takes hours but prevents days of rework, that trade-off is different from an elaborate ceremony that nobody uses. A small sample cannot prove a permanent causal effect, but it can inform the next practical decision. I would keep that uncertainty visible rather than present four changes as statistically decisive evidence.
Establish the shared facts
I would bring a short timeline and selected records to the meeting. The discussion should distinguish log evidence, documents, participants' recollections and untested explanations. Memory after an exhausting incident is not sufficient proof of exact sequencing. Differences between recollections need not indicate poor cooperation; they can reveal where the team lacked a shared operational picture.
I would also identify actions that limited the impact. A retrospective focused exclusively on mistakes can overlook an effective control or discard a useful improvisation. We need to know what to preserve as well as what to change. Each conclusion should connect to a particular situation rather than become a general judgement about the team's competence or effort.
Choose a specific intervention
When selecting an action, I would compare its likely benefit, cost and testability. A new platform is not always the quickest improvement. Changing one decision point, testing a sensitive assumption or arranging specialist availability may be more effective. Conversely, repeated material failures can require architectural change. Starting with a small experiment should not become an excuse to accept a permanently inadequate solution.
I would describe actions as observable behaviour. Instead of “hand changes over better”, an example might be: compare agreed sample outputs from both systems before deployment and retain the approved reference-date definition. This is my hypothetical proposal, not a universal standard. Its value is that someone can tell who should do what differently and whether the change occurred.
The team should also be able to reject a proposed action with reasons. Mandatory additions accumulate quickly when every incident creates another checklist item. If no one can explain which failure a control addresses, it may add delay without adding protection. A retrospective can simplify a process as well as expand it, provided that the reasoning is explicit.
Evaluate without false precision
I would interpret defect counts alongside the volume and complexity of changes. Fewer defects can reflect a better process, fewer releases or less effective detection. For the fictional export experiment, I would record ambiguities found before development and late returns as well as defects. Context reduces the temptation to credit an action merely because an improvement followed it in time.
A review date needs to be agreed before the meeting ends. Otherwise the organisation often learns only that the ticket was closed. At review, I would ask whether people used the action, whether it introduced unnecessary work and whether the expected outcome appeared. A negative result remains useful if it changes the hypothesis instead of prompting the team to defend the intervention at any cost.
Transfer the learning carefully
Another team needs enough information to recognise when a lesson applies. The full meeting transcript is rarely the best format. I would prefer a short statement of the mechanism, relevant conditions, intervention and evidence of its effect. Transfer still requires testing: two systems can use the same label for a problem while having very different dependencies and failure behaviour.
I would retain a factual, respectful tone. Knowing the eventual outcome makes it easy to say that a risk should have been obvious. The practical goal is to improve decisions made with the information available at the time. If people cannot describe uncertainty honestly, the retrospective loses precisely the material from which meaningful improvement can emerge.
Summary and recommendations
A useful retrospective changes a specific behaviour and returns to assess its effect. I recommend separating facts from interpretations, limiting the action list and assigning an owner, review date and evaluation method to every selected action. The process need not be elaborate, but it must be clear enough to move beyond a constructive conversation.
Google SRE's material on learning from incidents offers further reading. [2] My emphasis would remain practical: preserve what worked, alter the weak point and revisit the result when agreed. Whether the subject is a banking report or an operational service, improvement becomes visible when the next change proceeds with clearer responsibilities and better-controlled risk.