Restore the service, then explain the mechanism
When a banking service fails, the immediate priority is to contain the impact appropriately. After restoration, a second task remains: explain why the failure happened and what allowed it to spread. Recovery and root-cause analysis are complementary activities with different objectives and time horizons. A responding system does not establish that an ordinary future change cannot trigger the same problem again.
The phrase “root cause” can encourage a search for one final answer. I would usually expect a chain of contributing conditions: a technical trigger, an ineffective control, unclear responsibility and a factor that made recovery harder. Correcting one component may be necessary without providing sufficient assurance. The analysis must distinguish demonstrated mechanisms from plausible explanations that have not yet been tested.
A documented case: RBS, NatWest and Ulster Bank, 2012
On 20 November 2014, the FCA announced enforcement relating to banking IT disruption in June 2012. Its account distinguished a software compatibility problem from inadequate systems and controls for IT risk. This British case provides a concrete separation between technical trigger and wider control environment. The analytical method below is my proposal, not the banks' documented internal procedure. [1]
I would not stop at asking which file changed. I would want evidence about compatibility assessment, recovery planning and the signals that revealed the extent of impact. Public findings establish a starting point, not permission to invent a missing technical narrative. Where the public record does not explain an internal detail, that limit should remain explicit.
A hypothetical incomplete loan export
Imagine a fictional monthly export missing 600 of an expected 30,000 records. The first suggestion is to run it again. If the repeated run contains every record, the output appears restored, but the reason for the first failure remains unknown. Repetition also does not prove that source data and reference-date parameters stayed unchanged between the two attempts.
Before rerunning, I would preserve the original output, parameters and relevant logs. I would compare missing identifiers and look for common attributes: source, time interval, customer category or agreement status. A pattern is an investigative lead, not a conclusion. For each hypothesis I would seek evidence capable of disproving it as well as evidence supporting it.
The distinction matters because a successful rerun can conceal several mechanisms. A late source may have become available; a transient permission issue may have disappeared; an operator may have changed the parameters. Each possibility implies a different corrective action. Treating all of them as “temporary error” would leave the team without a defensible prevention plan.
Construct an evidence-based timeline
I would include source-data availability, job initiation, intermediate checkpoints and final file generation. Actual timestamps, time zones and delays between systems matter. A “completed” event may refer to sending data rather than confirming receipt. If teams use the same event label differently, responsibility can appear to move between systems without a shared understanding of what happened.
Every important statement should have supporting evidence. A log is generally stronger than memory for an operation's timestamp, but logs also require interpretation. They may be incomplete or record something different from what the reader assumes. An absent event can mean the operation did not occur, logging was disabled or the logging pipeline failed. Those possibilities should remain separate until investigated.
Try to falsify the first explanation
Suppose the hypothesis is that the export began before the source batch finished. I would compare event ordering and inspect other runs. In a safe environment, I would contrast processing with complete inputs against processing with an intentionally delayed source. It is not necessary to recreate an uncontrolled production incident to test the dependency logic. The test itself needs a clear scope and safe conditions.
If the hypothesis survives, I would still ask why the dependency was neither enforced nor detected. A timetable can work at normal volumes while failing after growth. Adding fifteen minutes to the schedule may help temporarily without creating a robust solution. A time allowance and a verified completion condition are different protections, and the decision record should make that difference clear.
Alternative explanations deserve attention too. I would check whether the missing population shares a filter condition or whether the source counts themselves use the same reference date. Finding one believable story is not enough if another untested mechanism could produce the same symptom. The analysis should explain why competing explanations became less likely.
Match remediation to the failure mechanism
For the fictional export, I would separate correcting today's file, checking input availability and redesigning the dependency. Each action addresses a different objective. Closing the whole incident after restoring the missing records can leave recurrence unaddressed. Waiting solely for a large architectural project can leave the next monthly run exposed in the meantime.
I would verify the controls under unusual conditions as well as normal ones. What happens if the source batch fails rather than finishes late? Does the export stop with an intelligible explanation, or create an incomplete file that resembles a normal result? Safe failure and clear escalation may be more important than forcing every job to finish and display a green status.
Make the conclusion useful to others
The final account would describe the established timeline, mechanism, impact scope, residual uncertainty and owned corrective actions. Important claims should be identified as demonstrated or provisional. The team also needs to know what was outside the review. Explaining one export's cause must not inadvertently imply that all other data journeys have been validated.
I would not accept “operator error” as the final analytical answer. I would examine the conditions that permitted an incorrect action and the protections that could have limited its effect. This does not remove accountability. It allows accountability to be attached more precisely to design, authorisation, execution and review rather than an undifferentiated search for blame.
Summary and recommendations
A useful root-cause analysis explains both what failed and why the control environment did not detect or contain it. I recommend preserving evidence before repeating operations, building a timeline and deliberately testing hypotheses. Corrective actions should address the demonstrated mechanism and be verified under both ordinary and exceptional conditions.
Google SRE's material on incident reviews provides further reading. [2] My contribution would connect technical detail with process impact and clear documentation. The purpose is not to remove uncertainty through confident wording. It is to help the team decide what to do immediately, what to change structurally and how to establish later that the correction worked.