Why this matters
A feature that fails silently is difficult to support and difficult to test. Observability makes internal state visible enough to answer what happened, where and to whom.
QA can improve observability by testing diagnostic signals alongside functional behaviour and by asking whether evidence is useful during an incident.
How to do it well
Identify critical questions
Define what the team must know when a flow slows down, fails or produces an unexpected result.
Add correlation
Use request, job and business identifiers that connect logs, traces and user-visible events.
Choose actionable metrics
Measure latency, error rate, throughput, queue depth and business completion where relevant.
Structure logs
Record event name, outcome, context and cause in a searchable format rather than unstructured prose.
Test failure paths
Trigger timeouts, invalid input and dependency failures and confirm that evidence remains complete.
Review alerts
Ensure alerts represent user impact, have owners and contain enough context for the first investigation.
What to avoid
- Logging everything without structure or retention strategy.
- Including passwords, tokens or personal data in diagnostics.
- Alerting on every technical fluctuation regardless of user impact.
- Using different identifiers across services so traces cannot be joined.
- Testing happy paths while leaving failure evidence unverified.
Practical example
A delayed campaign job should expose queue time, processing time, provider response and a correlation ID visible in support tooling.
Instead of “export failed”, a useful event distinguishes validation failure, authentication, provider throttling and internal transformation errors.
Lesson for practice
Observability is part of product quality because it shortens detection and diagnosis. QA adds value by verifying not only that a failure occurs correctly, but that the system explains it safely and usefully.
A good practice does not have to be complicated. It should be intentional, repeatable and explainable: the team should understand why the control exists, what evidence it provides and how feedback will improve the next iteration.
A real case: the Cloudflare outage of 21 June 2022
Cloudflare’s own incident report describes an outage following a network configuration change that affected nineteen data centres. The event began at 06:27 UTC on 21 June 2022, and the company reported all affected locations restored by 07:42 UTC. This is a supplier’s post-incident account rather than an independent audit, but it documents a concrete situation in which a change affected service and recovery conditions together.
My interpretation is that dashboards, application logs and remote administrative access can depend on the same infrastructure that has just failed. Testing whether an application creates an error message therefore misses a second question: can the responsible person actually obtain the evidence during a real outage? I am not claiming that Cloudflare lacked a particular monitoring product. The case motivates testing the whole chain from a signal to a useful intervention.
Hypothetical example: a stalled credit-document import
Imagine a system receiving a batch of 10,000 documents. The accepted-record counter rises, but processing stops after 3,000. There is no technical exception and users can still sign in. A simple availability check remains green while customers wait for results that never arrive. These fictional numbers describe neither a bank incident nor the Cloudflare event; they illustrate a common distinction between accessibility and completion.
For that process I would consider the age of the oldest unprocessed item, counts of accepted and completed records, and the time of the last genuine progress. Together these distinguish slow processing of a large batch from a stalled process. I would not choose thresholds arbitrarily. They should reflect the expected service, normal volumes and the time an operational team needs to intervene before the business impact becomes unacceptable.
Testing the chain from failure to response
I would agree a safe environment and a precise scenario with operations. A controlled timeout in a downstream service would create a known failure and an expected signal. I would check whether records can be correlated with the affected batch using a safe identifier, rather than exposing customer information in logs. Diagnostic evidence should support investigation while maintaining appropriate information protection and access restrictions.
Next I would determine whether the signal reaches the right role and explains what requires attention. “Service error” is less useful than a clear indication that a particular process has stopped progressing and what outcome is at risk. The test should cover acknowledgement, access to a response procedure and verification that the intervention restores processing. I would retain timestamps, not just a passed or failed test result.
Negative scenarios for the monitoring itself
I would also test loss of log collection, failure of the monitoring connection and inconsistent clocks. Missing telemetry must not silently become evidence that everything is healthy. If a process normally emits a status every minute, the absence of a status is itself meaningful. Another scenario would check whether a stream of minor alerts prevents the team from noticing a single critical warning that needs immediate action.
For a distributed process I would distinguish technical acceptance from business completion. An API response may confirm only that a request entered a queue, not that a final record was written correctly. I would therefore verify the outcome from the user’s perspective before examining intermediate events. The same distinction applies to banking reporting: successful message transport does not prove that the resulting report contains the correct information.
Correlation also needs boundaries. A batch identifier can connect processing steps, but it should not become a permanent unrestricted record of every customer action. I would clarify retention, access and the minimum diagnostic detail required with the relevant owners. This prevents a reliability improvement from unintentionally creating excessive exposure of sensitive information or making troubleshooting dependent on permissions only one individual holds.
Evaluation and handover
The output would be a simple map: which failure we simulated, what evidence appeared, who received it and what action followed. A missing link becomes a specific improvement task. Not every gap requires another tool. Some require a connection to an existing operational process, clearer ownership or an independent signal. I would select technology only after agreeing the question that the team needs the evidence to answer.
Summary and recommendations
Observability means being able to understand a service’s actual behaviour from usable evidence. Test detection, delivery, acknowledgement and confirmed recovery as a connected chain. Track business completion alongside technical availability, treat missing telemetry as a distinct state and exercise the response under realistic constraints. A good control helps a person make the right intervention precisely when the system is no longer behaving normally, without demanding that they first reconstruct the architecture from scattered logs.