I would begin performance testing before launch with a business question: which activity must the service complete, and what happens if it cannot finish on time? The number of virtual users does not answer that question by itself. Two applications with the same number of signed-in people can process very different workloads. Operation mix, arrival rate, data size and concurrency all matter. The purpose of the test is evidence for a readiness decision, including explicit limitations. A favourable chart from a brief demonstration is useful only if its conditions resemble the service we intend to provide.
A documented case: TSB migration in the United Kingdom
On 20 December 2022 the FCA and PRA announced penalties against TSB concerning operational risk management during an IT platform change. The FCA account describes technical problems and disruption to banking services in the United Kingdom following the April 2018 migration. I do not present this as proof of one particular performance-testing defect. The broader lesson for my proposed approach is that successful data transfer and readiness of the complete service are different questions.
I would translate that distinction into test scope. Alongside individual application responses, I would measure completion of an entire business activity, such as submitting an application and receiving its processed outcome. A quick response that merely places work in a queue can look reassuring while actual completion is delayed. I would distinguish acceptance, technical processing and the point at which the user has a usable result. Different timings support different conclusions. The readiness report should make the distinction visible so that a successful interface response is not accidentally presented as successful end-to-end service delivery.
Constructing a meaningful workload
I would describe an ordinary day, the expected peak and a relevant exceptional overlap. Assumptions might come from available operational records or a clearly labelled estimate for a new service. An estimate needs uncertainty and alternative variants. The scenario should include different file sizes, user pacing and activities competing for shared resources. Having every virtual user access one tiny record could mainly measure caching rather than the intended production workload. I would also include the initial state: an empty queue and a warm cache may produce a different result from the opening minutes after deployment.
The load generator itself needs validation. If its processor or network becomes constrained, the result can be mistaken for a limit of the application. I would compare planned, sent, accepted and completed requests. Differences help locate the boundary at which work is being lost or delayed. I would retain scenario settings, application version and relevant configuration. Without that record, a later measurement may be unsuitable for comparison despite carrying the same scenario name. Reproducibility is especially important when a proposed improvement changes both the software and the environment at the same time.
A hypothetical import campaign
Imagine an application expected to accept 500 files during one hour while ordinary users continue editing. For this example, set a target of completing at least 95 per cent of edits within two seconds and processing all accepted files within thirty minutes after the arrival peak ends. These are hypothetical requirements, not universal thresholds. I would vary file sizes and include reporting in the workload. An average response time cannot establish success because it can conceal a smaller group experiencing very long waits. The scenario should retain enough detail to identify which activities are affected.
Suppose files are accepted quickly but the processing queue keeps growing and takes another two hours to clear. The input interface was available, yet the stated business objective was missed. I would investigate the processing chain, perhaps database waiting or a constrained dependency, using measurements that support the hypothesis. Simply increasing worker count could overload a shared resource. After a change I would repeat the same baseline workload with comparable data. The comparison should show both completion times and the effect on concurrent editing, rather than celebrate a faster import while overlooking a slower customer operation.
Duration and recovery
A short run can explore immediate capacity without revealing gradual resource exhaustion. Depending on the service, I would therefore propose a longer steady-load run. Memory, open connections, queue size and response-time trends should be inspected through the run, not only as overall averages. I would also observe the period after load generation stops. A service that remains busy for hours finishing accumulated work presents a different readiness question from one that recovers promptly. Recovery behaviour helps determine how the next peak might interact with unfinished processing from the previous one.
Where the environment permits it, I would test a controlled dependency slowdown. This is a prepared experiment with an explicit boundary and stop condition. I want to understand waiting, retries and rejection of new work. In the hypothetical business process, a repeated submission should not create duplicate outcomes merely because the first response arrived late. Performance therefore connects to correctness and recovery. A fast response to an incorrect transaction is not a successful readiness result. I would make the expected outcome and the evidence needed to verify it part of the scenario definition.
Making the result usable
The report should state targets, executed scenarios, environmental differences and interpretation limits. If the test database is smaller than planned production, I would explain how that restricts the conclusion. I would not multiply measured capacity mechanically by a processor ratio without evidence that the workload scales that way. Readers need to see covered conditions and unresolved questions. A decision may accept uncertainty, but it should connect that uncertainty to an explicit operational measure or further check. This makes the evidence usable without giving a reassuring number more precision than the experiment supports.
Each finding needs an owner and a verification step. Changing a query, adjusting a limit and redesigning processing can have different side effects. After a fix I would examine activities sharing the same resources, not only the operation that was previously slow. Faster imports might delay editing. The final comparison should therefore return to the business objectives. Success should be visible in usable outcomes and completed work. If a remaining constraint is accepted, its trigger, monitoring and response need to be understood by the people who will operate the service after launch.
Summary and recommendations
I recommend starting with measurable service objectives, realistic concurrency and monitoring across the complete workflow. Separate acceptance from completion, validate the generator and preserve reproducible conditions. Longer runs and recovery observation address questions that a short peak cannot answer. The TSB case illustrates the importance of overall operational readiness; it does not replace evidence about another system. I would support the launch decision with a clear account of what passed, what failed, what changed and which uncertainty remains after verification. That gives the decision-maker a practical basis for acting on the test.