Load Testing a Payroll System at 10,000 Registries an Hour

Denis Sautin

Author

Denis Sautin

Denis Sautin is an experienced Product Marketing Specialist at PFLB. He focuses on understanding customer needs to ensure PFLB’s offerings resonate with you. Denis closely collaborates with product, engineering, and sales teams to provide you with the best experience through content, our solutions, and your personal journey on our website.

Product Marketing Specialist

Reviewed by Boris Seleznev

Reviewed by

Boris Seleznev

Boris Seleznev is a seasoned performance engineer with over 10 years of experience in the field. Throughout his career, he has successfully delivered more than 200 load testing projects, both as an engineer and in managerial roles. Currently, Boris serves as the Professional Services Director at PFLB, where he leads a team of 150 skilled performance engineers.

Why a payroll system is unusually hard to load test

Money moving from a company to an employee's account is not one transaction. It is a registry that travels through several processing stages, arrives through different integration channels, and is checked at every step. Along the way a registry can hold any of twelve statuses defined by the business process model.

That status model is the difficulty. A load test that only measures throughput will happily report success while registries silently land in the wrong state - and in a payroll system the wrong state means someone is not paid. So the testing had to follow status transitions for every registry, not just count completions.

The bank's goal was accordingly specific: eliminate the risk of payroll card service disruption, payroll accounting delays and failures during registry processing.

What we built

Off-the-shelf load generation could not cover a system this integrated, so a large part of the engagement was engineering:

The payroll system model: registries moving through their processing stages.

• 29 emulators reproducing the behaviour of seven adjacent bank systems, so the payroll platform could be loaded realistically without those systems being available
• 43 test registry generators in Python, producing the different registry types the system receives from different sources
• Additional load generated through Java and BPEL alongside JMeter and LoadRunner

How the testing ran

Each release carried at least three functional changes, and each release was tested against the previous one so the effect of every change was visible. Changes that failed their check did not ship - the comparison was a gate, not a report.

The test scope per release covered more than 60 business cases: receiving and processing registries from different channels and sources, generating reports, working with agreements, activating cards. Sixteen interface operations known to be resource-hungry - activation, obtaining document and task lists - were re-tested every time.

Two things were measured that a generic load test would miss: the processing time of a large registry (up to 9,000 records) and the average registry processing time under a load of 10,000 registries per hour. Throughout, the team tracked business-process status transitions for every registry and reported status defects as first-class findings.

What the testing delivers

The outcome here is prevention, and it is worth stating plainly rather than dressing as a percentage:

• Registry status defects were detected by tracking every registry's business-process status transitions through the run.
• DBMS behaviour was tracked with Oracle Automatic Workload Repository, with monitoring on Telegraf, InfluxDB and Grafana plus purpose-written Python and Go scripts.
• Releases carrying performance degradation are caught before production. For a payroll system that is the whole point: the failure mode is people not being paid on time, and the cost is financial and reputational at once.
• PFLB has run load testing for financial systems for more than five years, and this is the class of failure that work exists to prevent.

Questions this engagement answers

How do you load test a system where correctness matters as much as speed?

By tracking the business-process state of every object under load - here, up to twelve statuses per registry - and treating a wrong status as a defect equal to a slow response.

What if the surrounding systems aren't available for testing?

You build them. This engagement required 29 emulators covering seven adjacent bank systems before realistic load was possible at all.

How much test data does a payroll load test need?

Enough variety to match every registry type and channel - here 43 separate generators, because a single synthetic registry shape would have tested one path and missed the rest.

Can performance testing gate a release?

It did here: each release was compared against the previous one, and changes that failed the comparison were excluded from the release.