At a glance
- Client: a large governmental enterprise that processes a huge volume of documents of many different kinds
- The requirement: organise autonomous testing of the key subsystems that make up the automated information system - each one testable without the others being present
- The constraint: an out-of-the-box solution was ruled out by the client's own specification, so a custom tool had to be built
- The architecture under test: a transport subsystem (A) connecting everything, and subsystems B, C, D … n, each responsible for processing a specific data set, all communicating over REST
- What was built: a subsystem emulator written in Java, with a native graphical interface and a terminal run option
- Three switchable modules: the logger, the emulator and the retranslator, each independently enabled or disabled from the interface
- Configurable behaviour: response time can be set, an answer can be forbidden, and a custom answer can be returned - which is what makes it a quality-evaluation instrument rather than a stub
- Delivered with the tool: a user manual written for the client during the project
Microservices make the dependencies the problem
Microservice architecture keeps gaining ground for good reasons: it speeds development up, and it makes the finished product more agile and easier to scale. It also moves the difficulty. When a system is a single application you test the application; when it is a set of subsystems that talk to each other, testing any one of them properly means having all the others available, healthy and behaving the way you need them to for the case in hand.
A large governmental enterprise that processes a huge amount of varied documentation came to PFLB with exactly that problem. The task was to organise autonomous testing of the key subsystems comprising their automated information system - and the client's specification ruled out using an off-the-shelf solution, so the tool that made autonomous testing possible had to be developed to their requirements.
Work started with the specification rather than with code. PFLB studied the system specification and drew up a subsystem cooperation scheme, which is what made it possible to determine the real requirements and formulate the project charter for the emulator that would follow. The picture it produced was straightforward to state and non-trivial to work with: subsystem A is a transport subsystem connecting all the others; subsystems B, C, D and onwards each handle a specific data set; and REST queries carry the communication between them.
Java was chosen as the emulator's language.
What the emulator does
The tool has a user-friendly native interface as well as a terminal run option, so it fits both an interactive tester's workflow and an automated pipeline. From the interface an operator can:
The document flow under test, in 4 stages: generation, submission, processing, and getting the result back.
Subsystem A and the neighbouring systems it exchanges messages with - the dependencies that had to exist before it could be tested.
A request to the neighbouring subsystem and the response the emulator returns in its place, matched on 2 identifiers - MessageID and CorrelationID.
How the emulator sits in the loop: it answers, it logs, and it can replay what it recorded.
- switch the software modules on and off individually - the logger, the emulator and the retranslator;
- configure the emulator's behaviour;
- monitor subsystem load intensity.
Because it is universal rather than tied to one subsystem, it covers three distinct situations:
- when the system calling protocol needs to be analysed;
- when some particular system behaviour has to be partly emulated;
- when a non-existent or broken system has to be stood in for entirely.
Four requirements that shaped the build
Analysing the client's requirements surfaced four problems, and each one produced a specific piece of the tool.
- Subsystem isolation during functional testing. The emulator had to be able to substitute the subsystem actually in use. A log collection algorithm was developed for the subsystem; working from the log analysis, the testing team composes the correct and incorrect answers in advance and feeds them into the emulator. By the time the subsystem under test calls out, the emulator already holds the required answer.
- Supporting other teams working in parallel. Software, administration and monitoring teams were working on the same system concurrently. A retranslation module was added, which forwards queries from one subsystem to another and returns an answer when the question did not come from the same team - so one team's emulator does not block another team's work.
- Configurability, so the emulator can behave badly on purpose. The emulator can be configured to set the response time, forbid an answer, or return a custom answer. That flexibility is what turns it from a convenience into an instrument for evaluating the subsystem's quality: slow dependencies and silent dependencies are the conditions worth testing against, and they are the hardest to arrange with a real system.
- Query and answer logging. The emulator collects not only the subsystem communication logs but also statistics from them, and counts of the documents and queries used. All of it is available both in the log file and in the program's interface.
A user manual was written for the client during the project, so the tool did not leave with the consultants.
Results
- Key subsystems can be tested autonomously, without the rest of the automated information system being available, healthy or cooperative.
- A tool built to the client's specification where no out-of-the-box product was permitted, delivered with documentation the client's own teams can work from.
- Failure conditions became testable - delayed responses, withheld responses and custom responses are configuration settings rather than incidents that have to be waited for.
- Parallel teams stopped blocking each other, through the retranslation module that forwards queries between subsystems and answers on behalf of the ones that are not there.
- Protocol-level visibility, through logging that captures the communication itself alongside statistics and document and query counts.
Questions this engagement answers
How do you test a microservice when the services it depends on are not available?
You stand in for them. Here a Java emulator sat in place of each subsystem, primed in advance with the correct and incorrect answers derived from a log analysis of the real one, so the subsystem under test received exactly the responses the scenario called for.
Can you test how a service behaves when its dependency is slow or silent?
That was a specific requirement of this build. Response time is configurable, an answer can be forbidden outright, and a custom answer can be returned - which makes degraded-dependency behaviour a repeatable test rather than an incident you hope to observe.
What if an off-the-shelf mocking tool is not allowed?
Then the tool becomes a deliverable. The client's specification excluded a boxed solution, so the emulator was designed from the system specification and a subsystem cooperation scheme, built in Java, and handed over with a user manual.
How do several teams share one test environment?
Through the retranslator. It forwards queries from one subsystem to another and responds when the request originated outside the team, so software, administration and monitoring teams could work on the system at the same time.
Need to load test an API whose dependencies are not ready?
If your architecture is a set of REST subsystems and the one you need to measure is entangled with the ones you cannot control, the useful first step is deciding what stands in for them - and how it should misbehave.
