Yes, you can have it both ways: in-sprint QE automation improves code and accelerates development speed
Oct 03, 2018 • 6 min read
Oct 03, 2018 • 6 min read
In previous blog posts we have described the business need for automated and continuous performance testing, the role and place of continuous performance testing in the continuous delivery pipeline, and the design principles of performance tests. In this article we’ll describe the Grid Dynamics technology for CPT, Jagger, which we developed over the years based on numerous client engagements and have released under the Apache open source license.
A good CPT platform facilitates all common activities related to the lifecycle of performance testing, including:
In the following sections we’ll describe how Jagger supports these activities. It is an open source project that we are using in a number of client engagements to implement continuous performance testing. For a shortcut, you can download Jagger and read more about it on our website on GitHub.
There are some things Jagger doesn’t do internally in the platform that must be addressed separately. Two of the most important ones are:
While we would love to bring these capabilities to Jagger in the future, they are not part of the technology today.
Jagger core concepts
The main function of Jagger is executed by the Jagger Master, which coordinates the Kernels’ work on load generation for the system under test. These components are running on separate cloud environments and can be scaled by adding new kernels to the cluster and providing sufficient load.
Kernels collect basic performance metrics (response time, throughput, etc.), and for additional information we inject Jagger Agents into the application in SUT. Each agent provides system and application metrics to the Jagger Master, which stores all data in the database.
For analysis and visualization we use the Jagger WebUI.
Jagger can be managed in two ways:
Jagger is designed to make writing and running performance tests easy for developers. The tests are written in plain Java using Jagger DSL. Using plain Java for test project gives you the ability to add any Java library for describing endpoints and manage test data.
Developers can run the tests locally on their machine while they are developing them, before they are ready to be “released” as part of the CI/CD pipeline.
docker-compose -f docker-compose.yml up
You are now ready to start writing and executing performance tests.
Create new tests
Out of the box, Jagger supports HTTP and HTTPS protocols for load tests. While it is possible to create your own end points and add support for other protocols, it is a rather advanced feature which we will not cover in detail in this blog post. For other endpoints you can use classes as an example and create your own Test Definitions. For examples on how to create your own Test Definitions and more, please look at the Jagger documentation.
Define complex test scenarios
CPT framework has to offer rather sophisticated user scenario tests, similar to a sequence of user actions on an e-commerce website. Some example of actions in this kind of test case include:
For proper performance metrics collection, Jagger will measure the response time of every UI action and API call, and the duration of complete steps in our user scenarios. This allows Jagger to check scenario step duration vs. acceptance criteria, as well as review the impact of every action and call.
Jagger supports creating user scenarios and validating responses after each step and generating next step queries based on predefined context and system responses. This helps simulate complex behavior in test cases. More details can be found in the documentation here.
Managing test code
Performance tests are regularly stored as separate projects in the version control system, and they have to follow the same rules for versioning, branching and release cycle as the SUT. If we break this rule, we could create a lot of issues such as testing incompatibility and instability, which would cause false alarms and reduce the accuracy of the test results.
Another thing which we need to think about when determining the scope of versioning is the test data. From release to release, new features are going to come — and we need to support them by creating new test data for them.
Once the tests have been written, it is time to generate a load for the System Under Test. Jagger helps do this efficiently. Depending on the test requirements, the load generation can work in a variety of topologies.
Simple, single-VM configuration
In the simplest configuration, test load generation is performed within one JVM. The results are stored in In-memory DB and a pdf test report is generated at the end. In this configuration you can collect only standard performance metrics.
While this configuration is convenient for quick local debugging of the tests under development, it is limited in its capabilities and is not a recommended way to use Jagger. The default configuration of Jagger deployment provides a more robust functionality:
All load management and generation is still done from single JVM, however:
Distributed load generation
In a case where a single node is not able to generate enough load or you would like to load your system from different nodes/locations, you can run the Jagger framework in the distributed mode.
In distributed mode, load generators (Kernels) are started as separate Java applications on single or multiple nodes. Kernels are responsible for communication with the SUT, with monitoring agents, and for temporary storage of raw test results.
The Master/Coordinator is started in its own JVM and is responsible for load balancing between Kernels, communication with JaaS, final results aggregation, and storage in the results DB.
Collecting metrics and storing results
An essential part of every CPT solution is collecting metrics from the SUT during test runs. There are several types of metrics:
The Jagger CPT platform has the ability to store all these metrics and use them for making decisions based on defined limits or deviations from previously measured values. (More info here.)
Metrics collected from a single execution can provide a lot of helpful information, but this is not where Jagger shines. To meet the objectives of CPT such as production validation, performance stability, and capacity planning, we need to collect and compare results from multiple executions. When we have an initial baseline, the trend of how the system’s performance metrics are evolving over time is more important than a single result. This allows us to run continuous performance testing in smaller environments and compare the results to production so we can better understand whether the system performs well or not. We can compare results of the tests in a smaller “lab” environment with themselves and each other and use correlation models between lab and production to evaluate production behavior.
In order to do that, Jagger stores test results and uses them for the following purposes:
Collecting, storing, and comparing metrics is a typical performance testing tool feature set. But when setting up CPT, we also require automated decision making based on both standard and calculated metrics. This mechanism must be flexible and customizable in order to control and trigger based on metrics specific to our application.
Now that we are ready to execute performance testing automatically, it is time to integrate Jagger into the CI/CD pipeline. Since Jenkins is the most popular CI server out of the box, Jagger comes with two Jenkins integration plug-ins:
In case you use another CI server, you can still integrate it with Jagger via the REST API. JaaS provides a REST API with 4 main controllers:
The analytics part of the CPT platform has to cover two main use cases:
Without automatic decision making we can’t build a true continuous process. Based on predefined thresholds and baseline results, our engine has to fail fast and reduce costs for investigating performance issues.
First we need a Jagger support baseline and comparison with its results. You could easily execute your test suite in a CPT environment with a “well performing” application version to establish the baseline, then use its results for comparison.
Also, for each metric you can set up warning and error limits for comparison with a baseline.
A baseline build is established as a reference point and new build results are compared with the baseline.
Note that we want to fail tests and receive alerts not only when performance degrades, but also when it improves abnormally. In a way, Jagger automatically performs anomaly analysis on the system’s performance metrics.
Jagger also highlights threshold violations via its Web UI:
After detecting performance issues we need to find their root cause. In the WebUI we have a visualization which can help.
Building trends for comparison between test sessions.
Metric details during test execution
Also, all information about test nodes is stored in a data base and can be shown on the UI.
Implementing CPT gives us another great ability: the automation of all parts of the environment provisioning, deployment, test execution, and results storage. Now we have a way to reproduce results on demand, with the ability to add instruments such as profilers and other external tools to our test run. Our smoke and integration performance tests are executed regularly and often, so we generate a large amount of test results. To make those results easy to interpret, our CPT platform visualization tool follows these principles:
The CPT platform visualization tool provides:
In this post we have defined the main features required for the CPT Platform, and described how Jagger implements these features. As an open source platform available under the Apache 2.0 license, Jagger is freely available. Give it a try today, and feel free to reach out to us with any questions.