Introduction

Testing becomes crucial as the complexity and the adoption of a solution grows. In Kubecon NA 2020, Jian from Airbnb talked about 10 more weird ways to blow up your Kubernetes. The first key takeaway from the talk was to test the new features on test clusters. But how does one ensure adequate testing without adding significant time to build or deploy pipelines?

To do so, here are two preliminary tests that have to be performed by a team before moving their solution to a production environment:

• The solution must be running on such systems that can support peak usage. This would require simulating many users to test the peak performance.
• Each feature of the solution has the expected behavior and is seamless to use. For this, testing has to be done across multiple environments, different browsers, devices, operating systems, and inputs. 

Testing for both of these scenarios ultimately requires time and resources, which is where parallel testing is a powerful approach to ensuring your system is tested at scale before it goes into production. In this post, we will understand the concept of test parallelisation and some of the common approaches like load and functional parallelisation. We will discuss in depth some of the challenges associated with those approaches, and explore a Kubernetes native testing framework that can help us address those challenges.

Parallel Testing Overview

Parallel Testing is the process of performing parallel simulations and tests to validate the functionality and performance of the application under test. Parallelisation is done to both scale the performance under load, and perform functional analysis of the system under test to increase coverage and decrease overall test execution times. Let us understand load and functional parallelisation in detail.

Parallel Load Testing 

To test a solution for peak-hour performance, creating a load by sending multiple requests from a few anonymous users is not enough. We are required to simulate a real-time environment by generating a massive load from named users from multiple geographies, possibly across multiple browsers or devices.

Parallel load testing is the process of using parallel instances of load-testing tools to simulate a massive number of users from multiple nodes, possibly distributed across geographic locations. Solutions that face large volumes of concurrent users, such as social media platforms, banking applications, e-commerce websites, online gaming applications, etc. can all benefit from this approach to testing to ensure uninterrupted services for their users even under peak usage. 

Enabling load parallelisation has some challenges, let us understand these.

Challenges with load parallelisation

• Generate Heavy Load: Load parallelisation requires sending huge traffic to the software or application possibly from various geographical locations to test peak-hour performance and network latency. For example, for an online gaming website, we need to test if players from across geographies are playing in a multiplayer mode, they do not experience network lagging.

• Parameterized Virtual Users: To simulate specific user behavior by specifying individual characteristics, parameterized virtual users are required. Generating anonymous virtual user requests will not help if we need to customize test cases. Suppose in an online gaming website, players are chatting using the chat message feature in the game, it needs to be tested that their profile details are correct.
  
  
• Visualizing Results: We need real-time visualization of the test to simply understand the behavior of the application or the scenario that is causing the issue. It may also require aggregating such results to observe the behavior. For example, if the test shows large network latency when a massive load is created from multiple geographies, we need detailed results to visualize the issue and debug the underlying network infrastructure.

• Monitoring Test System: For the target system under test, monitoring can help us identify the resource usage such as CPU or memory utilization, and scalability behavior and anticipate performance issues when a massive load is introduced to the solution. For example, if the test shows a spike in CPU usage during peak hours then this analysis can be used to optimize the resource allocation.

Parallel Functional Testing 

Parallel functional testing is the process of performing parallel functional tests to increase test coverage and decrease overall test execution time. It is commonly paired with parameterised test execution by giving a specific input and checking to get the expected output. For example, using the browser version as input, we can test a feature across multiple browsers or operating systems in parallel and at the same time. This would not only help us with time to market but would also allow efficient testing and optimized usage of resources. In the coming section, we are going to talk about the need for functional parallelisation with the help of an example.

‍

Why is functional parallelisation needed?

For example, we have a banking solution for which we want to make two-factor authentication mandatory for all users at the time of sign-in. Now testing this functionality across multiple browsers sequentially would not only take a lot of time but also resources. Here the input would be a parameterized user for which the two-factor authentication is enabled and happens successfully in a minimal amount of time.

With functional parallelisation, we can test this functionality with various inputs across multiple browsers at the same time. Suppose the test fails for any of the browsers, we would get to know in a lot less time and work on identifying the root cause along with its solution.

Enabling functional parallelisation also has some challenges, let us understand these.

Challenges with functional parallelisation

• Test Isolation: Each test may have its own set of dependencies, configurations, and compatibility considerations. Setting up and configuring multiple test environments for different tests can be time-consuming and complex, especially when ensuring consistency across environments.
  
  
• Recording Results: In the previous example, we would require a system that can record different input cases and the combination browser and the actual results. These recorded results would then be used to compare with the expected result to determine the performance of functionality.
  
  
• Resource Allocation: Identifying which functionality test would require more resources and then distributing the resources accordingly could be a challenging task. Suppose for a banking solution, the sign-in and transaction functionalities have to be tested. Now these tests take time even though run in parallel so we need to determine the resource utilization limit for each.

Seeing the challenges associated with load and functional parallelisation, we need a testing solution that automates testing with parameterized virtual users or other inputs across multiple browsers or operating systems simultaneously. Also, the solution should allow dynamic resource allocation based on test requirements and optimize resource utilization. For systems under test, the testing solution should provide real-time visualization of tests and aggregated results to perform result analysis.

Parallel Testing with Testkube 

Testkube recently introduced Test Workflows, which leverages an execution and orchestration engine specifically built for executing any testing tools and scripts at scale, and since Testkube leverages Kubernetes as its runtime environment for test execution, it can scale and allocate resources for test execution in line with corresponding functionality provided by Kubernetes itself.

Test Workflows are defined using a dedicated YAML vocabulary, and can be created using both extensive Wizards and samples for different scenarios which can then be further enhanced in any way required to fulfill your testing requirements using the Testkube Workflow Editor. Workflows also give fine-grained control over resource-usage and allocation for your tests, helping you maximize the utilization of your infrastructure for test execution.

Parallel testing is one of the many functionalities available via Workflows:

• A parallel keyword makes it possible to run any testing tool across multiple nodes 
• A shard keyword makes it possible to shard tests across multiple nodes to ensure each node is running the right set of tests 
• A matrix keyword makes it possible to parameterise tests running both in sequence and in parallel
• A service keyword makes it possible to manage depending services to run distributed tests (for example with JMeter or Selenium)
• An execute keyword makes it possible to orchestrate multiple tests to run both in sequence and/or in parallel, allowing you to simulate more advanced usage scenarios - see our article on System Testing.

Workflows for Load Parallelisation

As described above, running a load test from multiple nodes in parallel is often required to simulate large amounts of user traffic. The parallel keyword in Test Workflows allows you to parallelize any load-testing tool or script for this purpose, for example:

• K6 - parallel can be used to distribute and parameterise k6 scripts far beyond what can be achieved with the k6-operator - Read More, 
• JMeter - service can in combination with service.count be used to spawn any desired number of distributed worker nodes which are then invoked from a master node for running the test.
• Artillery, Gatling - similar to K6, distributing and parameterising Artillery and Gatling tests is easily achievable using the parallel keyword

‍

Examples for distributed load-testing with several of these tools are available in the Testkube Documentation.

Workflows for Functional/End-to-End Parallelisation

Parallelising functional and end-to-end (E2E) tests for the purpose of increased test coverage and reduced overall test execution times is equally well supported by the parallel keyword. Depending on the testing tool you are distributing, you can also shard test files or input parameters across nodes, which is supported by the shard keyword, for example

• Playwright tests can be parallelised and sharded across multiple nodes using Playwrights built-in sharding functionality
• Cypress tests can be parallelised and the Workflow Express Language can be used to shard test files accordingly
• API Testing tools like Postman and SoapUI can be parallelised and parameterised similarly to increase test coverage for APIs
• Acceptance testing tools like Selenium and Robot-Framework can be parallelised and parameterised for different browsers and user input, increasing acceptance-test coverage before releasing applications into production.
• Any other testing tool or script can be distributed across multiple nodes, with parameterisation as applicable for each tool.

‍

As above, examples for distributed functional testing with these tools are available in the Testkube Documentation.

Composite Workflows for Test Parallelisation

The approaches described above with the parallel and shard keywords are meant to be used within a single workflow. If you on the other hand want to combine multiple workflows to run either in sequence or in parallel you can use the execute keyword to orchestrate the execution of any other workflow in any combination required. For example you can:

• Run multiple load-tests in parallel to see how the traffic they simulate affect each other
• Run an E2E or API test in parallel with a load-test to ensure that functionality is maintained under load
• Run a security test in parallel with a distributed load-test to ensure that security is maintained during high load on your applications.

‍

Read more about Composite Testing in Introducing System Testing with Testkube and about the execute keyword in the documentation.

Troubleshooting, Artifacts and Reporting

Testkube automatically captures the log output of any testing tool it runs on all nodes the tools are running on, to help you ensure that tests were executed as desired and debug issues with your executions. Furthermore, Testkube can be configured to capture any artifacts produced by your tests, including aggregate reports, videos, JUnit-reports, etc. When captured, Testkube will make these available via the Dashboard.

‍

Conclusion

Testing your applications at scale often requires a parallelised execution approach to your testing efforts, be it the parallelisation of load-tests to generate massive load, or the parallelisation of functionali/E2E tests to increase test coverage and improve test execution times. 

The recently introduced Test Workflows engine in Testkube allows for parallelisation of any testing tool, both load and functional, with additional support for sharding, parameterisation and management of dependent services to execute your tests.

Head over to testkube.io/get-started to learn more and give Testkube a try using either our demo environment or your existing tests running in your own infrastructure. 

‍