Distributed Systems face various challenges due to the system’s complexity, such as partial system failures, data inconsistency, deadlocks, high latency, and packet loss. These challenges can often be proactively addressed by proper functional and non-functional testing, including distributed load-testing to ensure your system can handle a high number of users at any given time. Using tools like JMeter, you can simulate real-world conditions and failures to ensure robustness and reliability.

However, simulating production-like conditions often requires running load tests in a distributed setup, generating load from multiple sources, which is resource-intensive and requires careful orchestration of tests and resources. Thus, you need specialized tools to make that process simpler. 

In this blog, we will show how Testkube can be leveraged together with JMeter to simplify the distributed load-testing process, resulting in efficient resource allocation, horizontal scalability, and simplified orchestration of tests.

Distributed Testing with JMeter

Distributed testing is executing tests concurrently from multiple machines or environments. The load is distributed across multiple nodes to more closely simulate real-life usage scenarios. This method is beneficial for testing complex systems, cloud-native, and large-scale applications like banking or e-commerce websites. 

In this section, we will discuss JMeter, a distributed load-testing tool, dig deeper into its distributed systems testing architecture, and understand the associated challenges.

JMeter

Apache JMeter is an open source software for distributed, performance, and load testing of applications. JMeter supports protocols like HTTP/HTTPS, FTP, SOAP, JDBC, TCP, UDP, and much more, allowing it to support the testing of different applications. It enables distributed testing with master-slave configuration, spreading the load across multiple nodes. This enables an extensive test of an application’s behavior under heavy traffic. Let us understand the master-slave architecture supported by JMeter for distributed testing.

JMeter Master-Slave Architecture

In the master-slave architecture of JMeter, multiple users are simulated across various machines. This type of setup helps test such applications where you need to generate significant loads but have hardware limitations. 

‍

Source: JMeter Distributed Testing

Here is an overview of each component in this architecture:

• Target: An application, service, or server under test.
• Master [Controller Node]: A controller that manages the execution of the test. It initiates the test execution, distributes the load, and collects the results from the slave.
• Slave [Worker Nodes]: A load generator that takes requests from the master, generates the load, and executes the test on the target.

How does JMeter execute a test on a target?

There is one master that handles the execution of tests using multiple remote slaves. For proper communication, the master, slaves, and target must be on the same network. The master communicates with the slaves using Java RMI(Remote Method Invocation). Each of these slaves generates a load and executes tests. The test execution by all the slaves on the target starts at the same time. The execution details are sent back by the slave to the master, who performs the result aggregation.

Challenges with distributed testing with JMeter

While this setup seems easy, there are some challenges associated with it:

• Resource allocation overhead: Even though the master-slave architecture distributes the load, which reduces the chances of over-utilizing a single slave, it is quite challenging to allocate resources to the slaves for optimum usage.
• Complex master-slave configuration: Ensuring the same version of JMeter on all the nodes, configuring the network to allow RMI traffic, and synchronizing time between nodes is complicated.
• Monitoring and debugging issues: JMeter requires configuring external plugins or tools for real-time analytics or monitoring. With a large-scale system setup and complicated configurations, this becomes more of an overhead for the team.

Thus, for performing distributed load testing on complex systems in a cloud-native environment, you require a tool to help with optimum resource allocation, easy configuration of master and slaves, scalability, and proper observability. The good news is that Testkube can do it all and gives you both a Dashboard and command line utility for test orchestration. 

In the following section, we will take a detailed look at the features of Testkube that can help achieve better test orchestration for JMeter distributed testing. 

Automate JMeter Distributed Testing with Testkube

Testkube is a test orchestration and execution platform that leverages the power of Kubernetes for testing cloud-native applications. It allows you to automate test execution irrespective of your testing framework, tool, or script using a powerful Test Workflows engine, and leverages a unified dashboard, for centralized test creation, execution, and result aggregation, helping you manage tests better and gain observability in overall testing.

How does Testkube automate JMeter distributed testing?

Testkube supports all popular testing frameworks, including JMeter, for distributed load testing. Testkube runs the test directly in your Kubernetes cluster, ensuring secure execution of your system or new features. Here are some of the benefits that Testkube offers when it comes to running distributed JMeter tests:

• Easy RMI configuration: Testkube simplifies the major challenge of configuring RMI in JMeter’s master-slave architecture by automating the network configuration needed, such as firewall and port setup. It also automates the installation of the same JMeter version on all nodes, saving time and reducing the risk of error. Testkube provides a sample configuration template that you can use to get started easily.
  
  
• Scalable system design: With Testkube, you can leverage Kubernetes native scaling capabilities and dynamically scale the number of slaves without changing your test scripts. Testkube utilizes the Kubernetes’ resource management to distribute load. This way, you can easily simulate a large amount of traffic to test your system or application without worrying about resource allocation.

• Centralized monitoring dashboard: While testing, it is important to have the ability to view the details of previous test executions and current ones for comparison. In the case of JMeter, you also need test execution details, errors, and performance metrics per slave. The Testkube Dashboard aggregates all the test executions in real-time and helps you track them for easy monitoring and debugging.

Testkube handles all the complexities of performing distributed tests in JMeter. This lets you focus on developing the application and testing rather than figuring out the infrastructure. Let us see in the coming section the JMeter distributed testing in Testkube.

How do you execute JMeter distributed testing in Testkube?

Using Testkube Test Workflows, we are going to execute the test with JMeter. We have created a JMeter test that runs on our Testkube website and performs the distributed test. Let us get started with the prerequisites and create a Test Workflow to execute the test.

Prerequisites

The basic requirements while configuring the master-slave architecture in JMeter with Testkube are as follows:

• A Kubernetes Cluster(We have used Minikube here.)
• A Testkube Pro account (free plan is fine)
• The Testkube Agent is installed on the cluster.
• A Testkube API token with Admin Access rights.

‍

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent.

Creating a Test Workflow

In the Testkube Dashboard, we are going to create a Test Workflow.

1. Login to Testkube and select Workflows from the left menu bar.
2. Click on “Add a new Test Workflow” and select “Start from an example”.

‍

3. Scroll to the right and select “Distributed JMeter”. Testkube loads an example YAML.

4. We are going to update this YAML to also process the JMeter artifacts. Provide the YAML given below:

‍

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: distributed-jmeter-example-config-artifacts
  namespace: testkube
spec:
  config:
    slavecount:
      type: integer
      default: 3
  content:
    git:
      uri: https://github.com/kubeshop/testkube
      revision: main
      paths:
      - test/jmeter/executor-tests/jmeter-executor-smoke.jmx
  container:
    workingDir: /data/repo/test/jmeter/executor-tests
  services:
    slave:
      use:
      - name: distribute/evenly
      count: config.slavecount
      logs: always
      timeout: 30s
      image: anasoid/jmeter:5.6-plugins-21-jre
      command:
      - jmeter-server
      - -Dserver.rmi.localport=60000
      - -Dserver_port=1099
      - -Jserver.rmi.ssl.disable=true
      readinessProbe:
        tcpSocket:
          port: 1099
        periodSeconds: 1
  steps:
  - name: Run tests
    run:
      image: anasoid/jmeter:5.6-plugins-21-jre
      shell: |
        jmeter -n \
          -X -Jserver.rmi.ssl.disable=true -Jclient.rmi.localport=7000 \
          -R {{ services.slave.*.ip }} \
          -t jmeter-executor-smoke.jmx \
          -j /data/artifacts/jmeter.log \
          -o /data/artifacts/report \
          -l /data/artifacts/jtl-report.jtl -e
    artifacts:
      paths:
      - /data/artifacts/**/*
status: {}

You can view the same running in the Testkube Dashboard as shown below.

‍

Let us split the above file into parts to understand the changes specific to JMeter.

1. Defining the slaves to have a dynamic value: Here we have used `config` to define the slave count. The name of the field is set as `slavecount` of type `integer`. We have set the default value as 3. This will be updated with the value you provide at the time of execution. Testkube allows easy scalability without touching the test script.

‍

spec:
  config:
    slavecount:
      type: integer
      default: 3

2. Path to the test that will run on target: Using the `content`, we have defined the Github repository path that has the test that will be executed on the target. In this test, the target is the Testkube website: `testkube.kubeshop.io`. With this, Testkube helps us version control the tests.

‍

content:
  git:
    uri: https://github.com/kubeshop/testkube
    revision: main
    paths:
    - test/jmeter/executor-tests/jmeter-executor-smoke.jmx
container:
  workingDir: /data/repo/test/jmeter/executor-tests

‍

3. Configuring the slave: Testkube creates JMeter slaves as Kubernetes pods. The number of slaves helps decide the number of pods to spin. Here, we have passed the JMeter image to be used while creating the containers in the pod. Also, we have defined the command that will run when each container spins up.

‍

  services:
    slave:
      use:
      - name: distribute/evenly
      count: config.slavecount
      logs: always
      timeout: 30s
      image: anasoid/jmeter:5.6-plugins-21-jre
      command:
      - jmeter-server
      - -Dserver.rmi.localport=60000
      - -Dserver_port=1099
      - -Jserver.rmi.ssl.disable=true
      readinessProbe:
        tcpSocket:
          port: 1099
        periodSeconds: 1

‍

You can see here the minimal configuration that needs to be done which is already part of the example that loads in the Testkube Dashboard when you select to get started with an example. Testkube internally does the firewall configuration to ensure secure and seamless communication between slave and master.

‍

4. Running test and processing the artifacts: The final step is to define the test. In the master-slave architecture, the master initiates the test execution on the slaves using the `jmeter` command with arguments that allow it to connect to the created slaves We also add arguments for generating a report for our execution, which we then capture using the artifacts configuration.

‍

steps:
  - name: Run tests
    run:
      image: anasoid/jmeter:5.6-plugins-21-jre
      shell: |
        jmeter -n \
          -X -Jserver.rmi.ssl.disable=true -Jclient.rmi.localport=7000 \
          -R {{ services.slave.*.ip }} \
          -t jmeter-executor-smoke.jmx \
          -j /data/artifacts/jmeter.log \
          -o /data/artifacts/report \
          -l /data/artifacts/jtl-report.jtl -e
    artifacts:
      paths:
      - /data/artifacts/**/*

‍

We used here the sample provided by Testkube to run distributed testing in JMeter and tweaked it to process the artifacts.

‍

5. From the dropdown, select “Create” to create the test workflow.

‍

‍

6. We have set the default slave count as 3. On the prompt, enter the number of slaves you want to be created and click on “Run”. We have entered the value as 6 here.

7. Testkube does the automatic resource allocation and creates 6 slave nodes along with performing the test execution as shown below. 

‍

‍

After the test execution is completed successfully, you can view the execution logs of each slave by selecting slave in Testkube Dashboard. In the image below, we have shown the execution logs of slave #1.

‍

Viewing the artifacts

Artifacts help with the analysis of an application under heavy load. Testkube integrates with JMeter and loads the UI which gives a detailed view of the application test. In the Testkube Dashboard, click on “Artifacts”.

From the `artifacts` dropdown, select `report` and click on `index.html`.

‍

‍

Testkube loads the JMeter Dashboard as shown below. This has a detailed view of the Application Performance Index, Requests Summary, Statistics, etc which you did not have to set up explicitly. Testkube configures all this for you, making it easier for you to gather results.

‍

‍

Testkube provides utilities to make testing easier for you so that you can focus on the application development better.

Executing the Test Workflow using Testkube CLI

Testkube provides the ability to connect to the Testkube account using the command line. In the previous section, we created and executed the Test Workflow from the Testkube Dashboard. We will learn here how to manage the Test Workflows using Testkube CLI. Install the Testkube CLI on your machine and configure the API Token. Once the context is set, you can view your Test Workflows and run them using the following command:

```

testkube run tw <testworkflow_name> --config <item=value>

‍

For our use case, here is how the execution from the CLI looks like:

$ testkube run tw distributed-jmeter-example-artifacts --config slavecount=4
Context: cloud (1.17.54)   Namespace: testkube   Org: SONALI SRIVASTAVA-personal-org   Env: SONALI SRIVASTAVA-personal-env
--------------------------------------------------------------------------------------------------------------------------
Test Workflow Execution:
Name:                 distributed-jmeter-example-artifacts
Execution ID:         66a2315f7fc8371cd690a9d3
Execution name:       distributed-jmeter-example-config-artifacts-2
Execution namespace:  testkube
Execution number:     2
Requested at:         2024-08-13 21:05:03.083826997 +0000 UTC
Status:               queued

‍

You can view the same running in the Testkube Dashboard as shown below.

‍

‍

This level of automation, from creating to executing the Test Workflow, allows you to work on the test and leave the execution to Testkube. With Testkube, we could run the JMeter test with minimal configuration and view the application performance-related metrics.

Conclusion

JMeter is a commonly used tool for distributed load testing. Testkube abstracts the complexities associated with it and handles the network configuration, resource allocation, and processing of the artifacts. In this blog, we have seen, using an example available in the Testkube Dashboard, how we can easily set up the master-slave architecture for distributed testing of an application with JMeter.

Using the Testkube Dashboard, we were able to configure and execute distributed JMeter tests with so much ease, and Testkube helped us view the execution logs of each slave and an aggregated results report which helps debug issues quickly.

By leveraging the Kubernetes features, Testkube simplifies the process of configuring slaves and gives the power to dynamically set the number of slaves. Testkube also supports k6 for distributed testing. So, to standardize testing for you, we invite you to try Testkube today. Witness firsthand how Testkube simplifies and empowers your testing process with its Kubernetes-native test execution capabilities. Join our active Slack community for guidance and support.

‍