Introduction

Functional testing tools like Selenium are essential for verifying that the different components of an application work together as intended. Integrating Selenium into a CI/CD pipeline can demand significant rework and high resource utilization, and running Selenium tests in a Kubernetes environment can be complex and resource-intensive. Running the pipeline multiple times for functional testing can quickly become inefficient and costly.

To address these challenges, we explored Testkube’s innovative Test Workflows for executing Selenium tests. Test Workflows provides a streamlined approach to defining, running, and automating tests in Kubernetes, allowing you to define your tests as code, manage test executions, and generate artifacts without disrupting your existing setup.

In this blog, we will delve into how Testkube’s capabilities simplify the functional testing process with Selenium. We will explore how to set up and automate your Selenium tests within a Kubernetes environment using Testkube, overcoming the common challenges of integration and resource management. 

What is Functional Testing?

Functional testing is the process of verifying whether the functionality of a product is built right.  It involves systematically testing each feature to ensure they function correctly according to specifications. This type of testing validates input, output, and user interactions to confirm that the product behaves as expected.

With functional testing, we can ensure that tests execute without errors, including handling exceptions and HTTP 404 responses. It also involves verifying correct redirections, ensuring usability, and assessing accessibility according to specifications. For web applications, functional testing can be automated using Selenium, which tests for expected outcomes in your web applications.

Functional Testing with Selenium

Selenium is an open source project that comprises many tools and libraries for browser automation. In Selenium, we can write test scripts in different languages like Python, Ruby, Java, Perl, etc. It supports different browsers such as Firefox, Chrome, Edge, Safari, etc. Selenium allows various types of testing such as acceptance, functional, regression, performance, load, etc.

There are the main Selenium components that allow testing:

• Selenium IDE: For developing Selenium test cases, Selenium provides a Chrome and Firefox extension. It records user actions in the browser using Selenium commands, making it efficient and educational for learning Selenium script syntax.

• Selenium WebDriver: WebDriver provides an abstraction layer over browser engines, exposing APIs that command the browser to run tests, simulating real user interactions. WebDriver is non-intrusive, allowing you to test the same application you deploy live.

• Selenium Grid: Once WebDriver tests are ready, we may require testing on multiple browsers, different operating systems, and their combinations. Selenium Grid allows parallel execution of test cases across multiple machines and also enables running tests remotely.

These Selenium components help create automated test scripts, cross-browser/platform testing, data-driven testing, and parallel testing with different language support across multiple browsers or operating systems. Let us understand in the coming section the working of Selenium and where these components fit.

How Selenium Test Works

Web browsers, despite appearing similar, operate differently beneath the surface. Selenium abstracts these differences, enabling code to execute seamlessly across browsers like Firefox, Microsoft Edge, and Chrome. This simplifies coding, allowing complex workflows to be written in just a few lines while ensuring consistent functionality across supported browsers.

To build a test suite in Selenium using WebDriver, we need to understand some of the commonly used terms:

• Driver: A driver controls the actual browser. They are also known as proxies. These are created and provided by the browser vendors in most browsers.

• WebDriver: The task of WebDriver is to command the browser but to do so, it needs the driver. WebDriver talks to the browser through the driver and receives information back via the same route.

• Framework: The task of the framework is to run and execute the WebDriver. These could test frameworks like JUnit for Java, RSpec for Ruby, NUnit for .Net, etc.

How does a WebDriver communicate with a browser?

In the execution of the Selenium test, the first step is to understand how the communication is happening between the WebDriver and the browser. Here are the two methods:

• Direct: In the direct communication setup, the WebDriver, Driver, and the browser are on the same system.

• Remote: In the case of remote communication setup, the communication between the WebDriver and the browser happens through a Selenium Server/Grid or RemoteWebDriver. The Selenium Server or Grid runs on a different system while the RemteWebDriver can run on the same system as the driver and browser.

Why do we need a Framework?

The WebDriver communicates with the browser but is unaware of the testing. It has no information on asserting a test fail or pass, or reporting an issue. This is where the frameworks help. A test framework allows the execution of WebDriver that runs the steps mentioned in the test.

The above image shows the working of a Selenium test with remote communication. Let us understand with the help of an example. Let us suppose, you have to create a Selenium test in Java. To execute this test, you will need a test framework like JUnit that will call the WebDriver which runs the test. Once the WebDriver is invoked, it communicates(remote) with the browser via the driver. Thus a Selenium test is run to verify the functionality of a browser.

Now we understand here that multiple components need to be set up in a Kubernetes native environment to perform tests. This comes with some of the challenges. Once such major challenge is scaling Selenium tests to run in parallel. Kubernetes can handle scaling, but configuring Selenium to run multiple browser instances effectively within Kubernetes can be complex. You need to figure out browser specific dependencies, ensure that Kubernetes nodes have optimum resources and that the test environment closely mimics the production environment.

Addressing these challenges requires a well-planned approach, leveraging Kubernetes features effectively, and utilizing tools like Testkube to simplify and streamline the deployment and management of Selenium tests.

Functional Testing with Testkube

Testkube offers a comprehensive solution to address the challenges of deploying Selenium tests on Kubernetes. With Testkube, teams can efficiently manage resource consumption by leveraging Kubernetes' scalable infrastructure, ensuring optimal use of CPU and memory resources. Its simplified setup process eliminates the complexities of configuring Selenium within Kubernetes, facilitating seamless communication between components while managing browser dependencies effortlessly.

As a Kubernetes-native test orchestration framework, Testkube simplifies the creation and execution of complex test workflows across various environments without the need for extensive scripting. Test Workflows provide granular control over test execution parameters, resource usage, and setup/teardown procedures. Also, Test Workflows are stored as Kubernetes Custom Resources (CRs), ensuring compatibility, scalability, and portability.

Let’s explore how to use Selenium with Testkube for functional testing in the coming section.

Pre-requisites

To follow along, you will need to set up the following:

• A Testkube Pro account
• A Kubernetes cluster - we’re using a local Minikube cluster (https://minikube.sigs.k8s.io)
• The Testkube Agent is configured on the cluster.

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent configured.

Use Case: Single Browser

We have written a test in Java that verifies a specific button on the website exists and is clickable. This test has been designed to check button functionality on Chrome, Firefox, and Edge. To build the dependencies and run the test, we are using Maven. You can find the Project Object Model(POM) configuration for Maven(pom.xml) and the Java Test(SeleniumTest.java) in this repository.

We will now create a Test Workflow in Testkube using Maven and integrate Selenium tests into it. In this first use case, we will run the test only for the Chrome browser and store corresponding artifacts to get an understanding of the process. 

Creating a Test Workflow

Here are the steps to create a test workflow from the Testkube Dashboard:

1. Navigate to the Test Workflows tab on Testkube and click on “Add a new test workflow”
   
   

This will provide you with the three options shown above:

• Create from scratch - use the wizard to create a Test Workflow.
• Start from an example - use existing k6, cypress, and playwright examples
• Import from yaml - import your own Test Workflow.

2. Select the “Import from yaml” option to create this workflow and provide the configuration as shown below.

3. Click on Create and a Test Workflow will be created. Here you can trigger the Test Workflow and execute the tests.

Before we execute the Test Workflow, let us glance at the Test Workflow and the Java test.

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
 name: selenium-single-browser-test #Name of the workflow
 namespace: testkube #Namespace where TestWorkflow run
spec:
 content:
   git:
     uri: https://github.com/cerebro1/selenium-testkube.git
     paths:
     - selenium-java #git repo and path that contains Selenium Java Test
 container:
   workingDir: /data/repo/selenium-java #Mount point for the repo
   image: maven:3.9.6-eclipse-temurin-22-alpine #Maven image for container
 services: 
   chrome:
     image: selenium/standalone-chrome:4.21.0-20240517 
     readinessProbe:
       httpGet:
         path: /wd/hub/status
         port: 4444
       periodSeconds: 1
 steps:
 - name: Run Chrome Test
   run:
     env: 
     - name: SELENIUM_HOST
       value: '{{ services.chrome.0.ip }}:4444' # get hostip from service
     shell: |
       mvn test #execute the test
       mvn surefire-report:report #generate reports
   artifacts:
     paths:
      paths:
      - target/site/**/*
      - target/surefire-reports/**/*

‍

In this Test Workflow, we have used the Selenium Standalone Chrome image that allows WebDriver tests to be run remotely. The WebDriver will use the chromedriver to communicate with the Google Chrome browser. We have updated the path of the directory in the Workflow that contains the functionality test case in Java for Selenium. In the above code, we have added comments to help you understand each step.

Here is the test that we are going to run. In this test, we sent a GET request to a website, searched for the button using the text on the button, and made a click. This test will check if the click was successful and return the outcome based on that.

Maven refers to the pom.xml file and installs the dependencies such as `selenium-java` and `junit-jupiter`. We are using JUnit as the test framework and enabled `maven-surefire-report-plugin` for gathering the artifacts.

Now that we have our Selenium test, POM, and the TestWorkflow ready, let us go ahead and execute it.

Executing the Test Workflow

Here are the steps to execute the workflow and view the artifacts:

1. Click on “Run this Test Workflow now” to trigger the first run. As the execution starts, you will see workflow running as shown below.

2. Click on the running workflow to view the log output.

The above image shows that the Chrome Test is running. The next step here is uploading the artifacts once the shell commands are executed successfully.

3. Once the test execution is complete successfully, click on Artifacts.

The above image shows the list of artifacts generated and stored at path `target/chrome-artifacts`.

4. In the site directory, click on surefire-report.html. Testkube loads the sure-fire report in the browser as shown below.

Thus with the help of Testkube, we were able to automate the execution of Selenium Java test on a Kubernetes environment using Maven.

In this use case, we ran the test only for one browser - chrome. In the real world, for testing an application development, you would want to perform the functional test across multiple browsers to ensure your application works for the majority of users. Here you have two options, you can either run a test on each browser serially or invoke parallel testing. Let us see with the help of the following use case how parallelisation is better than serial test execution and can help improve the overall testing process.

Use Case: Sequential test execution with multiple browser engines

We utilized the previous use case to create a serial Test Workflow for the test execution on multiple browsers. You can create and run this workflow as described for the basic use-case above - the execution will look as follows:

The workflow is configured to collect artifacts from each execution into a folder named after the browser:

As you can see we now have three browser instances (chrome, firefox, edge) and three corresponding test executions, one after the other. Although this works as expected, there are some points we noticed:

• Tests run one after another will make the entire testing process time-consuming, particularly for large test suites.
• If a test fails, it can halt the entire testing process until the issue is resolved, causing delays.
• Serial testing is not an optimal use of available computing resources.

In the coming section, we will see with the help of a use case how we can run the same Selenium test across different browsers using parallelisation in Testkube.

Use Case: Parallel execution with multiple browser engines.

With parallelisation, not only could we improve the use of infrastructure resources for test execution, but also execute more tests in the same amount of time. This would lead to earlier detection of issues and an increase in the overall efficiency of the testing process.

Let’s modify our Test Workflow to make use of two powerful Testkube features; matrix parameters and parallel execution. To achieve this, in the `TestWorkflow` we have defined `ip` and `name` of each browser `services` in `parallel.matrix.browser`. The execution of shell commands and storing of artifacts will happen in parallel for all browsers using the ‘parallel’ execution object (the full workflow is here):

...
 steps:
  - name: Run Cross-browser tests
    parallel:
      matrix: # define matrix of browsers to test against
        browser:
        - ip: '{{ services.chrome.0.ip }}'
          name: chrome
        - ip: '{{ services.firefox.0.ip }}'
          name: firefox
        - ip: '{{ services.edge.0.ip }}'
          name: edge
      transfer: # make sure each parallel node has the test
      - from: /data/repo/selenium-java
      fetch: # retrieve generated artifacts from node
      - from: /data/repo/selenium-java/target
        to: /data/artifacts/{{ matrix.browser.name }}
      container:
        workingDir: /data/repo/selenium-java
        image: maven:3.9.6-eclipse-temurin-22-alpine
      steps:
      - run:
          env:
          - name: SELENIUM_BROWSER
            value: '{{ matrix.browser.name }}'
          - name: SELENIUM_HOST
            value: '{{ matrix.browser.ip }}:4444'
          shell: |
            mvn test
            mvn surefire-report:report
  - condition: always
    workingDir: /data/artifacts
    artifacts:
      paths:
      - '**/*'

Create and execute the Workflow as we have done previously:

In the above image, we can see that 3 workers are running at the same time. Testkube not only has improved the process of test execution, but also made optimal use of available computing resources here. Since these workers are independent of each other, failure of any one of them will not impact the execution of the other, providing test isolation.

Performance Comparison Parallel vs Sequential Test Execution

On the completion of the above tests, we compared the execution times of 5 serial and parallel test executions. 

While the execution of the entire serial test workflow took roughly 73 seconds in avg, the parallel execution workflow completed in 48s on average, an improvement by roughly 34%. Drilling down into the actual times spent on running tests within each workflow (excluding setup/teardown/etc), the difference was 57s on avg in parallel flows vs 30s on avg for parallel executions - an improvement of almost 50%.

Conclusion

Selenium is a widely used suite of tools for browser automation. Leveraging its power in Testkube without the complexities of resource management, environment configuration, and setting up dependencies, streamlines the testing process for Kubernetes applications.

In this blog post, we have seen how functional testing with Selenium could be done using Testkube. We started with setting up Testkube workflow for single browser and explored parallelisation to optimize multiple browser testing. We were able to run a Selenium test in Testkube effortlessly for single and multiple browsers and explore the benefits with parallel execution in case of multiple browsers. Testkube abstracts the complexities and allows optimized test execution in a containerized environment.

Using Testkube, you can ensure compatibility across Selenium, browser drivers, and Kubernetes versions, reducing maintenance overhead. Overall, it offers a strategic and efficient approach to deploying Selenium tests on Kubernetes, enabling teams to focus on delivering high-quality software with confidence.

We invite you to try Test Workflows. Visit Testkube website to get started. If you find yourself struggling with anything, feel free to drop a note in our active Slack community, and someone will help you out.

‍