Automated Acceptance Testing with Robot Framework and Testkube

Introduction

Testing an application’s capability usually requires setting up a production-like environment to run tests with realistic insights into the functionality and performance of the application. Tests can be both functional and non-functional, all depending on the requirements on the application and its infrastructure.

Teams usually prefer a testing solution that can be easily integrated into the existing CI/CD infrastructure without much work. But what if your testing tool can not be easily automated with your CI/CD solution or run in your existing Kubernetes infrastructure?

We decided to explore Testkube, which can take the test as code for any testing tool, automate test execution and integrate with CI/CD, and generate artifacts in Kubernetes. This would allow us to perform testing seamlessly and leverage the benefits of Kubernetes. So let's learn in this blog about acceptance testing with the Robot Framework and how to automate it with Testkube.

What is Acceptance Testing?

Acceptance testing is a phase in software development where the software is tested to determine if it meets the requirements specified by the client or stakeholders. It's usually performed after system testing and before deployment. Acceptance tests are typically written from a user's perspective and aim to verify that the system behaves as expected in real-world scenarios. 

Acceptance testing also allows you to ensure software complies with legal and regulatory requirements. This reduces the risk of legal consequences or penalties due to non-compliance, especially in certain industries, such as healthcare or finance. 

Other testing practices, such as API, load, etc., contribute to different aspects of software quality, such as performance, reliability, and interoperability, and are usually done before acceptance testing.

The Robot Framework is an acceptance testing tool that is easy to write and manage due to its key-driven approach. Let us learn more about the Robot Framework to enable acceptance testing.

Acceptance Test with Robot Framework

Robot Framework is an open-source, keyword-driven test automation framework that supports Python, Java, and .NET and is widely used for acceptance testing, acceptance test-driven development (ATDD), and robotic process automation (RPA). It provides a highly readable, tabular syntax for writing test cases in a plain-text format, making it accessible to both technical and non-technical users.

Robot Framework stands out for the following reasons:

1. Keyword-Driven Approach: In Robot Framework, test cases are defined using keywords that represent the validation. These keywords are organized into test suites, which are further structured into test cases.
2. Test Data and Variables: Test cases in Robot Framework are written in a tabular format using plain-text files, typically in files with the `.robot` extension.
3. Built-in and Custom Keywords: Robot Framework offers a set of built-in keywords for performing common actions such as clicking buttons, entering text, verifying text, etc. It also allows users to define their custom keywords.
4. Library Support: Robot Framework supports integration with external libraries written in Python, Java, .NET, and other languages, allowing users to extend its functionality as needed. These libraries can provide additional keywords for interacting with specific technologies, such as web browsers, APIs, databases, and more.

Test cases written in Robot Framework can be executed using the `robot` command-line tool or through various integrated development environments (IDEs) and continuous integration (CI) systems. During execution, Robot Framework generates detailed reports and logs, highlighting the status of each test case and providing insights into test results and any encountered issues. 

Running Acceptance tests in Kubernetes

While running Robot Framework tests against legacy/non-containerized applications is usually straight-forward, there are several challenges associated with running these tests against a containerized application infrastructure built on Kubernetes:

• Resource Management: Kubernetes manages containerized applications, including their resource allocation and scaling. However, scaling requires ensuring optimal resource utilization for Robot Framework test execution. This would be challenging to deal with, especially in case of varying test workloads and resource requirements.

• Test Isolation: Ensuring proper isolation for individual test executions can be complex while running multiple tests concurrently. It will require careful consideration to avoid resource conflict and interference between tests.

• Persistent Data Handling: Robot Framework may require access to persistent data, such as test artifacts, test data files, or external dependencies. Managing persistent data in a Kubernetes environment, including storage provisioning, volume mounting, and data synchronization, can be challenging, especially when dealing with distributed test executions.
  
• Secure Access: Access to applications or components running in Kubernetes can be severely constrained for security reasons, making it difficult to run tests against them with tools that are not running in the cluster themselves. Relaxing network policies for the sake of testing is not always an option.

Thus, when using the Robot Framework for acceptance testing of your containerized applications, you will require a test-execution framework that eases the life of a developer and can orchestrate Robot Framework tests with all the above caveats in mind. Fortunately,there is such a framework specifically targeted at running any kind of testing tool, including the Robot Framework, in a containerized application infrastructure: Testkube.

Acceptance Testing with Testkube

Testkube is a test orchestration platform that supports the execution of any testing tool in a containerized application infrastructure. It leverages Kubernetes to provide a scalable and consistent execution environment for all your tests, and includes a unified dashboard for test definition, reporting, and troubleshooting.

To run tests in Testkube, one defines Test Workflows for managing the execution lifecycle of individual test(s), which will then be executed in your Kubernetes environment using the corresponding testing tool image. Let us understand more about the Test Workflows feature.

Test Workflows

Test Workflows provide a declarative approach for defining test execution lifecycles. It leverages Kubernetes-native capabilities to orchestrate and execute tests, collect real-time artifacts, and display them on the Testkube dashboard. A Test Workflow is defined as a single YAML configuration that is stored as a custom resource in your Kubernetes cluster(s), Test Workflows allow integration with existing testing tools and GitOps pipelines, and also provide templates for standardized testing configurations adaptable to various needs.

Execute Acceptance Testing in Testkube

To run a Robot Framework acceptance test with Testkube, you will need to define a corresponding TestWorkflow that uses a prepackaged Robot container image to run the test cases that you have defined - let’s have a look at how that works.

We will see two demos here using the Robot Framework in Testkube using the sample example, Restful Booker, provided by Robot Framework.

1. Basic: In the basic execution we will run the sample test case by Restful Booker. In this test case, we will get bookings for a user, create a booking for a user, and verify that the application performs the required action. All these user details are hardcoded in the test case. We will execute the test in the Testkube dashboard and collect artifacts.
   
   
2. Advanced: We will update the above test case to use variables instead of hard coding user details and make use of parameters to provide values. This will give us the flexibility to test different scenarios. In Testkube we will make use of parameters to process these parameterized inputs.

So let us see Testkube in action performing an acceptance test using Robot Framework. Before we get started, let us verify the following prerequisites.

Pre-requisites

To follow along, you will need to set up the following:

• A Testkube Pro account (free plan is fine)
• A Kubernetes cluster - we’re using a local Minikube cluster (https://minikube.sigs.k8s.io)
• The Testkube Agent is configured on the cluster.

Once the prerequisites are in place, you should have a target Kubernetes cluster ready with a Testkube agent configured.

Basic Execution

In the basic execution, we will run the example Restful Booker test code in Testkube using Test Workflows. We have pushed the test case to a GitHub repository. These files will by default be mounted by Testkube to the’/data/repo’ directory when the test executes. Let us get started.

1. Select Test Workflows from the menu and click on the ‘Add a new test workflow’ button to start setting up the workflow.
2. From the options to ‘Create a Test Workflow’, select ‘Create from scratch’.

Testkube provides an option to start from scratch or use existing examples to set up a workflow

3. Enter the following details to create a new workflow.some text
   1. Name: Set the name of the workflow as ‘basic-acceptance-test’
   2. Type: Select the type of test as ‘Other’
   3. Image: Enter the name of the Docker Container Image for Robot Framework. We are using ‘ppodgorsek/robot-framework’
   4. Tag: Search for the stable tag on Docker Hub for the image and enter it here. We have been using 7.1.0 since it was recently released.
   5. Shell command: By default, the repository that has the test case is mounted to the `/data/repo` directory. We will use the `robot` command line tool to execute the test, and using the –outputdir option, we will save the output to a specific location. Here is the shell command: ‘robot --outputdir /data/repo/output /data/repo/test-restful.robot’.
      
      

Create a test workflow with your custom details.

4. Select the Source as Git from the drop-down menu and enter the repository details.
5. In the final step, the UI will create the YAML for our Test Workflow. Let us go ahead and add the path where the artifacts will be stored in `spec.steps`. Here is what the final configuration looks like:

‍

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: basic-acceptance-test
spec:
  content:
    git:
      uri: https://github.com/cerebro1/robot-framework-test.git
  container:
    image: ppodgorsek/robot-framework:7.1.0
  steps:
    - name: Run test
      shell: robot --outputdir /data/repo/output /data/repo/test-restful.robot
      artifacts:
        paths:
        - /data/repo/output/*

‍

‍

6. Click on Create and your ‘basic-acceptance-test’ workflow is ready:

7. Click on ‘Run this Test Workflow now’ to execute the workflow. Once the Test Workflow executes successfully, you will see the Test Workflow in the ‘Recent executions’ list.
   

8. Select the Test Workflow to view the details. You can see each step has been executed successfully in the following image. This is helpful for debugging in case something goes wrong or you need more insights. Test Workflow has a separate section for Log Output, Artifacts, and Workflow.

9. Click on ‘Run shell command’ to view the details of the test cases executed. The execution shows that all the tests have passed successfully.

10. Click on the Artifacts tab to view the artifacts processed. You can download the artifacts generated by the Robot Framework here.

‍

Thus, with the help of Test Workflow, we were able to run a Robot Framework test seamlessly without any setup required on the local machine. The output shows all the tests have passed. Testkube has processed the artifacts and provided them for download. 

Now, let us suppose, this Robot Framework test has some parameterized values to be passed with the test. Don’t worry! Testkube has got you covered. In the Test Workflows, you can pass the parameters and provide custom values. Let us understand this in detail in the coming section.

Advanced Execution with Parameters

Variables in Robot Framework

In Robot Framework, you can define the parameters as variables, and the values can be passed at the time of execution. The default values to these parameters can be set in the Robot Framework test configuration as shown below:

*** Variables ***
${my_var}     my_default_value

*** Test Cases ***
Test Case 1
    Log    ${my_var}

While using the `robot` command line utility, these variables can be passed as parameters as shown below:

$ robot --variable my_var:my_value test.robot

Robot Framework Variables in Test Workflows

Test Workflows allows you to define configuration parameters for your workflows, which can be passed to the Robot Framework as input. Let’s add parameters for the `firstname`, `lastname` and `totalprice` parameters to our workflow:

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
 name: basic-acceptance-test
 namespace: testkube
spec:
 config:
   firstname:
     type: string
     default: Jim
   lastname:
     type: string
     default: Henderson
   totalprice:
     type: string
     default: "100"
 content:
   git:
     uri: https://github.com/cerebro1/robot-framework-test.git
 container:
   image: ppodgorsek/robot-framework:7.1.0
 steps:
 - name: Run test
   shell: robot --variable firstname:{{ config.firstname }} --variable lastname:{{config.lastname
     }} --variable totalprice:{{ config.totalprice }} --outputdir /data/repo/output-{{config.firstname}}-{{config.lastname}}
     /data/repo/advanced-test-restful.robot
   artifacts:
     paths:
     - /data/repo/output-{{config.firstname}}-{{config.lastname}}/*

As you can see we have defined three config variables which we then pass to the robot command, and use in the naming of the output folder. When we new try to run this test we are first prompted to provide values for our variables:

Just pressing the run button uses the default values provided as you can see in the log output:

Integrating Testkube with CI/CD

Although it is useful to run tests directly from the Testkube Dashboard as shown above, you will most likely want to start these tests from your existing CI/CD pipelines. Fortunately, Testkube makes this really easy: 

• Navigate to the CI/CD Integration tab for your Test Workflow (see below)
• Select the CI/CD tool you are using from the menu to the left - a corresponding example will be shown to the right that you can copy/paste directly into your tool

Conclusion

Using Test Workflows, we were able to easily configure the execution of our Robot Framework test, and also pass custom variables to Robot Framework in a seamless way, and automate the execution of our tests from our CI/CD system. The configuration is easy to understand and manage, which not only reduces the efforts of developers but also increases the maintainability of the configuration. 

For a scalable approach to test execution in a Kubernetes environment, Testkube is the solution. It takes any testing tool, provides a template, or allows you to start from scratch and automate your tests. Teams can manage the entire test life cycle in an optimized way from a single dashboard. This not only empowers them with the feasibility of scaling but also standardizes the testing process.

In this tutorial, we have seen how acceptance testing with Robot Framework could be done using Test Workflows in Testkube. Similarly, teams can create multiple test workflows to manage unit, load, API, acceptance, etc., tests from the same platform.

We invite you to try Test Workflows. Visit Testkube website to get started. If you find yourself struggling with anything, feel free to drop a note in our active Slack community, and someone will help you out.