Dogfooding Testkube: How to Test a Testing Framework

People are fallible. We often need to make decisions based on incomplete information which results in unpredictable behavior. Engineers are often trained to limit those mistakes, but we’re still humans - not robots -  and our code is not perfect and can often result in errors.

Engineers (who are also humans - yes - nerds are also humans!) for decades have faced the topic of testing their products. There are numerous tools, frameworks, and techniques for testing code. There are unit testing, functional testing, integration testing, performance testing, stress testing, and many more.

The problem is even trickier when we do things in Kubernetes, which is an “open-source system for automating deployment, scaling, and management of containerized applications.” It handles a lot of the deployment complexity for you but working with it requires a lot of knowledge - and not just for beginners or junior developers, but seniors also need time to become familiar with how everything works.

As Kubernetes is gaining a lot of momentum recently in modern companies, the tooling around it not always is good enough to balance the speed of development and complexity of the platform, so a lot of companies are struggling with in-house tools - they are great to write - but hard to maintain (like any tool). 

We, the Testkube engineers at Kubeshop, also want to test our code and the final product, which is a quite small but complicated Kubernetes service with operators, APIs, kubectl plugins, and various other components - everything merges to reduce the complexity of running tests in Kubernetes cluster for the end-user.

ut wait a minute we've just written a testing tool? right? Why not use it to test itself? Yeah!! - we love testing stuff - It could sound like a crazy idea but for example, the Go language (and many other languages) can be compiled by itself, so why not test our  testing framework with our testing framework? Also, we’ll get some additional feedback on what to test and how it behaves in a real environment.

Considerations

Now let’s get back a little - that crazy idea needs to be planned to make it real. To test our software we’re using a range of testing techniques, starting with unit testing. In Go and almost all modern languages, it’s quite straightforward: you can use the internal testing library and add some assertion library and you're done. 

But it’s only part of the problem - now you can be sure of the quality of the dots (parts of code) in your architectural diagram. What about connections between those dots? That’s the job of integration tests that could be written in a language that other parts are written with - you can take packages and simulate connections between them in the same way as they are orchestrated in real code - but it’s still not the same.

But what to do later? - there is still a lot of complexity that needs to be handled like networking, running Kubernetes plugins, getting results in isolated environments (which Kubernetes is), performance issues (those, fortunately, we don’t have a lot), or other system-related topics. 

Some of the above issues can be addressed by End-To-End (e2e) tests which can take part or the whole of the system and test in a semi-production environment simulating real usage. For more information on how to manage your End-To-End tests, check out our blog on End-to End testing of your Kubernetes applications with Cypress. We decided it would be a good fit for Testkube - as some abstraction over testing frameworks and techniques. 

Some questions arose during this work - what should be the scheduler for running tests? Should we write some shell test scripts or other Github actions manually? What if we want to extend testing suites with different executors? - Probably it would not be flexible. After some ideation sessions, we decided to use Testkube to run our already defined tests against our Testkube clusters, almost all parts were there (like authorizing to cluster with kubectl) so the only missing parts were the installation of the Testkube kubectl plugin and running defined tests.

So we’ve set up our CI pipelines in GitHub and started running tests to test testing tools. 

Testing Our Testing Tool

Testkube is cloud-native based on top of Kubernetes. Additionally, we’ve designed it using OpenAPI spec with API-design first in mind. We’ve started by preparing a testing workflow in GitHub which can run Testkube tests - you need only configured access to Kubernetes cluster with `kubectl` and then install and run Testkube against your test cases to see the test results. 

You can find the full Github workflow step for deployment here.

And for starting tests here.

‍‍Starting test suite with valid in-cluster configuration in GitHub workflows. 

Test Planning & Execution

To start testing you should need a plan - define what are the critical, most important paths in your service, what is the happy path for them, and how to manage automation of feeding test data (this could be tricky in some old systems without APIs or with complicated access - but if standard testing tools are not enough for you, Testkube will help you here too - simply write your own tests executor for test case execution).  

Next, you’ll need to create Postman tests. Several tips could help you write your tests against services in Kubernetes clusters. 

1. To pass data from outside you can use environment variables in the postman 
2. To feed external data use the `-p PARAM=VALUE` flag when starting Testkube tests.
3. You can use the `URI` variable to use the same test code for testing in different environments (e.g. http://localhost:8080 on your local machine, and https://my-service.my-namespace:8080 inside your cluster using internal DNS service) - so it’s almost the same as testing against your local instance of the system. 

Yay! We’ve noticed what has been done here - it’s the first time we’ve built foundations for testing Testkube with Testkube (like in the Inception movie)! 

Let’s get back to test cases - what was the planning/thinking process? 

So to start we need to define some tests - we decided to go with API testing first as it feels natural for us (automatic OpenAPI spec validation in Testkube is still in the phase of designing) - and it's the most critical component of our system - and additionally Postman was in place - it allowed us to make quick prototyping on top of our local instances of Testkube API server. It was a good decision to go with design first with OpenAPI spec - testing is really easy with that - just prepare some data, pass them to endpoints and check results - Postman is your tool here! - tests and assertions can be written in javascript. Additionally you have a templating engine in place that can inject Postman internal environment variables everywhere.

We’ve started by writing test suites against our APIs. They can be easily written and run against local development instances. After everything is designed, we could move test code into clusters. With parameterized test cases, you can easily write tests on your local development infrastructure and move it into a cluster when you’re ready (I was writing tests against a local go language API server running on my local MacBook). 

We’ve created a plan for the “happy path” of our test resource, like creating a new test, running it, and checking if it was run on top of Testkube.Postman has great ability to orchestrate requests and pass data from one to another - we’ve used env variables for that

The environment variables will behave like global variables for a whole Postman test run, you can easily set and get them. We’ve used them extensively to generate random test names and pass created execution ids or test execution names. 

There were also some assertions with the use of the integrated `pm.expect` assertions library in Postman. 

After the test suite was completed I could simply move it into the cluster with a little effort like defining valid cluster endpoints as we’ve already planned that all URIs will be changed in a valid test run on a cluster with a valid Kubernetes DNS based service address.

If you’re interested in the code of our test, feel free to check it here (just import file as Postman collection) https://github.com/kubeshop/Testkube/blob/main/test/e2e/Testkube-Sanity.postman_collection.json it’s just exported Postman collections JSON file that tests against our Testkube API defined in OpenAPI spec

It was nice that Postman - which we knew how to use (as most of us used it to call APIs for some previous development) - can be run so easily in the Kubernetes cluster. 

Everything was run in our GitHub workflows with the use of Actions. The test is run after successful helm chart build and deployment to one of our clusters. 

The output of this step is simply Testkube output which can be analyzed in case of a failing build.

Summary

When you’re testing your code in Kubernetes you’re facing a lot of problems, starting from an isolated environment for your service (where calling tests against your services is very limited), and at the end getting results from your test run is not so straightforward. Fortunately, Testkube can help you there, we can manage these (and probably many more in future) issues for you - test engineers should focus only on testing their code. 

As our work is not completed yet - just some foundations were built - the next steps will be for sure to extend this testing suite to increase coverage and to some edge test cases or even parallel tests running. 

The question always is: was it worthy putting quite a big effort into such tests? - the answer (like for most cases) is only one - YES! For us, 123 failed builds were found already! So it was great to have really fast feedback before our users notice any issues.

Additionally, the second step will be also to test our frontend app - but here we’ll use a different tool for testing React-based applications - look for part 2 of this post ;). 

Why not give it a go yourself? Sign up to Testkube and try one of our examples or head over to our documentation - if you get stuck or have questions, we’re here to help! Find an answer to your questions in the Testkube Knowledge Base or reach out to us on Slack. We’re eager to hear how you use our integrations!