Everything as Code - Test distribution with K8s and Testkube

DEVOPS DAYS WARSAW

Hello. It's really great to be here today. So thanks for the invitation, and let's start.

Everything is code. Test distribution with Kubernetes and Testkube open source. I will show you how to manage your tests, manage your execution, environments with the combination of Kubernetes and Testkube.

But first, few words about me. I'm Tomasz Konieczny. I am a test architect and quality consultant. And in general, I'm interested in various aspects of quality, but mainly, test automation, architecture, and infrastructure.

And today's presentation will be related to all of these. I'm also a conference speaker. I speak at conference conferences quite a lot. And I currently work as QA Lead at Testkube.

So I'll be able to tell you a bit more about how this testing tool is actually tested.

But first, let's start with something more, generic.

Let's start with one of the problems that are pretty common nowadays. I mean, the business requirement to deliver something fast and keep it high quality at the same time. I mean, you definitely know this one really well because that's often the goal.

And it's a problem from the technical perspective, at least, because it's like trying to combine two completely opposite things. I mean, doing something fast and keeping things high quality, it may be challenging.

But at the same time, everyone should be aware that's a prerequisite for a successful product. I mean, we need to deliver something that will meet user expectations that and user users can be pretty demanding nowadays. And at the same time, we need to do it fast because otherwise competitors will do it first.

So from quality perspective, we can simplify this goal to fast and complete quality feedback because that's the first thing we'll try to achieve here, getting feedback as fast as possible. And we'll try to achieve it in every way possible. We'll be trying to shorten quality feedback loop in every way possible by, for example, shifting left, trying to test earlier in development process, trying to detect any discrepancies or issues as soon as possible, often before the call is even implemented, testing other requirements, thinking about quality assurance anytime, just to detect these issues earlier.

The second thing is automation. I mean, you know this one very well, automating everything that makes sense at least, because automation can provide can make repeated actions, more consistent, more predictable. It can save time on them. And at the same time, if you think from a quality perspective I mean, for example, automated tests, they can be way faster than manual ones.

So that's why it's used so often.

And trying to achieve continuous feedback with all of these things mentioned. But at the same time, we need to make this feedback complete because getting feedback fast, getting feedback at any stage possible possible isn't enough. This feedback also need to be complete. So having tests on very different levels, having very different testing tools, and that's, trying to achieve the best test coverage possible just to get feedback that will be complete and, meaningful. But at the same time, there is a problem. I mean, adding all of these different tests means introducing different testing tools. What about this additional complexity?

And all of these different tests need to be implemented. They need to be maintained.

And the problem is, that's not all. What about infrastructure and pipelines?

Because we definitely need to run these tests somehow. And with very different test testing tools, we may need to do it in different ways. And in the end, implementing these things from scratch can easily become more complex than the actual test implementation.

And infrastructure and pipelines are just a prerequisite, something that we need to run these tests. It's not the goal. And today, I will show you how to simplify at least this part with the combination of Kubernetes and Testkube.

What is Testkube? It's an open source test orchestration and execution framework for Kubernetes.

So to simplify, Testkube provides you the the unified way of executing the test no matter which tool you use. So it provides the set of preconfigured executors. So these are the maybe I'll simplify the configurations for different test runners. So Postman, k6, Cypress, JMeter, etcetera. There is quite a lot of them. And also the last one named custom container executor, which is probably the most powerful of these, that allows you to run any Docker image.

But what's so special about it? I mean, it's not that difficult to run a Docker image.

The second part is that Testkube provides test-specific abstraction. So no matter which of these testing tools you will use, no matter if you will use this custom container executor and run your own Docker image, you will still get the test-specific abstraction. And you can interact with Testkube in exactly the same way no matter which tool you will use. What's this test-specific abstraction?

Custom Resource Definitions (CRDs)

Testkube is Testkube introduces, some additional test types on top of the ones available by default, for Kubernetes.

So that's, for example, the test. That's the simplest thing possible you can execute with Testkube, the BMB. For example, the single test. It may be some project. It may be just, Docker images that will execute some actions there. Everything can be simplified to the test. That's just the simplest thing possible you can execute with Testkube.

TestSuite, so that's the group of tests because, usually, you will need to group these tests somehow, to, execute a complete set.

Executor, so that's the cluster, executor config. So if you have some custom test runner that that's not officially supported, you can define the new custom executor, or you can adjust the existing one to meet your needs.

Test Source, which allow you to abstract away, checking out the test from the repository. So, for example, that's quite common thing to have multiple different tests or testing tools in the same repository. You can use the test source to abstract the way how they will be checked out from this repository.

Also, TestTrigger and Webhook. I will show this later.

Setting Up Testkube

But let's start with setting up Testkube first. What's even needed to run it? So there are prerequisites.

And if you already use Kubernetes, you will probably already have this installed. So that's kubectl and Helm.

And also, you definitely need to have some Kubernetes cluster. If you already have some cluster used for testing purposes, it's possible you will be able to reuse it because Testkube supports namespace installation. If you don't have it, you can or do do if you want to try things locally, you can use KIND or Minikube. That's probably the easiest way, to just try things.

Then you need to install Testkube-specific things. So the first dependency is a CLI. It's just the kubectl plugging that allow you to use this test script specific abstraction in easier way.

And, also, we need to install Testkube cluster-side components, in your cluster.

Creating a Kubernetes Cluster

So let's start with creating some cluster locally first in case you don't, have it. In this case, I just used kind because because based on my experience, it's just more stable and predictable than, Minikube. So kind create cluster, optionally name. Let's name it testkube-kind-cluster.

Enter, and it's created. The context is also set.

Now it's time to install a Testkube cluster site components. And if if you have CLI installed, you can use Testkube install command that will do everything you need.

It will make sure the context is right. You just need to confirm it. Optionally, you can also specify the namespace, but by default, it will be namespace testkube.

Alternatively, it's possible to use Helm directly, if you prefer this way, but it's easier to use a Testkube install.

Let's check what's installed there, what are the deployments.

So kubectl get deployments, namespace test cube, and we have a few Testkube components installed here. Let's start with Testkube dashboard. That's Testkube's web UI that allow you to manage it, manage your tests, execute them, check the results, and things like that. I will show you it, later.

There's also test Testkube API server. So that's basically the work orchestrator, and it manages the executors, gathers logs, processes them, etcetera.

And the last one that's important here is the operator control manager. So it manages and watches Test Script cluster resources and then, communicates with API server to trigger this.

k6 Test via CLI

So it's ready. Let's finally run some tests. And the first example will be the, k6 test. If you don't know it, that's pretty powerful, load testing tool.

So we have this test in the repository. It's open source, so if you will get if you will go to the Testkube main repo, to the test directory, you will find different tests there. These are the actual tests that are used for testing Testkube, so it's not just an example. That's something that's actually used.

And we want to execute it with Testkube, so we need to create it first. And it's possible to create resources with Testkube create command. In this case, we'll create the test. So Testkube create test.

And then name, which is mandatory.

So let's name it k6 example because we'll be executing the k6 test.

Then the type. So the type is the executor that will be used here because we need to, tell Testkube which kind of test we'll be executing, which executor config should be used. And because we have the predefined k6 executor, we can just use k6 script here, and it will be used.

Then test content type. So we need to tell the Testkube how to get this test. And because it's in the git repository, we'll just use git here, and then we have git-related settings. So git URI, that's the main test script's repo. Git branch, let's keep it main. And git path. So that's the path for this specific file that will be executed. Address file. One of k6 tests that we have in the repository. So test k6, executor test, etcetera. Just some JS file. Enter. And we have info that test has been created correctly.

Let's check what's there. It's possible to get any Testkube resources with Testkube get command. So in this case, we'll get the test. Testkube get test. K6 example, and we have the details for the test that we just created. And additional to that, we also have the labels, which are optional. It's possible to define them, but they may be used on some cases.

And it's also possible to use kubectl describe test directly because, everything here is still Kubernetes-native. So if you prefer this way, it's also possible. We have the test created. Let's finally run it. And as you can probably expect, there is a test group run command, and it allows you to run test or test suite. Then the test name, so it's k6 example, and the -f option is just to follow to get the results that logs directly after the execution.

What's also possible with CLI? It's possible to manage resources. I mean, creating, editing, deleting them. And it may be test as in this example, it may be test suite, test source, executor. Everything every single Testkube resource can be managed with CLI.

It's also possible to run what and about execution for both test and test suite. It's possible to download artifacts. So, for example, if your test produces some kind of artifacts, like, for example, you have Cypress tests that will generate some report or some screenshots in case of failure, you can use CLI to download all of these. And it all also allow you to manage the Testkube. That's how we installed it in the in the first place. But it's possible to also upgrade it, etcetera.

And there is also one more command that's pretty useful, at least initially, and that's Testkube dashboard, Because it allows you to easily display display Testkube's web UI without configuring ingresses, without doing port forward manually. You can just run Testkube dashboard, and you will get the web UI. And everything I will show here is possible in three different ways. I mean, every single action is possible with both CLI and and dashboard, so you can use, whichever you prefer. And so we can manage the tests here, view them, create them, check the results, etcetera. The same for test suites and executors, triggers, webhooks, etcetera. And you can also manage the Testkube itself.

Postman Test via the Dashboard

So let's create a test with a dashboard this time, and this example will be for Postman. Pretty popular tool. You definitely know this one.

So name is, again, mandatory. Let's name this test postman example test. But this time, we won't have this test in the repository. We'll just have this file locally.

So first, let's choose the type. You can check the list here. There is the, executor config for Postman, so Postman collection. And then the source.

Because we have this file locally, we can use file as a source. So it will be, uploaded and then executed, and then the file that will be, executed. Alternatively, you can also use, the git file source here, or you can even use string. For example, if you have some curl test, you can just paste a string, and it will also be then executed.

So let's create this test. It's created.

That's the test view for the new test. It's not executed yet, so let's do it. I did it multiple times already just to show you how it will look like with some additional stats here. So just the basic ones, pass/fail ratio, some metrics, percentage, etcetera. And the executions, it's possible to just view this easily to download artifacts here. And also, you have something additional, like CLI commands. So you can just check how the CLI command would look like for this specific thing you might want to execute here.

But there are also some additional test settings here here that are possible. So you can set labels. You can set additional description, then test-specific settings, execution, so that will be comments and arguments. You can set these. You can override them. Variables and ends.

Scheduling, so chron-based triggers. And also definition, which may be pretty useful. I will show it, now.

Cypress Test Using CRDs

So the third way of interacting with Testkube is using CRDs. So that's the Kubernetes-native way.

And let's start with creating another test. This time, we will create a test for Cypress, which is a bit more complex because, as I mentioned, all of these tests are actually used for testing Testkube. And this specific test is used for testing, the executor for Cypress, so we want to make sure that everything works fine there, that it's possible to set ends, variables, etcetera.

So let's start Kind test, so that's the type that test queue provides. And then metadata name. Name is, again, mandatory. So let's name it Cypress test example because it's just some example for Cypress. And then the namespace, you can put it here if you have, just a single namespace installation. Or if you have multiple environments at the same cluster, you may want to, set it on KipsView data apply.

Labels, optional. And then the spec. So, basically, instructions how to execute the specific test.

So type Cypress is, again, the executor type that will be used. We need to tell Testkube that it will be executing the Cypress test. And then the content, so instructions how to get this test. And in this case, we'll again use the git repository. That's, again, the main task repo where you can find all of these. So type git and repository, then repository settings, URI, main task repo, branch, main, and path. That's the path, but not for the single test in this case, but for the complete Cypress project.

So we have, different, service projects in this directory.

Let's choose the one for Cypress 12 just to make sure it will work with the specific version. And then execution request. So additional settings that we might want to set here. And it's pretty powerful. In this case, we'll just set the variables, so Cypress custom env to Cypress custom end value cluster to, check if it's set set correctly on the executor side, and also arcs cluster to make sure they are passed correctly. That's just very simple sanity test that we use.

Additional, option, config video false, just not to waste the resources or disable, Cypress Video recording.

And in addition to this, execution requests allow you to override everything that you need. You can customize the job template here and, for example, set resource requests, resource limits, and everything else that may be needed. So let's save this step in the YAML file. Let's name it cypress test and kubectl apply -f cypress test yaml. The test is created.

But writing these definitions, at least initially, may be a bit, challenging, and it may require, checking the documentation pretty, often. So it's also possible to generate this this definition in different ways. So in the first example, we created the k6 test with testkube create test command. And you can just append CRD only, parameter there, and you will get the definition without creating the test.

So we can use CLI. You can, adjust everything you need there. You can append CRD only, and you will get the definition you can then adjust and use. It's also possible to do the same thing with dashboard because, as I mentioned, you can interact Testkube in any way you prefer.

So if you will go to test settings and then definition, you will also get the definition here. You can even edit it directly here.

But as I mentioned, the custom container executor is the one that's the most powerful because it allows you to exit any Docker image that you may need. So let's assume this Cypress test needs very specific Cypress version to be executed. Let's assume it's 12.7. And we don't have this version, predefined. We don't have predefined copy for this specific version, so we will define the custom executor.

And that's kind executor.

And then metadata name is, again, mandatory. Let's name it in some descriptive way so we'll know what it will do. So container executor cypress 12.7, and namespace, optional, and then spec. So instructions how to execute things.

So first image, Cypress included 12.7. I just used the official Cypress image that includes everything, like browsers, everything that's needed to run this test. Executor type, container, because it will be the container executor. And also types, so that's the name of the executor we'll be we'll be able to then use for the test.

So let's name it container executor cypress 12.7/test.

And in this case, this definition is pretty simple, but, you can customize everything you may need here. So command, arguments, I mean, almost everything is possible here.

And then we have this, definition applied. We can just update our definition for test. And all we need to do is to update spec type and change it to the type we just defined. So container executor Cypress 12.7/test. And we can run these tests with this specific executor that we just defined.

But you will rarely have just a single test in the project. I mean, it's pretty common to have very different test types, very different tools, And that's what test suites can be used for.

It allows you to group tests and execute them as a whole. So there is an additional type that's named KIND test suite. And let's run all of the tests that we just created, so k6, Postman, and and Cypress.

And let's start with metadata name. It's mandatory as always. Example test suite.

Labels, optional. And then spec, so instructions how to execute these specific tests. What are specific steps that we want to execute?

So spec steps keyword. And we have something like stop on failure. We we can decide, do we want to stop the execution if specific step fails or if we want to continue it. So we have two steps here, and then we have execute test.

And in this the first step, we have the k6 example, and then we have the second one where we have execute with two different test steps because we'll execute Postman and Cypress test in parallel after the first one will be, finished. So it's possible to to execute the test in any way, you may need, both sequentially and in parallel. You can control this whole execution flow with test suites.

Triggering Tests from the Pipeline

But what about pipelines? How can we integrate test with existing ones? How can we trigger these tests easily?

So there are a few options here. You can run the test manually. That's still a pretty popular option. You can use use dashboard or CLI.

That's gonna be cron-triggered. It's, again, a pretty popular option because even if you will have pipelines, you may still want to cron-trigger some things.

You can integrate Testkube with your existing pipelines. We have, for example, GitHub Action that allow you to do it easily, or you can just use Testkube CLI and do it this way. But the most interesting way, and, one I will show you now, is to use cluster events, to run tests based on cluster events. That's what's possible with test triggers. They allow you to run tests on Kubernetes cluster events when something happens inside the cluster.

So let's assume we want to trigger k6 example test that we created, before, when we'll deploy some service. And just to simplify, it can be just NGINX and default namespace.

So let's create a trigger for that. Kind, test trigger, and metadata name. Let's make it as descriptive as possible because, this can get pretty complicated easily. So trigger k6 example on NGINX creation. It should be, fine.

Spec resource deployment. So that's the resource that will trigger, some action. Resource type deployment. Resource selector, that's how we select the specific resource. That will trigger some action. So name nginx will just choose this service by the name nginx in the default namespace, so that's namespace default.

Event created, so if this deployment will be created. And then what will happen? Execution, test, and action run. So the test will be test will be run.

And which test? The one defined by the test selector. So name, k6 example, and namespace testkube. So k6 example test in the test cube namespace will be triggered if we deploy NGINX in the default namespace.

Let's apply this trigger, kubectl apply -f example trigger yaml, and let's check how it work. Let's create this nginx deployment.

Kubectl create deployment nginx image nginx, just to get a keep it as simple as possible.

We have the confirmation that the deployment has been created, and that's the exact moment where the Testkube will trigger k6 example test because some action happened on the cluster side. And that's probably the the easiest and the most powerful way to build pipelines and automate at least some easy actions that are predictable. For example, we can use it to trigger things when we deploy a new version of our application or if we update some dependency. All of these can trigger action.

And in this example, I just mapped name to name, so a service with name nginx to test name, case, example. But it's possible to do any mappings here. I mean, we can map name to labels. So, for example, service with name will trigger tests that share some label.

So that's how we can run multiple different tests when something happens on the cluster side, or labels to name. Any resource with specific label can trigger some test, or even labels with labels. Any set of the services, any service can trigger a set of tests by labels. So it's pretty powerful, and, it it may be a bit unusual initially, but it's really powerful.

And, basically, all of these CRDs I showed you I mean, it's possible to do those tests given very different ways, but I just prefer to use CRDs because that's the Keytop's compatible test management. I mean, you can put specific definitions along with the test in the repository.

So, for example, here we have some k6 tests. We have three tests here. These are, again, the actual tests that are used for testing test executors in this case. And there is also one more directory named CRD.

That's why I where I put everything I may need to execute this test. So not only I have the test versions, but I also have the version instructions how to execute. So definitions for test, test suites, executors, triggers, or webhooks, whatever I need in a specific case. And that's also what we do in a DataScript project, because these three examples I showed you are the real tests.

We have, the test directory in the main repo, so we can check these.

And you will find very different directories there. I mean, Artillery, Cypress, Gradle, etcetera. For every single executor that we officially support, we have some example tests or products there.

In some cases, like, for example, Cypress, we even have five different versions of Cypress products there. And along with all of these, in every single directory, we also have the CRD directory, where we have the instructions how to execute this specific test. We have executable definitions. We have triggers. We even have webhooks just to post the results somewhere else.

And that's an example how powerful it can be when you match a test with specific instructions how to execute them. And it may be pretty powerful if you, want to do things on prem. You can run it entirely on your cluster side. You don't even need some external pipelines.

And if you would like to see more examples, I really recommend you to go to the Testkube documentation, so docs.testkube.io. All of these examples are based on real tests that we use, so they should be always up to date. And I also recommend you to go to Testkube's GitHub.

So, that's where you can find the examples that I showed you today. And if you ever go to this test directory, you will have very different cases there, that are way more complex than the one I showed you. Because, for example, there are examples for, overwriting top templates where we can customize almost everything, setting timeouts, setting resource request limits, etcetera, all you may want to do. So these more complex examples are also available there. And that will be all. If you have any questions, I can ask them now, or we can also discuss things after, the presentation.

Thanks.