.webp)
How to Scale Automated Testing Beyond CI/CD Pipelines
Table of Contents
.webp)
Transcript
Introduction
I'm Ole Lensmar, CTO at Testkube. I don't want to talk too much about myself, but my background is that I've been in testing for over 20 years. My claim to fame in the testing space, apart from Testkube, is that I wrote SoapUI about exactly 20 years ago. It was a popular tool for API testing. I was at SmartBear for many years, worked with the Swagger ecosystem, and was also the chairman of the OpenAPI Initiative. Now I'm happily CTOing away at Testkube, four years in, and I'm excited to talk about some of the challenges in this space that we're trying to address. I'll answer questions at the end.
What I really want to talk about here, and this comes almost to the question of why we started building Testkube, is that CI/CD tools and pipelines are ultimately, at least in my experience, not great at testing at scale. CI is built for integration and CD is built for deployment, and testing kind of fits somewhere in there. But when you look at needs specific to testing, around reporting and scaling, that's where these tools fall short. That has become more and more obvious with recent changes in how we build applications.
Why we need more automated testing
As applications, teams and organizations go cloud native and embrace AI, we're seeing a general need for more testing, and more automated testing specifically. The trends driving this are microservices, containers and Kubernetes as a container orchestration platform. Instead of testing your application as a whole, you're now testing individual microservices, which increases the number of tests you want to run and the complexity of how you run them when those services are deployed in Kubernetes.
We're also seeing a shift to dynamic and ephemeral infrastructure and infrastructure as code. Kubernetes makes it relatively easy to spin up infrastructure for testing or platform needs, which opens new possibilities for testing, like running integration tests in an ephemeral environment instead of a shared staging environment.
Cloud native also introduces a move to asynchronicity, GitOps and progressive delivery. Instead of one tool like Jenkins running your entire pipeline from code to deployment, you now have multiple tools strung together by events. That raises new questions: how do you inject tests into these pipelines, how do you coordinate them, and how do you get consistency?
Of course I have to mention generative AI. At LeadDev in Berlin I had several conversations with engineering managers trying to figure out how to keep up, from a testing perspective, with the increased velocity of code being generated by AI. And if you're building AI agents or LLM-based applications yourself, those need to be tested too. There's a new breed of testing tools around evals that also need to be automated, with trends tracked over time. All of this adds to the overall burden of testing.
Testing happens everywhere across the pipeline
I'm sure you've seen this picture many times. Testing has been shifting left for years, earlier in the SDLC, but we're also seeing a shift right, testing in production or collecting quality metrics in production. Testing is part of planning, you test locally while coding, you run tests when building, in the testing phase, before and after release, during deployment and progressive delivery, while operating, and in production. Testing that used to be confined to CI now happens everywhere across your pipelines.
And these pipelines multiply: you have one for every application and every microservice, deployed across multiple environments. As you adopt new testing tools, migrating from Selenium to Cypress or Playwright, from JMeter to k6, or from SoapUI to Postman, the sprawl of tooling and the ways you trigger tests grows too.
Generative AI adds even more. AI-generated code and configurations mean more code changes and more tests in your pipelines. You'll probably also need more end-to-end tests. AI can be reasonably good at creating unit tests for the code it generates, but it's not always great at determining the ripple effects of its changes on the rest of your system. That's why you need end-to-end tests in place to make sure AI-generated changes don't break something elsewhere. You should have those tests anyway, but AI makes it more urgent.
People are also using AI to generate tests, converting manual test scripts or requirements into automated tests, which adds to the number of tests you need to run and can further clog your pipelines. And you'll want to test AI components themselves: running evals against agents to make sure they're accurate, not hallucinating and not biased, integrating those tools into pipelines, collecting their metrics, and instrumenting AI components for testability, for example with OpenTelemetry.
Testing is continuous
What this boils down to is that testing is everywhere and it's continuous. It has to happen across every stage of the software delivery lifecycle. It should preferably be automated. I'm not saying you shouldn't do exploratory testing, but for deterministic test cases, automation is a great start to accelerate delivery. You need test observability, meaning logs, results, metrics and analytics from every test you run and from the system under test, which can be especially tricky in cloud-native environments. And many teams embrace quality gates to promote components or groups of components across delivery stages.
No presentation would be complete without a Venn diagram. Continuous integration, continuous deployment and continuous testing are all part of the continuous delivery paradigm, focused on automation, integration and observability. But CI and CD don't cater to the needs of continuous testing. Scaling automated tests, troubleshooting and analytics, gatekeeping and risk mitigation are not what CI/CD tools are built for. They're great at building and deploying.
Continuous testing challenges: consistency and observability
So let's all do continuous testing. But what are the challenges people bump into today? One is consistency of test execution. You've heard "works on my machine," and now there's "works in my pipeline." Companies often use multiple CI/CD tools and different pipelines for different purposes, and they run tests locally, in progressive delivery and elsewhere. You want the same result regardless of where you run a test, not a different outcome because one environment uses a different tech stack. You want that consistency across geographies, testing stages, local and ephemeral environments, and cloud providers, with the same configurations and network policies everywhere in Kubernetes. And you want it across asynchronous pipelines and CI/CD tools, whether tests run in Jenkins, Argo CD, GitHub Actions or GitLab.
Another challenge is observability. How do you collect logs, artifacts and results from the tests themselves and from the system under test, when an integration test might touch tens or even hundreds of microservices? How do you make those easily available for troubleshooting and reporting? How do you correlate failures with system events? Say you roll out a new version of an infrastructure component like NGINX and your tests start breaking. Can you correlate those two disconnected events, when one didn't trigger the other? And you need centralized tracking of quality metrics, so regardless of what tests run where, you can track pass/fail ratios and how teams are doing.
Infrastructure and process challenges
Another challenge is promotion with quality gates: what is ready for release, and where? Applications today are built from many components that are rarely deployed as a whole. They're rolled out individually or in groups. When can I roll out an update to microservice X when it depends on other microservices? What tests do I need to pass? How do I validate a group of interdependent updates and flag them correctly so they can be pushed to production automatically or manually? How do you inject those controls if you're using a tool like Kargo or Keptn to manage deployments? And this isn't only functional. You might want quality gates for performance, security and pen testing, any kind of quality of service.
A very common challenge when teams shift to cloud native is moving from synchronous to asynchronous delivery. With Jenkins, one tool builds, tests, deploys and maybe promotes to production. In a more cloud-native world you might complement Jenkins with GitLab, GitHub or Bitbucket to build artifacts, add a GitOps tool to deploy them, and a progressive delivery tool like Argo Rollouts to roll them out across environments. How do you inject testing into all of these, and keep consistency in how you run tests, report and troubleshoot? People come to us saying: I'm running Playwright tests in Jenkins, we've moved to GitHub and want to use Argo. How do I run my Playwright tests from Argo, and use my Postman collections to validate progressive delivery? It's doable, but it's a shift in the mechanics of how you test.
Infrastructure testing and platform enablement
Infrastructure testing is another big topic, especially in cloud native. You don't only need to validate the applications you build. You also want to validate the infrastructure you deploy them on, driven by infrastructure as code, GitOps or Terraform. If you provision an ephemeral environment, you want to make sure it has the correct network policies, the right Prometheus instance or whatever components you need before you run any tests. In a platform engineering context, platform teams provisioning environments for internal teams will want to validate that infrastructure before handing it over. If you're provisioning infrastructure for AI workloads, you want to make sure GPUs and memory are available and that network topologies and constraints are in place. This is driven by how easy infrastructure as code makes provisioning.
Finally, platform enablement. Platform teams strive to self-enable the teams in their organization to deliver new applications and components. When a team provisions a new cluster to build a new microservice, they need flexibility in testing it: which tools to use, how to run them, how to build the deployment pipelines, which metrics to collect, and whether to notify on Slack or Teams when tests fail. In large organizations the diversity of tooling varies a lot, and providing a consistent platform scaffold that caters to that diversity is a real challenge, especially for testing. You'll probably also want to integrate this into internal developer portals like Backstage.
Approaches to continuous testing
So continuous testing is hard. It touches many different aspects, it's being driven aggressively by developments in the space, and testing isn't always top of mind until it's maybe too late. Scaling, consistency and CI/CD bottlenecks can all block a continuous testing strategy.
So what's the goal, the nirvana for continuous testing? When we talk to customers, they want their teams to be able to launch any test, at any time, in any environment, without being blocked by CI/CD pipelines, waiting for a build to finish, or hitting concurrency limits. They want dev, QA and ops teams to have one platform for running, scheduling and monitoring tests, instead of some teams running tests in GitHub Actions, some in Jenkins and others elsewhere. That fragmented experience is hard to manage, especially for access control. A unified view matters both for the teams and for leadership who want to know what tests are running where, which tools are used and which tests consume the most resources. They want generative AI fueling their SDLC pipelines, not clogging them. And they want platform teams to be able to self-enable the teams they serve to test consistently.
How do people get there today? The most common approach is to leverage the CI/CD tool they already have and write a lot of scripts in GitHub Actions, Jenkins or GitLab. The upside is that everything stays in one place and is tightly integrated. The downside is that it doesn't scale well and is high maintenance, with lots of scripts and dependencies to manage. As soon as you introduce a new CI/CD tool, which happens, especially with GitOps tools like Argo, you have to replicate that behavior. And CI/CD tools aren't test-centric, so they don't give you centralized test reporting.
The next step is integrating point solutions on top, like a reporting tool or a test management tool. You get the right tool for the job, but you end up with fragmented infrastructure glued together by scripts, API calls and webhooks across multiple tools, which is hard to keep track of, especially as you add CI/CD tools.
Finally, you can use a dedicated platform for continuous testing. The benefit is that it's built to solve these problems. The tradeoff is that a general platform will never go as deep as you want in specific areas, so you might complement it with, for example, a business intelligence solution for really deep reporting.
Introduction to Testkube
This leads me to Testkube and how we think it solves many of the problems customers struggle with when adopting a continuous testing strategy. At a high level, Testkube is a continuous testing platform. It maintains a catalog of all the tests you run across your infrastructure, whether they run locally, in CI/CD pipelines or as monitors in production. It has an execution engine built on Kubernetes to ensure consistent and scalable test execution, leveraging Kubernetes constructs for scalability, resource allocation and test management. Because it runs your tests, it collects all results in one place for observability and troubleshooting, and provides analytics, insights and reporting. It strives to unite all your test orchestration and execution under one umbrella.
Testkube does not replace your testing tools. It takes your existing tests: end-to-end tests in Playwright or Cypress, load tests in JMeter, Artillery or k6, unit tests in pytest or JUnit, security tests, AI tests with DeepEval. Whatever testing tool you use today, as long as it can run in a container under Kubernetes, Testkube can run it for you, including custom scripts.
What makes it different? Test execution is extracted from your CI/CD pipeline. Instead of Jenkins running your tests, Testkube runs them. You can still trigger that from Jenkins, and the most common use case is running tests from your CI/CD pipeline while delegating execution to Testkube, which runs and manages those executions more efficiently than a CI/CD tool. It's tool agnostic. The main limitation is hardware-related testing, like embedded or iOS devices, which you'd still run on a device farm, though Testkube can orchestrate tests running there. It provides centralized test observability with a dashboard. It's Kubernetes native. It's GitOps ready, so you can manage all your Testkube primitives through your Git repository with Flux or Argo. And it's AI enabled, with an MCP server and AI functionality in the product itself.
Test workflows
Testkube abstracts the tests you run as test workflows. A test workflow is an abstraction for any kind of test in your Kubernetes cluster: functional, performance, chaos, security, infrastructure or AI tests. Some people look at our workflow engine and say it's just another workflow engine. It is, but it's fine-tuned for testing. It has primitives for things specific to testing and testing tools: sharding, parallelization, composition and reuse, multi-cluster orchestration, services and templates. You won't generally find those in CI/CD tools, because that's not what they're built for. Testkube has a rich vocabulary for testing, maps it to Kubernetes to execute it, and manages workflows as CRDs in your cluster, so you can fine-tune them for security and resource utilization.
Testkube architecture
Testkube has two main components. The Testkube agent is installed in your Kubernetes cluster and runs your tests. Testkube always runs in your infrastructure. It's not a cloud-hosted service that runs your tests in the cloud. It uses the capacity and resources you allocate, and you can have as many agents as you want: one in your development cluster, one in staging, one in production, agents in different regions, or in different namespaces if you're security-minded.
The more agents you have, the more you need to orchestrate and manage them, and that's where the control plane comes in. The agent is open source. You can grab it on GitHub, install it with Helm or our CLI, and use it through a CLI and REST API. It's a great place to start exploring the engine. For higher-level orchestration, RBAC and AI features, the control plane is our commercial offering. It's also available on-prem, and many customers run it in air-gapped environments, but we can also host it for you in the cloud.
Platform integrations
How does Testkube tie into platform engineering? It provides a common platform that lets individual teams choose whatever testing approach they need. Because it's tool agnostic, teams have a high degree of flexibility in which testing tools they use and how they trigger tests from their pipelines, whether that's GitHub Actions, Argo CD or Jenkins. Reporting is flexible, and so is who has access to what. Platform engineering can use native Testkube constructs to enforce consistency across teams, such as the same image repositories, Docker images and testing tool versions, with workflow templates, webhook templates for integrations, and RBAC for access. There's also a Backstage plugin if you want to surface Testkube results in your internal developer portal.
Testkube also fits nicely into GitOps. It supports pre-sync validation, for example of infrastructure; in-sync validation of each resource as it's deployed; and post-sync, maybe the most common, where you run tests after a sync wave finishes to validate that everything deployed successfully. It works with both Flux and Argo.
So why use Testkube? It empowers engineering to do any testing they need, lets you leverage AI for test orchestration, gives you a single pane of glass for all test results, and is cost efficient. Because it runs on your infrastructure and isn't consumption based, you can run as many tests in parallel as you want, limited only by the infrastructure you provide. And it ties into platform engineering constructs.
Product demo
Let me do a really short demo of the sandbox environment available on our website, which you can explore for free. These are example test workflows showing how you'd run different types of tests with Testkube: k6, Playwright, Cypress, JMeter, Postman, Robot Framework and more.
Here's a JMeter test. We can look at individual results, and at the artifacts JMeter produces, like the HTML report Testkube collects for you. We can see the CPU and memory consumed by the load test, and track that across executions. I can run the test by clicking Run at the top right, and now it's running in our Kubernetes infrastructure. If I want to run this test from GitHub Actions, CircleCI or Jenkins, Testkube gives you the commands to put into those pipelines. You can also run tests on a schedule, like a load test every day at 3 a.m. to measure performance, or every five minutes as a monitor. We can follow the output while it runs, and it finished quickly with results right here.
For integrations, we can push results to Grafana, Prometheus, Datadog or New Relic, notify on Slack or Teams, trigger an event in PagerDuty when a test fails, or create an issue in Jira. With webhooks you can integrate with almost any system that has an API.
On the reporting side, there's an extensive reporting module where you can drill into tests based on how you label them. For example, a CPU usage report shows how much CPU all my tests used in October, which I can translate into actual cost with my cloud provider, then drill into which tests use the most. You can customize these boards any way you want.
One more thing: a workflow in Testkube can run other workflows, so it's easy to compose multiple tests into more advanced scenarios. You could run an end-to-end test while a load test runs to make sure the load doesn't affect functionality, or run a security test under load. You can run tests in sequence or in parallel, trigger the composition from any CI/CD tool, and keep all the reporting in one place.
Wrap-up
Thank you so much for listening to why CI/CD tools aren't cut out for testing in a continuous testing world. If you're at KubeCon in Atlanta next week, we're hosting KubeJam, a spontaneous jam session where you're welcome to join us on stage and play. If you can't make it to Atlanta, we'll do the same at KubeCon Amsterdam in March.
Q&A
How does Testkube support testing applications that aren't cloud native, such as legacy apps running on Oracle WebLogic?
Great question. The agent that runs tests runs in Kubernetes, but the system under test doesn't have to. As long as there's network connectivity from where the agent runs to your system under test, you can test systems running on VMs, on WebLogic or anywhere else. If you're running a JMeter or Playwright test, it hits HTTP endpoints, so as long as you can reach that network from the Kubernetes environment running the agent, you're fine. It's actually a very common setup. In almost every case we've seen, there was still some Kubernetes infrastructure that could run the agent. And when there wasn't, teams could spin up a Kubernetes cluster on AWS to run the agent against their systems, with Testkube hosting the control plane. You do need some Kubernetes instance somewhere to run the agent, because we don't run tests for you in our cloud.
Has Testkube been tested with less common cloud providers, like Tencent Kubernetes Engine? Is there a list?
We have a large number of enterprise customers running a wide range of Kubernetes distributions, and we haven't hit one where Testkube can't run. The agent itself is very lightweight, and most testing tools come from a non-Kubernetes world, so they don't have demanding requirements on their interactions with Kubernetes. If there's a specific provider you want to evaluate, we'd support you through that.
Can we run the entire load in our own system, since we need it isolated and very secure?
Yes. This is one of the core values of Testkube. It runs entirely in your infrastructure, including air-gapped environments, and its capacity is constrained only by the infrastructure you allocate to the agents. If you need to generate millions of virtual users with k6, Testkube can orchestrate that test and fan it out across 100 nodes, as long as your nodes have enough CPU and memory. We have customers doing exactly that.
For large organizations with multiple dev and QA teams, how do the Testkube dashboard, RBAC, teams and resource groups support centralized visibility and collaboration across shared testing environments?
There are a few ways to map Testkube to your organizational structure. You can keep everything in one environment and use RBAC and resource groups to control who has access to what, assigning members to teams and giving a team access to specific load tests or to the tests for a specific component. Or you can create one environment per team, which works well when tests are never shared across teams, and still use RBAC within each environment for more granularity. Our documentation has a guide on these strategies.
Is Testkube in production yet?
Yes. Testkube has been around for four years, we raised a Series A a couple of months ago, and we have 40-plus commercial customers, including large companies like Adobe, Workday and UWM. These are deployments with hundreds of users running thousands of tests every day, across all kinds of test types. Feel free to reach out to me directly or in our Slack community with any follow-up questions.
