AI Test Automation: Fixing the Three New Testing Bottlenecks

Engineering teams are shipping software faster than ever. AI assistants write features in minutes, pull requests pile up, and release velocity keeps climbing. But anyone running a CI/CD pipeline already feels the catch: testing hasn't kept pace. More code means more tests, more tests mean longer pipelines, and longer pipelines mean more failures to triage. The tooling that was supposed to make teams faster is quietly creating new bottlenecks downstream.

In a recent webinar, Testkube CTO Ole Lensmar walked through how Testkube AI tackles this. Instead of bolting a chatbot onto a dashboard, Testkube builds AI directly into the testing platform, so it takes part in your pipeline like any other part of your stack. Here's what he covered, including highlights from the live demo.

What are the three bottlenecks AI created?

AI-assisted development has an irony built in: solving one constraint exposes the next. Ole broke the testing slowdown into three bottlenecks.

• Test creation. More AI-written code needs more tests, and writing tests for everything AI generates is its own enormous job. AI doesn't always see the whole picture either. Ask it to build a feature and it may miss the ripple effects, so bugs slip through.
• Test execution. Once you have all those tests, whether you wrote them or generated them, you have to run them. A build that took five or ten minutes now takes twenty-five, and the whole team waits. CI/CD becomes the choke point.
• Test analysis. Someone has to make sense of the results: sift through failures, spot anomalies, and figure out why a test is suddenly slow or flaky. As volume grows, mean time to resolution (MTTR) climbs fast.

Each bottleneck makes the others worse, and the result is stalled releases and shaky confidence in what you ship. Testkube AI is built to tackle all three.

What is Testkube?

Testkube is a testing platform that gives teams one place to create, orchestrate, execute, and analyze any test in their existing infrastructure. It works alongside your CI/CD pipelines instead of inside them, and it's built on an open source core. Three things set it apart:

• Any test. Keep the tools you already use: Playwright and Cypress end-to-end tests, k6 load tests, API tests, security tests, and more. Nothing gets rewritten.
• One place. Every test and every result, across teams, clusters, and pipelines, lands in one view, so you can see what's safe to ship.
• Your infrastructure. Tests run as native jobs inside your existing Kubernetes clusters, in the cloud, on-prem, or air-gapped, so nothing leaves your environment.

Testkube AI is the platform's native AI layer, not a separate product. AI Test Creation, AI Agents, and the MCP Server all work with your workflows, results, logs, and artifacts, and you choose the model. Those three capabilities map directly to the three bottlenecks.

Pillar 1: AI-powered test creation

This is the headline feature: describe what you want to test in plain language, and Testkube generates the test and runs it against your real application, in any framework and any language.

In the demo, Ole asked Testkube to build an API test. It explored the API, generated a Postman collection, and ran it with Newman. When the first version failed, it noticed, went back, and fixed the test on its own. He did the same for a k6 performance test and a Playwright end-to-end test. The Playwright test ran in parallel across multiple workers and self-corrected when one node hit a locator error.

AI writing test code isn't the point. Plenty of tools do that. The point is the loop: generate, run inside your infrastructure, see real results, refine.

Picture it: someone reports that the authentication service is acting up in staging. You ask for an API test against the auth endpoint and run it in staging right away, skipping the usual "build it locally, then wire it into CI/CD" detour. And because Testkube already holds the history of previous tests against that service, it uses that context in what it generates.

Why not just use my coding assistant to write tests?

An audience member raised the obvious objection: if you already use AI to write code, why not use the same tool to write the tests? Ole's answer came down to context and tuning.

• It knows your existing tests. Testkube understands what's already covered and how your suite is structured.
• Its skills are tuned for testing, the same way a dedicated code-review tool beats a vanilla LLM prompt.
• It tests against intent, not just code. If your code has a bug, a test generated only from that code may simply validate the bug. Feeding in requirements and other context produces tests that check what the system is supposed to do.

Pillar 2: Autonomous AI agents

If test creation is the front of the pipeline, agents cover everything else. In Testkube, an agent is something you define to handle almost any testing or quality task, such as:

• Deciding which tests to run, and in what order
• Tuning test configuration, like sharding and memory allocation
• Triaging failures and running root-cause analysis
• Generating reports
• Enforcing quality gates
• Analyzing flakiness

The demo showed three agents in action:

• Troubleshooting. An "AI Analyze" button on the dashboard pulls the logs and artifacts from a failed Playwright run and reports what failed, where, how to fix it, and why the healthy workflows stayed healthy.
• Trend reporting. Another agent produced a report across every execution of a workflow, with resource-usage patterns and optimization suggestions.
• Issue creation. A third created an issue in Linear straight from the failure analysis, with a human approval step before it acted.

Here's a telling detail: when the team went through their own backlog of customer feature requests, they realized many of them were now just agent use cases. Automatic reruns of flaky tests, flakiness detection and remediation, issue-tracker integration, and anomaly detection all shipped as built-in templates instead of one-off features.

Out of the box you get:

• Core agents: troubleshooting, optimize/analyze, reporting, and a general helper
• Templates: flaky-test detective, smart rerun, failure categorization, and an infrastructure validator that can generate validation workflows for you

You connect whatever MCP servers your agents need, choose your models, and decide what runs automatically and what needs sign-off.

Most flakiness isn't really flakiness

Ole made a sharp point here: what looks like flakiness is usually something happening outside the test that you can't see, such as a config change, a source change, or another job loading the system at the same time. Give an agent access to the right MCP servers (your source code, Grafana or Datadog metrics, your Kubernetes runtime) and it can connect those signals to find the real cause, separating infrastructure-induced flakiness from genuine test-design issues.

How much does it cost to run AI testing agents?

Agents that run often can burn through tokens fast. The team learned this firsthand when a noisy test with massive logs ran up the bill on failure categorization. Two lessons came out of it:

1. You don't need a frontier model for everything. Log analysis is a good job for a cheaper, faster, or even local model. Testkube lets you choose the model for each agent and point at your own endpoints, including local or air-gapped models through tools like Ollama. Token caps and per-agent, per-environment consumption views are on the roadmap.
2. Testkube doesn't charge per test or per token. Everything runs on your infrastructure, so you can run as many tests in parallel as your clusters can handle. The only limit is the resources you assign.

Pillar 3: The Testkube MCP server

The third pillar connects everything to the AI tools you already use. The Testkube MCP server exposes around 30 tools covering most of what the platform does:

• Running workflows
• Fetching results
• Creating workflows
• Searching historical data

That lets AI coding assistants like Claude Code or Cursor read your live testing context directly.

In the demo, Ole connected Cursor to a Testkube environment and asked about a specific workflow. Cursor called the MCP tools to build a high-level picture of how that test had been running. He also showed a flakiness analysis it produced across thousands of real executions, breaking failures into classes and rendering the results as a canvas, all from data pulled through the MCP server.

This makes an AI-native development loop possible: write a feature in your AI editor, use the Testkube MCP server to run the relevant tests against your changes, get results back, and let the model connect failures to the exact code you just touched.

The server currently uses token authentication. OAuth support is coming in an upcoming release, after which the server will inherit the authenticating user's permissions automatically.

Where to start with AI test automation

You don't have to adopt all of this at once. If you've already nailed test creation with your own prompts in Cursor or Claude Code, keep doing that and let Testkube handle execution and analysis. Pick the bottleneck costing your team the most and start there.

Ole was also realistic about the technology. AI is non-deterministic, and so are people. Getting agents to behave takes iteration: the team's first attempt at failure categorization went wrong before they tuned it. Two tips:

• Let AI write your agent prompts. AI is excellent at drafting the prompts and agents you'll use. Ask it for a deterministic agent prompt for a given task, then refine from there.
• Keep a human in the loop, especially before anything self-heals a test. Sometimes a failing test is correctly catching a real bug, and you don't want an agent to fix it away.

Frequently asked questions

{{banner-get-started}}