Your AI Writes Code in Seconds. Why Does Testing Still Take Days?

Speaker: Ole Lensmar, CTO, Testkube

Host: Harry Gagnon, StickyMinds

Welcome

Harry Gagnon: Hello, and welcome to today's web seminar, “Your AI Writes Code in Seconds. Why Does Testing Still Take Days?” sponsored by Testkube and featuring Ole Lensmar. I'm Harry Gagnon, and I'll be the technical support for today's webinar. Before the presentation begins, I'd like to give a quick walkthrough of the webinar interface. There's a panel on the right with features to chat, ask questions in the Q&A tab, and download handouts. If you minimize this panel, just click the chat button to reopen it. There are some helpful resources you can download in the handouts section, and polls will also appear in the chat panel when they become available, so please be on the lookout for those.

If you experience any technical issues, send me a private chat and I'll be happy to help. This event is being recorded and will be available to watch on demand. Once the recording is ready, you'll get an email with instructions on how to access it. And finally, to optimize your bandwidth for the best viewing experience, please close any other applications running in the background. With that, I'd like to welcome and introduce Ole.

Introduction: Where AI Fits in Testing

Ole Lensmar: Thank you so much. Great to be here. Hi, everyone. I'm Ole Lensmar, CTO at Testkube. I've been working in the testing space forever. I was at SmartBear for many years, which many of you might be familiar with, and I was the creator of a tool called SoapUI, which was very popular at the time. Now I'm working with testing, quality, and, lo and behold, AI. That's what I'm here to talk to you about. If you have any questions, please put them in the chat and I'll try to answer them, with help from our host.

Before I dive in, I want to make my position on AI and testing really clear, because I understand this can be an area of contention and debate.

I think AI is a fantastic and powerful tool to help you automate automatable tasks (obviously, that's up for definition). I do not think, at least at this point, that AI can replace manual or exploratory testing. That still has to happen as part of your quality initiatives and pipelines, however you want to phrase it. And people always need to be in the loop, especially when it comes to quality. We know AI hallucinates and doesn't always get it right. So I'm not here to advocate for replacing testers or QA with AI. I'm here to advocate for using AI as a tool to help you be more efficient. Ultimately, people are king. So that's a little bit of a disclaimer to start.

If we look at how testing has evolved over the last ten, fifteen, twenty years, testing is spreading across the entire delivery pipeline. Test-related activities happen early, when you're planning, coding, and building; when you're testing, obviously; but also as part of your release and deployment process. Many teams test in pre-production environments and in production. Testing is shifting left, which isn't new. Testing has been shifting right, too. And AI is now really turbocharging all of this.

AI is being used for test planning, to write tests, to write code, and to accelerate (or at least attempt to accelerate) many of the things related to testing and quality, or that have an impact on testing and quality. That's not a surprise, but it's where the people we talk to and our customers are. They're asking how they can use AI as a tool to help with testing.

Poll 1: Your Biggest Pipeline Pain

Before I move on, let's do our first poll. The question is: which is the biggest pain in your pipelines today?

Thinking about that pipeline I just showed you: Is it coverage gaps, where you can't write tests fast enough? Is it slow or overloaded CI/CD and test runs? Is it triaging and flaky test noise, figuring out why your tests are failing and what the results mean? Or maybe none of these are a problem. Let us know, and we'll come back to it later in the webinar. I see a few answers coming in, thank you. The poll will stay up for a little while so you can come back to it.

The Three Bottlenecks

If we look at the pipeline, there are a lot of bottlenecks. “Bottleneck” is a very trendy word in the industry right now. I'd like to focus on three of them.

First, AI is generating more code, so there's a bottleneck on coverage: do we have tests for all of it? That was actually the most popular response in the poll, at least so far. The second bottleneck is downstream from having more code: you need to run more tests in your pipeline, which usually slows the pipeline down, makes it a point of contention, slows delivery, and creates frustration. And at the tail end, even if you're able to run all those tests, you'll have more results to analyze.

That puts a burden on flakiness detection, root cause analysis, and anything related to analyzing test results. The consequences of not handling these bottlenecks are low test coverage, which results in buggy releases; contentious builds, which result in frustrated teams; and, if you're not set up to handle all the results coming in, a mean time to resolution that can skyrocket and stall velocity. So how do you tackle these bottlenecks? The question we hear from customers who have one, two, or all three of these challenges is: can AI help resolve this?

The Quality Loop: Eight Steps

We've been working on that, and we devised a concept we call the quality loop: putting AI to use to resolve some of these bottlenecks. That's what I'll walk you through today. I'll take it apart, look at the different pieces, and hopefully inspire you to start building a quality loop on whatever platform you're using.

If we take a step back: for every code change, you'd like to release it to production. But to do that reliably, a few things need to happen, especially when it comes to testing and test coverage. First, you might want to check whether the code change meets the requirements and whether you have tests in place to cover it.

The next step would be to create or update tests for any gap you've identified. Say you add a new feature and find there aren't any tests to validate it, or the existing tests aren't sufficient. So you create or update the tests required. The next step is selecting which tests to run to validate the code change. That could be the tests you just wrote or refined, but also tests from a library you already have, both functional and nonfunctional. It could be an end-to-end test that validates you can put something in your shopping cart, or a load test that validates you're getting the response times or throughput required for that feature.

Once you've decided which tests to run, you need a testing environment to run them in. Then you run the tests, and maybe optimize how you run them: how you parallelize or shard them, whether you run them in sequence or in parallel, whatever cuts down the overall execution time. At the tail end, which gets us to the last bottleneck, you analyze and troubleshoot any failed tests, or even tests that didn't fail. A load test might not fail, but you still want to analyze the results and make sure you're within the response times you've set.

If that analysis identifies things that need to be remediated, you'll want to remediate them and probably rerun your tests. And finally, you need to decide whether you're good to release. Even if the feature has passed all its tests, there may be other things to weigh at that point in time. You might want to correlate with other features being released, other things may need to be qualified, and you might have quality gates in place to validate that. So that's one way of breaking down a quality loop.

It's a loop in the sense that it goes back and forth. There are micro-loops and iterations within and across these steps, which I'll talk about in more detail. And of course, you always need a human in the loop. I'm not advocating that you just let AI run with this and do it all. If you trust your AI and your models, go for it. But I'd definitely advocate for having humans in the loop to validate, especially early in your journey: Have my AI agents done what they're supposed to do? Have they correctly identified the gaps?

Have they selected the right tests to run? Have they correctly figured out why a test failed, and have they remediated it? This is a process of refining your prompts and agents and building trust in your models and setup. But a human in the loop is super important for anything AI-related, especially quality.

The Mechanics: Agents, Context, Tools, and Models

So what do we need to make this work? We need discrete AI agents for each step. Each agent needs instructions, a prompt that tells it what to do: do gap analysis, run these tests, decide which tests to run.

The agents need context. The more context they have, the better they'll be at doing what they're supposed to do, and context is usually provided by MCP servers. I'm not going to go into detail on what MCP servers are; hopefully you know, and if not, look it up, it's pretty well established. Skills can also give agents more context and understanding to help them be successful. They'll need tools, maybe to run tests, create a pull request, open an issue in Jira, or post an alert on Slack. Those tools are also provided by MCP servers or skills. And of course, they need an LLM.

You might choose different LLMs for different steps. You might use a local or open source model for a step you feel is easier and doesn't require the latest frontier model, and a more capable frontier model for steps like remediation, where you want to give it your best shot at getting the right response. Guidelines and best practices are super important. They're really part of context: generic guidelines across all agents on how to behave, how to involve humans in the loop, how to make decisions, how to format things, and how to document what an agent is doing.

This is super important, and it's often something you work on iteratively, improving these instructions and guidelines every time the loop runs. You can also consider sub-agents for specific use cases. On the troubleshooting step, for example, you might have different agents for troubleshooting a load test versus an end-to-end test. Overloading one agent with instructions that cover every scenario will eat a lot of tokens, and it can also confuse the agent if the instructions aren't worded exactly right.

So this can be a good opportunity to have one controller agent that delegates to sub-agents for the specific tools or use cases at hand.

Orchestration and Governance

Once you have all those agents defined, you need to orchestrate them. You need a platform that starts the first agent, takes its output when it's done, passes it to the next agent, and orchestrates the whole execution of the quality loop end to end. There are open source and commercial platforms for agent orchestration. I won't point at a specific one, but you'll need one. You could even use an AI agent itself as the orchestrator, using Claude or Cursor or whatever tool you use.

That might be a good start. It'll be a little rough, but it can help you validate that the quality loop you're putting in place works, or has a chance of working. The last thing, which is super important, is governance over the agents and the orchestration. By governance, I mean imposing rules and guardrails: how many tokens are agents allowed to use? What tools are they allowed to use? You want to constrain agents so they don't start doing things you don't want them to do. I'm sure you've all heard the horror stories of agents going outside what they were asked to do.

RBAC and audit logs are super important so you can look at what an agent did to achieve a task. If you notice something weird going on, you can go back in time and see, for every execution of the loop, which tools an agent used and how many tokens it used, which is also important for cost analysis. And once again, human in the loop. I've mentioned it before, I'm mentioning it now, and I'll mention it again.

Crafting Prompts and Instructions

Another task at hand is crafting the instructions themselves, which is prompt engineering. You have to be specific about the goals, the input, the output, the tools you're allowing (or want) the agent to use, and the constraints.

You also need to adapt prompts to the target models. If you know you'll be using a frontier model, you can maybe be more elaborate. If you're using a model that's not the latest, you'll want to be really clear about how you want it to work. A really good way to craft prompts is to use AI to generate them. Ask a model: “I need to craft instructions for an agent that helps me select which tests to run.” Then have a dialogue with your LLM about the tools and context that agent would need.

You can also ask a frontier model to generate instructions that will work with a non-frontier model. There are a lot of possibilities. It's a bit meta: using AI to help you use AI to solve the problems or tasks at hand. And finally, governance once again. This is somewhat advanced, but we've seen a couple of our customers create evals: a way of continuously evaluating that an agent or prompt is doing what it's supposed to. You can do this at runtime by randomly picking responses your agents create and running them through an eval to check whether they're accurate and relevant to the task at hand.

You can also do it at design time while you're building your agents, and simulate. This can be a good way to select which model to use. You can write an eval for accuracy or relevance, then run it against a couple of models with different cost and performance profiles to see which works best for your use case. There's a lot to this, and it's a lot of fun, because it's really interesting to learn the ins and outs of your models. Evals are also really important when transitioning from one model to another. I'm sure you've been in the situation where you switch to the latest version from OpenAI or Anthropic and think, “Whoa, I'm not getting the responses I'm used to.”

If you've built a pipeline you want to be reasonably deterministic based on a specific model, and suddenly you're not getting the same responses, that can be very disruptive. Evals can really help there. A quick example: for step six, analyze and troubleshoot, don't prompt “figure out why the test failed.” That's a very vague prompt. You're giving the LLM a lot of leeway to do things you don't want it to do, or to decide it's done when it isn't.

Instead, be very specific about what you want it to do, which tools it should use, what your expectations are, and how to analyze failures in that specific case. This specific example was actually created with AI, so once again, AI is a great way to craft prompts and instructions.

Considerations: Determinism, Privacy, and Cost

As always with AI, be mindful of nondeterminism and hallucinations. Make sure your prompts are high precision, and use frontier models where you need determinism. As I mentioned, switching models can break your prompts. And especially with testing, sharing requirements, source code, and test results can expose very sensitive data.

So make sure you're using a private model, or at least a solution that doesn't hand the keys to the kingdom to an external provider. Cost is also super important. If you're feeding your agents logs, guidelines, and source code, that will consume a lot of tokens. I'm not saying it's necessarily a bad thing, but it's something to weigh. If you build a quality loop like this and trigger it frequently, it can incur a lot of cost over time. But maybe it's worth it if it improves your delivery confidence or velocity. That may well be a worthwhile investment.

Walking Through the Loop, Step by Step

Let's walk through the eight steps of the quality loop and tease out what we want in each one. I'll repeat: you don't need to do all the steps. You could start with just step five, running the tests, and maybe step six, analyzing and troubleshooting. The full loop is the long-term goal. As always, start small.

Step 1: Gap and Requirements Analysis

The purpose of the first step is to look at the code change that triggers the quality loop. Someone opens a pull request, which triggers the loop.

First, identify whether the code change actually fulfills the requirements that triggered the initiative. Next, identify whether the changes are covered by existing tests or by new tests that were part of the pull request. For an LLM to do that, it needs access to the requirements; otherwise, it can't know whether the code meets them. Those could be user stories, a Jira issue, something from a test management tool, or just a markdown file provided in the repo or through an MCP server.

It also needs the source code with the changes, whether that's a pull request on GitHub or a merge request on GitLab. Your existing tests are super important too; they might live in a separate repository or the same one, depending on how your codebase is structured. And then related components: if you have a microservice architecture, make sure the agent has access to those as well, to understand how the change impacts other components and whether their tests need updating because they depend on the code changed in this PR.

The output of an agent like this, in a perfect world, is a list of tests that need to be created, refined, or removed (maybe you removed something and don't need those tests anymore), and how that should happen. Another output could be an alert to Slack or wherever the loop reports, saying, “This feature doesn't seem to cover the requirements,” or “I don't have access to the requirements, so I can't do my job. Let's stop here and let people know.” That brings in the human in the loop before moving to the next step or rerunning this one.

Step 2: Create or Update Tests

The next step is to create or update your tests, in line with the analysis performed in the previous step, which is the input and context. Here, the agent needs standards and guidelines on how to write tests and which tools to use: how you write your Playwright tests, how you structure your Cypress tests, whatever guidance on code structure will help the agent do a better job. The agent needs access to the repository where those tests should be written, and previous test results can be super helpful for understanding how tests behaved in the past.

The output here would be a branch with the updated tests, and maybe even a pull request or merge request to integrate them into the codebase. Once again, human in the loop. We're not advocating blindly letting AI update your tests and run with it. Someone should check that it makes sense and maps to what you expected. You could even have agents governing these different steps.

Step 3: Select Which Tests to Run

Next is selecting which tests to run. That includes the tests that were created and updated, which feels natural, but you might also pull in other tests that weren't touched by the previous steps, like a load test to validate that the new feature's performance is up to par.

Here again, the agent needs to know the requirements so it knows which tests to select. It needs insight into a test catalog and what's called a dependency map, or metadata. This is really hard, especially in a complex microservice architecture where you have 50 microservices all entangled (in a good way, of course). How do you know that a change over here means tests over there need to be rerun?

This is where dependency mapping, tracing, or manually curated metadata is mandatory to give an agent the context to make that decision. The revised tests from the previous step are input here too. Previous test results can also inform selection. If a test always passes, maybe you don't need to rerun it every time; focus on the tests that are better at finding failures. You could argue you should always run all tests because you never know what will break.

It also plays into prioritization. The output of this step is a list of tests that need to be run, by priority. Some are critical; others aren't, and don't have to run every time.

Step 4: Provision a Testing Environment

Next is creating a testing environment, which you often already have, so it's not necessarily part of the quality loop. When you create a pull request, it might immediately deploy your application to a temporary environment that's ready for tests to run against. But you might want to dynamically provision an environment just for validating this feature.

You don't need to deploy the entire application, just the microservice that was changed. That's much more efficient and faster, requires fewer resources, and provides a more isolated, black-box testing setup. If your agent can dynamically decide which components it needs and deploy them, especially with something declarative like Kubernetes, an AI agent could be successful at this. To be honest, this is probably the hardest step on this list to get right.

But it depends on your infrastructure and the complexity of your application. Many applications are simple, with one back end and one front end, so this would be easy. You probably don't even need an agent; you can just do it before kicking off the loop. The output here would be an endpoint for the environment, which is passed on to the tests.

Step 5: Optimize and Run Tests

Next is optimizing and running the tests. We've decided which tests to run, we have priorities, and we have a testing environment. This is where AI can figure out how to run these tests: in sequence or in parallel?

AI could look at previous execution times and say, “These tests usually take longer than these, so let's shard or parallelize them accordingly,” so you don't have to wait for everything to run sequentially. Guidelines might say you don't run your full end-to-end suite for small code changes, and this step could make that call, or step three could. It's up to you to decide which step has which responsibility. The output here, of course, is test results. This is the heart of testing: executing your tests as efficiently as possible and producing results for the next step.

Step 6: Analyze and Troubleshoot

Next is the analyze and troubleshoot step. If a test fails, you want to look at the results, figure out why, and decide what to do about it.

You might identify it as a flaky test. Historically it's been flaky, so maybe you just rerun it. For some, that's an anti-pattern, but for many it's a brute-force way to move forward: if it fails, rerun it; if it passes, move on; if it fails again, raise a red flag and maybe stop the loop. You could also correlate failures with other tests or other things going on, which could result in alerting or rerunning tests.

The context for an agent here is test results, both for the current execution and previous ones, if you want to detect anomalies or do flakiness analysis. Metrics from the system under test are important too, such as memory and CPU consumption or any telemetry you can get. If you're running a load test, you might check whether memory consumption is stable or continuously climbing, which could trigger an action and stop the loop. And the requirements are important as well, though they're often encoded in the tests themselves, so a test fails if the requirements aren't met.

For nonfunctional requirements, though, an agent here could help analyze them. I'm sure this sounds like a lot. You could definitely start with just using AI agents to analyze your end-to-end tests or your performance tests. You don't have to do everything at once. If this seems like an intriguing approach, take a slice across the quality loop, decide which steps and which tests you want to focus on, start there, and grow from there.

The output here would be a list of issues to remediate, if there are any. Your agent might create a Jira issue or post an alert on Teams, however you build awareness of test failures within your team. An agent with access to the corresponding MCP servers or tools can do that for you.

Step 7: Remediate

Remediation is another really nice one. Say your test failed because of an error in the code. Can you kick off an agent that tries to fix that error, and maybe even opens a pull request or branch with the fix?

This is definitely doable. The agent would need access to the results, error logs, and artifacts that help it understand how to fix the problem; the source code where the fix needs to happen; and the requirements. Requirements really span the entire loop, so you'll probably want to provide them to all your agents. The output here could be a branch with the remediated code, and once again, human in the loop. I'm not sure you'd want to let the agent automatically fix the code and rerun the tests. Maybe you do, and you put the human in the loop later in the pipeline.

It's really up to you and what you feel confident with, which takes us to the last step.

Step 8: Release Decision

Say you've gone through all of this. Agents helped you close the gaps and write the required tests. They helped select which tests to run, ran them, found an issue, and remediated it, with you in the loop. Now your tests are passing. Is the code change ready to go into production? The easiest answer is yes.

All my tests passed, so we're good to go. That's perfectly fine in many cases. In other cases, you'll want to factor in other things, like other tests and metrics running alongside. Say you have an end-to-end test that runs in your staging environment every four hours. Maybe you want to make sure it passed the last two times and factor that into the release decision. You could define other heuristics for when you're good to release into production. A change that's working fine might still not be the right thing to release right now.

The decision to move to production usually factors in other aspects. The output here could be a score or a yes/no decision. It could be an event you publish. Maybe you allow the agent to tell your provisioning tool, “Green light, push this fix to staging,” where other mechanics take over to move it into production. And of course, alerts and notifications, which don't have to be negative. You could post in the team Slack channel that this code change has gone through the quality loop, here's the outcome, and it's ready to release.

And once again, human in the loop: someone can review and do a final manual check-off to move the release into production.

Poll 2: How Are You Using AI in Testing?

That's a lot of talk from my end, so let's do another poll. How are you currently using AI in your testing workflow? I'm super curious whether people watching are already using AI. Are you not using it at all? Experimenting informally? Using it for specific tasks? Or have you already adopted it in your daily workflow? Let me know in the poll, and we'll come back to it.

What You Need From a Platform

Since we're coming up to the end, I'll move forward. Say you think this is a good idea and want to experiment with building a quality loop. What do you need from a platform? We've already touched on this. You need to be able to define agents and orchestrate them to run in sequence or in parallel, passing input and output. They need to be triggered, for example, by a pull request, by local code changes, or through an API. You need to define which models the agents use. You need to integrate with MCP servers and tools so agents have the right context and tooling. And you need governance in place to enforce guardrails, token limits, human-in-the-loop checkpoints, and shared context management, whatever helps the agents in your quality loop be successful.

Where Testkube Fits Today

That brings me to Testkube. You might think I'm about to say Testkube is the ultimate quality loop platform. I'm sorry to say it's not. Not yet, at least. But there's definitely overlap, and it's something we're aiming for. I'll come back to that. So what is Testkube? It's a testing platform for AI-driven engineering teams. Ultimately, it's a platform for running any kind of test: end-to-end tests, load tests, API tests, Cucumber tests, whatever. Testkube is totally vendor agnostic.

It's not a testing tool per se. It takes your existing tests, makes sure they run as efficiently as possible in your infrastructure, and gives you one holistic dashboard, a single pane of glass, for all of them. It's deeply integrated with cloud native technologies and your CI/CD tools. It can be deployed fully on-prem. There's an open source component, and it's really tailored to teams that have embraced open source and cloud native technologies for testing. And it has a bunch of AI features. The AI components in Testkube include AI agents you can define for troubleshooting, flakiness analysis, and remediation.

Many of the quality loop steps we talked about are things an AI agent in Testkube can do for you. There's an MCP server you can use from the AI tool of your choice, whether that's Cursor, Codex, Copilot, or Claude, to run your tests, analyze results, and so on. And we're launching soon, at STARWEST (so if you'll be at STARWEST, come to our booth), the capability to create any kind of automated test. If you want to use AI to generate Playwright, k6, Selenium, or Postman tests, Testkube will be fine-tuned to both create those tests and run them immediately inside your infrastructure, with a lot of bells and whistles.

To close the loop on the quality loop: remember the platform requirements slide? Testkube can define AI agents and run them individually. Today, Testkube can't orchestrate agents in sequence or in parallel, which is one of the core requirements for a quality loop, so that's a gap at this point. Testkube can trigger your AI agents, for example, when a pull request happens in GitHub or when an event happens in your Kubernetes infrastructure. It can't yet be triggered from the CLI or API webhooks. Testkube does support local and custom models for your agents, and it does support MCP servers. AI governance is an area where Testkube needs to improve.

It has good support for human-in-the-loop notifications and guardrails, but things like token limits and shared context management are something we're continuously working on. So how well suited is Testkube to implementing the quality loop I described? We're not there yet. Testkube can do some of the steps, but definitely not all of them. We have a vision of a quality loop that we'd like the product to support at some point. If you're intrigued by the concept, Testkube might have a part to play in your implementation, but you could also use other open source and commercial agentic orchestration frameworks to achieve at least some of it.

I'll leave that open for you to decide whether this is interesting, whether Testkube is interesting, and whether there's an overlap for you.

Poll 3: Where Would You Trust an AI Agent?

Last poll coming up, and I'm super curious about this one. Where would you be comfortable letting an AI agent help you? Thinking about all the steps in the quality loop: Would you be comfortable with an AI agent creating tests, which was the second step? Deciding what tests to run and in what order? Rerunning a flaky test, or deciding whether a test is flaky? Opening issues for you, assigning them, and doing root cause analysis?

Or, the last option: you're not really confident in AI's capabilities, and you'd prefer humans to be in the loop or doing many of these tasks. That's a perfectly valid answer and concern, and I totally hear you. Okay, super interesting. Thank you for all those responses. Someone asked about a demo, and I'm going to say no, because of time. A demo now would just be confusing, since it's a vast platform, and the purpose of this webinar wasn't primarily to pitch Testkube. It was to make you aware of Testkube, but mainly to pitch the concept of the quality loop, which isn't a Testkube thing at this point.

Not yet, at least. So with that, thank you so much for joining. I'm happy to answer any questions. I'll be at STARWEST if you're going. It's a great conference. We'll have a booth where we can show all the features we have, and how we fall short of the quality loop. But I'd also just love to talk to testers, especially in the age of AI, and learn about your experience and your appetite (or lack of appetite) for AI. As a vendor, it's always good to stay in touch with reality. Thank you for listening, and I'm curious if there are any questions. Harry, do you have any on your end?

Q&A

Harry Gagnon: Yes, I do. Thank you, Ole, for your presentation. We have some questions that came in. Again, feel free to post your questions in the Q&A panel, and we'll get to as many as time permits. For those we don't get to, our team will reach out with answers by email.

Question from Jeff: For flaky tests, how do you use AI to self-heal?

Ole Lensmar: Good question. For flaky tests, the first part is figuring out why a test is flaky, which may be the hard part. You could have an academic debate that there's no such thing as a flaky test, that it's always something else causing the flakiness, but ultimately, you want to remediate. For AI to correctly diagnose why a test is flaky, it usually needs a lot of data, and the ability to pull logs from your infrastructure and any component in your system under test that could be the culprit. If AI has access to that, it can often also suggest remediation. Maybe you need to upgrade to the latest version because the issue is related to a bug, or you need to reconfigure something.

In that case, if you have a declarative infrastructure approach, you could even allow an agent to open a pull request. To be honest, in my experience it's really hard for an agent to remediate flakiness specifically. But it's a great way to troubleshoot. Those of you who've troubleshot flaky tests know it can be really hard, because so many things could be causing it. AI is really good at sifting through a lot of data. If you give it all the context and access to Prometheus or whatever you use, it can definitely help you figure out the root cause of flakiness, even if fixing it might be hard.

Question from Michael P.: For commercial off-the-shelf products like an ERP, where do the frontier models come from, and how are they trained? For example, do they need to be provided by the vendor?

Ole Lensmar: I don't know if I can answer that one. You'd have to ask the ERP provider what model they're using and how they've trained it. I don't have a generic answer to that question, sorry about that. Next question.

Question from Michael S.: Is making Testkube able to orchestrate the entire quality loop a priority?

Ole Lensmar: Yes, that is a priority, which is maybe not surprising, or I wouldn't be talking about it. Like I said, we have a lot of the pieces in place, but we don't have that overarching orchestration, as you pointed out. That's something we want to focus on in Q4 this year. Just today, actually, two or three hours ago, we released an initial stab in this direction: you can now trigger steps five and six of the quality loop from a GitHub pull request, both selecting which tests to run and doing an AI analysis. It's a very, very early version of a quality loop, and we're not calling it a quality loop either.

Sorry for the little bit of promotion, but check that out. So yes, it's a priority, but I think it's still a couple of months out. A lot of other things need to work nicely together before this works as well as we want it to.

Question from Ragini: If AI creates tests as we ship products, how do you make sure they have good coverage, while also making sure we're not bloating our test suite?

Ole Lensmar: It's a good question. The definition of a good test is hard. The long answer is that someone needs to review the tests being created, and that should usually be a human. So one way to make sure is simply to have a human in the loop governing and monitoring what's being done. That said, AI can help there too: you can have AI governing AI. I'm not sure AI always appreciates being governed, but that's part of the game. I'd probably start by putting an agent in place that has access to the requirements and to the tests being created, whether manually or by AI, and asking it, maybe at the end of the quality loop or at a higher level, to look at all the tests created in the last week and match them to the requirements.

Do those tests seem good? You can have a debate with your AI about what a good test is. There's a lot of experimentation there. I don't think there's a clear “this is how you do it” answer, but there are definitely approaches that use AI together with humans to tease that out.

Question from Michael S.: How do you envision this working in a regulated environment like GMP (good manufacturing practice)?

Ole Lensmar: That's a great question, and this is where AI governance becomes really important. On the Testkube side, many of our customers are in heavily regulated environments. Platforms need to be able to run in air-gapped environments and have full audit capabilities so you can see exactly what's going on. You need guardrails and kill switches for your agents to make sure they don't go off the rails. There's a lot you can put in place to rein in your AI agents and govern your models. AI governance is a big field in itself, regardless of whether you use it for a quality loop or anything else.

Building awareness and understanding of those challenges is a good start. Looking for tools that support this level of AI governance, and looking at what the requirements are from a regulatory perspective, is probably a good starting point.

Question from Jonah: If AI is generating the tests, how do I know the tests themselves are any good? Couldn't an agent just write tests that pass against the code it was given?

Ole Lensmar: That's a great question. To your point, an agent will look at the code and write tests to validate that code. But what if the code isn't doing what it's supposed to do? The agent won't know unless you tell it what the code is actually supposed to do. So this is about context: giving the agent the requirements. That was the first thing in the gap and requirements analysis step: does this code change seem to do what it's supposed to do, even before deciding whether we need tests?

To do that, you need good requirements, which is interesting. As we adopt more AI, at least in our internal product development, the need for good requirements, something we've talked about for ages, becomes even more pressing. Requirements are important, and requirements are hard. As you involve AI in acting on those requirements, it becomes even more important that they're accurate, so an AI agent can use them to validate whether the code does what it's supposed to do, and then create tests that validate the requirements. I don't know if that answers your question, but I think it's in the domain of requirements management: allowing agents to work with requirements to validate not just tests, but also the code that's supposed to implement those requirements.

Question from Kelly: You mentioned agents deciding what runs in the pipeline and in what order. What happens when the agent gets it wrong and skips a test that would have caught something? Who is accountable for that?

Ole Lensmar: That's a great question. Tactically, what happens is that a bug might escape into production, or at least further down the loop. This is where quality gates are important. This is where human in the loop is important. And this is where constantly monitoring and auditing the loop and the agents you've put in place is important. You can have additional guardrails, like we said earlier, with humans or with agents that validate that whatever you've tasked your agent with is the right thing to do.

Just like humans, agents make mistakes. And just like humans refine their skills and get better, so will agents and LLMs, but we need to help them. We need to refine the prompts, tease out where they're more likely to fail, and figure out how to help them get things right. And, as I mentioned much earlier, use AI to help you craft prompts. You can describe it to an AI: “Here's my prompt. The agent failed and did this incorrectly. Can you help me improve my prompt so this doesn't happen again?” And maybe splurge on some tokens for a frontier model to get that right.

There's a lot you can do, but ultimately, humans in the loop doing governance and oversight are super important, and will continue to be.

Question from Sarah: Auto-rerunning flakes sounds like it could hide real problems. How do you keep that from turning into “retry until green”?

Ole Lensmar: Great question. Again, all great questions. It comes down to the heuristics for auto-rerun. You don't want to rerun forever. Depending on the flakiness of that specific test or how it's been performing, you might only rerun once. If it doesn't pass the second time, flag it as an incident and either ask an agent to figure it out, or just raise an issue, post on Slack, abort the loop there, and get a human in the loop to figure it out. This goes back to something I said very early: I would never give AI full autonomy, at least not early in your AI adoption journey.

Always set some thresholds. Let it rerun once, maybe twice, but if it still fails, get a human in the loop to see what's really going on.

Question from Evan: Where do you recommend a team start? What's the first thing that's safe to automate, and what do we need in place before we do it?

Ole Lensmar: Great question. Like I've said, it's super important to tease out one slice across the quality loop and maybe select just one or two steps. The most obvious step to me is troubleshooting, because it's very straightforward. If you can automate a loop where, when you run your tests, AI automatically analyzes the results and either reruns tests or posts a result analysis to whatever channel you use, a human can then use that to remediate the problem. That's a great starting point because it's low touch and easy to add on top of what you already have, since you're probably already running your tests somewhere.

Preferably with Testkube, of course. Adding an AI step on top of that is a great place to start, and then you can work in either direction. It also depends on where your bottlenecks are. Going back to the three bottlenecks I outlined earlier: Is it coverage? Then maybe that's where you go next. Or are you good on coverage, and happy with how AI generates tests, but more constrained on the pipeline or on MTTR? Then maybe that's where you go deeper to try to alleviate those bottlenecks. I don't know if I answered Evan's question, but I hope so.

Closing

Harry Gagnon: Well, it looks like we're a couple of minutes shy of the top of the hour. On behalf of StickyMinds, I'd like to thank Ole for his time and insights, and our sponsor, Testkube. A special thank you to our audience for joining us, and we hope to see you at a future event. Take care, everyone.