Testing AI Systems: The Two Layers Teams Skip

As AI systems moved into production, most engineering teams pointed their existing testing playbook at the new stack. Load testing got extended to cover model endpoints. Security testing got extended to cover prompt injection. CI gates stayed where they were. That is how every prior technology shift got absorbed, so the instinct made sense.

It was also incomplete. AI systems introduced two new layers, the model's own reasoning and the orchestration logic that strings its decisions together, and neither behaves like the systems that playbook was built for.

Why this is surfacing now

Part of why the gap went unnoticed is that agentic AI systems look familiar from the outside. They sit behind an API. They deploy through the same pipelines. They show up on the same dashboards as everything else in production. That surface familiarity is what let teams extend the old playbook, and why the extension felt sufficient.

Adoption has also moved faster than governance. Most enterprises still lack a mature governance model for autonomous agents, even as those agents take on more decision-making responsibility. Over 40% of last year's shipped code was AI-generated, a number large enough to force a testing conversation on its own, and confidence in that code has not caught up.

That volume is why continuous testing for AI-generated code became its own conversation. But that conversation covers the code the AI writes. The AI's own reasoning sits outside it.

The four-layer AI stack

Any AI system, whether a single model or a full agent chaining tool calls together, breaks down into four layers. Each one fails differently enough to need its own kind of testing.

Layer 1: Infrastructure

At the bottom sits infrastructure: compute, the model-serving layer, the vector databases feeding retrieval. This layer looks most familiar to anyone who has run load tests or chaos experiments, except the signals changed shape. You watch GPU memory instead of CPU utilization. You check whether a model finished loading instead of whether a process started.

Layer 2: Model

Above that sits the model itself: the raw output quality of whatever is doing the reasoning. Testing gets strange here, because the model does not return the same answer twice given the same input, and most testing careers were built on the assumption that it would. Pass or fail stops being a clean binary the moment non-determinism enters the picture.

Layer 3: Orchestration / Agent

Above the model sits orchestration, the agent layer: multi-step reasoning, tool calls, decisions about what to do next based on what just happened. The custom prompts, context, and RAG systems you have built on top of your LLM. Errors compound silently here. A small mistake three steps into a chain does not show up immediately. It shows up much later, as a visible failure that looks nothing like its actual cause.

Layer 4: Application

At the top sits the application layer: guardrails, the assumption that a user might be hostile, red-teaming against the system. This one is closer to familiar security testing, with one important difference. The attack surface never closes, because instructions and data share the same channel, so there is no version of "patched" that makes the problem go away for good.

Layers one and four get tested well, because they look like problems teams have already solved. Layers two and three do not.

Why this keeps happening, and what it costs the business

The gap between the model and orchestration layers does not persist because nobody has noticed it. It persists because processes that made sense for deterministic software stopped making sense the moment AI entered the stack.

The root causes

Non-determinism

The first habit is how teams treat non-determinism itself. A test that returns a different result on every run has traditionally been treated as a flaky test, a bug to chase down and eliminate. With a model in the loop, that variability is a property of the system. Treating it as something to engineer away means the testing quietly gets abandoned instead, since it never behaves the way a "fixed" test is supposed to.

Ownership

Most engineering organizations have someone whose job is to keep the test suite healthy and trusted. Almost none have an equivalent owner for eval quality, the thing that would actually tell you whether the model's reasoning holds up over time. Without an owner, evals do not get maintained. They get built once, usually for a demo, and forgotten.

Cost

Evaluating a model's output is not free the way running a unit test is free. It costs real compute and real money per run, and most teams never build that into a budget the way they budget CI minutes. So evaluation happens occasionally instead of continuously, and occasional evaluation of a system whose behavior can shift on its own is close to no evaluation at all.

Using AI to test AI

The fourth is a subtler trap: using AI to grade AI's own output. It is tempting, since it scales in a way human review does not, and it reintroduces the exact bias the evaluation was meant to catch in the first place. A model that is confidently wrong does not become a reliable judge of another model's confident wrongness just because it is doing the grading instead of generating.

The business cost

In a multi-step agent, an error does not show up where it happens. It compounds across every step that follows, and by the time it is visible, the actual cause can be several steps back, buried under everything that happened after it. Root cause analysis gets expensive and slow.

Behavior can also drift with zero code change on your end. A hosted model gets updated upstream. A retrieval index shifts underneath you. The system starts behaving differently for reasons that sit outside your Git commit history, and regressions stop being something you can bisect your way to.

Prompt injection sits in a category of its own. Because instructions and data travel through the same channel, it cannot be engineered away completely. It is a permanent risk that has to be managed on an ongoing basis.

How leadership needs to adapt

None of what needs to change here is exotic or complex. It is the same operating discipline most organizations already apply to infrastructure, extended to cover the model and orchestration layers.

Start with what "tested" means. Right now it is a pre-release gate, a checkbox before shipping. That definition does not hold for a system that can drift after deployment with no code change involved, so testing has to become continuous, watching production behavior alongside what happened in staging last week.

Human judgment has to stay in the loop. Letting AI grade AI's own output is efficient, and it reintroduces the exact bias the evaluation exists to catch, so quality assessment cannot be fully delegated away.

Then comes the budget. Evals need headcount and funding the way infrastructure does, as an ongoing line item rather than a one-time project that gets built and left alone. The golden dataset behind those evals, the examples a team trusts to represent real behavior, deserves an owner with the same accountability a production test suite owner has.

Unifying AI testing across the stack

All of it needs to be visible in one place. Most organizations already run separate tools for model evals, red-teaming, and agent testing, and that fragmentation recreates the problem fragmented QA tooling has always caused: no single view of whether the system is actually trustworthy. Centralizing those results is what turns scattered signals into an answer. It is the same tool-sprawl argument that has been made about traditional QA tooling for years, applied to a newer part of the stack.

This is where a test orchestration platform like Testkube fits. Testkube does not run the evaluations. It gives infrastructure, model, and agent signals one place to live instead of scattering them across dashboards nobody checks in the same sitting.

Testkube's AI agents already do a version of this at the infrastructure and orchestration layers, correlating test failures with environment state instead of leaving that correlation to someone cross-referencing logs by hand.

Test triggers extend that one layer further, kicking off workflows that evaluate retrieval accuracy and hallucination rates the moment a deployment or a prompt changes. For most organizations, extending the same principle across model and agent-level evals is the next step, and it builds on infrastructure they already run.

The two layers to fix first

Two of the four layers get tested well because they resemble problems teams have already solved. The model's actual reasoning and the agent's decisions as it strings tool calls together are the ones going unchecked, and they are also the ones most likely to fail once a system is making decisions on its own in production.

Closing that gap is less about buying a new tool than about giving the signals that already exist, from infrastructure monitoring to model evals to agent-level testing, one place to live and be acted on together.

Part 2 gets tactical: a walkthrough of what testing the model and agent layers actually looks like, with a working demo.