Release Confidence Is the Metric That Matters

Release confidence metrics: test results from k6, Postman and Cypress feeding a Testkube release readiness dashboard that shows safe to ship
read
Katie Petriella
Senior Growth Manager
Testkube
Read more from
Katie Petriella

Table of Contents

Start your free trial.

read
Katie Petriella
Senior Growth Manager
Testkube
Read more from
Katie Petriella
Katie Petriella
Senior Growth Manager
Testkube
Pass rate shows that tests ran. Release confidence shows whether you can ship. The four pillars and eight metrics that tell engineering leaders a release is safe.

Table of Contents

Executive Summary

It's an hour before a big release. Someone pulls up the CI dashboard, someone else checks the load test results, and QA is still finishing a manual regression pass. One suite failed twice, then passed on the third rerun.

Everyone in the room is asking the same thing: are we good to ship?

Nobody answers with a number. They answer with a feeling.

If that scene sounds familiar, one of these probably does too:

  • You can't see everything being tested across your teams and environments.
  • A red build means “rerun it” more often than “stop.”
  • A person still signs off by hand before anything big goes out.

These are symptoms of the same gap. Your team measures testing activity, but what you actually need to know is whether you can trust the result. This post defines release confidence, shows what it costs when it's missing, and lays out four pillars and the metrics that tell you whether it's getting better.

What is release confidence?

Release confidence is how far you can trust your testing to give one clear, explainable answer to whether a release is safe to ship.

It's a property of the signal, not a mood in the room. With high confidence, everyone reads the same results the same way and acts on them without a side conversation. With low confidence, every big release needs a meeting.

For an engineering leader, it comes down to one question. When you're about to ship something big, does your testing tell you whether it's safe?

Why pass rate isn't release confidence

A 98% pass rate looks healthy on a dashboard. It doesn't tell you which 2% failed, whether those failures matter or how many of the passes came on the second try.

Pass rate is an activity metric. It counts what ran and what came back green. Release confidence is a decision metric, because it tells you whether to ship.

Here's where the two drift apart:

  • Reruns. A flaky test that passes on the third attempt counts as a pass.
  • Environment noise. An unstable staging cluster fails tests that have nothing to do with your code, and real defects hide in the noise.
  • Coverage of the change. Every test can pass while none of them touch the code you changed.
  • Partial visibility. If load tests live in one tool and end-to-end tests in another, the pass rate you're looking at only covers part of the release.

Some teams will push back on this, but a higher pass rate can mean lower confidence. If failures get rerun until they pass, the number goes up while the signal gets worse.

What low release confidence costs

When teams don't trust their testing, they fall into one of two traps. Most organizations fall into both, depending on the week.

Trap 1: You test too much

When nobody trusts the results, teams add checks. Manual regression creeps back in, and in some teams it runs overnight. After an infrastructure change, people from several teams verify the system by hand, and readiness evidence ends up as a folder of screenshots.

Releases slow down, and your most expensive engineers become the test harness.

Trap 2: You test too little

The other response is to stop listening. Tests get skipped to hit a date, flaky tests get tolerated, and a red build turns into background noise.

The risk doesn't go away. It moves to production, where it costs more to find and fix.

Why AI makes release confidence harder

AI coding tools have changed how much code reaches your pipeline. More code means more pull requests, bigger changes and longer reviews.

The research backs this up. Google's 2024 DORA report found that as AI adoption increased, delivery stability dropped by an estimated 7.2%. In the same research, 39% of respondents reported little or no trust in AI-generated code. A year later, the 2025 DORA findings showed AI no longer slowed throughput, but stability still suffered.

Teams feel it every day. Reviews take longer, more defects get through and they tend to be more severe. Many teams also aren't sure how to check tests that AI wrote in the first place.

If AI doubles the change coming at your pipeline, a manual sign-off can't double with it. Confidence has to come from the system, not from more people reading more diffs.

The four pillars of release confidence

Release confidence holds up when four things are true, and all four sit on automation that keeps pace with change.

1. One answer

Every tool, team and environment reports into one place. That sounds obvious until you count where your results live today.

Playwright results sit in CI, k6 results sit in Grafana, someone runs a Postman collection locally and a homegrown harness covers the rest. On release day, two people can look at that spread and reach two different answers. With one view, “are we good?” gets one answer instead of a debate.

2. A clean signal

A failure should tell you something. That only works when real defects are separated from flaky tests and environment problems.

Picture a staging cluster that throws false positives every few days. Within a month, the team learns that red means “rerun,” and a real defect gets rerun right along with the noise. Once the signal is clean, a red build means stop again.

3. Clear thresholds

Each test tier needs a pass bar the team agreed on before release day. A smoke suite might need 100%. A full regression suite might ship at 90% if the rest are known-flaky tests with a controlled rerun.

A good threshold also asks whether the tests actually cover the change, not only whether they passed. With thresholds in place, go or no-go becomes a rule instead of a negotiation.

4. Explainable gates

Every go, no-go and override should be logged with its reason. That matters even more when an AI agent is part of the decision.

If an agent labels a failure “low risk” and nobody knows why, nobody trusts the label. When each decision carries its reasoning, the team can accept it or challenge it. Over time, the history shows you which tests keep getting overridden.

The foundation: automate it

None of the four pillars hold up if a person has to assemble them before every release. Tests should trigger from CI, deployments, schedules or events, run in the environments that matter and report into the same place. That way validation grows with change volume, not with headcount.

Where Testkube fits

This is the problem Testkube is built for. Testkube is the testing platform for modern engineering teams, purpose-built with test orchestration at its core. It runs the tools you already use, such as Playwright, Cypress, k6 and Postman, as Test Workflows on a Runner inside your own Kubernetes clusters, then brings every result, log and artifact into one view.

Testkube AI agents help separate real failures from flaky tests and environment issues. Testkube works alongside your CI/CD instead of replacing it. DocNetwork used it to save 30 DevOps hours a week and gain more confidence in its deployments.

Release confidence metrics to track

No single number captures release confidence. Together, these metrics show whether your signal is getting more trustworthy or less.

Metric What it tells you Warning sign
Change failure rate Share of deployments that cause a failure in production (a DORA metric) Climbing while pass rate holds steady
Flaky test rate Share of tests that pass and fail without a code change Rising, or nobody tracks it
Rerun rate How often failed runs are retried before anyone decides Rerunning is the default response to red
Failure classification rate Share of failures labeled as real defect, flaky test or environment issue Most failures never get a label
Pass rate against threshold, per tier Whether each test tier meets its agreed bar One blended pass rate for everything
Time to release decision Time from release candidate to a clear go or no-go Grows with every release
Manual validation hours per release Human time spent checking before you ship Flat or rising even as automation grows
Gate override rate How often people override a gate, and why Overrides with no recorded reason

You don't need all eight on day one. Change failure rate tells you the outcome. Flaky test rate and manual validation hours usually tell you why.

Where to start

You can make progress on release confidence before you change a single tool.

  1. Map where test results live today. List every tool, suite and environment that feeds a release decision. If the list runs past a page, that's your first finding.
  2. Separate the noise. For a few weeks, label every failure as a real defect, a flaky test or an environment issue. The split tends to change how your team talks about red builds.
  3. Agree on thresholds before release day. Write down the pass bar for each tier, such as smoke, regression and performance. Decide which known-flaky tests get a controlled rerun.
  4. Log every gate decision. Record what passed, what got overridden and why.
  5. Automate the manual sign-off first. It's usually the slowest step, and it's the one that scales worst as AI adds more change.

Ship on a number you trust

Testing activity is easy to count. Whether you can ship is what your leadership team and your on-call rotation actually depend on.

The teams that get this right stop asking “did the tests pass?” and start asking “do we trust what the tests are telling us?” Measure that, and release day stops being a meeting.

Frequently Asked Questions

What is release confidence?

Release confidence is how far you can trust your testing to give one clear, explainable answer to whether a release is safe to ship. It depends on the quality of the signal, not on how many tests you run.

How do you measure release confidence?

Use a small set of metrics together: change failure rate, flaky test rate, rerun rate, manual validation hours per release and gate override rate. Change failure rate shows the outcome, and the others show why it's moving.

What's the difference between pass rate and release confidence?

Pass rate counts tests that came back green. Release confidence tells you whether those results are trustworthy enough to ship on. Reruns, environment noise and untested changes can push pass rate up while confidence goes down.

How does AI-generated code affect release confidence?

AI increases the volume and size of changes that need validation. Google's 2024 DORA research linked higher AI adoption to an estimated 7.2% drop in delivery stability, so testing has to scale without adding manual review.

What is a quality gate?

A quality gate is a checkpoint that decides whether a change can move forward based on agreed rules, such as a minimum pass rate for a test tier. Good gates log every decision and override so the team can see why a release shipped.

{{banner-get-started}}

Katie Petriella
Senior Growth Manager
Testkube
Read more from
Katie Petriella

About Testkube

Testkube is the testing platform for the agentic SDLC. It runs tests directly in your Kubernetes clusters, works with any CI/CD system, and supports every testing tool your team uses. By removing CI/CD bottlenecks, Testkube helps teams ship faster with confidence.
Get Started with a trial to see Testkube in action.