How to Run Tests Outside Your CI Pipeline

The T in CI/CD is silent

Most teams discovered test automation through their CI pipeline. You add a test step, the pipeline runs it on every push, and for a while that feels like progress.

Then it starts breaking in ways that are hard to explain.

Tests that pass locally fail in CI. Tests that pass in CI fail in staging. The pipeline takes 40 minutes and half of that is tests. Someone disables a flaky test to unblock a release. Someone else adds a retry and calls it fixed. The test suite grows but so does the anxiety around it.

The problem is not the tests. It is the architecture. CI/CD tools were designed to move code from commit to deployment. Testing got embedded inside that flow because it was convenient, not because CI was the right execution environment.

This post is about what happens when you separate the two.

What CI does well, and where it stops

CI/CD tools are good at what they were built for: detecting code changes, building artifacts, running deployment sequences, and gating releases. That is a well-defined job and most teams rely on it working reliably.

Testing is a different job.

Tests need to run across environments, not just on push events. They need to run on schedules, on infrastructure events, before a deployment, after one, and sometimes in the middle of the night when no one changed any code. They need to scale independently of build pipelines. And they need to surface results in a way that is useful to more than just the engineer who triggered the run.

None of that maps cleanly onto how CI pipelines work.

When you run tests inside CI, you inherit all of CI's assumptions: tests run when code ships, results live inside pipeline logs, execution is tied to whatever compute the pipeline has available, and the failure context is whatever the pipeline step happened to capture before it timed out.

That works at small scale. It stops working when the environment gets more complex.

What "outside CI" actually means

Decoupling tests from CI does not mean removing CI from the picture. Your pipeline still triggers test runs. The difference is that testing logic moves out of the pipeline YAML and into a dedicated execution layer that is built for it.

In practice that means:

Tests run as jobs inside your own containerized infrastructure, using the same networking, secrets management, and configuration your applications use. There is no environment mismatch because the test environment and the application environment are the same environment.

Tests can be triggered by anything: a CI event, a schedule, an API call, an infrastructure event, or a webhook. The trigger mechanism and the execution mechanism are separate concerns.

Results are centralized. Instead of digging through pipeline logs across multiple tools, there is one place to see what ran, what failed, and why.

That is the architecture most teams are trying to build by hand inside their CI pipelines. Testkube makes it the default.

The scenarios CI cannot handle well

Scheduled infrastructure validation. If you want to run a smoke test suite against your production cluster every hour, that is not a CI concern. Nothing changed in the codebase. There is no build event. CI was not designed for this, which is why most teams handle it with cron jobs stitched to shell scripts, and then spend engineering time maintaining those scripts when something breaks.

Pre-deployment environment checks. Before a big release, you want to validate that the target environment is ready: services are reachable, configurations are correct, dependencies are available. That validation needs to run inside the environment, not inside a pipeline that simulates it. Running it as a CI step introduces exactly the environment mismatch you are trying to catch.

Post-deployment validation. Smoke tests after deploy are common. Running them inside the same CI pipeline that executed the deployment creates a tight coupling that makes failures harder to interpret. Was it the deployment that failed or the test? Separating execution makes the signal cleaner.

Long-running load tests. A k6 or JMeter suite that runs for 30 minutes does not belong in a pipeline that should complete in under 10. Teams either skip the load test on every push and run it manually, or they accept the pipeline slowdown. Neither is a real solution.

Cross-environment test runs. When you need to run the same test suite against dev, staging, and production with different configurations, CI pipelines usually mean duplicating pipeline definitions. A dedicated orchestration layer with environment-level config makes this a single workflow.

How Testkube handles this

Testkube is a test orchestration platform that runs inside your own containerized infrastructure. Tests execute as native jobs, using your existing infrastructure rather than external test clouds or pipeline compute.

The result is a testing layer that sits alongside CI/CD instead of inside it.

Your CI pipeline still triggers Testkube via API, CLI, or native integrations with GitHub Actions, GitLab CI, Jenkins, and others. But Testkube manages what runs, where it runs, how results are collected, and how failures surface.

You get triggers from CI, schedules, events, or direct API calls. You get centralized results across every run regardless of trigger source. You get test execution that matches your production environment because it runs inside the same clusters. And you get a place to put the testing logic that should never have lived in pipeline YAML.

What teams keep from CI

One concern that comes up: if tests run outside CI, how does the pipeline know whether to proceed?

The answer is that Testkube integrates back into the pipeline. It returns exit codes, posts status checks, and surfaces pass/fail signals to whatever system triggered the run. Your pipeline gates still work. The quality signal is the same. The difference is that the execution and the result collection happened in a purpose-built layer rather than inside pipeline YAML that was never designed to handle it.

Your CI/CD tool still does its job. Testkube does a different one.

What this looks like in practice

A healthcare software company was running all of its automated tests inside Jenkins. As the environment grew, the testing stage was the single biggest contributor to pipeline slowdowns. Tests that needed to run across multiple environments required separate pipeline configurations per environment, and the team was spending DevOps time maintaining that scaffolding rather than improving coverage.

After adopting Testkube, test execution moved out of Jenkins into the containerized infrastructure the team already operated. Jenkins still triggered the runs. But scheduling, environment configuration, parallelization, and result collection were handled by Testkube. The result was 30 DevOps hours saved per week, with cleaner pipeline output and test results that were visible in one place across environments.

Where to start

The cleanest way to move tests out of CI is not to do it all at once.

Start with the tests that are already causing the most friction: the long-running ones that slow down pipelines, or the ones that behave differently across environments. Move those to Testkube first, keep the CI trigger in place, and let the pipeline consume the result as it always has.

Once that is working, scheduled runs and event-triggered execution follow naturally. The test suite does not change. The framework does not change. What changes is where execution lives and how results are collected.