How to Run Tests Outside Your CI Pipeline

The T in CI/CD is silent

Most teams discovered test automation through their CI pipeline. You add a test step, the pipeline runs it on every push, and for a while that feels like progress.

Then it starts breaking in ways that are hard to explain.

Tests that pass locally fail in CI. Tests that pass in CI fail in staging. The pipeline takes 40 minutes and half of that is tests. Someone disables a flaky test to unblock a release. Someone else adds a retry and calls it fixed. The test suite grows, and so does the anxiety around it.

The problem is not the tests. It is the architecture. CI/CD tools were designed to move code from commit to deployment. Testing got embedded inside that flow because it was convenient, not because CI was the right execution environment.

This post is about what happens when you separate the two.

What CI does well, and where it stops

CI/CD tools are good at what they were built for: detecting code changes, building artifacts, running deployment sequences, and gating releases. That is a well-defined job, and most teams rely on it working reliably.

Testing is a different job.

Tests need to run across environments, not just on push events. They need to run on schedules, on infrastructure events, before a deployment, after one, and sometimes in the middle of the night when no one changed any code. They need to scale independently of build pipelines. And they need to surface results in a way that is useful to more than just the engineer who triggered the run.

None of that maps cleanly onto how CI pipelines work.

When you run tests inside CI, you inherit all of CI's assumptions: tests run when code ships, results live inside pipeline logs, execution is tied to whatever compute the pipeline has available, and the failure context is whatever the pipeline step happened to capture before it timed out.

That works at small scale. It stops working when the environment gets more complex.

What "outside CI" actually means

Decoupling tests from CI does not mean removing CI from the picture. Your pipeline still triggers test runs. The difference is that testing logic moves out of the pipeline YAML and into a dedicated execution layer built for it.

In practice that means:

1. Tests run as jobs inside your own containerized infrastructure, using the same networking, secrets management, and configuration your applications use. There is no environment mismatch because the test environment and the application environment are the same environment.
2. Tests can be triggered by anything: a CI event, a schedule, an API call, an infrastructure event, or a webhook. The trigger mechanism and the execution mechanism are separate concerns.
3. Results are centralized. Instead of digging through pipeline logs across multiple tools, there is one place to see what ran, what failed, and why.

That is the architecture most teams are trying to build by hand inside their CI pipelines. Testkube makes it the default.

The scenarios CI cannot handle well

Five testing scenarios consistently break down when execution lives inside the CI pipeline.

Scheduled infrastructure validation

If you want to run a smoke test suite against your production cluster every hour, that is not a CI concern. Nothing changed in the codebase. There is no build event. CI was not designed for this, which is why most teams handle it with cron jobs stitched to shell scripts, then spend engineering time maintaining those scripts when something breaks.

Pre-deployment environment checks

Before a big release, you want to validate that the target environment is ready: services are reachable, configurations are correct, dependencies are available. That validation needs to run inside the environment, not inside a pipeline that simulates it. Running it as a CI step introduces exactly the environment mismatch you are trying to catch.

Post-deployment validation

Smoke tests after deploy are common. Running them inside the same CI pipeline that executed the deployment creates a tight coupling that makes failures harder to interpret. Was it the deployment that failed or the test? Separating execution makes the signal cleaner.

Long-running load tests

A k6 or JMeter suite that runs for 30 minutes does not belong in a pipeline that should complete in under 10. Teams either skip the load test on every push and run it manually, or they accept the pipeline slowdown. Neither is a real solution.

Cross-environment test runs

When you need to run the same test suite against dev, staging, and production with different configurations, CI pipelines usually mean duplicating pipeline definitions. A dedicated orchestration layer with environment-level config makes this a single workflow.

How Testkube handles this

Testkube is a test orchestration platform that runs inside your own containerized infrastructure. Tests execute as native Kubernetes jobs, using your existing infrastructure rather than external test clouds or pipeline compute.

The result is a testing layer that sits alongside CI/CD instead of inside it.

Your CI pipeline still triggers Testkube via API, CLI, or native integrations with GitHub Actions, GitLab CI, Jenkins, and others. But Testkube manages what runs, where it runs, how results are collected, and how failures surface.

You get triggers from CI, schedules, events, or direct API calls. You get centralized results across every run regardless of trigger source. You get test execution that matches your production environment because it runs inside the same clusters. And you get a place to put the testing logic that should never have lived in pipeline YAML.

What teams keep from CI

One concern comes up consistently: if tests run outside CI, how does the pipeline know whether to proceed?

The answer is that Testkube integrates back into the pipeline. It returns exit codes, posts status checks, and surfaces pass/fail signals to whatever system triggered the run. Your pipeline gates still work. The quality signal is the same. The difference is that execution and result collection happen in a purpose-built layer rather than inside pipeline YAML that was never designed to handle it.

Your CI/CD tool still does its job. Testkube does a different one.

What this looks like in practice

A healthcare software company was running all of its automated tests inside Jenkins. As the environment grew, the testing stage became the single biggest contributor to pipeline slowdowns. Tests that needed to run across multiple environments required separate pipeline configurations per environment, and the team was spending DevOps time maintaining that scaffolding rather than improving coverage.

After adopting Testkube, test execution moved out of Jenkins into the containerized infrastructure the team already operated. Jenkins still triggered the runs. But scheduling, environment configuration, parallelization, and result collection were handled by Testkube. The result: 30 DevOps hours saved per week, cleaner pipeline output, and test results visible in one place across environments.

Where to start

The cleanest way to move tests out of CI is not to do it all at once.

Start with the tests already causing the most friction: the long-running ones that slow down pipelines, or the ones that behave differently across environments. Move those to Testkube first, keep the CI trigger in place, and let the pipeline consume the result as it always has.

Once that is working, scheduled runs and event-triggered execution follow naturally. The test suite does not change. The framework does not change. What changes is where execution lives and how results are collected.

Key takeaways

• CI/CD was built to ship code, not run tests. Pipelines handle build, deploy, and release gates well. Test orchestration is a different job that needs a different execution layer.
• Decoupling tests from CI does not replace CI. Your pipeline still triggers tests and receives pass/fail signals. The execution and result collection just move into a layer built for the job.
• CI struggles with five specific scenarios: scheduled infrastructure validation, pre-deployment environment checks, post-deployment validation, long-running load tests, and cross-environment runs.
• The fix is test orchestration that runs in your own infrastructure. Tests execute as Kubernetes-native jobs in the same environment as your applications, accept triggers from multiple sources, and surface results centrally.
• Start with the tests causing the most friction. Move long-running or environment-sensitive tests to a dedicated orchestration layer first, keep CI as the trigger, and expand from there.

Frequently asked questions

Why should I run tests outside my CI pipeline?

CI pipelines were built to move code from commit to deployment, not to orchestrate test execution. Running tests inside CI ties them to build events, locks them to whatever compute the pipeline has available, and buries results in pipeline logs. Decoupling test execution from CI frees you to run tests on schedules, infrastructure events, or API calls, and to scale testing independently of builds.

What is decoupled testing?

Decoupled testing means separating test execution from your CI/CD pipeline. Your pipeline still triggers tests, but testing logic, execution, and result collection happen in a dedicated layer built for that purpose. The CI/CD tool handles builds and deployments. The test orchestration layer handles testing. The two integrate through triggers and status checks but operate independently.

Does decoupling tests from CI mean replacing my CI pipeline?

No. Your CI pipeline still triggers test runs and receives pass/fail signals to gate deployments. The difference is that test execution moves out of pipeline YAML and into a dedicated orchestration layer. Pipeline gates still work, exit codes still flow back, and status checks still post to your repo. CI does its job; the test orchestration layer does a different one.

What testing scenarios does CI struggle with?

CI struggles with five scenarios. Scheduled infrastructure validation that has no build event. Pre-deployment environment checks that need to run inside the target environment. Post-deployment smoke tests where coupling to the deploy pipeline muddies the signal. Long-running load tests that slow pipelines below acceptable thresholds. Cross-environment test runs that require duplicated pipeline definitions per environment.

How does Testkube integrate with my existing CI/CD tool?

Testkube integrates with CI/CD tools through API calls, CLI commands, or native plugins for GitHub Actions, GitLab CI, Jenkins, CircleCI, and others. Your pipeline triggers Testkube to run tests, Testkube executes them as Kubernetes-native jobs in your infrastructure, and results flow back to the pipeline as exit codes and status checks. The pipeline still gates deployments based on the test outcome.

Where should I start when moving tests out of CI?

Start with the tests causing the most friction: long-running tests that slow pipelines, flaky tests with environment mismatch issues, or test suites that need different configurations per environment. Move those to a test orchestration layer first, keep the CI trigger in place, and let the pipeline consume the result as it always has. Scheduled and event-triggered tests follow once the pattern is working.

What is test orchestration?

Test orchestration is a dedicated layer that manages how tests run, when they run, where they run, and how their results are collected. It handles triggers from multiple sources (CI events, schedules, API calls, infrastructure events), executes tests across environments, parallelizes runs, and centralizes results. Unlike CI/CD, which is built for shipping code, test orchestration is built specifically for running tests at scale.

‍