Responsive

Continuous Testing as a Best Practice for GitOps

Published
July 20, 2025
Ole Lensmar
CTO
Testkube

Table of Contents

Unlock Better Testing Workflows in Kubernetes — Try Testkube for Free

Subscribe to our monthly newsletter to stay up to date with all-things Testkube.

You have successfully subscribed to the Testkube newsletter.
You have successfully subscribed to the Testkube newsletter.
Oops! Something went wrong while submitting the form.
Last updated
July 19, 2025
Ole Lensmar
CTO
Testkube
Share on X
Share on LinkedIn
Share on Reddit
Share on HackerNews
Copy URL

Table of Contents

Continuous Testing & GitOps TL;DR

TL;DR

Continuous Testing & GitOps Essentials

  • 1
    Continuous Testing integrates automated tests throughout the entire software delivery pipeline, providing instant feedback and ensuring applications remain in a releasable state at every development stage
  • 2
    GitOps uses Git as the single source of truth for infrastructure and application deployments, with tools like ArgoCD and Flux automatically reconciling live environments to match desired state
  • 3
    Test timing in GitOps depends on frequency and complexity - teams must balance between running quick sanity checks for every change versus comprehensive testing on a scheduled basis
  • 4
    A practical 3-stage approach includes rapid sanity tests for each GitOps sync, scheduled comprehensive testing during low-activity periods, and proper artifact collection for failed test troubleshooting
  • 5
    Testkube provides specialized GitOps testing using Test Workflows that integrate with tools like ArgoCD to run tests in response to infrastructure events or as part of reconciliation processes

Both Continuous Testing and GitOps practices are gaining traction in cloud-native applications delivery processes. Let’s have a look at what they are and how they can be combined to ensure adequate testing of your cloud-native applications as they go through the software development lifecycle.

What is Continuous Testing?

Continuous testing is a software development practice where automated test execution is integrated into the entire software delivery pipeline. Its purpose is to ensure that software is always in a releasable state by validating the quality of the application at every stage of the development lifecycle.

Key aspects of continuous testing are:

  1. Automation: Automated test scripts are run continuously or triggered by specific events (e.g., code commits, artifact deployments).
  2. Integration with CI/CD: Test execution should be integrated with both Continuous Integration (CI) and Continuous Delivery (CD) pipelines.
  3. Shift-left testing: Testing is performed as early as possible in the development process.
  4. Instant feedback: Test results are available immediately, helping developers and testers to quickly troubleshoot and address issues before they escalate.
  5. Testing across different levels: Testing should include unit tests, end-to-end/integration tests, functional tests (FE and API), performance tests, and security/compliance tests as applicable.

What is GitOps?

GitOps is a set of practices that leverages Git as the source of truth for managing and automating infrastructure and application deployments. It brings the benefits of DevOps, such as collaboration, automation, and versioning, to infrastructure management. Tools like ArgoCD and Flux have driven the popularity and adoption of GitOps in the Kubernetes/Cloud-Native space.

Key aspects of GitOps are:

  1. Infrastructure as Code (IaC): GitOps relies on declaring infrastructure as code and storing that code in a Git repository. This approach allows for easy version control, audit trails,  collaboration and rollbacks of the state of your infrastructure.
  2. Git as the Single Source of Truth: All infrastructure and application configurations are stored in Git repositories. Any change to the infrastructure is done by committing and pushing code to the repository, making the state of the infrastructure always aligned with the state defined in Git..
  3. Automated Deployment: Once a change is pushed to a GitOps repository, automated systems (e.g., CI/CD pipelines) ensure the changes are applied to the infrastructure or application environments - often called the reconciliation loop. This is often handled by tools like Flux or ArgoCD which continuously monitor the live environment and the state described in Git. If the live state diverges from the Git repository, these tools automatically correct the infrastructure to match the desired state.

How to Apply Continuous Testing to GitOps?

As mentioned above, a GitOps approach is often set up to continuously monitor your git repositories for state changes and ensure that these are applied to your applications and infrastructure. Running tests to validate that these changes are successful fits naturally into both GitOps and Continuous Testing paradigms. 

The question then becomes: when do we run our tests?

  • For every individual state synchronization applied to your cluster?
  • For high-level application redeployments only?
  • Periodically - out-of-bounds of GitOps synchronization processes?
  • A combination of the above?

A combination of factors will influence our decision:

  • How often do we synchronize minor tweaks to our application infrastructure?
  • How often do we update our application as a whole?
  • What kinds of tests do we need to run?
  • How much resources and time do those tests consume? 

The answers to these questions are intertwined, for example if you decide you need to run a full set of E2E tests for every minor tweak, perhaps you will have to refrain from synchronizing your application state more than once every 4 hours. On the other hand, if your DevOps team needs to synchronize infrastructure state often and continuously, it might be better to run your full E2E tests out-of-bounds - i.e. every 12 hours - while still running a small set of tests for every update.

Furthermore, running out-of-bounds tests might require you to disable GitOps synchronization while your tests are running (so your application isn’t updated in the middle of running your tests..) - which you could solve either by scheduling support in your GitOps tool, or you could have a separate testing environment that is synchronized less frequently (daily) purely for test execution.

For any implemented approach, another challenge becomes what/if to perform an automatic rollback if a test execution fails. If it is closely tied to GitOps synchronization, your GitOps tool might have this possibility built in to trigger a rollback. If you are running your tests out-of-bounds, you will likely need some manual process to troubleshoot and figure out what caused your test(s) to fail and consequently rollback your application (or infrastructure) to a previous state.

A 3-stage Approach to Testing in GitOps

As often, there is no single solution to the above challenge, so let’s start with a basic approach from which you can evolve in line with your specific needs.

If your infrastructure updates frequently (hourly), an initial 3-step strategy could be as follows:

  1. Create a set of “Sanity E2E tests” that take less than x (2?) minutes to complete, these should make sure nothing major in your application is broken (service connectivity, basic functionality) -> run these for every GitOps synchronization. If possible only run tests that are required to validate functionality that could be impacted by the changes applied during synchronization - i.e. only run tests for components that were updated and not for your entire infrastructure.

  2. Schedule a full set of E2E/performance/security tests to run on a regular (daily?) basis - preferably at a time when there will not be any need to synchronize your infrastructure. If possible, block your GitOps reconciler while these tests are running to ensure that no application or infrastructure updates are applied while your tests are running.

  3. Make sure that failed tests are notified and handled accordingly. Troubleshooting of failed tests should be easy so make sure to collect any artifacts (logs, etc) in a location easily accessible for those doing the actual troubleshooting. 

Testkube for Continuous Testing in GitOps

Testkube provides a formidable platform for Continuous Testing at large, and for testing in GitOps more specifically. Testkube uses Test Workflows as a powerful abstraction for running tests with any testing tool whenever needed; in CI/CD pipelines, in response to infrastructure events or as part of a GitOps reconciliation process. Check out the GitOps-Powered K8s Testing Machine blog-post to see a hands-on example of how Testkube can be integrated with ArgoCD, or head over to Testkube ArgoCD documentation to learn more about how to use Testkube with ArgoCD.

If you’re new to Testkube - head over to testkube.io/get-started to get started!

Top 5 GitOps and Continuous Testing FAQs

GitOps and Continuous Testing FAQs

Essential questions about integrating testing into GitOps workflows

GitOps is a deployment methodology that uses Git as the single source of truth to manage infrastructure and application updates, ensuring automated and version-controlled delivery through tools like ArgoCD or Flux. Continuous Testing, on the other hand, integrates automated testing into every stage of the software development lifecycle to validate code quality continuously.

Key differences:

  • GitOps focus: Infrastructure and application deployment automation using Git workflows
  • Continuous Testing focus: Quality assurance through automated test execution at all development stages
  • GitOps tools: ArgoCD, Flux, Jenkins X for deployment orchestration
  • Testing tools: Testkube, Tekton, Jenkins for test automation and execution
  • GitOps outcomes: Consistent, auditable deployments with rollback capabilities
  • Testing outcomes: Early bug detection, quality validation, and compliance verification

When combined, GitOps triggers can initiate Continuous Testing to verify infrastructure or application state changes before or after deployment, creating a powerful DevOps workflow.

Continuous Testing can be integrated into GitOps workflows by attaching test execution to GitOps triggers. After a Git commit that changes the desired state, a CI/CD pipeline or a GitOps controller can trigger automated testing to verify the changes.

Integration approaches:

  • Inline with GitOps syncs: Execute tests after each synchronization to validate deployment success
  • Pre-deployment validation: Run tests before applying changes to prevent problematic deployments
  • Post-deployment verification: Validate application health and functionality after successful deployment
  • Scheduled testing: Run comprehensive test suites periodically (e.g., every 12 hours) for ongoing validation
  • On-demand testing: Trigger tests manually for critical deployments or infrastructure changes

Implementation strategies:

  • Use GitOps controller webhooks to trigger test execution
  • Implement test gates in CI/CD pipelines before Git commits
  • Deploy testing tools like Testkube as Kubernetes-native solutions
  • Configure ArgoCD or Flux with test execution hooks

In GitOps environments, a layered testing approach is recommended to balance speed, coverage, and resource efficiency:

  • Sanity or smoke tests: Fast, critical path validation after every sync
    • Basic connectivity and health checks
    • Core functionality verification
    • Execution time: under 2 minutes
  • Component-level tests: Focused testing for updated services only
    • Unit and integration tests for changed components
    • API contract validation
    • Reduces testing overhead and execution time
  • End-to-end tests: Comprehensive user journey validation
    • Complete workflow testing
    • Cross-service integration verification
    • Run on scheduled basis (e.g., nightly)
  • Performance tests: Load and stress testing
    • Response time and throughput validation
    • Resource utilization monitoring
    • Scalability verification
  • Security tests: Vulnerability and compliance validation
    • Container security scanning
    • Network policy validation
    • Configuration compliance checks

This approach provides fast feedback loops with comprehensive coverage while optimizing resource usage.

The decision to block GitOps synchronization depends on the severity and context of test failures. A strategic approach considers both risk and operational efficiency:

  • Critical tests that should block:
    • Security vulnerabilities or compliance failures
    • End-to-end test failures indicating broken user journeys
    • Performance degradation beyond acceptable thresholds
    • Infrastructure health checks failing
  • Non-blocking approaches:
    • Post-deployment smoke tests with alerting
    • Performance monitoring with threshold notifications
    • Non-critical integration test failures

Implementation strategies:

  • Pause synchronization: Temporarily halt GitOps sync during critical test execution
  • Automated rollback: Configure GitOps tools to automatically revert on test failure
  • Progressive deployment: Use canary or blue-green strategies with test gates
  • Alert-based workflows: Generate notifications for manual intervention when tests fail
  • GitHub issue creation: Automatically create issues for failed tests requiring investigation

Many GitOps tools like ArgoCD support sync hooks and health checks that can implement these blocking mechanisms effectively.

Implementing Continuous Testing effectively in GitOps requires strategic planning and operational discipline:

  • Test execution optimization:
    • Use lightweight tests (≤ 2 minutes) for every GitOps sync
    • Focus on critical path validation for fast feedback
    • Implement parallel test execution where possible
  • Comprehensive testing schedules:
    • Run full E2E, performance, and compliance tests on a schedule
    • Temporarily disable GitOps reconciliation during intensive testing
    • Use maintenance windows for resource-intensive test suites
  • Test artifact management:
    • Store test results, logs, and reports centrally for easy debugging
    • Implement test result retention policies
    • Provide searchable test history and trends
  • Automation and integration:
    • Automate test-triggered rollbacks for critical failures
    • Implement intelligent alerting to reduce noise
    • Use tools like Testkube for Kubernetes-native test orchestration
    • Integrate with monitoring and observability platforms
  • Environment management:
    • Maintain dedicated testing environments that mirror production
    • Implement proper test data management and cleanup
    • Use namespace isolation for concurrent test execution

These practices ensure reliable, efficient testing that enhances rather than hinders GitOps workflows.

About Testkube

Testkube is a test execution and orchestration framework for Kubernetes that works with any CI/CD system and testing tool you need. It empowers teams to deliver on the promise of agile, efficient, and comprehensive testing programs by leveraging all the capabilities of K8s to eliminate CI/CD bottlenecks, perfecting your testing workflow. Get started with Testkube's free trial today.