Continuous Testing as a Best Practice for GitOps

Both Continuous Testing and GitOps practices are gaining traction in cloud-native applications delivery processes. Let’s have a look at what they are and how they can be combined to ensure adequate testing of your cloud-native applications as they go through the software development lifecycle.

What is Continuous Testing?

Continuous testing is a software development practice where automated test execution is integrated into the entire software delivery pipeline. Its purpose is to ensure that software is always in a releasable state by validating the quality of the application at every stage of the development lifecycle.

Key aspects of continuous testing are:

1. Automation: Automated test scripts are run continuously or triggered by specific events (e.g., code commits, artifact deployments).
2. Integration with CI/CD: Test execution should be integrated with both Continuous Integration (CI) and Continuous Delivery (CD) pipelines.
3. Shift-left testing: Testing is performed as early as possible in the development process.
4. Instant feedback: Test results are available immediately, helping developers and testers to quickly troubleshoot and address issues before they escalate.
5. Testing across different levels: Testing should include unit tests, end-to-end/integration tests, functional tests (FE and API), performance tests, and security/compliance tests as applicable.

What is GitOps?

GitOps is a set of practices that leverages Git as the source of truth for managing and automating infrastructure and application deployments. It brings the benefits of DevOps, such as collaboration, automation, and versioning, to infrastructure management. Tools like ArgoCD and Flux have driven the popularity and adoption of GitOps in the Kubernetes/Cloud-Native space.

Key aspects of GitOps are:

1. Infrastructure as Code (IaC): GitOps relies on declaring infrastructure as code and storing that code in a Git repository. This approach allows for easy version control, audit trails,  collaboration and rollbacks of the state of your infrastructure.
2. Git as the Single Source of Truth: All infrastructure and application configurations are stored in Git repositories. Any change to the infrastructure is done by committing and pushing code to the repository, making the state of the infrastructure always aligned with the state defined in Git..
3. Automated Deployment: Once a change is pushed to a GitOps repository, automated systems (e.g., CI/CD pipelines) ensure the changes are applied to the infrastructure or application environments - often called the reconciliation loop. This is often handled by tools like Flux or ArgoCD which continuously monitor the live environment and the state described in Git. If the live state diverges from the Git repository, these tools automatically correct the infrastructure to match the desired state.

How to apply Continuous Testing to GitOps?

As mentioned above, a GitOps approach is often set up to continuously monitor your git repositories for state changes and ensure that these are applied to your applications and infrastructure. Running tests to validate that these changes are successful fits naturally into both GitOps and Continuous Testing paradigms. 

The question then becomes: when do we run our tests?

• For every individual state synchronization applied to your cluster?
• For high-level application redeployments only?
• Periodically - out-of-bounds of GitOps synchronization processes?
• A combination of the above?

A combination of factors will influence our decision:

• How often do we synchronize minor tweaks to our application infrastructure?
• How often do we update our application as a whole?
• What kinds of tests do we need to run?
• How much resources and time do those tests consume? 

The answers to these questions are intertwined, for example if you decide you need to run a full set of E2E tests for every minor tweak, perhaps you will have to refrain from synchronizing your application state more than once every 4 hours. On the other hand, if your DevOps team needs to synchronize infrastructure state often and continuously, it might be better to run your full E2E tests out-of-bounds - i.e. every 12 hours - while still running a small set of tests for every update.

Furthermore, running out-of-bounds tests might require you to disable GitOps synchronization while your tests are running (so your application isn’t updated in the middle of running your tests..) - which you could solve either by scheduling support in your GitOps tool, or you could have a separate testing environment that is synchronized less frequently (daily) purely for test execution.

For any implemented approach, another challenge becomes what/if to perform an automatic rollback if a test execution fails. If it is closely tied to GitOps synchronization, your GitOps tool might have this possibility built in to trigger a rollback. If you are running your tests out-of-bounds, you will likely need some manual process to troubleshoot and figure out what caused your test(s) to fail and consequently rollback your application (or infrastructure) to a previous state.

A 3-stage approach to Testing in GitOps

As often, there is no single solution to the above challenge, so let’s start with a basic approach from which you can evolve in line with your specific needs.

If your infrastructure updates frequently (hourly), an initial 3-step strategy could be as follows:

1. Create a set of “Sanity E2E tests” that take less than x (2?) minutes to complete, these should make sure nothing major in your application is broken (service connectivity, basic functionality) -> run these for every GitOps synchronization. If possible only run tests that are required to validate functionality that could be impacted by the changes applied during synchronization - i.e. only run tests for components that were updated and not for your entire infrastructure.
   
   
2. Schedule a full set of E2E/performance/security tests to run on a regular (daily?) basis - preferably at a time when there will not be any need to synchronize your infrastructure. If possible, block your GitOps reconciler while these tests are running to ensure that no application or infrastructure updates are applied while your tests are running.
   
   
3. Make sure that failed tests are notified and handled accordingly. Troubleshooting of failed tests should be easy so make sure to collect any artifacts (logs, etc) in a location easily accessible for those doing the actual troubleshooting. 

Testkube for Continuous Testing in GitOps

Testkube provides a formidable platform for Continuous Testing at large, and for testing in GitOps more specifically. Testkube uses Test Workflows as a powerful abstraction for running tests with any testing tool whenever needed; in CI/CD pipelines, in response to infrastructure events or as part of a GitOps reconciliation process. Check out the GitOps-Powered K8s Testing Machine blog-post to see a hands-on example of how Testkube can be integrated with ArgoCD, or head over to Testkube ArgoCD documentation to learn more about how to use Testkube with ArgoCD.

If you’re new to Testkube - head over to testkube.io/get-started to get started!