How to Test Kubernetes Operators and CRDs

Kubernetes operators and Custom Resource Definitions (CRDs) let platform teams publish their own APIs and automate how applications and infrastructure are managed. That reach brings risk with it. A CRD can accept a configuration it should reject, an operator can fail to reconcile toward the desired state, and a Helm upgrade can break a deployment that was healthy before it ran.

How does a platform team know that its infrastructure works? CRDs define the API that developers configure, operators act on the resources created from that API, and Helm charts and deployment pipelines introduce failure modes of their own. Manual checks, such as running kubectl apply and inspecting pod status, cannot reliably catch failures in any of these parts. Confidence comes from automated testing at multiple layers, with each layer validating a different aspect of platform reliability.

This article explains what to test across CRDs, operators, Helm charts, platform APIs, and deployments, and how to run those tests inside the cluster with Testkube across environments. We will look at why infrastructure code reaches production with less automated verification than application code. The causes are structural, and they shape any testing approach that works.

How Infrastructure Gets Tested Today

Infrastructure testing usually grows one pipeline step or manual habit at a time. A CI pipeline validates manifests and may run automated tests, but confirming what the platform actually does after deployment can still depend on manual checks, such as inspecting pod status and reading operator logs. Those checks show the state of the platform on the day they are run and leave no record afterward.

Coverage depends on whoever last touched a component, because that person decided what got tested. Two charts in the same platform can carry very different checks as a result, and a gap in one is invisible from the other. Results stay in a terminal session or a chat thread, leaving no record of what passed on the previous release and no baseline for judging a regression.

Teams pay for this later, as incidents that reach every consumer of the component. A CRD change that rejects manifests teams already committed, or a chart upgrade that leaves an operator unable to reconcile, lands on all of those teams at the same moment. The next section explains why infrastructure is harder to test than application code.

What Makes Infrastructure Harder to Test

Four properties of infrastructure make it harder to test than application code.

Cluster dependence: Kubernetes behavior depends on a running API server. Admission, defaulting, validation, and reconciliation happen only inside a live cluster, and clusters are slow and costly to create. Application logic can often be verified in milliseconds with no infrastructure behind it, so every check on infrastructure carries a setup cost that application code mostly avoids.

Eventual consistency: Operators work toward the desired state over repeated passes, as a reconcile loop acts, observes, and acts again. The state right after an apply is only an intermediate one, so a check made at that moment can pass while the final state is wrong.

Carried state: A CRD or chart upgrade runs against objects that already exist, a situation a fresh install never reproduces. Defects that come from old objects meeting new code stay hidden until an upgrade runs against that existing state.

Rare failure paths: An unavailable webhook, API throttling, and node loss all happen in real clusters, yet a healthy environment rarely produces them unless they are caused deliberately.

A plain manifest apply covers none of these cases, so each one calls for its own kind of test, and the next section organizes those tests into layers.

The Layers of Infrastructure Testing

Testing infrastructure well takes five layers, listed here from cheapest to most expensive.

1. Static checks on CRD schemas, chart templates, and rendered manifests run in seconds and need no cluster.
2. Unit tests check reconciliation logic and conversion functions.
3. Integration tests run against a real API server and verify admission, defaulting, validation, and controller behavior without full workloads.
4. End-to-end tests run on a real cluster, where pods start and services respond.
5. Upgrade and failure tests begin from existing state and inject faults.

Each layer catches a different class of defect and misses others, and both cost and run time rise from the first layer to the last. Cheap layers therefore run on every commit, and expensive layers run on events or on a schedule, keeping feedback fast while coverage stays wide. Running all five from separate places, such as a CI job for one and a hand-run script for another, would recreate the scattered results described earlier. A single platform that runs all five layers inside the cluster avoids that.

Testing Each Part of the Platform with Testkube

Testkube is a Kubernetes-native test orchestration platform that runs tests inside the cluster being tested. For infrastructure that depends on a live API server, that placement is the main reason to use it, and three properties of the platform make the case.

Defined as Kubernetes resources: A Test Workflow is a custom resource that names a container image, optional source content such as a Git repository, and the steps to run. Workflows are plain manifests that sit in Git beside the chart or operator they test, with the same review and versioning as the code they verify. That ends the situation where coverage depends on whoever last touched the component.

Run inside the cluster: Each run becomes a Kubernetes Job and Pod under a service account the workflow can set, in a namespace Testkube is configured to use, and the cluster's own scheduling places it. Capacity, node placement, and topology spread are controlled through ordinary Pod configuration. Because the test shares the cluster network, it reaches internal services and webhooks directly, with no ingress route to open for the test. A test scripted into a CI job runs on the CI runner, which usually sits outside the cluster and has to be given a path in.

Any tool as a step: Any tool that runs in a container image can be a workflow step. The Testkube documentation includes an example that runs Chainsaw, an open-source declarative testing tool for Kubernetes objects, from the official Chainsaw image. Existing tests carry over without a rewrite, and every layer of testing executes from one platform.

With one place to run every layer, each component can be given the tests that fit how it fails. Each of the five sections below states the issue, the tests that address it, and how Testkube runs them.

CRDs

A CRD extends the Kubernetes API with a new resource type and defines the schema that every resource of that type must follow. Teams write and commit manifests against that schema, so a schema change can reject or silently alter manifests they have already committed. Developers meet the platform through that schema, and it deserves the same scrutiny as any other published interface.

The tests cover four areas:

• The structural schema, its defaults, and the pruning of unknown fields.
• Common Expression Language (CEL) validation rules, declared under x-kubernetes-validations, which need both accepted and rejected sample objects to prove they behave as written.
• Transition rules, which need update tests that show which changes to an existing object are allowed.
• For multi-version CRDs, a conversion round trip from one version to another and back, which confirms that no field is lost along the way.

A Testkube workflow applies valid and invalid sample objects to a test cluster and asserts that the API server accepts or rejects each one, using a declarative tool such as Chainsaw. Running it on every change to the CRD files means a breaking change is caught before the CRD reaches a cluster that developers use.

The Chainsaw example in the Testkube docs shows the setup. The workflow clones a test repository, runs Chainsaw from the official image, and saves a JUnit-style XML report as an artifact:

kind: TestWorkflow
apiVersion: testworkflows.testkube.io/v1
metadata:
  name: simple-chainsaw-test
  namespace: testkube
spec:
  content:
    git:
      uri: https://github.com/cerebro1/chainsaw-testkube-demo.git
  container:
    image: ghcr.io/kyverno/chainsaw:latest
  pod:
    serviceAccountName: chainsaw-service-account
  steps:
    - name: Run test
      shell: |
        chainsaw test --report-format XML --report-path /data/repo --report-name chainsaw-report /data/repo/chainsaw-test/basic-test
      artifacts:
        paths:
          - /data/repo/chainsaw-report.xml
status: {}

The workflow runs under chainsaw-service-account. Chainsaw creates a temporary namespace for each test, so that account needs a ClusterRole that can create, get and delete namespaces. The docs page includes the ServiceAccount, ClusterRole and ClusterRoleBinding manifests.

Operators

Next comes the operator that acts on the resources a CRD defines. An operator's controller is level-triggered and asynchronous, which means a defect often appears only after the second or third reconcile, after a restart, or when two updates collide. A test that applies a resource and checks once will miss all three.

The tests start with the cheapest checks:

• Decision logic, checked with fake clients.
• Controller behavior, tested against a real API server and etcd with envtest, the controller-runtime package that starts those two components on their own. With no kubelet, no running pods, and no garbage collection of owned objects, assertions target the objects the controller writes, since no workload ever starts. Within those suites, the behaviors worth asserting directly are:
  • Idempotency of repeated reconciles
  • Finalizers and deletion
  • Owner references
  • Status conditions and observedGeneration
  • Requeue and backoff
• Three further areas that deserve tests of their own:
  • Leader election
  • Webhook certificates and failure policy
  • Role-based access control (RBAC) limited to the declared permissions

The Go integration suite runs as a workflow step in a container, and an end-to-end workflow installs the operator on a real cluster, applies custom resources, and asserts on final state and status conditions. Second-reconcile failures then surface while the pull request is still open.

Helm Charts and Manifests

Helm charts add a different failure mode: the templating itself. A chart can render valid YAML for its default values and invalid YAML for another combination, and because Helm installs CRDs from a chart's crds/ directory but never upgrades them, an upgrade can behave differently from an install of the same chart.

Most of the testing happens before a cluster is involved:

• Linting.
• Rendering templates across representative combinations of values.
• Validating values against values.schema.json.
• Checking the rendered manifests against the target Kubernetes version.
• Policy checks on the rendered output, inspecting security context, resource limits, and image sources.

The cluster-backed tests then install and uninstall the chart on a clean cluster and upgrade it from the previous release.

Install and upgrade workflows run helm in a container against the test cluster, while the static checks stay in CI because they need no cluster. That split places each defect where it is cheapest to find: most chart defects are visible before any cluster exists, and the remainder fail in the install or upgrade test before release.

Platform APIs and Services

After the packaging, the services it delivers need their own tests. Platform APIs and services have consumers, and those consumers break when a response or behavior changes, however healthy the pods look. A running pod confirms that a process started but cannot show whether the contract that process serves still holds.

The tests fall into five groups:

• Contract tests check responses against the OpenAPI definition.
• Smoke checks after each rollout verify the service levels the platform promises.
• Scale tests measure reconcile latency and work queue depth with thousands of custom resources, because those figures only become visible at volume.
• Failure tests simulate a dependency outage and API throttling.
• Security checks include container image scans and policy enforcement.

Because Testkube runs inside the cluster, API and load tests reach internal services directly, and no service has to be exposed just to be tested. A rollout is then judged by the behavior its consumers see.

Upgrades and Deployment Processes

The last component to test is the upgrade path, a scenario that a clean install cannot reproduce. Old objects, old CRD versions, and old operator state all meet new code during an upgrade, and the deployment process around it, from the pipeline to the GitOps sync to promotion between environments, can fail independently of the software being deployed.

The tests follow both concerns.

• For the software, they:
  • Upgrade the operator from the previous release with existing custom resources in place
  • Apply CRD schema changes against stored objects
  • Run Helm upgrade and rollback
  • Check Kubernetes minor version skew
• For the process, they:
  • Run a server-side dry-run and diff before the sync
  • Enforce promotion gates between environments
  • Define rollback criteria

An upgrade workflow installs the previous release, creates representative resources, performs the upgrade, and asserts on the result. The pipeline waits for that result before promoting a release, so upgrade and rollback paths are exercised before any consumer depends on them.

Taken together, the five test sets cover the API, the code that acts on it, the packaging, the services, and the path to production. What remains is deciding where and when each one runs.

Running the Tests Across Environments

Running every test in every environment would waste money and time. Each layer therefore needs an environment, an execution order, and a trigger that match its cost.

Matching Layers to Environments

Local kind clusters and ephemeral per-pull-request clusters serve developer feedback and integration tests. A long-lived shared cluster holds the upgrade and soak tests, which need a cluster with history. Reserving the expensive environments for the tests that need them keeps cost in line with the value each run returns.

Ordering and Parallelism

Once each layer has an environment, the order in which runs execute determines how soon an engineer sees a failure. Stages run in order of cost, with static checks first and cluster tests after, so a cheap failure stops the run before an expensive environment is touched. Independent workflows then run in parallel within what the cluster can hold, using Testkube parallel steps and Pod placement settings. Engineers wait only for the slowest independent suite.

Choosing the Trigger for Each Layer

With environments and ordering in place, each layer still needs a trigger that starts its runs. The CI pipeline starts the pull request run, covering everything up to integration tests. A Kubernetes event, such as a Deployment update, starts the post-rollout smoke checks. A schedule starts the upgrade, scale, and failure tests, which are too slow to run on every commit. With triggers assigned this way, each kind of defect is caught by the earliest and cheapest run that can see it.

Reading Results in One Place

Whichever trigger starts a run, its results should be readable in the same place. Every workflow keeps its run history and attaches logs and artifacts to each execution, giving a failed run its own evidence. Labels on each workflow for its component and its layer let the Testkube dashboard and CLI group results by either one, which gives a per-component view of which layers cover it. That history gives anyone a direct answer to the question this article opened with: what passed on the previous release, and what has changed since.

Conclusion

Infrastructure earns trust the same way application code does, through tests that run automatically and report to one place. For a Kubernetes platform, that means a test set for each component: schema and admission checks for CRDs, reconcile and lifecycle tests for operators, rendering and upgrade tests for Helm charts, contract and load tests for platform services, and rehearsed upgrades on the path to production.

Teams can build toward that in four stages. Static checks on every commit come first, followed by CRD contract tests and operator integration tests. End-to-end and upgrade tests on ephemeral clusters come next, and scheduled failure, scale, and soak tests with reporting complete the set.

The best place to begin is static checks together with the CRD and operator tests, because they cost the least to run and remove the most risk. The tradeoff is that upgrade failures stay invisible until end-to-end and upgrade tests exist, so that gap belongs on the roadmap from the start. A practical first step: list every platform component, record which layers test it today, and close the gap on the component with the most consumers.

Frequently asked questions

{{banner-get-started}}