Your Platform Is a Product. Test It Like One.

How do you know that the infrastructure your developers depend on actually works before you put it in their hands?

Ask a platform team that question and the answer is often some version of "we find out when something breaks." A chart upgrade goes out, an operator moves to a new version, a network policy changes, and the first sign of trouble is a message from a developer whose deploy is stuck.

That's an odd place to be for a team that ships software every week. Platform engineering has spent years arguing that the internal platform should be run like a product. We agree, and we think most teams stop short of the part that matters most: testing it.

Your developers are already your customers

Platform teams ship a lot: Kubernetes clusters, operators and CRDs, Helm charts, ingress and service mesh config, RBAC policies, internal APIs, CI templates, and the golden paths that tie them together. Developers consume all of it the way customers consume a product. They build on it and expect it to work, and when it doesn't, trust goes fast.

One e-commerce platform team we work with describes their setup as a single generalized Helm chart that gives developers "levers and switches" for each environment. That chart is a product with developers as its users, and every change to it is a release.

Product teams wouldn't push a release to customers without running tests first. Platform changes go out every week with little more than a spot check. Most teams have adopted the product mindset without the testing discipline that's supposed to come with it.

Where platform testing breaks down today

Infrastructure sits under everything engineering does, and it's still mostly tested by hand or tested unevenly. The pattern looks similar across the teams we talk to:

• After an upgrade, someone runs a few kubectl commands, glances at a dashboard and calls it done.
• A few components have tests. Most don't, and nobody has a full list of which is which.
• A broken chart upgrade or an operator regression shows up first as a failed deploy on another team.
• Teams lose trust and start pinning old versions or routing around the platform entirely.

At one trading firm we spoke with, the infrastructure team runs upgrades and security patches on weekends. When an upgrade finishes, a representative from each of three application teams logs on to check that their services still work. Each check takes about an hour. In their words, the process is "not completely manual, but most of it is manual."

The same team said the pace of upgrades and patching keeps climbing. Every upgrade means another round of checklists across teams, so manual checks have become the limit on how fast infrastructure can move.

Shaky infrastructure also makes application tests fail for reasons that have nothing to do with the code, which is a big source of flaky tests in CI/CD. For the technical side of the problem, see our guide to infrastructure testing in Kubernetes.

What it means to test a platform like a product

Testing a platform like a product means giving platform changes the same discipline product teams give releases. Four practices make the biggest difference.

Test the infrastructure layer on its own

Helm charts, operators, CRDs, cluster config, ingress rules, RBAC and internal APIs are all release artifacts. They need their own tests, separate from the application tests that run on top of them. We walk through how to validate each Kubernetes infrastructure layer in a separate post.

One fintech platform team uses Testkube only for this. They built a test workflow for each core component on their EKS clusters, more than 24 services including cert-manager and trust-manager. Every image update and every cluster upgrade runs those workflows automatically. When something fails, they know which component broke and where to look.

Run tests where the platform runs

Testing from an external CI runner misses a lot of what makes infrastructure fail: in-cluster networking, internal services that aren't exposed, permissions and resource limits. Tests that run inside the cluster see the platform the way developer workloads see it.

They also exercise the whole platform working together instead of one service in isolation. That makes it easier to trace a failure to the right layer, whether that's the infrastructure, the service or the test itself. There's a real cost to testing outside the cluster, and platform teams pay most of it.

Validate every platform change before rollout

A platform change should go through the same gate as an app release. Tests run automatically when the change lands, and the results decide whether it rolls out.

Kubernetes events make this practical. An upgrade can set off infrastructure regression tests on its own and post to Slack if something fails, so nobody has to remember to click run. Ephemeral test environments let you run those checks without risking shared clusters, and a GitOps workflow keeps them versioned alongside the platform config. We've written more about where GitOps and TestOps meet in platform engineering.

Give everyone one view of quality

Platform tests and application tests should report to the same place. When a developer's pipeline goes red, they need to know quickly whether their code broke or the platform did. Without a shared view, that question turns into a Slack thread and an afternoon of guessing.

Why this gets harder as you scale

At ten services, a broken platform component is an annoyance. At a thousand, it's an incident across dozens of teams. As the organization grows, more developers depend on every piece of the platform, and one untested change can break many teams at once.

Manual checks don't grow with the organization. One Testkube customer tests only infrastructure, across more than 300 clusters, on every deployment. At that size, checking by hand isn't an option. If testing doesn't keep up, the platform team becomes the bottleneck it was created to remove. Our post on scaling test automation in Kubernetes covers the mechanics.

What engineering leaders get out of it

What does it cost when your platform is down for an hour? For engineering leaders, the answer shows up first in release confidence. Application tests only mean something if the layer under the application is known to work, and a green test suite on a broken platform gives false confidence.

It also shows up in engineering capacity. Platform engineers who spend their week firefighting infrastructure failures aren't improving the platform, and developers waiting on broken environments aren't shipping. Testing the platform turns both costs into something you can measure. That's the foundation of continuous quality governance for platform engineering.

From scattered checks to test unification

Most platform teams already have some tests. The problem is that they're spread across scripts and CI jobs, with a wiki checklist filling the gaps. Test unification brings them under one orchestration layer that runs infrastructure and application tests in the cluster, across every tool, with shared results. It also lets the platform team offer testing as a self-service capability, the same way it offers deployments. We see test unification as the next phase of platform engineering.

AlphaSense shows where this leads. Their five-person AI platform team doesn't write tests. It builds and runs the testing platform for more than 500 engineers across about 50 teams, and its customers are AlphaSense developers and QA engineers.

Before Testkube, their tests ran through Jenkins, and clusters were registered by hand in the UI. Clusters were recreated or upgraded about once a month, so the configuration went stale every time. A move to run on AWS, Google Cloud and Azure made that approach unsustainable. The team deployed Testkube with a Helm chart, moved all test orchestration into Git, and stopped allowing resources to be created from the UI.

The results:

• About 5,000 tests across about 350 repositories
• CI execution time cut 2 to 3x through sharding
• Failed tests now block deployments
• Three tools retired: Jenkins, the Moon browser grid and custom Helm charts

"We always know the history of anything created, and we always know who to go to if something is broken," said Valerii Timofeev, Staff QA Automation Engineer at AlphaSense. Read the full AlphaSense case study.

The platform teams that win will be the ones that test

Platform engineering promised developers a paved road. That promise only holds if the road is tested before people drive on it. Teams that test their platform before every change reaches developers will earn the trust and adoption that make a platform worth building. Teams that don't will keep hearing about failures from their users first.

So go back to the question we started with: how do you know the infrastructure your developers depend on actually works before you put it in their hands? If the answer is still "we find out when it breaks," that's where to start.

Frequently Asked Questions

{{banner-get-started}}