Scenario: Scripts to manage the scripts

This is a fictional but representative engineering organization. It is not a customer, nobody in it is a real person, and nothing in it is a quote. The numbers about the organization are arithmetic, with the inputs shown.

The tooling is leaving with its author

Eight months ago one of the four built a small testing agent to run agent-written tests against staging, after the agents' own sandboxes had let two bugs through. It worked, so it grew. A wrapper for each coding agent, because each one calls tools differently. A results store, because chat transcripts vanish. A retry script. A script that decides what counts as flaky. A dashboard, because a VP asked for one. Together it is a testing platform that nobody decided to build, and one person understands it. That person has given notice.

Here's the math: one engineer, roughly a third of their time, for eight months, plus about a month for a replacement to learn it well enough to change anything safely.

Works, and the team knows it. Depends on whoever built it. Every team that needs one builds its own.

The caution behind it is sensible. Most teams do not let agents run unsupervised, and that is exactly why wrappers like this one get built.

What they did

They had three options: document the tooling and assign a second maintainer; rewrite it properly as an internal product; or replace it with a shared service and keep only the parts that held knowledge the service did not have. They chose the third.

1. Agents talk to Testkube directly. Each coding agent reaches Testkube over MCP, so there is no wrapper per agent to maintain.
2. Results live in execution history. Every run, with logs and artifacts, is kept by Testkube, so the homegrown results store is retired.
3. Tests run as native jobs in the team's clusters. No runner scripts. The same frameworks the developers already used.
4. The flakiness rules survived. The author's rules for what counts as flaky captured eight months of learning about this team's tests. They were kept as configuration rather than rewritten.
5. Some things were dropped. The one-off retry logic did not map to anything in the shared service. The team decided to live without it.

What changed

The developers did not change how they work. Their agents write tests the same way; the tests go somewhere now. The platform team is back to four people doing platform work, and the next person to join does not inherit a tool only one predecessor understood.

Migrating 60 developers' agent configurations took longer than planned. It is also fair to say that a team with a stable maintainer and tooling that had stopped growing could reasonably have chosen to keep it. The case for replacing homegrown tooling gets stronger as it grows and as the number of people who understand it shrinks.

If this is you

Stop building the testing tool.

Run tests where your systems run, with results your whole team can see.

Start free trial