Testing in your CD Pipelines – the CDEvents way | CD MiniSummit

CONTINUOUS DELIVERY MINI SUMMIT

I am Ole Lensmar, CTO of Testkube, as we heard. Thank you so much. Been in the space for a long time. If you're familiar with OpenAPI, I was the chairman of the OAI, OpenAPI initiative, bought a SmartBear and other companies in the API space.

Testkube is an open source project, which is a, Testkube is also a member of the CDF, but the project itself has not been donated yet, something that we should talk about, Laurie, perhaps.

I'm gonna this is probably the most technical talk you're gonna hear today. So for the developers here, this is gonna be very hard on, and it's also gonna be very, the slide design is very Nordic. I'm from Stockholm, so it's very minimalistic. IKEA, not a lot of memes and colors. There's some purple here.

Okay.

What is CDEvents

So let's just start with CD events. Also, I want to just say, we've heard three talks now, I think, saying how important testing is. So I think that's great to hear.

I think everyone kind of agrees with that, but we've also seen a lot of changes in testing and testing practices, and that's what we're trying to address here. Just as a quick recap, CDEvents is was born out of the need that realization that different CI and CD tools and platforms need to communicate as we're building out more and more complex pipelines. We want those to be kind of, not we want them to be loosely coupled, and that would preferably be through an events eventing architecture. And the CDEvents protocol was one of the projects then that was created.

Andrea, who's sitting here, is the lead for that. So if you have any technical questions about CDEvents specifically, he can answer all of those. It's built on top of CloudEvents, so it's which in itself is another project and and a pretty established standard. So this is something that's kind of well, rooted in the space.

And it has a it's incubated in in the CDF website where you there's a website you can go to.

The CDF defines, events for a bunch of things. This is just from the website itself. So there's events related to source control. There's events related to continuous integration. There's events related to testing, which is what I'm gonna talk about, continuous deployment and continuous operations and cloud events binding, etcetera. So it's it it covers a rather broad area of within, CI/CD, but I'm gonna be focusing on specifically on testing.

Testing Events in CDEvents

And this is something, obviously, Testkube, the project I'm from, is about testing, which you can probably guess by the name, Something that we've, very, you know, enthusiastic about. And we when we discovered the CDEvents spec, we kind of quickly realized there's maybe a need to to extract what was at that at that point a very small support for testing, within, I think it was the continuous integration events. There was, there was a couple of subjects and predicates.

Why Test Events

So why do we why did we think that there's a need for new events specifically for testing?

Well, I think as we've already heard a couple of times today, today, in a distributor architecture, tests are now run not just as part of your build jobs in Jenkins or git or GitHub, but often run asynchronously. They can be run manually. They can be run as a response to events happening in your clusters or from your incident management. There's people doing testing production.

We have people who don't even run tests as part of their CI/CD pipelines because they take too long, and instead, they schedule their tests to run every thirty minutes and use that as a way to quality gate their systems. So people are not running tests maybe as strictly as they were ten, fifteen years ago. And so to that to kind of cater to that and to those all those use cases and those the decoupling of testing from your traditional static build jobs, it felt to us that the need for events related to testing was kind of pretty obvious.

Testing Events

So what we did then was we defined a couple of events, and as I said already, these replaced the previously defined test suite and test case subjects that were in the continuous integration category. All the documentation and the adjacent schemas and the examples, they're all on the repo, on GitHub. So it's all there, and this was released as version 0.1.0. It's part of 0.3.0 CDEvents. I guess this was in June something. I don't remember earlier this year at least.

Okay. Let's just dive. This is gonna be very techy and lots of monospace fonts, which means code.

So what we've basically done is we've defined three, subjects, a test case run, which models the execution of a test, a, test suite run, which models the execution of a test suite, and the test output, which models an output from a test case run, specifically. So important to note here that we're not modeling the test case itself or the test suite itself. It's it's about the actual execution, and we heard similar earlier today about pipeline runs and task runs, etcetera. So these are defined as separate, subjects. And then for each of these, we've defined predicates, queued, start to finish, for the first two, and then it published. And I'm gonna walk you just through what that looks like, and I'm gonna show an example at the end, of course.

testCaseRun

So testCaseRun, and if you look at the CDEvent spec, there's it defines a couple of common, fields that all objects have or all subjects, so, and some of those are inherited, from there. But specifically, the last three here, for each testCaseRun, there's an environment, so you know which environment it's running in.

You can provide a test case that is kind of an abstract reference to a test case somewhere, and this is very abstract. The test case could be a JUnit test. It could be a Postman ate collection. It could be a Cypress test. It could be, you know, whatever testing tool you're using.

And a testCaseRun can also be related to a testSuiteRun, and this is when it comes to more complex orchestration of tests, which we're seeing more in integration test scenarios where you might, as part of an integration test, run both API tests, UI tests, security tests, load tests, either at the same time or in parallel just to kind of see how everything works out. And then you orchestrate all of those into a testSuiteRun. And maybe a little bit complex, but that's how usually things usually end up if you're doing this kind of thing anyway.

So just looking at the queued, predicate for test case run, so this is an event that would be emitted when a test case is being scheduled to run. It hasn't actually run.

So and it might be waiting for some applicable constraints to come into place. Right? It could be resource availability.

It might be waiting for something else to finish. It might be, you know, waiting for something else to pass or maybe not fit well, maybe, before but something to be fulfilled before executing. So, for example, Jenkins could potentially emit a test case run queued event when starting a build job where it knows that it has a test case or a test step later on in its build. Right? But it hasn't actually run the test. So, on this receiving side, at least you'd be knowing, okay, this might be coming up.

This is not a required event. So these are many of these events, and we'll get back to the heuristics about which events would you actually actually expect.

And you'll see that a trigger object is also commonly used here because you might wanna know how is this actually being triggered. Is it a manual trigger? Somebody might clicking run this test button, and you might, in the end, wanna ignore manually triggered tests. You maybe you're only interested in tests that are triggered by, a pipeline in Jenkins or some other mechanism.

So trying to give some opportunities on how to alternatives on how to man handle these events. TestCaseRun started, not surprisingly, is admitted when the actual test case starts. Once again, you'll know have to know which environment it's running in, which test case it's related to, which test run it might be related to, and what triggered it. And the queued event that I mentioned earlier is not mandatory. So many times, you'll just get this event emitted from your system, and you probably won't get a queued event.

And the only thing that's actually mandatory here is the ID and the environment. So all of these others are optional just to kind of give the receiver more context. Test case could definitely be interesting becasue yo might want to aggregate all test case runs for a specific test case over time. You wanna might wanna track how is this test case performed over time when it comes to status, path fail, etcetera, etcetera.

And not surprisingly, there's a finished event at the end, which then adds, once again, the ID so you can kind of go back to which test case run you're talking about, which environment, and then the outcome of that test if it pass, fail, cancel error. And we have a pull request now for adding a skip opportunity alternative here. As as always, small things can result in big debates, so it hasn't been approved yet. So you can go to the GitHub issue and and weigh in.

We've also added severity, and this is obviously, or not obviously, but this is a little bit of a stretch because severity is is subjective. You know, if a test fails in one context, it might be really, really bad, but if it fails so if your load test doesn't pass in production, that's really bad, but if it doesn't pass in testing, that's maybe not so bad because you know that's a constrained resource.

So severity is something that is not mandatory, and reason is just a string that, you know, tells you gives you more context on the receiving side. Once again, the only thing that's required here is the outcome, pass fail studio because that's at least the least thing we thought would be helpful.

TestSuiteRun

Now we're coming to test suites, and testSuiteRuns. Once again, not surprisingly, testSuiteRun models, the execution of a test suite, and it also has reference to the environment that it's running in. So once again, staging, production, whatever, and a reference to a test suite object that could be defined in an external system.

And here we have the similar, queued I'm not gonna read these tables to you or at you.

It's the same kind of queued started finished, events that we saw for, test cases, and they have similar properties as you would maybe expect.

testOutput

The last one is the testOutput subject that I mentioned earlier. And this is interesting because often a test emits an output, not cluster pass fail ratio. Right? So if you're running a Cypress test, you can get a video of the recording, or if you're running Postman, you can get a hard-to-read log output or whatever testing tool you might be using.

And it would be nice, obviously, to get a reference to that. And there's a test output published event. And note that a testCaseRun can publish multiple outputs. Right? So you could get both a log, a video, and a PDF containing some kind of report.

It's totally up to this testing system to decide what it can kind of produce. The things that are required here is the output type.

Obviously, somewhat subjective of what we decided on initially as, the valid, values here, and the format, which be would be a MIME type, so application PDF, etcetera. Nice to hear if you the sources you can see is required. So here, we thought it was mandatory to help the receiver of this event actually receive or actually retrieve the artifact, because if you know there's been a PDF produced or a video, you might wanna know where to actually get it, instead of, okay, there's because that's probably why why you're interested in it in the first place. So that's why, the sources is mandatory. The URI is not mandatory, which maybe it should have been, which is a more straightforward reference to the actual output that was published.

And a testCaseRun allowing you to associate the output with the actual execution of a test case. Please note that this is not mandatory, so the only thing you might be receiving on the end is testOutputs.

Not very helpful, but once again, depends on the implementation of testing events, that you're using.

Objects

Also define a couple of objects. I've I've mentioned testCase, testSuiteTrigger. I don't know if we need to go through these in detail.

TestCases is a test case, and a testSuite is a test suite, and a trigger is a trigger. For the triggers, maybe a little bit interesting to know if as I mentioned, triggers can be for queuing and starting test cases and test suites. So these can be manual, pipeline, and event, a schedule. So if you have a scheduled trigger, like, every hour or every whatever, and other, which always references.

And then there are URI reference. And this is, I think, for all these objects, there are URI references optional. So if the system that holds the test case or the test suite or the trigger can provide a URI where you can actually look at it in a web interface or an API call to retrieve the definition of that, object. That's what you use.

Testing Events Heuristics

Okay. Slightly more colorful and confusing slide.

So just trying to model all of these. So we had the test suite run subject with the predicates queued started finished.

The bolded, tags or prop fields are required. So as you can see, on the only thing that's really required here is where it's running and the ID of the test suite run and then the outcome when it's finished. Same thing for a test case run. The only thing that's really required is the ID and the environment and the outcome. And for the test output, what's required is the format.

And then the sequence or the heuristics of kind of if you're on the receiving side and you're gonna build something that reacts to these events, How you're gonna want to know, "Which events am I gonna get?" Right? And this hasn't really been defined in in the specification, and that's maybe a shortcoming, and maybe something we should at least attempt to write down. Because you you probably wanna know that if I have a test suite, is there always gonna be a started event?

Will I if there's a queued event, will there be a started event after that? Or what if it doesn't start? What if it just gets queued? Is it and it cancels?

Will there be a finished event with a specific outcome, etcetera? And I don't think we've really it's maybe something for us to we've talked about it a lot, but we haven't really formalized it. Specifically, also around if a test case, if there's a testSuiteRun, that test suite run can contain multiple test cases. And as I mentioned earlier, those test cases can run-in any order.

They can run-in parallel. They can run-in sequence. So the the order of those events will is totally undefined. Well, hopefully, it'll get the finished before they started, for a specific test and the queued before they started, although that's not guaranteed.

So just a a little bit of an attempt as heuristics here. A test suite is an orchestration of multiple test cases. A single testSuiteRun event can contain to I'm not gonna read this at you, but I think you can I'll get it.

What's maybe important then is that the queued events are optional.

The test outputs are optional. So I think the only thing today that you could expect is, started and finished events for test suites and test cases.

But, of course, it's up to the person or the project implementing these events to, implement them in in, you know, as much as they want.

How to Use Testing Events in CI/CD Pipelines

So how would you use these events in your CI/CD pipelines?

So a couple of use cases. Right? So one is notifications. Right? You wanna be notified in Slack if the test fails. It's kind of an obvious thing. If you want maybe you want to even you've had users who wanna automatically create Jira issues, for example, if a specific thing fails or if you have an incident management tool, you might want to get notifications into that.

The other is aggregated or centralized test result management. What if you could pull all these events coming from all your different testing tools, dump them into one system that calculates your creates reports and creates, you know, quality metrics and pass rates, pass rates, field ratios, etcetera. And you might have might have application life cycle management tools like, Keptn or or others who might be interested in listening to these events and then, deducing from them if they can promote releases from one place to another.

So there's bunch of different ways you can use these.

And just for some little bit more color, just to kind of visualize that. So today, you'd have CICD systems, testing tools, test orchestration frameworks, which would all kind of publish these testing events to a CD events broker, and then you could have notifications, reporting, application life cycle management, all all listening to those events and then acting accordingly. So, obviously, your test orchestration framework could be listening to other events and using that to run testing tools. You could have testers or DevOps people, you know, run your tests ad hoc, as I mentioned earlier, which is actually not that uncommon.

And then kind of weaving this all together, to hopefully kind of build out more dynamic pipelines as you deploy your applications.

Test Events Example

And I'm gonna do a little bit of an example because you may be curious about what do these events actually look like. Let's see if this works.

So, Testkube is an open source project, which is if we look at is one of these test orchestration frameworks.

I just have a simple curl test. It has, support for CD events that you can configure webhook. I'm using webhook.site here where I'm gonna receive these events. So I'm just gonna run start by running a simple test, which is a curl test. This is running locally on my machine, hopefully.

And as you can see how we first got a test case run started event with some, you know, URI, kinda which would actually take me to the dashboard in SKU to look at that test, and then we got a corresponding finished event. We didn't unfortunately get a test output event, which is in the backlog. I'm gonna nag the developers to do that. But I could, of course, I could go in here and then here see the log that was produced by curl, and this is ultimately what you would maybe want to have a, test output event for. So you could retrieve that log for curl.

And then correspondingly, Testkube has the concept of test suites. So this test suite here only contains that same curl test. But if I go back here and run that now, it's running locally.

You can see all these events coming in. So there's a test suite run event, and then there's a test case run, and then there's a test case run finished, and then a test suite run finished at the end. So all those four events that you would expect to come in sequence come here. So Testkube allows you to orchestrate tests both in parallel and in sequence. So if I had a much more elaborate test than this one, you would have gotten a much more elaborate sequence of events, that you could then react to on the other end, however you might want to if you had plugged this into some kind of CD events, bus or broker.

And that was really short.

So so I'm gonna stop there.

As always, these standards really depend on people in getting involved.

And it's easy for me from a Testkube project and and Andrea and others to kind of, you know, think of what people will need, based on what we've seen. But, ultimately, it's based on what you guys do and how you do testing and how you wanna make testing part of your pipelines that drives this. So please get over to GitHub and open an issue or just talk to me or Andrea or anyone else.

And I'm, of course, happy to discuss. I'll be here for a couple days, and I'll be at KubeCon too. And you can get these really nice plushies. They're, like, this big. That's like the most popular thing.

That's it. Any questions?

Nope. Okay. Thank you so much.