If you're working with Kubernetes applications - making sure that they are fast, reliable, and stable is of the utmost importance. To guarantee your users have a great experience with your software, you'll want to be able to spot bottlenecks in your APIs, maximize resource usage efficiency, and setting appropriate Kubernetes resource limits. How can we do that? Let's take a look at running Performance Tests.
Performance Testing is a popular testing technique that helps you check your application’s speed, scalability, and stability - but how and why should we be performing these types of tests? And what tools can we use to really take advantage of this practice? More importantly - what does this mean for our infrastructure and toolkit when working with Kubernetes applications?
In this livestream, we’ll cover:
Who should attend?
Whether you’re a tester, developer, SRE, or anything in-between - this event is perfect for those who wish to dive deeper or are curious about the world of testing applications in Kubernetes.
Hi everyone, we're live! Welcome to Testkube's office hours. This is a virtual space where we open our office doors to you to talk about Testkube, Kubernetes, and testing in general. Today, I am joined by our amazing QA Lead, Tomasz. Tomasz, say hi, introduce yourself, and tell us a little bit about yourself.
Sure! So, I'm a QA Lead at Testkube, and I'm responsible for QA-directed things in general. Basically, my role is making sure our users get what they expect and that everything works correctly. I design testing processes and introduce different testing tools just to make sure everything functions right. This way, we get feedback as soon as possible, so we are not only guessing that it should work, but we actually know from our configuration that everything is fine constantly.
Perfect, yeah, you're our savior. Thank you so much for joining! Today we—oh, by the way, my name is Ali, I didn't introduce myself. I'm Ali, a Developer Advocate at Testkube. I currently focus on hosting these office hours.
Yeah, we're ready to talk about really important things regarding testing. A few releases back for Testkube, specifically in version 1.7, we released a really important executor: the JMeter executor. JMeter is a tool used for load testing, and it's pretty much used for analyzing and monitoring web applications. But we wanted to dive deeper into how to use this when Kubernetes is involved, what happens, and how to do it easier with Testkube, obviously. We have a lot of questions that we get asked frequently by our users and our community, so we wanted to talk about it in general and discuss the advantages of using JMeter, especially with Testkube.
Tomasz, maybe you can tell us a little bit about what load testing and performance testing actually are?
Okay, so I will start with performance tests first, because that's the biggest group and the overarching kind of test.
Basically, the goal of performance tests is to understand how the product behaves in terms of performance. For example, what are its limits? Will it handle a specific required load? Will it auto-scale if we get a huge amount of users suddenly? Or, what is the point where it will break? That's something that we also need to check.
Performance tests can be divided into different categories, like load tests, for example. The goal of a load test is to identify areas for optimization, like possible bottlenecks that we need to spot and probably fix.
On the other side, stress tests are just trying to "kill" the application. They try to find the specific load where it will break and when it won't be able to work correctly anymore. It's not about randomly trying to just kill the application; it's more about analyzing everything, applying a really detailed test plan for performance tests, executing it, and taking a lot of notes to analyze every single aspect that may be important. Then we need to analyze these results, which will allow us to understand how the application behaves in general in terms of performance, what the limits are, and basically, if we can depend on it to meet our expectations. It's basically trying to build confidence in the product in terms of performance as a whole.
Amazing, a super powerful tool. So, a tool that does this is JMeter. Can you tell us a little bit about what JMeter is?
Yeah, so JMeter is an open-source testing tool that is among the most popular ones. It's definitely a product that is really mature. Because of that, sometimes it may seem like it's too powerful because there are so many options that can be used there, which may be frightening at the beginning. But in the end, it's a really powerful tool that can be used for almost everything in terms of performance or load tests, to be more detailed. And that's one of our latest executors that we support, meaning Testkube now supports running JMeter tests easily.
So, more specifically, what kind of tests can we run with JMeter? Is it strictly for Java-based applications, or what's the gist with JMeter tests?
Sure, so it can be used to test almost any web application—that's the goal. I mean, the limitations are so high that in most cases, you won't be limited by the tool at all. It's more like performance tests are usually limited by your ability to run them, which is one thing, and the second one—which is sometimes even more difficult—is how to design the performance tests properly so they will answer specific questions.
It's not just that we want to hit the application with some random load or blindly increase it. Every time we create performance tests, we need to design them so they will answer very specific questions. For example, if we have some expected user load that we need to meet, we need to make sure that the application will handle it. Say we have an important event coming up where we expect a huge, massive load on a specific date; we can prepare for it, simulate it, and confirm that it will actually work. That's what JMeter can be used for: to just answer very specific questions that we want to find answers for.
Awesome. Speaking of challenges, what about when you're running JMeter but Kubernetes is involved? For example, from a developer perspective, if someone is not familiar with Kubernetes, they may get really confused as to how to navigate the whole process, where to find the logs, how to navigate the pods, or maybe even get access to them. What do you think are the more specific challenges that you can identify from a developer perspective, but most importantly, from a QA perspective, which is your perspective?
Sure. For me, Kubernetes is a really powerful tool, but it's also quite difficult to learn. I mean, the learning curve is pretty steep there. When you want to run a JMeter test with Kubernetes on your own, you need to configure everything manually and prepare everything yourself to achieve what you want.
Testkube just makes it simple because you can just create the test directly. Testkube provides you with a nice abstraction over Kubernetes, like our Test Custom Resource (CRD), which makes it simpler to define it. Alternatively, you can just use our dashboard and create everything with a graphical user interface in a very convenient way.
Yeah, way, way easier! We actually will show you in a bit how it actually works. But can you dive more into the specifics of how the JMeter integration works and what's happening under the hood? You mentioned a little bit about how we have it as a custom resource, but can you tell us a little bit more about that?
Yeah, sure. When you have your testing project, all you need to do is just save the project to a .jmx file, which is the standard format for JMeter. Once you already have this file—meaning you created your own performance test in JMeter—that's all you need to run it with Testkube. You just import this file, and we'll pre-configure everything that's needed. We'll create the Kubernetes job and run the test, and you will get very detailed logs directly from JMeter.
If you need something more customized, you can still introduce customizations over it. You can customize things like the pod size or the hardware resources used for running that specific test. All of these things that are possible with Kubernetes can be done with Testkube much easier and in a really convenient way.
That's the biggest benefit for me as a QA because I have various types of tests that I need to run every single time. One of our goals is to get quality feedback fast and constantly, so I will always know the status of the application, whether it works correctly, or if there are any problems that need to be fixed. With that many different test types, it's a great thing that I can integrate all of them with Testkube.
No matter if it's a Cypress test, a Maven test, or a JMeter test (and I'll show some examples today), I can run all of these tests in exactly the same way. That's a huge benefit because I don't really have to care what the infrastructure beneath it is; it just works correctly. It's really important for me because I don't need to think about the maintenance, which can become very complex.
Another interesting thing is that we use Testkube for actually testing Testkube itself. This whole self-testing concept is very interesting here. The great thing is that we have every single one of our tests defined with a Test CRD, which is the custom Kubernetes resource that allows us to describe the test at a high level. That's the abstraction Testkube provides. We have all of these different tests—like Cypress, k6, and any other executor that we support—stored in our repository. Then, you can use Testkube for actually running them, getting the results, and analyzing logs. So, we are basically the first users of Testkube (meaning the development team), and that's a great situation because we can notice things easier and faster than if it were just left to the end users.
And that's how you know it works and that it's good, because our own engineers are using it on a daily basis. Well, maybe we can jump into some examples. Ready?
Yeah, so I will just share my screen.
Okay, so you should now see my screen. Here is an example of a JMeter project I created; it's really simple. There is just an HTTP request in this case to our Testkube website, and then we have a response assertion checking if the website returned a 200 response. Then we have an assertion listener to get the results of this assertion and display it.
To run the project—and in this case, it's really simplified on purpose to just be a clear example—even if you have a really complex JMeter project, all you need to do is just save the test plan to get this .jmx file. I already saved this one as jmeter-executor-smoke, which is one of our smoke tests that we use for testing Testkube.
Running it is pretty straightforward. I already have the dashboard running in one of my local environments, at least. If you want to create the test for this JMeter project and you have this .jmx file ready, all you need to do is click "Add a new test" and fill out the form. Let's name it jmeter-office-hours-test. We can optionally add a label so we'll be able to use filters and gather results as a whole. For example, we have a label for our core tests like executors, and then we can just filter the dashboard using it, but we won't look at that right now.
Let's just set the type: there is a type named jmeter/test, and that will run the test using the JMeter executor. Then we need to define the test source—where we get the test from. We can select "File" and just attach the file that we just created and exported from JMeter. Let's use this option. You just get the standard file dialog, so let's navigate to the desktop directory I have locally. We have different test types, and there is the JMeter folder with the exact file I just saved and exported from JMeter. We just hit open, click "Create," and the test is created.
Now you can run it. Let's just click "Run test" now, and the execution has just started. You can see that our pods for the JMeter executor started running. You can display the execution logs here, and as you can see, the result is green—the test has just passed! There is just a simple assertion, so the returned 200 response was correct, and we just executed our first JMeter test.
You can check the CLI tab to get the CLI commands if you prefer this way of using Testkube. That's another advantage of Testkube for me as a QA: I can work in the way I prefer. I can use the dashboard with a graphical user interface, or I can use the CLI and just use the console. It's sometimes faster for me when I already know the specific parameters or commands. On top of that, we have CRDs (Custom Resource Definitions).
For example, if we want to get the result of this test in the CLI, we can just copy the get execution command that we have here, and it will allow us to get the result in the CLI. This shows exactly the same test result that we just saw in the dashboard, but this time in the CLI.
We can also create the test using the CLI, so I'll show you this one as well because it's interesting. This time, we will use a Git repository instead, so it won't be exactly the same. We'll use the exact same file because we have it in the repository, but we'll create the test in a different way. All you need to do is use the Testkube command. You can use --help to check what's possible; for example, we have the create resource command. In this case, we'll use this one to create the test: Testkube create test.
Then you need to define the name, like jmeter-office-hours-two, which will be our second test created here, this time using Git as a test source. So, we have the test name; now we need to set the test type, which is the --type parameter. Or maybe I will show you another help menu for creating tests first, so we can display the help for any of the commands available here. You can get a list of possible parameters. Obviously, you don't need to set all of them; usually, setting just three or four of these would be sufficient, but there is quite a lot of configuration options available so it will match your use case in almost every scenario that may exist.
But let's go back to creating this test. We already have the test name; let's extend it with the test type, so in this case, that's jmeter/test. Then we need to set the test source or the content type, which is the --content-type parameter. In this case, it will be git-file because that's just a single file in the repository. We do this so it doesn't check out the whole repository, making it faster; it will check out only that single file.
Then we need to set the repository URI using the parameter named --git-uri. I will just copy this one. In this case, we use GitHub, so you could just copy this directly from GitHub. Then we can set the Git branch; it's optional, but let's set the branch name in this case. And finally, the last one is --git-path, which is the path to this .jmx file in our repository. In this case, we have all the tests in a specific directory, which is test/jmeter/executor-test.
(To Ali) Do you think you could zoom in a little bit, Tomasz? Just a little bit.
(To Ali) Yeah, yeah, that's better. It should be better now just so that we can see.
Yeah, way better. Okay, yeah, so that's the directory with executor tests where we have all of the tests that are used for making sure our JMeter executor works as expected. We have the smoke test that we just ran, we have some tests for features, and we even have tests that should fail by design so we can make sure everything works as expected before a release. That's also how we use Testkube for testing itself, which is really powerful. If we make sure the results are correct first on tests running with Testkube, then we can use Testkube for almost everything else once we can trust it's working correctly, and that's extremely powerful.
So, we have this executor test directory and then the name of our test file, which is jmeter-executor-smoke.jmx. Let's create it... Ah, I just made some mistake here, some incorrect parameter probably. Oh, I think you wrote get-path instead of git-path. Yeah, let me check what was wrong. Okay, so now it is created correctly, and we can now run this test. Let's try it with the CLI again.
There is the run test command; you can also display the help here and you just set the resource name here. So, we can run the test and use the name we just created: jmeter-office-hours-two. The test is being executed. You can use the --watch command to get the results immediately and track the whole execution.
Ah, there was probably another typo... Let me copy the comment that I'm sure works because I probably made a typo here. Okay, so this test already exists. Now this one is created, and let's run it. Or we can use the dashboard to check that it is created and executing. Yeah, everything looks fine now! I just made some typo earlier; that's what usually happens during live demos. I know it works, but then I just made a typo because I wanted to type everything manually instead of using the autocomplete that I usually use. But now you can see it works correctly.
We can also check the test execution now—it's successful. There is additionally a JMeter report artifact. You can use the execution data, analyze it, or import it into other tools just for some additional processing. We just ran a JMeter test really easily with Testkube instead of configuring everything manually.
Exactly one more feature I need to mention: when you create a test with the CLI, you can additionally use the --crd-only option. This way, Testkube won't directly create the test in the cluster, but it will generate the YAML CRD that can be used for creating this test using kubectl apply. That's how we handle our tests internally. Basically, we write or generate the CRD, which is the Kubernetes abstraction for the test where we can define everything related to our tests, and then we can use kubectl apply to apply it to the source. It really simplifies test maintenance because we don't need to edit everything manually. If we want to change something, we can just go through all of the files easily, replace whatever we need, and just run it. It really makes things simple, and with the --crd-only argument, it's pretty straightforward to generate these files.
Amazing, thank you so much! Thank you for that amazing demo.
We wanted to—oh, we have a question from a user, Jonathan. He asks: "Besides JMeter, can you recommend a load or performance testing tool that also works with Testkube?"
Yeah, so currently we support JMeter, k6, and Artillery as load testing tools, and I can recommend any of these. I mean, all of them are really powerful. In the past, I preferred JMeter because it was possible to do absolutely everything there, but in real life now, I prefer to use k6 because it's just simpler to set up. It's really simple to implement some initial tests and then extend them, and you don't need to configure everything in a graphical user interface; you can just write your tests in JavaScript. It can be pretty powerful because you can use external libraries and create exactly what you need in a way that feels more development-friendly.
And speaking of k6, we have our office hours covering k6 coming up next week, so be sure to get notified for that and don't miss it!
But going back to JMeter, let's summarize to wrap things up. What would you say are the main advantages that you have as a QA using Testkube to run JMeter? What do you think are your main advantages?
Okay, so maybe I will speak about non-functional testing in general. In most cases in the past when I created a test, I had to configure everything separately—like setting up functional tests first—and then it was difficult to introduce proper non-functional tests like performance tests. This is because they always require different setups and different tools, and these environments can be tricky not only to set up but also to maintain over time.
That's what Testkube makes simpler for me: being able to configure every single test in exactly the same way, whether you're using CRDs or using the dashboard, no matter which underlying tool is used. I can then customize it when I need to. For example, in the case of a JMeter test, the first thing I would need to do is customize the hardware resources that will be used for running that test. With Testkube, that's much simpler because we can overwrite the default job template that we prepared. We provide a baseline that's suitable for most standard cases, but then you can easily customize it using the job template parameter in the CLI or directly in the CRDs.
Through this, you can overwrite any option related to the test execution job. For example, if you want to limit the CPU or you want to guarantee that the execution pod will get a specific allocation of resources that will be enough for running a really complex test in JMeter, you can customize it easily.
So for me, it's a combination of two things: being able to run tests easily in exactly the same way no matter the tool, and also having the power to deeply customize things whenever I need to. The combination of these two things makes Testkube the perfect tool for QAs because I can do everything in the same way instead of just focusing on infrastructure maintenance. Test maintenance is already difficult enough on its own just ensuring they are always up to date, so being able to handle the infrastructure in an easy way is a huge benefit.
Yeah, I do agree. I do agree that even as a developer, that's a huge advantage—just having to focus straight on what you need to do, which is testing. So thank you, thank you so much, Tomasz!
Yeah, thank you so much, that was an amazing demo. Thank you all for watching! If you want to read more, Tomasz is releasing an article pretty soon about JMeter, talking a little bit more in-depth about everything we discussed in this video.
Please make sure to check out testkube.io. You can check out all our docs there, find all our information on how to use JMeter and other executors. We're also very open to contributions—we're open source—and we look forward to having your feedback. So, make sure to reach out, join our Discord of course, and check us out on GitHub.
Any closing thoughts, Tomasz?
Yeah, so I think the best thing is to just try Testkube yourself, and then you will realize what you can do with it. I would recommend you to just try things out because that's probably the most exciting part, at least for me as a QA: trying new things, experimenting, trying to understand how it works, what I can do with it, and basically seeing how it can improve my everyday work.
Perfect! We have an amazing interactive demo up at demo.testkube.io where you can just play around with it. And yeah, we'll see you in the next office hours. Thank you so much for watching, and yeah, have a great day! Thank you, bye!