How 3Key Company Utilizes Testkube for Automated Microservice Testing in Kubernetes

Summary

In this case study interview, Roman Cinkais, the founder of 3Key Company and the CZERTAINLY platform, with over 15 years of experience in information security IT projects, discusses their use of Testkube to enhance automated testing framewo§rks within a Kubernetes environment. The CZERTAINLY platform, which focuses on public key infrastructure and encryption, sought an efficient automated testing solution for its microservices architecture deployed primarily in Kubernetes clusters. They discovered Testkube in mid-2023, which aligned well with their goal of high automation in testing. The integration of Testkube into their operations was primarily motivated by the need for a robust testing framework that could handle multiple microservices and provide end-to-end testing capabilities.

The technical infrastructure of CZERTAINLY includes several microservices running in a clustered Kubernetes environment, with automated CI/CD pipelines implemented through GitHub actions. The team, consisting of developers, platform engineers, testers, and consultants, is focused on improving and maintaining the platform. Before adopting Testkube, the testing process was limited to code reviews, unit tests, a small amount of API tests, and manual testing. The implementation of Testkube has allowed for more structured testing, enabling the team to automate tests, run them as needed, and access detailed reports on the results. This transition to Testkube has not only streamlined their testing process but also significantly enhanced their ability to maintain high quality and reliability in their software development lifecycle.

Introduction and Background

• Can you briefly introduce yourself and describe your role within your organization?
  

My name is Roman Cinkais, I am the founder of 3Key Company and the open-source platform CZERTAINLY. I have over 15 years of professional experience with a focus on information security IT projects across various industries (financial, retail, banking, telco, postal), including knowledge of cyber security and risk management framework. I am heavily focusing on the public key infrastructure and encryption related topics.

• How long have you been using Testkube? When and how did you first learn about it?

We were trying to design a good automated testing framework for the CZERTAINLY platform that contains several microservices and is deployed primarily in a Kubernetes cluster. We have our QA strategy defined and we were looking for a tool that can be a good fit for that purpose. One of our main goals is to highly automate. We have discovered (I think through Google)  about Testkube and Tracetest in the middle of 2023 and that it might be something we are looking for. We have started to work with Testkube, exploring how to use it and how to integrate it with our so-called automated testing framework.

Infrastructure and Team Composition

• Could you describe your technical infrastructure, focusing on elements like microservices, clusters, and CI/CD processes?

The CZERTAINLY platform is an open-source software that is deployed in the Kubernetes environment. We currently do not offer SaaS and therefore we would like to be sure that everything is working according to the expectations during the development and before the release of a new version. It has several microservices, typically running in clustered Kubernetes in production. During the development, we have automated CI/CD pipelines through the GitHub actions. Testing is implemented using these actions. For the microservice platform, we need something that would be able to test the functionality not only for each separate service, but also as a use case, end to end testing, and have the possibility to measure and report results. Therefore we have prepared a tool, utilizing Testkube, through which we can define what and how we would like to test and report. It is still under development, but it can be found here: https://github.com/CZERTAINLY/CZERTAINLY-Automated-Testing-Framework.

• Tell us about the composition and roles of your teams, including developers, DevOps, testers, SREs, QAs, and any other relevant roles.

We are a small team that is improving and maintaining the CZERTAINLY platform. We have 4 developers, 3 platform engineers, 1 part time tester, and the rest are consultants and PS that are helping with the implementation of the platform. We understand the importance of tests and therefore we are looking for a way to make it effective yet still be heavily focused to deliver new features.

Challenges and Solutions

• What specific challenges or problems led you to seek a solution like Testkube?

See above, lack of resources and ability to automate testing as much as we can. Also, testing of the microservice platform in the Kubernetes environment where each microservice can use a different set of tests. The QA strategy contains description of what we are trying to achieve: https://docs.czertainly.com/docs/qa-strategy/overview#automated-testing

• Before implementing Testkube, how did you manage your testing processes?

It was typically covered only by code review, unit tests, a small amount of API tests, and manual testing before the release. This requires resources we do not have and cannot currently afford, therefore we have invested our time to build the automated testing framework that can be easily extended and supports various testing tools and use cases.

Impact of Testkube

• In what ways has Testkube transformed your testing process? 

We are able to create tests for our use cases through the code, with various tools, run them when needed, and access reports with results. That gives us better visibility on what is the impact of introduced changes and timely notification if something has been broken or does not produce the expected results. Before Testkube, we did not have any systematic approach for executing tests.

• Can you share any specific results, metrics, or improvements observed since using Testkube?

Not yet, we need to improve the automated testing framework and include other tests. We are planning to use it for performance testing where I think we will find some interesting results, and we would like to automate some parts of security testing using ZAP. While we do not have metrics we can share, we have built a tool (in the GitHub repository), where we can automatically run our tests in an ephemeral way. Based on the definition, we can build the environment, run the tests, and notify reports. It is a complex testing of microservice based platforms in about 10 minutes, fully automated, fully managed in the code.

User Experience and Feedback

• How would you describe your overall experience with Testkube in terms of usability, efficiency, and support?

I have a good experience with the team that is behind the implementation. Initially we tried to use Testkube for something that was not yet implemented. I am talking about ephemeral environments, accessing testing reports while the agent is offline, and having the possibility to manage multiple agents for multiple testing environment. Our primary goal is to build an ephemeral environment using GitHub actions that is always destroyed after tests are done. Although we have the development and staging environment, this way of testing provides better overview and is running from scratch which eventually eliminates some changes that can be introduced unintentionally by developers and testers to development and staging environment. The support we have received was great and I enjoyed talking with the team, discussing the use case. It is very valuable for open-source platform like we are maintaining.

• What feedback have you received from your team about Testkube?

The feedback is positive, as we can focus more on the implementation of new features, keeping tests running and notifying us when something is not according to the expectation. We are still at the beginning, but having Testkube gives us the flexibility and automation we were looking for.

Comparison and Contrasts

• Have you used any other testing tools or solutions prior to Testkube? If so, how does Testkube compare to them?

Of course, we are using various tools for testing like Postman, Cypress, SOAPUI, Jmeter, however, we were looking for a better and automated way, how we can do this systematically and consistently in a Kubernetes environment. I think we cannot compare Testkube with these tools, as they have a different role. We did not use any similar tool like Testkube before.

“We are able to create tests for our use cases through the code, with various tools, run them when needed, and access reports with results. That gives us better visibility on what is the impact of introduced changes and timely notification if something has been broken.“

Future Outlook and Suggestions

• Looking forward, how do you see Testkube fitting into your future projects or workflows?

 For the open-source project it is important to have access to automation. Spending resources effectively will lead to a better adoption and higher community. Testkube fits very well into our goals. We are also experimenting with the OpenTelemetry and in combination with Tracetest, it can be a very strong help to keep such platforms as CZERTAINLY in good shape. I am also very happy we have learned about Testkube, as from the consulting and implementation perspective, we can help our customers adopt Testkube for effective testing of services in Kubernetes and automate testing.

• Do you have any suggestions or features you'd like to see in future updates of Testkube?

We have talked about having the possibility to “attach” different agents to different testing environments, which can give us flexibility in choosing where we would like to run the tests. But in general I think we can use it for our purposes and do not have any specific suggestions right now.

Overall Impact and Marketing

• In your opinion, what is the most valuable aspect of Testkube for your organization?

Having an open automated testing framework where anyone can contribute and write tests. Having the possibility to utilize different tools for testing and running it in an ephemeral environment. This not only promotes the quality of the open-source CZERTAINLY platform, but makes it more robust. Anyone can write tests that are executed independently.

‍