Scaling Testing for the Age of AI-Generated Code

Ole Lensmar: Hello and welcome to another episode of the Cloud Native Testing Podcast. I am super thrilled for the first time in the podcast history to be joined by two fantastic guests. I have with me Dimitri, co-founder of TestCube, who is going to share immense wisdom, but I have someone even more prominent, Atul, who we've been working with for a long time. Atul, it's great to have you here. How are you?

Atul: youThanks a lot, Oleh. Nice to see you, Dimitri, here. And happy to be here. Things have been going pretty well. I was occupied with KubeCon India last week, but happy to be discussing interesting things today.

Ole Lensmar: Yeah, it's great to have you. And actually, when we prepared for this, we talked a little bit about your background. And I was amazed to hear that you started off in manual QA, like the unsung heroes of testing. So I'd love to hear, because it's a great story. And you've come so far in doing amazing things right now in the cloud native community. But let's go back a little bit to your testing roots. So how did that start off?

Atul: Sure, think it's been quite an amazing journey. It's been a roller coaster ride. I was fresh out of college about just over a decade ago and I joined one of the largest service companies in India and was part of a manual QA team. And back then I was like, what will I do being a manual QA? I that was the thought that I had when I was getting into the project. And interestingly, that project was around virtual desktop, so it was around virtualization already. I was already in touch with a little bit of cloud virtualization, if I can say so. The solution was more around creating virtual desktops, VDIs for end users. As part of my job, I was handed over an Excel sheet and it had about 50 to 100 test cases. I just had to go to the portal and create a new VDI and then go into the backend and check whether the storage was provisioned correctly or not, and whether the right CPU was given, so on and so forth. So just kept on marking pass and fail in those Excel sheets for a few months. And I think one of the most interesting things that happened early on in that state was it was time for a major release from the team. And I remember failing a test case because I found something to not be right.

Ole Lensmar: We.

Atul: And that was my first experience where all the architects and developers came at one side and I had my testing manager and myself. You know, the typical it's a feature, not a bug sort of a debate. And we spent a few hours discussing, debating on it. And eventually the release didn't go through and it didn't go through for good reasons because there apparently was an issue with how the storage was functioning. But then I spent a few months doing that. But always I was interested in

Ole Lensmar: Thanks.

Atul: working with code, you know, always wanted to write code in some way or the other and eventually stumbled upon automation. And now I was actually writing scripts to do the same thing, which I was doing manually. So I did get to write code now and you know, that was proprietary on C sharp. So I moved from manual testing into C sharp where now I was writing scripts, which were, basically recreating my actions. So clicking on buttons, clicking on icons, validating what's happening.And then that led me to a different project where I was now building Android applications using C sharp. I did that for a couple of years, working with GPS functionality, as well as a lot of locking and locking mechanisms on Android. And after that, I transitioned into my current role as a developer advocate. And this role has been completely different from what I've been doing for the last decade because now I'm focused more on cloud native tech and especially from not knowing what the entire landscape was, what Kubernetes was, what Docker was, getting into this for the last about four years, it's been interesting. And I think how the whole testing or QA phase helped me through this entire journey was instead of just figuring out how to break things, I was now able to understand

Ole Lensmar: ThankThankThank

Atul: systems better because now I'm at the other end of building systems. So earlier, you know, when I was just fresh out of college, the whole idea was, hey, let's just break this. Let's just go and see what happens. Usually getting into the edge cases. But over time, this entire shift from manual to automation to a developer, a developer and now a developer advocacy. I think I've got a better sense of how systems function. What are the different points of failures? And I think

Ole Lensmar: ThankThank

Atul: Anyone who has been into QA initially, who gets into development or similar roles would naturally have a better sense of how systems function. So I feel I have that edge among the folks I meet.

Ole Lensmar: Yeah, I'm sure you can make an argument that all software engineers should start with testing or at least do testing in some part of their career just to be on the other side. But before we dive deeper, because I'd love to learn more about kind of...that journey on moving to more cloud native applications. Just I know that in the testing community, the practice of manual testing or exploratory testing is still a very cornerstone of QA. How do you feel about that? And how do you see the move to automation balancing still the need for manual testing or maybe exploratory testing? Maybe those are different things. What are your thoughts?

Atul: I think automation definitely brings in lot of velocity that is required. A lot of heavy lifting could be done using automation. There is no doubt about it. But then I think having the human in the loop or the exploratory part, like we discussed, is critical because that domain understanding is something that only you, me or anyone who is building the system or part of the team would be able to understand better. I could write a lot of scripts, especially in this era of wipe coding, I could have AI generate test scripts for me, but it would still not have a complete context of what application needs to be tested or what infrastructure is on which infrastructure the application is deployed. And I think that's where it's very critical for all of us to be a part of the system because we do understand areas or places where AI or automation could completely fail and not give us the right results. So we need a good balance of it, but we definitely need people within the entire automation system.

Ole Lensmar: And do you think, I mean, I don't want to jump to AI too quickly here, but it does seem like an obvious or like maybe a natural next step here is to have AI agents like test your system, not based on a script, but more based on an understanding of requirements, etc. and more dynamically performing actions. you, have you seen any like ventures in that direction? Does that make sense to you? Or do you? How do you think AI fits in into testing?

Atul: think right now you see a lot of things are happening mainly on generating code and maybe reviewing PRs. These are the two major areas that I've seen a lot of tools, a lot of things happening here. So a lot of developers are using AI to write codes, write scripts, maybe even review code, review scripts, things like that. But when it comes to actually using AI for testing, I haven't seen a lot of use cases, you know, outside of especially what we have done with TestCube earlier, right? The few demos that we built. I think outside of that, I have not seen a lot of work that is happening in that area. And I think it's something that a lot of people need to still explore and understand what are the benefits that AI could bring in, especially into testing because, you know, A, it has a complete understanding of your code base now.

Ole Lensmar: me.

Atul: Obviously domain is the second part, but it understands your code. It understands how the request would flow through different modules and different layers that you have. And with that knowledge in place, it can definitely have a good way to cover more functions, more utility. So basically when you talk about test coverage, using AI will definitely help you cover a lot more surface when it comes to testing applications.

Ole Lensmar: Specifically on that, do you think that as teams like Embrace AI to generate more code and configuration, do you think that also puts pressure on doing more testing or more test automation to make sure that, I'm sure that like generated code is reviewed by... experts, like any code, things can fall between the cracks. maybe especially this early in AI, people are hoping for a big velocity improvements and generating a lot of code to do things. How do you feel like there's a need for testing to be more scaled up as code is more generated with AI?

Atul: Interesting because just a few weeks back I wrote a post on LinkedIn and that post was around this very discussion that we are having on, you know, a lot of developers using AI to generate code. imagine earlier, let's say, let's consider the time when AI wasn't there, right? We would have five developers writing code, raising five PRs. You had your pipeline in place, which was able to handle all of it. But now with the advent of AI agents who are writing code for us, you suddenly see that the entire thing has gone 10x. So previously, five developers raised five PRs in a day. Now you have 50 or 100 PRs coming in a day. So at that velocity when the code is being generated, our CI CD pipelines kind of are the bottlenecks here because they are not yet scaled to handle the velocity at which things are thrown at it. They are still stuck with the five PRs, right? So Definitely scaling is required, not only focused on testing, but even your entire CI pipeline that you have in place because a lot more code is being generated at much faster pace, which are existing pipelines were not even built to handle. So we definitely need to address that bottleneck first. And then obviously, because once we handle that part of it that now we have scaled as CIs, so it can handle a lot more code coming in. testing can come in and then that also needs to be scaled accordingly so that it is able to handle the number of changes that are coming in basically addressing the velocity.

Ole Lensmar: Yes, to me it sounds like it's multiple dimensions. One is just being able to run tests more efficiently because you have more PRs and pull requests and changes coming in. But the other is also on coverage, like you were saying. How do we make sure all this code that is being generated is sufficiently tested? So the existing tests, we can run them a lot. But if they don't really cover the code, the stuff that's been generated. obviously things will still fall through the cracks there. So I think that would be interesting. You could, of course, I'm sure use AI to generate tests as well. But maybe there's a self-bias there. How much do trust the tests generated by AI? So it will be really interesting to see how this plays out. think we're still very early and people are obviously very enthusiastic about potential velocity gains. But it will be interesting to see as that settles maybe a little harder how that does change pipelines and velocity over time.

Atul: Yes, in fact, just a few weeks back around at CubeCon, I had some discussions with different set of people around, you know, why coding and AI generating code and the whole security aspect of it as to, you know, how do you actually trust the code that AI is generating? You know, how do you validate, you know, let's say you ask it to generate a script. It did. But then what's the image that it is using within the script that it created, right? From where is that image coming in or how is it generating the code? So a lot of software supply chain concepts coming in here because that also will become critical because earlier AI was just helping you probably, you it is giving you some code that you're using. But how I see maybe a few years or maybe a few months down the line, it would become an integral part of your entire software development life cycle. while you were and you know, both you and me are there writing code, now you have an AI agent sitting there and generating code. So The whole security aspect becomes equally more critical here as well.

Ole Lensmar: And do you, going back a little bit to like you with cloud native and Kubernetes, so you've been, you know, going deeper and deeper into that community and. I guess one of my reflections has been over the last year is that testing is still not very prominent. If you look at like the CNCF landscape and if you go around the booths at KubeCon or the talks, testing is rarely like a headline topic, right? There's maybe one or two topics and they're usually very low level. Do you have any thoughts on that and what could be the reason of that?

Atul: I think I might not have the exact answer to this, but then I completely agree to what you said. have been to a lot of cube cons and a lot more emphasis has been on observability. know, Bazanin, you'll see a lot of boots, a lot of tools around observability and now more of AI agents are sitting on top of observability solutions that can help you figure out what's going wrong. But I think A lot of times what's happening, especially when it comes to testing is traditionally applications. mean traditionally developers focused mainly on application functionality, which meant just your application, your database, maybe a couple of services here and there. Now with Kubernetes and cloud native coming into place, there are lot of different pieces that are working altogether right now. So you have your containers, your applications you need to. test how orchestration is happening, how networking is happening, policies, service mesh, a lot of different things that are happening. So ⁓ I feel they are blinded by a lot of things so that the focus remains only on their application. And this unfortunately takes a backstreet. But then eventually it actually comes to forefront when things break down. That's when they realize the value of testing everything that is possible, not just load testing your application, not just checking the functionality or just

Ole Lensmar: One, two, three.

Atul: checking the checkout functionality of your e-commerce application. You need to check how the network is being handled by your service mesh or you need to understand how your containers are being provisioned on the infrastructure. I feel there is bit of blindness, I would say. I don't know if that's the right word. But then a lot of focus is right on the applications, whether they function well or not. While the whole ecosystem is still a little bit untouched.

Ole Lensmar: you

Atul: how I've seen it. And I think that's where, you know, we need to talk more about why this is required. know, why testing has to be a critical part since your infrastructure has completely changed. You have a lot more moving parts over here. And everyone talking about shifting left, you know, everyone talks about platform engineering. You you want to reduce cost of fixing bugs at the end, bring it to the front. I think a lot more awareness, a lot more

Ole Lensmar: Thankit.

Atul: things need to be done to talk about the effects of not testing things at an earlier stage or not doing a comprehensive testing of your application as well as the infra. And I think once we get that piece out there, I'm sure this is going to pick up amongst the overall community as well.

Ole Lensmar: I mean, I can definitely see the argument that... Kubernetes and Cloud Native is more about operations and infrastructure and testing is maybe more in the application domain, like you were saying. And you'd say, so if you want to talk about testing, well, then you go to a developer conference or a different part of the lifecycle. But at least from what we can see, and I think what you're also alluding to, there is a big need, especially as infrastructure is code. part of Cloud Native to test infrastructure. Infrastructure testing is a pretty big thing from what we can see. It's security, obviously, and observability also kind of leads to testing or troubleshooting in quality. it feels like, well, I think... I'd be hoping for too much to say that there's like half of the booths at KubeCon about testing. That's obviously not gonna happen, but it feels like it could be something that is more top of mind and talked about and kind of in this context. I mean, there are several... Areas of course, where it is interesting, I think you look at ephemeral environment provisioning, which is something that's usually used for testing. So that's something where it usually comes up. So I'm not saying it's not there at all. just got, obviously, know, we at Tuskube would love to talk more about testing. So some self bias here, of course, as well.

Atul: No, I completely you know, echo that. In fact, again, as part of CubeCons, I've had discussions with folks who are in the QA and testing side of things. And they are still, you know, not not not leveraging Kubernetes or cloud native tech for testing, which which you know, with test cube, or all of us have have seen or, you know, are actually using it. They're still stuck in the traditional way of using it. So whether it's load testing, whether it's API testing, they're still stuck in the past and they are not leveraging Kubernetes itself. that's when I just told them that, why don't you just go back and see if Kubernetes can actually help you to improve your testing itself as an infrastructure or as an orchestrator in this case. I think that there is a fair chunk of people or organizations who are still not aware of the benefits that they can. get from Kubernetes or Cloud Native when it comes to testing, running that testing info over here. I think that is one area which I would probably want to discuss with a lot more folks I meet from testing because I've seen a lot of benefits myself, even whether that was working with other clients or just experimenting things. So I would definitely would discuss more as and when I get more opportunity to talk, raise more awareness that how this can be helpful as your infrastructure where you can run your testing on top of it.

Ole Lensmar: Yeah, I mean, I think to your point, it'll take time. think also as people are moving maybe more to a cloud native wave delivering software. So it's not just a monolithic CI CD pipeline, but it's more of a event driven pipeline where you have one CI tool, another CD tool, you're doing get ops and you're doing progressive delivery. And then how do you inject testing into all of those? I think that's, I think as people kind of go down that path, they also kind of realize, or maybe they see the need to run tests, not just as part of, you know, from a Jenkins build, for example. So I think definitely it's something that's, it's coming. But it's interesting to follow that journey from our end, course, because that's kind of what we're trying to facilitate with, with TestCube. I think one thing I wanted to ask you about is one thing that has come up in previous podcast episodes and also when I talk to people is about testing in production. And it's one of those things that, you people say you should never test in production, but almost everyone we talk to say, well, of course, well, yes, we test in production. I think we haven't had someone say if you it's testing in production is the only way to test. I'm exaggerating, but if you know, if that's kind of the that's the place that you have to know that it has to work. there could be barriers to that. Do you have any thoughts on that or any experience from like testing and production pros and cons and when

Atul: I don't have any direct experience being a part of this, but well, know, I've been interacting with teams here at InfraCloud or elsewhere as well. And I think a lot of them emphasize on this. And one of the major reasons is being the difference within the environment itself. A lot of times you set up your testing environment, which doesn't mirror the production. And I think that that is a major concern amongst a lot of application teams across the globe. Because eventually when you have all your integrations and systems and unit tests and everything working perfectly in test environment but it just goes off when it comes to production and that is where I think a lot of emphasis is given to testing in production and It may not be like a hundred percent test there but then obviously something like an AB test or Maybe you know when you speak about something like a progressive delivery, right? You are not testing the entire load or the entire stuff that's happening in production, but then testing critical areas that you feel, you know, that can that can probably go wrong. So you try to use the signals, you try to use the observability metrics that comes in from the production and use that to refine your test, refine your application and use some chunk on your production to test how things are happening and then use that as a feedback mechanism to give it back to your team to understand that, these are the things that eventually create a bottleneck in this type of a situation. you know, even a percentage of information coming in from the production is gold or it's very much valuable for the team when they're actually building the application in the development stage. So I've had few discussions and, you know, everywhere teams have been moving to the phase where they want to test into production. But obviously, all of them are wary about not breaking things. And that's where they are getting into the whole mirroring the production. So that's where your ephemeral environments come in, for example, which recreates the exact production setup. You try to do a lot of chaos testing to understand how the actual real-time effects can happen, make or break your system. So yeah, that's what I've heard from my discussion with people.

Ole Lensmar: It make sense. mean, it always did. matters in the type of tests. I'm guessing like penetration testing is maybe something you want to do in production because that's where you want to make sure that you don't have any, you know, security vulnerabilities. But like massive end-to-end tests, maybe it depends on kind of like how they alter the state of your application. Chaos testing, I know some, you know, some companies do in production because that's kind of where it has to work. So it's definitely, I think, I think when you say testing, most people think functional testing, but I think there's so many different types of testing so definitely that can be more nuanced as almost anything else. Great, Atul it's been a pleasure having you. Thank you so much. Dibicchi, do you have any last minute questions for...

Atul: Absolutely.

Ole Lensmar: Well, how about this one? A tool, if you had a magic wand, and well, the constraint is it has to be about testing, and you could wave it and solve one problem for people. What would it be?

Atul: Wow. This will take, this will take, I need a few seconds to think about.

Ole Lensmar: That's fine. I'll come back, we'll put it in the comments.

Atul: Then I'll, I'll, I think magic wand for, I think, so focusing on the cloud native space we are in, right, with so many pieces, multiple pieces, we being part of the Infraender application, I think if that one can give me a complete list of various integration points that are available, what are the different modules that are involved, then it would give me a better visibility into the entire application and the info that I have. And that's when I know what are the places where it can fail. you know, write everything from my source repo to the voids where I'm storing my secrets to service meshes and everything else that's part of it. Because I know when you have an application, I might miss out on certain things like, you the repo from where it's coming in, right? Or maybe the void that's part of it. Maybe I'm just focusing more on the service mesh or how it's functioning or more on how the containers are being, you know, orchestrated. So while my focus is there, I would definitely want want some command where I just run it and it tells me that hey these are the hundred points of failure that can happen and I just take that and write my test according to it. So that would be where I would want to use the logic code.

Ole Lensmar: Yeah.Okay, great. I love that answer. Thank you so much, Atul. Thanks for joining. It's been great talking to you as always. Thank you, Demici, for that very interesting question. Thank you. Thanks for everyone listening and looking forward to talking again soon. Bye-bye. Bye.

Atul: Thank you.Bye bye.