Peter: Good morning, good evening, good afternoon, everybody, wherever you are in this wild world. Welcome to episode four of Quality Over Coffee, continuing to be brought to you by The Letters A &I. We had a session two weeks ago and we're on to session two now and I'm here with Jeff again. Jeff, how are you?

Geoff: Hey, Peter, I'm fine. And hi, everybody. The famous phrase, good morning, good afternoon, good evening, wherever you are in the world. And thank you. Thank you for taking time to listen to us. Feedback. Give us some feedback. It would be great to see some feedback on how this is going. But yeah, yes, no, great stuff. Peter, a question for you. We talked about AI and we talked about testing with using AI, which was kind of, I think, summarized, kind of said,

Peter: Yeah. Yeah, of course.

Geoff: actually there's still work for testers. You just got to know your job a bit better, but the biggest challenge is the requirements. And if they're never good, it doesn't matter what we do with AI, we're not going to test it any better than we can. But the other side of testing that we initiated in the last call was actually testing AI systems. So what's your perspective on testing a system that's got AI in?

Peter: Yes. Absolutely. Well, that's a really good question and kudos on you for asking me before I asked you and putting me in the hot seat on that one. So, so like you, I don't have a lot of experience in actually testing AI. mean, I can take it back 20 odd years where, you know, I was trusting Falcon software, which is self-learning software that provides fraud protection for banks. And it, you know, once it learns what a fraud is, it stores it. And so when it sees it again, it's able to flag it stuff like that. and testing that was quite hard. You had to teach it something and then see if it reacted to it. And that doesn't always do that. And therefore you don't know if your test has passed or failed. Of course, traditionally you don't know if the AI is taking the information from what it's been given. And therefore your problem is actually in your data. And gosh, when you're testing, And there's a whole episode we could do there on the importance of test data. And in fact, just as an aside, I actually had a ⁓ bit of a rant with someone over dinner in India a couple of weeks ago that actually the testing world is more going to be about data management than anything else as we start to use AI because the data is Everything else is crap as well. you've got to look at is it my data? Is it what I've told it? Is it just not learning? So you've got three different things to look at. So when you're testing AI, you're probably not looking at your traditional, if I do X, is it Y? It's if I do X, do I get something that's close to Y? And do I have to do more of X to get Y and get it to learn and teach it and things like that. it's kind of like you training somebody. This just happens to be a computer. And that makes it harder because I've always said that the only grey area in testing is usually around things like performance. When someone says it has to be done in three seconds and it takes 3.1 seconds, materially does it matter? The answer is generally no, but your test has failed because it didn't make three seconds. So you have that grey area. With the functional side of stuff, it's very black and white. If you do X, you don't get Y, bang, it's failed. Whereas when you're looking at AI, it's if you do X, have you got close to it and then if you keep doing it do you end up there? So you could end up running the same test 10-15 times to get the outcome you want but then you've got to look at how do you set that up in a production environment because you're not going to be testing in production and you're testing on known data that hopefully you've taken from production but how do do it and then how are you confident and how much time do you have to put into preparing your production environment so that when you put in X that you always get Y back. Because that's what it's the customer, that's what the customer, as a customer of many companies that use AI, that's really what we want. We don't want an iffy answer, we want the answer. yeah, that's kind of my opinion on the sort of approach to it. It's just not straightforward.

Geoff: No, no, No, I think you've hit it the nail on the head. It's just not straightforward. it's not as simple. Well, data is the key, isn't it? You're right. So I guess it depends again, like you, my experience in actually testing AI. I've tested machine learning before, a machine that learns rather than actually AI. where the data is stored and you're asking it, for example, with the agenting AI, you're asking it to answer a goal. So agenting AI, you can say to it, write me a test strategy and it will go off and come back, whatever. To me, the interesting thing, I was watching a video blog from Google recently on testing agenting AI and they got quite excited about, I think I said this on the last. two weeks ago now, I can't remember quite so well. But I I said this on the last podcast was that they announced that this was really exciting and a really new way to test. But it wasn't, it was the same old stuff. And I think to a certain extent, when we're testing AI, we start to have to go back to the real core of what we have to do. What is the risk? is the area? But it's interesting. So there's suggestion for testing an agent, an agentic AI implementation. was that you have to accept that whatever answer it brings back, you almost accept that it's whether it's right or wrong, doesn't matter. The first thing you're checking is that it does it quickly and performance is key. So actually, this is all about, years ago, I worked on a project where we implemented a quote system, the first online, accurate quote system. There were the days when you could put in your details and It would come and get set right. Your premium between 50 and 60 pound a month. Yeah. This was absolutely accurate. Do a little bit of underwriting, whatever else. And, the company wanted this to go onto these, sister, these websites where lots of companies compete for the business. And so we had to be back within a certain time slot. And the only way we could test that properly was that as it was built, we tested almost each module. for its piece, its speed. So that actually by the time we put them all together, it easily met the criteria because we'd focused in on each module and we'd gone into it and we'd said, right, that one's too slow. We need to speed up. How can we speed that module up? Because if you get three slow modules, know, just, it's just exacerbating the problem. And that's a very, very high level view of this. It was never quite as simple as that. And that's what I think is important today. You were saying, you know, things are different. So actually with with these things, it's about responsiveness firstly. But then it's where did it get its data to be able to give you the answer it's given you to know whether the answer is correct. And again, this is where it's, I was saying in the last one how developers are now interested in testing for the first time and they're starting to take testing training is really how. How do you know where it's gone? Unless the developer who's put all these little agents together has built a chain that's actually something you can check. And if they don't build it to be testable, you can't check that data. So you can ask it any question you want, but have to accept the answer is correct because you just don't know. Unless you answer it, you can ask it lots of questions you know the answer to and see if they come back right. And that could be your decision making process.

Peter: Exactly.

Geoff: But it's never quite as simple as the answer is an answer to a question because these days AI is being used to help the administration processes, the customer interaction. So, you know, when you're on Amazon, it picks up that you've I know I might go in and buy Thin Lizzy, for example, and it will then say, well, there are these 12 other things you can buy that are Thin Lizzy. Did you know? Now that may it may have 75 things. in its library that it could give me, but it's chosen to give me 15. Why is it chosen those 15? So you just have to, you're almost, because it's undeterministic, you cannot non-deterministic. You cannot predict what it's going to give you. You might have a view. So this is where the process of testing becomes incredibly important. And that process needs to be bedded in from day one. So if you build something and you've got a clue how it gets from A to Z. your loss. you don't get the right answer, you've got no way of I mean, how the devil would you work your way through an AI system? And you probably want to run that same test ⁓ a ⁓ multitude of times to make sure no bias comes into it. And that it doesn't find other data as it goes through. each test could be you could run it 1000 times. ⁓ So automation is going to be important for that. But at the moment, I don't have a lot of faith in the automation when it comes to AI. that's that there's no disrespect to the automation teams out there. But I think it's important that when we use automation, we get the right automation because if you're building it into an AI system, you know, it can be quite complex.

Peter: I think that's a really good point because I think a lot of the tools out there are fighting to say they have an AI component to them but actually using them to test AI is probably something you've really got to wrap your brains around and find some very very clever people to work it all out. I find a lot of the tools that are saying we're AI this, we're AI that are really learning bots. They're sticking them in. They've got to go around, they've got to find all this information and they've got to collate it all and then for you to then to present to you how you should test something. And again, if you're building that from code and not from requirements, although requirements for AI are also could be incredibly vague, right? You just don't know where you are. I think what's listening to you just talk there. think more than any time, mean, software testing has been around. You and I have been in it since the 80s, probably officially.

Geoff: Hmm? Yeah.

Peter: really born through Y2K. So let's say we're in our 26th year of existence or whatever, it's never been more important now to have what we asked for back then, which is testers or a test person in meetings on day one of a project. So they can help guide you through it. And quite frankly, stop your pissing a load of money up the wall because you're always in danger of that if they're not involved.

Geoff: Yeah, that can help. Well, and take that one step further, I would say that it's important that everybody in the development life cycle has a quality mindset. So they're all thinking about actually how are you going to test this? How would you test this?

Peter: ⁓ yes, yep. Well, that's interesting, because I, again, had a few talks with a few people over the last few weeks about, you know, well, you know, anyone can test and the quality mindset. And I always said, you know, when Agile took off, a big time, 10 years ago, whatever, and Safe Agile came out and was going to be the way and enterprises were going to use it. And some of the training that was done was

Geoff: Yeah.

Peter: putting 20 people in the room and then just asking the question who's got a growth mindset and of course 20 hands go up straight away and you wonder why you're in the room anyway right we've all got a growth mindset why are we here but it's the it's not everyone has that and and it's all right not to have it I'm not saying you shouldn't have it but having that quality mindset is different from having ⁓ a solution driven approach to something you might be able to work out if something works but how do you

Geoff: Thank

Peter: prove its quality might not ring true and that's why there's always kind of been developer testers and I know there's sdets and don't stomp me on why they should be sTeds and not sDets as well maybe that's a future episode right but it's that thing that's been around for years that anyone can test and they just they can't and the quality mindset is hard to teach some

Geoff: education quality. Yeah, yeah, I have a friend in the UK and I mentioned his name, Stuart Reid, Professor Stuart Reid as he's Exactly. And he used to enjoy playing with the handset for television and would eventually stop the television working or have to go back for a repair shop. Yeah.

Peter: ⁓ well, now he owes us 20 quid,

Geoff: That is what I call an investigative mind in that you're not just doing the job, you're thinking further than the job. And this is where the big thinkers come in. I think you're absolutely right. Intellectual people. as well as non-intellectual people who are thinkers and challenges and will actually make a difference. But it's interesting, we're talking about tool vendors and the AI element. I met a tool vendor at a conference last year and I'd been told that that tool vendor was advertising the fact that their tool with AI in it can find 95 % of the tests that are required to test any application automatically.

Peter: That's a bold claim.

Geoff: ⁓ It's a very bold claim. Now I think I mentioned a couple of weeks ago in the last ⁓ blog where we talked about the tool that we have within the UK and Ireland testing board that can check the ability of people or computers to do the right testing. It has no AI in it at all. It's just basically an application where we can inject defects where we want to and we can do that. each time you run it you can have a completely different set of defects in which case you can then see whether people can genuinely find a good at the analysis side of it and the intellectual side of it. So I went over to see them because they said they I'd heard they were giving away a free North Bay raincoat and I thought this is brilliant. £100 benefit. So I wandered in and started asking them about this.

Peter: ⁓ well, can't be passed up, yep.

Geoff: this this tool and this 95 % whatever and I got I actually happened to immediately find the technician who was explained to me how it does it and whatever else and then I said oh well we've got a tool that can check that so that I said have you verified that you said well it's just our best guess on what we've achieved today I said well would you like to verify it accurately it's happened brilliant that would be really helpful and it would be a great thing to say even if it means we change from 95 to 90 we can be absolutely sure that we're telling the truth you know so i said well that's fine what do do next so he pulled over his the senior guy on the stand who was some kind of sales manager i think i don't know and i explained ⁓ this and ⁓ he very discreetly sidelined this technician so that he eventually went away ⁓ and was really trying to move me off the stand completely and ⁓ so I said I was actually the reason I visited is because I wanted one of your raincoats you know can't have that they're for customers I said no I'm not a customer anyway and we kept I kept pushing we should do this test your technician thinks it's a great idea and we're more than willing to do it because it'd be a good validation of our tool if it can find none it proves that ours does that thing as well so

Peter: Yeah, absolutely. Yep.

Geoff: Anyway, he eventually pulled this young lady over and said, can you get this man a raincoat please? And he came back with a large and I said, well, if you've got a 2XL that, I can't forget that. So I got my jacket and I said, all right, all right, give it up, give it up. So I walked away and I told it, I mentioned it to our team and they all that's disappointed. That'd been quite a good test. About a week after I got back from the conference, this guy reached out to me on LinkedIn, the technician and said, we were talking, but we got

Peter: Yeah.

Geoff: way late, I'd like to restart that chat. I ⁓ brilliant. So I replied to him, never heard from him, never heard from him. I then got the sales girl call me because they got my details as part of the process. Who didn't know me from Adam. And I said, I'm looking for this person because we were having a chat when we were there. She disappeared without a trace. So I think that went to prove to me that the claim they made was just that. It was a claim. There was no validation behind it.

Peter: Awesome. Good.

Geoff: So you have to be really careful. And the trouble with AI is if you're buying a tool to test it, that's quite a complex process to prove that it's going to work properly. So it's a big investment on time on both sides, you know, to actually be comfortable that you've bought something that genuinely will find what you want. And this was based on the code, not on the requirements.

Peter: Yeah, that, yeah, that's, mean, a bold claim, again, in this world today, you can just about say anything you like as long as you don't name names, but I can't understand why didn't give you a coat straight away. That would have been my solution. ⁓ well done. You found us out. Don't tell anyone. Here's your coat.

Geoff: Yeah. Well, if sees it, exactly. If it's raining in the UK and you meet me on the street, you'll know who I'm talking about because I wear it quite a lot. The badges sewn into the front and I can't get rid of it. So if I sew something over it, I'm going to make a hole in my jacket and it's going to come through. But I think it's an exciting time for people who test AI. And over time, we will learn more. But I think what this is going to do is bring the life cycle together.

Peter: Yeah. Yeah, I... Yeah.

Geoff: because with a quality mindset everybody has to think about. Because actually what fool in their right mind would release an application that makes decisions for not necessarily makes decisions because actually in fairness none of today, none of these systems really make decisions for you. They provide you with an answer and you then have to decide what you do with that. I'm in some of these high tech, very high tech companies are probably experimenting with something a bit more sophisticated than that. today, and we talk about AI, there are lots of companies that use it, but it's not predominant. Unless you're an IBM or you're an Ernst & Young with loads of money to spend on R &D, most companies can't afford the time and effort required to make this successful.

Peter: You're exactly right and it was really interesting on that note I was at ⁓ a presentation probably about 18 months ago now by MIT So ⁓ I'll name names a very nice gentleman called Alan Thurragud I'm not sure if he's a doctor or not, but we'll just call him Alan for now came and did a presentation on AI and the use of it and at the time

Geoff: Mm-hmm.

Peter: you know, kind of everyone was going, ⁓ you're experimenting it or using it for testing, know, churning out test cases, a lot of grunt work type stuff. and you could, you know, sense his hesitation, but, but when he did his presentation, it was, was quite clear that he said that, you know, and the crux of it and Alan get in touch if I've got this wrong, you know, my memory is still relatively fresh to 18 months ago was as a bit, the biggest benefit AI brings to a business is if you drive your business. with AI. If you want to use it for little bits and pieces over here and there, that's fine. But your biggest benefit is if you want to get it to drive your business. So it's not just something that your backend technologist does, it's right there in your business process. The start and how it designs them, defines them and all of that stuff is driven with AI in mind. And that's a big leap for many companies to make. But the data he had to back it up was

Geoff: Thank WHAAAAT

Peter: quite startling in the terms of success. The only thing is as a company you have to be aware year two is pretty tough. But once you get past year two, everything's on the up and up. So it was fascinating to see, because floating around technology, all you hear about is, it will make testing quicker, you'll be able to do coding without, by just telling it what to code rather than doing anything.

Geoff: Mm-hmm.

Peter: And yet if you don't look at it from a ⁓ full-on business perspective, you're never going to realize the benefit of having it. It's fascinating.

Geoff: No exactly, It's interesting actually, I ordered, ⁓ I've ordered, I haven't received yet, in October last year. Yeah, this is the sad part. Well I'm losing patience to be fair very fast, but from a company in America that will remain nameless, there was a release, I'm a thin Lizzy fan as you probably gathered, there's a whole series of the albums that were released in multicoloured vinyl.

Peter: You're a patient man, Geoff. You're a patient man. Yep.

Geoff: and being ⁓ a collector I wanted them and I could only get them in this one place this was the only place I could get them so I ordered six of them and I couldn't actually well there's nothing on their website that told me other than I bought them there was no they've been sent or anything so after about a month I wrote to them said I've noticed this hasn't been sent can you tell me what happens and I get an AI message back and it absolutely announced it was AI you know dear Jeff this is the AI agent for this company what I can tell you is we've got your order and we're waiting to fulfill it all I said that's great. If you want any more information, please write back to me and I will get a human to give you more information. I wrote back and said, OK, what are you waiting for? Which album? Is it all of them? Whatever else. And I got the message that this is your AI agent. Hang on, I'm in a bit of a circle here. so if you'd like a human interaction, please answer. So I replied again. And this time I said

Peter: Fair enough. ⁓ wow. Yep.

Geoff: please pass this to a human at the top. And then nearly six days later, I get a reply to tell me that there was one album that they've not managed to source yet. And their process is such that they post everything together. And I said, okay, fine, rock on. When you expect to that. Anyway, let's just say they said that Friday, this was before Christmas. And so every few weeks I'd write and say, well, that Friday's passed. So eventually we agreed. Well, no, and then every time I contacted them, I had to go through this process with the AI at least twice before I could get to the human. And then after four months. The system, although my orders still there, stopped recognizing old orders because it's only got so much it can do. So the AI engine came back and said, I'm sorry, we don't recognize your order. You've obviously picking up. You come to the wrong place. Please go somewhere else. ⁓ my God, what am going to do? Because a human would find it. Yeah, but from the number. But obviously this thing couldn't. So and I still haven't got it. In fact, what they've done in the end is told me they replaced one of the albums. I think one of the albums has been taken off for a different. They found a different one with a different color. They said, would you like me to have this one instead? was. Yep. Fine. And even today, their website isn't updated to show it's on its way. So I wrote again and they said, we're still trying to source your god.

Peter: Yeah, absolutely.

Geoff: I've got an email saying it's posted. Why hasn't it arrived? So yeah, it can be very, very frustrating when, AI has a very, there is a limit to how much you store in your system in the way of information that enables AI to give you a response that's good enough, you know? And that definitely was a good, I'll never buy from them again. And at the moment they've got a fairly large chunk of my money. Yeah, they've certainly got a fairly large chunk of my money.

Peter: It doesn't feel like you're both unreacted.

Geoff: which I am in the process of thinking, well, I'll just get it back and give up. But as it still is the only place in the world I can get these, I'll keep my fingers crossed these have them survive. Maybe when they do, we can talk about it on here and I can tell you how long it actually really took. Yeah, that was an AI bot supporting my questions. But I think testing AI, going back to that, obviously if it's traditional AI, it's a little bit easier. But these days, to do the job properly, everybody's using agentic AI.

Peter: Yeah. Yes. Yeah. And then you can name.

Geoff: ⁓ And I think I said to you in the ⁓ In the in the thing we did in Bangkok the bug hunting Bangkok with it ⁓ so many thousands people and The agent AI users were finding it was harder To code congenit a gente girl to give a gentle guy the right direction They get there in the end and that would be then it would be far quicker because it could just turn them out hundreds of time But actually in that situation the manual tester was still king ⁓ and could find the tests and write automation scripts for them far quicker than AI could. So, yeah, agent AI is good, but it takes a lot of prepping to make sure you get the right answers.

Peter: Yes.

Geoff: I don't think anybody's come up with the answer. So I was asked by the BCS to review a book, a potential book the other day, on testing AI. it called itself, I can't remember the exact time, let's say Silver Bullet for testing AI. And it was six people who had spoken at different conferences with a chapter each. So I said, the two things don't match. You can't have the Silver Bullet and six people give you different answers. nobody has solved this problem completely and perfectly. Although there probably are people in universities or very high tech organisations who have, but in the general world we're learning as we go.

Peter: Ironically, if you asked Jack GBT, you probably would get six different answers. So maybe it's a maybe it's as accurate as you can get. Right. But but but interesting concept for a book that you get six people to tell you what it means to them. And then you've got to settle on which one you agree with. So, yeah.

Geoff: Yeah, exactly. one you agree with and you then use. Well to be fair there has never been a single view on what testing is so I guess why we should ever think there'll be a single view on how you solve the AI problem I don't know.

Peter: ⁓ Well exactly, so just before we finish our coffees for the morning, from an ISTQB perspective then, what's being adjusted to cope for the rise of AI in testing?

Geoff: So at the moment, what we call the core things like the foundation, the advanced and the expert, they're not updating with AI. What we've chosen to do is produce two separate things, which is a GenAI syllabus, which is predominantly about prompt engineering and how to do that and how to do it properly. And also a testing AI syllabus, which doesn't repeat, but references a lot of the techniques that you'll learn in all the previous in the core syllabi. At the moment, we don't see AI as a core discipline in test. It's like agile was five, 10 years ago, but it was a great idea and we're making it work. So we're keeping the separate at the moment. The moment it becomes 80 % of the world is using AI when it's testing or testing AI. in the world your professor talked about then I think you'll do more. But yeah it's that at the moment and of all our syllabi those two are updating probably every two months with new information as it comes to light.

Peter: Awesome. Thank you, Jeff. I think we've talked AI now for a couple of episodes. Maybe next time we'll delve into a little bit of our history and come up with some stuff that might amuse you or amaze you. But for now, Jeff, as ever, been a pleasure. Coffee's done. And we'll talk to you guys in a couple of weeks. As Jeff mentioned at the start, feedback would be great. Hit us up on wherever you can find us, LinkedIn.

Geoff: ⁓ What? Yeah, adieu

Peter: anywhere else you're connected with any of us. We'd really appreciate your feedback. Thanks everybody, see you soon.

Geoff: Yeah, definitely. Thanks, all. Bye.