Geoff: Hello, good morning, good afternoon and good evening wherever you are in this wonderful world that we live today. Today we're looking at episode six ⁓ of quality over coffee and we're going to focus on the cost of testing and the value of testing today. ⁓ An area that I know Peter and myself have had interesting fun with over the years. So I will start by throwing a question over to Peter. what is the most useless metric you've ever come across?

Peter: Look, there's two answers to that. Given the subject we've chosen today, one of them is the cost of quality. ⁓ But my all-time most hated is test case... ⁓ steps per test case or defects per test case or anything like that. Because no two test cases are alike and trying to measure defect ratios against test cases is just time-consuming and pointless.

Geoff: Yeah.

Peter: That's my most mind-numbing thing, but the one that really gets my goat is the cost of testing or the cost of quality. And the reason for that is no one really knows what the right answer is. See, with a lot of metrics, if you think about defect detection and all that defect detection efficiency and stuff that would kind of started out in the early 2000s, you had a kind of finger in the air, yeah, 80 % per test phase removing the... total of of the total of defects you've got something to hang your hat on but with cost of testing it cost me two million well so what so what and then what contributes to making that cost of testing and if you build it out to the cost of quality it becomes even harder and and we'll probably as I get to the back of this and slow down my rant a little bit we'll start talking about the difference between cost avoidance and and actually cost saved because they're two different things. But I'll give you my view on what should be measured on the cost of testing and let's see how far I am off from your view then Jeff. I think the cost of testing is the actual cost of testing. If you want to get really scientific it's the cost of the environment, it's the cost of the licenses, it's also the cost of the resource.

Geoff: Go for it.

Peter: that you require to do it and when I say licenses I mean licenses of the software you're testing and licenses of the automation tools and everything you're using. Then there's the time and materials cost of the people but then the tricky bit is the time to fix. How much is it costing you to fix every defect? How much time does it cost to analyze every defect? So it's all that extra stuff that is pure time and materials gets added up as to me as what the cost of testing should be. Now, that usually comes out as a large number and scares everybody. But that's what I think if you're going to measure a true cost, that's what it is because some of that cost the developer's time, BA's time, project manager's time wouldn't appear if you didn't test and you didn't find defects. Perhaps you can give me the ISTQB view and then the Jeff Thompson view after that.

Geoff: There isn't really an ISPQB view on this to be fair because it is very much in the eye of the beholder I think. And it depends what you determine cost to be. If cost is judged by satisfaction in the business for what they've got, then you've got a different measure to finance. And obviously when we talk about cost to testing and value, everybody immediately looks at cost, physically the money, and says, Well, actually, as you say, all those things get added together and you say that's what it's cost me. Whereas this is what you spent on development or whatever. In fact, there seems to be a sort of very interesting urban myth out there that testing costs between 50 and 30 percent of the cost of the delivery. And so where's that come from? And actually, surely you can't say that because I could work for a I could be the test manager for a business guy who wants everything tested and is not prepared to accept anything less and is prepared to pay for it. So I've then got as much as I need and I determine what that is. There are others that will be more specific saying, right, you know, I want all the finance functions if I'm dealing in you know, like say an insurance company, I want all the finance functions to work properly back to back, but I'm not so worried about this bit over here. So actually you talked about avoidance of challenge. That's part of the cost of actually not having testing.

Peter: Yeah, yeah, absolutely. Absolutely.

Geoff: So a mixture of different things in there. And I'll always say it depends is my answer to a lot of this stuff. ⁓ But yeah.

Peter: But so you really hit on a good point there. And really the crux of the matter is cost. Because you don't actually want to measure the cost. Because the cost could be five million, it could be two million, it could be 200 million. That really doesn't matter. What you really should be measuring is the effort. Because it's the effort that it takes on the scale of a project that shows whether you're testing enough or your testing is effective. if you like and so if you look at you know a project is 200 000 hours if you're spending 150 000 hours of that project testing then you are testing too much or you've just not developed or understand what it is you want now if you take that kind of golden 30 % metric and say you've spent you know 60 000 of 200 000 hours on testing and fixing and things like that you're probably looking back and go yeah you know what i think we've done a pretty good job And if you can work through that and what the effort is, the reason I don't like cost is because in this day and age, let's take a country, maybe like New Zealand, so that's away from both of us. You know, I can reduce cost of testing by offshoring it. Just like that. Cost of testing will go down. So what are you moaning about if it's the cost you're worried about? It's never the cost you're worried about. It's generally the amount of time it takes. So one of, actually in my career, one of the things I'm proudest of when I joined a company, their goal was to reduce their effort in testing from 37%. And they had a clear way of measuring it. And so whether it was the right way to measure it or not, they had a consistent way of measuring it. And they said, it's 37 % today. We want it at 22%. And when I left, it was down at 17%.

Geoff: Okay.

Peter: ⁓ and a lot of that was through automation and many other things but it was just about pushing back onto the other disciplines within the project lifecycle to say well of course I'm spending 37 % of my time testing because here and this is a genuine story number one non-functional requirement was it must work good

Geoff: Excellent. Great English as well!

Peter: Yeah, yes Yes, and that genuinely was the number one non-functional requirement. Well You're not gonna tell me what good is so I'll just keep going right and then of course the effort goes up But if you tell me it must work to this specification The amount of effort I'm gonna take to find that is a minimal. So Yeah, it's interesting

Geoff: I think that one of the biggest challenges we have as an industry and I have feeling we're flicking back to a few ⁓ episodes ago really, AI may very well start to help this because it's enforcing some structure that we don't really, we've kind of got out of thinking about and that's coverage. it's like, so you were saying earlier, like effort, time, whatever. And if you're measuring your end to end coverage incorrectly, you could do a million hours testing and still not achieve what you set out or know you've achieved what you set out to do. So actually it goes back to, know, this is a simple principle, I guess, a metrics principle, not actually anything else, is that before you start, you need to decide what you're measuring. So you collect the right data to enable you to control and measure that. you need to understand the thing called coverage as well. Within IST, you'd be funny enough, I was talking to somebody about this the other day, well, when we were, when we were our last general assembly, because coverage to me is one of the weakest elements of our testing process today. We all think we understand it and I'm not suggesting I'm brilliant or you, whoever we're all, we're all at fault in some instances to not understand coverage well enough, but it depends where you are in the life cycle, what coverage means. And ultimately the value of switching costs to value, the value of testing.

Peter: Absolutely.

Geoff: me is that your sponsor, the guy who's funding the build, gets what he wants and it works for him first. The critical elements of that all that you've agreed with him all work first time. So that's you can't do that two months before you finish on a year's project. You can't suddenly come in and say right now I need to when actually you need that needs to be defined at the very front. So your value of testing and therefore your cost of testing is based on how you start your project and if you don't start your project in the right way. Bye. Actually thinking about from a business perspective. What's the outcome? I'm expecting and how's that going to be materialized and how I'm going to measure that How can anybody ever build a test process or even a development process? Because even as a development process, there's a level of prioritization needed for what you build so actually if you don't know what the critical elements of the business process is or Components are so it's a really interesting thing. So like you I don't particularly like the cost of testing and one of the main reasons I say that his colleague, when I was at Legal and General, I built partnerships with different companies and one of them was at High Street Bank and that High Street Bank had its own testing team, quite a large testing team and they decided they were going to measure the cost of testing and it was a lesson for them and for me actually because immediately they'd realised the cost of testing. The world just threw like brick bats at them. How much? You know, are we really doing this much?

Peter: Yep. Yep.

Geoff: I know you've got a room full of people. I didn't expect them to cost that much. Right. You've got to bring your cost down by 10 % every year for the next five years. And in year two, he failed and the whole lot was outsourced for actually significantly more cost than using this team internally. But it was an apparent cost and it was an agreed cost. So it's a really interesting angle. But so I think we're probably better talking about what's the value of testing. And that flicks us a little bit back to our previous conversation on shift led. And actually, as you said, quality, building it into the life cycle as you go along.

Peter: Exactly and I really like your point on coverage because I think actually understanding coverage is one of the evolutions of the art of testing since you know maybe let's say the year 2000 right more people became conscious of why you needed to test because of the year 2000 than anything else and I think coverage has kind of evolved from there in its importance and getting people to understand it. So much so that actually now we're at the forefront of testing is about risk management. It's not necessary about validating the software anymore. It's understanding the risks of doing something or not doing something. a lot of people who work for me get fed up with me talking about testing is really your insurance policy. You could choose full excess. Well, and that's going to cost you a lot. Or you can choose to buy down that by paying premiums.

Geoff: Okay.

Peter: And the premium you pay is what you pay for people to come and tell you it's either working or it isn't. Yeah. But it's also about understanding the risk. I'll jump into again, one of my favorite stories and probably one of the ones that kind of helped smarten me up in this world was, you know, having to sit in front of a CEO with my leader who did an exceptional job and ⁓ having

Geoff: going to

Peter: 15 or so people from a project in the room with me and saying he won't sign this off because it was more of a governance role that I was doing at the time and we're prepared to go live this weekend and when the CEO looked at me and I said why and I said basically they don't understand the risk they are taking and he turned back the project and said can you explain what the risk is and the project manager says no but we've got 25 people prepared to work through the night if anything eventuates And the CEO just looked at them and said, go away and come back when it works. Now the real sting in the tale was the following day, the project manager came to see me in a very grumpy mood and said, just tell me what you need me to do. And I said, well, you've got to show that this risk is not going to eventually, so you got to run. It wasn't something like my five or six tests to validate that this can do this and this can do that. said, and then you'll know if it doesn't, you'll understand why you'll either fix it or you'll accept the risk. It's not going to do it. and he went away and I didn't hear anything and I didn't hear anything for a week and then I didn't hear anything for another week and then on the third week I got sent a test summary report and buried in the last page was a severity one defect they'd found in the five test cases that they ran and he said will you sign it off now and I just said absolutely here's my signature and you're welcome and he said welcome for what I said saving your job my friend saving your job Because if that had hit production, would have just... Well, the level of damage and the cost to fix and the reputational cost would have been huge. And that's kind of when I started realizing, if you understand the risks, then you can go live with 100 defects, as long as you can manage it and you understand if something goes wrong, what you need to do and the risk you're realizing and the value of that risk as well.

Geoff: Yeah, no, absolutely. And I think actually your point on risk is well made because if we are actually the guardians, shall we say, of quality in testing, then part of our role is to communicate accurately out to the right people where we are at any moment in time. And if you're communicating out to a business sponsor who ultimately owns this thing when it goes live, and as you say, his job is on the line if this doesn't work. just telling them you've done 30,000 tests and you've got 50,000 left to run is no measure at all. And the ironic thing about that when I go back to coverage is the worst case of coverage measurement that I've ever seen is where people write, let's say, 50 tests because they think that's fair. And that's what they measure coverage against. If they're 50 % complete on coverage, they've got 25 tests that have gone through successfully.

Peter: Yeah.

Geoff: Well, that's not coverage. That's coverage of tests. That's not coverage of the application. It's not coverage of the solution, which is what we really are talking about. the key when you are actually in a scenario of looking to provide value to your customer, risk is the only way they'll understand. However, I say that because that's what the books say. The reality is. And I remember this so clearly and one of my stories here, we have a, I was responsible for a delivery of a massive application as the test. They call me a test director there. I've got a promotion and it was ginormous. There's about two, 300 people working on it for a couple of years. And we had a sort of critical moment, I suppose, when everybody realized, oh, we've got some problems. And the only reassuring thing that the sponsor could say is, I trust you to get it there on time. And he just walked out of the room. And it's like, what does that really mean? What does that really mean? And then afterwards, when we did get it there on time, having worked, you know, hours to do so, he said, I told you you'd do it. He didn't even say thank you. It was just kind of expected. And therefore he had no view and we didn't give him any view. Did you know the risk you're taking? Did you know? And in The reality is that he probably wasn't having a risk because the developers, the testers, the designers, we just put our heads down and we made it work. That it was that critical moment when we realized we were in trouble that focused us. But actually there was no measure of where we were at the time other than we knew we were going to not make the date if we didn't do something different. So it's interesting that even our sponsors don't always physically understand the value of testing either. In that same organization, actually, I had a tester who felt that she needed to do at least another six weeks testing after we finished. I said, why can't you go to Well, I found all these extra coverage that we haven't done. And it's like, well, why have you found that now? You've been dying for six months to a year, you know. So actually, if if we go back to cost of testing again, you know, we are then paying for and I don't mean this harshly because actually there's significant amount of brilliant testers out there.

Peter: Yeah.

Geoff: But we are often paying for people that have just got the word test in their CV because we're caught in a trap and we need bodies. So boom, boom, boom. We just load them in thinking that bodies will solve the problem. It's the old argument, isn't it? don't you don't you don't ask a lady to ask nine ladies to have a baby in a month. It's still going to take you that amount of time for one person to complete the task. So it's like. So again, you go back to what's the value, what's the cost and. because just dumping bodies on the floor to get the work done doesn't it increases the cost and decreases the value necessarily because you don't progress in the right direction. But the last thing I was going back to your very first comment, the last thing I ever want to see and I hated seeing was the step counting. You're at and it's like so I had ⁓ to work on a project where there was a business a business person for the company and I placed from my company a young

Peter: Ugh.

Geoff: graduate tester in there to do. Now he was a real tester so he thoroughly checked everything yeah and he found lots of significant defects but didn't progress through the steps as quickly as the business person who found nothing and zoomed through them all in 30 seconds flat. I was asked to remove my tester and give them someone like the business tester who could get through the process so much quicker and I said well it depends what you want doesn't it. Do you want an app that works or just want to say, because again, value of testing, a lot of people just think they've got to have it and they don't understand why they didn't even ask why.

Peter: Yeah, and I think the ⁓ real thing for me is, coming back to one of my original points, they focused on the actual cost. But that's not really the cost of testing. Just simply because even if you had all the development and reworking, every defect you find, and let's just say if you're using a wonderful defect scale, let's say every defect of two and above.

Geoff: Mm-hmm.

Peter: you find has a cost avoidance associated with it because if that was found in production you could be talking 50,000, 60,000. Self-ones that stop businesses can be up to 150,000 dollars, pounds, euros, whichever we want to say to fix plus reputational damage and that's not happening because of what has been done. But that number is always hard to come up with and that's why I say

Geoff: millions.

Peter: Now lot of the metrics where you try and prove the value of testing is about demonstrating the cost you're not spending because you've spent the money in the first place. And if you're dealing with a true accountant, you know, sometimes that doesn't true up. It's the risk managers you need to deal with to say, look, if you don't do this, your risk is X, Y and Z. And then you need the appropriate sign off, you know, because it's all right to go live with a risk that's going to cost the company 10 million.

Geoff: No, you're absolutely right, yeah.

Peter: as long as the CEO signed it off because he's probably the only person who can underwrite a 10 million dollar write-off, right? yeah.

Geoff: Exactly. Yeah, that exactly. That makes you think about, I can't remember the name of actual vehicle, was in America, a Ford truck of some sort, I think it was a Ford truck, where they knew they had a problem because they put the petrol tank far too near the very back of the truck or car, whatever it was, I can't remember. And they actually calculated the risks of not fixing it and to hit the deadline. and whatever they calculate, let's assume it's $100,000, whatever, I'm sure it was a lot more than that for a cost of a death, what you would pay somebody who died because of it. The actual impact to them was nearly 100 times that by the time they'd been through the courts. And it would have taken them a few days to change the, before they started the production, to change the position of the tank. and have it running through the actual build process. So there wouldn't be that problem. But they wanted to go live. The other story I used to tell people when I used to go out selling testing was Nokia. I don't know if you remember Nokia around the year 2000. They were like the big thing. Everybody had a Nokia mobile phone. was like, if you didn't have Nokia, you didn't have a phone really.

Peter: Fun fact on Nokia, they produced the first ever camera phone. I think it was Nokia 7650 or 2650 or something like that. Tested. Yeah, well I was the test manager of the team that tested. Yeah, yes.

Geoff: Mm-hmm. Hey, I won't be careful what I say now then. What I used to say was my daughters, so my daughters around the year 2000 were in there, mid teens and ⁓ no probably slightly younger than that but young, old enough to have phones let's say and they, there was almost a secondary industry for Nokia where shop, local shops were established where you could take your phone in and they'd reinstall everything and correct it because it never worked first time, there was always a problem and They never really improved in on that model for a long time for a long long time ⁓ And so then you got the Blackberry came along which suddenly was stable Could you can you know it was kind of a strong stable thing followed off see by the smartphones? Which have been a lot more stable and are easier to update we can update you know to take to a shocker But Nokia lost a lot of trade a lot of business because they either didn't see that coming or they prepared to accept that they were releasing things that didn't work. They were making the wrong risk-based decision fundamentally I would say. ⁓

Peter: Okay, another fun fact. Tested the first BlackBerry. Yep.

Geoff: Really? There was me thinking you were only ever tested in finance. You live and learn something every day. Really? Okay. That's fascinating. Well, here's my fun fact. ⁓ I tested that I was one of the first testers of IBM Lotus. Is it was it Lotus Notes mail? Whatever it was. I forgot the Regent's Park. I was sent up there by my company. had to go in a closed room. Having never done had email in my life ever. And

Peter: Yep. Yes. Yes.

Geoff: They said it was the most usable system that you could develop. So my job was I had loaded questions. I had to go and do whatever I thought was possible. And it was recorded all around me. It went wrong. And I had to phone up the help desk and what didn't click was the first phone desk that was supposedly the other side of London. The guy was in my office in minutes flat fixing it. And you don't think about it. But actually what then is there was a double sided mirror. Everybody's watching me. It was really interesting and not at all oppressive.

Peter: Hahaha ⁓

Geoff: because you are on your own in the room and you don't think about it because you are focused. But yeah, I tested one of the first people to say the Lotus Notes mail system. So we have our history, don't we?

Peter: I would say... yes, but I would say ⁓ given the history and reputation of Lotus Notes as a mail system, you're a very brave man to admit to testing it, because it never bloody worked. So how bad was it when you were working on it?

Geoff: Yeah, Balik, what we gonna do? To be fair, I had some very simple tasks to do. Yeah, I think it was all that stuff where you had to, can't remember what they called it now, where you had to, it didn't maintain the thing, you had to keep taking up loads and things, was, I can't remember much about it to be honest. But what I do recall is I worked for one of the large system integrators in the mid 2010s when they were actually migrating away from Lotus Notes to Microsoft. They were one of the first companies that were still using it for everything.

Peter: haha ⁓ no, no, no, Big in Australia when I first turned up. Not hard for Microsoft to come in and take over with Microsoft 365 and even Office 2006, I think, was what we migrated to in about 2012 here in Australia to Lotus Notes. But ⁓ yes, Lotus Notes was...

Geoff: ⁓ there's still something round there, is there? Yeah, yeah, yeah, yeah, yeah, okay. I didn't realize that. But yeah, that's certainly my company. yeah, it was anyway. yeah. But back to the value. So as we get near the end of the half an hour now, I guess what we're saying is, that, and again, please correct me if you think I've got not picked this up, that we really are in a situation where understanding of the value is not clear. and still isn't clear today. The real value is to the person receiving the software at the end of the day, but they don't understand the same things that a tester understands. So somehow that needs to be translated to enable them. I've done 30,000 in my 50,000 tests. What does that really mean? That kind of thing needs to happen. But actually, cost of testing is a dangerous measure because it can't be accurate. because you don't know all the costs and you're measuring something that isn't giving you necessarily the value that you need in a sense of measurement. So would that be fair?

Peter: Yes, I think that's fair and I'll just stay on my soapbox for two more minutes. We never measure the cost of development. Why's that?

Geoff: I don't know. What? Because actually, but that's interesting. Is it? Part of me says it's because testers are the we like to prove ourselves. So we want to tell people what we've caused.

Peter: ⁓ well, maybe. I I would say that from an introvert perspective it's probably a bit of both for development and testers. But I always think it's, they don't want to worry that it's going to cost $10 million to build something, but they'll worry it's going to cost $2 million to prove that it works. And it always seems like the false economy to me that you don't measure. If you're going to measure one part, why are you not measuring everything else? Because...

Geoff: Mm-hmm.

Peter: you can only improve one part if you improve the life cycle. And I need to probably say I think a lot of this is improved and yes this is going on the record. I think this is improved since more people looked at doing hybrid agile approaches to stuff. There are very few people who do pure agile but one of the benefits of any hybrid or agile approach is the ability to measure stuff at any given point in time. Measuring the right stuff.

Geoff: Mm-hmm.

Peter: No, because you can... Yes, yeah. yeah.

Geoff: If you're doing it properly you're absolutely right. Absolutely. You're burning down charts and all those kind of things. Yeah, no, no, absolutely. Absolutely. Absolutely. so to me, the big issue with not with cost of development, cost of test is actually that organizations still don't still struggle to understand where they sit. Are they separate? Are they together? And actually, the cost of test is actually just part of the cost of development. It's just part of it. So if you consider it separately, it seems expensive. you consider it as part of the overall cost of development was 12 million pounds, nobody would argue. But if you say I'm paying 10 million pounds for this and 2 million pounds for that, oh, why am I? It's the old argument. It's what I used to call the bike shed argument. I'm a governor at a school. Guy walks in. He's been asked to prepare a plan to redevelop the whole school. and he presents this plan and it's 35 million dollars. I'm doing this in Australian for you. And everybody's excited, no questions about the cost at all. And then one of the people in the room says, well, there's no bike shed. Where are kids going to put their bikes when they're coming? Ah, right, I can put one of those in for $4,000. $4,000 for a bike shed, you're joking. And when the next six hours talking about the $4,000 and not.

Peter: Yeah

Geoff: the 35 million that they're about to spend. So the smaller the number, the easier it is to challenge. The larger the number, the harder it is to challenge because people just accept it's a number. And that probably has something to do with the value of testing as well. So anyway, well, Peter, thank you. That's been a good one today. So yeah, absolutely, absolutely, absolutely. So I guess we leave it here and we say thank you everybody for listening to us.

Peter: Now, good full circle moment at the end there.

Geoff: Obviously, if you've got any feedback, please, please do do let us know to drop us a note or put add messages after these episodes in Spotify and Apple. But thank you for your time.

Peter: Yes and very soon we'll be able to go live on YouTube as well. ⁓ I've had a good shave so we'll be alright to go. Very good. Thanks Geoff. I'll speak to you in a couple of weeks.

Geoff: Wow, ⁓ Cheers, my friend. Thank you.