Isaac Askew: Welcome to Never Rewrite, I'm Isaac Askew.

Jeffrey Sherman: And I'm Jeffrey Sherman. And today, we're going to discuss how AI doesn't save you any steps on the modernization projects. Well, so I'm saying it doesn't save you any steps. It might make the steps faster, but it doesn't remove any of the steps. And I think that's an important distinction with what I'm trying to say.

Isaac Askew: Ooh, spicy opinion. I was told by Silicon Valley CTOs that AI is a magical genie lamp and will solve all my problems.

Jeffrey Sherman: Yeah, AI is full of li- ⁓ sorry. Silicon Valley is full of liars.

Isaac Askew: Continue.

Jeffrey Sherman: So at work, I was working on a project and we have in our code base a very, old library that got mixed in with our regular code. And I want to get it out of the regular code so we can replace it with something more modern or at least just not have it entangled this weird dependency that's not even ours and get it out. And so in this case, I had AI take the first step. You know, I worked with the AI, I came up with a plan, I'm like, here's the 18 steps that we're going to need to do, more or less. I'm like, okay, cool. You know, that seems reasonable. Let's take it one step at a time. Do the first step. And I'm like, okay, here's the first step. It's basically moving the code from a directory with all the rest of the code to a directory that is not with the rest of the code. Just off to the side. I'm like, okay, so that you can then start cutting the ties and have some sort of separation. you know, move it back to being a library. And the response, they showed it to some people and the response they got is, ⁓ this thing is so terrible. We shouldn't even bother trying to remove it, like incrementally remove it. We should replace it with a modern library that is totally like unrelated, but does the same thing, but a more modern version. And somebody else was like, ⁓ we shouldn't even use that library. We should use this other library because we've, you know, it works so much better. And the two of them went off and then they had AI generate a plan. like a five step plan to do this. like, okay, cool. So I started with, I have made this first step and I don't know how to test it because it's so terrible and wacky and I'm not even sure where we're using it. And now you've got a plan. want to replace this entire library. How are you going to test that? And they looked, you know, and that there was silence in the Slack channel. I said, right. So we're back to step one of if you make any changes to this code, how are you going to test them? And again, silence.

Isaac Askew: This sounds familiar. Yeah.

Jeffrey Sherman: Because no matter how good the AI is, if you've got a piece of legacy code that requires manual testing, it's still going to require manual testing at every step. And the AI can't solve that.

Isaac Askew: Okay.

Jeffrey Sherman: There is no magic step that will get you across that. I don't know how I would test this.

Isaac Askew: So essentially you're saying that characterizing how it should behave at this point still requires human intervention. Because AI can read it and make assumptions, but you still need to make sure it does the thing it's supposed to do before and after, regardless of the underlying technology.

Jeffrey Sherman: Right, you can start with having some characterization tests and you can have some level of confidence. But if you've never, like if you bolt on these tests at this later point, you still need to be able to manually test this change, at least the first time, because you still have never, you won't know if the tests that you've just added are sufficient. Look, if you build up code from the ground up and you've been testing it all along, you can be pretty confident that the tests are sufficient. But if you're doing a regression of, now you make me some characterization tests, you won't know if it's right.

Isaac Askew: Right. Yeah, I feel like, so I've run into this recently, well, not maybe this exactly, but something similar recently where I was tasked with looking at JIRA tickets and just seeing how little effort I could put in to guiding the ticket. Like if the ticket's written well enough, can AI just do the thing for me? And so one ticket came in the pipeline about adding a feature to a product that I was unfamiliar with. point.

Jeffrey Sherman: Mm-hmm. Mm-hmm. Mmm.

Isaac Askew: And I said, OK, do the thing. And it made code for me. And looked at it, and there's tests, and it looked great. ⁓ And then I'm like, how do I know this works? Where does this live? I mean, the tests pass. I got clean-looking code. But I need to know, especially because this particular feature talked to a third party, that in the end, that works. Because usually, you mark the third party in the test, right? Well, what if the API contract is slightly off?

Jeffrey Sherman: Yeah. Right you.

Isaac Askew: And in this, actually interestingly in this case too, the API for the third party service was wrong. They said they were, if you, they said that if you provided like the same ID twice, they would change some of the data if you were providing an update, but they actually, was immutable once you, once you give it to them. So their contract was wrong about the mutability of the underlying data.

Jeffrey Sherman: Mmm.

Isaac Askew: So it read the contract and went, ⁓ OK. And if their API contract was correct, it would have worked, but it didn't. So you actually had to manually test it to make sure the integration worked. So that was good for me because after I generated it, I'm like, OK, well, let me understand where the service lives, how to test it, what its success looks like.

Jeffrey Sherman: Right.

Isaac Askew: and that required human intervention. So I feel like it's somewhat parallel to what you're talking about here, where you still need to know what it's doing, everything that it's doing first. And a lot of times people go in with the whole, AI will do it. But yeah, it'll give you code. But there needs to be something that confirms. And how do you even confirm that?

Jeffrey Sherman: Yeah, that's exactly it.

Isaac Askew: Maybe you could write AI that logs into the third party service and sees that your API call is successful because there's a row there in the third party service, but that requires manual stuff too. It's bespoke.

Jeffrey Sherman: Right. Yeah, I think what we're saying is AI will not remove any of the manual steps in your system. Like, you can't use AI to remove a manual step without at least manually confirming it at least once.

Isaac Askew: That's a good way to put it. Like you can automate it, but the first pass will be manual to guide the automation because the context is still not quite there.

Jeffrey Sherman: Right, one of the great, you know, the good and the bad about AI is it will encourage you to start taking bigger and bigger steps because it can do more and more of the coding parts. But every time you take a step, if there's a manual piece in there, that becomes a gate. Any manual piece in your code, in your system, has now become a gate that you cannot have a step bigger than one manual step. One manual piece. Ideally, it greatly encourages you to get rid of your manual pieces, which you should have gotten rid of them anyway, but you didn't. If you had full test-driven development, then you wouldn't have any manual steps in the first place.

Isaac Askew: Yeah. And another thing that you brought up that I was interesting was the argument over which technology to pick. So you said, they said, ⁓ we're going to do it in this one. And they're like, ⁓ no, this one's actually better than that one. And so they were bickering back and forth about fully replacing the old system with one of two newer, cool technologies. And this kind of like throws back to my concept in a previous episode about how everything, like the rewrite conversation is the same. The timeline is just compressed. So I can rewrite the whole system faster.

Jeffrey Sherman: Mm-hmm. Mm-hmm. Yes.

Isaac Askew: But then now what if we rewrite that one? And then someone's like, well, now there's a new technology. And the other one, you know, we rewrote that one in a month and wasted all this time because now we've determined that rewrites can happen faster. But why would you do them if features can happen even faster too? And then now you're just like every month everyone wants to rewrite it in their new preferred favorite system because it could be done so quickly. And it's just a constant cash burn and token burn.

Jeffrey Sherman: Mm-hmm. Well, to me, was like, I didn't care. Like, yeah, you want to replace, you know, this library hasn't been updated in 10 years and you want to replace it with a modern library that's being maintained. sounds great. But the first step is still, how do we get this thing out and into a position where we can replace it? Or at least off to the side or, you know, encapsulate it. You know, you've stepped, you've completely stepped over the, how do we, you know, how do you get out of where you are? which is the, the magical thinking that goes into rewrites of, well, you know, we're just going to replace this thing. And then it'll all be peaches and cream and sunshine and rainbows and cats and dogs living together. And then Zool comes along and is like, I am the gatekeeper and I need, I need you to, I was going there. ⁓ I need, you know, you might have this, this vision and all this new technology can do all this great stuff, but

Isaac Askew: Mm-hmm. These references.

Jeffrey Sherman: Still, right here is this immutable object and you have to do all the manual work to get it out. And you can do it now, you can do it later, but that manual work doesn't go away. AI cannot do the manual work.

Isaac Askew: Mm-hmm. Agreed.

Jeffrey Sherman: Well, thank you all for listening. I'm Jeffrey Sherman.

Isaac Askew: I'm Isaac Askew and this is Never Rewrite.