Isaac: Welcome to Never Rewrite. I'm Isaac Askew.

Jeffrey Sherman: And I'm Jeffrey Sherman. And today we're going to revisit solutioning in the age of AI. So solutioning is an evergreen pro or it was an evergreen problem. It probably still is. the idea of solutioning, our definition is it's when product comes to you and they say build this thing, and they don't tell you the problem that you're trying to solve and the outcomes that they want. They say build this thing. And you build the thing to spec, as it were. And inevitably they come back and they say, well it doesn't do this, that and the other thing the you know, so it doesn't achieve the outcomes that I was hoping for or expecting And you're like, Well, no, of course not, because that's not what you tell me to build. You told me to build this thing. And that's got the term it's called solutioning. It's when, you the somebody else comes to you and says, Here's the solution, build the thing, and you don't understand the problem. And so we've historically said push back on that.

Isaac: Yeah, it's like I f I feel like this is like a to begin with, it's like junior versus senior versus principal engineering concept where i now as a principal engineer, if somebody came to me and said build the thing, I would be like, Hey, hey, hey, back off. I would look at this story first 'cause it's more like especially I I think in one episode we were saying always like ask a question. Like if if the if there's a story that you pick up that hasn't been discussed, then it's not ready yet. Like don't just take it literally.

Jeffrey Sherman: Mm-hmm. Right.

Isaac: And that the engineers who took things literally like, well in the acceptance criteria, I think those will be the people who eventually get like fired because you yeah, y it doesn't really make any sense for someone to hand you just a a ticket like it's like a a restaurant order, in in this particular industry. 'Cause the idea is you need to be understanding the bigger picture around what's being asked on this thing. So

Jeffrey Sherman: Replaced. Right.

Isaac: There is ne there would never be a story in which somebody should hand you something and you just go, okay, and just start working on it. There should always be some discussion. And if there's already dis discussion done on it, then you presumably you would have already understood that information that there's this the bigger thing that they're asking here. And you wouldn't run into those edge cases or the cases where products go in, I'm so confused that you delivered the thing that wasn't asked for. Because as you're looking at it, you would look at it go, Hey, so you realize whenever I deliver this, it's not going to have this and this, right? Yeah, those conversations should be had well before you start writing code.

Jeffrey Sherman: And now the question is, how do you treat AI? Do do you allow AI you know, as you work with AI, do you as the developer say, Hey, build me this thing? Or do you say, Hey, here's my problem, here's the the outcomes I want, and build the thing. what do you do?

Isaac: this is a good question because it it bleeds into prompt engineering essentially and how people use AI in general. because everyone I talk to uses AI a little bit different. And we have this weekly meet another company I'm working with that will use if you use Claude you can use like the insights command to see how you're using Claude.

Jeffrey Sherman: Mm-hmm.

Isaac: And then we'll we'll s send it back to this person and he kind of compares and sees how people are using it, which is kind of a neat way to see habits that you're you you know, any anything like any kind of thing where you're saying, check your work or are you sure this is right or any kind of repetitive things like that. so prompt engineering in general differs between different people. And I know some people who do what you're talking about, which is they they solution it and they tell AI exactly what to build. Here's my 20 point plan on what to do. and that's fine. If it works for them, that's great. For me, I don't do that actually. I just say, here's a complaint someone gave me. Like, I'll start with a problem. A customer says, it's it's buggy in this particular responsive mode. Like if you're on a screen less than 900 pixels wide, then it does this weird thing where you can't click this button. Enter. I don't say how to fix it or you know what libraries to use if they need to use different responsive libraries that are standard or anything like that. I just say fix it. And then I watch and see how it fixes it. And then when it says I'm done, I go, well, okay, we'll boot it up locally and let's confirm. Like I want you to resize the window and make sure it looks good. but I don't even tell it to resize the window. I just say confirm locally that it works before we push it up. So I'm very You know, I I try to treat it as the engineer. I don't tell it how to solve the problem personally. then there's some people in between. I think you mentioned before the show that, you know, you you're a little bit of both. Like so

Jeffrey Sherman: Well, yeah, yeah, I I don't mean that I'm a little bit of well, I I do it both ways. So there are times but I'll get a bug report. and I think I'm a ver you'll be shocked to know that I'm a very incrementalist in in both my thinking and the way I work. and so what I will also often do is I will say, he you know, here's a bug report, do an investigation and determine what the problem is and add that result to the ticket.

Isaac: Mm-hmm.

Jeffrey Sherman: And so it'll say, I've investigated and I think this is the problem and blah, blah, blah, blah, blah. And I say, Cool. Now do the work. And then it will go and do the work. but and it you know, and occasionally the the work it comes up with is I don't agree with. I'm like, this is a bad solution. And I go back and I, you know, you you look at the what it determined the problem was, you're like, look, there's a better way to do this. You tell it the better way. but again, it it's still it it's

Isaac: Yeah.

Jeffrey Sherman: You know, the somebody else has come up with the problem and the description of you know what the problem is. And I just say, okay, do the analysis. the thing that I think I do that is different than most people is I want that analysis as an artifact. I don't want it to say, look at this ticket, fix fix the problem. Right, and then come back to me with with like here's the MR or the PR, like this is the solution. It's like, no, no, no, I want you to create that intermediate artifact of the description.

Isaac: Yeah. I like that actually. W I I do something similar here. We have like a skill for investigate task where it will go through and pull the the

Jeffrey Sherman: Hmm.

Isaac: Body of the JR ticket, it will pull conversations and Slack around the discussions around that ticket. and then anything from Confluence that already documented that particular feature area that we're touching. and then it'll build out, it'll investigate, and then after that, there's a build plan step for an investigated task that will build out the plan of what we want to do. Well, we actually just don't commit that. We just use that, that that actually just gets deleted and discarded after we're done. But actually committing that, not to the repo per se, but like a actually just adding the small MD file to the story, I might be, yeah, we ha we're not doing that. And I kinda like the way that you're you're documenting it there. That way someone can go back and look and go, this is what they were basing it off of. or even Claude could go back through and be like, we see this was implemented and marked done and this was the plan they used. And I looked through their steps and I can see why that was the wrong way to implement it or something. So that's a that I might change my process to include that.

Jeffrey Sherman: Right. good. Yeah, because I I like doing it this way because then they the explanation of the problem ends up in the description of the merge request. It's like, You know, this is the the problem, this is the solution. And and right now at work we do a lot of we have a a checkerbot so you have whatever agent you want, but then there's a checkerbot that whenever you try to do something, it will say it will compare. And so this results because it's showing the plan, the checkerbot knows what the plan was. It's like it knows why. More than just here's the bug and here's the solution. It's here's the bug, here's the analysis of of how it's going to be solved, and then I've done the implementation and that gives the checker bot right so I've got these two bots playing against each other not against the working ping-ponging not playing against and it makes them it gives them more context for it and so in that way I'm sort of I'm I'm not solutioning the AI at all the agent at all in that case I'm like hey problem goal go but because I'm I

Isaac: Yeah. Yeah.

Jeffrey Sherman: again, 'cause I'm a very incrementalist, I'm all you know, always doing the iterative thing. make create, you know, artifacts of every step along the way that you're going. And so that I find works really well. But then in the opposite way way, when I'm doing something at the other end of the spectrum, I never have it come up with a giant plan. like a thing that I'm doing this week. Jira ticket comes in And I want it to you know webhook go scrape the log scrape our internal logs, throw that into a file, zip the file, attach it to the ticket, and now the logs are a permanent artifact part of the ticket so that the AI can now get it without having to go back because there's you can't keep logs forever, but also other you know, other stuff. It's like okay. I had it go in a very incremental fashion. I'm like, okay, you know, write me a script that scrapes the ticket details, and then write me a script. It way back, God, over twenty-five years ago, I spent time doing ETLs as a DVA. and it was all steps and it was all i incremental that you wanted them all to be idempotent so could just keep rerunning the thing. and so this is always how I approach anything like this, where it's like, okay. I need a script that's gonna get the details, and then I need a script that's going to get the log, and then a script that's gonna s just a statement that's gonna zip it, and then another script that's going to attach it, and then I need a a controlling script that's gonna call all these things in order and pipe them together.

Isaac: Yeah.

Jeffrey Sherman: And so each of these becomes an artifact and it's each of them because it's so incremental, it's very easy to check. It's like, okay, you I've got this script. So give it, you know, the JIRA, the key, and the ticket number, and it spits out the details. And then the next ticket is, well, if you have these details, can it scrape the logs? So hey cool. And then the next one is, you know, bye. But boop, boop, boop, boop, boop. And I don't tell it like I have the solution in my head and I tell it to do each incremental step that I would normally do. But in that case, it's it's a different case because when you get solutioned, right, there's this giant disconnect between you, the developer, and the customer. You've got Right i i if you've been solutioned by a product person, the product person talked to the customer, the product person decided what the solution was, you implement the solution, then the product person goes and shows it to the customer. In this case, I'm solutioning the I'm not solutioning the AI because I am the customer. I say do this step and it does it, and then I say do the next step or I change my mind or whatever. But like the there is no s there is no space, there's no daylight between what what the customer wants.

Isaac: I th yeah, i i I think it's the same it's i we're still acting in the same way of the progression of a junior engineer to senior to principal, where the the main idea is that we want to reduce work that is incorrect. We want to make sure that the thing we're building there's a reason behind it and that we've already done all the work to prove that out. So the main disconnect before is that There was no discussion. Like the artifact that existed before AI was the c the discussion on the thread or any specs on the Word document from discussions with product to understand what needed to be done. So there's a there's a the old world of doing things where that always needed to be done to understand that we're building the correct thing and pushing back to when we don't have everything understood so that we can never get to the state in which we would release a product that doesn't serve all the needs that was required.

Jeffrey Sherman: Mm-hmm.

Isaac: And in the new flow where we're putting on the product hat or the customer hat, we need to also be doing the same checks, the iterative steps that you're taking. And I agree with those, which is essentially saying in between every step of the way, we have a deterministic flow that shows that we've done every checkbox state of the investigation, of what features need to be had, what the end done state means, a way to check against everything we've done to prove that. We've built the th we've solved the problem. So if it's a customer complaint and we say build the thing, we need to know some deterministic thing, which we can can run over and over and over again to make sure that if it still passes, we definitely solve the problem. My example from earlier was the responsive nature of something. It was hiding a button. So if I can resize the window and the button still shows and I can prove from a screenshot or any kind of Selenium test that the button is still visible and clickable, then

Jeffrey Sherman: Mm-hmm.

Isaac: It worked. And there's no argument to be had there about that. So if there was no regressions on other features and the thing that we've tried to solve was solved, then we are done. And so acting on that same state is essentially replicating that process. But I think that the kind of the gap that you're explaining is almost jumping from the junior aspect of, work. Let me work on it without asking any questions, solve it, even when I'm confused, about, you know, end states.

Jeffrey Sherman: Yeah.

Isaac: And then just push it up, which is what happens if you just give a a dumb model something without context.

Jeffrey Sherman: Right. And along the same lines, the amount of work that the model will do, which you know you have to pay for, is also like if you give it an open ended problem, who knows what kind of a solution it's gonna get. 'Cause it it could go hog wild. Whereas if you

Isaac: Mm, yeah. Yeah.

Jeffrey Sherman: have it d work incrementally, say, well the the solution is going to be a bunch of text explaining what it thinks the solution is. And it's never gonna get itself into a state where it's changing code and then watching the build then failing the tests and then changing the code and failing. Like those though that's where AI can really chew through your tokens. But you couldn't do that because it's working incrementally. It well, it can still do that, but it's harder. Mm-hmm.

Isaac: What those checkpoints are important and it kinda reminds me of the previous episode where we're talking about hourly based billing versus token based billing, where you can if you want to, and you if you want to give like somebody that you're you're paying hourly a really abstract problem and say, All right, I'm gonna be ac back in eight hours. I hope you got it. And then you come back in eight hours and you were vague and they didn't get it done like you wanted to, and you're mad because you have to pay them, you're gonna run into the exact same problem with token based billing. You give it an open ending problem and you're not specific enough and you don't have these steps that we're talking about. with these checks in between to make sure you've done the things and you haven't regressed, then you're gonna be a lot happier with the output. It's just about that iterative check. These each checkpoint's to make sure that you've done the main goal without breaking anything else.

Jeffrey Sherman: Yes. Don't don't break other things.

Isaac: Don't regress. Yeah.

Jeffrey Sherman: So I I think what we are saying is work incrementally, whether you're using AI or not, is always going to be safer and more effective and faster. right the old s something is smooth, smooth is fast. I forget. somebody ha th there's a yeah.

Isaac: Well even even the iteration itself will give more context to the to the thing work you're work the the model you're working with. Just the fact that you iterated, it might look at the history and go, I see that I see what we're getting towards. So the iteration itself provides context.

Jeffrey Sherman: Right. And it focuses and it exposes things that you may not have thought of. Right. If you tell the AI t to do something and it comes back and it says, I'm gonna solve it by deleting everything Which it might do. I I have

Isaac: Which it might do. The Thanos approach to delete half of it without asking you.

Jeffrey Sherman: I I have fed it vague. A thing that does not that I have not had good success with is I'm going to feed it errors from my logs and tell it to fix the errors with no other context because I don't have any other context because I'm just randomly sampling errors in my logs, fix them. that does not produce the results. That that is very likely to produce a, well this is a transient database issue, and there's nothing you can do about it, so I'm just gonna delete the warning.

Isaac: Mm-hmm. Right. It's like saying, Hey, I have a couple of dry patches of grass in my lawn. Random person, fix it no matter what you need to do. And you come back and everything's just paved over. Like, okay, well I didn't mean to kill all the grass. It's not a problem anymore.

Jeffrey Sherman: Ha ha ha. Yeah. Well I mean the c can't yeah, k grass can't grass can't go, man. It's gone. Yes.

Isaac: The Amelia Badilia version. All right.

Jeffrey Sherman: I think we should leave it there. Thank you all for listening. I'm Jeffrey Sherman.

Isaac: Yeah. And I'm Isaac Askew, and this is Never Rewrite.