speaker-0: The agent doesn't change your standards, it enforces them. And the lower your standards are, the more enforcement you get. And the best way to get reliability is to create a specific reliability band that you're willing to accept. The model is non-deterministic. You're never going to get the same thing out every time, but you can narrow your reliability band to something that's acceptable. Businesses already do this with risk assessment. Now it just has to do it for agents.

speaker-1: you So today on the show we have Ian Johnson. ⁓ I think I discovered Ian Johnson through LinkedIn and I realized he'd written a book called Harness Engineering, which I have been reading for the last week or so ⁓ and is fantastic. I'm really enjoying it. It's just got lots of really good thinking and really good skills and tips on how to set up your harness. Really, I guess that's what I'm getting from it at the moment. ⁓ But I could tell that Ian's really you know, done his research and learned how to do it well. And I was really excited to, to kind of hear more about Ian's setup and how he develops day to day. And then maybe we'll sort of dig into some of the techniques you use ⁓ a bit later on as we go. so yeah, do you want to introduce yourself in anything else we should say about you before we start?

speaker-0: Yeah. Yeah, so ⁓ I'm Ian Johnson and ⁓ I am a staff engineer at Parento. ⁓ and that's parent with an O at the end. And what we do is we build paid parental leave programs to attract talent. ⁓ so I recently wrote ⁓ a book and the Many of the lessons and learnings and experiments and things like that come directly from work. Some of them also come from open source and experimentation, but many things ⁓ I learned while we were implementing these things at Parento.

speaker-1: Cool. Amazing. All right. Well, we haven't got too long. So let's dig straight in. So maybe tell me a bit about sort of how you started with AI coding. presume you're a developer before before AI. So what were your first kind of experiences and how did that then develop?

speaker-0: Yeah. So my initial ⁓ my initial experience with AI coding was with GitHub Copilot whenever it was initially out, whenever we were still in the autocomplete stage. ⁓ I used that for six months roughly, ⁓ but didn't really see very much of a productivity boost, a little bit, but it wasn't that impressive. so I took a little bit of a break from that and ⁓ about a year. After that, that's whenever agenc ⁓ coding really started to get big. That's whenever Claude Code came out and many more of these bigger models. and that's really whenever the process changed. So that's whenever I decided to give that another go and try a different method. ⁓ and so really started to dig into that. And I've been working with ⁓ harnesses and with ⁓ agentic engineering for about eight or nine months now. ⁓ actually maybe a bit more. And ⁓ some of that has been deep in the weeds and then other parts of it has just been consuming it, figuring out how to actually work with it on a day to day basis.

speaker-1: Cool. That's incredible. you, mean, the fact that time is now so compressed that this has happened in nine months that you've gone from kind of copilot to really sophisticated harness engineering with and writing a book about it is that's quite something. so, so, so tell me about your, what does your setup look now look like now? What's your

speaker-0: Yeah, so ⁓ my setup right now, I just started using a product called Orca. ⁓ I decided to try that out. ⁓ so far it's been really great. prior to that, I was using CMUX for a long time. I really like CMUX. ⁓ so far I'm really enjoying ⁓ Orca, and we'll see where that goes. many times I'm using Claude code just in the terminal or in some kind of a tool. I also use a couple of other harnesses, Codex, Pi. but ⁓ in terms of working, there's two main methods that I use, either ⁓ dispatching agents in parallel using CMUX or Corca. or the newer thing that we're actually building right now is very exciting. It's actually a factory. And that is a really exciting piece of software because ⁓ that raises our abstraction level at where we think. ⁓ so the factory is able to ship entire tasks as ⁓ context. So we basically built a Kanban board that looks like Argyra and the agent automatically picks things up and moves things across the board. The thing that's great about that is that you get a lot of parallelism. you can have as many threads as you want. ⁓ so long as your computer supports that. And so for us, for me personally, that's usually about four or five threads because we use Docker. It's a lot of RAM.

speaker-1: Yeah. Wow. Okay. Before we jump into that, I think that's, and actually what it took you to get to that stage, because I think if anybody just kind of does that setup, you're probably going to get some pretty chaotic code straight away. But I know from reading your book, you put a lot of thought into ensuring the quality is right, ensuring that you're giving it the right information and the right context before you let it go, let it rip effectively on your Kanban board. Tell me a bit about Orca. I don't know about Orca. is that?

speaker-0: Yeah. So Orca is ⁓ an agentic development environment. ⁓ I think it's been out for a little while now. ⁓ but it's very similar to, say, Warp or maybe CMUX. ⁓ it's a little bit more integrated than CMUX. It has some more features, so it's probably closest to Warp, ⁓ if anybody's used that before. And essentially what it allows you to do is to be able to drive many agents in parallel. I would say the main ⁓ benefit and selling factor that differentiates Orca from me is the ability to use multiple harnesses. So I don't have to register like a particular harness or a particular API key. Instead I can fire off different accounts with different harnesses all within the same product.

speaker-1: Okay, okay. So it's kind of orchestrating your harnesses and your the different sessions that are running. And does it I mean, does it pop up and say, okay, it's time for to review this, this one kind of thing? Does it have a code review piece in it? Or how do you how do you have any sort of verification stage, I guess, is the is the question.

speaker-0: Right. Yeah, so ⁓ in terms of verification, the way that we do that is through ⁓ code review essentially. And I know that a lot of folks have been saying that code review is going to die. And I I think they're probably right, ⁓ at least in the form that it is right now. But we still wanted to be able to validate things before they went out to production. You know, ultimately we're responsible for our changes and the impacts that they have. We own. The product. The LLM's not accountable for that. ⁓ So many of the things that we do are based on creating more guardrails, more determinism, and more checks around our product. In terms of the checks that it took us to get there for review, we have a set of review agents that fan out and ⁓ focus on different things, feedback review initially, and then the agent automatically responds to that review. ⁓ so it they are sub-agents, but it is still the agent kind of doing his own fixes for that review. But the thing that's really nice about that is that whenever I get a code review, ⁓ I don't have to worry about linting or security or any of those sorts of nitpicky things that have already run through the linter and so forth. Instead, I can focus on what the actual change is. and in addition to that, we also have a hook so that if a PR has or if a code change has way too my many lines, it'll be rejected. So it's also got to be small enough. Again. For the context for a human to be able to understand that. If I can see a PR for a thousand lines or less, ⁓ probably actually smaller than that, five hundred lines or less, you that that's that's readable, that's reviewable, that's something that I'm going to pay attention to and actually do a good job. If I get a five thousand line diff, everybody's just gonna click, yeah, approve. Looks good to me. So that's a lot of what I was trying to get rid of is to be able to consolidate that. So part of that workflow is Holding the PR size to a maximum and if it violates that to break up the card into separate individually releasable cards to not make excuses, to not ⁓ say that ooh this particular thing is an an exception to the standard, but instead to be very ⁓ explicitly rigorous to that standard.

speaker-1: Okay. And are you setting the expectations on the agent on, on PR size at the beginning as well, presumably you're not otherwise you'd spend a lot of time, I guess, breaking things up.

speaker-0: Yeah. Yeah, I do have ⁓ some initial skills running beforehand that will break cards down to a small amount and it will in the agent does have directions saying that these changes should be small and if the change starts to get too big to stop and suggest breaking things out to another card. ⁓ we also have a rule that one card equals one PR. So as soon as scope starts to get a little bit bigger, as soon as things start adding, then Claude's initial reaction is should we split this up? So it does happen proactively, but if we get past that, then the hook catches it in case I missed it or in case Claude missed it.

speaker-1: Okay. Okay. Cool. All right. So you touched on spec in there. So what do the cards actually look like? mean, are you, it seems one of those things where, I mean, if we give it something vague, obviously, we're going to get different, different results, we can be unexpected, we'll probably blame the agent for not, we're not understanding what we thought. so how do you describe your requirements in a, a very clear way? And I mean, some of that, guess is functional requirements in there. Do you also put any sort of non-functional requirements in? How do you kind of spec these things out?

speaker-0: Yeah, so in addition to functional and non-functional requirements, I also make sure that there's explicit acceptance criteria that are falsifiable. If the acceptance criteria are not falsifiable, that does not help. So that's definitely an important thing. And then I also usually have an out-of-scope section where I will explicitly declare these are the things that are out of scope. And that helps the agent to plan because then it can look at a smaller set of files. ⁓ and I usually get better results and better planning out of that. ⁓ so the main way that we do ⁓ tasks is through JiraCards. That's our source of truth. And essentially the description is our spec and it includes mostly explicit acceptance criteria, screenshots, ⁓ different things like that in order to give the agent as much ⁓ as much information as possible up front.

speaker-1: Okay, cool. Do you use AI to help with the speccing as well? is that the kind of, you keep that?

speaker-0: Yeah, so we we do that ourselves, but also we have ⁓ an AI assistant to do that as well. We have a skill, a refined skill, that will take a look at a card and it'll analyze it and say, ⁓ this acceptance criteria is not good enough. It needs to be better here. So it kind of double checks our work for us before we feed it off to the agent because we might have had a gap or missed something.

speaker-1: Okay. That's great. So that's like a reviewer, but it's not making assumptions for you. It's actually, you still have to do the work in a sense. That's interesting. So where would you say now how much your, your, your standard day, how much do you spend speccing? How much do you spend reviewing? How much do you expect managing your harness? I imagine is a, a, is a reasonable piece of work as well.

speaker-0: Yeah. Mm-hmm. Yeah. Yeah. I mean it it really does depend on the day as far as where my efforts lie. But usually the specing does not take me very long. I I can spec out a card in ⁓ a few minutes usually. It's nearly not that large of a of a problem in terms of time for me, for the size of our cards. And that's also because Of the size of our cards, because we explicitly set a rule for ourselves. We want to have small cards that are independently releasable. It makes all these things go smoother. If we had larger cars, they would be a bigger problem.

speaker-1: Okay. That's a real skill in itself. how do you get to, let's go back a step from there. How do you get to the business feature, like getting that down into cards? I guess we're moving a bit away from the AI bit, but I think, I think with AI, this is going to become a more important skill again, right? This is like, it's not about the coding anymore. It's about giving the right instructions to the agent. So yeah.

speaker-0: Right. Yeah, and I think that what what I'm doing is it's sort of an alternative to SDD. ⁓ so SDD, ⁓ fair or not, folks are equating it with waterfall, but with SDD you have this big spec and then it follows you along in the process. And as you're making changes, discovering things, finding things out, you update the spec, and then that's what carries you through the whole way. and so it is sort of a waterfall process, but it really does depend on the size of the card. But if you're doing a full feature, Which you probably are with SDD, then that's a big hunk of work. so instead we are kind of taking the agile approach of instead let's plan the whole thing out like an SDD document, but the SDE document is an epic for us. That's where we put our projects. Our projects are epics. And then once it ⁓ is able to add a spec to that, then we'll break it down. Then we will go through a planning session and we'll break down each of those epics into ⁓ a set number of tasks. We'll ⁓ relate them to each other so that they have blockers and all of that. So that way as much information as possible is in Jira. And once that's in place, it's really easy for the agent to tackle the the correct thing next because it can just use the Jira ⁓ API and discover. ⁓ this is the car that's unblocked. This is what I should work on next. That also helps a lot with parallelism. I can spin up a bunch of parallel agents tell to work on different things and it'll choose the things that are not dependencies.

speaker-1: Hmm. Okay. Amazing. Yeah. So, so, okay, so there's, so we've got a great spec. How many cards do you think you're getting through in a day? Give you some sense of that.

speaker-0: Yeah. ⁓ I would say maybe six to eight cards a day. Something around that.

speaker-1: Is that for the team or for a person?

speaker-0: ⁓ that's per person. And that is also with the caveat that we do a one to one PR to card ratio. So, yes, we are pushing through a fair amount of PRs, but our PRs are also small. And that's actually what keeps us able to continue that velocity is because we can actually still review our code because what comes out at the end is maybe a two hundred and fifty line diff.

speaker-1: Okay. Okay. Okay. So that's great. So you're getting it down to a small size. It's still human reviewable. You're using agents to kind of help catch lots of the kind of the basic things that they can catch. And so when you're actually reviewing, what are you, what are you actually looking for?

speaker-0: Yeah. So there are two things I'm looking for. number one is just functional correctness and ⁓ that it's doing the thing I expect it to do, that is touching the files that I expect it to be touching, that it there wasn't any scope creep. ⁓ and then the other thing that I look for specifically are patterns. if there is something small about a diff that I don't like, then that's something that you just change that one small thing. But if there's a pattern that's represented there, then that's something that needs to be updated in the harness. ⁓ because if you update the harness, then that pattern becomes undoable in the future. ⁓ and so a lot of my PR reviews actually turn into harness suggestions. Okay. Because whenever I find the same pattern coming through. I use the rule of three. The third time I see that pattern I say this belongs in the harness and then I promote that into the harness and then I run through more cards and I validate that it's no longer giving me that solution.

speaker-1: Okay, this is great, right? So the feedback cycle is there. In the same way as a traditional agile, you do the retrospective at the end of the sprint, and then you say, okay, we're not doing this again, or we're going to start doing more of this and you put some kind of things. So now we're doing that with the harness. So, okay, so I've, I've read some of the chapters in your books on, on the harness side. So just for everybody else, just take us through a little bit of the basics of how you think about that, right? How are you thinking about? setting the context.

speaker-0: Sure, yeah. So I would say in terms of the broad ⁓ term itself, ⁓ harness engineering, there's a lot of ambiguity there. the word harness kind of is having a lot of semantic diffusion right now. ⁓ so what most people would call the harness is everything around the model. ⁓ but that's way too large and way too vague. I define the harness a l slightly differently. And I say that it's everything that wraps around the model that executes what's happening. and the reason why I do that is because it keeps other environmental factors out of it. And the other part of the harness that is necessary is what I've been calling the charter. A lot of folks just call this context, and I'm just talking about. The claude folder and the claude.md file, just that. But that context, in my opinion, is special, and that is why I call it something different. ⁓ if you put something in your claude.md file, it's gonna be in your context every single turn. If you put something in your skills folder, you're gonna have this new ability to have a skill. ⁓ these are specialized things that are in your project, if they're stored in your project. And the reason why I call it a charter is because it's an agreement. It is a charter between the development team and the harness, an agreement that says here are the standards we're gonna follow, and here are the ways that we're gonna check those standards. And so the charter sometimes gets folded into harness engineering because folks are talking about ⁓ my harness being a collection of skills and commands and rules and so forth.

speaker-1: Hmm.

speaker-0: But that's actually the configuration of your harness. It's a separate thing. And it still does the same work, just at a different level. Your charter is able to restrict and restrain behavior on that project level, ⁓ but your harness is able to do that across a broader level, across many projects. And the more projects you need to be able to handle, ⁓ the bigger your organization. That's whenever those higher level patterns start to emerge that are really important with not only building the harness, but also with building other ⁓ more complex types of harnesses, not just coding agents, but also factories and orchestrators that do jobs with many agents and orchestrate their results together, fan out to smarter or ⁓ more cheap models, those sorts of things.

speaker-1: I think, I think I really liked this concept of the charter and I've been struggling in the conversations I've had with the name of this thing, right? Cause you were going, the context. then you mean, well, the Claude MD and then, actually it might be the agent MD. it's like, it's like, okay, that's, and it's more than that, right? It it can be, like you say, it can be the skills and things, but it's, it's also not what the hard, mechanics of what the harness is doing, what the tools the harness uses and things are a different piece. Right. And if you're optimizing that you're optimizing. taps, perhaps the effectiveness of the agent in general. But if we're talking about the charter, we're talking about, okay, how the, how the agent knows to create good code on your repo, guess, right. The right code that you want without you having to specify every single time in each of your tickets. Right. And I'm really pleased to hear that actually it's quite quick for you to create a ticket. And I think that's probably only because you've got the charter correct and it kind of, it can quickly kind of place that ticket in the right context. So yeah, go on. Yeah.

speaker-0: ⁓ yeah, and I I was just going to say that that, you know, that is one of the main benefits of it is that you get this positive feedback loop. And when you have a feedback loop in place, it can be positive or negative. ⁓ so it's all about putting the right things in place to ensure that it's a positive feedback loop. Because for example, if you have messy code, then the agent is going to just write more messy code and it's going to amplify that. ⁓ so fighting entropy is ⁓ a very ⁓ being part of harness engineering.

speaker-1: Yeah. I mean, it's a big part of every software engineering, It's always as your project gets bigger, bigger, it kind of gets to a point where you go, can't work on this anymore. It just breaks all the time, right? Because the entropy is too high. think, and I think, you know, there's going to be this big skill in controlling that entropy and, the things you talked about here with the charter, think are a good, a really good starting point. So I'm kind of, I would love to know a bit more about the kind of things you're putting in the charter. Right. I'm probably just starting to get to those chapters in your book. So I'm of skipping a bit, but tell me about the types of things and, your guidelines for other people. Right. Because I think everybody is starting to think about this now and, you know, the agents are getting so good and the models are getting so good. They are capable of writing the right code. If you give them the right background effectively, if you give them the right instructions. So the limitation is now ours rather than the, the coding agent, which I think it kind of was. maybe a few months ago, right? So I'm super interested. What kind of, what does it look like, your charter and what kind of things.

speaker-0: Yeah. So my charter is it's a Claude file. So it is we use Claude as our daily driver and so ⁓ it is effectively just the dot Claud folder and the Claude file. There's a couple of extra files that we added in there as well. ⁓ for example, in our Claude folder we have an index.json file. ⁓ and that is ⁓ to make searching everything faster. So our Claude file actually says before you go looking for stuff, look at the index.json file, see if can find it there, and then go point to that file directly. So that way you don't have to read through a ton of files and do searches. ⁓ So that helps a lot as far as the indexing. ⁓ And then ⁓ everything lives in their standard ⁓ folders. So in our rules we have rules that are specific ⁓ and that are scoped to the folders where they're present. So we don't have rules that say when you're making API changes, ensure that you are using actions to do that. ⁓ instead we have a rule inside of the API folder that's clogged down to that and that says own you know use that during this time. So we never have to have those predicates beforehand. That's one of the tells that we've discovered is when you have to say when you have to set up a condition or scenario, then it probably belongs in its own rule file with more restrictions. and another thing that we've discovered is the quality of rules. That rules need to be explicit, they need to be specific and implementable, they need to be falsifiable. You need to be able to take a look at a rule and say yes or no, this this is adhered to it, and it shouldn't be vague. So a rule like use composition over inheritance does not help a model. ⁓ and it actually it it makes things worse because first of all, you're spending extra tokens on it. Second of all, the model already knows that, it's trained on that. And then in addition to that, what you're actually doing is you're telling the agent to lie. You you're giving the agent all of this information saying, ⁓ yeah, we do it this way but then it looks at the code and it sees that's not how you do it. And so whenever you ask it for a diff, which way does it go? And that's part of the variability. Does it implement the new way or the old way? And in general, it seems to lean toward the old way. And that's because the imitation surface of the code is just so much more broad than the instruction surface of the charter. So the charter can say a bunch of stuff, but if you have one line or one rule in your charter and you have a hundred instances that do the exact opposite thing in your code base, then it's gonna follow your code base. ⁓ and so that's another thing. Rules have to tell the truth. A rule that lies teaches the agent to lie, and it also doesn't help you out because it doesn't actually get you to where you're going. Instead, it is better to record the truth as it is, and if you want to change it, to have a migration. So instead, you would say something like, Right now all of our services are using or yeah, right now all of our APIs are using fat controllers. And we don't like that. So we want to refactor. So The rule should be right now they're all, you know, fat controllers. And then there should be a separate migrations rule file that says this is the direction we're heading. We want to get to the point where we have these nice thin controllers and here's our plan to get there. And this is transitional. And once we're there, once that's the truth, now it becomes a rule. So rules have a life cycle. They're not just static.

speaker-1: That's great. Yeah. Cause was going to ask you my next question was, was a lots of people listening will be working on legacy code bases. And obviously you're, in a startup. It's probably it's, I'm sure it was engineered well from the start, but on legacy code base, you've got a bigger problem. So I was thinking, okay, this, this is, this is great for you, but this isn't going to work. But then I really liked that approach, right? Just to go, this is, this is what we've got. And, and this is where we're going and have that in a separate thing. And yeah, I think, I think there's going to be lots of interesting. ⁓ kind of challenges around, we actually end up, can we do bulk refactorings safely on older databases and start doing on older code bases and start doing that? Or do we do it incrementally now? And what's better with an agent, right? Cause we, with a human, was almost impossible to do a big bulk, you know, change kind of thing.

speaker-0: Yeah, and that's actually one thing that I found Yeah, that's one thing I found a lot of success with. That was actually one of the first things that ⁓ we did at Parento. It was it probably took about two months in total for us to do. ⁓ but ⁓ we ⁓ basically refactored the whole thing. and the code wasn't bad to start with. It was reasonable. ⁓ but it was, you know, fat controllers and things like that. Not the worst talus in in code, but Things that do make the maintainability of it harder. ⁓ and so ⁓ the first thing we did is covered the entire thing in ⁓ characterization tests. ⁓ that was, in my opinion, that's required. ⁓ because then I was starting to change things. And the only way that I really could trust that the AI wasn't breaking things is if I had those characterization tests and they continued to be green. And so once I had the characterization tests in place. Then we created this migration plan to break everything down and refactor en masse ⁓ one thing at a time. And as we were doing that, we updated ⁓ the the migration document as we were moving. So it was a living document as we were changing things and updating things. And then we also implemented each of the our newer classes ⁓ using TDD. So once we had the characterization tests. in place that allowed us to really break down and mark off sections to do T D D in order to refactor things under the hood. And so now things are ⁓ interface driven. we use ports and adapters a lot. things just simply delegate. There's not a lot of things going on in the controller. Every controller just sends a delegation to an action and the action handles everything.

speaker-1: Amazing. That's super cool. so you have actually done a refactoring, you know, using the refactoring legacy code kind of sort of principles that have been around for a while, but using the agent to actually do that. I mean, was that problematic? Did you get a lot of kind of regressions when you were doing that or did you manage to do it fairly cleanly?

speaker-0: I would say in general it was pretty clean. And the reason for that is because whenever I did the characterization tests, I hit a hundred percent coverage. ⁓ that might not be reasonable for many teams, but for the size of our application and for me feeling comfortable that as I was changing things, that little that other things were not breaking without me being aware of it. ⁓ I wanted to make sure I had a hundred percent coverage. So that gave me the confidence then to start to refactor things because at that point everything was just a test suite away. And that doesn't mean that it's always correct, right? The test suite can give you false confidence, but it did give me enough confidence to know that at least the endpoint is giving the same information it was before. It's acting the same way, it's getting the same status codes. So as long as all of that is the same, the front end should be fine. And as long as my tests pass for the things that I refactored, then that's enough evidence for me. ⁓ then it goes to a staging environment and that's whenever it gun undergoes UAT and that's whenever we really get that confirmation.

speaker-1: Amazing. And I mean, you obviously put a lot of effort into call code quality, right? You, you've even though you're using an agent and you know, they they're taking, they're doing a lot of code ⁓ quality is still really important. And I think that's an interesting question for people, right? A lot of people are, should I care about code quality as long as the functionality works at the end? Right. And if you've got all these tests saying the functionality works, why invest in goods, your structure? Why invest in kind of, yeah. Why does that still make sense?

speaker-0: Good question. ⁓ and I I would say it's the difference between launching a product and maintaining a product. the reason why most of the software can and exists is in order to encourage software that is easy to maintain and easy to change. ⁓ that's why things should be testable and modular. ⁓ and so forth. It all of that is basically about the maintaining of the code. That's what's important. So those principles help us to maintain the code. And whether an agent or a person writes it, if they don't use those principles, they can kind of paint themselves into a corner and then we have to do a lot of work to remediate that. ⁓ so in my opinion, the software canning is action is actually more important now with agents. Because the agents turn the volume up on everything. The agents amplify everything, which means if the agents aren't following software canon, we essentially have a bunch of cowboy coders out there. You know, so ⁓ by holding it to the canon, ⁓ I'm able to number one, I'm able to understand the changes as a person and I can take accountability for them and I can review them and I can own them. LM will never do that. That's a business need. ⁓ And then in addition to that, what I really care about is the ability to change things easily and quickly. So if whether I write that code or whether the agent writes it doesn't matter to me. What matters is that it's easily changeable, that it's testable and so forth. So I am gonna tell the agent to write code like that. Because if I get a page at 3 a.m., then I'm gonna have to look at that code and I'm gonna have to debug it and understand it. And Maybe I can use an agent to help, but I still own that code. And so I need to have an understanding of it going in there before ⁓ I can just start to debug and things. And that kind of closes the gap for me in terms of comprehension debt. I keep myself in certain points of the loop in the software development process, basically to ensure that I don't. End up with too much comprehension done.

speaker-1: Hmm. Amazing. Yeah. So you have the comprehension debt is something you've really put front of mind and you're absolutely prioritizing it where I think other people potentially are like, you know, giving the cognitive surrender, right? Okay. I'm just going to let the agent, it's doing a better job than me. Who am I to have to tell it what to do? Right. I think it's really interesting. And I guess it gives a bit of hope for software engineers in the future, right? It sounds like the level of software engineering you're doing with an agent is certainly no less than you were doing. Before, before LLMs, right? It's just the actual coding. I mean, what do you think the future of software engineering holds and for developers coming in now, what kind of skills do you think, do you think they would need to be able to work say on your team, right?

speaker-0: Yeah. so in terms of juniors coming into the industry, that's that's definitely, I would say, a very big deal. It's the hardest part of what's happening right now in our industry, and it's something that we have to solve. ⁓ I think this is super important. I very strongly believe that AI should not replace people, but that it should help to make people more effective. And if you replace all of your engineers with AI, Then in five or ten years, you have no seniors left. And that is a very dangerous thing to have happen in a company, let a clone let alone in an entire industry. ⁓ so for the junior, I would say that the the number one job, the number one thing is comprehension. ⁓ yes, it is easy to have that cognitive surrender. It is easy to just keep hitting yes, yes, yes, yes. ⁓ don't resist that at all costs. Have a specific plan to explicitly reject that notion. Every change it makes, look at it and understand it. If you don't understand it, look look at it again until you do. Try to figure it out yourself. Ask the agent more questions about what it's doing. ⁓ you should be able to recreate it in your mind. You should be able to understand what's happening. By the end of that change, once you get that into a PR. If somebody gives you some feedback or some comments about the PR, all of a sudden you know what you're talking about because you've done that comprehension. ⁓ you haven't just thrown something out there, a w a wall of AI slot. Instead, there's reasoning behind things. And maybe you made a bad decision in this particular case, but that's what the code review is for. And so you can talk about it and that's the way that the junior grows. ⁓ I also think that ⁓ companies need to be doing more to support juniors. ⁓ In this age of AI, charters and harnesses and the like are very important. They are a way to standardize our workflow. They're also a way to onboard. So that I think is really important that the environment is set up to be onboarding friendly for a new engineer, especially for a junior, and then to allow them to be able to integrate with their team to To be able to develop some of the skills that make you a senior. ⁓ I think a big part of this is pairing. ⁓ and pairing might be a thing of the past with agents. I know people pair with coding agents and and everybody's about productivity right now. But I do think pair programming is still worth it for a lot of reasons. and so I suggest for juniors joining a new team, ⁓ Pair with a senior with the agent. and as you're coding, make sure you're understanding everything. And pay particular attention to moments where the senior pauses and says, Mm, I don't know about that. Why? When you can answer those questions, when you can catch an agent in a lie, whenever you can have a an idea for design above and beyond the code, and have the ability to pause and reflect and challenge and think about those things, you're a senior.

speaker-1: That's great. Yeah. I love, I I spent a lot of years pairing and, just the, you know, the sharing of knowledge, the way just brings everybody up to the, to the bar of the best developer on the team, because they're all kind of working with that person. They all understand the same thing. I think the point you made there about the charter as well, right? Because normally when a new developer joins the team, there is so much unwritten knowledge on the team that yeah, maybe they'll get through pairing. Maybe they'll get through reading some old documentation that nobody really had any. kind of interesting keeping up to date because all it was, was documentation for new joiners. But now it's the, it's the bread and butter of the agent, right? So it has to be accurate. And if it's wrong, you're going to see it in the code reviews. So you iterate on it and fix it. And so, you know, this, this in the same way as a unit test is right. The unit test doesn't lie in the same way as the charter probably doesn't lie. Well, maybe it's not quite as tight, right? The coupling isn't quite as tight, but it should be, there's a reason to maintain it. Yeah. Do you do? Do you do? Yeah, go on.

speaker-0: Yeah, and I I was I was actually gonna say like th there is a little bit of a of ⁓ a parallel there with testing because there definitely is debt that happens. If if it doesn't line up, then debt can happen and it can happen a couple of different ways. It can happen where the code changes underneath the rule and the rule never gets changed. It can happen where the rule is not firing and nobody discovers that for a while.

speaker-1: It's interesting with, you know, with the Claude MD being, being up to date, right. And, obviously, like you say, if it's not in alignment with your code base, it's probably doing more harm than good. It's just confusing the agent to some extent, right. I often hear these stories and I think there was a lot when people went to Opus five and people said, I cleared my Claude MD. And suddenly, you know, the, all the avowals were, were like so much better. Right. And it's like, this thing is working without the charter. But I also wonder maybe that the charter maybe wasn't well, the Claude MD wasn't perhaps so accurate in the first place. And maybe they would have done that without Opus five, right? It's like, how do you ensure that what's in Claude MD and, your broader charter is having a good impact? Do you do any kind of evals on it or is it, it purely that you're getting the feedback from the code review?

speaker-0: Yeah, so ⁓ I would say that the the process working in the code review is definitely anecdotal evidence for it. In addition to that, we do have evals. We have evals at our harness level and at our charter and our at our charter level. ⁓ so that way we can assess things with A-B testing. So we can update a rule and run two things through that as an experiment and see if we get the same results out. ⁓ that's been really helpful. and ⁓ in general I would say that those sorts of evals are necessary. They're the they're really the tests of your harness and you need to be able to to ⁓ validate them somehow. And if your original claude file wasn't that great, ⁓ and deleting it with Opus did cause a lot of improvements, th there are s you know obvious reasons for that is in terms of so if your claude file didn't have the best content, then of course it would be working against your agent. And as the new models come out, they do get better and better. So it is a good idea to clean up or clear out your charter from time to time. But I think it's always going to be required. Models will continue to get better, but they're gonna get better and worse at different things. And no matter how good they get, they will never know your particular project, your particular domain, your particular business. That's what you need to give it. So that's what game's entry into Claude and to the charter. There's a couple of questions I ask myself. Like, would this if I put this rule in here, would it survive a process change? would it still be true if it was a double project with a different stack? Like what if we change the tech stack? Is it still true? that tells you placement of things. If something is project based, then it belongs in the charter. If something is team based or organizationally based, then that gets promoted to the harness.

speaker-1: I just wanted to tap a bit into what's the software development team going to look like in the future, right? We see there's this real skill in developing the charter. There's a real skill in reviewing the code. There's a good skill in specifying, you know, what, what the agent is going to do. Do you see those roles becoming more specialized in the future? Or is this a good, does does it make sense that you do all of the stages because they're all so interlinked?

speaker-0: Yeah, so I I do think that in general ⁓ it is going to be a cohesive, universal skill set that that ⁓ developers are going to start to move toward not building the thing, but building the thing that builds the thing. And ⁓ that's gonna be focused on the harness, it's gonna be focused on the charter. ⁓ instead of working cards directly, you will curate your charter and harness to produce better outputs such that it is the thing that knocks out your tickets. and I do see this as having a natural separation also. ⁓ I think that software developers are gonna be in a good position to ⁓ do things to the charter and to the harness, depending on their level. ⁓ and I think that the platform team is really the team that owns the harness or the multiple harnesses in a company. Because a lot of these things are really the codification of platform standards. So a company can say, we don't deploy on Fridays. Well, that can become a hard check in your harness. So that way, even if a developer tries to push, tries to develop on a Friday, it fails and then rejects it outright because it's been codified in the harness. So all of those policies that

speaker-1: Thanks

speaker-0: we're always at the platform layer, whether explicit or implicit, can be all of that can be collapsed down into one level of that particular harness.

speaker-1: That's, that's really interesting as well, because, you know, historically, like organizations probably want more standardization across their teams. It would always have made sense, but teams have their own kind of preferences and they will turn to these, they might use different frameworks or just different coding standards and things like that. But actually if the, if the agents writing the code and the, and the platform team specifies that in the charter for the, for the agents, then they'll get those standardization without having to battle. trying to persuade different teams to do different things in different ways, right? It kind of made more sense. It might make the standardization and therefore that kind of debt that you get from project A looking completely different to project B, you know, even when they were created at the same time, which you often get, right? And you're like, God. Yeah. That's a nice bonus.

speaker-0: Yeah, yeah. And and the the nice thing is that you you can have as much standardization as you'd like with that. So if you have a company that is very strict, maybe you have to deal with HIPAA, PHI, something like that, then all of those things can be ⁓ codified at the harness so that way all projects inherit them. ⁓ but also you can have ⁓ projects ⁓ override things at their own level. So there could be some policies that are locked and others that are able to be overridden. So that for two different projects with two different tech stacks, you could each define how you run tests differently. But you gotta run tests. That's required.

speaker-1: Yeah, yeah, it's perfect, isn't it? You can really decide, you know, how tight or how loose you're going to be and it will depend on the organization and the diversity across the organization. Okay, that's amazing. think, I think I've asked, you know, I've sort of covered the life cycle. We probably didn't quite get into factories as much as we, I guess we could have gone deeper into that maybe another, another time we'll do that. Is there anything else that I've missed that you think is really important at this point? Apart from your book, obviously, and feel free to plug in.

speaker-0: ⁓ yeah, I I think I think we covered quite a bit here. And I think that the the thing that I take from this as like the most important thing, what I keep in my mind, is ⁓ the amplification thesis that the agent is going to turn up the volume on whatever whatever you point it at, ⁓ whether it's good content or bad content. So the Agent doesn't change your standards, it enforces them. And the lower your standards are, the more enforcement you get. And the best way to get reliability is to create a specific reliability band that you're willing to accept. The model is non deterministic. You're never going to get the same thing out every time, but you can narrow your reliability band to something that's acceptable. Businesses already do this with risk assessment. Now we just have to do it for agents. And the way that we do that, the way that I've done that with agents in terms of my work with a with ⁓ agentic coding has essentially been to remove bad pieces and what's left over is acceptable. So if it passes tests and lints and static analysis and all those other things, then that's that's kind of good enough. And the The the reason why I believe this to be effective is because you are reducing the search space of your coding agents. ⁓ if you tell a coding agent implement such and such, it has so many ways it can do that. So many things that it could do to to to implement that. but if you also layer on that it has to be tested and that it has to be green, that it has to lint, if you add on these extra requirements, then anything that It tries that doesn't meet that bar, gets tossed away.

speaker-1: Yeah, absolutely. Just about the right constraints and constraining it in a way that it still allows it to write great code and boost functionality, but without going off onto things that are unacceptable in your organization. think it's, yeah.

speaker-0: I think another part of that as well is also the difference between feed forward and feedback. Many people put in rules to feed forward ideas, but it's also important to feed things back. That's one thing that a lot of folks ⁓ neglect. ⁓ a hook is more powerful than a rule. ⁓ and usually they work in tandem. And that is the best way for them to work. And in terms of life cycle, I see rules as moving from undocumented tribal knowledge to inferential rules to deterministic hooks, to deterministic checks in the harness. It ladders up and as you get closer and closer to that determinism, your gate gets stronger and stronger.

speaker-1: Amazing. Yeah. Yeah. I feel like this would be a point for me to plug Sonar's Vortex, which has hooks, which are running the analysis every time you change a file. But yeah, I'll hold off there. Cool. All right. That's amazing. ⁓ I think I'm done on questions. Is there anything else you want to say before we go?

speaker-0: ⁓ no, no. I think we covered everything. It was a great conversation.

speaker-1: Yeah, thank you very much, Ian. It's been, I've really enjoyed it. I'm looking forward to just going back and finishing your book because it's quite long. ⁓ Yeah, I'll get into more depth. think it's just a really important skill that is just emerging. And like you say, it's happened so quickly, right? Six months, eight months ago, nobody was really doing this, right? And it's very quickly people are discovering what works and what doesn't. And it's a super exciting space to be in as an engineer, I guess. And it's great to hear from you, how you've been working on it.

speaker-0: Yeah, yeah, thanks for having me on. It's been ⁓ great to talk about it.

speaker-1: Cool, thank you.