Isaac Askew: Welcome to Never Rewrite. I'm Isaac Askew. Jeffrey Sherman: And I'm Jeffrey Sherman, and today we're gonna talk about token-based billing is nuts. So, ⁓ on this show we are longtime fans of a guy named Jonathan Stark. He wrote a book almost 20 years ago called Hourly Billing is Nuts, where he talks about how hourly billing developers who charge hourly, ⁓ but it works for any in ⁓ industry where they charge hourly. It creates this tension where the provider and the buyer are not on the same side and it it creates this tension and you get worse results and they're more expensive and everyone is unhappy all the time. And therefore his solution in the book is you should switch to value based billing. But I thought now you know it's twenty it's June twenty twenty six and ⁓ the GitLab has switched from ⁓ monthly plans to tokens e for their billing. And I'm watching Blue Sky explode as people discover that their $70 plan now costs $4,000 or they've now spent $4,000 instead of their $70 plan. And the arguments, the anguish that is coming out, it resonated with i in my head. I'm like, ⁓ this this is exactly the same thing that Jonathan Stark thought was talking about. And so I thought we could go through the the major points that he makes. and debate whether it's exactly the same problem. Isaac Askew: Sure. Yeah, let's do it. Jeffrey Sherman: All right. So again, these are the discussion of how the billing model makes you adversaries with the provider. So if you're going to pay for tokens, you know, instead of like an all-you-can-eat plan or just a fixed price to solve a problem, ⁓ there the first problem is there is no guarantee that you're going to get anything of value from your spent. Right, where and this puts the pressure, it puts the onus on you as the requester. Like now you are responsible if you ask ⁓ whatever you're talking about. So copilot, since we're talking about GitLab or GitHub, it if you ask Copilot to do something and it does the thing and that was the wrong thing, that's a you problem. Right? Copilot's got Microsoft as the owner of Copilot has no skin in the game to interpret your request as anything other than literal. Isaac Askew: Yeah. Jeffrey Sherman: It you you ask it to do something, it does it. If it's the wrong thing, spend more money. Isaac Askew: I see where you're going with this. Jeffrey Sherman: Right. And this is the same thing that ⁓ Isaac, you've done l more consulting than I have, but if you get a customer who asks you to do something and you just do it without asking enough questions and d and pushing back and you deliver it and it's not what they want, how do they feel about that? Isaac Askew: Mm-hmm. Well, they're pretty annoyed and they're Where's the value in this? Yep. Jeffrey Sherman: Yes. A and and I've been here as well where I'm like, Well, this is the thing that you asked me to do and it does this and it solves the problem that you described to me. And they're like, But yeah, but it doesn't it's not useful. Isaac Askew: Mm-hmm. Mm-hmm. Jeffrey Sherman: ⁓ and so Isaac Askew: No, yeah. I was just taking a breath to to agree with you essentially. Jeffrey Sherman: ⁓ yeah, so to me you you have this now this you by paying for tokens, you have recreated the same dichotomy of it's no longer anyone's because when it was an all-you-can-eat plan, Microsoft had some skin in the game, at least of well, if if you ask me a question and I deliver what you want, what you've asked for but not what you want, you're gonna ask me a second question and I'm gonna have to do the request again. It's gonna cost me, Microsoft, money to redo it. So I'll Isaac Askew: Mm-hmm. Jeffrey Sherman: I am now incentivized to figure out what you really want and not just what you've asked me. ⁓ okay. Moving on to Jonathan's second point, it would be a mista i if you are selling tokens, or in this case are ours, it would be in your best interests or it would be against your best interests to optimize any of your flow. Right. One of the his big w analogies is, Well, if you bought a faster computer that lets you compile the s software faster or lets you d work faster, it would cost you money. Right. So if If as a consultant, if I bought spent two grand on a computer and it made me twenty percent faster because my code compiles faster and everything else, and therefore and I'm billing by the hour, I just cut my hours by twenty percent and I spent money to do it. And that's not in my interest. Isaac Askew: Hmm. Yep. Jeffrey Sherman: ⁓ but it's in the customers' interest, right? But ⁓ so again the same problem here with tokens is anything it now if you're paying for tokens, it's no longer in Microsoft's interest to make their tokens more efficient. Isaac Askew: Yep. So I've run into something similar. I don't know if you want to go through all the points. Are you care if I just interject? ⁓ okay. ⁓ I've gone through something similar just playing around with clo both Claude and ⁓ codex. ⁓ so I ended up spinning up I I wanted to try both of them out. So I ended up buying the plan for each and quickly ⁓ realized ⁓ like the monthly plan for each. And quickly realized ⁓ with the basic plan that you hit the limits. Jeffrey Sherman: Yeah. The interject. Isaac Askew: ⁓ quickly. And by limits, for anyone who hasn't used the monthly plans for these, ⁓ there's to begin with, there's these arbitrary limits for how much you can use the software. And it's just ⁓ here's your five hour limit. You can use a lot of compute at this time and you can set it to like max thinking or a different model that costs more or whatever, but you just got this arbitrary five hour limit depending on how much you're using. Or the weekly limits. Jeffrey Sherman: Is that a win sliding window or is it like you start it at nine so that window ends at two and then at two ⁓ five you can Isaac Askew: It starts yeah. If you max out in that five hour window, then you have to spend your own tokens that you have to buy separately to continue. Or you can just wait. ⁓ but that's changed a lot. At first, I don't remember the fifth the five hour window was I don't remember when that was introduced. But for a while there was like this fun Wild West era where what I did was I bought the twenty X plan for each, basically, which is two hundred bucks a month. Jeffrey Sherman: Mm-hmm. Ha ha ha. Isaac Askew: ⁓ because I was like, I'm gonna use this a lot. Let me see. You know, assuming I get a ton done, two hundred bucks is not that much, if you think about it. ⁓ so Jeffrey Sherman: No, it's nothing for relative to developer costs. Isaac Askew: Right, exactly. And so I just had a ton of stuff going. I remember that early on, like I don't know, maybe two months ago, ⁓ I was just like building like crazy. ⁓ you know, sp because like it felt like the spin was unlimited. That's what it felt like. Because I didn't hit any limits at the time. I could get a lot of coding done and not hit the limits at at twenty X. Jeffrey Sherman: Mm-hmm. Isaac Askew: And then it changed to where suddenly there was a little more limits, new models came out that seemed to eat up more tokens than usual or be more expensive. So your your window limit would get, you know, it was like they're A that's like they're using us as the A-B test to see just how far they can push you. Cause you don't know what they're doing or how expensive this really is in the back end. They're just saying that. It was almost like they were testing out how much it would cost you, the user. Jeffrey Sherman: Hmm. Mm-hmm. So do you feel like they shrunk the limits or the newer models just needed more tokens? Or both? Both. Isaac Askew: Yes. I'm not sure. or they just was like they just ⁓ had a button that's just like, yep, you're at your window. Because we decided, but you couldn't tell, right? There's not much insight. And there was like a there's a ton of very funny commentary on like the ⁓ clawed communities and stuff in Reddit about the newer models being terrible and eating up your window. Like you'd ask it a questionnaire or like Jeffrey Sherman: Yes. Isaac Askew: They were joking about ⁓ so I'm almost like penalizing you for thanking. Usually you say thanks, Claude, like as a joke, but like it would spin your entire limit to say thank you. You know that token's a token. Jeffrey Sherman: Yeah, I could see that. It means a token is a token. Regardless, the th it's unrelated to value. Isaac Askew: So I ended up like looking up all this stuff for like ⁓ ways to optimize cost. Because what I noticed is ⁓ when she would exhaust that limit, if I wanted to keep working, like especially because I'm like, ⁓ I'm about to go to the gym, but I want this thing to keep working autonomously. ⁓ dang, I hit my limit. ⁓ I should have thought about this different. Okay, I'll spend a little bit of extra money to put some like sp $250 worth tokens in the tank, so to speak. Jeffrey Sherman: Mm-hmm. Mm-hmm. Isaac Askew: ⁓ that way I can have it going autonomously for me while I'm at the gym because I want this thing to get done and it's still worth money to me. and that's when I noticed how much it was actually eating up if I spent $250 versus token wise, this is the hourly billing, versus the what I thought was the value. ⁓ for $200 a month, I get this incredible thing. But compared to how much it ate up of just tokens after this, you know $200. Jeffrey Sherman: Mm. Mm-hmm. Isaac Askew: limit it was qu quite different. Like the economics were completely different. And the I ran into a similar thing with Codex too, had the same twenty X plan. They they have a kind of like a similar setup with limit windows and all that kind of jazz. Jeffrey Sherman: Right. Yeah, that I mean that's one of ⁓ the h his critical points is when you when you're buying hours or tokens, you don't know how much something's gonna cost until you're done, which then makes it very hard to judge whether it was worth it. Right? If you're if you've got an all you can eat plan or if you've got a twenty X plan or whatever, you can say, Okay, this is gonna cost me two hundred dollars a month and I can get a pretty decent concept of whether or not that's gonna Isaac Askew: Right. Jeffrey Sherman: pro provide enough value to me. But if you're like I want to get this done today and I have to pay tokens, maybe it was worth $250 to me. Maybe not, but I don't even know if $250 will get the job done. Isaac Askew: Right. Right. And then when I ha when I had the open Wild West kinda ⁓ I can do anything with this and I never hit my limits, that let me keep talking and like fine tuning because I didn't really know how to engage with Claude yet. You know, what are some ways it'll screw up, what are some limits you can give it such that it gets the job done better or faster. Jeffrey Sherman: Mm-hmm. Isaac Askew: Lately I've had to almost tell it every time. Audit your response to me and it'll go, I overstated this almost every single time. Right. So like there's like this weird engagement where I have to like, can you double check what you just did? And then I'll even tell it before you even get back to me, double check. And it goes, Okay. And it still doesn't do it, so I have to keep telling it each time. So like there's like weird ways to engage with it, right? ⁓ and whenever it was two hundred dollars a month, the value based, and I didn't hit any limits, then Jeffrey Sherman: Yeah. Hehehehe Mm-hmm. Isaac Askew: That gave me complete freedom to play around and like, ⁓ that wasn't what I wanted from you. Let me reapproach that so I get the value that I'm looking for out of this. And then once it swaps to 250 in tokens that just like get exhausted quickly, I'm like, whoa, whoa, whoa. I gotta be really careful about, you know, the the genie wish here. Because it's quite expensive if I get it wrong. It costs me money to undo it. Jeffrey Sherman: Mm-hmm. Yeah. And that feels very much the same way it is if you hired a developer to work hourly of you know, you might have some idea of like, ⁓ I've got this problem, can you fix it? And they might say yes or no, but w if you're paying a developer by the hour, you might have to clarify, don't spend more than two hours on this. It's not worth it. ⁓ or yeah, or spend an hour figuring it out and then come back to me. Isaac Askew: Mm-hmm. Right. And that that Yes, you know, ⁓ if you don't set those kind of limits there, especially like I mean I've I've heard this story on Reddit, it probably wasn't true. But they were talking about how one company spent like, you know, half a million just you know, one of the develop d one of those silly developers let Claude keep running and billing spend half a million in a day. And later I w you know, I read some other thing about how that was overstated or, you know Jeffrey Sherman: Yeah. Isaac Askew: a myth or whatever. But I could see that happening. ⁓ I could see, you know, and even even with me, like ⁓ not through cloud spending, but like writing a dumb script that built Google Places. I've had it accidentally bill me 200 bucks for Google Places. ⁓ because it told me how much something cost it lied to me about how much it costs to use ⁓ this particular query for Google Places. And it was actually Google Places enterprise version because I asked for some particular field that's not available in the basic version for like if you're you know making the API request, but if you request some of those other fields, Google charges you differently. It costs more. And so it costs a lot more for that request than Claude let on to. And I'm like, ⁓ man, imagine doing this at scale. Jeffrey Sherman: Mm. ⁓ so Isaac Askew: You know, I've got a limit on my Google Clo ⁓ cloud thing because you I guess it didn't cost me any money because technically it was a I had $200 in free credit and no billing thing attached. So it just used my free credit on that. ⁓ but imagine if it wasn't there. Imagine if it's just a card that's just or I I see this a lot in Claude, like it there's an option to automatically re-up. Jeffrey Sherman: And then it stopped. Right. Isaac Askew: your credits once you get to one thing. Like imagine if you just keep exhausting it and it keeps re upping. And I'm sure some people have run into that or crazy API expensive request that Claude has written an infinite loop to keep billing you for. So yeah, it's it's really similar to an unbounded developer and you haven't told him, Okay, make sure you only spend two hours on this. Don't work overnight because AI can and bill me for all of those hours overnight. Jeffrey Sherman: Yeah. Mm-hmm. Right. Well, this goes also to something we've covered before where it's also a developer who has no concept of the costs. Where I have definitely in my career seen this of developers have no idea what something costs, and so they build a solution, and that solution is ten x more expensive than it needed to be, because they had no idea that any of this stuff was expensive. And once you tell them, ⁓ this is, you know, way too expensive, can we use these Isaac Askew: Yes. Yep. Right, right. Yep. Jeffrey Sherman: cheaper ways of to getting the exact same answer. Like, ⁓ yeah, sure. If you told if I'd known it was expensive, I wouldn't have done it this way. Isaac Askew: That's a really good point. And I've definitely run into some ⁓ at least one DevOps engineer that I've talked to and they've architected this really beautiful system. And I'm like, Well, how much does this cost? And they're like, like to to run this. And they're like, I don't know, but it's the right way to get it done. And I'm like What what? No. ⁓ the right way to get it done is how much you can afford, dude. Like it's nothing to do if it's beautifully architected, if I can't afford to run it, you know. But that wasn't that wasn't what they were getting at. They were just thinking, okay, if if we're building this kind of queue system or this or that or scaling this, they knew how to make a really well architected thing. But if they're being paid by Google, sure. You got a lot of money to work with. If you're like Jeffrey Sherman: No. Right. Isaac Askew: If you're not profitable yet and you're gonna have layoffs if you don't get something going soon, then no, that is not a good solution. ⁓ so that context is very valuable, and that's still the same thing in previous episodes. We've discussed like what will set you apart from that person who takes the acceptance criteria literally and he's like, ⁓ I got it done, but he doesn't like think about the context, the bigger context. Claude still is doing better at it, but it still needs that bigger context of ⁓ be careful about how much this costs. ⁓ and to rem remember that. It's almost like you'd want to put that in the Claude MD file if it would ever keep reading it because it ignores it half the time in my experience. ⁓ to make sure that it, you know, it stays in those guidelines or it doesn't architect something silly. Jeffrey Sherman: Mm-hmm. Ha ha ha. Yeah, like i in my day job I'm using ⁓ Devon AI a lot. And one thing I have it do, which is really good, is I say, here's a problem, search the code base for all instances of this problem, and create a epic level Jira ticket that describes the problem and points out all the you know here all the instances, and then under that epic create a subtask for each instance so that we can clean it up. And Devon does a great job at that. But I notice That in there, like you know, cause in the prompt script, it it talks about it has to load understanding of Jira and understanding of Markdown and all this stuff. And I wonder to myself, every time I do it, I'm like, I wonder how much cheaper this would be if I told us to just give me markdown files and I copy pasted it into Jira, if that would save money and be more effective. But I you know, I'm not paying for tokens. Isaac Askew: Yep. Jeffrey Sherman: And my company's not worried about tokens at the moment. So ⁓ I don't know. Like but like to me in my head, 'cause I'm an optimized, I'm like, I wonder I I wonder if this is if I'm just burning tokens here. Isaac Askew: Well Mm-hmm. And eventually people will care. ⁓ I think we we've seen some people like Uber and Microsoft already cut costs going well, Microsoft's was different because they're going, We're gonna cut Claude here because it's the beginning of the year and they're gonna cut it before they they were new and they want people to use ⁓ copilot. So they were I think they were like basically try like they were wondering why all their their developers loved Claude so much and were not using their own tool. And I think they were using that as an experiment to understand why they liked it so they can build it house and Jeffrey Sherman: Eventually they will. Mm-hmm. Theirs. Isaac Askew: I don't know if you've followed this past week where they there were just yesterday they released a new tool. But I think they're they're going the the correct way with it. ⁓ and by that I mean everyone's blowing up now with Chat GPT and Claude and then freaking out when ⁓ Deep Seek or something else comes out and undermines some of the open source software. ⁓ but I think that's what what we're gonna get at. Eventually Jeffrey Sherman: Mm-hmm. When the bill comes. ⁓ Isaac Askew: The competitors are gonna shrink. There's only gonna be a few out there. They're gonna start charging exorbitant prices or illegally price fixing in some faction fashion. And then ⁓ people are gonna go, Well, wait a minute, and they have already been doing this, I can just host it on my own Mac Studio with this open source version of Open Llama or whatever. Jeffrey Sherman: Yeah. Mm-hmm. Right, and it's not as good, but the value proposition is better. Isaac Askew: But it's essentially I mean you're just paying for the cost to run your your own local one. And it's slow and not as good, but new models come out that are better. But if you're your own company and you don't have the money to spend, or they you know this company's start charging you too much for it, or maybe they're not even charging you too much for it. Maybe actually just this entire time we've been paying the subsidized cost because they're trying to get everyone to use it, right? Probably the Jeffrey Sherman: Yeah, I think that fundamentally that's the economics of what's going on is it was all subsidized to get people to to use it and now the subsidies are come are ending and the Yeah. Isaac Askew: Yeah. And everyone's going, Well, wait a minute, what? This is supposed to replace a developer. Now it's the same cost as a developer in some cases. And and and worse context. Jeffrey Sherman: Right. Yeah, yeah. I mean that's a lot of the Yeah, and and that's I think part of the just the the thing turning on you know, developers are gonna be fine over the long term, ⁓ just the short term that that's been the problem of Yeah, but once you s you know, th this could replace a developer, but it's by the time you fully load the costs, it's as cl as much as a developer and it's not as good. Isaac Askew: Yeah. Right. And I've I've actually already seen a case of this where I was kind of like ⁓ lurking in a in a ⁓ a slack room, for lack of a better term. ⁓ and I I saw an issue and I knew what the issue was. I didn't have time to respond to it because I was just like in transit essentially. And before I could even respond, as I'm like going through I was going to the gym or doing something. Jeffrey Sherman: Mm-hmm. Mm-hmm. Isaac Askew: Whenever I finally checked the thread out, a lot of people had talked in this thread. And one person was like, Why ask Claude? And Claude says this and explained what was going on for this particular bug. And I read it and I'm like, That is the absolute wrong answer. Because I knew the system. And I'm like, that is not what you should do. And they're like, ⁓ we're gonna delete this record so this thing happens. And I'm like, Okay, well Jeffrey Sherman: No. Isaac Askew: One, I'm not part of this right now. I'm just gonna play it through. But they just came up with the wrong answer and just trusted it blindly and ran with it. And I'm like, I I expect that from people who are not tech savvy. And the people who were responding here were like seasoned engineers, and I was like really disappointed in the idea they would just run with the first thing Claude said here, because it was absolutely not correct. And I'm like, ⁓ my god, these guys. So The context is still important. Developers are still important here. Now everyone might be like, for now, it's gonna get better. And the economics are gonna get cheaper. I'm sure they will. ⁓ but you know, I s I I still if Claude was writing code and I had to rely on it to launch me into space without my ship blowing up. Jeffrey Sherman: Yes. True, it it will get b I expect it will. Isaac Askew: I would not get on that ship at all. And it might be another ten or twenty years. And even still I would be like, but we had somebody review this, right? Please don't make me get on the ship yet. Jeffrey Sherman: I would not get on that ship. Yeah, it I get terrible motion sick, I'm never getting on the ship. But I won't let my kids on the ship either. Yes. All right, so to Isaac Askew: Hm. There you go. Are there any other point yeah. J ⁓ there any other Jonathan Stark points that he equates to it? Hmm. Jeffrey Sherman: I I mean I think we got the the top ones of there's no relationship between spend and value. You don't know what anything costs until it's done. So you don't know if it's gonna be worth it. ⁓ and there's no incentive for the vendor to do things efficiently because that hurts their bottom line. The only you know, in this case, the only thing that would in both cases, the only that constrains them is the free market. Right? If they don't if they're too expensive, you'll go somewhere else. But other than that Isaac Askew: Mm-hmm. Right. Jeffrey Sherman: They should be as inefficient as possible. It's in their interest to be as inefficient as possible. Yeah. So answer the question. Isaac Askew: ⁓ well, I was about to say actually that that was I was on half a thought from earlier when was like, I think this is what Microsoft got right. Or is getting right. Not just the clock cutting costs, but like the the the incentive thing you're talking about here is Microsoft now or these other companies, they want to make sure that they understand your request right. They I mean, 'cause cause now they're not, then th then you get pissed off, right? Going, I spent how much on this? and they don't look good, right? Jeffrey Sherman: Mm-hmm. Isaac Askew: So that's when they're just spending time to make sure they understand your requests better and the models are ⁓ better. But also they're the new computers they're shipping out. They're they're trying to get them like I think they have the new ⁓ NVIDIA ⁓ GPUs or I think actually CPUs too, if I'm not mistaken, the the game like they were gonna get in the CPU game. But either way, they're partnering with Microsoft to get essentially it on your own computer, your own AI assistant on your own computer, hosted on your own computer. Jeffrey Sherman: Yeah, I saw that. Isaac Askew: Right. So I I I think, yeah, over time they understand that these models need to be really personalized to you and self-hosted to be as fast and efficient without you getting pissed off and wasting time because I'm a stupid human who can't spend time understanding how to form formulate my request perfectly to this thing so now it can help me out. Jeffrey Sherman: Right, 'cause Well, if it's running on your computer and your hardware, then it's back to the same value proposition for development that it always has, which is, hey, you've got a machine and it can compile at a certain t speed. And if you want it to be more efficient, get a better machine with more compilation. It'll compile faster. Run your models locally. You can run as much as you want because you're it's on your hardware and your stuff. You're paying for it. And you know how much electricity costs or reasonably. And so then it it it pushes it back. Isaac Askew: Right. Yep. Right. Jeffrey Sherman: It gets Microsoft and and the vendor out of the equation. Like instead of that you being on the other side of it, it's like, ⁓ well, here's some software. We're selling it to you where it's free because it's open source or something. You run it on your hardware and we are no longer part of the discussion. And therefore, you know, we're not you're not gonna be angry at us because we're not in this thing to be angry at. Isaac Askew: Right. Yep. And I think I think you'll see that across every industry too, because hosting it on your own computer or in your own like Jeffrey Sherman: Mm-hmm. Isaac Askew: People again, like back in the day, we had your own server racks, right? And then it went to the cloud. And now I think we're gonna go back to this kind of concept of, ⁓ we have the local AI here because it's HIPAA compliant, it's secure, there's privacy to worry about. We're gonna have what like at at your hospital there'll probably be a server room with an AI thing that can understand records, but it's locked down. I bet you we see an explosion of ⁓ Jeffrey Sherman: Right. Isaac Askew: Rackspace essentially. A lot of people having their own self-hosted models ⁓ because of that value ⁓ over time. Yeah. Especially for privacy and security. But yeah, so I I think that delivers the value back there. So I think that's why I I think Microsoft's being clever here is to kind of swap back the value to the to the user. Jeffrey Sherman: Mm-hmm. And there you have private clouds. Right. So we're saying that would move you away from token based billing to almost shrink wrap software, but I'm sure it would be more of a subscription based of you always get the latest models, it'll be a hundred bucks a year or something, just like Office three sixty five or whatever. Isaac Askew: Well wait helping ⁓ Mm-hmm. Yeah, and they'll probably up the cost of Windows or it's built in or maybe it's built into the Windows subscription. Like they'll find a way to monetize it, but it won't be anything a crazy like four grand a month. Jeffrey Sherman: Yeah, yeah, though it Right, 'cause it won't cost them anything when you run com questions 'cause Right. They're s they're back to sending software. Awesome. So is out is token based billing nuts? Isaac Askew: On your own computer, yep. Yes. Jeffrey Sherman: Awesome. ⁓ thank you all for listening. I'm Jeffrey Sherman. Isaac Askew: And I'm Isaac Askew, and this is Never Rewrite.