speaker-0: Agents use more tokens than humans for the first time in history, and probably that will be the last time that humans will ever use more than agents. So if you are building for humans, you should pivot and build for agents first. I don't think it's unreasonable to think that might happen in the next call it six or nine months. I know you probably read a ton on X about how open weight has come to demolish the closed labs. It's a little hyperbolic. If you take token share, increase it from 25% to 71% for open weight models, the spend share goes from 10 to 26. So what if you took it to 99? It wouldn't even get to half of the spend. I'm not sure that there's ever been a time in technology where there was so much planned cannibalization of your own work. This is the fastest growing market in probably the history of capitalism. There's going to be more than enough pie for a lot of people.

speaker-1: So before we start, I guess. So you're head of insights at Open Router and before you were head of insights at Carta. Yep. yeah, like what does that mean, head of insights?

speaker-0: Well, at Open Router, it's telling the story of what's going on across the six hundred trillion monthly tokens that we see. so that's everything from the easy stuff, the debates between open AI and anthropics. But the stuff that I actually really prefer is all the data that we have about one, how launches are going broadly across the ecosystem, and two, maybe even more importantly, the vast difference in inference providers that you all are probably seen and experimented with and felt over the last couple of years. I think When you talk to people outside of this city, it feels as though they only consider which model to choose. And that's about as basic as you can get, where where you're actually serving that token from is becoming more and more the hot space. That's incredibly competitive.

speaker-1: How would you how would you describe what open router is?

speaker-0: it's one API key to 500 models. But I think more expansively than that, it is the ability to flexibly interchange models in and out, either personally or in your production apps, and understand that we are going to route you the best token at the lowest price whenever possible. So we've kind of abstracted the layer, not just again on the model side, but I think on the inference side, it's even more important. And as you dip your toe into open weight and then eventually into open source, I feel like people need to a little bit holding hands on that journey though, especially if you're a CTO or CFO that is not coming from San Francisco.

speaker-1: Also, are you still considered head of insights at OpenRouter or is it head of insights at Stripe now?

speaker-0: Nay yeah, nay open router, exactly. no, we have we're super excited to get into Stripe. I think that'll happen in the next couple of weeks. finally, you know, switching out the laptops and everything. So Yeah, yeah. Yeah. Cool. Yeah We're we're fully remote though, and so you can imagine, I'm sure some of you engineers recoil at the idea of going into Stripe offices, but we're smoothing it out, I promise. It's gonna be fine.

speaker-1: So so maybe you already mentioned this, but like what kinds of analysis are you doing right now? And then I wanted to ask like once you get access to Stripe data, like what will you be able to analyze that you can't do right now? I'm just

speaker-0: Not allowed to talk about when I'm going to be able to do it Stripe, unfortunately. Okay, so the analyses that we put, how many of you have spent time on our rankings pages to date? Open Rider rankings, et cetera. Okay. that is what we consider one of, if not the most authoritative source on multimodel usage across the ecosystem. Again, it's I think this week we'll be at 180 trillion tokens across 500 models slash rankings. Either one. Yeah. And if you go to the rankings, one up on the right. there you go. Yeah. across all sorts of different attributes. So you can track these by task, cost per session, market share of any of these models, benchmarking, all the way down to context lengths, et cetera, where you're gonna see things like JEV and other decision models now pop up. So if you want to understand how a wide ecosystem of people is actually ingesting every one of these new releases, I think you can't do better than this page.

speaker-1: So one thing that I wanted to get your thoughts on was I guess last was it last week was tech week? you guys did this you guys did this talk on the state of models. Yep. what was the like what was the purpose of doing that? I think

speaker-0: That it I know it feels this way in in these sorts of talks where it feels like everyone is already swimming in the same waters. I think it's really important to take a step back and say, just six months ago, about I'd say 90% of the topics that we care about barely even mattered or weren't on the radar. And the first one, if you flip down a few slides on this, the first one that shocks people is just the way that AGENTEC token use has exploded and continues to explode over the over the remainder of the year. So if you keep going a couple more. Down down. Okay, so on February sixth of this year, at least according to our data, some people have it on different dates, agents use more tokens than humans for the first time in history. And probably that will be the last time that humans will ever use more than agents. As you can see, the gap is about eight X now and increasing basically week over week. So if you are building for humans, you should pivot and build for agents first. And the second is that all of these labs, and we, Kim EK3 coming out this today is a great example of this. The gap between what we would call the frontier closed models and the best of open source and open weight at this point in the beginning of the year was something like five to six months. I think there's actually a pretty reasonable gap. So this is percentage of a Gentec token share for each of these. Every successive capability increase, it seems as though that lag is narrowing and narrowing and narrowing. So Will the will we see open weight or even open source models come out with a capability that is far surpassing a frontier closed lab model? I don't think it's unreasonable to think that might happen in the next call it six or nine months.

speaker-1: So one question that I have about the total number of tokens is how do I contextualize how you guys fit into the overall number of tokens like that are going maybe like directly to open AI and directly to Anthropic?

speaker-0: Well, I think that you can split that into three categories. So there's one is tokens spent through subscriptions to the major model labs. Very difficult to tell how many that is. If you work in Anthropic and want to tell me, that's awesome. I would love to know. the other is tokens that are flowing through your personal devices and nobody gets to see them. So those two pools of tokens are effectively dark to us. It's really difficult to tell. On an API basis, we have done a lot of work with a ton of different institutions on measuring what this is. I think it's about one and a half percent, one percent to one and a half percent of global token flow. the vast majority of which still goes through Anthropic and OpenAI. So I know you probably read a ton on X about how open weight has come to demolish the closed labs. It's a little hyperbolic. they are seeing far, far, far, far more tokens than any one of the open weight players, but the trend is certainly towards Open weight, open source and a more diverse model ecosystem.

speaker-1: So if I go back to this graph. Yeah. So basically the number of tokens generated is like exponential right now. Do you see that trend continuing like for the next, I don't know.

speaker-0: Not only continuing but increasing. Wow. So the change, if you look at this, so half a trillion to six trillion over the period of twenty twenty five. Per week. Per week. Six trillion to one hundred and sixty-four by the end of September. And the week over week percentages continue to increase. There are some gaps in that because we host a lot of stealth models, and you all love stealth models because they're free. And in the moment that they're not free, you don't use them anymore. I don't know how good that is for the model makers, but it does seem to be true. how many of you tried Ox Alpha or Space Bunny or any of those free models on Open Router at the time? No, nobody? Couple. but this will this is by far the fewest number of tokens that we'll see. if you look back next year, you're gonna be it'll be dwarfing the number tokens. Like I I just don't see that slowing down.

speaker-1: So one other thing I think you guys have here is a graph on like the revenue. so this is like just broken down by open source and closed source yep tokens. Maybe you can talk about this. Have you guys

speaker-0: Go scroll it up one more. Across OpenRouter, which again is an specific population of people, it's in the, you know, close to 10 million individuals and and many hundreds of thousands of organizations use open router on a daily basis. But you would see that the open weight tokens are now about sixty percent of all the tokens on the platform. Depends on the day, depends on the week, but I think that that trend will probably continue. Again, if you look at the dollar spent though, so if you go down a little bit further, there is Still, it's still true that the two closed labs plus Google and a couple of the other smaller players are taking the majority of the spending from enterprises and individuals across the ecosystem. So it can be the case that Anthropic and OpenAI are quote unquote losing share in terms of tokens and are completely fine in terms of economics. those two things do not need to run together. So if you're freaked out about the IPO, we'll see what happens.

speaker-1: So one thing that Ara Karazin, he's like chief economist at Ramp. Yeah. he like posted this today basically. I want to get your thoughts on this. basically he's saying like as companies get I guess, more sophisticated with how they use these models, they actually decrease the no the amount of money that they spend. So you can see in the graph that he's showing, basically like even though tokens are increasing, they're getting better at using them. And so the cost goes down for them. and so like if we go back to This is so difficult. If we go back to this, if if that's the case, like how would you see this trend changing over time? Like would you see more revenue? I guess like the revenue to closed weight actually shrinking. Yep.

speaker-0: not by much. So the I think it's paint I think it's painting it really naively to assume that changes on a weekly basis to some of the biggest one percent of companies on their AI usage is going is indicative to the rest of the market for a couple of reasons. One, it's still very, very early. Two, and I think more importantly, the close weight models have a lot of opportunity to change the landscape if they so choose. So Luna and Haiku is the perfect examples of this. Before Luna There really was no incredibly cheap, really, really highly regarded closed weight model that effectively was at a Pareto cost efficiency with some of the open weights, Deep Seek in per in particular. And then the moment Luna came out, it really changed the token spend on a lot of places.

speaker-1: Yeah.

speaker-0: So one, discounts work. If you are making models and you want people to try them, discounting them is a great way to get them to try. Who would have guessed? Second, maybe more importantly, there was a ton of retention on these models after the discount period ended. So it wasn't just people trying it for free. You found it, you inserted it into a specific use case that worked, and then you continued on it. And so what kills token usage on this? The release of the next Luna. I think one of the problems with A lot of this analysis is I'm not sure that there's ever been a time in technology where there was so much planned cannibalization of your own work. I'm sure the people at OpenAI, the good folks at OpenAI spend blood, sweat, and tears to get Luna out the door. And then they know that they're gonna need to get a new Luna out the door in six weeks. And so that's gotta suck, Apologies to the folks at Google have to do that. but it is true that every opportunity, every one of these model increases is an opportunity for switching behavior. And I don't know if we the very first slide on this is that more models are coming out more quickly every single year. We've released on open router, I think eight hundred and sixty-two over the last two years and th and more than one a day so far in twenty twenty six. So how many of you have tried all eight hundred and sixty-two of those? Yeah, me neither. But every time something comes out, it's an opportunity to experiment, to try all new things, et cetera. So the switching moments that are that individual labs cannot control are just getting faster and faster and faster. So I expect retention to go down.

speaker-1: Alright, so I wanna get your thoughts on one other thing. So I think this is also something you guys supposed to do right? Maybe you want to talk through this? Yeah,

speaker-0: We we basically

speaker-1: we So do you see that do you see that trend incre basically continuing where the share of share of tokens going up is gonna go up?

speaker-0: you take token share, increase it from twenty five percent to seventy one percent for open weight models, the spend share goes from ten to twenty six. So effective what if you took it to ninety nine? It wouldn't even get to half of the spend. Yeah. There is I think it's again, I I I do not think that the closed weight frontier labs are in any way screwed by the fact that open weight competitors exist and are awesome. This is the fastest growing market in probably the history of capitalism. There's going to be more than enough pie for a lot of people.

speaker-1: But if you go to let's see. If you look at slide 18 here, you're basically saying over time, organizations are actually using more and more models. And then if you look at slide twenty-one, basically you're saying that companies are switching models more often, which tells me that like they view them as commodities. And so if over time they view them as commodities, they're switching them in and out, yep, then I would assume the margins go down. And also we're also seeing the share of tokens going to open source increasing. So not only is the share is their share closed source models decreasing, their margins are also gonna decrease. So is that a like do you agree with that?

speaker-0: So, most of the switching is actually from a flagship to a flagship within a family. So you would go if your daily driver's Opus five and then Opus five five comes out, you're very likely to try Opus five five. And if you find it to be useful for you use cases, you might stay in the anthropic family. Anthropic doesn't lose your API dollars in that case. So that's one. Two, and maybe more importantly, I think it is very difficult for us as humans to understand the exponential. Share battles, losing Percentage points of share does not matter in a market that's grown 33,000% over the last year and a half. Now, maybe maybe I'm completely wrong and the tokens won't continue to grow that way, but that's what I would be betting on.

speaker-1: get your thoughts on something that I feel like has been on Twitter recently quite a bit.

speaker-0: Do we call it Twitter or X? Twitter.

speaker-1: I call it

speaker-0: Okay, raise your hands if you call it Twitter still. Okay. Raise your hands if you call it X. Okay. Yeah, that's right. What do you call I appreciate you old school people. That's right. Twitter forever. Twitter forever. Come on, it's more fun.

speaker-1: What do you All right? So this is from social capital. Basically, this has been big. I feel a lot of people are saying that, hey, the top one percent of c companies, I think you brought this up, are spending most of the money on on tokens. So like the revenue quality is really bad. so I wanna see like from your vantage point, like can you actually see that, like can you see this trend in the data that you guys have access to?

speaker-0: Yeah, ours isn't quite this extreme. I think we see a different kind of swath of businesses, but it's definitely the case that the top one percent of big AI spenders are going to be driving a ton of the enterprise revenues. I don't know if it's, you know, ninety nine in one, but it's pretty close. Yeah. Okay. So what do I think about this? Yeah. power laws exist in every single industry. I'm not that worried. Yeah. Honestly. I mean Maybe again, maybe I'm way too complacent about this. And if Meta stops spending money on Anthropic, that IPO is screwed. But I really don't think that the overall growth in the use of AI is going to be stopped. Now, this has no bearing on whether or not you'd buy at the valuation that the IPO at. That's a financial question, like maybe you think they're worth a trillion or two or five hundred billion, vast differences. But the idea that this is somehow a net negative for AI as a whole, I think is is a silly thing to say.

speaker-1: Uhhuh. So one other graph that I think you guys have is on Jeff with his which is like a system one model just seeing the how many

speaker-0: you tried JEV, you have to write it with the explanation point. So I just came from a I just came from the open router offsite and Diogo at TypeSafe was kind enough to join us for a talk. That guy's awesome. So much energy. But I think this is a perfect example of an unknown unknown that just comes out of it some complete blue and changes the way you think about models. So like is JEV a model kind of, right? It's sort of a decision architecture that you can use in Conc in concert with many other models. The week after Jeb launched, we had 12 decision models on the platform. So, one, innovation diffuses really quickly. And two, maybe this opens up a further new change in the architecture. we are so early. There's going to be so many more changes, like JE that introduce new primitives or new ways to think about using models or using intelligence. so I'd really be, I'd just be kind of. It'd be so cynical to say this is the best that we have. Now let's fight over the pie.

speaker-1: Are you seeing this trend continue? 'Cause this is a month old, this data. Yeah. So it's like increasing their share.

speaker-0: Ev as a share of all decision models is still ninety five percent of it. As a share of what we would call classification requests, which is return one or binary requests, et cetera, I think it's actually up to forty percent. So yes, it has completely shifted the game. Yeah. for specific tasks, Jev has almost no equal at the moment, but it's only been a month. So maybe in the next month you'll have a lot of other different people. And I also think branding matters. some of the how many of you look at artificial analysis benchmarks or eval benchmarks, et cetera, et cetera. Okay. How many of you care about them though? Slightly, slightly less, right? They're fun to look at. They're awesome. But what are you gonna do? You're gonna put the model in your own use cases and see if it works. there is no benchmark that is not usage effectively when it boils down to it. So the best thing you can do is make usage benchmarks as easy as possible and make internal evals as easy as possible. I just think we're The human mind is not capable of reading twelve hundred benchmarks and deciding amongst them what I should be doing. I think w we're a little benchmarked out. If not bench masked.

speaker-1: Do you happen to know what the percentage of tasks are classification tasks? I feel like I saw a graph in here somewhere about that.

speaker-0: It's not a it's not a super high percentage. Okay. classification tagging like, you know, on the lower end of percent of class. But but it is an incredibly important one because it happens a lot in multi agent step tool calls. You know, yeah. So it's like it's if you if you have a long agent run, it's almost invariably going to be classifying in some way, shape or form.

speaker-1: So So this is the CEO, I think, of Hugging Face. So he basically has said that hey, for most tasks, you pretty much are doing classification. That's probably like what what you need for most cases. I wanna see if you agree or disagree. And so would you expect that share of classification tasks to increase?

speaker-0: I'm not sure it'll be ninety percent, but yes, I think that a lot of what we end up doing is t classifying things and are is is a language model the right way to do that? Often no. So but I but I think the bigger point is there have to be other places like this that exist in the current workflows that will change. and this is just the first one.

speaker-1: Cool. that was all I wanted to go over through your presentation. Is there anything else you wanted to go over in there?

speaker-0: Can you go back to the deck? So if there's one thing that I think again is a benefit to the entire community, it's just that, especially after agents became a real thing in January, February, the entire ecosystem is now optimizing themselves for agentic usage. And the first time, first where you see that is the provider cache. Rates across all of these inference providers, which was pretty crappy back in January. And now if you scroll a little bit, all of these providers have gotten massively better at cash. They're faster, the throughput is higher, the time to first token is lower, and the cash is much, much better. That's a combination of the models themselves, the inference providers, et cetera. A lot of people trying to help this.

speaker-1: But this is another view of that. So if you do it by cohort a model. Yeah.

speaker-0: Yeah. So models released in the most recent cohort had an 88% cash rate on average. Models released in the same cohort last year was 56. That is a massive improvement that is just helping you cut costs effectively. and you didn't have to do anything for it. Now, choosing between the inference providers is obviously hard. That's where we can be helpful, but it's cool to see basically if you look back 12 months ago, everything you're doing today is so much better, no matter what model you're using. So we there's just gonna be continue this tailwind. Over time.

speaker-1: Feel like I saw a graph here about the cost per token. Do you happen to know? Yeah.

speaker-0: More scroll that. So across OpenRouter, token volume is up thirty three thousand percent since the beginning of twenty twenty five, and the cost per token has fallen from seventy seven cents to fourteen.

speaker-1: So one thing that I was looking at, I think on artificial analysis is the cost per task. Cause to me, I don't care about the cost per token. It's like if the tasks are getting more expensive, then who cares what the cost per token is? Yep. and I feel like what I saw was the cost per task is basically either stagnant or it's getting more expensive. have you seen that or can you guys look at that?

speaker-0: We I I mean I I've spent a lot of time in the artificial analysis data. I don't our internal costs per task are not getting more expensive. So maybe there's a difference in d definition in that case. this I agree with you. Cost per token can be deeply influenced by the number of tokens used, all those kind things for the actual tasks that you want. I still think, especially for some saturated use cases, that will just continue to go down and down and down. But we always seem to find good uses for very, very intelligent models.

speaker-1: Yeah, another thing that could be happening is just the types of tasks that people are doing are more complex. Yes. And so yeah, that's why maybe

speaker-0: Push the ambition further. So if that wasn't possible before, it's kinda weird to compare the cost. Yeah.

speaker-1: Exactly. Yeah. Well yeah. is there anything else you wanted to go over in this?

speaker-0: That was it man. I appreciate the time. Awesome.

speaker-1: Awesome. Yeah.

speaker-0: Thanks everybody.