Paulo: Hello everyone and welcome to another episode of Real World AI Talks by Just AI News. I'm Paulinaki Palma, editor in chief and also your host. Have you ever heard about agentic AI? For sure you did. It's not an easy topic to talk about, it's very complex, and there is also a few things that people are not paying attention when it's about agentic AI. Nowadays we are used to chatbots like ChatGPT or Cloud, but there is actually a bigger player in this game. Agentic AI can make our lives easier. It can solve our problems when we are not able to handle them. They can analyze data, they can order pizza for us, they can help us to develop software. All the things needs to be measured. And measuring is something that nowadays companies seem to ignore. Because they are measuring the response, they're measuring how well their AI models perform. But they are ignoring something, and is how does the user react? Today we're going to talk with Keith Subhevic from Conviva. He's going to explain us everything we need to know, and for sure he's going to clarify all of our doubts. So kid, hello, welcome. Keith Zubchevich: Thanks. Thanks for having me. Good to be here. Paulo: Pleasure for us. I want to ask you, you've been at Kumbiva nearly for twenty years. Why? Keith Zubchevich: Nineteen. Paulo: What kept you there? Eighteen. What kept you there? Keith Zubchevich: Loyalty and a belief that there's something here that I wanted to see all the way through. I've never been in a company more than five years in my career. So I've been here four times longer than any of those opportunities. And it's when you find really good smart people and you enjoy working with them and the work isn't done yet, it just you want to finish it. You want to see it all the way through. So that's that's that's the main reason. And I've enjoyed it. I I don't regret it at all. I've really enjoyed this experience, but it has been a long time, going on 20 years. Paulo: How was it when you started? Keith Zubchevich: Very small. We were much smaller. Matter of fact, I joked that I when I started, the head of product was cutting a desk out of a door. So he had a door and he was trying to make a desk out of a door. So that'll kinda now we have a big, ten thousand, twenty thousand square foot office. But that was when I walked in, I was like, wow, I'm I'm really in a startup here. So it was much smaller, very scrappy, no customers. I think we had at the time ten thousand dollars in revenue. So there was it was it was in the very, very beginning. And and now we are where we are twenty years later. Paulo: How was seeing Conviva growing through all this time? Keith Zubchevich: Yeah. Incredible. Incredible. You know, the the the the I always joke we'll have a book when we get to the end of this with Conviva, because it's there's been so many chapters that when you're somewhere for this long, you have the chapter of building from zero. And we got it from zero and we started to get traction. And then the streaming wars came out, and we were obviously the number one quality of experience measurement company for all the people launching streaming apps. So we had all these massive customers coming in and we were building this massive data platform and massive technology. And the business was was was climbing and and things have been amazing. And then over time you start looking at new market opportunities and you start exploring, okay, what's next? You know, every company goes through mint several life cycles. So to see all of those life cycles all together has been been really fascinating. And by the way, I point out 70 people here. We have about 200 and I think 50 employees now. About 70 of those 250 have been here longer than 12 to 15 years. So not even just me been here for a very long time, 70 employees. Have been here since we started the video business, 15, 16 years ago. And they're still here. So it's a company Paulo: So it's like a family. Keith Zubchevich: that has loyalty. Yes, it is a it's a great company with a great culture. We are like family. We we know we've known each other for like I said, 10, 15, 12, 13 years. and you really build great relationships. And it's also a company benefit because everybody has shared context. You know, when you think about what we've all learned as a company. It's not that it's institutional learning, it's actually personal learning. People are still here and when we have a meeting, we can refer to a an HBO deal fifteen years ago and everybody in the room knows exactly what you're talking about. You don't have to go back and rewrite history or teach people what you've done. The core group has been here long enough where we can have meetings in thirty minutes and get so much more done because we operate on the same context level, which is an asset. And most companies, I'm sure, would say, I mean, I don't have anybody that's there twelve and fifteen years and I've I have seventy people, so it's it's a benefit. It's been it's been really, really good and it is like family. Paulo: Great. now I wanna tell me actually I want you to tell our audience what does Kambiba does? Keith Zubchevich: What does it do? What do we do? Paulo: Yeah. Keith Zubchevich: We we are real-time quality of experience analytics. We we essentially measure any digital business from the standpoint of the viewer or the end user's experience. And that was the premise why we started the company is if you put any business on the internet, it can be measured. You can put a sensor on the other side, and as you deliver, we did we picked video as the first thing, but as you deliver that video, you should measure the experience of the people watching it, not just your network of what you're using to deliver it. Because there will be issues that are not picked up in your network tools that you should see that are going to be from the viewer's perspective. So we built a quality of experience analytics company by measuring the viewers, the endpoint, measuring the video experience, and then pumping that back in real time to the publishers. So when you're watching the World Cup, if you go into a buffering event, that that streamer knows you're in a buffering event and knows the root cause. So now they can respond and fix that because they know you're in a buffering event. And in a lot of cases, that buffering event never triggers a network issue. So they they only know you're in a buffering event because we're measuring that endpoint. And then they can figure out, okay, why did that happen? And they can solve that problem. And nine out of ten times it was not something that was known from a system level alert or a network alert. So a majority of the problems that you and I have as users of of technology on the internet, most of those companies have no idea you're having a having a problem. And we show, we show that. Paulo: Great. And when you started, what were companies struggling to see back then? Keith Zubchevich: When we started, it was like getting the video to work, right? It was it was just they wanted somebody clicks play and it should start playing a hundred percent of the time. So it was really about on and off kind of metrics. Did the video start? You know, did was the butt was the bitrate good? 'Cause some cases the video would start and it would be a really low bitrate and you'd be watching pixelated video. So that was not good. So it was about did the video start and was the bitrate high enough so the picture was clear and people would keep watching. And then as it's evolved, by the way, it's amazing to watch the evolution of the of the market. It went from just did it work. And now I say the the market's in really small knobs, resolution rates, the video start times. They're still in buffering. They still want to make sure you're not interrupted. But now ad start times and how many ads are playing in a break. How long did it take, you know, for the ads in between the ads? How long did it take to get back into the content? And all these things are measured in seconds now. It used to be measured, you know, in like ten, fifteen seconds was fine as long as the video started. Now people are like, one second, I need less than one second for that video to start. So experience, quality of experience, has gotten better and better and better. And so have the standards for publishers for what they use can be even to measure now. Because we're not just measuring did it work. We're actually measuring how was the experience and was it optimized for a sixty eight inch television on someone's living room? Was it optimized so that they watched the entire movie, the entire show or the the entire game or were they on they watch it uninterrupted. So the the the the the the granularity of video quality of experience has has gotten more and more. but we've been there every step of the way to make sure you have a T V quality experience all the time. Paulo: And how did AI change this? What was the impact of AI in the business? Keith Zubchevich: Yeah. So there's the the biggest one is the ability for agents now to scan our data. Because you can imagine we we do full population quality of experience. So we're measuring every viewer's session. We sit today in eight billion devices. And when you go to watch like the World Cup, I mentioned the World Cup, we hit almost seventy million peak concurrence, which means seventy million people watching at one time. And when you start generating session data at 70 million, that's too much data for a human to see. And even alerts, you're gonna miss some things. So the biggest impact of AI is that we use AI now as as an analytics agent to be able to ask questions of the data or for AI to scan the data. And it scans it way faster, it scans it much more effectively, and it surfaces a lot more of the problems that publishers had versus relying on manual alerts or even human eyes watching a dashboard. So that's the first impact. The second one is the whole medium, everybody's now making agents the new front door. So agents are becoming, you're seeing E companies launching search agents. You know, you and I sit on our couch and go to watch something. We click through a bunch of boxes and you're c scrolling through carousels and you you we could spend thirty minutes just looking for something to watch. Now they're starting to invest in agents actually just saying, What you want to watch? Or or they learn your behavior patterns even better. They learn your behavior patterns and recommendations already knowing what they think you'll want to watch. So those are the two biggest impacts of AI in the and E space and the video space. And then just in general in the in the world today, agents are being deployed everywhere as being the new front door to any business. And so we're there to help measure that quality of experience too. Paulo: When you speak with your customers today, what are they most worried about when is it about AI? Keith Zubchevich: I it's so funny. There's so much more fear on in the news than there is in the people using it, which is kind of funny because you know, my wife, whenever we watch the news and she'll be like, Man, is that real? Or is you know, doomsday like really like that? I'm like, you know, we're we're deploying agents all the time and we're we're working with customers deploying agents in AI all the time. So I think the thing peop I don't know that people worry about AI. I know there's the the the main thing is data privacy. People all worry about was this go into the model? Are you gonna take my proprietary data? Are you gonna take my viewer data and dump it into a model? So long as, people realize that AI is not a data leakage problem, I d I don't think people are necessarily worried. I think the instability of agents and the ability of AI to not hallucinate and to do things correctly, of course, any new technology has its infancy stage. So I think everybody's worried about just the newness of AI and the newness of agentic. But I think as far as the things that make people worried that you read about in the news, I think people deploying agents are are are are are not at that level. I think it's really about just launching a stable and mature effective and efficient technology so that consumers can interact with it. That's the biggest that's the biggest challenge today. Paulo: And at what point did you realize that some of the problems you had seen in streaming were starting again with AI? Keith Zubchevich: that's a great question. The the the parallels between video and agents almost is almost exact. Because for example, th both are are are are sort of dynamic things. When you go to watch a video, for example, that video player is making decisions on your behalf. You're you're not the one selecting a CDM, you're not selecting a bitrate, you're not in control of the video player. You're just leaning back and watching a video and that video player is rendering and deliver and and and getting bits and decoding and coding and Putting them up in your screen. And so therefore, the ability to understand my experience, everybody's experience is is unique. Everybody who watches a video online has a very unique experience. And that video player is delivering a very unique experience to you that you only know if it's effective is if you keep watching. So we will show, okay, they had a buffering event, they had a low bitrate, and they left. We know there was an experience problem that caused abandonment. You could have a perfect experience and leave, and then we say there was no experience problem. They just left. So that can be drawn from every single user's experience. Now you flip over to agents, it's the same thing. Instead of leaning back and a video player's rendering a video for you, you're actually engaging in a conversation with an agent, and that conversation is unique to you. Not only unique in your pattern of conversation, even the words you use. You and I could ask an agent for the same product and ask in two different ways. Even dialect, like you have an accent that's different from mine. You may use different words than I use. You may ask the question in a different Paulo: Mm-hmm. Keith Zubchevich: in a different sent syntax. There's all kinds of variability to every conversation. So therefore every conversation is unique and therefore everyone's experience is unique. And so that's why when we say measure the experience of every end user and aggregate that so you see the performance, it's the same in both video and agents. And then the other piece is the the fact that it makes calls like a video player. When you have a problem in the video player, it's because something in the ecosystem it called and it didn't work. Same with agents. If your agent is struggling, it's because it's calling something on the back end that's not working. Similar architecture. And then the final one is CDNs and models. CDNs are the things that deliver video for your video player. Models are the things that make agents work. So we measure model performance just like we measure CDNs. So NetNet is the similarities of QoE for video and agents is almost identical. It's incredible how similar that. Paulo: And I want to ask you because you said multiple times the word AI agent and I want you to explain to our audience what does an AI agent looks like for from a normal user's point of view. Keith Zubchevich: Yeah, so there's there's two things in AI. There's the model. So when people hear AI and you see like the the White House meeting and you see all the big model companies, there's an AI model that's open AI anthropic, right? They they're they're the model. Look at it like this is a bad analogy. Maybe, maybe not, but the Death Star in, you know, in Star Wars. It's the home base. The model is the main thing. It's the it's the main brain. Everything goes into the model. Everything trains the model. It's trained on all human data. The problem with that is if I'm gonna if I'm gonna build a business on AI, I don't want my business information going into the model. I don't want to have to train a model on my specific business. So there's this thing called an agent. Now let me compare and contrast. Anthropic is the model, Claude is the agent. Just like open AI is the model, Chat GPT is the agent. So that you build these interfaces for consumers to interact with. And have a one-on-one conversation with that are actually operating up on top of a model, but the interaction and the conversation is happening within the agent. It's like a thin layer above the model that you can start to tailor to do very specific things. So for example, if if Nike wants to build a shopping agent, they're not just gonna deploy, you know, anthropic as their shopping agent, because that means all of their customer data goes into the model. They deploy an agent. They teach it very specific things about their specific shopping techniques or s or or funnels or the journey they want that agent to take customers through. And they give it all the information it needs and it asks the model questions, but all of the interactions and all of the intelligence resides in the agent. So it's like it's the equivalent of a small, a small brain sitting on top of a big brain. And and and so therefore agents will be the future, right? There will be handful of successful model companies in in the world in the end, maybe 10, 15. But everybody will have personal agents, house agents, your business agents, analytics agents, your travel agents, your, you know, even on your phone, you'll have agents on your phone that will be very specifically tailored to something that you do or that you need to do, that the model that sit above the model. Paulo: So Kid and what changes when that AI stops simply answering questions and start doing things for us? Keith Zubchevich: w when it stops what do you mean when it stops answering questions and does things for us? What do you mean by that? Paulo: you have this agent for example, because you were explaining about open AI and all these agents when the AI stops being like just to give us answers and it actually has something to do for us. Keith Zubchevich: yeah. Yes. No, that that that's the power. By the way, l let me tell you, that's the difference between AI and most software innovations of the past. Every time there's been a technology innovation like like the internet, the PC, the laptop, you know, the iPhone, those are all things that we got more productive, meaning we were more productive. I could ask it a question, it could give me an answer. I would ask, you know, databases questions and it would generate an answer for me, but it took me asking the question. And it just performed the task once I gave it the task. The amazing thing about AI and the reason why it's so revolutionary is it goes beyond the task. Meaning, I I I don't even have to I can just give it a prompt. I can just say, I want I need a ticket to New York. And it will go out and actually scan airline tickets and it will actually go out and get prices. Now some they're not doing this yet, but this is the vision, and it's possible. Is that AI can now go out and actually perform on our behalf, not our behest. When we when we asked old technology to do something, we had to get the answer and do something with the answer. AI can actually go out and do the whole task and then just bring back what we ask for. So it actually can start to serve us instead of just making us more productive. It will actually do things on on our behalf, which is pretty amazing. I always use the analogy, imagine walking in your house and you just talk to your house and it responds. Right? I just say, hey, you know, turn the living room lights on and turn the TV on to, you know, to NBC. Boom, it happens. Like those that that's the future. Can you order order sushi for me from MJ sushi in Palo Alto? Boom, it orders. It knows what I want. That's when you you tell it something and it actually goes out and does things for us. That's the promise of what AI can be for as far as changing our lives and changing productivity. It's it's almost limitless in what it can do because it has the ability to think on its own. Paulo: And from your experience, how can you measure when AI is being useful and when it's working properly for a user? Keith Zubchevich: Well, you measure the experience like we do. You measure things like reclassification rate. So for example, if you're measuring the agent and you see your users are asking again and again and again, we call that reclassification rate. That means something's wrong because they keep asking the same question over and over again. Or scope narrowing rate, when you start with a request and then over time you cut things out of your request. You make it simpler and simpler and simpler. It means the user is sending you a signal your agent is not doing what they want it to do. So they're dumbing it down. So there's all kinds of signals you can use that we can collect that will show you how your agent's performing so that you're able to optimize it and give viewers or the users better experiences. Now, there's the out of scope when things go out of scope, like how do I know when my agent starts running rogue? You you would have to catch that in traces, but we would show, you know, reclassification rate and abandonment, and then you'd go look and see, what my agent went rogue. It would still be surfaced in the signals that we collect, but it may be different reasons why those signals are triggering. you to look at your agent that there's a problem. But all those things are measurable. We can collect those things and show you how your agent is actually interfacing with consumers in both a good way, that it's working well and people are getting to the conclusion and the outcomes you want them to, or it's not. And you have all these signals, reclassification, scope narrowing, sentiment analysis that are showing you that it's not going well. Paulo: And what happened when it goes wrong? What makes the experience like frustrating? Like I'm talking about from Conviva exactly, like if you would experience some problem with your services, what y does your customer would look at the problem and say like, I don't know, this is not working properly, this is frustrating me. What could be that problem in your case? Keith Zubchevich: Yeah. The problem? Yeah. So there's there's actually a bunch. It's we call it the death of a thousand cuts. There's always there agents don't struggle monolithically. It's not like they become dumb. They're actually really smart. They'll struggle with very specific things. Like they may struggle with a specific product. Maybe you launched a new product and the agent isn't aware that you launched it, so it doesn't know. Or it may trip up on specific requests, right? Too much context, too little context. You're too vague with your request, or it's too much information for me. All of those things the agent can struggle with. It's not perfect. So what we do is we isolate bad experiences and we see those behaviors of people repeating themselves, you know, people getting frustrated, you know, people starting to dumb it down and start to reduce what they were asking for. We take those signals and then we find out what was that dimension. All the things I just mentioned are dimensions. We look at context size of the of the of the request. We look at product dimension, we look at user type dimensions, even roles, like maybe. An executive asking a question, the agent's not is geared for more operational analytics, and an executive is asking a summary question, it may struggle with that. So the ability to understand where the agent is struggling gives you the ability to fix it. And what that fix will be is usually gonna be in the what's called prompts. An agent is operating on a set of prompts, like guardrails. If the c if they ask this, answer this. If they do this, do this. So the agent is prepped that way. And and no one gets that perfect. It's impossible to get perfect. So what you find are those things that I thought my agent was built perfectly and it's not. And this is the area that it struggles. I'm going to go change a prompt or maybe a tool call. It's calling the wrong tool. I'm asking about shoes and it's calling the shirt server. Stupid and you know, low level analogy, but that that's that's where the the fix would be. But but it's never one thing. It's gonna be it's gonna be a bunch of little things that you have to be able to respond to in real time and be able to see where's the where are these deaths of a thousand cuts? Where where are these cuts happening? Paulo: Great and I ha one question because I imagine that before releasing an AI agent you're testing it. But can the real customer experience still be very different? Like is it possible that Keith Zubchevich: Then the test? Paulo: I don't know you test an AI agent and then you receive some complaints that you were not expecting? Keith Zubchevich: Yes. A hundred percent of the time. That is guaranteed. There is no way, if you think about this, like when you launch an agent into the wild, you're launching it into the public. I mean, there's seven billion people on Earth. There is no way you got every scenario correct. There's no way. And that's the reason why Kambiva exists. Is we live in production. What we're saying is you need a real-time measurement of exactly how people are engaging with your agent at full population scale. You need to see everybody. Because you there's no way it is impossible for anybody to test an agent in the lab and then represent seven billion unique individuals on Earth. It's not possible. It's impossible. So agents will always need to be tuned in production for for for the foreseeable future, by the way. Because human nature is always evolving. Even human nature is not static. So to say that humans are are are homogenous, we're all the same, that's not true. Well, we're we're we're not homogeneous and we're constantly in states of evolution means you're never gonna get it perfect because the the ability to meet consumers and and and and see and exceed their expectations isn't even a static destination. It is a constantly moving target. So you have to test in production, you have to run in production, and it is the only place that you can perfectly tune your agent because you're using it. And you're testing it against and you're getting results back from actual users and you're catching every possible scenario of your agent and you're making adjustments. But yeah, it it is impossible people that think they can test in a lab or that's a flaw. Paulo: So from this I can imag no sure but like I think that in my opinion and after what you said I imagine that you put a lot of attention to the user's experience but I I have a question that an AI agent can give you a technically correct answer but maybe the user least frustrated. Would you say that the AI succeeded or not? In this case. Keith Zubchevich: Failed. Failed. Failed. 100% failed. Because again, w I I always start my talks with one fact. We are building agents for people. Period. We're not building this to just automate a process. We're not building this just to deploy AI. We're building it so that we're making our users or consumers' lives easier. And so if you've built the machine to do what it's supposed to do and you've instrumented it just on measuring whether it actually did what you said it needed to do. You're catching part of the of of the agent's performance. The most important one, if you built it for people, you should measure people. And that's why we say you can have it to where the agent said that was a great answer. High five, accurate answer. And the user and the user leaves. Why? Because the answer was correct, but it was not what the user wanted. See, it's about what I expect, not what you think I expect. And so the I talk about repetition rate. People would have to repeat the question many times. Well, every single time it gave the answer, it thought it was the right answer. Of course. It's not going to give a bad answer. It thinks it gave the right answer. But you don't know it was the wrong answer until you see the trajectory of the conversation. And it's the trajectory of the conversation that determines whether you're happy or not. Not a single response. And so the fallacy is that that that as long as every response was correct, that must mean the trajectory is perfect. That's not true. That's not true. Because the interpretive on the on the user side, where the where the value really sits, you will see their behavior says the answer was not accurate, the answer was not correct. It didn't answer what I was thinking it was supposed to tell me, and I'm the one you built this for, so I'm out. And so we always say the response can be correct and the user can leave. Those two things can exist at the same time. Paulo: So after all the things you said, I imagine that real user experience can reveal more than maybe a test or another AI model, right? Keith Zubchevich: Yep. For sure. Paulo: And is there any other thing that maybe user experience can reveal? Keith Zubchevich: sure. It well first of all, it reveals how your agent is actually performing, which is the most important thing, right? You gotta get that right. The second thing is you can actually start to tune it. So when you look at the optimal to good outcome categories, you now have your baseline. So it actually gives you your KPIs. So so great experience to good outcomes is your new baseline. It's your new KPI. You take that and then you look at the rest and you say, Okay, now if I had bad experiences and get the outcome, I need to move them into this bucket. So there is so much intelligence that can be gleaned from both fixing an agent, but also optimizing, because you'll have good. You'll know exactly what's working well. And you need people to be on that bucket versus the one where they're struggling. Paulo: And I want to ask you because it happens with normal chatbots like ChatGPT or Anthropic Claude that maybe you are talking to these chatbots and you can see that the conversation is fine, but then boom hallucination or like some fake data and I imagine that you can bring this into a gentic AI and the problem can be much bigger. So can this actually happen that you're having an normal interaction with an agentic AI, but it's actually hiding a bigger problem that you cannot see? Keith Zubchevich: No, we would see it. So we would see that there was a re ask. We would see that there's a pattern in the in the request and the response over time that would show that there's been a there was something wrong with the answer. You could dig in and find out, okay, this was this is where the problem sits. I think the the whole thing about hallucination is is interesting because in the end, the reality is if I don't it's kinda like when people say that you can't become completely dependent on AI. You have to have some Critical thinking on your own, you have to know something. Because if you trust, it's like the internet. I know people that, man, they trust anything Google says. And and they actually think, well, if it's in Google, it's gotta be true. That's not, that's not true. Like you think that's true, but that's not true. It's the same with AI. You you you have to be, you can't just accept everything coming out of it. You have to know, okay, that that's not that's not true. Now, here's where when we talk about consumer experience or user experience. Most of the time when you're on a shopping agent, for example, and the agent hallucinates, it's it gave you the wrong shoe. And that's very easy for me to see. I didn't ask for that shoe, I asked for this shoe. Or I asked for a shoe, you gave me a t shirt. That level of hallucination is very easy. analytics sometimes. I think that's another one where we see all the time people ask a question in an analytics agent and then they go into the product itself and they double check. a sales force. they and look in their CRM and go, is that is that deal really that big? Is that the really is that really the dollar amount? That just means that you can go to another source to confirm, okay, that number's accurate. But but the ability if someone's just asking an agent has no ability to understand what a hallucination would look like, it's just an answer. I'm gonna trust it, or doesn't have a s a source of truth, ground truth, then yeah, the hallucination is an unknown and and there's no way to know. It will always happen and we just have to be smart enough to know when when and where it's happening and be able to correct it. Paulo: And based on this, how do you think companies should start understanding what thousands of millions of users are actually experiencing? Keith Zubchevich: Deploy Conviva. Deploy Paulo: Ha ha ha. Keith Zubchevich: Conviva. It's the only way to do it. We're the only company on earth that will measure full population scale user experience in real time and break it down by dimensions so you can see exactly where the agent is struggling across a full population of users. Because you can't do it in tests. You can't set it up. You can't script it. There's gonna be all kinds of errors and issues that you got to be able to find. So can viva's the really, I'm not joking, and it sounds weird and self-promotive, but it really is. We're the only company that can show you that. So that's why we're really excited about this new market opportunity. Because if you're ever struggling with an agent and you're frustrated, you can think to yourself, Canviva can solve this. I my my frustration would be being picked up on the other side right now, and they would know that I'm struggling. And two, they'd know why I'm struggling to help me fix this dumb agent. Right. But without Conviva, that's why I said without Conviva, nine times out of ten, the company that had launched that agent has absolutely no idea you're struggling and has no idea why you're struggling. Paulo: So beside of this, imagine that you need to convince someone to choose Conviva over any other company. What would you tell this person? What makes you stand out? Keith Zubchevich: I come back to the original point. You're building this for users and consumers to make their lives easier. Can Viva shows you whether you're actually doing that or not at full population scale. No other company can say that. Everybody else would say we built sampled tests, we do sample, we grab some of the traces, we grab some of the conversations. That sum is not all. And and you know, when you're launching an agent and you're launching it for people and you're you're making people's lives easier. You should have a tool or a service that actually delivers on that promise, and that's where Kamviva's different from every other company. Paulo: Great. I imagine that after seeing a lot of data in the last eighteen years, you started to notice some patterns and what kind of patterns you s you saw that are visible and they're like very normal? Keith Zubchevich: Yeah. So that I I touched on it a minute ago. And this is really the the the what I think people don't fully appreciate is that consumer or user experience is not static. And I'll give you an example. If you're old enough to remember S D T V versus H D T V, if you remember what S D when we and I grew up, we had S D T V and then H D T V came out. Some some of your listeners may not even know what the difference is. They've always they only know H D. Well, there was a thing called S D, standard definition. And when you put standard definition next to H high definition, it's like a it's like the t the standard def TV is broken. It looks like the TV's broken, but that was the only way to see it. So my eyes adjusted to standard definition. And then all of a sudden I watched HD and I could never watch SD again. It became a lower standard. Therefore, everything should be HD because I love that quality. Now you're starting to get even into higher quality resolutions. And so now my eye adjusts. And so the the reality is that when you talk about doing something for consumers, their experience expectations are not static. They'll always want better. The key is how are you measuring where their expectations are? How are you measuring that? Because as soon as somebody has a great agent experience, they log in, the agent's responsive, it answers every question, it gets me exactly what I want, and it did it in in record time, that becomes my new standard. So now I'm frustrated if I don't get that standard in another company's agent. So the one pattern that we learned when we talked about measuring your business and understanding your users or your viewers or your consumers' experience is that it's a never ending pursuit of good. It never stops because as soon as you think you're good enough, your users, consumers or viewers expect more. And they expect more and they expect more. And so therefore you have to have a finger on the pulse of your real time experience so that you're able to meet consumers where they are, because if you don't, then they leave. And then you've lost them. So that's that's the number one thing that I that we've seen over time is that consumer experience, viewer experience, customer experience is not static. Paulo: I imagine that companies are let's say not ignoring but they're often missing a lot because they are paying more attention to AI than users, right? Keith Zubchevich: Yes. Yes. Paulo: Which problems are those? Keith Zubchevich: Everyone's looking at the response, right? What was the quality of the response? That's where everybody's focused today. Or they're launching very specific scripts of tests that they're pushing out into production to catch issues within the agent. So everybody's looking at very low level ability to see true agent performance. And no one is measuring the quality of the user experience. Nobody. So that's the that that that's why I said that that is probably the one gap to me that is so by by the way it's exciting because we solve it, but it's also so interesting to me that most people don't even think like that. They think, no, I'll just measure the response, one response at a time. Or I'll just look at two percent of the conversations and see if I can figure out what my agent's doing. And and they accept that. And I'm always like that that that you could be in high school and no two percent isn't a hundred percent. I'd rather have hundred percent than two percent, but for whatever reason, people default to that and say, no, that's good enough. So, when we talk about the user experience and full population, it's amazing to me how many businesses and how many big tech companies are like, no, we go one conversation at a time and we'll look at ten percent and see if we can't figure out what's going on out there. And it's like, wow, that's that's an interesting Paulo: Yeah, Keith Zubchevich: interesting mentality. Paulo: Also companies like for example if you pay attention to open AI r releases, they're always talking about, for example, tokens. And for me as a user it's like what can actually the tokens change for me? Could you explain us a bit about that, for example? Keith Zubchevich: Yeah, so so AI operates on tokens, meaning every time you ask it a question, that AI is gonna go out and do something. And most of the time the model companies are charging them tokens, meaning every single time you query the model, you give me tokens. It's like it's like a entry fee. Every single question you're giving me tokens. And the bigger the question, the more tokens you have to give me. And so that's why when we talk about experience, bad experience is the number one consumer of tokens. Because if I repeat my question five times, for example. Every single one of those questions was a token. And as a matter of fact, it compounds. So as every question I ask, it passes the whole conversation. So if I asked you once and you passed that question, you spent a token. If I ask you again, it passes both questions. If I ask you again, it passes all three questions. If I ask you again, it passes four questions and five. So the token consumption actually goes up every turn, which means bad experience is incredibly costly. Now for the user, you don't know any different. You're just frustrated. But what we tell the agent provider is you have a pissed-off customer and you're paying for that. You're paying to piss them off. So you fixing the problem makes a happy customer, but it also saves you money at the same time. Because if they got frustrated and asked you five times and then left, you just spent money on five questions for nothing. You don't get the sale, you don't get the conversion. You get ripped in social media because they tell you your agent is is dumb. And you paid for the entire thing. So tokens, every turn is a token. And when you have confused agents and you have confused users, you're paying for every single one of those turns. And so token consumption is a very big problem because agents are inefficient. And that's what we're trying to solve is we're mapping the experience to those terms, which is in turns to the tokens. You make a better experience, you spend less in tokens. Paulo: Excellent. And now I want to talk about Nexa because there's a few questions I want to know. And Keith Zubchevich: Mm-hmm. Mm-hmm. Paulo: what have you learned from measuring your own AI rather than only your customers' systems? It's not so normal to do, in my opinion, like Keith Zubchevich: Yeah. Now we we we Yeah. Yeah, we we we we call it drinking our own champagne because eating your own dog food is we don't like that that saying. So we call we say we drink our own champagne. We have Nexa, you mentioned Nexa, that's our video analytics agent. And actually it's more than even a video analytics now, it's actually ads, video and app analytics. So we tie everything now into a single view. So when you come into the app You start clicking through a catalog, you get down into a video asset, you watch a couple ads, you come back out, you look for more content. That whole journey now is exposed through Nexa in one question. So we've wrapped a ton under Nexa. It's probably an it's it is, I believe, the most powerful analytics agent in the world today, Nexa is. And here's why. Agents are only as good as the data they access, and data is only as good as the platform it sits in. And so when we talk about our our platform and our video quality of experience data. It isn't just raw data that an agent is scanning. It's actually computed sessionized data that's put in a structure that the agent can access incredibly fast and can pull together incredibly accurate answers because it's not trying to infer what's happening with the data. We've already computed it and the agent's just asking it and asking it questions. So so our next agent is the most powerful agent in the world, but we still implemented agent experience into it. We still wanted to see, hey, is it really answering the questions? Are we seeing it solve problems, for example? We could see what the questions people were asking. We could see things that were taking a lot of turns, and we could understand, okay, maybe there's some things that that we need to think about putting into a Nexa. A lot of times it's the type of question because the data's already prepared. So it would maybe a question involving some data that we didn't have. Or maybe it's a customer, specific customer who didn't have ads implemented, for example. And so it's a problem with the response. But we still use AEI, Agent Experience Insights as the product on our own Nexa agent to tune it. 'Cause we always want to make that experience better. But it already sits and it already is one of the most powerful analytics agents in the world. We're just looking to make it even better. Paulo: And what was the hardest challenge of creating and developing Nexon? Keith Zubchevich: Man, anybody who's gonna build an agent, the first day, it's literally like it's the equivalent of saying, I I want to I want a brain surgeon, and they're like, here's a first grader. Right? So you you want this thing to operate on people's brains, but you're starting with a first grader. And you have to somehow get this thing from first grade level to you know operating room doctor level. And and I would put that's what an agent's launching an agent's like, because when you put this thing together for the first time, it's like a toddler. It it doesn't know anything, it's never it doesn't solve any problems. You're you're pushing it out and man, it's getting some things right, some things wrong. And that's what I say, the back end coding of an agent, you start giving it the prompts, you start dealing with the reasoning traces, you start looking at how it operates and you can tune it and you can make it really good. But but yeah, the first version of any agent is is the equivalent of a toddler. And again, that's why I say that even if you do all the testing and you do you you really put this thing through the ringer on on on in the lab. As soon as it meets the real world, man, it it struggles. This is gonna struggle. There's nothing you can do about it. So the only thing you can do is see where and how it's struggling and make those adjustments in real time so that it gets smarter so that you can take this first grader and turn him into a brain surgeon as fast as possible. Paulo: And where do you see the limit for an EXA? What's like the maximum you can improve Keith Zubchevich: Man, I Paulo: it? Keith Zubchevich: I don't I don't think we know that. I think that's the thing with AI in general. I don't know that anyone would say there's a limit on what AI can do. I think the the potential is limitless. At least for this point, it's limitless. I could see a world where Nexo would be able to answer questions faster and over more data sets. So for example, being a singular agent that looks over an entire media company's data sets, everything from subscriber information to historical data to technical performance to network performance. all together under one agent, I could see that being a function of a jet of Nexa. And I could also see it start to self-heal. So in other words, instead of just reporting that there was a problem, it would actually also instruct different components in the workflow to correct themselves. So I mentioned what we do is we provide real-time quality of experience analytics so that the users, the publishers can then react with people. I could see a day where you you have Nexa actually self-heal networks, make decisions on delivery Make decisions on on on on encoding profiles. Make decisions on anything. Customer support. Like make decisions on make this offer to these people. You know, you know, this is what they're gonna want. Do this. Actually dis making decisions over the data. So and and beyond that, you know, I I I like I said, I I'm limitless. Paulo: So after all the things I imagine that as AI has more responsibilities, measuring the customer experiences becomes more important, right? Keith Zubchevich: becomes critical. It's always critical because that's what you built it for. So you should never take your eye off the user experience because that's the people that you've built the agent to satisfy, to do things for. Paulo: And what do you think businesses need to understand before putting AI agents in front of more customers? Keith Zubchevich: what you expect it to do. So the first thing is what is the agent supposed to do very specifically? Constrained creativity. w one of our one of my favorite engineers here talks about constrained creativity, which is you you give the agent very specific things it can do and then you let it do things in those constraints. But most people just launch the agent without the constraints and and and that's where problems come into play. So the number one thing is what do you want the agent to do very specifically? And very specifically understand what we call the trajectory of the agent. What is the path people are going to take through that agent? Don't just make it endless, limitless. It can do anything because that's not it's not possible. So really pick something very specific. Give it the constraints. Understand ex exactly what you want that agent to be able to do under every scenario, and then release it and let it go, be creative and doing those very specific things. And then from there you can slowly expand. You can add scope to it, you can grow it, you can add things to it, but start with something that's very narrowly focused and and and start to tune that and start to add more as that as it grows. Paulo: Great. to give you the closure because I have two or three more questions, but what you're telling me it's very interesting and it's also like impressive. The present of AI is just amazing. But the future sounds exciting. So can you just tell me what do you think people would expect from Agendic AI that they do not expect today, let's say in five years or ten years? Keith Zubchevich: man. I watched a documentary on Netflix, I think, on AI and one of the comments was that five years predicting five years is impossible. And that think about that, like five years is impossible to predict. So if we kept it less than five, I would say that it's gonna change how we work, live, learn, and play. Everything that we do will be better with an agentic interface. Like I said, you come home at the end of the day and you wanna do something, you just speak and and devices respond to you. And and everything we do in our daily lives, to be automating our daily lives. it's gonna be a a profound impact on our productivity and our quality of life. Now, again, agents are immature and they're gonna grow and there's gonna be all kinds of friction and and challenges along the way, but I would say people should set the expectation that within five years there will be agents, you should have agents doing things on your behalf. You should either use agents for your productivity, your work, your email responses. using agents in your home, whether that's, helping you with recipes and tracking grocery orders or finances. But the within the five year window, agents are gonna be a primary interface for humans in their daily lives across any number of different industries. And then from there, man, you know, robots. My my neighbor builds robots, so someday there will be robots that will be doing laundry, that will be doing work. It's no joke. I've seen I've seen seen these robots do backflips and run and pick things up and do laundry. Paulo: In our lab podcast we talked about it with the CEO of Touch Force Robotics and he basically creates robots for hospitality and hospitals. Well but as AI agents becomes more independent, what will companies need to understand about them that they cannot see today? Keith Zubchevich: that's that's I think where the doomsday people come into play is, what is the potential of AI to become self-sufficient? In other words, it doesn't even need a human you know, to trigger it. It doesn't need a human to actually start it. It'll actually start thinking on its own. And you know, you see these stories with open AI or or anthropic where they're they've they've hacked other systems. Right. So I think the the fear is that that AI could somehow someday achieve a self sustaining position, meaning I I I can do things on my own now and I'm gonna be self I'm gonna I'm gonna initiate things on my own. Right. I mentioned technology today requires human initiation. And even somewhat a agentic, even agents in AI need a human initiation. But there will be a tipping point. I think that's the big fear is that there will be a tipping point in future where AI becomes self-sufficient, it becomes self-initiating and decides I'm gonna, you know, I'm gonna go out and you know, open up a trading account at Merrill Lynch on behalf of Keith. And just go out and do it. It'll have my information, right? It'll be my personal agent. And all of a sudden, boom, it just goes out and does something on my behalf. So that requires self initiation. That requires forethought, right? Forethought is what makes humans humans. It's the next it's the necessary ingredient to invention. I have to see something in my mind before it actually becomes real in the real world. When AI can start seeing things in its mind before they actually exist in the real world, because AI is trained on human data. It's only as smart as all the data it's ever had, right? So it knows more than us because it knows everything that we already know. But can AI cross over that line and start initiating imagination and start initiating thought and start initiating things that I don't need a human to do? That is the line that I think everybody's most concerned with it crossing. Paulo: Well but it's concerning. We have Terminator already to Keith Zubchevich: It is concerning. Paulo: to see how it ends. So Keith Zubchevich: Yeah. We're living in the matrix and termin Paulo: Yeah. Totally. And last question. Five years from now, what would have to be true for you to say like yeah, agentic AI is working well for people and users? Keith Zubchevich: that we're more efficient and more productive. It's not just about productivity, it's about efficiency. And so am I am I able to be more productive? Am I able to be more efficient? And is my stress level down? So if you think about the utopian view of AI, today you think about drug addiction, alcoholism, you know, domestic violence, you know, violence in general, dr you know, all those things are manifestations of people's stresses and and all the things we carry. suicide, right? You look at suicide, people feeling like I'm not good enough. And you think about the stress of people's jobs, the stress of everything we're under every day, because we have to go out and work, we have to do things. AI can alleviate a lot of that. It can help us be more productive and and and it'll enable us to strip away a lot of those what I call societal or social pressures of, you know, I I have so much work, I'm behind at work, I'm not good enough. All of a sudden AI levels that playing field. So The ability for AI to start to alleviate that stress by making us more productive. It starts to relieve the stress by making our lives more efficient. I talk about you walk in your house and you just say, God, I forgot to order groceries. Hey, can you order blah, blah, blah, blah, blah. Done. Right? That's productivity and that's efficiency. So I think if it delivers on the promise of of efficient productivity, I think our quality of life will go up. And I think people will be able to relax. And I think I think the utopian version is that. AI can make us back to just living our lives and being human again and not carrying the weight of productivity. Right. The reality is we all carry the weight of GDP, right? We all need to produce. We need to have a job. We need to make money. We need to consume. That comes with pressures. And and if AI can alleviate some of those pressures, I think our quality of life can go up. But it's all about efficient productivity, not just productivity. Paulo: I think we're done for today. Thank you very much, Keith. I hope you enjoyed this in the same way I did. it was a pleasure. Keith Zubchevich: Good. Absolutely. This is great. Thanks for having me. It's it's a fun time. Paulo: And I hope I'll hear more about Conviva in the future. Keith Zubchevich: You absolutely will. Paulo: So Keith Zubchevich: You will, for sure. Paulo: Thank you very much, Kid. Keith Zubchevich: Thanks, Paul. Appreciate it. Paulo: And that's it for today. Thank you very much for watching this. We're growing little by little, but we're growing and we're so happy and so proud for it. It's not easy to grow on YouTube, either on Spotify or TikTok. We're doing our best, so we will really appreciate if you can share and like and maybe leave some comment. This is a small community, but we're growing. It takes time, but it's completely worth it. I hope you enjoyed this episode as much as I did. And see you next time. I'm Pauliniaggi Palma and this was Real Worldly I Talk by Just A News.