speaker-0: It's not going to be erased to the bottom, it's going be erased to the top. The minute that someone is 10x more productive than you, you're going to have to just step up your game. You can create millions of these ads and every single person can get a completely different ad. Like that's actually possible now and that would have never been possible in the systems of the past. You can go to IKEA and get a bookshelf. You could also go to a master carpenter who builds artisan furniture. If you rewind hundred years, everyone just went to their local artisan. I think AI Slop is the counter balance to the argument that all these jobs are going to go away. It's not even about slop, right? Because slop implies that it's just bad. It could be like objectively good. Yeah. The problem is, is it's not differentiated. You use things like chat, GPT. It's like intended to make you really excited, happy. That's your thought partner who's has like literally zero critical thinking skills. Then like you're just going to become duller over time and not sharper.

speaker-1: Today's episode with Supriya Gupta is really interesting and is close to my heart because she was a VP of product at Injuwit Credit Karma while I was still a PM there. Not only that, but she was also leading the Gen.ai team when they were just getting that off the ground. She was also a product lead on the ads team at Meta when they were just scaling that business like crazy, so she brings a really unique perspective on everything going on in Gen.ai right now. In this episode, we talk about everything from how ads are solely going to be generated on the spot per user so that the image, the copy, the video, all of it's going to be unique. We also talk about how even though you can theoretically infinitely scale ad testing that in practice that's not actually possible and much, much more. You don't want to miss it. Let's get right into it. Everyone's talking about like AI right now. I guess like more specifically, Gen.ai, but like a bunch of tech companies have been using AI for like forever. And I kind of want to know like, what is the transition between what those tech companies were doing then and kind of like what's happening now? So could you just start with like, when you were at Facebook, you were doing a bunch of AI stuff. Like what were you doing there?

speaker-0: So when I was at Facebook, I was working in their ad tech ecosystem and I was helping them with a variety of products around optimizing showing the right ad to the right person. And then eventually evolved into the right ad creatives. So, you you could think of it like the right types of ad assets, like copy, images, et cetera. That was what dynamic ads was. The right mix of those assets for the creative. for the person and that's as far as optimization really got before Gen.ai came into place. One ⁓ of the big transition points you can think about is AI before generative AI was really focused on optimizing based on numbers and now we're optimizing based on content. like actual language and that kind of thing. so that's... That's why you see such a wide variety of different types of products. That's what GEN.E.I. opened up and enabled us to go do.

speaker-1: Yeah. So you worked at Meadow, which is obviously the biggest ads company ever at that time. Like you're saying you're doing more like classical ML and now Gen.ai is like a very different kind of AI. But I think what's really interesting is now you have these insane image and video models. You have like VO3 for making videos. You have Sora. I've been watching these Sora videos on YouTube. They're so crazy. Have you seen some of these? These shorts where I get these random shorts where it's like.

speaker-0: ⁓ yeah!

speaker-1: Einstein boxing Muhammad Ali or like Stephen Hawking just like on a skate ramp. I don't know if you've seen those, but they're insanely good. But anyway, the fact that you can, and they do, they perform really well. Like they get like tens of thousands of likes. So obviously like they're really good. So now that you can generate content like this, you can generate videos, you can generate images. I think I saw that. I think this makes sense. Like as a long-term vision for Facebook, which is if you're a business, you just upload your products and it'll do everything for you. It'll target for you. It'll generate ads for you. Whatever. What do you think those teams are doing right now at Facebook or like what what are they trying to do or do you think they're using these models?

speaker-0: Yeah, okay. So just to put a little vocabulary in terms of like the before and after, there's predictive AI and generative AI. And I think predictive AI is still being used. It's not like it's gone. It's not like it's like just randomly disappeared, right? You still need classic recommendation systems to serve things up to people and things like that. think that actually, let's just take dynamic ads as an example. So back then when I worked at Facebook, that was when dynamic ads first came out. And the idea of that was really like, hey, let's figure out what to show, like which ad to show you. And then I'd have maybe some decisioning on which element of the ad to show you as well. Like do I pick this picture or this other picture to show you based on numeric funnel data on what makes sense to show you, right?

speaker-1: And that would be basically like an A-B test. Like it sends it to two different people and say, okay, what's the conversion rate on this one, conversion rate on this one. Okay, this one's doing better. So let's focus on this one basically.

speaker-0: Kind of, it would be like, okay, I'm gonna flight, I'm just gonna make up random stuff here, but it's like, ⁓ I'm gonna flight like, you know, 10 different headlines. And I realized that men in San Francisco really resonate with headline A and, you know, women in Phoenix, Arizona really resonate with headline B. And so it'll find its own local maximas of like, okay, I'm gonna serve up the headline A for these types of people and headline B for these other types of people. And so. It works something like this, but the base point is that it's not like the models are understanding the specifics of the headline. They just know that asset A versus asset B, numerically they tend to perform in these areas really well. What's changed now with generative AIs is actually has some level of understanding of what the headline is because it's language, right? So it's able to take that language, turn that into some sort of vector that it can understand. and like work with that asset, similar thing with video or imagery, where it's able to like somewhat quantify what that actual asset actually is versus just treating it like a black box. And so that's the difference that exists today with generative AI. So now if you have that rich level of information, what could you do with that? And this is where I think the next gen of dynamic ads, just like you described is going to go, right? It's like, well, if I know what you're selling, who you are as a business, et cetera, why can't I just generate ads for you on the fly? Okay. If I have all the context, I should be able to figure out the perfect ad for Basel. I should figure out the perfect ad for Supriya. Why not? Right. It doesn't. And then you can create millions of these ads and every single person could get a completely different ad. Like that's actually possible now and that would have never been possible in the systems of the past. I think that's the big delta.

speaker-1: What do you think is going to happen with all these content agencies? Because obviously that's a huge thing for all these different e-commerce companies or whatever. They go to these agencies, they make all of the content. Does that mean that you should not be in that business anymore?

speaker-0: I think it will be, okay, I'm gonna answer that question with a little bit of an analogy. So like, you can go to IKEA and get a bookshelf and it'll cost you like a hundred bucks. And bookshelf will be pretty good, you know, to hold your books. It might show some wear and tear in a few years. It's fine. It works for most dorm rooms. It's great. You could also go to a, you know, a master carpenter who builds artisan. furniture out of solid wood and create something extremely specific and bespoke for you with their 40 years of experience and all these things, right? And they'll create this great bookshelf that'll last you a lifetime. And so what I think is going to happen is there's an IKEA option, there's a target option, there's a what's mid-range, West Elm option, there's an RH option, and now there's a, you know, and then there will continue to be the artisans. But like, if you rewind, you know, 100 years, everyone just went to their local artisan to get their bookshelf, like there was no IKEA. I think you're going to see something like this with digital media too, right? There will be like, like the farms that create the things where I'm okay to give up that last mile of polish that a human could create with human judgment on top of AI in order to get get scale and I'm okay to sacrifice quality. And then you're going to have like varying layers of human judgment involvement on top of that, which will then limit scale, but add quality and people are going to have different preferences on what they need depending on the type of business they are. So it's a long way of saying like, I don't think they're going to disappear. I think they will have to reinvent themselves into different categories and pattern match to the needs of tomorrow versus today and really focus on like, what is that last mile polish? And so I think, I think that it's going to evolve. do think, I do think it has to shrink because in some cases using a tool is just going to make more fiscal sense because, because there isn't willingness to pay for quality beyond what a tool can provide.

speaker-1: Yeah. guess like another argument that I've been hearing people say is, well, if we can just generate the ads, who am I to say what is good and what's not? Why not generate a million ads and just see what works and just test them all against each other versus, you know, having a human, you know, add that polish.

speaker-0: So I think the thing is like, I could deliver a personalized ad to you, I could deliver a personalized ad to me. I don't think that that's an A-B test. I think that's actually just like a way of doing things. What you cannot necessarily test effectively is a million different variants of an ad. The reason why you can't do this is because you're still beholden to the laws of physics that govern predictive recommendation engines, right? Because I need to see enough Supriyas to know which ad is gonna work well for Supriya. And so at some point you do get limited by that. So what I actually am seeing in market, when I talk to other, clients and customers and things, what they're doing is they're actually leveraging AI to come up with say like a thousand variants of an ad. And then they're picking their favorite 10 or 20 or whatever based on their intuition. And then, and they're sending that off to their agency to go flight or do whatever they need to go do. I'm seeing that as a way to like take advantage of the volume. and scale of AI without actually running into this like testing barrier. So anyway, I just thought I'd throw that out there as a counter example for that theory of going nuts.

speaker-1: thing to think about. Like, I guess my other question was going to be along those lines on like AI slop. It's like now that you can generate all these videos for free, obviously there's going to be tons and tons more content. And basically it's just going to be the TikTok algo in my mind everywhere. It's just going to be whatever content is getting engagement, just surface that. So it's going to be like this giant like power love. There's like a couple of videos that do really well. Everything else is just like going to get no traction. So what do you think about like AI slop?

speaker-0: I think AI slop is the counter balance to the argument that all these jobs are going to go away. Like because, because low quality stuff is going to produce a lot of slop and it's not even about slop, right? Slop is one dimension to think about because slop implies that it's just bad. could be like objectively good. Yeah. The problem is, is it's not differentiated. And so when you think about advertising and marketing, inherently it has to be differentiated. That's where humans can add a layer of like serendipity and judgment and taste to actually create the differentiation. That's where I think the willingness to pay is going to start to come in to create that separation between the two. for some reason, sorry, for some people, that slop is actually going to hit a quality bar that is actually good enough for them. like, there is that reality of it'll probably be fine for certain people because it's fine. And if you don't have to overdo it and that will actually create economic expansion for companies is actually a good thing. So I'm not anti using AI and it's all bad because that that's actually ridiculous. I, and I do think that autonomous AI in certain circumstances will work for certain types of businesses. But I do think that this differentiation trap is actually going to be. the thing that's going to make people think twice about fully automated version of every single asset for themselves, that I feel like it's a harder reality for me to wrap my head around that that's going to really work that way.

speaker-1: Yeah, that makes sense. The other thing that I keep thinking about with all these companies that are saying, ⁓ we're going to totally automate advertising is I think before you used to have these super productionized ads that were super high quality and they're like in a studio that that sort of thing. And then things basically move towards like UGC. So now you see all these ecommerce businesses basically just having sending up like just telling somebody, hey, just record something on your phone. And a lot of the times that phone recording will have a higher conversion rate than the super high end, like super expensive production thing that people would run. So if everything starts looking the same, everyone can't tell like, ⁓ are these just like AI videos? I don't know. You know, there's got to be something that changes. There's got to be something where it's like, I don't know, something's got to change for somebody to be like, okay, well, this is a real video. Cause I think at some point what's going to happen is everyone's going to be like, I can't tell if these videos are just fake. They're just like, whatever. So I don't know if the companies are going to start mandating that if your video is AI generated that you tag it with AI and that will reduce conversion rate on those videos and so you'll have to do real videos or what's gonna happen there? Do you have thoughts on that? Do you think they're gonna mandate, hey, if this is made from AI, you have to tag it as AI?

speaker-0: I think that in certain circumstances, the reality is that it doesn't matter if it's AI generated, if it's entertaining. So, and if it's entertaining, it elicits an emotional response. The emotional response creates a connection between you and your brain and your heart of hearts with that brand, right? Like that's how advertising works. And in some cases, like it doesn't really matter if it's an AI generated avatar human person model thing that that's now a trend or it's generally an, like just, just a AI generated video, if it works. Think, I think there's like the dimension of like, is it differentiated? So it actually stands out and it works or it doesn't work because you decided to just create something at the mean. And I see that as the bigger Delta in terms of like distribution. And then I think the other thing, your point on UGC is also, I think a societal. reaction and backlash to the fact that non differentiated AI slop is just all over the place. So then you're like, well, all this is like BS. So it should be down ranked because I don't want to see a bunch of this generic garbage. It's doing nothing for me. I am not connected to this brand. It's just cluttering up my feed and actually. from a very strong practical business sense, the TikToks, the Facebooks, the Instagrams of the world should down rank that actually, because it's cluttering up the feeds and it's gonna reduce engagements. It's all very rational from a business perspective. I think what would be a shame is if you start tagging things as AI generated and that drives down, artificially drives down the conversion rates of the AI generated content, because at the end of the day, If you are really good at providing a good prompt and good creative ideas, I mean, we've used AI for years, well before, you know, VO came out to create stuff. mean, I don't know, Godzilla, whatever, like pick your favorite random fictional Hollywood video, right? All of that stuff is AI generated. It's not like we've never used it before. It's just, it turned out to be really good and engaging. So think. Like this is just talking from a very business perspective. I think there's maybe like an ethical quandary around, hey, if it's AI generated and these people don't exist or like this thing never happened, that there should be some disclosure that AI was used to like simulate this. But, you know, I think similar disclosures exist even for real world videos where they're like, this testimonial is a simulation or, you know, when you have actors acting to give a test. when they aren't actually customers and things like that. So I think there are smarter ways for us to actually bridge this gap and we just haven't really figured out what those things are yet.

speaker-1: Okay, okay. That makes sense. so after Facebook, then you went to Credit Karma. Yeah. And I credit karma like I guess you were also doing a lot of like recommendation stuff. But do you want to talk about specifically what what are some of like the biggest stuff that you worked on there?

speaker-0: so when I was hired originally I was hired to take on the lightbox program ⁓ lightbox ⁓

speaker-1: I interned on lightbox. Hmm? I interned on lightbox. Yeah.

speaker-0: think it was probably, ⁓ maybe after I, after I, or I, like I moved over to Rex and light box had a, they had a, they had a divorce, they had a split, they had a breakup. I don't know what to say, but we we were still friends though. It was an amicable split. Yeah. When I first joined though, light box had just, I think they had one partner.

speaker-1: Yeah, after you left.

speaker-0: more or less, which was Amex and was just getting off the ground with cards. And I was hired out of Facebook as like an ad tech expert, moreover also someone who's actually had some enterprise experience and whereas credit card model is mostly consumer. So the reason why I share that is just like, it was interesting, this is an interesting time for the company where you're like actually trying to create products for partners, not just relationships. And And the interesting thing about that was like it was an incredibly sophisticated targeting platform. The idea was to get extremely sophisticated criteria from these banks and build out this targeting criteria ⁓ that would then carry through our platform and drive the labeling of the different products. ⁓ Interestingly enough, over time, ⁓ the Lightbox platform, which is effectively a very sophisticated targeting layer, layered on top of the recommendations engine, which you could think about as the optimization layer from from Facebook days or the Facebook parallel. And over time, both Lightbox and recommendations covered all all product lines. But actually, when I first started, it was only on cards, believe it or not.

speaker-1: That was because it was hard to do it for loans, or it was just easier to make money from cards.

speaker-0: It was the largest, I mean, I think it was just a sequencing and prioritization thing. was like the largest product line at that time by far was card. Loans wasn't even that big for Credit Karma at that point. But the personal loans was like the next largest thing. I would argue that recommendations and Lightbox in combination actually started to spike the growth of all these other product lines in ways that just wasn't happening before. They were kind of like rounding errors before.

speaker-1: Yeah.

speaker-0: those programs came out. ⁓ It's kind of crazy to think we were like, I think we like for extra revenue, more or less from when I started to like when I

speaker-1: Yeah, it's pretty crazy. Yeah. So that was like more like the classical machine learning stuff, right? You have a bunch of these signals and we're optimizing like, okay, how to rank these offers in the marketplace. And then I guess like that's a lot of the stuff that you were doing on the recommendations team.

speaker-0: Yeah, that was fun. So I took on Lightbox and I took on all of recommendations. Then I took on Core as well and we reworked the whole app to bring a more AI first engagement experience into the application. So that was the stories on the front door, if you remember those, the dynamic content on the front door and bringing dynamic content across the app. So this was actually published content. as opposed to being published ads, i.e. the offers from our financial products, our financial institutions. so that was the next undertaking. And then the last chapter, at least when I was there, was actually the financial assistant. And so this was the Geni consumer product, which was also a very, very interesting endeavor, very different than the previous projects in predictive AI.

speaker-1: I had left by the time the financial assistant thing started. like, what were the conversations that even led to that becoming a thing?

speaker-0: so, I mean, it was interesting because I would say that the financial assistant was probably the one of the first very major investments that Kurtekarma made in terms of changing the experience of the app that was kind of like somewhat driven by our own leadership, but really largely driven by Intuit leadership. Intuit wanted to make a very big play. in Gen.ai and they were building up this whole platform. And the expectation was all product lines would integrate Gen.ai in interesting ways. Now, of course, they weren't super heavy handed. We were able to come up with our own vision that made sense for us and our customers. But ⁓ that I would say was like an it was an interesting undertone of the whole thing. And then the idea there was ⁓ actually it started off incubating as part of core for the first year. So we were actually building that as part of the core product. And once we actually started to see it take off, the financial assistant became its own business unit. ⁓ And partially for that was because the systems themselves were just so incredibly different than the rest of recommendations, that it made more sense for it to be its own standalone product because it needed its own infrastructure. ⁓ Of course, as a smaller team, you're more nimble and you can move faster. The idea there was like integrating the AI throughout the app, not just having a chatbot. The chatbot was, we put that in because everyone expected it and you know, whatever, we needed to check a box. But the real needle mover was actually embedding ⁓ contextual AI entry points across the app. So think like you're on your credit score page or whatever. And it's giving you proactively serving you up an insight about your credit that's like specific and personal to you. And then that is a, that is a moment to like interact with the AI and actually get answers to your questions about your finances. And that was actually like, I don't know, 80%, something around there in terms of our interaction volume was actually coming through those, those entry points. And so, ⁓ it was, it was, it was very different because it was, It was content, but it was dynamic, ⁓ which was very different than the static content we had been dealing with previously in terms of the engagement product stories. And it had this just like, I don't know, like a 10x effect or potential for an effect on engagement within the app. And there was like lot of very interesting learnings that came out of that.

speaker-1: Were you guys using the OpenAI or you had your own model that you were using?

speaker-0: We were using off the shelf LLM. Yeah, yeah. ⁓ I just going to say we did explore building our own LLM to our own foundational model. Now non-zero chance they're still working on it, but at the time it just wasn't ready for prime time.

speaker-1: I'm like, ⁓ sorry. Yeah, and like, how are you guys thinking about the hallucinations? Because this is finance, so that's obviously not okay.

speaker-0: Yes, so it's very interesting. ⁓ I mean, one was we had to have obviously had disclaimers everywhere that you're interacting with AI may make mistakes. But we did we did a variety of things. So we actually had ⁓ guardrails built into the system such that you're able to we're actually able to basically check that, the data is coming from where it should be, that the right tools were being used, things like this before it served up to the end user. So we had a bunch of checks in place before it got served up. ⁓ And then on a post check or post deploy, we actually had a whole team of people in the ⁓ customer service department that were actually dedicated to effectively QA the QA logs. So we would do samples of of logs and then we would review those logs through criteria. And that criteria and that scoring and those logs, it actually serve as training data for our guardrail models that we then go put into production so that they're smarter about finding edge cases. And so we have this kind of like feedback loop that we had going to ensure that we didn't have any issues. The funny thing was is like the The times when we had issues, was like, ⁓ it was like, there was very, very few and far between. You would actually think it would be happening a lot, but it was like very, very, very small percentage of times where there would be any sort of like mistake. And most of the mistakes were actually benign and we kept a close eye on it. ⁓ So yeah, but I think one of the pro, one of the challenges though, is just like when you're dealing with millions of people and millions of conversations, ⁓ multi-million conversations, you can't monitor every single interaction. ⁓ And so that's where it becomes challenging.

speaker-1: Like what's an example of one of these insights that would pop up? So I'm looking at a credit card recommendation or something and it's like, what would it say?

speaker-0: It would surface things like, we noticed that your credit, I'm going to make something up because I don't actually remember the full cop. It was like something like, ⁓ we noticed that your credit score went down. It's because we noticed that your credit card balance increased and that was the cause of it. So you don't have to go dig into the app. It'll just pull that forward because it knows that, ⁓ your credit score went down. And the first thing you're going to wonder about is like, why? And so instead of

speaker-1: Yeah.

speaker-0: to search for that it's just there.

speaker-1: How did the team think about some of this dynamic content? Because I know there were specific content people who were on every team. So if now the AI is generating content, did you guys have to change their roles? What was that?

speaker-0: actually ended up happening was the content creation or the ⁓ content designers at Credit Karma actually did end up folding into the design department and were asked to take on broader design responsibilities that weren't necessarily content specific. But the reason for this was actually not because of the specific things we were doing at Credit Karma. It was actually a broader initiative from Intuit. And the idea of that was that product copy can be created through various AI tools and it can created much more efficiently. So it's not like their expertise is not needed. It's just they would be so much dramatically more efficient with the tools coming in. It doesn't make sense to have one dedicated person for 100 % of their bandwidth only focusing on copy related decisions.

speaker-1: Meaning the people being folded in didn't like that?

speaker-0: Yeah, I think some people embrace it and some people, you know, like they're they love the content part and they don't really want the other design pieces. And so that's like that is a little bit of a job change for them. And that was like a structural thing that was coming from, you know, from at the higher corporate level above above credit karma. So was kind of like it is what it is. And, know, you're going to conform or not. I do have to say I have like one small anecdote on this, though, to like the contrary, which is when this was first rolling out and we had to do this, I still remember the designers who are not content people being forced to create copy because it went both ways. So designers were told they have to do copy and then content people were told they have to go do design. So both worlds were merging ⁓ because the idea was like, ⁓ you can just use tools to create copy. ⁓ So they tried, they tried. I love them for trying. This was not their forte, nor was it the product manager's forte. And the things that I saw for like copy was just not good. And ultimately, it was like, kind of pulled the string and pulled a favor and I like called one or two of the actual content designers we had on the team, even though we're technically not supposed to be copy only. I was like, hey, can you just join this meeting and give us your two cents?

speaker-1: Yeah

speaker-0: And like within five minutes, they were able to dramatically improve the copy situation of the things we were trying to ship. So I think that there's a cautionary tale here of like, to me, that felt like a little bit of a over pivot or, or a lack of understanding of taste and judgment that comes into these types of decisions. Like, yes, AI can create copy as in it's an LLM. Language is part of. It's large language model. Yes, it produces language. Is it good though? Right? Like there are practitioners who have spent their careers figuring out what works well and resonates with people and is easily understandable. And, ⁓ I think, I think AI has a ways to go before it can replicate that level of taste and judgment.

speaker-1: What are like some other roles that you think are going to change in like these bigger companies because of LLMs, I guess? Yeah.

speaker-0: mean, I think that there's this ⁓ I know, I think it was like one of my old professors called it like a garbage can, it's just like a big swirling of stuff. So I think like product is totally different, right? Product managers now prototype like crazy ⁓ as part of their day jobs. I've heard, I've actually talked to a handful of them since I've been out of the game for a year. I prototype every day, but that's because I'm a founder now. But like, actually at real corporate jobs, PMs are prototyping all the time. And I was like a 50-50 hit rate and I was like, oh, well, what tools are using and how are you doing this? Like, actually, I'm just doing it on my own laptop because they won't give me access to the tools, but it's all prototype, which I find to be totally crazy, but you kind of have to do it. And so that's kind of one of the interesting things that I'm seeing right now. What is the implication of prototyping is the following. It means that product managers actually need to lean in to product design, to engineering a little bit, and into content design, which is like, and maybe even a little bit into UX research, which are all of your supporting functions, but you're like actually moving into the territories of all these other job functions. ⁓ So I think that's interesting. And I could see a collapse of the product role or ⁓ dramatic evolution of the product role as a result over time, especially as the tools get better.

speaker-1: So does that mean like you're going to need less designers, less UX researchers?

speaker-0: I imagine that their work might just look different. Like they're just gonna, maybe a lot of the grunt work that they, that are associated with their jobs, holds into this prototyping step and maybe that's actually taken care of by the product manager versus the designer. I still think like, I mean, come on, you've probably used lovable and cursor or whatever, right? Like, are their designs great? No. Do they get the point across? Yes. Would I still pay a designer to like make it amazing? Right now? Yes. I have not seen tool that does that yet.

speaker-1: Yeah. So do you think there's going to be more PMs, less PMs?

speaker-0: Oh, loaded question. I think there's going to be just as many PMs. I think it's really hard to shrink the PM population. And these are the dynamics that I'm putting in my head in play, which is I think that there's just going to be a lot more stuff shipping. because it's easier to ship the cost of executions gone down, which means more things are going to go out the door. So if more things can go out the door, you need just as many humans to think through, what is it that should go out the door? Right? And I feel like that economy of scale is like very hard to achieve. I have a hard time believing that we're going to have less PMs. It feels like you still need probably still roughly the same number. Maybe each PM is like shipping more because our engineers can do more, but it'll probably be the same number of PMs. That's my prediction.

speaker-1: Yeah, it's kind of like how everyone says, because you have AI now, you don't need as many people. But it's like, well, if you can do more with the same amount, a company is just going to say, OK, well, why don't we just do 10x more with the same amount that we have now and just beat all of our competitors.

speaker-0: Exactly. I actually think it's going to be an arms race in that direction more so than a cost cutting thing and I've honestly I personally called this like a year ago and everyone thought I was crazy ⁓ And I'm like actually very delighted that the world is like You're actually seeing it right it's like it's not saving on costs. It's just helping you go faster

speaker-1: Yeah, it's kind of like game theory because it's kind of like if you just cut the number of people and do the same amount that you're doing now your competitor is just gonna keep the same amount of people and do 10x more than you. Exactly. So then you're just not gonna beat your competitor. So yeah, that makes sense. So then why did you leave Credit Karma at the point you did? Because it seems like that was like a big big change in the company that was happening.

speaker-0: I don't know. believe in chapters. I think that I stayed for the chapter of learning and building and zero to wanting the Gen.ai product on this frontier of technology. But honestly, ⁓ it was my time to start my chapter of my own company and building my own thing. The last revolution like this was the internet. AI is going to change everything and there's no other time to build the now. And I've actually always wanted to build my own company. ⁓ Well, before this moment, I actually bootstrapped a company 15 years ago when I was in business school. ⁓ And ⁓ I needed money. So I took the job at IBM and came to San Francisco and my whole plan was I'm going to join a couple of startups and then I'm going to start my own company. And then 15 years went by and I didn't start a company. but I, but in the process I fell in love with AI. So I've been working with AI for so many years. And when Gen.ai came to town, basically AI, working with AI and using the power of AI to build a product turned from a big company game to like one that was much more accessible to pretty much any entrepreneur. And I think that change for me was the catalyst of like, all right, the time is now. Like, ⁓ I was very happy to have a bookend and have that like last launch with Credit Karma, but it was just time to transition and, and, and start my own thing.

speaker-1: Yeah, so like when you left, did you already have an idea or is that when you were doing more of the consulting stuff and then you move towards your own product?

speaker-0: When I left, I didn't have a specific idea, but I had a couple of principles in my head. One was around ⁓ this idea of tacit knowledge. So like knowledge that lives in your brain that has not been digitized and may never be digitized because that data would never really be used in LLMs. And I wanted to work with that kind of... data and see what I could do with it in terms of value creation because that felt unique to anything that general purpose platform could do. The second notion I had in my head, which comes, think, from my leadership background is just the fact that AI is going to first make it really, really easy, fast and cheap to do things. So people are going to do more things. So where is the burden or the cost going to move as a result is actually going to be at the leadership layer. It's like, figuring out what to do, which is actually what we're talking about with PMs. I actually think it's also going to level up into leadership because ultimately leadership makes calls, unblocks things, diffuses fires, all these things. And so my idea was around playing at that layer. So combining that with tacit knowledge and like figuring out what I can do in that space. When I started the consulting, the consulting practice, It really was mostly to have a legitimate store face, so to speak, so that I could have conversations with customers. And if I could help people along the way and learn from, learn from their problems and provide value, that was great. But actually it was more from a lens of entrepreneurship. I just wanted to learn from what people's problems are out there to figure out like what's a meaningful product that I can go build that can really move the needle for people.

speaker-1: Yeah, actually back to the leadership thing. you do you think because I've heard of a lot of these companies that are like firing managers and making more of them like ICs. Do think that's going to happen more?

speaker-0: think it's the same thing as the notion of like, we're going to cut headcount at some point. ⁓ It may or may not make sense. I see why it does make sense in that your best, some of your best operators are actually managers. And if they can manage AI tools instead of people and produce the same or better results, you might actually be better off with that. And I think that is the rise of the super IC. ⁓ you're basically giving them leverage and through AI as opposed to human capital. And I think that's one strategy that works for some people. For other people, you know, that may or may not be the right strategy for them. You may still need leaders who are just like making decisions and dealing with people and figuring out org charts and like, you know, using their influence to to move things around across an organization and working across teams, across functions, et cetera. And so I think that in whereas I can kind of see that first one taking shape in a way, it feels like a temporary holding pattern. think you're like, effectively, there's a new type of role, which is like a super IC that's like supercharged by AI. And the first place people want to go grab humans to go fill those roles that are now available to be filled. are these like very senior people and management positions because previously that was the path. There was only one ladder and it goes up and that's where you end up if you're really good. Now there's like, so that's like the most obvious place to find that senior talent. But I think it's one of those like that senior talent will exist, but then you will also need managers for many other reasons. And so I think it's just a... an opportunity and then like a vacuum that came to like suck up talent into that opportunity more than it is about anything else. So where I'm going with this is that in the future, if you fast forward like three, five years from now, I imagine that people will graduate into those roles versus being covered in from senior positions because it'll be more natural and normal for those things to exist. And there will be a more natural and normal path to like land in those positions.

speaker-1: So you started doing the consulting thing and I guess like, what did you see? Like what were the commonalities in like successful deployments versus unsuccessful ones? So I think that's a really big thing. You've been, you probably saw that study that everyone keeps talking about the MIT one. It's like, oh, 95 % of all these like AI projects are not actually being productionized. like, what commonalities did you see in the, successful ones versus the unsuccessful ones?

speaker-0: I think the most common thread that I saw was people just trying to bite off way more than they can chew. Like when you think about AI, it's like, and the way it's been romanticized, quite frankly, in media, it makes you think, oh, it's just like magic fairy dust. can just like sprinkle on top of this. It's oh, can't the AI do this, this, this, and this? like, yeah, I mean, technically it could.

speaker-1: You say hi.

speaker-0: But people don't realize like the amount of effort it takes to like do each individual project and so it's mostly the thematically that is that's been like the biggest learning across the block is that especially people who don't come from technical backgrounds tend to massively underestimate the investment they have to make and so and just to kind of break it down a little bit further it's like there's like two places where people get caught one is like they don't have their One is they don't have their data story pulled together, and that's assuming that they understand what they're trying to optimize. The other piece is like, they actually don't know what they want to optimize or how it works today. So by that, I mean, it's just like most AI implementations are about some sort of like automation of a workflow. If you can't articulate your workflow, you can't find the bottlenecks. You can't find the opportunities to like 10X productivity. Like you actually have to do that granular work to figure all that out. And then you could figure out if you have the right data assets to do the thing. And then you can start building and people want to go from like, you know what? think my marketing could go faster. Can you just throw AI at it? And like, it's just like, that's just not how it works. We need to get a lot. Like the level of specificity required is pretty extraordinary. And I'd say that. One last thing to add is at least from my background, I have an engineering background. Like I have a master's in EE. And because of, I think because of that, my brain is very structured with respect to understanding systems and flow charts and that kind of a thing. Just as I'm like, I'm an engineer by trade. ⁓ most people's brains don't actually work that way. And, and I think that was like an, that was probably one of the most interesting things that I realized as I was working with different people across different organizations. They have an intuitive sense that AI could help in certain places. Really struggle to actually do the detailed process flows and trying to understand it. What is it that you exactly are trying to accomplish? And that's, that's where I've seen most people fail is like they, they, fail at the start block actually.

speaker-1: Yeah, were you mostly working with companies of like a similar size or were there like some enterprises, some like startups?

speaker-0: It's been kind of all over the map. Like in some cases I was like more playing more of an advisor than it like actually doing any implementation, especially those at the bigger companies. And there's some some that are leaning more smaller that I've that I did a little more implementation with. But in all those cases, what I just described, whether it's a big company or a small company, this was the biggest failure point.

speaker-1: Interesting. Like, did you see any differences between the way that the enterprises were thinking about AI versus how the startups were?

speaker-0: The enterprise are definitely thinking bigger. They would swirl a bit more and like run into brick walls here and there, but eventually they would figure it out. Because I think partially because enterprises have a bunch of engineers in-house on hand. So there's like opportunities to ID and pull in resources and over time triangulate into something and kind of figure out where they want to start. So it's like, yes, it would take several months, but they have the fortitude and conviction to stick with it and eventually figure it out. And yeah, they're putting like, probably 10x more engineers than I probably would have against them. But they figure it out eventually, right? And they eventually move forward or they come up with some sort of tool or something. It's never perfect, but they figure it out. The smaller companies are very like, because they're so lean, like prioritization and like, really tightly binding the scope is so much more important that

speaker-1: Yeah

speaker-0: they try to do it in-house, ⁓ is ⁓ hard unless you're very savvy or you have the right people in-house to help you. Otherwise, they're almost always better off just hiring somebody to just figure it out for them because it's hard to have that kind of expertise in-house.

speaker-1: Yeah, like what specific expertise just you've used all the latest tools that kind of thing.

speaker-0: Yeah, like even technical staff, like, um, trying not to name names since I don't have, um, that it's all confidential stuff, but there's certain companies I work with, like they, just straight up don't have, they have ideas, they've got a great business. They don't have an engineering team. And even, even though theoretically there are no cold tools out there, there's always these like system and data layer pieces that are just like hard to grok and figure out. And so. a lot, especially if they're, it's a largely non-technical organization. It's just, it's just too difficult to translate what's in your head to like what you actually need to go do. And then also, ⁓ just having that, like, I don't know, ability to get in there and just learn what you don't know to like figure out all the technical stuff. It's just, it's very daunting. ⁓ so more often than not, it ends up turning from like advisory to me suggesting or recommending like, a dev shop that I work with or like ⁓ an AI engineering company that I work with to like just come in and actually help them do the thing. And it's like, they're doing scoping and building. It's not just building.

speaker-1: Okay, so what does a project look like from start to end in the successful deployment? it like you start with the evals and then you go to everything else? How does that work?

speaker-0: Yeah, I mean, it's really ranged, honestly. Like there's like one company that's like trying to, I don't know, scale to venture scale. We're helping them all the way from the very beginning of the business strategy. Like what area of the business do you want to go after? What types of things? And we were helping him with not making his business efficient, but actually shipping AI products, which is a little bit of a different lens. So that's interesting. That's like ongoing work. So no end yet, but. Definitely a strong start. And then there's the other ones which are like a little bit more operational in nature. And with them it's just like, it's really, it's still a lot of like, I'm first principle, like, what is your business objective? What are you trying to achieve? And starting there in terms of like, what you want to go build. Because also people have an idea of what they want to build because they're like focused on a symptom versus the root cause. And so you have to go through root cause analysis first. And then of course, evals come flows out of all of this, but it's like actually very important to take like, what is their like stated business problem and then actually get into like the weeds of why is that the problem and then start there. And then we build like eval, a sense of evals and also just general success criteria off of that. And then we can start doing prototypes. We usually start with prototypes though, right? Like just don't over-engineer it. Just like make sure it actually is kind of working. And then we start to go on to bigger displays.

speaker-1: That's what I've also seen. It's like, if these guys already knew what they wanted, then they would have just went to a dev shop just to go build it. But the reality is most people don't actually know what they want. And so really what they need is that person to figure out what they actually need. If that's what they should be paying for, it's not necessarily like, just build me this. Because again, if they knew that, then they would just go to somebody to build them exactly what they wanted.

speaker-0: You know, the other very interesting thing that I've picked up on too is there are the people who have the wherewithal to know that they don't know exactly what they want. And then there's the people who think they know what they want. They try to just demand what they need. ⁓ There's like one shop that I work with and actually occasionally they will tap me on the shoulder to come in because they're just like, Hey, they thought they know what they want, but they actually don't know what.

speaker-1: Yeah, yeah

speaker-0: So did Al with like, can you work with them from a business perspective so that we like build the right thing? Because like, at the end of the day, they're like, well, I could make money, but there's no point if they like run out of cash because they're spending money on the building the wrong thing. Like, that's not good business or their business.

speaker-1: Yeah, yeah. So like, is there going to be a new like role for this type of person? Or is it just going to be like all people should just get up leveled on AI? It's just going to take a few years before that happens.

speaker-0: really hard, hard time making a call because conventional wisdom would say everyone has to or needs to get up skilled. But it's kind of like everyone saying, hey, everyone has to and needs to get up skilled on like coding skills in general, which is what we've been saying for like, past couple of days since the beginning of the internet, you know? And it just, it never happened because I think people's brains just work really differently. So I still feel like it's like that systems mindset that really Like and not everyone has that right every and not and I don't know that everyone should I think different people bring different strengths to the table. That's like the beauty of humanity. So I think that there will probably be just like champions across various organizations that have this skill set and it'll be a new a new type of role versus an upskilling broadly. I think people will like learn to use AI just like they learn to use the internet like no one's going to ask you about your web skills, you know of quick, like your AI skills is like asking about, you have web skills? Like in this day and age would be stupid. That would be insane. Like a couple years now, but like, so people have basic understanding, but as far as this like architecture kind of stuff goes, I, still think like, I think there's going to be engineering support or engineering AI engineering, like support for pretty much every function across the board, whether it's centralized or decentralized unclear, but like I I think that's more where things are going to go.

speaker-1: I see. Okay, I have one more question. Maybe I should have asked this earlier, but like, are most of these like execs thinking about AI is like, it's going to save them money or more like how do I make more money by using

speaker-0: By far everyone is thinking about cutting costs. ⁓ I'm starting to, it's the more interesting forward looking ⁓ executives that are starting to finally see the opportunity for upside versus costs. And I think it's because of competitive pressure. And this goes back to the original note I made earlier where I just like, I've been waiting for this moment. Cause like for me personally, it's so obvious that like at some point it's exactly what you mentioned. It's like, ⁓ You know, it's it's not going to be erased to the bottom is going to be erased at the top the minute that Someone is 10x more productive than you you're going to have to just step up your game And so I think the year of my personal prediction is the year of 2026 Is when we'll start to see the paradigm shift towards that mentality and less about having

speaker-1: Yeah. And that's kind of like the idea for your product,

speaker-0: Yeah, so my product, the agency, today we help executives create executive content on a regular basis. But the larger vision for it is to actually be an executive's twin, so to help them with everything that they do.

speaker-1: Yeah, like how did you come up with that idea?

speaker-0: I think I just looked at my life previous to being a founder, being an executive and thought, you know, if I had AI, how would that change how effective and productive I would have been? ⁓ I even had micro moments during my time as an exec thinking because I was working on AI. I'm like, ⁓ my God, I wish I could just like commission a project to automate this or help me with that. And I realized it was like three facets, right? It's like you're thinking all the time, you have to communicate and package that communication and you need to ship it out to lots of people. You're just constantly doing that as an executive. And so a lot of my past life was an inspiration. And then also my current life of like working with executives constantly in my consulting capacity and just in my day to day learning that actually I'm like not alone in those desires and We're very executives are kind of an underserved market. So I'm to serve them and I think that it's going to, it's going to 10 X the executives once they realize that they need a 10 X their companies. And so I'm excited to catch that wave.

speaker-1: Okay, cool. Wait, it's kind of like executives aren't really sharing their insights online and maybe they should be. And so we can help them kind of do that. That's sort of like the overall goal. Yeah.

speaker-0: That's ⁓ the current version of the product, but ⁓ future vision is like ⁓ going beyond social media, like just in general, inside and outside the building, all the different things that they need to communicate on a regular basis. A lot of leadership is communication. so communication, coordination, disseminating decisions and insights. And so how do you actually disseminate all of that in a very structured, sharp, timely, efficient way to basically activate hundreds, thousands of people sometimes, depending on how many people you're affecting with your decisions. And so how can you do that in ways that takes like hours instead of like weeks or months? And like, by the way, I was on the receiving end of the weeks or months version of this on a regular basis as a VP at Credit Karma. you know, having people on the same page within you know, within a day versus weeks or months would have been like a game changer for a lot ⁓ of initiatives. So, you know, take that and then multiply that with the productivity gains you're going to get for each individual function with all these other AI tools coming out. ⁓ I think this could be kind of a force multiplier for smart businesses.

speaker-1: Yeah, so what's the current iteration of the product and then where do you see it like over the next 12 months, I guess.

speaker-0: I see it for the next 12 months is what I just described. The question of the product is ⁓ on thought leadership for social media.

speaker-1: Okay, cool. So like over the next six months, like what are some features you're going to be shipping?

speaker-0: Yeah. So, ⁓ right now we're in this like alpha stage, private beta. I don't know what you want to call it. ⁓ I guess we're in alpha. we're going to move to private data in a month. ⁓ the alpha is basically an email only interface. So my first handful of clients have only been using it via email. There's no web app or anything, which is actually kind of nice. They like it a lot because you literally don't have to log into anything and you, just text into their phone and send it and it's great. ⁓ But we're going to release a web app in the next month or so. And so we'll start to open up a private beta with a web app, we're able to just take on more customers. And so I'm very excited about that. And then the big thing from there is like, once we have that, that web app ⁓ in place, ⁓ we'll, then start to lean into more features that feel a bit more like, like having a thought partner. So it's not just about you. telling us about your day or sharing your meeting notes or, or transcripts from your podcast or whatever. It's like, you're actually, ⁓ you're actually able to interact with the agency and like pressure test thoughts and come up with ideas and ideate. So there's, there's a big, there's actually a big large roadmap around that sort of hope to start to switch gears and release features around ideation and just being a thought partner. So that's, that's probably the next ⁓ Three to six months for us will be building that out and sort of setting the stage for for something that feels more like a digital twin, but we're sort of working our way there.

speaker-1: Cool. Interesting. So I don't want to like go too, too, like overboard here. ⁓ I know like we're already like 15 minutes overboard, but like, if anyone wants to get in touch with you, how can they get in touch with you? If they want to work with you on the consulting staff or if they want to use the agency. Yeah.

speaker-0: Well, feel free to find me on LinkedIn. My handle is Supriya G. Also, you can visit my website. It's getagency.ai. And there's a waitlist there. see every submission from that waitlist. It's not a black hole, I promise. And if you leave some comments there, let me know that you heard about me on this podcast and any context about your situation. And I'd be happy to be in touch and get to know what you're working on, what I can help with.

speaker-1: Cool, awesome.