Dan: Welcome to AI and Design, where we explore how artificial intelligence is reshaping the world of design. I'm Dan Safer.

Nik: And I'm Nick Martillero, and we're faculty at Carnegie Mellon's Human Computer Interaction Institute. Each week, we break down the latest AI developments, dive deep into topics that matter to designers, and talk with fascinating guests who are right at the intersection of these fields.

Dan: Whether you're a designer working with AI or an AI practitioner interested in design, we're glad you're here. In this episode, we talk about the ways you can train your design senses.

Nik: Christine Ju gives us what systems thinking looks like for PMing AI products.

Dan: And our final bit of news is a project that helps you change the default settings on your LLM of choice.

Nik: And our guest today is Dan Macaron, founder and CEO of Charming Robot, a digital product studio. Dan wrote an article about revamping his studio around AI that really caught our attention.

Dan: But before we get into it, a quick word about something we're doing at the HCII. Every conversation that we have with industry partners right now is about AI. Every single one of them. Product leaders want to know where AI actually creates value, which bets are worth making, and which ones aren't. These aren't new questions. The HCII has been working on them for over 30 years. AI just happens to be the technology of the moment. But many companies are moving so fast to ship AI that nobody gets to just explore. That's what the MHCI capstone is for. You get two semesters with an interdisciplinary team of graduate students and faculty to explore a new area. You don't walk away with just a slide deck. You walk away with working prototypes, design systems, and documentation that your team can build on. If your organization wants to sponsor a capstone project, please reach out. The link is in the show notes.

Nik: All right, so Dan, let's talk a little bit about the stories that we read this week. the first one, we talk a lot about design judgment here on the show, but we often don't talk about how you should do it or how you should develop your judgment. and there was a really nice article that came out by David Huang called Training Design Senses. it's pretty short. It's actually, a five minute read, but I think it actually lays out four ways that you can better train your taste and your judgment. And these are things that probably professionals here have done before and they they might even do these things without even thinking about it now. but for our, listeners who are maybe earlier in their careers or students who are listening, this is actually a nice little article to think about how you train your design judgment. Dan, do you want to talk a little bit about what some of the four areas were?

Dan: Yeah, so the first one is all about training the eye. To train the eye, you need to spend time doing three things, which is capture, collecting, and curate. So you first have to capture things that you notice. You capture them with a physical notebook or a camera or your phone, or anytime that you're out walking around or surfing the internet, you want to capture the ideas, and then you're basically collecting them. You're collecting what matters, like things that are important to you. And finally, Finally, you want to curate what you capture. You Don't just have it like sit in your photos. You actually take the time to curate, like you're a museum curator, looking for new things. Put your observations somewhere that you can revisit, whether that's a Pinterest board or a notebook or an actual physical mood board that you can put up.

Nik: Dan, this reminds me a lot about when I went through school, they actually had us take magazines and clip just stuff that we thought was cool out of them. And then we would pin up the pages in the studio and just have them around us. And actually I've kept this habit ever since. Actually in my office now I have postcards and pictures and printouts or even small objects and trinkets that are pinned to my wall and they they live above my desk because it's something that, I really enjoyed doing, but it it is something to kind of train how you're looking at the world and get inspiration from so many things.

Dan: Yeah, I used to have a collection called No Ideas Button Things. And it was just a collection of mechanical interfaces. It was buttons, it was knobs, it was dials, it was dashboards, but none of them were digital. They were all like from everything from stuff I would find on the internet to things I would just see out in the world. And it really was Pretty formative to me and I I kind of lament that I let that URL and that collection go. But it was it was really valuable when I was a junior designer to start to really train my eye to look for those kinds of things and the different ways that something as simple as a button or something as basic as a dial could be rendered. So the second thing that he talks about in the article is design dexterity. And this is all about design making and being able to really focus on how things are made. And one way that he suggests in this is to copy from Masters and take a screenshot of your favorite app and figure out why they did it. Reconstruct it from scratch. Why is why is this here? Why is this piece like there? And this is something that we actually give our advanced design students in the masters and undergrad programs who take advanced design with me. This is the first exercise that we give them is to deconstruct a bunch of interfaces to figure out why they work, what might be improved, but also just how it was constructed, what was it like to actually have to build this from scratch as a designer in a company. And then the the other thing he mentioned is, how are things finished? what's the polish to create pixel perfect artifacts? that's the other thing that he recommends is you know, it's one thing to kind of just kind of slap things on the screen, it's another to really figure out what makes it look very aesthetically pleasing.

Nik: actually, Dan, this one to me, there's an argument here of recreating by hand say an interface or a piece of a product. But I actually wonder from the perspective of working with AI systems as we do design work. There is a way to potentially train this design dexterity by utilizing AI to recreate things, but not copying and pasting a screenshot, and as in the article saying, like have codex or you know, Fable just recreate it, which usually can do pretty well at that. But actually the idea of building it up, say component by component, and working collaboratively with an AI system to do that. For example, I know I think last week we talked about how I've been using Fable V for stuff. And actually I had it build a lot where I just sort of gave it a goal and it felt awesome. And then I came back a couple of days later and I was like, this is garbage. Like I because I didn't think about it. I actually didn't think it through. And actually I didn't have this sort of tacit feel for how to even, prompt the system correctly. I just gave it a goal, which didn't work. I've since actually been working with Opus V. But I've been working in a much more deliberate way where I'm kind of building up interfaces component by component. And actually, I think that if you were to look at and say, just have say the reference image up, and in the same way where you might try to paint or draw something in a similar way, if you work with the system and say, don't give it the image, but say, I'm going to describe and I'm going to basically work with an AI system and prompt this to actually recreate it, you might be able to develop a better sense of how to. Get these systems to actually create what you want and what you see in your mind's eye as opposed to surrendering to what it comes up with.

Dan: I think there's always this gap between what you know is good and being able to make it good. And I think what he's trying to say here with the design dexterity is like how you take that time to decrease that gap between what your eye knows is is off and what your hands can do or or what you can tell the AI to do. So I think any time that you spend doing reps to close that gap, I think is valuable time. The third one here that he mentions is fluency of aesthetics. And this is this idea of being able to study what people have said is is beautiful, to look at art, to to create basically a a taste to kind of get your design and art history here to increase your fluency of just everything that's out there and everything that's been done. I myself I look at things from Tibor Kalman or Dieter Rams or all the Muji products or read a lot from other designers. Particularly a big influence on me was Michael Beirut of Pentagram Design. A lot of his writing really speaks to me. And being able just to have other points of reference and to think about, well what would this person do in this situation? Or if I look at this through a Bauhaus lens, what might that be? And I think it's just giving yourself another lens to look through things. the last thing he talked about in in terms of building up your design sense was what he calls establishing rhetoric. And it's funny because Carnegie Mellon design under the old head of the school, Dick Bucan, was all about rhetoric. And it's funny to see this here. I mean, rhetoric, for those of you don't know, is the art of persuasion through language and and visuals. And it is all about visual communication. And what he is saying here is that it is all about presenting your work. It's about persuading people that this is a good rationale. It is influencing them to be able to go forward to you know to either fund the product or go with your idea If you take rhetoric even further, it's about products being able to persuade you, hey, you should incorporate us into your life. I'm a good product. you should use me. And he talks about presenting your work even before it feels finished. And it's not about necessarily winning the argument, it is about being able to sharpen your judgment and explain your design decisions.

Nik: Yeah, I really like the point here that if you can't explain a decision beyond it just feels right, your intuition may be ahead of your reasoning. And one of the things I like about this is that he's not discounting your intuition. Like you may have actually developed a good sense of what is tasteful, what is good. You just don't have the language to be able to convey that. And I think that that is something that as we mature as designers, we develop. And that is what sort of sets apart, good practitioners to like great design strategists.

Dan: So yeah, what d generally like about this article, and I think it's great for all of us to be reminded of these fundamentals because this is something that everyone is yelling right now, taste, taste, taste, judgment, judgment, judgment. And it's like, okay, well how do you actually do that? How do you sharpen that? How do you not let the tool take over that for you? Well Here it is. He just gave us four ways to do that.

Nik: in our next article, we're talking about systems thinking for AI products by Christine Ju, who is a PM at Intuit and is actually working on agent systems there. So, the main argument of Christine's article is that product managers who are building AI products, and especially agenc AI products, where the AI is actually operating on its own and doing tasks, is that they need to shift from building features to growing systems that these AI agents actually work with and can evolve around. And the argument is sort of that because AI models and agents behave In this organic way, PMs should act like gardeners in creating the environment and some of the rules that determine how these AI agentic systems are operating. There are a couple of points that Christine makes in the article on sort of how to approach this. The first one is decomposing the product. For agent experience. And we've talked about agent experience before. And the idea here is that whereas a lot of products are typically built for human consumption, so that's user experience, if you're building a tool that needs to have AI interact with it, you need to be thinking about an AI agent is going to process that. Actually, just sort of wholesale taking a UX perspective is probably not the best idea. You have to actually think differently about how AI systems and agents use it. And the way that Christine recommends that PMs think about this is decomposing workflows into a set of core primitives. This is the context that the agent needs, so structured context, the actions that the agent can take, the preconditions for when an AI should take action when it may need to raise an issue or ask for a human intervention, and then the logic around coordination between that agent and arguably probably other agents within the system.

Dan: I mean, I think designers think about at least some of this all the time. This is kind of what our design system thinking had been kind of all about, but now we're just adding a lot of extra layers onto it I think is basically what she's talking about here.

Nik: Yeah. And then the next area is defining contracts over the features that you are developing. And so the idea here is that instead of building sort of one-off features that operate within their own silos, you need sort of this set of contracts between things. The metaphor that she uses is it's kind of like an electrical outlet. So the electrical outlet is a standard. Every appliance you plug into it it's built for it. And so that standard interface, allows the electricity to get to the product. And so I think the idea here is sort of thinking about what are these interfaces and the standardization around those, sort of between how parts of the system work together. And you define those and you provide those so that your AI agents basically know how to work. with these things and they know how to behave with these different aspects of a system. the third area is designing feedback loops. We've talked about feedback loops a lot on the show. and I think this is something that that sits with designers, very, very well because so much of the design process is about feedback loops. But the idea here is that you want to be building feedback systems into your products. and especially into anything where agents are operating. There were a couple of ways of thinking about these. In the primary area where, a feedback loop can work really well for a agent system is the correction signal. So this is the difference between what the a agent does and then what the user had to do to correct it. So the example that's given here is if the user edits a field that the agent updated or accepts the draft but rewrites the last paragraph in something, that's a correction signal that you get. And you should be capturing all of those. There's two big benefits of that. One is that that lets you know that your agent is failing. I mean, it's not doing what it was intended to do if the agent was supposed to do it on its own. And this is something that you may have not seen or thought about in the benchmarks that you were developing and the second aspect of this is it gives you information on how to improve. you can actually feed that back. into your agent building system, which likely itself is an AI system, to then say, hey, this is a place where the agent didn't do so well. Let's actually figure out how to improve this. you can imagine giving it into like an auto research type thing of saying, hey, fix fix your prompt, fix your system settings so that in the future this will happen correctly.

Dan: Yeah, I think that is the real key to this is this kind of self-improving system where it's like, I keep noticing that this thing is going wrong. Maybe I should generate something completely different. And having that learn loop. built right into the system is really powerful and really is speaking to the future of product design where you have these things that are slowly adapting to particular users and to particular user groups, I would say, well you're doing this thing? well this is the right way to do it or here's the right pathway for your agent if you're trying to do these things.

Nik: Yeah, and I think that a lot of design teams out there probably have pretty long running loops that do this work of getting, product feedback and then figuring out how to improve it. The question is, is how do you get an AI system and an AI enabled product that can start to collect those signals and maybe do some of that work on its own. So it like you've said, it's a self improving system.

Dan: Right. mean the good product teams are definitely doing this. It is just those loops take weeks, months, sometimes years to really make their way through the system. And this could be hours if you deployed it.

Nik: the last part of the article talks about, hey, okay, I'm asking you to think about systems, but actually, that's a lot of work here. I gotta decompose stuff, I gotta define all these rules and contracts and define feedback loop systems and potentially build systems for it to auto improve. that sounds really long and tedious. and of course, we gotta move fast. Like we gotta move really fast. But Christine makes an argument here that. That actually this kind of systems thinking can eventually become a velocity advantage. that if you put the work and the effort into building up these core aspects of the system, that you'll ultimately be able to more quickly roll out and test new things and learn from them fast without having to sort of re-implement the basic infrastructure of what you've created. So there's a lot of interesting ideas in this article. And a lot of the thinking is rooted in some really great systems thinking from authors like Danella Meadows, whose book Thinking in Systems where many of the ideas were rooted. and so we'll we'll put a link to that in the show notes alongside the link to to this article for you to check out as well.

Dan: Yeah, thinking and systems and the Stuart Brand pacelayers are definitely two design thinkings, systems thinkings classics that only become more and more relevant as time goes by. our last thing that we wanted to talk about is a concept project. It's from Nadia Piet of AIX Design called LLM Counter Defaults. Now, AIX Design is a nonprofit community organization of quote practical radicals making AI work for the rest of us. And I really love this project. so the the idea behind this project is, and I'm quoting from the website, which of course we'll link to in the show notes, is that your AL LLM came with settings you didn't choose. It agrees with you, it sounds sure, it writes in your place, and every dimension of that is one of those factory defaults. And on the website you can move the slider to tailor the result to your own needs and values, and then you paste the resulting instructions into your LLM once and then it follows your rules. So what this is basically doing is helping you figure out what instructions are the defaults, because most of these I guarantee you probably aren't even thinking about even if like us you are fairly sophisticated working with AI but if you're not even more important because so many of these are what I would call hidden settings and being able to go in and use this visual slider to kind of figure out where your own preferences are is I think a really great way to visualize something that is otherwise completely invisible and I think that's one of design's great superpowers here. Now I will say that this website has one of the worst default website design that has come out of AI this year. like it's challenging to read and it does not look great. So I'm caveating this, but I think the idea behind it is really good about really empowering people to take charge of of their AI and tune it the way that they want.

Nik: Yeah, I really like the concept behind this because we do a lot of work in the lab here where we're trying to get AI to act in specific ways that aren't like the default that we usually get, sort of sycophantic, people pleasing. You know, we actually want it to be, for example, adversarial. we develop agents that have different personas and work in different ways. And so the fact that actually there is actually a default that you get when you open up Claude or ChatGPT and that you can change it. Actually the fact that you can manipulate that is something not everyone knows. I imagine a lot of people here know that, but actually knowing how to change it is kind of tough. And so it's cool that the tool I assume is kind of built on maybe some best practices of how to Tune these types of settings with instructions.md files. So basically giving an addition to the system prompt to the language model so that it it acts more like the way you want it to do. And yeah, I mean I like the fact that also it exposes like there's a lot of things that you can set and it make it does make it kind of easy to set with sort of the sliders. It gives you examples of what it means by those things. So for example you know, changing the voice or setting the persona to be more person like or more tool like. I quite like the ban LLM speak where you can just turn off dashes, inflatable vocabulary, filler adverbs, not just X but Y. Like you could just basically say stop, don't ever give me these things. yeah.

Dan: Right. Yeah, here's here's another example. So, you know, it there's a slider for divergence, and there's basically four ticks on the slider. One is where the default setting is, that it gives you the most common expected answers. And then it goes up to the next one, which is default answers occasional alternatives. The next one from that is the obvious answer alongside less common ones. And the farthest away from the default setting is that it surfaces under explored angles and can give you contrarian takes. And depending on what you want or what you're doing, you you can just push that slider to any of those different places and get your AI to work differently One other thing I I like here and wanted to call out was that there are a couple presets here that are just if you don't want to fiddle around with each of the little sliders, you can just be like, hey, start with a preset like Feed My Curiosity, where it gives you different alternate frames. It gives you a lot of cross-disciplinary connections, and you can just click the preset and then copy the instructions and take them over to your AI of choice. So stuff like that I think is really nice. I w I'd love to see something like this built into the actual settings of these AI, but until then, I love that we have LLM counter defaults.

Nik: And with that, that's our last bit of news for the week. now we'll jump to an interview that Dan and I did with Dan Macaron, of Charming Robot to talk about how he has been changing the way his studio works, given his team's use of AI.

Dan: We are here with Dan Macaron, who is the founder and CEO of The Digital Product Studio Charming Robot. And

Dan Maccarone: That's right.

Dan: we're excited to have Dan on because We haven't had a guest on that is from the design studio world, which I'm very excited to hear about how AI has changed your process. And that is what we wanted to talk about. You wrote an article a couple weeks ago, and I said to Nick we should we should have this guy on to talk about this. So you wrote an article called Nevermind the Prompts. Here's the thinking about what happened when you started to apply AI to your company. And before we hear the results, can you take us back to the beginning, tell us what you did, how did it start, and let's just start there.

Dan Maccarone: we've been working as a team in the world of AI now probably for four or five years. using it for helping us craft, PRDs and, get through our research user research transcripts to help with themes. And that's been fine. But what we ended up doing late last year was we would we've been working on a s with a startup, we were a lot of work with a lot of startups and things weren't going well with this particular client. And they weren't going well because the CTO and I was the fractional CPO of this company and I just we're not seeing eye to eye. We weren't seeing eye to eye on process. We weren't seeing eye to eye on how to build things. And it wasn't my fault and it wasn't his fault. We just there was a disconnect. And we had gotten through making the whole product basically and basically they coded something that wasn't what we designed and it really it was a really bad situation. And he and I talked and he was Look, I think that the problem is that you're approaching things in a very traditional way, which he was not wrong about, and I wanna just clod code things. And I was like, Well, we it's never a good idea in product to just start coding. It's also never a good idea and product to just start designing. we need strategy and we need a reason for this product to exist. And so I said to my team, I'm I think this is a reflection moment for us to rethink how we how we do things. And I wrote a whole basically a strategy document on our process, on how we should rethink it. And by the way, this has changed slightly since we first did it because we're always learning and getting better. But basically I was like, I think we can do all of our UX and all of our not maybe all of our design, but all of our UX in cloud code and do it in sprints if we write basically an experience brief for every sprint that can be our prompt for that sprint. And it can probably get us, fifty to sixty percent there, maybe seventy percent there, and then we can use prompts to evolve it from there. I had talked to a friend of mine who is a much more well known UX person than I am about this, who who's been using AI even longer than I have, and I suggested this idea and he's We haven't even done that, but that he's done a lot of vibe coding and he's That's a really interesting approach. And so I tried it. And basically it became something that really evolved over the past seven, eight, nine months, something that, as to how to make your UX in prototypes and connect it to Figma with design systems and component libraries. and and I find it fascinating whenever we get to show this to people because they're like they're like, my goodness, that we've never seen anything like this before. I mean I we're early stages in AI, there's a wow factor to it, but really it's changed our whole way of be able to work faster. not faster in terms of that Sprint still takes five days. But we can get so much more done in five days than we could have done, ten years ago.

Dan: Mm-hmm. And what goes into the experience brief? what is it that you kinda gather as the seed of this now?

Dan Maccarone: I'll answer that question, but let me back up and say before we ever get to the briefs, we always create a strategy document, right? And then we create a prompt to start the project. So it's like I'm working with this company with this name, they're in this industry, giving basically Cloud Code all the context it has. But then I'll give you an example. One of the companies we're working right now is media company that has different tiers of subscription. So we're like Here's all you all you need to know about like the different tiers of subscription. so if you're logged in as this different user, we need to know that these are the permissions they're granted. And then these is what this is what you need to know about mobile and tablet and breakpoints. and then also we want to create three branches of this prototype. One is a whiteboard, one is a client review, and one is approved. So you have a foundation to start. Then you start the experience brief. So I'll use the same example from this media client. Sprint one was a article. And the couple company we're working with is infusing AI into its media product as a service to its audience, right? So it's not just this article, it's information that can be relevant to you depending on your tier of subscription. So in the experience brief, we say these are all the different elements that need to go into the article: headline, byline, content, recirculation, insights into things, whatever it might be. And here's the ad placement rules. And then it can create all that in a basically an interactive wireframe very quickly. And it's part of that brief also, and and one of the big factors in this is everyone has to approve the brief before we put it in there. And so it's basically like we are not gonna have scope creep on this article. This is exactly what this is gonna be, this is what the functionality is, this is what success metrics look like. This is what we're tracking on the analytics side. So all that goes into the brief so that we can make sure that it's in the memory of Claude when we're creating the prototype. Now, on the other side of the

Dan: It sounds a lo it sounds a little like a PRD at this point, right?

Dan Maccarone: It's like a PRD. Yeah, I mean it is. I mean it's a PRD with a different name. I found that people really hate the term PRD. So

Dan: Ha

Dan Maccarone: I basically renamed it Experience, babe. Shh, don't tell anyone.

Dan: Sure.

Dan Maccarone: but here's the greatest thing about it is that so you finish that sprinter whatever. Claude can then and whatever you use. I we use Claude, but we use whatever the one we use. Claude can then create tech specs. And what the component library should be that you're gonna create in Figma eventually so that everything's tracked. So when you're handing things off to developers, you're not using the code from the prototype. That's gonna be garbage. because you're gonna revise things and it the problem with cloud code is when you revise things, it just adds more code, it doesn't remove code. you have all the tech specs, all the documentation, so you kinda have this suite of things to hand over to developers that would have taken us, weeks to craft in the past. it's really cool to look at when and when you show it to people, you're my god, that this must have taken months. It's no, it that took five days.

Dan: Right. I mean what I thought was interesting and for me coming up from the same era as you, yeah, we used to do all the documentation up front, right? you do the flows, you do the the wireframes, all the different pieces. But you are flipping that. So you put it in the PRD and then You create the prototype and then from there you create the documentation.

Dan Maccarone: Yeah.

Dan: it expands out from there.

Dan Maccarone: When I and I think what's interesting about that, and Dan, I think you nailed it with back in the day, I remember my first job working at an agency in nineteen ninety nine. I was a new information architect, I think at the time. And they gave me these like hundred page requirements documents that were written by engineers because IA wasn't really a respected industry at the time, it was still new and whatever. And they gave me these and then and then you had to translate these requirements documents into wireframes in Visio, I think we used at the time. it was so complicated. It was it was so challenging, especially if you're trying to work an enterprise software type of product. it's still complicated in a different way, because we have to figure out the foundation of it. But man, the AI does really understand so much more about the technical requirements that we don't have to say in order to make it work, if that makes sense. We can be brief about

Dan: Yeah.

Dan Maccarone: our briefs, if you will.

Dan: if you're not using the documentation to build the product, what are you using documentation for at all? isn't the product and the prototype the isn't that basically the source of truth?

Dan Maccarone: I I totally agree that the brief or the PRD, whatever I call it, and the prototype are the source of truth. But you gotta remember that not everyone who's looking at this is a product person, right? you have a founder maybe, you have a or an executive, you have a marketer, you have a developer, you have myriad people on the team who have to understand what we're creating, why we're creating it, who we're creating it for. And that documentation is there for them, right? someone new joins the team who hasn't been part of the process so far, needs to be able to see and read and consume all that, which they might put into cloud anyways, or chat GT, whatever, to get the TLDR version. But that stuff is so important because context is king. we have a source of truth. But if you think about a prototype, this is something we've actually spent a lot of time talking to developers about. The great thing about the prototype is I can send it to you or Nick here and you guys can use it and you can feel it and it feels tangible and you understand the interaction. But all the details that are in that prototype that have to be actually engineered need to be documented so there's no there's nothing hidden. if you have nuances of language, of different states of things, they can be missed if they're not documented. That was the ideal from years ago was Even though it was a pain in the ass, you could you you'd have to spend days doing ten different states of a screen so that it was clear that, you had thought about that and you accounted for it. and error states and whatever. Now those errors are all there, but you gotta document them so that someone knows that they're there and they can find them.

Dan: And I also like the idea that you mentioned the article that in a week or two you're gonna completely forget why you did something. and I'm constantly telling my students this when I'm when I'm like, yeah, you gotta document some of this stuff. Because in a month your brain is not as good as you think it's gonna be that you're just you're gonna forget why you made these certain choices. And that's you, much less somebody else out in the world who has to, pick up your work in a couple months and make it real or try to build it or just try to add to it any of those kinds of things.

Dan Maccarone: the bait of my existence is having the same conversation five times because people forget that they made decisions two weeks ago. I mean, look forget about the prototyping process. It's why I love having AI note takers. It's like, I got it I got it here. We made this choice. it's it is it is written and recorded for me to bring back to you when you don't remember that you made that decision. So I I totally hear you. And about the that that's also probably true for me and my team can do the same thing and say, Dan, you said this two weeks ago. So that's why we did this and I'm like, yeah, all right, fair enough.

Dan: I know you talk a little bit in the article about high speed mediocrity and maybe explain a little bit about what that is.

Dan Maccarone: Let's Yeah. so with high speed mediocrity I mean you can this goes back to builder without strategy. if you're yeah, you can start by coding something tomorrow, it doesn't mean and just 'cause you can put it out there, and this is classic product talk, right? Just 'cause you can doesn't mean you should. And that's what high speed media I see it so often where someone has an idea or they'll hear from a customer, we really want this feature And then boom, boom, boom, I vibe coded it last night. And but you didn't think about how that feature fits into the product as a whole. You've shoehorned it in because you can. That's that's forgetting the whole idea of why we're doing things, why we sequence things, why we don't watch with every feature we could, and why we test things over time. I think the question about whether this is successful is still out there. it's an evolving process. when we did it the first time, we talked about earlier, when I blew up blew up our process and said, guys, this is what this is what we're gonna do from now on. We've evolved even just that thinking so much since then because of what we've learned by failing, or not even by failing, by just, being wrong about stuff or being I think by educating ourselves more about, how to how to talk to Claude or how to talk to AI. And so I can't say that it's 100% successful, what I can say is that it is much more effective and we are learning how to partner and collaborate with engineers on the handoff side of it, which has been the hardest part so far because it works really well. Everyone looking at the prototype being this is what this should be. You look at design components and design and design systems and Figma, that works out really well. But there's still a handoff part that is challenging. and that's not against developers. I think they're trying to figure it out too. that's why the documentation is needed. and I think we're getting better and better at that, but I don't think there's a perfect system yet. I think we're probably, eighty five percent there.

Dan: But wait, I thought that we're beyond handoffs now. That isn't the code for your prototype good enough to just launch?

Dan Maccarone: No, no, no, no. No, no, not at all. god, no. I I think you're being glib, right? Like like

Dan: I am I'm I'm a hundred percent being glib.

Dan Maccarone: I I was I was going through the code for one of the prototypes we're working on. I think it was like two hundred and fifty thousand lines of code. and then I looked at it the next week and we hadn't made changes other than removing features. And it was then three hundred thousand lines of code because at least in Collide Code Cloud doesn't delete code, it just comments stuff out or it writes code to override things as opposed to being efficient in its code. So it's so it's so messy. there's so many dangers we're trying to launch something just by via coding it. Never mind the security issues if we don't know what you're doing and whatever. as we've seen, breaches in security already happen through vibe code products. But yeah, I I'm not a fan of that idea at all.

Dan: Mm-hmm. I think it must also vary by client too for you, right? if you're working with startups, they may be a little more fault tolerant but if you're working with a major media company or a bank or something like that, they're like, no, we we'll engineer this. Thank you.

Dan Maccarone: Yeah, I mean I have some clients who won't let us use AI for anything. one of my clients is Finra and they are just like, yeah, no, you can't even have an AI recorder on a call. And you can't if you're doing user research for us, you can't record those meetings with a AI recorder. yeah, I think with banks it's similar. I have a couple big media clients right now where we're using the process that we're talking about, in terms of prototyping. But they're still coding, the product. I but I have worked with CTOs at startups over the past year or so who definitely are using things like Cloud Code and Chat GPT to do some of the work for them. And I and I don't mean that in any disrespectful way. I mean just I'm using it to prototype things, they're using it to code things. And so that's but that goes back to that question you had about the mediocrity, high-speed mediocrity, because the knee-jerk reaction be to be like, we can just code this quickly in cloud code negates the need for strategy and UX and product thinking and why we've been doing this for so long and why we make successful products. So I think there's a danger there, especially in startups, to just start coding something and use cloud code because you can, I'd rather be let's take a second. If you want to cloud code it or use whatever AI you want to use for your code, that's fine. But there still needs to be a strategy, there still needs to be a real I think a good set of thinking around it so we all agree on what we're doing,

Dan: Mm-hmm.

Nik: Dan, one of the things that you had mentioned a little bit back was that your teams had a lot of failures along the way. I'm curious, what are some of the failures that you've had that you've then changed your process to remedy? I'm imagining some of our listeners are also in the process of changing their process. And if there's wisdom you could impart of hey, avoid this thing that we did that actually wasn't great.

Dan Maccarone: Yeah. I mean, other than fighting with Claude on a regular basis, because Cloud's a jerk sometimes, and wanted to throw our laptops out the window, there are a couple of things that we learned the hard way. So one of the first projects we did, the goal was let's make this prototype in, black and white we would wireframes back in the day, and then we'll eventually, connect the Figma design system and component library to the prototype. The problem was that we didn't know that Claude doesn't code in style sheets unless you tell it to. And so every single font call, every single color call was unique to that particular line. hundreds of this goes back to the line of code is terrible. So in order to connect that system, we would have had to redo the entire prototype because of how much bad code was in. So that was one thing we learned. a little too late on the project, but we it worked out fine. I remember I talked about the foundation prompt that we give on before we actually start building. that's one of the things that we talked about is we are gonna build this in CSS, we are going to connect this to Figma, you have to allow for that. I would suggest actually having even a fake design system that you can connect on day one, even if it's all black and white and you can change it later. the other thing we learned was I mentioned the branches we use. So we have a whiteboard branch, a review branch, or approval branch. We did not have that on day one. And that it became so valuable for us because if we're showing if I'm showing you some stuff in the VU branch, my wiper branch can be a mess. my actual whiteboard is a mess. So those are things we started adding these things on to make it easier for us to put the right things in front of the right people at the right time. and that's the same thing true, is we export it to Bruxelles for public consumption. We can export parts of it, not all of it. So our whiteboard branch never gets exported, but our Ruby branch does. So those are a couple of things we had to learn along the way because we just didn't know what we didn't know. And I'm sure there are things that we're still gonna have to figure out as we as we evolve this process. But those have made us more efficient and I think smarter as to how we work on this stuff and are able to complete our sprints. I also think that the mistake we made is I think I said this in my article, I thought we could do this faster. I was like, We could do this in three days instead of five days. And my COO was like, Dude, I'm gonna guarantee you you can't. I'm like, No, no, no, it's like we we're gonna vibe quit it. And my CO Eric, I've known him for almost thirty years and we've been working together in one way capacity or another for about twenty plus years. He's very blunt with me and he's like, Dan, I see what you're doing, I see what you want us to do here, I get it, but You're missing this piece. And he was totally right. it did still took five days. It just was more stuff packed into those five days. But that was a lesson that I unfortunately learned the hard way.

Dan: even though it takes the same am amount of time, it seems like the outcome is richer than you used to be able to do.

Dan Maccarone: my God. So much richer. we've evolved our process collectively in the world of UX and product over the past three decades. you used to have to make wireframes and with backgrounds and different states and whatever. And then and we did that in Visio, we did it in Omnographical, and we did it in Sketch and whatever. And then we used Sketch to hook up a prototype to an envision, which was its own problem. Now you can do it all in one place, And it like I said before, it's tangible. every interaction can be defined and maybe it's not high res, but at least it's it feels real. and I used to say this to clients all the time, the UX part is the boring part because it's all black and white and it doesn't feel like you're actually looking at the product and then when you get into design that's the sexy part of it. I think that this is sexy now. It's yeah, you kinda can dig in and click around and it feels like you're actually using the product that you're talking about, even if you're in the early sprints of and using part of it. so I yeah, I think that is it resonates so much better. And I don't have to sit there at the beginning of a wireframe review and be like, let me tell you what a wireframe is and this is what you can expect and this is what not to expect. It's not design. These aren't the right fonts, blah, blah, blah. Like you have some of that, but it's a, it's a little easier to get across because people can hop on their computer and use it.

Nik: One of the things we've talked about actually in a past episode was the fact that it feels like using these code agents to help us build the interactive prototypes actually helps us focus more on the interaction design as opposed to just the visual design. I know that myself, that's what I've felt as I'm doing it. I'm curious, when I'm listening to you talk. It's about of course letting the customer be able to really feel the interaction. But do you feel there's a difference between you and your design team and having this capability now instead of working only in, say, a Figma like tool?

Dan Maccarone: yeah, I do. And I think about how we had to approach interaction design in the past, right? So you could use but if we go old school sketch or Photoshop to design something, but then if you want to show how a transition would happen or how a drop down menu would work, or how what's the timing between something, you had to then go to another program and animate that it was laborious and expensive, I remember having to do prototypes or concepts for clients where it would take us days to do something because of the tools that we need to use. And now to your point, Nick, in the prompts that we're giving, it's like, yes, this is how this mouseover should work. This is how this animation should work. And it's 10 minutes versus 10 days. And that really that really does allow us also to before we show things to anyone, make mistakes. should this be five seconds? Should it be three seconds? Let's look at this. Let's play with this. does this drop down work this way? Or does this button look this way? all those things really do become faster decisions because you can clearly see where you're making the mistakes. except when Claude doesn't listen to you. But not a whole degree.

Dan: Yeah, I liked I liked your example that you have of the article of okay, give me all the error states here and put up. And it's yeah, stuff that used to take forever to figure

Dan Maccarone: Yeah.

Dan: out and put together find, figure out, put together all that stuff.

Dan Maccarone: Yeah, and document and you'd have a spreadsheet of all the errors and the error messages and you'd have to go through it manually and create them and maybe if that's in the PRD also, but it's also in the wireframes and designs as to what those look like. And here it can be documented pretty fast and AI can identify a lot of them for you, which is it's always what I love about what I love about using AI is that I don't believe it's replacing UX designers. I don't think it's gonna replace graphic designers or visual designers. or strategists. I think it's letting us do the thinking that's what my article's about. that's letting us do the thinking and taking over the stuff that the busy work we don't want to do. I don't want to think about aerostates ever I want to think about language, but you can teach AI the voice of the product, right? I'll give the brand book to the AI I'm using so it understands the words and language and colors and ideas and philosophy that we have so that when it's generating language it it's at least trying to do it in the right place. It's not gonna be right all the time, but it's a good start.

Nik: When you talk about having the AI do the busy work though, there are some arguments that actually doing some of that work or especially some of the manipulation work that we would do, drawing, whether it's physically or digitally, I don't know, I feel like sometimes we lose something and then the nature of the work. has also changed. I have found that when I use AI tools, everything's really sporadic because it's like, all right, hurry up and wait while it's doing something and then I have

Dan Maccarone: Mm-hmm.

Nik: to go do something else. And then it comes back and says, do you want me to do this? I'm curious how you've and your team have maybe felt about this. because we've talked about on the show this idea of this is not the same flow-based workflow that I might be used to when I use other medium.

Dan Maccarone: Yeah. Well, I mean, you're not wrong. I mean that it's there are a couple of things that get in the way. One is depending on what your your subscription is to these things, you might run out of credits, right? one of my UX people who I've been developing this process with, she can work for two hours and then she has to take a break because she runs out of credits and has to come back a few hours later. I mean gets she get she gets a lot of a lot done in those two hours, but The other thing you said, which I think is really actually important, Nick, is this idea of drawing. I was taught when I first started working in digital years and years ago, you always start with your pencil and paper or pen and paper, or whiteboard, whatever, you want. I've really kept that true in my head for the entirety of my career. And and I impart that to the people I work with, who collaborate with me. And we still do that, if we have ideas, I have a notebook with me all the time where I can draw something and send it on Slack or send or even upload it to Claude, which is what we do. Like we we will upload our drawings to Claude to be this is what we want this to be, so that we're not starting from whatever Claude thinks it should be. It's what we think it should be. and that's a really big part of it. And I that's not really in the brief I was talking about, but it because we're often doing hey, here are three different concepts for this particular screen or whatever. we gotta draw those out first. Now, I would say that my team is better at that than me because I have the worst handwriting in the world. everything I do is a squiggle. But but they but like they do get specific in the drawings so that we're starting from a place of our imagination, not AI's imagination. Which I don't find AI's imagination to be that great.

Nik: one of the things about Charming Robot is that you are an independent and I feel like over the last few years We've seen a decline in independent design studios. There was a big phase where I feel like a lot of the larger independents got bought by banks. And then there's still a few left around. But I've actually thought that AI-based systems could potentially be a boon for people who want to create their own independent design studio, right? You have your own craft, your own taste, your own skills. You can use these tools now. And I'm curious about your thoughts. on being an independent and also if you had advice to those who might be wanting to start an independent design studio. I recognize that maybe makes them a competitor to you, but

Dan Maccarone: No, I I welcome it. I mean my advice would be don't start an independent design studio. I I mean that somewhat sarcastically. I've I've started two, this is my second one and both of them have been luckily, knock on wood successful. But I think that first of all, I think independent design studios go through waves of success. I find that when the economy's bad, we do well because no one wants to hire a big agency. They want to hire smaller agencies. but then I also find that the people who do these smaller agencies like ours, we came from the big agencies years and years ago. We just wanted to have a bit more autonomy and not be tied to, the holding companies that buy agencies as well. we've been around for fifteen years, Shaming Robot, and we've had multiple acquisition offers and we've always turned them down because of how we wanna be independent. I don't have an answer as to how AI makes independent studios better or worse. I know that if we're not leaning into AI, we will become irrelevant everything we do, especially this is true of independent studios and especially of how we are product folks, if we aren't thinking about what's next before everyone else's, then no one's gonna hire us. someone asked me the other day, like, what training do your does your team have on AI? And I'm like, We started building AI products in twenty twenty one. we've been doing it before anyone else. And the best parallel I gave, and I don't think this is in the article I that we're what we're talking about, but I think there's another one I published in the past week or two about this, where if you look at the ways of the internet, dot com one point you look at dot com two point you look at mobile, you look at crypto and whatever. I think mobile is the best parallel to this, which is none of us knew how to design for mobile when the iPhone came out, but we had to learn pretty quickly because everyone needed an app or everyone needed a mobile and we didn't even know how important it was gonna be. for that mobile web version of something, right? Just in a web browser. It was so important for us to learn that stuff so fast and probably be wrong about a lot of it, but at least get ahead of everyone else. And so I think bigger companies have problems with that because even though they have bigger budgets and more time, money to spend, there's also a lot of Kool-Aid that has to be dragged from their clients where we on the other hand push back very, strongly on things because we really dig in we do so much user research on stuff. And so with AI, I really likened it to mobile because we didn't know how fast it was going to change everything, but we had to lean into it so so incredibly hard and I think intensely. and I think for the right reasons. I'm thinking I think we have learned a lot and we've been lucky to be able to be experimental like that. a bigger company I don't think have that flexibility of being experimental because of the types of people they work with. yeah, that's how I how I think about it. I don't know that answers your question, Nick.

Nik: Yeah, it does. Thank you. Dan, if people wanna read more of your work or if they're interested in seeing more of what the studio does, where can they find you online?

Dan Maccarone: Well, I'm at Medium at and my username is Dan Macron, so easy to find me there. Also on LinkedIn, same thing, Dan Macaron. and chumingrobot.com is our website. I have a podcast called Story in a Bottle, where I talk to people at all walks of life, from product designers to Tony winning actors to comedians to founders. I recently had Nolan Bushnell who founded Petarion. and everyone and the whole thing is I get them drunk. that's the other thing is I get them drunk to spill their life story and secrets that they wouldn't tell anyone else. It's a hundred percent true. And we often do it at a bar that I own in New York City. Called Fool's Gold.

Nik: All right. So yeah, if you wanna if you wanna catch all of those things online or head out to New York City and potentially run into in person, now you know where.

Dan Maccarone: Come in. Yeah. also I have a book. I have a book. Sorry, I should I should play the book too. my book is called The Barstool MBA. It's on Audible and it's about the parallels between running bars and starting a Starbucks.

Nik: Awesome.

Dan: All right, we will link to all these things in the show

Dan Maccarone: Awesome.

Dan: notes.

Dan Maccarone: Thank you so much, you guys. It's go so great to meet you and I hope we get to have more than this conversation at some point.

Nik: Yeah, definitely.

Dan Maccarone: Awesome.

Dan: And great. And thank you everyone for listening. And we will be back next week with more AI and more design. We'll talk to you then.