Dan: Welcome to AI and Design, where we explore how artificial intelligence is reshaping the world of design. I'm Dan Safer.

Nik: And I'm Nick Martillero, and we're faculty at Carnegie Mellon's Human Computer Interaction Institute. Each week, we break down the latest AI developments, dive deep into topics that matter to designers, and talk with fascinating guests who are right at the intersection of these fields.

Dan: Whether you're a designer working with AI or an AI practitioner interested in design, we're glad you're here.

Nik: This week, Mobbin releases its MCP, Perplexity and NVIDIA announce an all-local AI agent, and we talk about David Huang's article, The Ledger of Design.

Dan: Well, before we dive in, I have two upcoming speaking engagements that listeners might be interested in. I'm giving my talk about the six futures of apps at Chicago Camp's Leadership by Design Conference on September 18th. It's a remote conference, extremely reasonably priced, lots of great speakers. That one's a super easy one to pop into. The following week, I'm in Brooklyn for Shift UX, which is a Hybrid remote in-person conference about changing the conversation around AI and design. I'm teaching a workshop called Designing AI for Humans, and this is the first in-person workshop that I've taught in several years, so I'm excited about that. And I'm also giving a new talk called Tuning the Turing Machines as part of the human-centered tooling cluster of talks. My talk is going to be making the case that we should consider designing tools like instruments, like a cello, and having that be our design target so that people's capabilities compound with use instead of the AI absconding with it. We judge tools by what they make, we judge instruments by what they make us. And that's the big theme of my talk. I'll put my discount code in the show notes. but let's get into the articles. Let's start with Mobbin's new MCP server. I had never really heard of Mobbin, had you, Nick?

Nik: Yeah, I've actually heard of Mobbin and mostly because it is one of the sponsors for some of the software development YouTubers that I watch. And they're often, you know, saying things like, Hey, software developers, check out this really awesome resource to look at designs for applications so that you can get inspiration. in many ways it's sort of a a database of app designs. That you can query and look up to get information about and actually to see the screens. Basically, it's a search engine for design screens.

Dan: Yeah, and apparently like two million people use it. So I am just I'm I'm out of the loop. but yeah, it has like over a thousand mobile apps on it, it's got like 200 sites, and they just launched the mobbin mcp and that connects your AI agents to 600,000 plus real product screens. So you can prompt your AI with commands like, hey, find me an example of e-commerce product pages that have a floating ad to And Nick what do you what do you think about this?

Nik: I mean, I'm a big fan of any kind of data repository being opened up to agents via MCP servers. I think that this is a potentially really cool thing because as designers and really lots of people work more with a code agent like Claude Code or Codex, the ability to pull this information in using the code agent as opposed to having to say manually go into a search. probably is really, really useful, right? The more I'm spending in sort of my tool and the more connections it has, the the more I can maybe stay in flow and say, hey, go off and do a search for these types of things, like add to cart buttons, and then you know, I'll go work on something else. And then you come back and give me results. I will say that one of the things here is that it's it's part of their paid plans. And so you definitely need a paid plan to do this, which I think makes complete sense. You know, it's gonna cost them money to to host and run this. But I could imagine this being a really good business move for them because lots of people might say, wow, I can point my agent and say, go look up the best things that I you can find on this design kind of area, and then come back with three proposals for me. And it'll just work on those based on the say inspiration.

Dan: And I guess what's a differentiator for them is that they have all these outside sources. They have tons of product screens and stuff like that. And so they're not completely reliant on whatever was built into the training set for, say, Claw Design or Figma. They're going they're going far outside that. Now I guess the question is, will Claude Design and Figma basically build this and put it into their own products?

Nik: Yeah, possibly. I mean, I think that any time you you see a feature like this now, there's always a question of will they will they build it themselves and take it. that being said, I think that Mobbin's done a a lot of work already. I mean, they have a huge platform that they've been running for years, that allows people to do this just in a manual way, right? As a search engine. And I think they have a big head start. and so I think that Being connected with an MCP server is probably something that they have solved a problem well enough and hopefully for cheap enough that, even the Claude design team might actually say, like, that's cool, we'll just point our server at this, we'll pay the we'll pay the monthly fee. As opposed to, I think one of the differences was say Claude Design and and Figma was that the designers at Claude we're trying to kind of rethink what it meant to design really in this agent way. And of course if you are limited by the the tools that you have, then that's going to potentially prompt you to build a new tool. In this case, this is a more about like information retrieval. I, you know, the MCP server might have some things of like say summaries or finding the best designs, but I imagine really what it mostly is is about pulling the the reference designs for you in a more semantic, you know, way where you can describe it in natural language, but it's maybe not doing so much work for you. you're still using the agent however you want.

Dan: Right, one of the things that they tout at Mobbin is that they have all these things and they're very well labeled. And so it is easy to get and find them. Where anyone else who is trying to do this would have to build all that up, and that's a considerable effort. And so maybe maybe they're an acquisition target. Who knows?

Nik: Possibly. I mean, I I'll say I would in some ways I would hope not, just because, you know, they

Dan: Ha ha.

Nik: they as an MCP server, right, they're they're widely available to whatever agent you want to use and they show you right on the page, right? You can you wanna use Claude code, you wanna use Codex you wanna use open code, you wanna use any of these different things. Like I think that being available to more people is better. Like I like that. And I think that hopefully that works for works for their business. But yep, I mean they could they could definitely be an acquisition target.

Dan: Right. Well, as we've talked about here on the show in the past, it's not a given that any one of these tools will last for more than a couple years and before something switches. And in this era it it may not be a given that it lasts for more than a couple months before something new appears. So we just we just don't know. So having a resource like that, I think is Like you said, Nick, I think a really interesting to have a third-party resource that all these different tools can connect to. Now let's jump to perplexity. We've had a couple perplexity stories over the last couple months, and I'm starting to think I need to get on the perplexity train. Do I need to do I need to get Another subscription to another AI service? I don't know, but boy, some of this stuff has been really intriguing and they have something new. Nik what is it?

Nik: Well, actually, Dan, you may not have to get another subscription in this case. You could probably just buy a new computer. so what this story is about is that Perplexity and NVIDIA have partnered to introduce Perplexity Computer, which is a local AI agent that runs on your own machine, your actual computer hardware that you you own and is physically with you. And so the idea here is that Perplexity had computer, which was their sort of agent system. In many ways, it was sort of their answer to open claw. and it it was a way to have an agent, that will go out and do stuff more automatically for you. Like you could really automate lots of tasks. But that actually ran on Perplexity's servers. Like you paid for the subscription and it ran on their servers. It didn't run on your machine. You you controlled it from your machine, but it didn't run there. But now they've partnered with NVIDIA to get portable computer or basically this agent harness working on your local machines if you have very high-powered and expensive NVIDIA GPUs like RTX GPUs or even NVIDIA's DGX workstation, which was which is kind of like NVIDIA's answer to the Mac Mini. It's a tiny little workstation with a with a a big GPU that's good for local inference. Now the benefit of something like this is that it eliminates token billing because you're not hitting a cloud Basically you are just running the models on your own machine. And so it just costs you the electricity costs of running that machine in your in your business or your home. In addition, it also allows you to run completely severed from the network in the internet. It will run locally. Now, you might have aspects of your agent that go and do things like web search, but let's say you had sensitive company files that you wanted to work with, and you had an agent that was processing them, you know, summarizing them, pulling information out of them. You could use something like portable computer on an NVIDIA-based machine to process all of that without ever connecting to the internet, ever sending that data over the network, or have it hit anyone's servers. And so there's some some benefits there. You know, I know for example, I have a lot of friends, and myself included, that I'll do I'll use a local model to process my email. You know, I don't actually want to send my email even though if I use Google, they're gonna process it all. But if I want to build a a little tool for myself, I'll actually do it often locally. So Portable computer packages, a bunch of things. you can choose different models. so folks out there may have heard of things like Qwen which is one of the Chinese distilled models that actually runs on much, much lower powered hardware, and you can run it locally. NVIDIA has a number of models called Nemotron that are of this class, and so the idea is like you could also run the Nemotron models. and so you get models, you get tools, so lots of different software tools that can come with this and a security. Secure sandbox to kind of run portable computer as an app on your machine. And yeah, this allows you to basically sort of get away from the cloud billing model as subscription, or if you're paying in pure tokens, and actually run things on your own machine. Now it is only for Linux users now. And eventually will be for Windows users. And I will say that this might not be as relevant for most people in our audience. I imagine most of you out there, like Dan and I, probably run Apple Macintoshes. but

Dan: Wait, wait, wait. So are you telling me that this is finally the year for Linux on the desktop?

Nik: man, is it the year on with Omarky and and portable computer? I mean, yeah, 100% it is the year of the Linux desktop. I don't I don't think so. I'm I'm sure that for for maybe many of our our listeners, you're not maybe gonna go run out and buy a Linux-based machine to to run something like this. There there may be, of course, you know, folks who are on Windows, and I imagine there's lots of folks who are on Windows, you know, for your company and stuff like that. you'll hopefully get this soon. But I will say that this sort of trend and excitement about local AI is something we're also seeing a lot in the Mac world. Apple actually last week just released pre ordering for their new Mac Mini and Mac Studios with the M5 and M6 processors. And one of the big things about those is that they have a bunch of the memory, the RAM, is shared with the GPU, meaning if you get high capacity systems, you know, 32 gigabytes, 48 gigabytes, 64, 128, you can run many of these pretty performant local models now, things like the new. Models of Qwen things like Meta's Glimmer, you know, coding model. and they work pretty well and pretty fast because it it turns out that the series processors are incredibly good at running LLMs locally. And so I think there's a lot of people who are really excited about this, and I think in the next few years. if you're updating your machine, you might consider getting a higher memory system. I know that it costs an eye-watering amount right now, but hopefully they're gonna figure out the supply chain stuff. And in the future, we're we're just all gonna have machines with 128 gigabytes of of RAM. but because I think what it means is that people are going to be able to start building a lot more around utilizing these local AI models, which can be more cost effective. for users and again it's a little more private and and secure.

Dan: Yeah, from your mouth to God's ear about the prices. I I think that may be a a a long term play and maybe this is this is like oil. Let's see how high it can go before people really, really complain. And we'll see next week at the Apple event how they're pricing the new phones and new hardware because that's That's gonna be a big signal about how pricey things are gonna be going forward, I think. But yeah, I I'm very interested to try this. Yeah, basically is a heartbringer of what to come. I mean, the rush on the Mac Minis harkens back to the ClaudeBot back in Back in March when that was like a huge thing. And yeah, I wonder I I mean, obviously it's gonna be a pretty small group of people who are going to pay for a second computer just to do these things for them. But I wonder hopefully we will get to the point where some of these small LLMs can run on lesser hardware and I think that opens up a whole new world of personalization and customization and and you know wearable agents that you can use to walk around with.

Nik: Yeah, and I think for the next couple of years, we're probably gonna start seeing some interesting explorations and competition between cloud-based, AI built-in services and potentially some new experiments in local AI running on your machine. Now, one of the biggest challenges from a user-facing design perspective is that. We as designers can't control the hardware our users have. And so we have to be designing for all of the hardware. And so it it right now it just it doesn't make it much sense to try to run local AI in in apps that we build, because I don't know if you have a 16 gigabyte machine or if you have 128 gigabytes. And it and I can't assume any of that. And so by of course running it on the Claude, I don't have to worry about it. Like I control and our our tech ops team controls, you know, how much we use and we try to make it cost efficient and stuff. But I could see, for example, from a from the perspective of like design tools, you know, for for the the things that we use to build. if it's really costing you a lot in token-based spend, you know, while you're while you're developing new design concepts and some of these newer models are actually pretty good at at doing, you know, at least one or two tasks, this is where I could see local models and building local models into tools being valuable. Or people sort of doing it on their own. I actually imagine what we're gonna see here is probably much more exploration in the open source source space. perplexity being sort of the one, I think, industry leader here in in trying to get you a little bit more of a polished local AI experience.

Dan: I'm wondering does it make a difference what the kind of tool is? So for example, something like Figma, could some of what Figma does currently in the cloud be pushed down into a high powered machine so that things like small changes that you are making, could be done right on the canvas and and locally rather than this expensive trip b back and forth to the cloud

Nik: Yeah, definitely. Actually, one of the neatest strategies I've started to see among the software development folks out there is that they'll use a really big model like Claude Fable and they will have that as an orchestrator. That actually can call other models like tools. It can basically create subagents and call those. And one of the folks that I watch actually is like, I taught Claude how to use Codex because the Codex was super efficient at certain tasks like back-end development tasks. And so they had Codex do all of that. And actually one of the things they started doing was also saying, I I also have machines that can run local models. And so for really small things, I just have it hit the local model. You know, do a lookup here, make a change here. Like all the things, for example, when we talk about say like the d direct manipulation, and actually many of the apps, when you do a direct manipulation change, it actually says, okay, now here's a prompt to tell the code to make the update. Those are the kinds of things I could totally imagine working really well on a fine-tuned local model where you can sit there and really do that and hit it, hit it, hit it. And then when you come and ask a question like, all right, now I want to do an exploration of four new design concepts, it goes, okay,

Dan: Right.

Nik: we're gonna hit the cloud model for that because you definitely don't have enough there. And so I could imagine some interesting sort of system architectures that start to call local models for specific tasks to just help reduce cost and latency.

Dan: Yeah, one of the trends that I've seen lately is is this idea right now so much of so much of picking the model is a manual thing. You know, unless you go for just the default, you're you're s choosing the the level of effort, you're choosing the model. And for most people, that's like, what you know, I what's the difference between these models? I don't know. But we're starting to see a lot of more of these coordination, these orchestration things appearing on the back end that are doing exactly what you said. Like the movement between different models. Now I haven't seen what you're talking about, which is moving like here's a whole different company's model. I think that's that's pretty interesting too. And yeah, w once you start adding in the local mix into it, that's gonna be a really interesting area as well. So lastly, let's discuss the concept of design ledgers. I hadn't really thought of a concept like this, but now that I have, I think I'm gonna start requiring students, especially to set up a design ledger. So this idea comes out of designer David Huang's article called The Ledger of Design. And in brief, the concept is this: that we have gotten really good at producing a lot of work quickly. What we have not gotten very good at is making progress about recording that work, especially decisions that were made and the context that fuels the ultimate outcome. But right now, we have an abundance and a slop problem. And our teams can produce more work at higher volume than we ever could before. But the process behind the work, the thinking behind the work, becomes easier to hide, harder to change, harder to go back and discuss. And so we get all these finished looking outputs that are cheap now, but There may not be a lot of thinking behind that. There may not be a way for people who aren't in the room when you're writing the prompt, which is probably a lot of people. And that's where you need this idea of a design ledger so that you can basically show your work. here are the ways that this that this concept came together. Here's what I was thinking when I put behind it. Here's why I think it's a strong concept. those kinds of decisions and here's the prompt that led to this concept in case people want to go back and do variations on it. What do you think about this concept, Nik?

Nik: I think this is an interesting idea to consider, especially given the challenge of we are producing so much now. We can basically it it's almost effortless to to produce a design artifact, but Who knows the thinking that went into it. But this has been a a challenge for a long time. actually, capturing design rationale is a really hard thing. I know that designers often will talk about, like, it's very important. We always need to have rationale, and capturing the rationale is, critical. And then when you ask how do you do it, they're like, well, you know, kinda. Maybe write some stuff or I create some slides or do this. And and actually it's really hard to do. So it's it's it's a known thing that's important and yet it is still really hard. And in theory, AI could potentially help us, but I actually think right now we don't have very good models of how while doing design work with AI, should be capturing your your intent and your and your rationale. Because I think that from a software development perspective, right, there's a lot of people who say, well, the code is self-documenting. And actually a lot of the the aspects of your decisions can be written into the code. even though I don't know if that's entirely true, and you may still need to have documents or some type of ledger outside of that. But git commits are really good at that. Actually, you can you when you commit things, you can write a little thing of this is the decision I made, here's why I did this, and here's the change. But for design, right, we we first off we have it actually we don't even have anything really like git commits entirely in a lot of our tools like a Figma or Photoshop or Sketch or anything like that. now though we maybe we do have a lot of that because we're working with code agents, but we might not be doing that.

Dan: Yeah, and basically what you just described about using AI to create this is basically the the solution that he recommends in this article. He says, coming from the tools people already use, you know, the documents, the commits, the prompts, agent runs, artifacts, and he points out, I think rightfully, that if this stuff isn't easy to capture, then it becomes bureaucracy, it becomes busy work, it takes a lot of effort, and you may do it like after you're done, and you may f completely forget how you did something or why you made a certain decision. And so I don't think it is just as easy as, keeping a list of all the prompts you made or those kinds of things. So I feel That feels facile. I don't have a good solution to this because like you said, this is something that has been a problem for a long time, but it's now a real problem in the fact that so much stuff can be produced so quickly and it all seems so polished, but you can't go back and interrogate the thinking behind it.

Nik: Yeah, this is actually an area that I have lightly explored. Actually, Brad Myers, who is a faculty member here in our department, focuses primarily on software development tools and sort of the HCI of developer tools. And we ended up working together for a little bit trying to actually think about how to capture design rationale. this was a few years ago. And there were like MCP servers in Figma we would be able to build with. And so we actually conceptualized a little bit of of AI agents maybe trying to capture rationale, trying to ask and elicit rationale from you so it you could capture it quickly. But even us, we we ended up putting the project down because it just got really hard. It was just it was such a challenging thing and we just didn't feel like we had good good answers for it at the time. And we ended up deciding that we were gonna work on something else, you know, things that maybe were it would just be a little more fruitful. But I actually think that reading this article again, it it it brings up sort of questions that we had. What is a good ledger? How do you actually get people to do the work to get the thinking out of their head and into something which is a a document or some type of, captured resource on this. I don't know. I mean I I would be interested to see, how people are thinking about doing this. I know that within something like when you're using a code agent, sometimes they will actually write design decisions. Like a code agent when it asks you questions and you answer them, it will it will write those answers and like some of the decisions you made. But that might not also incorporate the why. Like you might be using the the little question answering system Which is the multiple choice, and you're just selecting quickly, but you're not actually saying, and this is why I'm choosing to go with this direction.

Dan: Right, well that's what I was wondering. Like could that could this be an add-on to start to make a log basically of design decisions? Okay, like I'm giving you a menu of these three options. you're picking this one. Can I ask you why? and maybe then it just is another prompt for you, like, hey, why are you choosing this? Pick these three options. could it be another step? in that process. Now obviously that is adding some friction to the process, which I don't think either of us think is necessarily a bad thing. I certainly don't, but when you're trying to work on a deadline and stuff like that, it could become annoying. But you'd have to show what the value of it is. you'd have to be able somehow to go back and review the ledger or the ledger actually helps the AI start to design better things because it's designing things like you would do them to talk about tuning the Turing machines, making the AI a little bit more like an instrument that you're playing versus something that is feeding you all the answers, that it's it's working back and forth with you. I think having to show the value of of why you would be doing this is hard I think in this era and maybe in every era of going back and documenting things. Who it it's it's a rare set of people that really enjoy that.

Nik: I think though, Dan, you're making a really interesting point on the utility of the ledger and the utility of the capture of rationale. Because if you can solve where it becomes most useful, and it might be because AI agents find it very useful, that may be the thing that actually motivates the field to to start developing systems that capture these types of things in better ways than we've done before. Cause I think one of the challenges in the past has been that when is the rationale useful? When someone asks me a question, like why did you do this? Cause if I give you something and you're like, wow, that's awesome. I love it. They're they don't That's it. I don't care like what why you made these decisions. It's great. I like it. Or if I don't like it, you know, I'm it's it's I'm in a meeting and I'm asking you a question, you know, why did you do it this way? Can you explain this to me? I mean, users don't, users don't do that. So it's also a very internal process, right? It's a very business-oriented process, which always ends up feeling a little bit like, more work. And again, when done poorly, it feels like make work.

Dan: Right. And I don't think that let's put it this way, I have my doubts that people are going to do this out of the generosity of their heart to improve Anthropic's model. Like I don't see that happening. But if it can improve my work or my team, that's where I think that it becomes valuable. Where it it where my version of Claude or Is starting to learn about me and then maybe that is also something that I can share with my team internally and and maybe that all improves the work because it is starting to understand how I personally work. And I think that's That's that gets past the the altruism because it's it's helping me personally. It's selfish it's a selfish thing. And I think if we can convince people to do that, then I think it's more likely to succeed. It is very hard to Be like, well, I'm gonna make this because it may help another designer two years from now, or it may help my future self, because we always screw over our future selves by not doing the the right thing that we should be doing doing. Like, I'll have another beer. That's tomorrow's problem. You know, those kinds of things. So I think that this is the case here too. it can't just be a ledger for a ledger's sake, it has to be a ledger for improvement.

Nik: Yeah, It's gotta do some kind of work for me. So I'll I'll say though, there has to be some job to be done here. but it is it is a real thing. Actually, just today on the product that I'm building, I made an update. I just pushed it and one of my teammates was like, Hey, I really like what you did here. That's really cool. And then this one I'm I'm I it's not bad, but they're like, Why did you choose this? And I'm sitting here being like I don't remember exactly like or like I kinda I kinda remember it. It wasn't a willy-nilly decision, nor was it a complete it wasn't a decision that I I left to the AI to make, right? The rationale isn't only in the AI, but actually, man, I really wish that I could just pull it up and say it's here, or like please look at the log or please look at this. This is where my thinking is. Or literally if I could just have, an agent living in our Slack channel. That instead of asking me, it goes and asks Nik's design collaborative agent that he's using to do all this work, and it comes in and says, here was the decision that we made yesterday, into your point, and I can just answer your question without me having to go and search my logs for it.

Dan: So this was the first week of class last week, and right out of the gate, one of the first questions that we got was, well, how and this is in the advanced interaction design studio, it's like, well, how do we start to develop our taste? And I'm starting to think maybe something like a ledger can help us develop our taste. When we're able to look back at the ledger or the AI itself is able to pull some insights out of the ledger about some of the decisions that we've made that maybe subsequently have turned out to be bad decisions? Like, are there ways that we can use this as part of a reflective practice in the Donald Schoen vernacular? To improve our tastes. Like, is this something that we can be part of part of what we do in terms of getting b getting better and not only helping the AI get better but improving our taste, improving our judgment based on decisions that we've made in the past that haven't worked out well or maybe have worked out well. Like, hey, you know, these color schemes that you keep picking are amazing. Or this flow you keep going this way and it keeps not testing well. we've need to redo this three different times now. what's going on here? those i think those things could be really really interesting particularly when you're starting new iterations and being like well what changed between last point and this point that feels like a a great point to reflect upon like well this thing didn't work or this doesn't really work and Really getting into more the why. I think the AI is gonna be great about documenting what happened and the steps, but getting into the why, that's where stuff gets lost. And I think that that is something that we could use AI to help us reflect on that and help improve our taste and judgment.

Nik: Yeah, I love that idea. And I actually think that that would be super exciting to to see some experiments out there. So, you know, hopefully if people are inspired or maybe you're already out there working on something like this to capture your own thinking. If you are, maybe yeah, send us send us an email to share what you're doing with us. but yeah, I'd love to see more systems out there trying to help people keep a a ledger.

Dan: Or if any company wants to sponsor a research project on this, we're all ears.

Nik: All right, so that's our episode for today. One last thing before we go. we are just shy of a thousand followers, and we would love to get into four digits. so I know we don't do this a lot on the show, but actually if you would take a moment to rate and review the podcast, it would be a big help to us. That is how the algorithms sort of start sharing it with more people. so yeah, thanks for being listeners. for those who are followers, And if you've got any ideas or stuff remember you can reach out to us. and we're happy to hear from you.

Dan: We'll be back next week. More AI, more design. We'll talk to you then.