Dan: Welcome to AI and Design, where we explore how artificial intelligence is reshaping the world of design. I'm Dan Saffer

Nik: And I'm Nick Martelaro and we're faculty at Carnegie Mellon's Human Computer Interaction Institute. Each week, we break down the latest AI developments, dive deep into topics that matter to designers, and talk with fascinating guests who are right at the intersection of these fields.

Dan: Whether you're a designer working with AI or an AI practitioner interested in design, we're glad you're here. This week, taking a look at what Apple launched ⁓ ⁓ last week and what that means for designers.

Nik: Airbnb's CEO is starting a new AI lab, but it's not focused on what you think.

Dan: Sarah Gibbons names the four design jobs that AI has created. Is AI Podcaster one of them?

Nik: And finally, we discuss an article from Perplexity Research on how AI agents reshape knowledge work. But first, let's get started with what Apple launched last week at WWDC, which is their worldwide developer conference. There were a lot of new announcements, especially with a big focus on Apple intelligence and the new Siri AI. addition to sort of these platform announcements that often are very consumer facing, so very exciting to see what. are going to get on their new in the weeks and months. There's also a of really good information as part of WWDC for designers and developers to learn about the new tools that available to them to build their app experiences So big news of course was that they launched a long-awaited update to Apple Intelligence and Siri with Siri AI. ⁓ Which will be a lot more conversational. It'll have more to access what they call world knowledge, where can access information in your messages, your mail, or on the screen. And they're promoting this as they usually do with a lot of privacy considerations, also touting their private cloud compute, which is their own cloud compute solution, which doing AI work on their systems. ⁓ and with privacy in mind rather than sending that off to say third party model providers. this includes even their their partnership ⁓ with Google, They're working on a way to ensure that customer data is still private, even when using services. For example of the kind of demo that gave, there was a cute one where they where a user about, hey, what was that cookie recipe that Maria was messaging me about? And Siri AI can then search your messages, pull up that message, and then conversationally says, Hey, you want me to pull up a recipe for that, which it could then search the web for, or would you like me to draft a message to send back to Maria to ask about the recipe or ask about the cookies? So this is a lot of what I think we have been expecting since Apple Intelligence was introduced, this it does appear that these demos are And so that's We're finally kind of getting to the point there with what Apple Intelligence's was supposed to be. We're getting that.

Dan: one of the things that Apple is been known for for decades now is privacy and trust. And this is a time to probably lean into that especially when you're talking about tracking user behavior. And ⁓ Apple is instead trying to be like, nope, we keep all stuff So I think that's a smart move on their part. but this also the question, can other app developers opt into this so Siri AI can look into them? So can Siri AI find things that are in a Notion page, for example, or something in for me?

Nik: Yeah, so this is something that is coming up developers start to integrate these features into their own applications to give access. So pulling actually from a newsroom article that we'll link to here, it says with the latest enhancements to Apple Intelligence and the introduction of Siri AI, developers can make their app's content and capabilities more discoverable and accessible across the system. And the way that they're doing this is through app intents. There's at the App Intense framework, basically ⁓ app developers to connect things and Siri AI the understanding, information, app actions, all these things. But basically you have to be using App Intense. Now this is a little different from say, I'm just providing say an MCP server to the built-in Apple intelligence frameworks. So that's a little from how we're seeing, for example, integrations between OpenAI or Claude with all kinds of apps. Basically, app developers are gonna need to start using these sort of these app intent ⁓ framework to start updating their apps. we won't see as much this until these apps become updated with the new app intent features.

Dan: if want to at all, for example, Spotify might say, it. I'm not letting you in into Spotify. you have Apple Music, go do that.

Nik: companies may want you to use AI that's baked into their product and the experience, which they could use either Apple or other third party providers for. So there's an interesting question of control here, right? In the same way that when you open up an MCP server and you allow, your Claudes and your open AI chat GPTs of the world to interact with your app, you're kind of seeding a little bit of that interaction design to the frontier model developers. ⁓ same thing here. You kind have to choose like do I have built-in intelligence features within my app do I actually open it up ⁓ for Siri?

Dan: I was thinking you could use for Amazon, but then I remembered they killed Rufus. Rufus is now just Alexa, so pour one out for poor old Rufus. I don't think Amazon would l would allow this anyway.

Nik: Yeah. ⁓ so a couple things here. Actually, this this brings up a good point. One of the things about WWDC that's important is that this is the developer conference, and so there's a lot of other content that's out there on sort of new tools and ⁓ frameworks, platform features that are really on the designer and developer side of things. a couple things to, maybe highlight if you want to have some guidance on where to look on the specific presentations that they gave. If you are building agentic and AI experiences, there's a lot that looks like it came out on how you're evaluating how good good that is. And so they have a new evaluations framework. There I saw a lot of videos there on that. in addition, ⁓ if you're using ⁓ to develop your apps, ⁓ they have some new agentic capabilities that can do UI prototyping, where you sort of in the same way, like a vibe coding style of things, you can say what you want and it'll start laying it out. I actually think that would be really nice because unlike say, those of us who work in more of a web-based frameworks, ⁓ who have maybe learned how to do our our interaction design there, the Apple UI frameworks are a little bit different. and so actually this could open up native app design to more people. This has been something that where a lot of vibe coded or even just agentically coded things are not ⁓ utilizing native capabilities as much. Because they're built on web framework type things. So that's kind of cool to see. lastly, maybe just as a as a note, ⁓ because I saw this in one of the articles that I didn't even totally expect, ⁓ which was that you're gonna need some new hardware. the newest Apple intelligent features are only gonna work on devices like the iPhone Air, iPhone 17 Pro, M4 iPads, and M3 Max, each 12 gigs of memory or more. That basically means that if you don't have the device, you're going to have a stratified user experience. So if you are starting to use these more Siri AI features, your users have to have the newest devices to able to access them. If you have older devices, they did say that Apple Intelligence will work, but just not with the newest best models, and likely probably not with the best experiences. Just something to note, I'm sure that everyone is excited to go purchase some new hardware.

Dan: And by newest, you mean like this year's models and next year's models. my poor ⁓ iPhone 16 Pro left out this. It's like a year old. It's I yeah, it's a little bit a gut punch to me. I won't lie. So sadness.

Nik: Yeah, and again, I think that the really the interesting thing here is it in this sort of space of stratified user experience, as you start to build these features, this is one of the things where if you're using cloud-based, models, you didn't have to really worry about. I mean, someone's device might not be super fast, but in theory it would be able to render and show these things. So it's interesting to see, there's pros and cons of using, say, the on device machine learning capabilities. I think that Apple still has a leg up here, because of the devices.

Dan: Yes, but first, let's roll back. a year ago, people were like, Apple doesn't know anything about AI is cripple And this year, I think because of the landscape and because they have the hardware, They might just pull this off. You've got OpenAI and Anthropic that are shifting towards enterprise, they're doing coding use cases and stuff like that because that's where the money is. Meanwhile, Microsoft is not really at consumer products. It's looking more at the stuff that we talked about last week, which is Microsoft 365 and enterprise tools on their side too. Meta is thinking about its own smartphone ecosystem and the the Ray Band glasses and stuff like that. nothing really happening there of of too much note. And then Google, it's one big competitor. It's got Android ⁓ and it it's also getting Warby glasses and stuff like that that I'm very interested to try. But is also now feeding Apple the foundation model here. So that's probably pretty good for them. So they're actually looking like they're gonna be in pretty good shape. Apple is well, these other people are doing this stuff in AI, but we are actually gonna make AI for the rest of us, the kind of Apple, it just works. They're going for simple. They're gonna treat Siri as this centralized way to handle all these AI flows across your devices, and it's gonna be with a of trust that these other companies just can't match. So I think that this strategy has ⁓ somehow worked out for them. ⁓ And great. I keep thinking about this is the AI for the eighty percent who aren't really using AI tools now.

Nik: that leads us a little into the next story here, a short one is that Brian Chesky, CEO of Airbnb, a designer-founded company, is reportedly developing an independent AI lab aimed at crafting novel models and reimagining user interactions with AI services. ⁓ while other tech executives are focusing heavily on conversational AI, ⁓ chat based AI really focusing on sort of the intelligence component of it, Chesky is prioritizing the design, user experience, and interactive systems as the driving force of the next wave of AI innovation.

Dan: What do think this is? Is it is it hardware? Is it software? Is it just baked into different kinds of interaction models? I mean, I'm all for all of that, really. I know that OpenAI is definitely looking into the hardware space with ⁓ Johnny Ive ⁓ and that'll turn out better than Johnny's Ferrari did. But I'm pretty dubious. So I wonder what this is. I mean, literally, this is this is all we know. It's top secret. the only other piece of info here is that he's not gonna be the CEO of it. Somebody else is gonna do it.

Nik: I am not sure. first off I will say I was excited to see the the concept around user experience being the driving force behind this frontier lab. I could imagine there would be exciting ways of exploring how to design new user interaction with frontier models. I guess the question there though is like I'm not sure and if you want to start a lab and do all this, that's awesome. the question is is is this gonna run like the frontier labs with a product, i.e. the open AIs, the anthropics of the world where they're still selling something. Like ideally they were a lab that was focused on developing new capabilities and that ultimately is making its way into products, which get more usage, but that they have something to sell. And the thing about this is I'm not totally sure what there is to sell. Like do a lot of this kind of work here at the HCII and many other HCI labs across the country are really excited about the way in which AI can improve and change user experience and interaction design. But a lot of our work oftentimes our goals are to share what we learn with the world and then hopefully have designers out there like yourselves pick up and employ in any product. ⁓ we're trying to figure out how it is that people can utilize these new and techniques. And so when you talk about this as a lab model, it suggests to me you want to do something like that. but if there needs to be a product associated with it, and this was something that I read and one of the short about this, was that it was like, ⁓ it that this is gonna drive the experience of all kinds of different apps. The question is, is are they gonna build these apps?

Dan: Right. Are they or are they going to it open source and give out components or give out their models? What yeah, what are they doing? I'm wondering if they're gonna be like thinking machines, you know them? ⁓

Nik: yes, I think that Thinking Machines might be a good comparison. So Thinking Machines, this is one of the companies that actually spun out ⁓ from some of people who left OpenAI. This is Mira's company. and their focus was a lot on interaction. Actually, they had a pretty cool we'll link to it in the show notes where they were showing us a voice-based interaction that was super fluid, right? Back and forth. it could see what was on the screen, it could see what was in the world. so it was actually at like if you had a camera and it was looking at you. and it seemed like they were focused a little more on really the very fluid interactivity, getting away from still conversational, but getting away from sort of the chatbot model. ⁓

Dan: Right, it was a much more back and just like this conversation is, but with an AI. ⁓

Nik: Yeah, and I think with this, so maybe in that way, right, the way that Thinking Machines is exploring super real time interaction capabilities. This new might be focused on what are the AI native ways of with that we might have. But again, I you know, for both those cases, I'm not entirely sure what the product is, but I mean again, that's fine. I'm I'm all for R and D and then seeing what what people do with it. So Who knows? Hopefully they're gonna be a hiring so folks out maybe you should keep an eye on this and see if if you're interested in being a part of one of these interesting frontier lab type ⁓ initiatives.

Dan: Yeah, it's great to see initiatives in this space because so many are they're not. they're being led by the technology at this point, which is understandable because it's technology that it's changing all the time. But it'll be interesting for places to start taking a design lens and putting that front and center brings us to next article. This is Sarah Gibbons from the Nielsen Norman group, the ⁓ NNG. I wonder if ⁓ Norman is even involved. Is Don involved this anymore? I don't know, Jacob certainly is. but that's n neither here nor there. so ⁓ Gibbons wrote this article called The Four Design Jobs AI Created So Far. And it is, is taking a look at four places where AI has been impacting or our design work. And the big is, of course, the one that we talk about the most, which is designing with AI. ⁓ And this is what your class is all about, Nik the AI augmented designer. And this is where designers are today. They are using AI in their design process. But then she out the next three that are less less talked but growing important. So the next one. designing AI products and this is we teach our the class designing AI products and services. That's what this is about. How do you use AI as a design material to start incorporating that into existing products and creating brand new products that are AI And then you get the two smallest which are designing for AI agents and designing the AI itself. And this is a small slice that is growing demand, but there's Not a lot of people that are working at this, and there's very little supply. People who have, who work at anthropic or Gemini or OpenAI or or those of who are actually working ⁓ on the actual foundation models and that are coming out of that, which I guess would be AI products. and so but people that are actually the things to make other things, is what those last two buckets are. So kind of an interesting breakdown of what I don't know if I would call these jobs per se, but I think this is more about where AI is affecting designers in different kinds of ways. I think for that it's a really interesting model

Nik: Yeah, I also think that the breakdown, I mean, it's a rough sketch how many designers are working in these areas, but I generally agree with what's presented. Most people are probably designing with AI, with fewer kind of designing the AI themselves. I do think though we're gonna start seeing a shift. So, with more and more people designing with AI, actually one of the interesting things here is depending on ⁓ Your flavor of designing with AI, you might be utilizing products and services that other folks have created for you to use, but you also might be putting together your own systems. We talk about this here on the show a lot about putting together your own workflows and then mixing agents together to do stuff. We've promoted articles of people doing this because we think it's cool. And the interesting thing there is I think actually the more that designers start doing that. The more they'll be able to transition over into that category of designing for AI agents, as they start to develop their own agents for, say, their own workflows, I think they're gonna start to understand this materiality of AI agents better. And that can help them to start designing for AI agents and also start designing AI products and services that use agent backends in a in a better way. So I mean maybe the the way to start in this area is to actually start just designing your own ⁓ work processes to kind of get a feel for it.

Dan: Yeah, and is probably the place to do that. And that brings us to our final article, which is all about perplexity. Perplexity did a bunch of ⁓ research along the Business School looking at how people using their product computer. Computer, and I love saying that like Star Trek style. it is well computer if you've never used it, it's essentially an AI assistant that can see and navigate and control your desktop to execute complex multi-step tasks on your behalf. So a very powerful Claudbot like ⁓ agent. And the question that they were asking was how does this change the nature of knowledge work professions and which from these kinds of tools might we expect? And the finding that they had was that using computer expanded The breadth and depth of what users were accomplishing, doing that at a lower cost. They were getting more work done at higher levels of abstraction and crossing disciplinary boundaries. And is all interesting. Now I want to Before the before this becomes a computer ⁓ I am gonna give a bunch of caveats here. So, number one, this research was early with a product that had just launched, and the adopters are really skewing pretty heavily towards AI natives, people who are very used to AI, people that are actively experimenting. ⁓ those patterns are evolving. They're rapidly shifting across the product landscape. And so what is working now may not be what in six months from now, or or will have entirely changed in six months from now. The second caveat is that the estimates that they did efficiency are possibly way off because it's very hard to say how much time and attention a task will take in the real world compared to what an agent can do. So all of the efficiency and stuff definitely take those with a grain of salt. But a couple interesting things that they found here. ⁓ So The first one, if you've ever done anything with an agent, that this should be pretty obvious that it requires a lot more upfront effort you've gotta tell it what to do, you've gotta tell it what not to do, what the goal is, and it a of effort up front, but less as the overall amount of effort. per unit of work because the agent's doing stuff for you. So it makes it really effective on long multi-step workflows. So the work ⁓ deeper and cheaper, but there is this longer time period up front getting What are we talking about in terms of length? So they looked at kind of two big tasks. One was search, that usually returned a response in seconds. But then when it was doing things that were more agentic, the kept for like minutes or even hours. And it would do searching, browsing, editing, running code, all those kinds of things. And Those could take I think the average was like nine minutes, so a long time. But the longest was in hours, not like an hour or either, like 20 hours, like a long period of time. And I think that's a really interesting design opportunity. How do actually start to design for things that take a nine hour time period? but a couple interesting findings with those long time periods. It didn't actually translate into more abandonment. the user stop events were similar across both search and using. And were both like three percent three 3.7 3.4 so not a big variation there. ⁓ one taking so much longer than the other. The other thing was that quality didn't fall with a higher autonomy. On turn turn sessions, that next turn was only 1.3% in the computer versus 3.4% in search sessions. So actually search did worse in that evaluation. here's things start to get a little wonky where they were trying to figure out how long these things take They looked at eighteen different domains and they found that there was a seventy nine to ninety two percent time savings and an eighty seven to ninety six percent cost savings. Now again, big caveat here. It's very hard to estimate how long things would have actually taken doing them by hand or without an agent's aid. But they found not shockingly that programming was the most extreme here. So something that was taking five hundred and ninety-six estimated minutes was only forty eight minutes. For computer and humans. So that was like a ninety-two percent time reduction. And so, and that yielded a ninety-six percent cost reduction. And the places where they found that this was the most advantageous were a couple different ones business, technology, education, writing. And then they looked at a sample of 8,000 users from eight different occupation clusters, they found that fifty-nine of them were actually using computer to work outside of their primary occupation. And unsurprisingly, one of those increases of that 59% was in arts and design. The other ones were management ⁓ and digital technology, which probably also includes design, and healthcare human services. So interesting couple tidbits in there computer about how these things are actually affecting work.

Nik: I think one of the cool things that you're seeing in this is the fact that they're trying to their best to measure the kinds of things like time reduction, cost savings potentially. they are getting down into the kind of tasks that people are doing and so I'm not surprised, for example, to see that programming is where a lot of efficiencies are coming from. ⁓ this is a question that a lot of companies are now starting to ask, which is like where are we actually saving, say on time or actual cost of the work to be done. then of course, are you getting the quality from that? So that's one thing I like about this that they're they're trying. And I think that if you're working there to implement AI in different aspects of your workflows, whether it's of this computer use, you know, fully agentic or even smaller things, like this article might have some good inspiration for how you might measure that and see that you're actually getting some impact and return on that investment.

Dan: Yeah, I think what I would love to see is more of this. Like some real actual studying ⁓ what the actual savings and time actually are. 'cause all all we get now is hype. And I don't I mean, maybe we even put this into the hype category as well. but It does have some numbers to back it up. It does have the Harvard Business School researchers, so it so it does have some you know, academic cred to lean back on. But I would like to see more numbers how this is actually working, how this is actually affecting knowledge work jobs would be really interesting. and I also think that rather than saying, well, this is saving us so much money and so much time, which I wonder if this pre-token hikes in terms of how much ⁓ money was actually being saved here. but being able to say like, hey, the benefits to this are things like being able to work broader ways and pull in more information from areas that ⁓ I don't know well. And that alone is valuable, whether you're in healthcare or in technology or management. I there are benefits and negative drawbacks that we should be more research on. And I'd love to see more companies partnering with academic institutions to do this kind of research.

Nik: I agree with that.

Dan: Including Carnegie Mellon. Shameless plug. Hashtag shameless plug.

Nik: Yeah. Yeah, it came up.

Dan: Alright. Well, thanks for listening to this episode. Brief as it was. you're enjoying your summer. One note is that Nik and I are off next week. I will be at Config to catch all the product announcements in person and to reactions and some speakers attendees. If you see me, come say hi and stay tuned for that special episode, which will drop on. June 30th. So hope to see you then. Enjoy your break, and we'll see you in two weeks.