What if the future of AI wasn’t louder, but more considerate? Joshua To, VP of Product Design at Meta, shares how empathy—not just intelligence—is shaping the next generation of wearables. From AR glasses to audio interfaces, he unpacks what it means to design for context, not attention—and why AI might finally deliver on the promise of invisible tech that just works.
In this episode, Joshua To, VP of Product Design at Meta, shares how AI is reshaping how—and where—we interact with technology. He walks us through Meta’s evolving approach to AR and wearables, why notifications are still the killer use case, and how AI is becoming the “brain behind empathy.”
We dig into what it means to build interfaces that understand you, why audio might be the future’s most underrated platform, and how designing for emotion changes everything—from form factor to function. Joshua also reflects on his path from launching a clothing brand to leading design at Google and Meta, and what those worlds taught him about craft, context, and human-centered systems.
This one’s for anyone designing AI into the real world—where every interface choice carries weight, and intelligence starts with listening.
Key takeaways:
LinkedIn: Joshua To | LinkedIn
Website: Home - Joshua To
Meta: Meta Careers
00:00 Intro: Fixing Notifications With AI
00:54 Meet Josh: VP of Product Design at Meta
02:06 From Hoodies to Hardware: Josh's Journey
03:53 The Google Experience: From Ads to Product Management
10:37 The Evolution of Google Glass and AR
19:12 The Role of AI in Josh's Career
22:25 Designing the Future: AR, VR, and Attention Management
32:49 Contextually Aware Suggestions
33:38 Leveraging Generative AI in Design
34:52 AI's Role in Concept Art and Storyboarding
41:24 AI Tools and Model Capabilities
45:54 The Future of AI and Wearables
51:58 Reflections and Takeaways
[00:00:00] Joshua To: the number one use case that I'm excited about when it comes to aR and AI is fixing notifications. Like literally when I interviewed this job, everyone asked me what's the killer aR use case? And I literally looked at them dead in the eye and said, notifications. I just want to like, get notifications in a way that is helpful to me. Don't give me notifications that aren't useful in a certain time. And like, everyone's got this vision to like, summarize notifications, do these, all these things. But I think like really, I talk about this all the time, um, I really think that aI with AR is gonna like allow everyone to have like a president of the United States sort of level chief of staff, kind of doing work, doing life with you
[00:00:54] Joshua To: Hey, i'm Josh. , I am the VP of Product Design at Meta and, I get to work on wearables and ai, and I'm super excited to be with you all to talk about AI and how we use it at work and, , how we think about the computing products that people are gonna wear, and, , how all this soup of things are gonna sort of create and deliver a profound, , impact on people's lives. I.
[00:01:22] Jeremy Utley: I am thrilled to invite my friend Josh to the conversation. Josh is a proper designer like capital D. I always joke at the D school that I'm a lowercase d designer. Um, Josh is a capital D designer. I feel. Josh, how do you, how do you react to that framing of your, of your, uh, professional credentials?
[00:01:44] Joshua To: I mean, I don't know. That gives me a bit of imposter syndrome. I like, I'm a, I'm a property designer, but capital D, that's a, that's a big statement. You know?
[00:01:53] Jeremy Utley: now, josh. Okay. So when you and I met, call it 10 or even 15 years ago, maybe somewhere around there, I think, if I'm not mistaken, you were doing hardware at Google, is that right?
[00:02:04] Joshua To: That's right.
[00:02:05] Jeremy Utley: Okay. So how did you get from kind of hoodies to hardware, and then how'd you go from Google to meta just to, just to finish tracing your hero's journey?
[00:02:14] Joshua To: I mean, it's a pretty long and winding journey, but anyways, um, I do hoodies and then, you know, on a personal level, like I, you know, like the, impetus behind hoodies was like sort of twofold, at, least, right? One was I just thought it was cool to run a clothing brand and to see people that I really looked up to, like vibe with our brand, if you will, and wear it around. Um, so, you know, it's like, whoa. The Rock is wearing our hoodie. Like, that's cool. but really like the the thing that really drove me, um, back then was like, we thought that, clothing, t-shirts, hoodies were like a medium for people to communicate things that they really care about. And so that was kind of like our whole thing It was like, um, like, uh, I think what, what was our tagline? Clothing, people love, um, messages, society needs was sort of like our tagline. and, uh, so much of what we did was about like raising money for causes, raising awareness about issues that were going on, but doing it in like a very graphic forward like design forward way where, It wasn't just the message that mattered, it was like, it looked really cool too. but, uh, I guess like two years outta college, i'm like grinding, um, on hoodies and it's like, it's really hard. It's like doing as well as I could have dreamed of doing. And yet I can like, barely pay myself more than minimum wage. And I'm like, oh man, I think I'm gonna live at my parents' house for the rest of my life.
[00:03:49] Jeremy Utley: And, and the rock's already wearing my hoodie. So like, what am I, where else am I supposed
[00:03:52] Joshua To: yeah. And so, um, I, I was lucky enough to get my first job at Google, but it, it wasn't , you know, sunshine and daisies. Like, I kind of went through the school of hard knocks at, Google where it was like pretty hard to kind of go from being top 15 college student in America and having rock stars wear our clothes to being like a contractor at Google. Uh, my first job was like, I was basically like a customer service associate for, Google AdWords. and it was a really hard job to get. Um, and, uh, you know, like, so the first year of, Google, uh, was me like memorizing this like five inch binder of policies for. The text ads on, on Google, and then my job was to like approve ads and to look for spelling mistakes and grammatical, things like that. And there was like a fear of getting fired. Every Thursday someone can tap you on the shoulder, do like a quality review with you, and then, uh, you can just get fired. Like folks would be like crying, vomiting, like really worried before these quality reviews. Um, and you kind of had to go through this gauntlet. but anyways, like for me it was like, man, I was like a, I, I studied design in college and so what I would do is like work really hard, approving ads for 45 minutes of every hour, and then I would spend 15 minutes coming up with design ideas, product ideas for Google, uh, kind of how I stayed sane.
[00:05:24] Jeremy Utley: Just from your own personal kind of time management. It's not that somebody's asking you to do it. you just said, Hey, I'm carving out.
[00:05:30] Joshua To: In fact, it was like price sort of like frowned upon. It's like,
[00:05:33] Jeremy Utley: No, you just say it's my 20% time. Right.
[00:05:36] Joshua To: There was no 20% time for contractors doing the approval bin. Uh, let's just put it that way. Um. But yeah, like from there I, I like managed to get through that. And then, , the progression was you go from, , doing ad approvals to doing, , email support, to doing chat support. And then you get the final kind of thing is , you do phone support, , with, um, Google customers and that's kind of got how I got my start in tech. Um, and. , You know, credit to Google. Uh, it was just like a really crazy, fun time and things , were growing like crazy. And it sort was like once you got your foot in the door and once you became a full-time employee, it was sort of like, there's no problem at Google that you can't at least attempt to try to solve. , Or at least that's sort of just the feeling of the time. And so even when I was a contractor at Google, I was able to suggest things that I. made it to google.com. But these are like super subtle things. Like I noticed that the spacing and the, I'm feeling lucky and Google search bar were not aligned correctly. And the padding around the search, , box was wrong. And, , it was like pretty thrilling to, to, to make these small tweaks and then see millions, if not eventually, billions of people see these like small tweaks you, you made. Um, and so that was like pretty cool. Um, even though like, you know, looking back it's like, oh, you know, that's not that big of a deal. But for me, really big deal.
[00:07:04] Jeremy Utley: Would you say your kind of 45, 15 split, would you say it kind of. Prepared you, it was almost like exercise for looking for opportunities. Because I'm just looking at my notes here. You said once I got my foot in the door, there's no problem you can't at least attempt to solve. But I think there's always this tension because like, I've got a job and it's usually not my job to solve this other problem. But it sounds like, I mean, if I'm kind of narrating over your character arc, you had started practicing being deliberate about, you know, making the most of your time, carving out time to do stuff beyond kinda your scope. do you resonate with that characterization?
[00:07:41] Joshua To: I would say even today that's sort of like what makes my job most exciting and what maybe gets me in trouble, um. kind of at the same time is that, , the easy version, not easy, but like the, the safer version of doing my job is to focus on purely just design things.
But I also think like design in general is kind of about problem solving and., I think our work gets most interesting when it kind of butts up against like, , all the other things, whether it be partnerships and new technologies and, um, different brand collaborations we could be doing. Um, and, uh, but you know, when you do that, your job just sort of involves a lot more people and you, you have to be a lot more thoughtful and, um, just empathetic about how to sort of connect the dots and work with the right people to make things happen.
But, , that definitely resonates. Like, you know, I think just getting your hands dirty and putting your hands in, , different cookie jars just helps like the, , cross pollination of things. For sure.
[00:08:42] Jeremy Utley: Love that.
[00:08:44] Joshua To: anyways, um, continuing the story arc. Um, I, get to phone support, which is like the final boss, and then you're supposed to kind of slowly become like the sales executive over time and you, um, help big companies, um, advertise on Google and use Google's latest technologies, , to help your company grow. And, uh, that's when I kind of like broke off from that and just like had really cool opportunities that I got to jump on. And so then got into like. Marketing and program management, and then eventually became a product manager. Uh, but at the time, , Google didn't really hire designers unless you studied HCI at Stanford or you know, Carnegie Mellon. And I just, you know, I studied design media arts at ECLA and I was like, sorry, we don't understand how that maps to user experience. Um, and so I eventually left Google, , started my own company, , which was like kind of a crazy concept, but it was like a VC fund attached to a coworking space attached to, um, a design consultancy design studio. And we did a dozen investments over a number of years. had like a pretty big design team, uh, that both serviced our portfolio companies as well as some selective outside clients. And, , kind of unexpectedly, a couple years in Google came knocking and acquired the entire design team, including me back into Google. And so, um, it took kind of leaving Google to come back to do the job that, , I actually wanted, but it kind of all worked out. And so, , that's when I kind of like came back, led a bunch of like enterprise software, so like gmail docs and spreadsheets, Google Drive and those types of things. And then, um, that kind of led me to, , like Google Cardboard and AR and VR and hardware. And so, um, it's been, it's been really fun.
[00:10:37] Henrik Werdelin: Did you ever get to work on the original classes over at Google?
[00:10:40] Joshua To: Uh, I worked on it. I was not on the original Google Glass team, but , I did get to work on it some, um, kind of the years before I left Google,, when they were starting to look at how the Google, , glass could be, um, well applied in enterprise for folks working in factories or, , folks out in the field. And so
[00:11:00] Henrik Werdelin: do you think? I think sometimes when. People like yourself have invented these kind of early versions of things that then becomes kind of like more mainstream, but it takes like a few years. There's like a few things that kind of like is obvious now, but wasn't that obvious back then. Do you have any of those that you kind of remember from like the early Google class days where, 'cause I think a lot of people now go, oh, you know, it was too early and like, but I think a lot of us had them and tried them and started to understand how visual language would be when suddenly you had to, you know, like touch, uh, your side of your head instead of a mouse and stuff like that. Whether, any of those kind of like, uh, things that kind of comes to mind?
[00:11:43] Joshua To: Yeah, I mean, , the most obvious thing is that like, you know, coming kind of a bit from like a street you wear fashion, , standpoint, um, you gotta make sure people wanna wear it and you gotta make sure that people around you want you to wear it. Right. And I think that's probably the miss, if you will, because like, I think in many ways, like the experience on Google Glass for the time was like way. Ahead of its time and like, it was actually pretty well stitched together. Um, and uh, you know, you can do like turn by turn navigation, with the thing, but it just looked crazy, you know? Like it just was not the thing you, you, kind of wanna be seen, , wearing and people around you were really uncomfortable.
I.
[00:12:34] Jeremy Utley: Yeah, I want, I wanted to hit that point. You said people around you, because I get the, the two parts, the first part of you gotta make sure you wanna wear it. That makes sense. But then the part around you want people around you to wear it. How do you feel like the latest kind of, you know, iteration of technology addresses the, do people around you want you to wear a problem differently than the original Google class did?
[00:12:57] Joshua To: Well, I think we take privacy pretty damn seriously. And so that's why, early on it was just like, Hey, when you, when you're taking the photo or when your video's on, um, it should be really obvious. , And so when you take a photo with the glasses the, the LED turns on, so you know, Yeah. folks know when the, the camera's on. ,
[00:13:20] Henrik Werdelin: Do you think that will continue? And I'm asking because after COVID we all did Zoom and then now, then everybody started to transcribe and record their Zoom. And now obviously. You know, the classes, but also like all these new devices that records everything. And it seemed to be kind of like just slowly becoming kind of a thing where people, I guess, assume that at least all Zoom calls are recorded. And, I guess the question is, do you think that will ever just become commonplace? , We assume that all our conversations when we talk to other people are recorded.
[00:13:53] Joshua To: I mean, I think we're probably like in the most recorded era of time right. In human history. , And, so I really do think that like over time there, there, will like continue to be more, , things recorded, but like, these things take time, right? Even early on when people started getting, , you know, smartphones, um, I mean it's been a like, you know, even like the smartphone arc as like even like the most successful consumer electronic in history, um, there was sort of like a period of time, right? And, cultures matter a lot too, right? Like in Japan, , to this day, , I I, if you sell a phone in Japan, um, the camera needs to have like an audible. Like shutter button, , sound, right? like it's privacy, um, is really, really important. I think what aids in, in, in sort of these, uh, things too is, that as new technologies, become available, they also often like help, groups of people. Um, so like specifically with our glasses, probably one of our, like most fanatic, product users are sort of the, the, um. Low vision blind community. And I think like, kind of early on in kind of product design, product adoption, curves, right? Like folks who find like a really unique, spiky way to use it and has like a sort of pretty like, um, meaningful and , profound impact on their lives kind of helps pave the way, for, um, other people, um, to accept using it. So, 'cause I think in general, if there were folks around me like recording me, I'd be like a bit uncomfortable. But if one of those folks was someone who was using that recording, um, and technology to, , help them get through the day or help them get things done, it would be like, Hey, like you could have 10 cameras pointed at me it helps you, , do your thing. Um, but I do think in general, right, , , it's really important that there's trust in the, the social cues kind of all exist, right? um, I do think that as like the utility and the usefulness that you get, um, outweighs, , some of the privacy concerns and you do a good job of the social contracting and. You know, when someone's recording, you know, when someone's not, um, you know, there's gonna be a bit of a shorthand for like, Hey Jeremy, do you mind turning it off Or like, having a, a more personal conversation and, you know, I just don't want that logged. And I think that should be totally okay.
[00:16:28] Jeremy Utley: So can we get to, uh, I would love to hear how you think, Josh, about the aesthetics, the want to wear and people want you to wear, kind of now that you are building what you described to me, at least as one of the most complicated, if not maybe the most complicated consumer electronic of all time. Right. How do you think about preserving the, call it street culture, you know, aesthetic mentality when there's now so much, you know, a hoodie's, one thing the most complicated consumer electronic of all time is, is something totally different? How do you think about kind of merging those worlds?
[00:17:05] Joshua To: Yeah. I mean, I think to, to, be clear or to clarify the, the thing I showed you, Orion is like our internal prototype for air glasses. Um, that's, uh, many clicks, more complex, um, than the RayBan meta, though I think the RayBan my glasses is like also an incredible feat of design, and engineering, right? Because it's like, literally it's the RayBan Wayfairs or the headliners, and we managed to put as much tech as we've put into it, and we're really proud of that. But, um, it's really the Orion's that, you know, I sort of make this claim that it's the most complicated consumer electronic in history. But, um, I think the, the, question is a good one. And, i really do think that, um, it's, , kind of what I shared. It's like does the utility and the usefulness you get out of it, , outweigh sort of the fashion hit, if you will, for most people, right? It's like you give Orion to a supermodel and they put it on like, Hey, this looks good. It's kinda look like my, uh, Dior sunglasses, but. The average person, uh, wouldn't wanna get caught dead, maybe wearing that like, you know, i. Unless folks knew like, oh my God, when you put it on, you got this like wide field of view display. You've got ar and ar eye, like, you know, popping off at the same time and doing really good things for you. And so, , these things will sort of take time. And like right now I think we're very, very proud of Orion because it's like under a hundred grams. Um, it looks like a bulky pair of glasses. , It's clearly not your normal pair of glasses because you know, at certain angles you see kind of some cool light leakage, , things coming through. And so, um, it's got a bit of like a cyberpunk vibe to it. And, you know, you can maybe kind of own it, but I think the form factor that you particularly saw is like, you know, like we're aggressively working on getting the form factor to be lighter, , so that more people, um, will be willing to wear it
[00:19:05] Jeremy Utley: in your career arc. One thing we haven't touched on but is obviously relevant to our audience and our conversations is ai. Can you weave AI into your journey? When did it first kind of hit your radar and specifically generative ai? And then are there any kind of moments of early moments of delight you remember going, oh, this is, this is something special.
[00:19:27] Joshua To: Um, I mean, I think in, when I was at Google, I got like some really early glimmerings of it, um, where, I don't know, maybe like three out of every 10 times you prompted it. You'd get like a, a, a, pretty incredible answer and it would almost feel like, oh my God, like , is it like a real being or a, a, real thing? , But then seven outta 10 times it would just like hallucinate and give you like an absurd answer. And so it's like, oh, it's not quite there yet. , But I would say like probably, you know, GGPT is probably when I was like, oh wow. Like it's really progressed. Um, and a really big leap has happened. , And uh, I, you know, like I think we've all kind of been a bit on this journey and, it's pretty wild how many things are changing in, um, the, the, short period of time. And so I think the other thing beyond just like large language models and um, i. text is like the stable diffusion and the image generation stuff, like, I think as a designer, as like a creative, , seeing like the quality of visuals coming out of Dolly and Midjourney kind of a couple years ago was like mind blowing to me. And then now the most recent gPT-4, like you basically have like comfy UI embedded at your fingertips without having to do the crazy workflows is also just like kind of unbelievable.
[00:20:57] Henrik Werdelin: Why are you in trying to build like the new kind of worlds that we will live in? So when you have. You know, VR and AR build into the classes. Um, lemme tell you what I asked. One of the things that I found profound when using the Oculus for meetings were not necessarily that the graphics at the time were kind of like, I didn't feel I was there. But what I noticed was that I had also like the depletion of all other inputs. And so the meetings were actually pretty intense 'cause nobody had their phone, nobody brought their computer like their email up in front of you. So, you know, like going to the cinema, like I was having this conversation with the other person and it was the only thing that happened.
'cause it's the only thing I could do was to be there. And so, as you're thinking about these creating kind of like the world that we will most likely live in, in the near future, right? These worlds that have part reality and part like augmentation on top of it. What are some of the things that you kinda like are most excited about that will emerge and how do you think about maybe even like creating kind of like a Yeah. Environment that is cool for people to be within because that you are in many ways now creating the imagery that we all will see through, you know, like, you know, being awake every day.
[00:22:23] Joshua To: Yeah, I mean, that's a good question. I mean, I guess just to go back to Jeremy and the demo you had of Orion, I think the biggest difference between, um, a, a, Quest headset and an Orion is the fact that, , you are present in the world around you, right? And so sort of. Early signs of how, , the physical world and the digital world kind of come together. And so I think that's like a big part of our vision is that like, , there'll be moments where you wanna be like fully immersed like you were mentioning, and there's like nothing you are doing, but the thing you're doing. Um, but I really do see the worlds blending together. Um, and I think the trick for, for me, our team, the company, the industry is to like sort of work alongside people's sort of like attention stream, right?
Because we don't wanna be doing this chat and then I'm doing nine other things at the same time, But at the same time, it's like very nice for me to like have my phone to my left and be, be able to be aware of like the people walking by, , in the office or whatnot. And so I think a big part of. The design challenge is figuring out sort of like, well, one, we want to give people control, right? , Over what's the right kind of like flow of visuals, information interruptions,, and coming up with really good defaults, but also like helping people be able to kind of tune these things. Um, so that, you know, I think the last thing you want is for people to feel overwhelmed. But you know, you're also not gonna go very far if like, people are just really bored. And so I wanna help people kind of find the, the happy, happy medium.
[00:24:19] Henrik Werdelin: What's that phrase? The devil is in the default.
[00:24:22] Joshua To: Yeah. I mean, I think, I have not heard that phrase, but, um, that, that definitely does ring true. I. Um, but also like, you know, like I think , There's like a pretty massive spectrum of ar, right? Like ar could be as, as small as, , you've got one AirPod in your ear while you're at a museum. And it's like you have your, a private docent in your ear kind of telling you about a history of a work it knows a bit about your preferences and the things you've seen in the past. Um, you get this like really discreet and awesome kind of experience. And then there's like, sort of like full ar where it's like you are in, I don't know, van gogh's studio and you're painting, a painting right alongside him your entire sort of 360 world is like, you know, you're in a studio. Um, and, . And so we think a lot about that, right? And so many of these things are, um, and like getting products to people. There's like cost considerations, there's weight considerations, there's social acceptability considerations, there's power thermals, um, silicon. And so, , we think about all these experiences we sort of want to have for ourselves or we want to deliver to, to people. But um, kind of along the spectrum, like we have to sort of like intersect technology availability and what people are sort of ready for. And so, um, that, that, I mean, that's what makes this. Work pretty fun and interesting, you know? I.
[00:26:09] Jeremy Utley: How do you think about attention. I, I, want to come back to you. Said, um, we wanna work alongside someone's attention stream. There's a real, I would say I feel a, a tension in regards to my attention because i, I, I'm sure others have had this experience, but you know, where time kind of evaporates where you go, I did not mean to be watching , YouTube, you know, shorts for 45 minutes, and now it's like, I actually wanted to go to bed, but then like, my amygdala got hijacked right. Or whatever. Um, and so there's, there's this weird thing of like, what do people want? What do they need? And , I think in uh, more immersive, more blended world. I, I, think I fear in some way, My inability to control, you know, what I want or, and what might be good for me. You know, I have to do increasingly kind of savage things like leave my phone outside of the room to be able to go to bed at night, things like that. And I just wonder about, in a world that's increasingly integrated, how we, uh, give users or customers the control that, i, you know, you use that word control, like the control they need versus the control they want. I don't exactly even know what the question is, but I think there's probably lessons from the, call it the social revolution about unintended consequences of, you know, things that we thought people wanted but actually resulted poorly. , How do you think about all that?
[00:27:33] Joshua To: Yeah, it's a really good question. I mean, I think like, A person, a human's attention is actually like. Pretty narrow. And I also think the attention spans of each individual person is also pretty different, right? Just kind of like, depending on how you're wired and I don't know, like, I think in many ways like that's, the job, like for us, right? Like I think as designers, our job is to sort of like choreograph attention almost. Um, and it's sort of a, a skill to like be empathetic to how a person's gonna be experiencing the, things that we're doing and how that like, sort of intersects with like the life they're doing, the specific. Task that they have and, and um, that they're trying to get done. , And so I don't know, like we, take that responsibility really seriously, right? Like in, in many ways, like, I don't know if any of you have watched the hyper reality video. It, it's like this it's this YouTube video that, a, group of designers made, called hyper Reality. I think there's this guy named, um, kechi,, matsuda, um, you can look it up, but it's like really scary. Basically like it's black mirror kind of vibes. you, uh, put on glasses and your wearables and it's sort of like what, you know, 2030 is gonna be like where, , you're just getting bombarded with. Digital things. And in order to unlock a, you know, 50 cent coupon to buy the broccoli in the grocery store, you gotta like collect seven coins by watching ad, you know, it's like everything is just like, you're like, you know,
[00:29:36] Jeremy Utley: is the worst possible version of the future.
[00:29:38] Joshua To: orders possible version. Uh, but it's like in a, I think it's been like a powerful cautionary tale for, technologists and like, I think especially designers, like we, we get, , caught with that. And so like, I don't know, uh, I like whenever, I'm reviewing designs, whenever we're like looking at, what features we should launch. Like I, I kind of have like hyper reality video.
[00:30:02] Jeremy Utley: kinda going that that's part of the criteria. Would this make hyper reality? If so, please reject.
[00:30:08] Joshua To: Yeah.
[00:30:09] Henrik Werdelin: that, we, I feel a lot of us on the entrepreneurial side, when the, maybe specifically when the EarPods came up, we were like, that's gonna start a whole revolution in new applications and new use cases of basically having these always on kind of like, uh, things in our ears. And I think we haven't seen as many as would've imagined. And some of the, I would say, successes that has been, has been this very weird kind of success like podcast. Like who would have known that like long form spoken word content would suddenly be the thing that, or the outcome of like really good headsets. . When you are looking at, you know, building glasses, obviously what you can see is a big thing, but what you can hear I would imagine is incredible important also. And meanwhile, the tone of the voice, the type of voice, how they talk, when they talk, what they talk about, all those things seem to be like an incredibly unexplored world. Because at least I haven't seen too many of those. Like it's very reactive right now. Like, Hey meta, what do I see? You know, and it tells me what I see. And all, like you have a notification, which is basically I read out from whatever I'm used to have my phone. Where, where do you see kinda like the next few steps, uh, in the, in the audio side of all these amazing devices that you're building?
[00:31:37] Joshua To: I mean, to take a step back, right? I think in many ways, like designing for screens have been a lot easier because it's like you assume that that's the thing that they're paying attention to, and so you kind of have their undivided attention. I think designing on a AI and, and wearables is like, you kind of wanna like. Slipstream into people's attention kind of moments. Right. And so to your point, like if you can see what someone sees and hears what they hear, you should never give them like a Groupon coupon when they're in the middle of like giving a big talk. Right? It's just like, hey, not a good time.
[00:32:20] Jeremy Utley: Well, you're saying is, you know, it actually that's interesting. If you, when you say if you can see what they see and hear what they hear, what you mean is as the designer of the product, if you know their context, then you can, you, as the designer can be contextually aware about what you're serving up. Is that, is that what you're saying?
[00:32:35] Joshua To: yeah, totally. And like that's what the person wants, right? It's like I actually am like a huge fan of advertising, like when, uh, it helps me find a product that I really want or is helpful to me. But like, please tell me about the product at a, at a good time. Like
[00:32:53] Jeremy Utley: Well, Is there, Is there an interruption there that's like, Hey man, like in the middle of the talk, slow down pace yourself. Right. That's actually a pretty interesting contextually aware suggestion, right.
[00:33:05] Joshua To: and speaking of audio stuff, right? I am, i'm the most annoyed these days where, um, I'm saying like, Hey, thank you so much for having us for dinner. Exclamation mark. Can't it know I want an exclamation mark at the end 'cause of the tone in which I'm sort of dictating, right? And so, , if you think about like, sort of. How high the ceiling is for, how much better these things can be and how much less friction it'll be to help people communicate and get things done. It's like super exciting. Um, it's super fun.
[00:33:38] Jeremy Utley: So i, I want to turn harder towards AI just 'cause I think maybe some people are going, wait, hang on. Isn't this beyond the prompts? Um, 'cause we've been all, we've been like design, we've been in, uh, lots of, lots of great places. Can you talk for a second, Josh, about how your team, uh, or how you even personally leverage generative AI in your work? And one kinda seed I would just put there, if it's interesting, you know, feel free to run with it. If not, feel free to take it however you want. But I heard Mark talking recently on a podcast about how the llama team in developing AI is really using a lot of internal AI products, which are advancing their development, which is great. , and so he said there. That the llama team's general kind of use case is not the world at large and kind of any general use case, but really for almost you could, for lack of a better word, dogfooding or dog AIing. Right. Um, do you find your own experience, you know, somebody who's not coding necessarily or primarily do you find that you're able to kind of, uh, leverage aI models in unique and differentiated ways? Or are you kinda doing more off the street, off the shelf retail kind of stuff? Just tell us how you and the design team are leveraging ai.
[00:34:52] Joshua To: Yeah, I mean, I think just to give you all like, um, a little bit of a view under the hood, um, we're, we're sort of just like super curious about it all and for sure. I think if nothing else, like I don't think ai, replaces taste just yet, or, um, AI isn't gonna sort of do our job for us, but boy, is it like an incredible sort of like brainstorming partner. . Imagination machine. Right? Um, and, , don't get me wrong, I, I do think like AI also does like quite a bit of like speeding up of some of the more repetitive tasks or even some of like the, uh, early steps of design tasks, right? Like, I think, you know. Nine months ago, or a, a couple years ago, concept artists were like a pretty big part of our process, right? Because it's like before you actually build the glasses and spend tons of money on building glasses might be useful to kind of sketch out, like in what situation people would use this, when they would wear it, how it all work. And was like, kind of like a a big part of our thing. Um, and like in, many ways, like, you know, sort of the need to like rely on a human to do the storyboarding, work has like kind of evaporated and vanished, but as a result it's like almost every designer is unblocked to be able to like do the quick pre-visualization in scenarios. and
[00:36:31] Henrik Werdelin: Do you happen to have a good prompt that you use normally? Like, um, you know, if you were to like just, Hey, I had this idea of like, you doing something with whatever. Do you then just literally, you know, speak, you know, write your mind? Or do you have like, like a little bit of a process that you normally go through?
[00:36:47] Joshua To: yeah, I mean, I think like it used to be when we were trying to get good concept art, the prompt would be like half a page long and now I feel like you can just like go type like seven words, right? Like, you know, for example, um, we sometimes use it just to like sort of simulate what future devices might look like. Um, sometimes it's just like, hey, imagine, , a person in a library reading a book, um, and you know, um, AI helps them, suggest other books that they may be interested based on the research project they're doing or whatnot. And just that is like enough for us to get like a. A few rough visuals to kind of then sculpt into like a kind of a key frame of a storyboard that's like useful.
[00:37:37] Jeremy Utley: You know, there's, there's a phrase here, Josh, that you'll like, um, , shout out to former podcast guest Bryce Shamal, who's the head of AI at Moderna. And one thing he's, he mentioned, which I really liked is quote, he said, the end is the beginning. you know, you think, like if I think about storyboarding, right? And you said the need for a human storyboarder has evaporated by this. And some people might say, oh no, poor storyboards. Right? But the other hand, everyone's now effectively capable of crafting storyboards, right. The accessibility. And Bryce gave the example of chess, right? When ai, you know, beat humans, they, they said it was gonna be the end of chess. And the truth is, it was a Cambrian explosion of grandma and new moves and understand. And and there are more people playing chess than ever before. And the end is the beginning. And I wonder if there are other, if there are lots of areas like, like storyboarding, where you go, what used to be this, like this very niche kind of ability is now actually democratized in a way that advances the craft.
[00:38:38] Joshua To: Yeah. And also, um, I think it's really important to note that it's not like, oh, poor storyboards, they don't have jobs anymore. In fact, I think the storyboards are like having an awesome time, right? Because they have spent decades of their life honing a craft to like get a lot of information with very few strokes on a piece of paper. And I think like they're way more well equipped to use AI to like create and deliver like incredible narratives and , visuals. So I think like they just don't have to sit in front of a page and you know, sort of, rock out on like a piece of paper as much, even though I'm sure they still use that skill, um, they're super well like equipped like, like think. about just who are the best AI prompters right now for video or for photography. It's like folks who are photographers
[00:39:39] Henrik Werdelin: There's this really beautiful ad, uh, test out that somebody did for Volvo where you see this car kind of like Volvo drive through a and uh, kind of like a town. And then all these flowers kind of start to bloom wherever it drives. And then I kind of like try to figure out who made it. And obviously it's like a guy who normally makes Com advertising, right? Because obviously you can't just write, make a cool ad for Volvo. You have to be like, you know, first I need this image doing this specific thing. And you have to be actually pretty good at articulating what is it that you see for your inner eye, which is what I'm sure storyboards do well.
[00:40:14] Jeremy Utley: It's, I think that's actually, it's a hugely affirming idea. I've, and I've kind of been floating around with it in my head this, crystallize it in a new way. Um, and so I just wanna put a point in it uh, for listeners, your expertise is not irrelevant. It's your differentiation and it's what enables you to get differentiated output out of a collaboration with ai.
[00:40:37] Joshua To: Yeah, totally. And I guess another like fun use, um, of AI has been, you know, like on our team, right? Like, for us to kind of do a better job of understanding how people will use our product or even who we're building our products for, we kind of use this, you know, and I think this is like a d school concept too. It's like the idea of personas. Like, you know, we sort of like named people, that were building products to serve. And as we kind of build out a persona of a person, we've been able to like feed it into AI
[00:41:13] Jeremy Utley: Like build a GPT, that's the Jeremy persona. And then you, you interview Jeremy about, Hey, tell me about your last crime.
[00:41:19] Joshua To: yeah. And so that's been actually really, really, , useful and, and, fun. Um, the other thing I was gonna get to is that like we, obviously use m all the time. That's like , the thing we build at Meta. But we almost like we've, we've, we've built like a switchboard of almost like every single AI tool that's out there. Some that are more developed, some that are really early. And, um, we like turn up the dial and switches and of these various things and , just do a lot of like testing. Right?
[00:41:54] Jeremy Utley: interchange, you're saying interchange models.
[00:41:56] Joshua To: Like, it starts with GBT, it ends with llama, starts with, uh, ends with perplexity, you know, it's perplexity, hands off the clo and then goes to llama. Like, we we're playing with that all the time because I think, , the models all have like a very, , they all have things that they're like particularly good at. And so a lot of what we do is like, you know, try to exploit the, the spiky, um, strengths of each model to kind of like get the best kind of end user, , experience during our kind of prototyping, , stages.
[00:42:30] Jeremy Utley: , Are there interesting spikes that you've observed between different model capabilities?
[00:42:37] Joshua To: Um, I think, like right now Claude is like probably the, tool of choice for, um, designers in terms of like its ability to generate UIs. Where, um. GPT like four oh is like really good at sort of like reasoning about things, , in a pretty impressive way. Um, the other thing that like, and there's a bunch of models that do that, but like, I think like one thing that has been also really useful is just like using aI for like, rapid, rapid deep research. Um, and, , you know, we, kind of like. Have a bunch of models compete with each other
[00:43:19] Jeremy Utley: yeah, I mean the, just on the deep research front, Josh, the way I think about it is, you know, just take three deep research tools, gemini, deep research, um, Chad, GBT, deep Research and perplexity deep research. And I know because, you know, I've, I've heard teams talk about it. that Gemini's goal, or deep research goals, basically to obliterate the million dollar McKinsey research project. You know, it's like, it's a five person team at Google, and their job is basically to get rid of the million dollar, you know, consulting, research. Um. I, I, the way I see it is I and Parallel can commission a McKinsey team, a, B, CG team and a Bain team, and I commissioned them all at the same time. They all come back to me within 10 or 15 minutes and then I feel, I mean, what would've taken a month for a world class consultancy? Granted, perhaps it's not quite to the same level, but it's close enough that I go, if I can go make coffee and come back and it's on my desk. That's pretty, and, and to your point about kind of interchanging models and seeing there's actually, there's zero cost in terms of user cost, uh, to commissioning parallel experiments and seeing which one delivers a result. Either that's particularly useful to you or particularly inspiring or thought provoking, whatever it might be. We now have more ability than ever to kind of commission parallel experiments, which is incredibly valuable.
[00:44:38] Joshua To: Yeah. And I think, um, one kind of unexpected thing too is like, you know, , I was kind of joking that like AI hallucinations are sort of like the new D School, random word exercise or, you know, pick your favorite kind of like, um, explode a bunch of ideas and look for random associations. And I think that like, , in many ways, like sometimes like a, a good AI hallucination is sort of like the, , the sea crystal of some new idea or a new direction that I was like, oh, that's like. Way off from what I was looking for, but I can sort of see what it was trying to do. And that's like an interesting thing to kind of pull on and, um, do more about. And so I think like that's, um, that's been kind of a really fun to notice as
[00:45:31] Jeremy Utley: I, I think hallucinations are what we call creativity among humans. You know, creativity is hallucination and in models we see it as a, as a bug in humans, we see it as a feature. There's a great New York Times piece, maybe we can link it in the show notes about a Nobel Prize that was recently awarded, I think, in chemistry. And it actually came from a, a model hallucination. That's actually the seed of the idea.
[00:45:53] Joshua To: Yeah.
[00:45:54] Henrik Werdelin: I have a last question for you, Joshua. Um, when I was watching, uh, as I was praying for this, conversation, a, , uh, interview that, , mark did, and one of the things that he was pointing out why things might move not as fast was because of the hardware constraints. Right. You know, like on the AI side, I think he was specifically talking about like, you know, before that you can get to these new models, you need to build like, you know, big centers and assembles to be used. So like, well, less used, but like you, you have to assemble all these different things and so with ai, of course these days you feel every morning you wake up, there's like a new foundational model release and like it can do something new and it's pretty incredible. You, however, with classes like. You have to bend metal and you have to do all the heating and you have to like get a manufacturer and all this stuff. Do you think that, well, two questions. One is, do you think AI will help you then get to this future vision faster or do you think one of the issues of now kind of doing stuff in the physical world is that it will just be slower because of, of basically the physical world's constraints.
[00:47:05] Joshua To: It's a great question. Um, in in my view, I think AI is like making wearables like sort of way more relevant and way faster than I think any of us would've expected. Right? Like, because like we sort of design wearables not just to be worn, right? Like we actually design wearables so that they could understand. And aI is sort of like the brain behind empathy, right? Because it's like if someone's wearing the glasses or any other wearable and you have a better sense of their context and the situation that they're in, , it gives it like a 10 XA hundred advantage against the other competing devices that you have in your life. Um, and , we kind of used to think that like, oh, to get people to wear glasses, you gotta give them like hD big screen tv. You gotta blend the digital world with a physical world perfectly for it to be worth their wear. And AI has like lowered the barrier to entry, I think in like a huge way, right? Because, , I think in like single digit years, you'll like never forget where you left something. , Or, like that's, that's like profound. Like, I think that's like really, really cool. and that's why like the, the patterns that we're thinking through have changed too. Like it used to be like, you know, what's the weather augment on your coffee table to tell you the weather? And Really, like, that's really cool and it's gonna be a thing, but really the thing we need to nail is like to figure out how to say, Hey, Henrik, hey Jeremy. I like have something for you that might be useful to you in a non, sort of disruptive kind of way.
[00:48:52] Henrik Werdelin: I mean, I just, your point for earlier, it's super uncon contextual right now. Right. You know, it will basically disturb you when whatever your phone tells you to just disturb you. Right? Like it does not know that I'm, you know, like hugging my wife goodbye, leaving for a two week, uh, business trip.
[00:49:08] Joshua To: Yeah. To your point earlier, Henrik, I, I, I don't know if you know this, and maybe you did, um, you did a good deep research on me or something, but like the number one use case that I'm excited about when it comes to aR and AI is fixing notifications. Like literally when I interviewed this job, everyone asked me what's the killer aR use case? And I literally looked at them dead in the eye and said, notifications. I just want to like, get notifications in a way that is helpful to me. Don't give me notifications that aren't useful in a certain time. And like, everyone's got this vision to like, summarize notifications, do these, all these things. But I think like really, I talk about this all the time, um, I really think that aI with AR is gonna like allow everyone to have like a president of the United States sort of level chief of staff, kind of doing work, doing life with you where there's like a, um, three minute gap between meetings, right? Where you're like, you're tying to the prime minister of a country then three minute gap and you're completely context switching to meeting the, the CEO of like the big oil company. And it's just like in those three minutes, like, what, what, what do I need to know? Like what do I need to know about the next meeting I'm have, What's some crazy thing that happened between the last meeting this meeting that I should know about? And like, I think AI has like the ability to give that superpower, um, that usually only like a few people have access to. Everyone.
[00:50:52] Jeremy Utley: Dude, you literally are reading my mind. Josh, you're reading my mind. , I've been saying recently, I have a lifelong fixation with Winston Churchill and how he was able to dictate his national addresses from the bathtub because he had this assistant who had all of his context, who knew all of his speech history, who knew the audience and his intentions and objectives. Therefore, he could lay in the bathtub dictating a speech and it's done. Right.
[00:51:19] Joshua To: you think about all those like West wing shots, right? Of the,
[00:51:22] Jeremy Utley: Right in between. Yeah.
[00:51:24] Joshua To: And all these people scurrying with papers to like, you gotta sign this. Like, we need you to do that. Um, and it's just like, go back to those like. Go back to those clips on YouTube and imagine like all those people scurrying around with the papers, um, or you know, like you got an, you you got a earbud in your ear and , it's telling you exactly what you need. You can ask it questions, you know, and I think it's like pretty awesome.
[00:51:49] Jeremy Utley: It's an incredible view of the future. And as you said, I don't think we're too far off. that is the perfect close, my friend.
Okay. So, uh, henrik, first of all, I love inviting friends to this show just to get to hear their stories. 'cause I've known Josh for like 15 years and I learned parts about his hero's journey that I had never known before. So it was just super cool, super fun. I, I feel like we covered way more ground than even I was hoping we would get to cover. So I'm super delighted. You, you obviously just met Josh for the first time today. What stood out to you?
[00:52:25] Henrik Werdelin: Mean, like, I had. Three. When you say this, there's one thing that actually stand out to me and that is just how incredible it is to, for you guys to be an environment in San Francisco where a lot of people kind of randomly know each other. And there's so many people that do so many interesting things that, uh, you know, like I. I'm very, uh, very happy that I am pissed. The audience too can kinda like tap into the people that you have to stumble into. Um, on what he was saying, there's three things that kind of like made me, uh, that I wrote down, which we thought first is like, really blown away by this whole concept of how we'll have to design audio experience much better than we have and how little actually innovation has been in that. So most audio right now is basically you talking to an AI model and it says something back, and increasingly, it obviously says it back like a human. But as we were talking about in the conversation, this idea of talking in the right way at the right time is kind of like very important. So that's the second one I think from audio going into being context aware, I mean like. To, Josh Point, like they have like, uh, you have glasses that can see stuff, but still, you know, those systems don't seem to be connected yet. And so any glasses or any devices, the ring or whatever will just notify me, you know, even if I'm running late to the plane, it'll be like, you know, you got a notification, somebody liked your podcast. You know, like, and obviously clearly not now. And so I
[00:53:52] Jeremy Utley: I'm I am late.
[00:53:55] Henrik Werdelin: And so, oh, you know, you're doing a big presentation and then it goes like, you know, like, you know, your mom is ringing. Um, so I think that both like what's gonna happen in audio and what's gonna happen in context awareness is kind of pretty nascent kind of spaces. And it just like so fascinated to hear a little bit more about that. And finally, I mean, like, I obviously drank the Kool-Aid on technology and I love this thing, but. I think as much as a lot of people think of technology comes as evil, I'm always struck by how kind and thoughtful and human, a lot of the people that are designing these projects are, and so. Who am I to say? But when I hear somebody like Josh talking about how he's inventing this thing that you and I and the audience and probably our kids at one point is gonna wear, I feel a little bit slight sense of comfort because. He doesn't seem to be somebody who kind of thinks, you know, I'm just gonna think about like, the way that I can shove more ads into Henrik's face. I, I realize that there's a business model and there's like structural thing that create like other things, but at least you know, like the person sitting and drawing is somebody that. Just came across as a very earnest, very kind, very thoughtful soul that is designing something to make life better and not something to increase the, the share price.
[00:55:24] Jeremy Utley: Yeah, that's great. Yeah. Highly considered individual for sure. Um, and folks didn't hear his part because it was off air. But one thing that Josh. Said as we were wrapping is the job of a designer is to make people's lives better. You know? And I, I agree. I can attest for sure to his, uh, Sincerity in seeking to do just that. To me, one thing I loved was this. conversation had a little bit for folks, young and old you might say. I think on the young end of the spectrum, taking a page out of Josh's playbook of. Uh, One purpose, you know, leading with purpose and his fashion brands. You know, even having a mission and a message, a social message, et cetera, I think is great. But then also just the tactical thing of, hey, 45 minutes, do your job in 15 minutes. Find things beyond your job that probably need to be done. I thought that is a hack that literally any intern day one could take really seriously. Sure. You might be under the gun, so to speak, to every Thursday people are really concerned about whether they're gonna be fired. You know, who's not gonna be fired is the kind of person who's doing their job and 75% of the time. And then spending 25% of the time solving other problems, that person's never gonna be fired. And
[00:56:35] Henrik Werdelin: And you know, another thing on that specific point, when. I also, that's been my experience when I've worked in big companies. If you have something that needs to change or you notice something, like if you notice that like the pattern in the search bar, uh, I was facing in the, in the search bar is not right. Nobody's gonna get angry at you for pointing it out in a polite way. And so I do also think that seems to be like a good, kinda like takeaway of like, if you can come up with somewhere to add value, uh, you know, people will listen. That you don't just have to sit there and like draw on your own sketchbook.
[00:57:07] Jeremy Utley: Yeah. Yeah. I thought that was, I thought that was really great. Anybody could take that. And then I think I said young and old, I think for older folks, you know, kind of a newer obsession to me is this idea that experience is a meaningful advantage with AI and the commentary, and the discussion about storyboard artists that their expertise isn't irrelevant now, their expertise is actually their differentiation. And What they're, how they're, able to achieve differential outputs out of a language model. And I would say if you've got expertise. I. be afraid that now it's commoditized. Now anybody can create a storyboard. No, you are uniquely equipped to get spectacular storyboards out of AI in a way that I'm not, 'cause frankly, I've never made a storyboard. I have no idea how to make one. Right. I don't know how to prompt for it. Right. So that was super cool. Um, the other thing that I thought was, uh, really great was just the. Idea around hallucination. Um, sometimes being a feature and sometimes having a kernel of a great idea. And We know this in creativity, right in the episode we did with Devin recently as well, like randomness and fun and unexpected angles are all a part of the creative concoction and. Hallucination sometimes deliver kind of amazing nuggets and kernels of possibility. I really think that the last thing you want is a model that never. Hallucinates, and so folks shouldn't be trying to get, you know, if you're primarily working in facts and trying to ground your, process. In fact, perhaps find a tool that's a fact based tool like notebook, LM or perplexity or something. But if you're looking for, and, but I would say most of us aren't looking for fact, Most of us are seeking to explore and imagine and interpolate and extrapolate there. You need hallucination. And I just think that's become like the H word, um, in companies, you know? And to, , I wanted to, uh, bring hallucination back to its proper esteem. And then the last thing that stood out to me, or at least here in my notes as I'm just looking down, is this idea of, uh, the. Assistant, everyone's gonna have a chief of staff who's capable of kind of whisking them between, you know, in the three minutes in hallways to have hyper heck, uh, contextually aware, you know, spin up, update, preparation, et cetera. I think that's a really invigorating and exciting vision of the future. And, um, yeah, I mean, wearables, I, I, he convinced me, he convinced me that wearables are a thing that we should take seriously. If you don't maybe play with them, Uh, and, and see how you can start to incorporate them into your suite of tools in your toolkit. kit. Yeah,
[00:59:51] Henrik Werdelin: I think that's it, man.
[00:59:53] Jeremy Utley: I think we did it. Thanks for listening. As always, if you enjoyed this episode, please hit like, please hit subscribe. Please share with 14 designers. Or young people who need to know how to be good interns Or older people who have checked out because they think the game has passed them by. Just share with 14 people. In fact, go into your phone right now Find 14 people who need to hear this episode and let 'em know 'that you love them. The secret password is Petya
[01:00:23] Henrik Werdelin: Patia Patia,
[01:00:26] Jeremy Utley: Petya. It's a flower Henry. It's a flower. Patia patia.
[01:00:31] Henrik Werdelin: I'm gonna Google that. Nice.
[01:00:33] Jeremy Utley: Until next time.
[01:00:34] Henrik Werdelin: time.
[01:00:35] Jeremy Utley: Goodbye. Bye.