Published in Pioneers

Andy Matuschak on physically-informed digital interface design

By Devon Zuegel


60 min read

Andy is a software engineer, designer, and researcher working on technologies that expand what people can think and do. In past lives he helped build iOS at Apple and led R&D at Khan Academy. Now as an independent researcher, his methods bridge the gap between academia and Silicon Valley.

DEVON: Hello, and welcome to Tools & Craft. I'm Devon Zuegel and today I'm talking with Andy Matuschak, a programmer, designer and researcher who works on technologies that expand what people can think and do. The way Andy approaches his work is an interesting hybrid between academia and Silicon Valley. He explores theories by building and studying real world software systems, improving the systems with insights he's gathered and then repeating it all in a cycle. Andy was part of the early iOS team at Apple and after that, he co-led the research lab at Khan Academy.

He's a true craftsman who puts tremendous care into everything he does, whether it's cooking a complex meal, learning a piano sonata, or writing an essay summarizing a finding from his research. Andy had a fun and wide-ranging conversation. We talked about the importance of peripheral vision, the emotional experience the different technical systems evoke, what it means to build a serious context of use when building tools and much more. So, Andy, thank you so much for taking the time to talk with me today. So for starters, how does the emotional experience of an e-ink screen differ from using a normal LCD?

ANDY: How does paper differ from using a normal screen? One thing that's funny is that when I was a kid, neither tablet screens with digitizers and pens nor e-ink screens were an option with these Wacom digitizers. And that had its own interesting feel where you had to keep your eyes here and move your hand here, and you can see a cursor on screen where your hand is. And that felt very different from paper, because I guess on paper you see where you're holding the pen, you see the inking and everything is this continuous experience. And somehow it's separating where I'm holding from where I'm looking, created this distance, but a different kind of distance was created also by the emissive display. Something about drawing often in faux watercolor or faux acrylic or faux graphite on an emissive screen, it felt very strange.

Something about it felt very active and energizing and not, I don't know, contemplative in the way that I feel when I just sit with paper. E-ink for me is somewhere in between when I sit with e-ink, there's a slowness, and I don't really know what to attribute this to. One idea can be that, well, e-ink devices usually don't have notifications and web browsers and things like that, but I've actually experimented with connecting e-ink devices to my normal laptop. So then the e-ink display now has wifi and notifications and all those distractions, and yet it still feels different, there's something slow about it. And not just because the displays are slow because they are, they're calmer somehow, but they're also frustrating. So I do use an e-ink display for reading and taking notes when I travel.

But even though basically every ink display on the market bills itself as being just like paper, they're not, and they're actually much, much worse than paper in most ways, except that when I bring them to the beach, I don't end up with a notebook full of sand I have to shake out. The really powerful thing about e-ink displays for me has just been outside is wonderful. And it's nice to have a little bit of friction, like if I am away from my desk, it's harder to follow citations and references, but I still can. So I can get there and I have a folder of 30 papers and I'll flip through and pick one that seems interesting. And then if a reference seems really important and I'm like really excited about it in that moment, I can follow it in that moment, but the sun doesn't ruin the display.

DEVON: Which is important. A similar problem I struggle with is I find I do much of my thinking while I'm walking. But I also find just really useful to have something to work off of and push off, like something to write on whether it's my phone or tablet, and it's difficult to write while you're walking.

ANDY: Yes.

DEVON: Stephen Wolfram from his thing that he brings his laptop and so on, I haven't tried that yet, I probably should. I've obviously used my phone while walking. Sometimes I use audio notes, but I feel like outside and especially mobile outside is a mode that very few people have thought about when designing devices.

ANDY: Right. Right. A class of interaction I'm really excited about is persistent audio. We're lucky that there are now these really cheap well-engineered long battery life devices, like the AirPods that allow us to have persistent input and output all day. And so between the watch and that I actually find it's very easy to... I'll go out for a walk and I'll just stick one in my ear, and I can really rapidly start a recording by tapping on my wrist, and then say a thing and then tap again to stop, and that's a relatively low inertia way to do it.

One thing I really like about audio notes is that they remove my editing instincts. If I'm trying to write something even here, not on a walk and I'm struggling to figure out how to say what I'm intending, then I will often start dictation on my computer and then turn away from my computer, and walk out here with an AirPod in my ear and just walk in big circles. And I'll come back and a thousand words of texts will have been generated, I will probably use none of them, but at this point I'm ready to sit down and start typing.

One thing I really like about audio notes is that they remove my editing instincts. If I'm trying to write something... I will often start dictation on my computer and then... walk out here with an AirPod in my ear and just walk in big circles.

DEVON: Yeah. Turning off that editor brain is extremely important.

ANDY: Right. Right.

DEVON: On the topic of specific implementation details and how they shape the way that we think and the way that we work. In 2015, you had a tweet that I liked, which said, "Animation APIs parameterized by duration and curve are fundamentally opposed to continuous fluid interactivity."

ANDY: Okay.

DEVON: So this is the standard model for a lot of animation APIs, how has this low-level technical detail constrained what designers create?

ANDY: Yeah. The biggest place that you see problems around this is that it's very difficult to specify animations this way and have them be interruptable. So the failure mode that you will see as a consumer is you press the button and start doing the thing, and then the screen starts to like, "I'm going to go over here." And you're like, "Wait, no, actually I wanted to tap this one." And it's like, "Nope, you got to wait until I stop playing." You're like, "Okay, fine back." You wait 500 milliseconds, now you're allowed to tap again. And it looks just like laziness, but there's really something much deeper there, which is if the way that you specify animations is go from here to here over this amount of time, then it's not necessarily clear what it means to stop part way and start going somewhere else instead.

There's a particular feel that you get when someone has made something where animation is specified say through other means, and that's that it feels like you have your finger on the thing to a much greater extent, that the inertial scrolling on the iPhone is a really good example. The rubber banding experience also where you slide this thing and while your fingers on the screen, it really feels like an object on the surface of a glass table with no friction, that you're sliding around. And then as soon as you let go, it moves exactly as you'd expect for some object of mass, we've all internalized what the mass of these virtual screens are, it'll decelerate in a way that makes sense, but you can grab it at any point even while it's decelerating.

So this is a physics-based approach to the animation rather than an iteration-based one, and it feels like it's in your control the whole time, it's very difficult to pull off often on the web... When I'm building stuff for the web it often feels like I'm having to first create the universe, like let there be light, let me make things that are interactive, even if I don't know if you've made stuff with the pointer API. If you just want to make something that can respond to touches effectively and mouse events effectively, and responds reasonably to when there's multiple touches on the screen, it's like this huge burden and overhead on the web.

And so I'll notice that I'll spend hours just getting to step one, whereas if I've been working with a platform that was built for this kind of stuff, I'll start at step one. But that's hiding a huge amount of work and tons of decisions that the platform has made for me, and that the designers of the platform have suffered through in order to figure out how to make multi-touch work a particular way.

DEVON: I've mostly done web design and many people have worked in single systems. And I think it's interesting that to break out of those systems and look at how others will do things, because they encourage or enable you to implement different things and think different types of thoughts. Something you've done a lot of work on is spaced repetition. What would it look like to build space repetition into an OS or into one of these fundamental systems, so that it's a core building block of a system?

ANDY: I think spaced repetition is a special case of something much more general, that I've been labeling programmable attention. Just this idea that there are so often lots of little things in your life that you'd like to orchestrate happening maybe with some regularity, maybe periodically, maybe repeatedly over time, maybe with that schedule responsive to some kind of input or feedback. And so when I imagine the very general version of space repetition, I imagine something where maybe I start reading a paper and I get tired, my attention's slipping. And so I can program my attention later and say throw this into some reading spool, sometime in the near future and just trust that's going to happen, my attention's going to be guided back to it through some orchestration later.

Then there are other things that feel more ritual, maybe I was really moved by the way that this particular essay expressed an idea, and I'd like to bring that into my life more regularly. A thing that people with physical studios and workspaces sometimes do is to print out inspiration and put it all around them. And I like imagining that you could do that, but have it be constantly shifting, almost like the seasons, like the trees around you changing. But you also have maybe these portraits of people that inspire you, around you and changing every day, and perhaps even these things are active somehow.

So I might have some kind of interaction around people that I care about who live far away, and maybe every day one of their faces appears in this magical frame I have, and brings their presence into my world for the day. And I may then be inspired to reach out to them in conversation, or remember that I saw this article last week that they would really love, these are all different ways to orchestrate our relationship to things over time.

DEVON: Apple Photos and Google Photos, and a bunch of other photo apps have some versions of this that I like. And I have a widget on my phone that shows random pictures that I've taken over the years. And when a picture of an old friend pops I'll often send them a message and be like, "Google Photos just sent me this picture." Often results in a nice conversation, it's a lot more natural too than just reaching out out of the blue. I would like to think that I would do, but realistically it doesn't happen organically, it's sort of an out of sight out of mind thing. Or even if I were to think of someone, I don't have an excuse to reach out, so it's just like, "Hey, what's up." And so I think socially it also gives you a nice excuse as well.

ANDY: I find that when I'm orchestrating this kind of stuff for... I want to have a relationship with this person, or I reach out to them regularly, or I want to think about this inspirational quote or see this art that is related to a project. I usually can't specify in advance what kind of schedule I want or all of these options, like delete this, never show this to me again, like that's too heavy handed. So I think it's really important to have some kind of feedback mechanism that's more like a thermostat, turn the knob down, turn the knob up, kind of thing, or you have a momentary reaction. Imagine if this widget you have on your home screen you could just double tap the photo and that would say, "Hey, I like this." And then maybe you saw more of things of that kind of person on your widget in the future.

DEVON: I do use snoozes in my inbox, which works quite well, but it has a particular problem, which is if I say, "I want to be reminded of this in three days." And it comes back and I'm like, actually I want it in two more days, if I do that enough times, I start to feel guilty, I'm like breaking some promise to myself.

ANDY: Yes.

DEVON: Then actually it's more of I just don't want to see it yet. And probably it's fine that I don't get to it until next week, it just happened to be that I thought that three days was the right number at the time.

ANDY: I think inboxes in general need a redesign. The way they work right now, you have this overwhelming set of stuff. And people either feel overwhelmed and only deal with what's new or feel an obligation to deal with everything. A lot of people will try to do inbox zero, "So I'm going to snooze everything to the future that like I can't do right now, and I'm going to delegate stuff and handle right now what I can handle right now." But then in order to actually deal with all of the items on their list, they might have to like you were saying snooze a thing three or four times. And the problem each time was not that they needed to not see that thing again today, it's that they needed to live in a system assured that they triage or saw with some consistency the things that remained to be done to make sure that nothing slipped through the cracks.

I think that can be handled in a way that's much less heavy handed and requires much less active and repeated action. I do have a system that works fairly well for me, which is I assign very constrained windows to deal with my email, it's about half an hour, three or four days a week usually. And in that I will get through whatever I can get through. And then outside of that time, I just completely put it away, and my thermostat knob operation is part of my weekly review. I ask do I feel I'm kind of getting through this at the pace that I need to be getting through it. Do I need to devote more time to this or less time, otherwise I just let it accumulate with my apologies to any of the present day viewers who are waiting on a reply from me to something.

And initially I feel bad about this because heartfelt and personal messages arrive and there's real correspondence that doesn't get action. And yet it is also just the case that responding to this is less important than using my time to do novel work. And this is one of the other problems with I think a lot of the way that we think about inbox design is that it's kind of a menu problem, a menu problem is like it's difficult to order what's not on the menu. And often the correct menu choice is not responding to everything and just spending less time and dropping stuff on the floor.

DEVON: Yeah. And the inbox not only can constrain what you think you should be doing, but it feels really wrong not to respond to them all. The affordance is that you should respond to these or deal with them in some way.

ANDY: That's right.

DEVON: I often find myself drawn to my email inbox when I'm not quite sure how to solve a problem on something important.

ANDY: Yes. Absolutely.

DEVON: And because it's this really easy win-

ANDY: That's right.

DEVON: ... where I'm like archive, archive, respond, archive.

ANDY: You feel like you're doing things.

DEVON: The number goes down, it's like a little mini video game, it's a little addictive, but also not generally that important.

ANDY: I've had to completely ban myself from doing this. So I'm not allowed to touch my email during like real work time, it's not real work. Meaningful correspondence with a research colleague can be real work, but it is in a cordoned-off section of time because I will absolutely do that. But the writing inbox is working better for me. And I have a little bit closer to a dynamic system, where I'm kind of constantly collecting the material to write about whether it's when I'm out walking, or I've read something that interests me. And I throw it into a writing inbox and I have various mechanisms to let me view a slice of older stuff, repeatedly view things that I'm not sure how to deal with yet.

ANDY: And I don't really force myself to clear it, it feels more like a candy store or I want it to feel that way. And often it feels that way, but my aspiration for it is I walk into this room of juicy things and it's like, "Oh, what should we write about today? Aha. That is an interesting idea, and I have something else to say about it right now." If only my email could be made to feel like this and usually it's like some burdens.

DEVON: Why is there a difference? Is it mostly about the types of contents that are in there or is there also something about the way that the inboxes are structured and the way they work?

ANDY: Yeah. I actually have some rituals around the writing inbox that help its recur in interesting atemporal fashions. A challenge with email is that it's very easy to do in most recent in, most recent out, you see the top of your inbox and you can rearrange it to make that more difficult, but that's the default arrangement. But with my writing inbox, I've set up tools to make things resurface at different times. So I'm always seeing interesting strata of the writing inbox. And when something's resurfaced a few times I'll push it out of mind eventually if it doesn't generate anything interesting, I'll archive it or it'll move to some very slow stratum.

DEVON: So essentially to keep it like a candy store, you have to do a lot of curation over time and make sure that everything in there is high value or you really strongly feel like it has the potential to be high value?

ANDY: Well, it's funny I want everything to feel like a candy store, but I also don't want to feel the burden of curation, because this thing about having to manage the inbox down to zero that sucks, making these decisions about am I going to snooze this? Am I going to respond to this? I feel guilty about it, I hate that. I want the good stuff to float to the top and the bad stuff to sink to the bottom. And I think you can arrange that just by these temporal strata that I was referring to where if you don't respond to something after it's been surfaced to you for enough times, then you can take out the rotting vegetable and put it in the trash bin, and you'll probably not notice without explicitly curating it.

DEVON: Just because it sinks to the bottom, you just don't see it again.

ANDY: You don't see it. Right. Right. And so the question of whether the system actually removes it or whether they just practically speaking, it's going to be a long time until you see it, that's kind of a detail, but I think actually it doesn't matter. If it's three years until you see it again then it's basically gone.

DEVON: So in other fields like Yo-Yo Ma practices basic skills, Steph Curry stretches, lift weights, this is very common. These are really simple core exercises that help them perfect building blocks, which they then build up into bigger chunks of excellence. What are the equivalent core exercises for a practicing doctor?

ANDY: Oh, that's a great question. So I have a very secondhand view of this, my wife is a doctor. So case series are an important part of their practice, they have a tradition of assembling as a department and have a member of the department present an interesting case which occurred recently. And the attendees are intended to try to guess at a diagnosis as the information is being revealed. And as I understand it the goal is to try to arrive at a diagnosis as early as possible. And sometimes there will even be structured competitions around this where some people will be put on the spot, trying to be the first one to diagnose it or to get the diagnosis right. Another important part of their practice is that when they're seeing and this is true at academic hospitals, I don't know how common it is everywhere. When they see a patient who has an unfamiliar condition or a condition that they've only seen a couple of times, then they'll go dig into the literature and read a bunch of papers about what's happened here.

What do we know about this? What remains unknown? What are the interesting counter-indications that I have to make sure not to do. And so every day's swath of new patients might be stimulus for the equivalent of and now you need to work on this stretch. You need to learn about this kind of disease because you have this patient. There's also an interesting practice of continuing medical education, where I believe it's the case that in order to keep their license, they actually need to earn a certain number of units of continuing medical education, which you can get a certain number. You can attend a workshop that's teaching you about latest techniques in performing this particular kind of procedure. Or my wife helped edit this journal called Continuum, which for neurology is kind of these big review articles, it's like what do we know about how heart diseases interact with the brain right now?

What do we know about this? What remains unknown? What are the interesting counter-indications that I have to make sure not to do.

I don't know how much deliberate practice there is for that. There's certainly physicians who advocate for these more core things to be actively developed. Maybe you've read Atul Gawande's written these pieces about having coaches even for very, very core practices in medicine, and that's in addition to the checklist that he helped develop as well. But I'm not sure how widespread that stuff is. I think actually the things I've articulated are more about learning subject matter, and it's similar for programmers, for instance. Programmers I know will very often be studying a new programming language or a new library or something and it's very different from, I don't know, honing their skills within.

DEVON: Why do you think this is more common with athletes and musicians than it is with programmers and perhaps physicians, working on those core skills?

ANDY: I'm willing to basically accept the traditional answer to this question for now, which is the core skills of an athlete and a musician and a dancer, and these kinds of things are really well for structured. So we kind of understand there's a certain number of muscle groups that you train in a certain number of ways, we can assess their current state really easily. We have known practice methods for developing those skills, if I need to learn a particular scale on the piano, we know ways to drill that, if I'm deficient we know how to improve me. And there's a tradition of training and teaching that has developed to support all of these things. And so only some subset of those things exist for knowledge work. I think if you're a contemporary software engineer, some parts of that job we might call it instrumented, maybe we can know what you know about red black trees or something like that, if you want to call that a skill.

But what about something like breaking down a software deliverable into tasks or into an architecture to be implemented. I think that's a fairly difficult thing to assess. We don't culturally really have agreed upon a set of levels, or know what it is to be better or worse than that except intuitively. I think it's often not clear to people that they might need to develop in that until it's too late or maybe not even then. And we certainly don't have well developed practice methods. You can just architect a bunch of systems and that'll probably work, but that's pretty different from improving your agility on the piano by playing a certain kind of Czerny exercise or something like that.

DEVON: Yeah. Or if you're improving your tennis serve, if it doesn't go over the net it's pretty clear that it didn't work so well. Or if it wasn't going a certain speed, there's things that are much more measurable, probably throughout tennis or throughout piano.

ANDY: Right. Right. A lot of it is because it's physical and it's external. While I bet if there were a tennis pro here they'd be saying something it's not just watching it go over the net or going the right speed, there's all this subtle stuff about the musculature that needs to be right. And that is really difficult for the tennis player to assess themselves and maybe someone else in the court even to assess. But if we have a good camera on the sidelines, then we can watch the tape and slow it down and make sure they're moving thing in just the right way. And likewise with piano if I videotape my hands I'll sometimes notice that I'm playing with only with my fingers, it's like a bad thing to do. I need to be playing like with my whole arm and my elbow and my wrist and my body. And so game tape could be helpful for them in a way that sometimes it's helpful for knowledge workers, but I think often it's not.

DEVON: Because it's in your head or...

ANDY: It's not externalized. Exactly. I've done a few experiments where I've recorded the game tapes sometimes publicly of I'm thinking through a thing I'm confused about conceptually and it's just an idea and I'm writing. And I think it's very difficult to identify where my thinking processes are weak and where they're not. There's a lot of ways in which my ability to synthesize and think through problems and ideas could be improved, but I don't really know how to systematically identify them.

There's a lot of ways in which my ability to synthesize and think through problems and ideas could be improved, but I don't really know how to systematically identify them.

DEVON: I found that pair programming can be a very interesting practice, both when you're the driver and when you're the navigator, because you can see how people think through the problems very differently than you would. And sometimes they have a better way to approach it or it's just a different way if you happen to be stuck in that moment, that can be very useful. The live streams that I watched of yours, it seemed like you were very much in the mode of teaching people or modeling how one could think, was that an implicit or explicit choice?

ANDY: No, it was explicit. And that's like a limitation or an opportunity to form that. I don't really know what I'm doing with those videos yet. Is the purpose of this video to faithfully record what it's like to actually watch over my shoulder, which would have no talking whatsoever, it's just me with a text file. Or is it a way to get an example of how to use a particular method to think about a thing, in which case now we're distorting the method, because normally I wouldn't be speaking and it's slowing me down and it's making me self-edit, maybe perform a little bit. I haven't quite figured it out and as you've probably seen, I stopped recording these in part because I don't have satisfying answers to these questions. I think there's value in making this game tape to learn from others, to help learn about myself, maybe to get feedback conceivably, I don't know, but I haven't quite pinned down where the right value is.

DEVON: So you've spent a significant amount of time in Dynamicland, what do you think they got wrong or what they maybe didn't emphasize enough?

ANDY: So Dynamicland is a research lab and a project led by Bret Victor and executed as well by a number of collaborators and it is difficult, it resists summarization and it has an ethos slightly of illegibility. So I will necessarily butcher it by trying to summarize, but something like it is trying to portray a vision of computing that is physical and embodied. So we interact with computers using physical objects and using our bodies and interacting with each other socially in an environment, which is highly malleable, so that the line or the boundary between the creator of the computing system and the end user is blurred as much as is possible, is highly learnable and legible. So that if I see some computational thing in the environment that I find interesting, I can always open it up and see how it's made.

Ideally not even having to take an action. I can just look down and see how it's made very directly. In many ways Dynamicland wants to make computing approximate a wood shop, which is a very interesting ambition. It's tied up with a whole bunch of very rich values about how the future of personal computing might evolve, which are difficult to summarize. Dynamicland is trying to be a highly social computer. It's not just a computer which takes place in physical space, it's a computer which takes place in a shared physical space. This is a large environment that many people inhabit and it is all the same computing environment. And there are no seams in the computing environment, so what this means is that if I'm sitting at a table where you're doing something on the computer, what you're doing on this table may reach out and interact with what I'm doing over here at the other end of the table.

And while that sounds like it could be annoying in practice it's extremely rich socially because it means I can look over and see that you're doing this very interesting thing on the table. And I can now riff on what you're doing, it feels a little bit more like computing as a jam session or something like that. So it's a very exciting vision in the future of... Computing or end user programming for my money, it's one of the only like interesting serious projects going on in trying to figure out what the future of computing could be, and so it's precious in that way. As to what they've gotten wrong, these are maybe some observations which they're already aware of, I don't want to speak out of turn here, but limitations are opportunities for future work.

One thing which comes to mind is that Dynamicland is a strange laboratory. It was a space in Oakland that is no more, but it's a physical environment where the primary activity being undertaken was creating this very unusual computing system. And so insofar as there was pressure in the environment to evolve a new kind of programming computational system, that pressure was being able to make new programming and computational systems. So there's a kind of recursion there that maybe leads the design of the system away from the needs of visitors to the space. So the aspiration was we're going to have artists and musicians and scientists come to the space, and they're going to use the system and they're going to push around the development of the system in important ways. But in practice, it was very difficult to get a scientist to like spend a bunch of time doing their work in the lab.

And so most of the work being done and most of the work being observed was about building new software systems, or we'd have a class of artists come in for two weeks and they'd do some stuff. And some of that would be interesting and rich and we'd improve the system based on that, but then the artists would go home and then it's back to building more computer systems. And so I think that in order to grow the system they'll need to situate its growth in some serious context of use that is not about building a system. And in fact, that's exactly what the principal investigator is doing right now. He's picking up and relocating the work to very interesting synthetic biology lab, where maybe now that the further development of the system will happen in a way that's meant to support this professor's research.

And so I think that in order to grow the system they'll need to situate it's growth in some serious context of use that is not about building a system.

DEVON: An idea that you've talked about a bit that I think overlaps with some of the ideas of Dynamicland is peripheral vision. How would you design a messaging app from scratch if you actually took the idea of peripheral vision, seriously?

ANDY: Oh, I love this, this is great. So a problem that I have with messaging apps is that sometimes like you Devon will send me a message, and I'm like not in a position to think about it or take it seriously right now, and so it disappears out of sight, out of mind. I can mark it unread and we have this really janky way to make sure that I'm keeping track of like, "I owe Devon an answer to this question." But I notice that I don't have this problem with physical mail, if I get a note from someone in the physical mail, it will live in my physical environment, it'll be on the dining table or on my desk or something. And every time I walk by it I get a little bit of stimulus that like, "Ah, Devon sent me this postcard, maybe I should send her a postcard."

And it will naturally stay there until I feel authentically and organically inspired to respond to it. And that's really different from as a task I now need to go triage my inbox. Instead it's like I was walking by, I caught this postcard in my peripheral vision and I was like, "Oh, yay, Devon. I miss Devon. I'm going to write to her." And so I think you could make a messaging app that worked like that physically in your environment where the messages they just get pinned, like post-its around your house. And you could scribble on the back of the post-its and it would go off in the reply and similarly, it could kind of just hang out there until I feel inspired. And maybe group chats then begin by I can kind of pick up some message and then put it next to some others, and now we're in this kind of republic of letters together exchanging weird postcards.

DEVON: That's such a fun idea. During the deep pandemic, I experimented with a Figma board with a group of friends.

ANDY: Oh, cool.

DEVON: And we were using that as our messaging app. It had the problem that I don't open that Figma board for any reason.

ANDY: Just check it.

DEVON: I have to think to check it, so it really actually breaks the idea of peripheral vision. But I was wishing that I could just project it onto a wall and have it open at all times, and then just over time see when they've added things.

ANDY: This is one of the other things, another important principle of Dynamicland that they'll sometimes describe it as breaking us free of the tiny black rectangles. We're all stuck to the tiny black rectangles. So we have the infinite computing power, practically infinite of the Silicon in my M1 Mac, this endless computing power I'm not using. And also there is a bunch of light space on the walls of this home, there's a bunch of windows too so it's a little hard, but this space you could project. But I'm limited to this monitor and setting something else up beyond that feels really onerous, it's like a big burden. And so there's this aspiration that what if you could get something that's more like a lampshade.

Fairly recently this corner of the home was feeling a little dim, so we went out and got a lamp and you could plug it in and it's like, "Cool, now there's enough light over here." Saying like, "I'm feeling a little confined computationally. I want my computing to spill out into this part of the home, you can't go buy a lamp." But you didn't do it, you had this idea and you could buy a projector, projector's like cheap, projector's like $150 or something, I imagine the cost wasn't the issue, it's like it's too much overhead.

DEVON: I have done the projector thing on the wall though it wasn't for this particular project. It feels like for it to work well, the way that I put things on the wall matters too. And I think actually for the Figma art board it would work better because other people putting things on the wall, and so it's like I'm coming by to see them. But I was using it for other things almost as a second monitor, but on the wall instead of on my computer and it was just awkward, it like felt weird sort of like the Wacom tablet thing you were talking about before. It's awkward to put things over there because I feel like I'm here, it was just off in a bunch of different ways, so that didn't work out so well. But if it felt more like a bulletin board that I walked past and other people have put things there, I think that would work a lot better.

ANDY: Figma is neat because it's highly visual and it's very easy to make things colorful and weird and varied. A problem that many computer interfaces have is that text has this fundamental sameness to it. So my writing inbox, it's just like a big list of propositions. And my email inbox is a big list of subject lines, and they're all kind of these identical little tick-tacky houses. And so it's very easy to not just lose object permanence and lose sense of where anything is, but also just to become totally desensitized to the list, it's like, "There's always going to be a list it's overwhelming, those lines of text." For all time, there are always going to be lines of text in my inbox. And that feels totally different from some weird space where today Omar decided to make this big purple square to make his point, because he was excited about, I don't know, a field of wildflowers and what it had to say about pollinating computation or some strange Omar thing.

A problem that many computer interfaces have is that text has this fundamental sameness to it. So my writing inbox, it's just like a big list of propositions.

DEVON: Yeah. Something I've done is for my most active group chats, I've set the theme to different colors.... there's this like a cottage theme on Facebook Messenger, which is very adorable. It just makes it much nicer to go into that chat because it's like, "I'm home."

ANDY: Wow.

DEVON: It feels very different from all of my other group chats. And I really wish that more messaging apps did that because otherwise they all feel like you're in the same room, and talking to the same mannequins, because there's just the blobs. And I think if there was less sameness and more difference, they would feel more real spaces that you're inhabiting.

ANDY: This is I think another opportunity for the pervasive audio input, output tech. I will very often send messages to these groups by dictating when I'm walking because it's annoying to type when I'm walking. And there's information being thrown out, so there's the expression of my voice that's being thrown out when we do that. And I could record an audio message and send it to the group, I find audio messages annoying to receive so I don't do that. But you can imagine some strange type of graphic presentation, that's trying to actually like take the effect of my voice into account. If you search around you'll find all kinds of experiments with this where people make the letters big or small, or they stretch them out or they change the color, or the tone of the letter according to timber to try to really make every message feel visually unique.

DEVON: It's like the old Mac audio visualizer for music, but for someone's voice.

ANDY: I don't know this.

DEVON: Did you ever play this?


DEVON: I might be misremembering what it's called, but it was this cool thing that you could like... it was in iTunes I think. And you would just play a song and it would like visualize the music based off some of the input output. And I really enjoyed it when I was younger and you could have something like that, but for someone's voice. And if they're speaking really quickly or excitedly, maybe you have neon colors. And if they're saying something really sad, maybe you don't want to animate that at all, because it might be disrespectful, but you could play with different things there.

ANDY: Something that's interesting about style guides in art directions for modern systems is that they kind of necessarily need to be practical and functional. And so that means that we need to choose say a font that's highly legible to use throughout the system. And we need to not be irreverent in places where it's inappropriate to be a irreverent. I remember in Messages the Apple Messages app, you may notice when you scroll there's a little bit of lag in the balloons, it used to be much more, it used to be basically jaunty and playful, "Oh great, these chain of balloons." And then Jony Ive... he was using a beta version before we'd shown it to the public, and he went to a funeral of a friend and he was texting to coordinate for the wake or something like that, there were these jaunty bubbles bouncing into each other and he is like, "Oh my gosh, this is totally inappropriate."

It was inappropriate in that moment, but only because there was this assumption that the same art direction should be applied to every conversation in every moment. And so what if we could take this cottage aesthetic that you're describing and extend it not just to the background and the color scheme of the bubbles, but to the typography and the way that we treat the interactions. Maybe there's a dance party aesthetic that layers all the bubbles chaotically on top of each other, in order to make a crowd feeling.

DEVON: That's so fun. It feels like a theme of the last 10, 20 years has been around software scaling widely and being used in a lot of different situations, and that makes a lot of business sense, you can lower marginal costs and so on. But you end up losing a lot of context, which could allow you to build more special software that creates you feel a particular thing or solves a particular problem. And it seems like I've been hearing more people talk about specialized software systems, or homegrown software and that sort of thing, I'm very excited about that. Because I think that we've really lost something by having tools that are used by billions of people, and they have to fit all of them. They probably won't, they have different goals. Sometimes they're at funerals, sometimes they're at a rave. Sometimes they're doing completely different things. And so if the software can reflect those moments in your life, I think that the software would just mean a lot more to people.

ANDY: Yeah. Absolutely. Robin Sloan has this great piece, "Software can be a Home-Cooked Meal." I really love this. And if we take it seriously, we get to this observation that I can cook dinner without having to go to some central industrial kitchen because I actually have a kitchen in my house. And that kitchen has the appliances I need and pots and pans and stuff, I have the tools of the trade in order to make a home-cooked meal. And they're less serious in certain ways than the industrial kitchen, and so I'm not allowed to make certain kinds of meals even, but I can do it. And I think that kind of reveals an issue in software, even with decades of work on end-user programming research, it's still very difficult to make a homemade meal kind of piece of software, even for an experienced person like me. Just on a whim, you and I are in a particular message group and I want to make a messaging app just for our little chat group, it's like that might be a multi-week project even for a very experienced person.

You can probably hack something together in an afternoon, but it wouldn't be good enough. And this is I think what a lot of projects like Dynamicland are pointing towards, a malleability, this sense that I can take ingredients and manipulate them with my hands and with tools that I have available to me, and possibly hands metaphorically, if we're not talking about Dynamicland. And then I can make a home cook meal that really serves my purposes, I find that really exciting.

I don't know whether it's going to come through new kinds of programming environments like expanding something like Excel but for Messaging apps. Or whether it's going to be something more low code, no code kind of environments where they figure out... I guess the analog might be a highly customizable messaging app. And so it's not that you're making an app for you and your group it's that there is an app which someone else has made, which has the affordances of allowing you to highly, highly, highly customize the feel of individual chats, much more so and that's it's differentiator, it's like a no code Messaging app, I'm not sure which direction we'll end up going.

DEVON: It might be all of the above.

ANDY: Right. Right.

DEVON: Because each one will probably end up solving for different uses. Different people will be attracted to different approaches.

ANDY: Right. And so much of this is not programming, that's one thing I find really interesting about it. If we just want to talk about people like to use the word vibe. No, if we want to create a different vibe for a different relationship, for a work situation, that doesn't necessarily involve programming at all it may just be much more about say art direction. And so if we think about enabling this kind of thing, as equivalent to as hard as solving the problem of end user programming then we may be creating a much more difficult situation for ourselves than is necessary.

This is a huge problem for the field, it's that independent people coming up with ideas in this field, I want to say this field, I mean kind of like, I don't know, novel software interfaces for empowerment we could say, they tend to be programmers. And that means usually that design is a secondary skill for them, but often the problems are fundamentally design problems. If we want to talk about like how to make a world where you and your little communities can set up a vibe for your communication environment, whatever that happens to mean, that's mostly a design problem, the programming challenges there aren't interesting.

If we want to talk about like how to make a world where you and your little communities can set up a vibe for your communication environment — that's mostly a design problem.

And so mostly we need like design thinking in order to solve that problem. But unfortunately, because of this chicken and egg, we still don't have end user programming world that we live in. Somebody who's just a designer, they can't on their own actually make such a thing. They can design some compelling prototypes maybe, but I think they're disincentivized to do that because they know they can't make it real. And so they'll often just do like a screen as kind of a concept, but they don't really flush it out, I think with good reason because it feels like a waste of time.

DEVON: I think the good news is there's actually a lot of low hanging fruit where currently people who build messaging apps, have it in their head that they have to all have the same UI like you were talking about. And if they just added a few basic things, like more ability to change the theme or ability to, I don't know, pin pictures in weird places. There's a few more degrees of freedom that I think could be added that would unlock a lot of ability to make it feel more your space, that are just not even difficult technical problems that currently most designers are just not thinking about at all.

ANDY: Yeah. That's probably right. Kate Compton, another researcher in this space has a phrase that she uses for describing the systems, she's like, "You want to make them deep enough that you can access the poet lode." As if there's a bunch of oil hiding under some surface and that's where the poetry is. And so these systems, if you make them really shallow in one respect, then there are things that only programmers can use or only nerdy technical engineering people can use, and so there's no poetry from those people a lot of the time. And then another way if you make them so shallow that all you can do is like change the background color and change the name of the chat or whatever, then cool everybody can use it, but there's also no poetry.

Kate Compton, another researcher in this space has a phrase that she uses for describing the systems, she's like, 'You want to make them deep enough that you can access the poet lode.'

You actually have to make it deep enough so that real people who are poets, whatever that means for your field can express themselves in ways which surprise the creators, because the creators are usually not poets. And that's very difficult to make a system, which is that expressive when you yourself can't express in that way, usually it requires pretty deep collaborations.

DEVON: There are many different niches that a talented person might fill in a field, like a researcher to uncover ideas, engineer to build things, communicator to spread the ideas and then spread the things that have been built and so on. You've filled many of these different niches in different times of day, different times of your career. How do you think about wish niche makes most sense for you to focus on right now and in the future?

ANDY: I try to guide my work first through just a sense of visceral excitement. And I'm really lucky to have my life structured so that I can do that. But given that my life is structured in that way I feel like it's almost an obligation. I want to be really excited about whatever it is that I'm thinking about. And sometimes what I'm excited about is communication, sometimes what I'm excited about is building, that varies. And then if I have a big set of things that I'm very excited about, then I will ask things like counterfactual impact and things like that. Where basically I'm curious about what do I think is going to happen if I don't intervene here versus if I do. And that has often guided my work in the direction that it's taken, at least for the past seven or eight years in my career, I've been trying to focus on things that I think are really underrated, or underrated by people who have the kinds of skills I have, where I think about if I don't go work on this problem what do I think is going to happen?

I think maybe like not very much progress is going to happen, it may also be the case that if I work on it not very much progress is going to happen, but at least I think there's some a shot where the counterfactual doesn't look nearly as good. I've noticed that the people working in this space tend not to be very interested in design as a discipline, or to have that as part of their background. And so when I was looking at the people who were working in the space, basically, I just didn't really see any designers. And I thought that if I kind of come to this space with both a willingness to engage with the literature and with that background, then maybe I can solve some of the problems that are keeping people from using these systems.

DEVON: You've expressed skepticism for software tools that have aimed to augment the human intellect in the past. So for example, you've mentioned that the digitization of dictionaries hasn't changed the way we think or act very much. What would you expect to see if digitized dictionaries had changed the way we think more?

ANDY: I'm not sure. I think it's easy to imagine or many people did imagine that if you give people word processors, if you give people dictionaries at their fingertips, then people are going to use a much wider range of words, people are going to be much better at thinking through their ideas. If it were really transformative, what might we expect to see, maybe we should expect to see a huge blossoming in the publishing world. There should be so many more people writing books, writing at a particularly high level of scholarship, should feel more accessible. And my gut feel is that it just doesn't feel like this has exactly happened, it feels like maybe it's a little easier. If I take an outside view though I'm not sure I believe myself, it is the case that more nonfiction is being published than ever.

Some people on Twitter pointed me towards some studies that suggested that actually people do use a wider vocabulary now than they did before. And perhaps this is because of the instant accessibility of dictionaries and so maybe I'm wrong about this, but I think at a gut level I still think I'm not wrong about... it's maybe not as transformative as we might have hoped. I think that should give people like myself working in the space, trying to make tools that change the way we think pause because these tools seemed like they would have this absolutely enormous outsized impact. Maybe they had some impact that was pretty good. I think it's difficult to make the argument that it was totally transformative civilization or something like that. So what do we think about that? Does that mean that it's just not possible to make tools that are quite that transformative and that they're very, very rare. Or maybe we just need to accept relatively more incremental results, I haven't quite gotten my head around how I think about that yet.

DEVON: Well, one thing that might make this challenging to view is that a lot of it is an internal experience that somebody has. And also often even if you were to be able to see it on their screen, you're probably not there to see it. I haven't found dictionaries to be very useful in my own day to day life, but thesaurus' have been really helpful. I actually always have open in my tab. And it's not so much to find synonyms to make my language more complicated, in fact, it's usually the opposite where I have a complicated word and I want to find a simpler word. But the thing that I really like using it for is I have a word that doesn't quite capture what I want to say, and I want to sharpen the thought.

And then I like go through this web of the's words and then I hop through and then I'll see a word that I actually wasn't thinking of, but I'm like, "Oh, that describes another facet of this thing I'm talking about-

ANDY: Yes.

DEVON: ... and I want to use that word." And so I find that a really, really useful thing, whether that's world-changing is unclear, but it helps me get unstuck, it creates a path that I can follow.

ANDY: Stepping stones.

DEVON: Stepping stones. Which is I think not what thesaurus makers probably had in mind when they started making thesauri.

ANDY: No, but it's real. And I think you're pointing to something that's important about the digital edition specifically, this hypertextuality that you can jump around, that's something that would be painful to do with a physical text, you might not do it. One way to interpret this is that maybe the most eminent writers are basically writing the same thing they'd written before. Certainly if I read 19th century literature, 18th century literature, 20th century literature, it's not like you can identify some breaking point where, "Aha, now the computers are here and look the writing got so much better." That's clearly not there. But it is also the case that now everybody's got blogs and mailing lists and that's kind of strange, more people are writers in some capital W sense than has ever been the case before, and maybe we have these tools partially to thank for it.

DEVON: Yeah. To wrap up with one last question, how do you imagine the world might be different if your research and similar efforts were to dramatically succeed?

ANDY: So taking just my current research narrowly for a moment, we can ask what if it's the case that everything interesting in every book you read, everything compelling in every conversation that you had, everything interesting that you saw that was visually striking and so on, you just had trivial access to in your memory, you could learn things very rapidly. And so you might find more people being multidisciplinary. For instance, I really enjoy conversations that I have with my friends in synthetic biology, kind of a lightweight interest in that, when I think about entering that field, it feels a little onerous, a little overwhelming. But if the cost to learn a new field were somehow cut by a quarter, cut by a third, cut by half, then I might be tempted to add another hyphenate onto my list of skill now and identifiers, that might in turn lead to some very surprising creative and scholarly inventions and discovery, we might be really excited about that.

But also just from a more creative artistic perspective, as a designer I often feel that my strength as a designer comes almost from my mental Rolodex or catalog, of past work that I'm constantly referencing and drawing on for patterns. And my understanding is that the same is very much true for composers, architects, things like that. And if it were much easier to build that kind of catalog or to add to it, I think that my artistic work, my creative work would be more compelling. And so culturally, you might see more interesting production as well. I also think emotionally there's this very interesting opportunity to use memory augment organization, to connect more consistently and deeply to experiences that were very rich and to keep them vivid in your life and for others around you to do the same.

As a designer, I often feel that my strength comes from my mental Rolodex of past work that I'm constantly referencing and drawing on for patterns.

And that there are probably some interesting network effects in that. For instance, you and I have had past conversations where you've said things to me that I now come back to repeatedly because of these systems. I think that's interesting, and if you told me that the same were true for some things that I'd said, I would find that very interesting too, and that's for intellectual things. But if we're talking about experiences that we'd had together, I knew that you were coming back again and again to that time we went and hiked this place, that would make that part of our shared experience more rich in my mind too, it would make me feel closer to you, so it's a ritualistic way that we can share.

DEVON: Well, thank you Andy. This was a really fun conversation. I really enjoyed the excuse to dive deeper into your work.

ANDY: Thanks Devon.

Share this post

Try it now

Get going on web or desktop

We also have Mac & Windows apps to match.

We also have iOS & Android apps to match.

Web app

Desktop app