Joseph Smarr

Thoughts on web development, tech, and life.

AI and the future of jobs

I was invited to give this year’s keynote address to the German International School of Silicon Valley, where both of my kids attend. The high school students have started taking internships and thinking about jobs, so the school wanted me to share some thoughts on how AI may change the future landscape for work and how the students should think about preparing. This is something I have a lot of thoughts about, although (as you’ll hear) I’m the first to admit that no one has a clue how the future will play out, especially these days.

Nevertheless, I was excited to pull together a number of strands from my own career and related topics I’ve read and thought a lot about like wealth inequality and even music, and deliver it in what I think turned out to be a pretty engaging and broadly consumable format. I was pleasantly surprised how many students (and also their parents) came up to me later (even days later) to say how inspiring and thought provoking they found it, so I thought others might like to hear it to.

My wife recorded the talk on her phone from the back of the room, so the quality isn’t the best, but I’ve extracted the audio, cleaned it up a bit with Audacity, and made a (lightly edited) transcript using Descript. I’ve included both below. Let me know if you have any feedback or if you see things differently! When I polled the students at the beginning of the talk, they were generally more worried than excited about how AI will impact their future job prospects. I hope my talk injected some cause for optimism. 🙂

Transcript

[00:00] Now that you’re in high school, it’s natural to start asking “what’s the world going to want from you all?” as you get out of school the next five years, or what’s it going to want for you in the next 10 years, 25 years, 50 years, there’s a lot of future to be had. Right now is when you’re making choices about how to prepare yourself for that. Trying to figure out what’s the optimal thing to learn and do. Honestly, and I say this as a parent, it is really hard to figure out right now. I think that you’re entering one of the most uncertain times in our future that I can certainly think of an analogy for, for a number of reasons that we’ll talk about.

So the short answer to my talk is I don’t really have a clue what to tell you to do. And anybody who thinks that they do and gives you a confident answer about what to study or how to learn or what’s going to happen, what jobs robots are gonna take or not is full of shit (excuse me). But it’s important to know that we’re entering an era of pretty big uncertainty.

I do think, however, there’s a lot that we can think deeply about that will help you, I hope, ground your own decisions [01:00] about what you want to spend your time doing. And so that’s what I thought we could talk about today. And there’s a number of sources of uncertainty, but obviously AI I think is one of the biggest ones that’s the elephant in the room because it is the thing that could be displacing a lot of traditional knowledge work.

So I’m curious first, just to make sure I know what your own experience is, raise your hand if in the last month or so (i.e. recently enough) you’ve used one of these frontier large language models like ChatGPT or Claude or Gemini, or one of these things. Looks like not everyone, but almost everyone.

That’s good. How many of you are using a paid version that either you or your parents are paying for that’s higher power than even the normal free version? Not many. I mention it because there is a difference, and I think one of the best ways you can start to think about how to change your own approach to things with AI is to just use the latest and greatest stuff. So tell your parents, from me, I think it’s a wise investment.

How many of you that raised your hands have done more than just played with [02:00] it, but have actually say made something or built something or learned something that you’re genuinely proud of and that you probably wouldn’t have happened if you hadn’t used AI? A few of you. I certainly feel that way. I think you have to taste it to realize what’s behind it.

Okay, last question we’ll do, this as an A/B test, but there’s no wrong answer. How many of you, when you think about your own future, think that the presence of AI in the future of technology generally makes you more excited than worried? And how many of you are more worried than excited? Excited first. Yeah, very few. How many more worried than excited? Oh, okay, the majority. Interesting. Interesting. Okay, maybe I’ll be able to change your mind a little bit, because I’m fundamentally an optimist about this stuff.

Just a very quick bit about me, so you know where I’m coming from. So, as you can hear, I’m American born and raised in America. My connection to the German school comes from the fact that I had the good sense to marry a nice Austrian girl 20 years ago. And, [03:00] so I have kids here in sixth grade and ninth grade. So that’s why this is a personal topic for me as well as a topic of professional interest. I came out to California in the late nineties to study AI at Stanford, because actually even 30 years ago, it was clear to me then, and, obviously not to as many people as it is now, but that AI was going to completely change the world and usher in a sort of second industrial revolution and have a really profound effect on society, and I really wanted to be a part of that.

So this is something that I’ve been working on and thinking about ever since. I did academic research for a while. I’ve been in a number of startups. I was at Google for over a decade, including working on Google Assistant, which was the predecessor to Gemini. So it’s something I’ve done a fair amount with personally. But I also have thought a lot and read a lot about the sort of economic and societal implications of technology change in AI in particular, because I’m very concerned about [04:00] wealth inequality and making sure that we can not only create the future, but all share in it. So that’s where I’m coming from on this, and that’s what I’ll talk about today.

So why do I say that we’re in such an unpredictable time? From my vantage point, and I’m curious if you agree with this or not, I think we could be on the brink of a kind of technological utopia in your lifetime. That is the stuff of only science fiction until fairly recently. If you think about it, we could have abundant clean energy and self-driving cars and flying cars and limitless intelligence helping us to solve long unsolved problems in science and climate change and curing diseases and brain computer interfaces and exploring space and all that. These are all things that have just seemed completely out of reach for pretty much all of humanity, right? But you can just go around here within a 10 mile radius and see people working on all of these things with really credible paths to making, [05:00] continued progress, right?

So I think if we can stick around and if you can be part of ushering that in, it could be an absolutely wonderful thing there. Humans were subsistence hunter gatherers and farmers for almost all of human existence, right? And it’s really only since technology started accelerating that we’ve seen this sort of massive ability to bring everybody’s standard of living up and there’s no reason why we can’t go significantly further and we better hope we can.

Because we still have a lot of problems in the world. Despite all that optimism, we also, I don’t know about you, but if you read the news recently, it’s hard not to escape the fact that we might be on the brink of civil war and pandemics that we’re totally unprepared for, and runaway climate change and cyber attacks, or AI running amok or mad men running around with nuclear weapons. It’s pretty bleak out there, right? Let’s be honest. And that’s happening at the same time, right?

I think the more you study history, the more you come to appreciate the fact that the future is not foreordained. There’s a lot of fairly random things that can cause it to go down [06:00] very different paths. You are all going to have to navigate these, sort of real high highs and real low lows that we’re both staring at. A lot of possibilities. And on top of that, I think even if we are able to create a lot of technological abundance, there’s no guarantee that it’s going to be shared widely enough to maintain social cohesion. It could very well be that it continues the path that we’re on right now of concentrating wealth and power in a few hands and everybody else not sure how to participate. That’s something that you’ll have to deal with as well. We’ll talk a little bit about that

The last thing I would say is that it really was true, maybe not for your parents, but certainly for your grandparents, maybe even your parents, that you could think “I’m going to school for 10 or 20 years, I’m going to learn some skills, and then I’m gonna practice those skills for the next 40 or 50 years, maybe even at the same company the entire time, and that’s sort of it”. I just think those days are over for sure. I think the rate of change is accelerating itself so quickly [07:00] that you’re going to have to be lifelong learners. You’re going to have to change your identity, you’re going to have to do multiple things. You may live a lot longer, if we cure a lot of diseases. But you may also just have to reinvent yourself and think of school as more of a launching pad for more fundamental meta-skills like learning and curiosity and so forth. So I just think that makes it that much harder to predict, because it’s not just “oh, I’m gonna learn X and that’s gonna gimme a good job”. What the AI can do, and what you can do is going to change and the set of possibilities are going to change.

So anyway, that’s the backdrop I would think about. Hopefully I haven’t depressed you too much! I do think there’s good news. What I try to do when I think about my own kids’ future, when I think about the world more generally, is try to think more deeply about what is the way that you can add value to the world that is fairly universal? What is it that causes a job to be valuable in the first place in a more fundamental way? And also what are human universals that are unlikely to change even in the world of rapid [08:00] technological change? So that’s what I wanted to talk through with you, because that’s how I at least maintain some grounding and some optimism despite all this uncertainty.

So I don’t know if you’ve thought much about it, like why do some people get paid more than others when they’re both working hard and the same number of hours? It’s a huge range, living in Silicon Valley, there are people working super hard, making no money and there are people working super hard making unfathomable amounts of money and everything in between. Anybody have any ideas about what sort of fundamentally drives that? You can shout out if you want.

Economists will differ on this. I’m not sure there’s a well defined agreement on this. But the way I think about it, for what it’s worth, is I think there’s sort of two factors that are interrelated. One is how much supply versus how much demand exists for the skills and knowledge that you have and the connections you have and all in what you can offer. The other is how much [09:00] leverage do you get in the world based on the application of that effort.

So as an example, if you’re a Uber driver or you work on building construction or there’s lots of jobs like that where it can be hard work and you can spend a lot of hours doing it and you still don’t make a lot of money. The reason is lots of other people could also do that job without a ton of training. And you’re only driving one car at a time or fixing one house at a time or whatever. Even traditionally high paying jobs like doctors and lawyers and that sort of thing, they top out because you’re very well trained, so there are not that many people who can do what you can do, but you’re only fixing one person at a time or writing one contract at a time. Whereas the sort of amount of unbelievable wealth that’s been generated by Silicon Valley, or Hollywood for that matter, comes from the fact that not only do you have a lot of very specialized knowledge and a lot of very specialized connections and work and experience and so forth, but you’re changing, you all have cell phones in your pockets and you’re all watching Netflix and all these things, right? The amount of impact you can have from the leverage of your impact.

[10:00] It always made me a little humble at Google. I would write some code and push a change and it was like, “okay, a hundred million people are gonna wake up tomorrow and see that”, it was like, “I’d better not fuck it up”. It’s huge leverage. So I would just say in general, there are lots of different problems in the world that need solving that you can go out and decide you wanna get excited about, but I would ask you to keep in mind: where can I build up a set of, like I said, not just, book smarts, but like skills, practical, real world experience, people, like the whole set of what it is you amass? Where can you get on the right side of that supply and demand curve, and how can you find ever more leverage to use from that? Can you, instead of fixing something in one place, can you build a system? Can you teach other people to do it? It doesn’t have to just be traditional technology, but other ways to find leverage.

The problem of course, is when you’re trying to think through supply and demand, what AI is going to fundamentally do is add a lot more supply of a lot of different skills that previously didn’t exist, right? So like in my own [11:00] field of software engineering, AI is getting really good at writing code. So does that mean it’s no longer going to be valuable to write code because the supply and demand is going to get totally messed up? You can ask that question both in terms of what’s in demand and also where is the AI going to provide the supply. Of course, it’s hard to predict because the history of AI, which dates back now well over 50 years, is constantly a history of people saying: no computer can ever play chess or translate human languages or talk in a natural voice or compose new music or, like all these things people have said very confidently. And of course computers have steamrolled over all of those expectations. And they’re showing no signs of slowing.

Even in the real world, robotics are still lagging behind AI just in a digital format, but I think there’s a lot of progress going on right now with robotics and actually embodied cognition is one of the things that really unlocks, like having a good brain from AI makes a lot of robotics tasks possible in the real world that [12:00] work. I think the most salient example, maybe you don’t think about it this way, but if you’ve seen Waymo’s driving around in the streets here, those are robots, right? They’re machines that are using perception to navigate the real world and not bump into things and decide where they want to go. They’re robots. They just have wheels instead of legs. But it really does work and it really is a total game changer. And it is potentially displacing a huge amount of human labor, right? Actually, driving a car is one of the top professions in most places, right? And, totally unclear if that will still be a thing 10, 20 years from now.

So you have to be careful about what AI can and can’t do. But if I look at what I’ve gone through as a software engineer, just give you some personal experience, which is, on the one hand, it is really amazing how much good code AI can write and how much more it knows about esoteric aspects of different things that I don’t know about. But it’s also been amazing for me to see, as someone who uses it every day now for years, how much value I still have to provide on top of that. I don’t say that to brag. It really is just [13:00] fascinating to me that I wouldn’t have been able to tease apart the parts of my job that are replaceable by AI from the parts that aren’t. But there’s actually a ton of both.

For example, if you tell the AI: write a program that does X, Y, and Z, it might do a decent job of that, but it is not keeping up with what’s going on in the industry and what was just happening in the meetings and what’s the overall roadmap of the team and what’s our unique advantage and, oh, this thing you used, we were planning to tear that down next year anyway, so please don’t use it. There’s so much additional context that you as people bring to the puzzle. There’s also, for lack of a better word, the wisdom I’ve accrued as a multi-decade software engineer, which is there’s lots of ways to build things that are technically correct today but are unlikely to be as good in the future. They’re too brittle or they make too many of the wrong assumptions, or they’re making things unnecessarily complicated. One of the things you learn in any profession as you get older is those areas where you’ve done it a bunch of times and you know the sort of attractive pitfalls, right? [14:00] If you ever work with more junior people, that’s really the difference: they can be really smart, but they make a lot of “rookie mistakes” because they just haven’t been in it enough to really understand those second and third order effects. And you really see that too.

Maybe the coding systems will continue to get better at that, but unless they’re going to be sitting in all the meetings and chiming in on the slack threads and going to industry events and reading the news and so on? You could imagine a world in which my AI text editor is doing that, but it’s a pretty far stretch from the way things are today. Then putting even on top of that: I have to still wake up every day and decide I want to work on this and what I want to work on, and I have to motivate the team around me to get excited to work on stuff, and I have to get people outside to be excited about it or to get people to want to give me money to build my thing. All of that human stuff: building trust and getting people excited or having people feel like they need to work harder because they don’t want disappoint me, and vice versa, right? All that stuff is very human and it’s just not the same when [15:00] there’s blinking cursor on the screen, right?

So even as much of a optimist as I am about the continued progress of AI. I’m very aware of all the stuff that it’s not really on a path to do anytime soon. And that’s where I would suggest you anchor your own thoughts about how can I take an area. So in medicine, if you’re just being trained to read a radiology image or something like that, that narrow task might be very well doable by AI. But if you think about what it takes to be an effective doctor, let alone a medical researcher, let alone someone who helps change health policy in the country, etc., there’s so many layers above that that really need that human touch and I think will for a long time.

Now, like I said, I think you want to figure out how to be part of creating a lot of value, not just because you wanna make a lot of money, but also because you want to use your precious time as effectively as possible, right? I still think there is this big question, and I’ll be honest, I don’t know how we’re gonna fix it, but I encourage you to think about not just what can I do to [16:00] create a lot of value, but also how can I be part of shaping our future so that value gets shared broadly enough that the whole system is sustainable? Because, I think the history is pretty clear here that when wealth and power get too concentrated. a the economy slows down because there’s just not enough money circulating, there’s not enough people able to buy stuff. We already see today there’s a huge affordability crisis that most people have. Then at some point, political, societal stability breaks down too. And ultimately you get the French Revolution or things like that, right? And so we could absolutely be down that path where you’ve got a few trillionaires in their bunkers and we’re all using their stuff and we’re serfs to the AIs and we don’t want that world. They have solar panel arrays everywhere and so forth.

So it’s just interesting for you all to think about. It doesn’t mean you have to go into politics necessarily, but I do think that positive sum thinking is the secret to Silicon Valley. It’s basically “we can grow the pie”; you don’t have to lose for me to win. We can just make things work better. We can find new ways to distribute value. But I really encourage you to think about [17:00] that as well. what, any field you go into, what’s, where’s that opportunity to create that kind of positive sum game?

Alright, let me just wrap up here. I don’t wanna talk too long. The last thing I would encourage you to think about when you’re trying to figure out what you can do uniquely in the world is think more about what makes humans special. I talk a lot about this with my kids, but you really have to remember that as advanced as we are, we are monkeys, we’re social primates, right? That is not going to change any time soon. We are hardwired for social, understanding what each other thinks, building status in groups. That is a lot of what humans want and a lot of what drives us. And that is something that we really don’t want from machines. I think you can probably see that around you already.

I see a lot of FuĂźball Trikots here and there’s a lot of passion around that. There are robots that are pretty good at playing soccer already, but I just don’t think anybody is going to [18:00] be as passionate about a robot soccer player or a robot soccer team anytime soon as they are about the actual players that they care about. Why is that? It’s not because robots can’t be good at playing soccer. It’s because you don’t just care about the execution of soccer; you care about the human drama. Watching a player go from a rookie up or the team that you’ve stayed with through the highs and the lows or all the fans that are around you. It’s actually the human aspect.

This really hit home for me when, last summer we were lucky enough to be in Munich, when Taylor Swift played there in the Olympiastadion, and I don’t know how many of you have been to that place, there’s this whole Olympia Park around it and it was just completely full, probably 50,000 thousand people all dressed up with bracelets and singing songs and posters and shirts. Half them didn’t even have tickets for the show. They were just hanging out. And it was such an amazing energy and I was just thinking it’s so cool that people do this, but would anybody do this for an [19:00] AI-composed piece of music, no matter how good it is? Even if it was super catchy? No, of course not. It’s not just that it is Taylor up on stage, it’s the whole ethos of the culture that is around that and that’s what we care about, right? None of that is going to change even though AI will get better at writing music and maybe even really good music, but there’s still, people want to still see a human, right?

I don’t about you, but like I love live music, and I always think it’s funny, you can go see somebody play live even in like a coffee shop or whatever, not a big stage, and it’s still way more enjoyable than listening to recorded music through the speakers, even if the person who recorded that music is more talented than the person who’s playing it. It’s weird, right? Because if the goal was just, I want the best music possible to enter my eardrums, like the sort of very narrow idea of the work, then you wouldn’t ever have the need for live musicians, right? Because we already recorded it once and we can spread it everywhere. And yet I think if anything, the opposite is happening, which is we really enjoy that [20:00] live communal experience. We really enjoy seeing the person play. I love playing music myself too. Even though I’m not very good, it’s incredibly fulfilling.

So I think that’s the human universal stuff that I think is not going to change. And that’s to say nothing of, if you’re actually caring physically or emotionally for a loved one, a, child, a parent. Again, I think robots will have a huge impact on healthcare, but I think there’s nothing that’s going to replace the human touch; the feeling that a teacher really believed in you and made you feel like you wanted to be more because they inspired you or that sort of stuff. Again, we’re monkeys. So embrace your inner, monkey! Don’t, don’t run away from it.

The last thing I would say about that is that, I don’t know if you’re familiar with the psychology literature on human drive? You can read Daniel Pink or one of these people, but basically the thing that motivates humans is a combination of desire for autonomy, mastery, and purpose. Have any of you heard that framework before? So autonomy is you want to be able to [21:00] be your own boss or march to the beat of your own drum; mastery is you want to feel like you’re getting really good at something; and purpose is you want to feel like the work you’re doing actually matters to someone. Pretty much everything humans do is trying to get one of those things. Actually, making a lot of money is not one of them. You have to make enough money that it’s not a problem, but after that, more money isn’t nearly as motivating as these other factors.

You see that in Silicon Valley, right? You see a lot of people who have more money than they ever spend, but are working really, hard. Why is that? Why aren’t you just on a beach? But laying on a beach doesn’t give you a lot of mastery and purpose, right? Whereas getting back into building something new, or even back to the street musician example, why do people become starving artists and make no money, but play music or make art or whatever? It’s because it really does fulfill a lot of those drives, and they’re willing to do it despite the fact that it doesn’t really pay well. So if, you strive for something that actually lights your candle, gets you excited, makes you want to go push and do things to make some change in the world you want to see, I think you can start to optimize for that.

And, maybe one of the good sides [22:00] is maybe it’ll even become less important that it creates a lot of economic value. Because if we are successful at creating overall abundance and sharing it widely, then it probably won’t matter as much that what you’re doing personally is creating a lot of value. A lot of people who make a lot of money really hate their jobs, right? They really don’t love it, in fact, a lot of them have a “side hustle”, or they’re like a 12th level Orc on World Warcraft or whatever, because that’s the thing that actually fulfills them, not the work. So, as much as possible, try to find work you can do that you can throw yourself into for a long time. It’s just such a hedge against what will and won’t be the “hot job” at the time. Does that make sense?

To sum it up, if you take away one thing from this, it’s that I think you should all know that the future is very uncertain, but that there’s a lot of promise and if you can just try to, maybe it sounds cliche, but cultivate a sense of agency, have a growth mindset, remember [23:00] that there’s lots of things in the world that should and could be different, and if you can just be curious about them and then have the courage to learn things and try things and fail and iterate and just get out there and try to make a difference. That set of skills will always be valuable no matter what the set of “building blocks” that are out there are. Remember, everybody else is going to have to same building blocks as you. The reason why having ChatGPT do your homework is not a good idea is not just because this is the way school is set up. It’s because everybody else can press that same “yes” button that you could have, right? So you’re not differentiating yourself in any way if you’re just the like passive conduit. What you want to do is figure out how to use that technology to do something you couldn’t do before, or to do something better or faster, or to have or be more ambitious because you’re like, “I have no idea how to do this”, but I can learn it on YouTube and I can try it with this. That’s, to me, the positive way of thinking about it. That’s where you’re still providing that unique set of supply and demand skills.

So, be curious. Have a growth mindset. Care about other people, right? Don’t be a robot yourself. Humans care about [24:00] humans. The more you understand, not just technology (I do think it’s very important to stay up on the forefront of technology), but I also think it’s really important to understand what drives art and culture and fashion and empathy and all these things.

And schnall dich an, because it’s gonna be a wild ride, but I think it could potentially be a really great one. I hope that’s helpful. Thanks.

Leaders in Tech podcast appearance (part 2)

After my initial conversation on the Leaders in Tech podcast, the host asked me to come back and follow-up with more of my thoughts on AI and the future of work and what we humans will or won’t still want or need to do in the future. We discussed what we can learn from the study of the human brain, and in particular how the pattern-matching cerebral cortex is distinct from the goal-oriented “old brain”, the latter of which is still largely missing from the AI models we’re building. While a lot of knowledge work will undoubtedly be augmented if not replaced by AI over time, we reflected on how much of being an effective leader in tech (or in most professions) still comes down to innately human characteristics of passion, empathy, group coordination, and so on, as well as how we will continue to be driven by work that affords us autonomy, mastery, and purpose, even if it becomes disconnected from how we provide for our basic needs.

Throughout the interview, you will hear why I am still fundamentally optimistic about “team human” and our potential to thrive in a world of technological abundance, which AI can help us usher in (if we don’t mess things up in the meantime, of course!).

Leaders in Tech podcast appearance

I was recently a guest on the Leaders in Tech podcast. We covered a lot of ground, from my childhood and how I got interested in tech and AI, to many lessons I learned working in startups (first Plaxo, now Triller) and inside big companies (Google+, Google Photos, Google Assistant). In particular, the conversation focused on the advantages startups have when it comes to driving innovation, and why, despite their advantages in terms of resources and distribution, it’s hard to get the same results inside larger organizations. We finished with a discussion of how AI is likely to impact the careers of software engineers (my bet is it will remain more of an amplifier than a replacement for years to come).

I think this is one of the best summaries of my experience and thoughts on Silicon Valley and entrepreneurship that I’ve managed to capture. I hope you’ll find it useful and would love to hear your feedback!

Returning to my startup roots

After nearly 12 years at Google, the last 5 of which I’ve spent leading core conversational technology for Google Assistant, I’m excited to share that I’m joining TrillerNet as Chief Technology Officer. I will always love Google and remain super bullish on Assistant’s future, but here’s why I found this opportunity so enticing.

The internet and social networking were supposed to enable us to connect in meaningful ways with our friends and the artists, thinkers, and brands we care about. But it’s nearly 20 years since my last startup, Plaxo, helped usher in this “social web” phase, and it’s clear by now that this is not the future we were promised. Sure, the people you want to follow are now on Social, but we mostly receive undifferentiated, one-way broadcasts from them. They don’t really know who we are, what we’ve seen and done, or what we’re interested in, and what personalization does exist comes more from “surveillance capitalism” than real two-way connection.

It doesn’t need to be this way: the data we need to differentiate ourselves as consumers (what we read, watch, listen to, purchase; the places we visit, etc.) all leave “digital breadcrumbs” now (in addition to our social media comments/likes). Recent advances in AI (including Natural Language Processing, unsupervised clustering, large language models, and more) have given us the tools to understand all of that data well enough to enable a new level of two-way personal engagement at scale. But since this is all happening across multiple sites and services, a neutral arbiter is needed to tie it all together–a company that deeply understands the needs of both creators and consumers and can develop the technology to help them connect like never before.

Enter TrillerNet, the unlikely but oddly ideally-positioned rocket ship, built from a remarkable convergence of multiple startups spanning technology, entertainment, and the burgeoning creator economy. It combines (1) the “AI-driven conversational superpowers” of Amplify.ai (which originally caught me eye), (2) deep experience and credibility with the creator/influencer community from Triller and Proxima’s backing, and (3) the ability to repeatedly create marquee cultural moments with Verzuz and FITE TV. Bold new initiatives like CLIQZ hint at the massive potential at the intersection of those three core differentiators. And it’s still early days.

I’ve been excited about Amplify.ai’s technology and success in both the commercial and political arenas since their CMO (and my longtime friend and collaborator from Plaxo), John McCrea, joined them several years back. Earlier this year, after their acquisition by TrillerNet, Amplify.ai CEO Mahi de Silva became CEO of the overall TrillerNet conglomerate. He sensed the coming “1 + 1 = 11” opportunity to pair the strong creator relationships and cultural engine of Triller with the breakthrough conversational AI capabilities of Amplify.ai and recognized that, with my two decades in Silicon Valley focusing on social networking, identity, and data portability on the one hand and NLP and conversational AI on the other hand, I was the “unlikely but oddly ideally positioned” CTO to lead the company’s next phase of transformational growth.

It’s surely going to be a wild ride. I’m excited to get back to my startup roots and see how the world has changed and what I’ve learned from my time at Google that does and doesn’t carry over. I welcome any advice or support and will have lots more to say as things unfold!

Why I’m all in for Warren

Note: This is the first time I’ve written about politics on my personal blog. It’s also the first time in my political memory where a lot of smart people who live near me don’t already share most of my political views. Given the unprecedented stakes of the 2020 election, and before the voting starts, I feel compelled to make this pitch and hope you’ll see it as a good faith attempt to help us all reach a better outcome.

Warren 2020 logo 02

I feel strongly that Elizabeth Warren has the best chance of beating Donald Trump in 2020 and also the best chance of delivering non-incremental change to the health and well-being of America and its citizens once elected. She’s the candidate this moment in our history calls for, and we should do everything we can to answer that call.

If this doesn’t strike you as obviously true, yet you know me well enough to respect my intellect and opinions on other matters, please read on and get in touch with me to discuss / debate further until I can convince you or we can figure out in detail where our values or views of the world differ. 🙂

Warren is the most electable candidate

All savvy political observers believe that, to win in 2020, the democratic candidate has to excite and turn out the base in record numbers and hold/persuade enough moderate voters, esp. in battleground states. I worry Biden won’t be able to sufficiently excite the base (I’m having flashbacks to 2016). I worry that Bernie will turn off too many moderates (due to flaunting his “democratic socialist” brand, his “revolutionary” policy proposals, and, more importantly, his general lack of specifics on what his policies would look like in detail and how he’d get them done).

In contrast, I believe Elizabeth Warren can excite the base as much as Bernie without scaring off everyone else. After all, she’s a Harvard Professor who grew up as Betsy, the Republican girl from Oklahoma. She knows both worlds. She wants to restore Capitalism to its proper working function (better regulation and balance of power between capital and labor), not blow it up and replace it with a different system. She has detailed plans for what she intends to do, how she intends to pay for it, and how she intends to get it passed and enacted. There’s no uncertainty about what you’d get with a Warren presidency. And despite being a wonk’s wonk, she’s incredibly relatable and charismatic in person.

Beyond policy and character, she’s shown she can win elections against Republican incumbents, even when starting far behind in the polls. She’s running on a message of anti-corruption against the most corrupt President in history. Everyone (including her opponents) agrees she’s running the best political organization in the field. Put it all together, and there’s no one I’d rather take my chances on competing against Trump in the fall.

There’s (justifiably) so much fear about Trump getting re-elected that many people are feeling risk averse and clinging to what looks like the safest choice (usually: an old white man who won’t rock the boat too much). I believe this is a fundamental misreading of the dynamics at play this year. We should be playing to win, not playing not-to-lose, and that means backing the candidate that we truly believe in. Warren is that candidate for me, hands down, and if you haven’t taken a good look at her, please be open to the fact that the same may well happen to you once you do.

Warren will be the most effective President

As important as beating Trump will be, it’s only the start of the job of fixing what’s broken in America. I’ve spent hundreds of hours over the past many years reading and discussing and trying to understand the root causes of our political and economic dysfunction. I believe Warren has the best understanding of what’s really going on and how to actually begin to fix it. Political capture by moneyed interests and out of control wealth and income inequality have been growing problems for my entire lifetime, yet Warren is the only candidate focused on tackling them head on and with plans bold enough to meet the scale of the challenge. As she points out, you can’t win on any other policy areas we care about (climate change, expanded safety net, etc.) until you can solve the root problem that’s been preventing progress on them (“corruption” writ large).

She’s also the savviest when it comes to how to make real change in Washington. Her ability to create the Consumer Financial Protection Bureau is justifiably legendary, and she’s the most clear-eyed about what she can accomplish through the Executive Branch and how she can get the rest through Congress even with a narrow majority (e.g. eliminating the filibuster and passing what she can via budget reconciliation), something Biden and Bernie still refuse to touch, thus essentially neutering their policy agenda. She understands that “personnel is policy” and has been spending considerable time identifying the right cabinet and other positions to best achieve her goals. In short, she’s got a plan for her plans.

Finally, if she can start enacting her plans, I believe real Americans will, for the first time in a while, really feel a noticeable positive impact on their lives (access to childcare and education, student loan forgiveness, etc.). This in turn, I believe, will strengthen her mandate to do more, since it will shift people’s perspective on politics from essentially just a “tribal team sport” to something where the outcomes actually matter to them. This is important not just for momentum but also to counter the degradation of our political institutions, which is possible today partly because people don’t feel sufficiently personally invested in protecting and strengthening them. But once the government starts actually delivering in noticeable, critical new ways, which I believe Warren is best positioned to accomplish, I think we have a chance to finally turn the tide.

Don’t just take my word for it:

PS: I wanted to keep this post short, but there’s so much more I could say or share, so again please either help spread the good word or let’s talk more if you’re at all open to the fact that I might not be crazy here. 🙂 This election is too important to just watch from the sidelines. We all need to sacrifice and get involved to the best of our abilities. I’ve started donating and volunteering and speaking publicly. This is new and uncomfortable for me, but I hope by sharing this, it will help some of you feel empowered to take your own next steps.

Starting a new conversation…

Google AssistantI’m excited to share that I’ve recently joined the Google Assistant team! Like a lot of people (including our CEO), I’ve become convinced that natural language conversation represents the future of how we’ll primarily interact with computers and the myriad smart devices that will soon proliferate around us. This new “UI” will be personalized based on both knowledge of you and your history of interactions. It will also be proactive (reaching out to you with pertinent questions and updates) as well as responsive. And it will execute tasks across multiple, interconnected services on your behalf.

Which is to say: it’ll be a lot different than how we work with computers today. And it promises to be a lot better, too — if we can get it right.

I’ve been fascinated by interacting with my Google Home (which I’ve had early access to for a while). It highlights both the challenges and opportunities of this new conversational modality, and it surprises in equal measure with how far we’ve come and how far we still have to go. For instance, my 5 year old daughter walked into our living room the other day and proclaimed, “Ok Google, play some Lady Gaga”, then started dancing to the music that immediately began playing. Think about that: She would never have been able to accomplish that task with a traditional desktop/mobile app, nor would I have been able to help her as quickly as she was able to help herself. She didn’t have to be unnaturally precise (e.g. select a particular track or album), and it was an enormously empowering interaction with technology overall. It feels like “how things should work”.

I’ve had countless similar “wow moments” asking Google questions about the world, my own upcoming flights and schedule, or streamlining tasks like playing music or showing some recent photos I took on our TV to the grandparents. But for all the magic Google can deliver already, this is still very clearly early days. The dialogs are often too fragile and require too much custom crafting by too many Google engineers in a way that clearly won’t scale to the ambitions of the medium (and the team). There’s not yet much deep learning or true language understanding under the hood. And only recently has there even been a focused effort to build The Assistant, instead of just “voice enabling” individual products here and there. The industry as a whole is still only starting to figure out how “chatbots” and “conversational commerce” and so on should work.

Which is why it seems like an ideal time to get involved–we can tell “there’s a there there”, but we also still need many foundational innovations to realize that potential.

On a personal level, this change also represents an opportunity to get my career “back on track” after a wonderful decade+ diversion into the emerging world of social networking. I actually came to Stanford in the late nineties to study Natural Language Processing with the aim of empowering ordinary users with the superpower of Artificial Intelligence, and even published several papers and helped build Stanford’s first Java NLP libraries while earning my BS and MS. I originally joined Plaxo, an early pioneer of social networking, to build an NLP engine that automatically parsed contact info out of emails (e.g. “Oh, that’s my old address. I’ve since moved to 123 Fake St.”), but eight years later, I was still there, serving by then as its CTO and trying to open up the social web that had sprung up around us as the new way to stay connected. That in turn led me to join Google in 2010 as part of the founding team that created Google+, and I’ve been at Google ever since. But now I’ll actually be working on NLP again, and I have a feeling my years advocating for user-centric identity and data portability across services will come in handy too!

I’m uncomfortably excited to be starting this new chapter of my career. If you think about all the exhilarating potential surrounding AI these days, realize that a surface like Google Assistant is where you’re most likely to see it show up. One of the senior-most engineers on the team remarked to me that, “I’m sure a full Turing Test will be part of our standard testing procedure before long,” and I think he was only half joking. If you’re building similar things in this space, or have ideas about what we should prioritize building in the Google assistant or how you’d like it to integrate with your service, please let me know. I’m ready to learn!

When Dreams Become Real

So much has happened since I first wrote about my “side project” to help Dan Ambrosi apply DeepDream to his multi-hundred megapixel HDR landscapes. Here’s a rapid-fire rundown. Be sure to click on the photos below to see a series of (smart!) albums from each event.

Close encounter

The first chance to view our creations at near “life size” came on March 31st from Calit2 at UC San Diego where Dan was invited to present his work on their 66-million pixel wide room-sized VROOM tiled wall display. Calit2 (which my father founded in 2000) has a long history of collaboration between science, technology, and the digital arts, so everyone felt right at home, and it provided us with the first visceral validation that seeing this work displayed with sufficiently high scale and resolution was indeed a transformative experience. Here’s a video I shot of Dan controlling the wall, and here’s Calit2’s news release from the visit. It left us energized and determined to see these works printed at scale in the real world. Turns out we wouldn’t have to wait long…

The great jailbreak

The very next day, Dan installed a series of large format printed Dreamscapes at the massive GPU Technology Conference, which is held annually by NVIDIA in the San Jose Convention Center. We’d shown NVIDIA our work earlier, and they loved how it showcased the power of what you could compute with GPUs and CUDA, so they agreed to purchase three 8 feet high x 16 feet wide (!) Dreamscapes (printed on tension-mounted fabric backlit by LEDs inside a free-standing aluminum frame) for the main conference hallway. Dan, Chris, and I also gave a well-attended talk (video) on the art and tech behind the Dreamscapes, and Dan gave a couple additional interviews. The pieces were a huge hit, and it was so fun to see everyone enjoying them, taking selfies in front of them, standing back to take in the whole picture and then walking right up close to see the surprising details, and grabbing their friends and colleagues with “you gotta see this!” We’re so grateful to NVIDIA for their support, which allowed us to finally unshackle these works from their digital prison and experience them with the full freedom and resolution that only reality provides (for now).

Bringing the tour back home

Ever since I started working on Dreamscapes, I’d been sharing my progress on Google’s internal G+ and asking for feedback. After Dan published his work online and photos started flowing in from Calit2 and GTC, demand grew to present a tech talk and exhibit Dan’s work inside the ‘plex. NVIDIA generously agreed to loan us their Dreamscapes from GTC for a week, so on May 20th we set them up in one of Google’s largest auditoriums, and the following Monday Dan delivered an hour-long tech talk (video) followed by a reception. This was the first time I’d heard Dan go deep on the art history and iteration of technique that drove him to be able to produce these compelling giant landscapes, so I learned a lot and it sparked a lot of discussion among Googlers. The pieces remained up that whole week, and as word spread, there was a constant flow of people sneaking in to check them out and share in the unique experience.

Entering the fine art world

When it rains, it pours. During the time we were showing our work at NVIDIA and Google, Dan was approached by several fine art galleries about exhibiting his work, and he ended up creating installations in contemporary spaces in both Miami, FL and Steamboat Springs, CO (more photos). He gave talks at both places, and the enthusiasm from fellow artists and the community was enormously validating. He’ll also be showcasing his work at the upcoming 9e2 art + science event in Seattle this October.

A parting gift

After all of this excitement, I couldn’t help wishing I could take home a physical memento of our adventure. Obviously 8′ x 16′ is way too large to display inside a normal home, but Dan also produces his works in 4′ x 8′ wall-mounted light boxes printed using Duratrans, which is even crisper than fabric (though it can’t be printed as large). My favorite scene of his is the Point Montara Lighthouse, shot just a few minutes away from my house, which became something of a signature image for us. I celebrated my 35th birthday this year, and my parents decided to commemorate the occasion by purchasing that piece for me. I couldn’t imagine a better present, and not a day goes by that I don’t pass by and stare at it with a big smile on my face. 🙂

Onward

As you can see, it’s been a busy time in Dreamscape land. Yet remarkably, Dan has simultaneously undertaken an end-to-end upgrade of his workflow, from the camera (Sony RX1 → Sony RX1R II, which with its 42.4 megapixel full frame 35mm sensor nearly doubles the effective resolution he can capture, enabling him to shoot scenes that call for a narrower panoramic sweep) to the software (Photomatix → Aurora HDR, which provides better color balance, especially with blue skies; PTGui → Autopano Pro, which stitches more accurately resulting in fewer artifacts; and even upgrading to the latest version of Photoshop CC, which can finally handle images larger than 30,000 pixels on a side). He’s currently in the process of “re-mastering” many of his previously captured images with this new suite of tools, as well as shooting new scenes and exploring new DeepDream settings to run on them.

Oh, and he’s also started collaborating with an additional AI / deep-learning image-oriented software team on what may turn into “Dreamscapes 2nd Edition” soon. I can’t wait to see what dreams may come…

Home Sous Vide: What the Books Don’t Tell You

After years of passive enchantment with sous-vide cooking, my wife bought me an Anova immersion circulator for my birthday so I could finally try it at home. Despite reading the book everyone recommends, I was still left with a bunch of newbie questions that didn’t have immediately obvious answers. I forged ahead with a mix of phoning a friend, asking the Internet, and just giving it my best shot, reciting the mantra that worst case the food will just be inedible. In the end, things worked out fine–though there’s much more experimenting to be done!

For the benefit of other first-timers, here’s some collected Q&A based on my experiences so far. I hope this will help remove a roadblock or two and help convince you to “take the plunge” (see what I did there?).

Q: How should I prepare the water bath? How warm should it be?

Turns out it doesn’t really matter, since the circulator can heat it up pretty quickly on its own. I just filled the pot with warm water from the sink, which turned out to be about 110°F. In my case, I was cooking at 142° and the circulator brought it up within several minutes. No need to heat the water on the stove or use a kettle first (remember, you’re almost always cooking at temperatures well below boiling).

Q: How high should I fill the pot with water?

Technically, it just needs to comfortably cover the food you’re cooking, but it’s nice to give it some buffer. The Anova has a min and max line and you need the water to be in between. One thing I hadn’t thought of is how much water can evaporate over a long cooking time (more on this below), so that’s another reason not to cut it too close. The circulator moves the water pretty gently, so there’s no danger of it sloshing over the top of the pot or bubbling over like with boiling water.

Q: If my meat is frozen, do I need to fully thaw it before starting the sous vide process?

Not if you’re doing a longer cook. For instance, I did 24-hour beef short ribs at 142°, and even though the cubes of meat were 2-3″ thick, there’s plenty of heat and time to fully thaw the meat and bring it fully up to temperature. That’s one of the beautiful things about sous vide–since it’s low and slow, you don’t have the same worries about unevenly heating the meat like you would on a grill. And most meat defrosting techniques involve placing it in warm water, which is exactly what you’re doing. I just pre-defrosted the meat in warm water for ~20 mins and then put it in the bags and had no trouble. This is handy since otherwise it can take a long time to defrost meat, which means you need to plan days ahead to cook it.

Q: Many books/sites say you can use Ziploc freezer bags to seal the food. What about Ziploc storage bags?

I had a minor panic attack when I realized that all of our gallon-sized Ziploc bags at home were “storage” not “freezer”. I obviously didn’t want to risk melting the plastic or rupturing the bag during cooking. But according to the Internet, both are plenty well qualified for the task, being made of heat resistant food-grade polyethylene. If the food isn’t sharp and pointy, you’re not going to poke a hole in the bag during cooking, and the water is not going to be anywhere near the melting point of the plastic. In the end, I decided to brave the storage bag and it seemed to work just fine. I’d probably still prefer a freezer bag if I had one handy (just to be paranoid), but I don’t think it’s required. Just remember in either case to get all the air out of the bag using the water displacement method, which you do right in the sous vide pot.

Q: How do you keep the bag from getting sucked into the circulator during cooking?

When I first started the cooking process, I just naively put my ziploc bag into the pot with the circulator clamped onto the side. Nothing in the book or the instruction manual said to do otherwise! But inevitably over time the circulator would suck the bag towards it, ultimately impeding the water circulation and making some weird noises. Definitely didn’t seem like it was working as intended. Turns out most people clip the bag to the side of the pot with a chip clip or clothespin. That fixed the problem. I have no idea why this isn’t specifically mentioned in any of the basic instructional material!

Before

After

Q: Why does the circulator sometimes make a metallic grinding noise?

The instruction booklet said it means the case isn’t tight enough, but I couldn’t obviously tighten it or loosen it. It definitely sounded wrong though, so I tried a few more times and eventually got the right amount of twisting and pulling to remove the case and then reattach it with a satisfying snap at the end, after which point the sound went away. It didn’t seem to affect the cooking process to have that noise, but it was definitely somewhat worrying and annoying, so I’m glad I finally fixed it.

Q: If you set the cooking timer, will it turn the circulator off when the timer finishes?

No. The Anova timer is just FYI and does not control anything. It’s confusing that they don’t tell you that, but it means you don’t need to worry about the exact cook time, which is part of the point of sous vide (it’s hard to overdo it, since the water is the same temperature you want the food to end up at, vs. an oven or grill, which is much hotter). For instance, I’d planned for my short ribs to be done around the time I would get home from work, but my schedule is not always super predictable, and I didn’t want the circulator to shut off and let the water cool down if I came home later than expected. Fear not, it’ll just keep going until you stop it.

Q: Is it safe to cover the entire top of the pot with cling wrap to reduce water loss from evaporation?

A recipe I read suggested that I do this, but I wasn’t sure if that would cause lack of airflow or pressure to build up or something. Fear not, it didn’t seem to cause any problems, and it significantly reduced water loss from evaporation, which is especially important for longer cooking times. I let it cook over night uncovered and it lost an inch or two of water in the process, so if I’d let it continue all day, it might well have gotten down below the min line by dinner time. The only hard part was keeping the cling wrap from sagging down into the water (don’t want to cook it!), but by using some longer pieces, pulling them tight, and wrapping the extra around the pot handles and edge, leaving room for the circulator on one side and the clip on the other side, it was stable and effective. Definitely recommended.

Q: What happens if the power goes out in the middle of cooking, esp. while you’re away at work?

You’re in trouble. 🙂 This happened to me, and even though it was only a brief outage, the Anova frustratingly does not restart itself after waking back up! So the water just slowly cools, leaving your meat under-cooked and potentially in the danger zone for bacterial growth. This is what I came home to, and at first I thought the meal was ruined. But after some research and talking to some scientists in the family, I felt a bit better. It usually takes at least four hours in the danger zone for bacteria to grow, and the meat had been cooking above that zone for many hours already, so most bacteria should have been killed by then. When the power went out, the water didn’t immediately cool, so it only spent some of the post-outage time in the danger zone. And most preparations call for searing the outside of the meat before serving, which re-kills anything that might have started growing on the surface. In the end, after much soul searching, I decided to crank up the circulator for a bit and then pull out the meat and sear it. The meat was not as tender as I was hoping (possibly because the cooking time had been cut short) but it was definitely edible and, as I write this the next morning, I have no indications that my stomach is any worse for wear. The advice from the Anova folks seems to be to buy a UPS battery backup and plug the circulator into that.

My first home sous vide: 24-hour beef short ribs

Q: Can you reuse the cooking water for a subsequent sous vide run?

The consensus on the Internet seems to be yes, provided the water is not obviously funky or contaminated. It’s possible some bugs could grow in it between cooking cycles, but you’re keeping the food sealed in plastic and turning the water back up to a temperature that will kill anything in there, so you should be fine. Hey, it’s drought-tolerant cooking! After my scare with the power outage, I wanted to “get back on the horse” and do another run for the next dinner. I could have poured out all the water and started over, but it seemed wasteful, provided I wasn’t risking my health. I’ll post an update if this turns out to be a mistake, but I don’t think it will. 🙂

Conclusion

The promise of sous vide is “set it and forget it”, which as an engineer I love. It’s precision cooking that’s not fussy on technique. And it can produce some pretty stunning, unique results. But the danger is that by cooking food at lower heat for longer time, you’re at greater risk of growing bacteria if you’re not careful. So most of the first-timer questions above ultimately boiled down to how to minimize that risk without giving up entirely. That plus stuff that I’m sure you get used to after you’ve been doing it for a while, but that’s not obvious the first go around, and that the books don’t tell you about one way or the other.  So I hope this addendum finds and helps someone. In the meantime, I’ll keep at it. And if you’re a home sous vide cook and have any tips to share, or if you found any of this useful (or confusing, or provocative), please let me know!

Dreaming big

dreamscape-detail

I’ve recently been working on a side project with my good friend Chris Lamb to scale up Google’s Deep Dream neural net visualization code to operate on giant (multi-hundred megapixel) images without crashing or taking an eternity. We recently got it working, and our digital artist friend (and fellow Plaxo alum!) Dan Ambrosi has since created some stunning work that’s honestly exceeded all of our expectations going in. I thought it would be useful to summarize why we did this and how we managed to get it working.

Even if you don’t care about the technical bits, I hope you’ll enjoy the fruits of our labor. 🙂

The ”˜danorama’ back story

gridDan’s been experimenting for the past several years with computational techniques to create giant 2D-stitched HDR panoramas that, in his words, “better convey the feeling of a place and the way we really see it.” He collects a cubic array of high-resolution photos (multiple views wide by multiple views high by several exposures deep). He then uses three different software packages to produce a single seamless monster image (typically 15-25k pixels wide): Photomatix to blend the multiple exposures, PTGui to stitch together the individual views, and Photoshop to crop and sweeten the final image. The results are (IMO) quite compelling, especially when viewed backlit and “life size” at scales of 8’ wide and beyond (as you can do e.g. in the lobby of the Coastside Adult Day Health Center in Half Moon Bay, CA).

“I’d like to pick your brain about a little something”¦”

Dan AmbrosiWhen Google first released its deep dream software and corresponding sample images, everyone went crazy. Mostly, the articles focused on how trippy (and often disturbing) the images it produced were, but Dan saw an opportunity to use it as a fourth tool in his existing computational pipeline–one that could potentially create captivating impressionistic details when viewed up close without distorting the overall gestalt of the landscape when viewed at a distance. After trying and failing to use the code (or any of the DIY sites set up to run the code on uploaded images) on his giant panoramas (each image usually around 250MB), he pinged me to ask if I might be able to get it working.

I had no particular familiarity with this code or scaling up graphics code in general, but it sounded like an interesting challenge, and when I asked around inside Google, people on the brain team suggested that, in theory, it should be possible. I asked Chris if he was interested in tackling this challenge with me (both because we’d been looking for a side project to hack on together and because of his expertise in CUDA, which the open source code could take advantage of to run the neural nets on NVIDIA GPUs), and we decided to give it a shot. We picked AWS EC2 as the target platform since it was an easy and inexpensive way to get a linux box with GPUs (sadly, no such instance types are currently offered by Google Compute Engine) that we could hand off to Dan if/when we got it working. Dan provided us with a sample giant panorama image, and off we set.

“We’re gonna need a bigger boat…”

18-KolobCanyon_megoHDR60-1600Sure enough, while we could successfully dream on small images, as soon as we tried anything big, lots of bad things started happening. First, the image was too large to fit in the GPU’s directly attached memory, so it crashed. The neural nets are also trained to work on fixed-size 224×224 pixel images, so they had to downscale the images to fit, resulting in loss of detail. The solution to both problems (as suggested to me by the deep dream authors) was to iteratively select small sub-tiles of the image and dream on them separately before merging them back into the final image. By randomly picking the tile offsets each time and iterating for long enough, the whole image gets affected without obvious seams, yet each individual dreaming run is manageable.

Once we got that working, we thought we were home free, but we still couldn’t use the full size panoramas. The GPUs were fine now, but the computer itself would run out of RAM and crash. We thought this was odd since, as mentioned above, even the largest images were only around 250MB. But of course that’s compressed JPEG, and the standard Python Imaging Library (PIL) that’s used in this code first inflates the image into an uncompressed 2D array where each pixel is represented by 3×32 bits (one per color channel), so that the same image ends up needing 3.5GB (!) of RAM to represent. And then that giant image is copied several more times by the internal code, meaning even our beefiest instances were getting exhausted.

So we set about carefully profiling the memory usage of the code (and the libraries it uses like NumPy) and looking for opportunities to avoid any copying. We found the memory_profiler module especially helpful, as you can annotate any suspicious methods with @profile and then run python -m memory_profiler your_code.py to get a line-by-line dump of incremental memory allocation. We found lots of places where a bit of rejiggering could save a copy here or there, and eventually got it manageable enough to run reliably on EC2’s g2.8xlarge instances. There’s still more work we could do here (e.g. rewriting numpy.roll to operate in-place instead of copying), but we were satisfied that we could now get the large images to finish dreaming without crashing.

BTW, in case you had any doubts, running this code on NVIDIA GPUs is literally about 10x faster than CPU-only. You have to make sure caffe is compiled to take advantage of GPUs and tell it explicitly to use one during execution, but trust me, it’s well worth it.

Solving the “last mile” problem

new-dreamWith our proof-of-concept in hand, our final task was to package up this code in such a way that Dan could use it on his own. There are lots of tweakable parameters in the deep dream code (including which layer of the deep neural net you use to dream, how many iterations you run, how much you scale the image up and down in the process, and so on), and we knew Dan would have to experiment for a while to figure out what achieved the desired artistic effect. We started by building a simple django web UI to upload images, select one for dreaming, set the parameters, and download the result. The Material Design Lite library made it easy to produce reasonably polished looking UI without spending much time on it. But given how long the full images took to produce (often 8-10 hours, executing a total of 70-90 quadrillion (!) floating point operations in the process), we knew we’d like to include a progress bar and enable Dan to kick off multiple jobs in parallel.

Chris took the lead here and set up celery to queue up dispatching and controlling asynchronous dreaming jobs routed to different GPUs. He also figured out how to multiply together all the various sub-steps of the algorithm to give an overall percentage complete. Once we started up the instance and the various servers, Dan could control the entire process on his own. We weren’t sure how robust it would be, but we handed it off to him and hoped for the best.

“You guys can’t believe the progress i’m making”

dreamingOnce we handed off the running EC2 instance to Dan, we didn’t hear much for a while. But it turned out that was because he was literally spending all day and night playing with the tools and honing his process. He started on a Wednesday night, and by that Saturday night he messaged us to say, “You guys can’t believe the progress I’m making. I can hardly believe it myself. Everything is working beautifully. If things continue the way they are, by Monday morning I’m going to completely amaze you.” Given that we’d barely gotten the system working at all, and that we still really didn’t know whether it could produce truly interesting output or not, this was certainly a pleasant surprise. When we probed a bit further, we could feel how excited and energized he was (his exact words were, “I’m busting, Jerry, I’m busting!”). It was certainly gratifying given the many late nights we’d spent getting to this point. But we still didn’t really know what to expect.

The following Monday, Dan unveiled a brand new gallery featuring a baker’s dozen of his biggest panoramic landscapes redone with our tool using a full range of parameter settings varying from abstract/impressionistic to literal/animalistic. He fittingly titled the collection “Dreamscapes”. For each image, he shows a zoomed-out version of the whole landscape that, at first glance, appears totally normal (keep in mind the actual images are 10-15x larger in each dimension!). But then he shows a series of detail shots that look totally surreal. His idea is that these should be hung like giant paintings 8-16’ on a side. As you walk up to the image, you start noticing the surprising detail, much as you might examine the paint, brush strokes, and fine details on a giant painting. It’s still hard for me to believe that the details can be so wild and yet so invisible at even a modest distance. But as Dan says in his intro to the gallery, “we are all actively participating in a shared waking dream. Science shows us that our limited senses perceive a tiny fraction of the phenomena that comprise our world.” Indeed!

From dream to reality

galleryWhile Dan is still constantly experimenting and tweaking his approach, the next obvious step is to print several of these works at full size to experience their true scale and detail. Since posting his gallery, he’s received interest from companies, conferences, art galleries, and individuals, so I’m hopeful we’ll soon be able to see our work “unleashed” in the physical world. With all the current excitement and anxiety around AI and what it means for society, his work seems to be striking a chord.

Of course the confluence of art and science has always played an important role in helping us come to terms with the world and how we’re affecting it. When I started trying to hack on this project in my (copious!) spare time, I didn’t realize what lay ahead. But I find myself feeling an unusual sense of excitement and gratitude at having helped empower an artistic voice to be part of that conversation. So I guess my take-away here is to encourage you to (1) not be afraid to try and change or evolve open source software to meet a new need it wasn’t originally designed for, and (2) don’t underestimate the value in supporting creative as well as functional advances.

Dream on!

« Older posts

© 2025 Joseph Smarr

Theme by Anders NorenUp ↑