In today’s podcast episode, Ben, Matt, and Cassidy talk with Utkarsh Srivastava, VP of Engineering at Patreon. We cover the product roadmap, deployment best practices, and UX philosophies. Matt and Cassidy, both Patreon users, confirm its delightful user experience.
Srivastava reflects on his upbringing in India, learning to write Assembly, and going to Stanford University to complete his Ph.D in computer science.
He shares his early career experiences at big tech names like Yahoo!, Google, Twitter, and Google.
The group reflects on some of the engineering challenges at Patreon including technical debt, migrations to open source services, and troubleshooting bugs.
Srivastava walks us all through upcoming product features that his engineering team is working to implement.
Andy wins a Lifeboat Badge for answering this question about a list of all tags on Stack Overflow.
Follow Ben, Matt, Cassidy, and Utkarsh.
[intro music plays]
Ben Popper Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I am Ben Popper, the Director of Content here at Stack Overflow, joined as I often am by my wonderful colleague and collaborator, Matt Kiernander.
Matt Kiernander Hello!
BP I'm joined not only by my wonderful colleague and collaborator, Matt, but also by my wonderful colleague and collaborator, Cassidy Williams. Hi, Cassidy.
Cassidy Williams Hello!
BP We're going to be chatting today with Utkarsh, who's heading up engineering at Patreon, a company near and dear to content creators' hearts. So I feel like a lot of us have used it. I give a dollar a month and I get my little receipt and it makes me feel like I've done some good for the artists of this world. Utkarsh, please, welcome to the show.
Utkarsh Srivastava Thanks, Ben. I'm really excited to be here today and thanks for having me. I'm looking to talk all things technology.
BP Yeah. So we always ask people to start out, just tell us a little bit about how you touched your first line of code. And I know you've worked at some big name Silicon Valley companies, so walk us through a little bit of your journey and how you landed where you are today.
US Yeah, sure. So I grew up in India. The computer access there started a little bit later than out here in the west. First brush was, as with many other folks, computer games and Prince of Persia 2D version, and figuring out the cheat codes for those and wondering how those actually worked. And then getting into some Assembly coding to sort of bypass some of the most critical parts where it was really hard to fight the Vizier. So that was probably my first brush with programming and I kept getting more and more fascinated with how powerful computers really were, and how with a few lines of code you could achieve what took otherwise a tremendous amount of labor. And just that huge enablement angle of software is something that's always excited me.
BP Very cool. And so I know you made the jump to Silicon Valley. Can you tell us a little bit about that?
US So I went to school at Stanford which was right in the heart of Silicon Valley, working on databases, getting my PhD, working on query optimization. So that was the time when big data was really starting to become a thing, and so I ended up going to some of the big tech names. At that time it was Yahoo Research, working on things like Hadoop that really revolutionized distributed data processing. Because until then, all the databases had been really centralized. You put all your data on one machine and you query it. That was the time when data was really breaking out, and you would put it on these large clusters of thousands of machines and then try to query them all in parallel. So I had a really fun time there building systems for data query analysis and some of that stuff –the system that we built– was called Big and it ended up being used pretty much across the valley at companies like LinkedIn, Netflix and at Twitter. And that was how I got connected with some of the Twitter folks and the open source community there. And then ended up going there and wanting to work on something completely different. And I remember chatting with Dick Costolo who was the CEO at that time, and he was like, “Hey, what do you want to work on?” I was like, “I want to work on something greenfield.” He was like, “Yeah, we haven't really figured out how to make money yet. You want to go figure that out for us?” I was like, “Okay, I guess we're doing ads, right?” So when we started, I was the first engineer on the ads team. And that was a really fun ride, scaling the business from zero to a billion dollars in three and a half years or so. Lots of distributed systems problems, click prediction, machine learning. I think we were the first ones to pioneer native ads in the feed. I think we did it even before Meta did it. Of course the rest is history with that. Then I ended up going to Google working on Android communications products. I was really fascinated with how even though social media was obviously a very heavy consumer of people's times, but we were still seeing a lot of trends around messaging and people wanting to message in smaller groups. And I went to Google to help really establish Google's efforts around this, and we ended up working on the Messages app, which is the default text messaging app on Android, which now has over 500 million users and we are trying to work with Apple to make sure that we can get rich messaging across iPhones and Android. And then my latest gig here at Patreon started just a year ago. I was really fascinated with how Patreon has a chance to just change how content is produced and consumed on the internet. I helped create the ads business in this space, but then the whole direct payment from subscribers to the creators they follow is a very, very appealing model. And I wanted to help create that as well, to serve as a good sort of counterbalance.
BP Interesting. You helped to build the business at Twitter, and now you're thinking about moving from advertising to subscribers, and that is all I will say on the matter. Apropo of nothing. Very interesting, very interesting.
US Thank you.
CW Yeah, that's awesome though– the different approaches to everything that you've taken. They're very different problems but I'm sure there's some interesting overlaps in the actual technical side of things as you figure things out over time.
US Definitely a lot of technical problems repeat across these things just in different contexts. And I think one of the things that I enjoy is taking the overall business context and distilling it down to the engineering problems, because that business context can help inform the approach you take to some of these problems and what you prioritize first and how you go about doing things.
MK So what kind of technical challenges have you kind of encountered over your last year and a bit at Patreon? Just because I'm familiar with how they work and the features and stuff they provide, but in terms of the actual technical challenges that Patreon is facing, can you speak a little bit about that?
US So Patreon has been around for eight to nine years now, and with that amount of tenure comes a fair amount of technical debt. And specifically when Patreon was getting started, a lot of the technologies that we take for granted now were still in their infancy. Something like GraphQL that has now become really popular and is the de facto way you would build an API wasn't really around eight to nine years ago, and that has led to us being in this sort of batch of companies that got started with some tech and there's now far better things available in the open source. And this applies not just at our API layer, but pretty much all layers of our stack, going from all the way to compute even. At the bottommost layer we were using EC2, but at this point most folks go with containers because of the additional flexibility and the quick startup times and all those benefits that containers offer. So one of our challenges has been that we have all these technologies that we would want to migrate off of and move to more open source better alternatives than in-house ones. How do we do that while still maintaining product velocity and continuing to ship value to creators? And just kind of balancing those two considerations has been an interesting challenge, and we have an amazing engineering team that's able to help us navigate how we can incrementally do many of these migrations so that we are not pausing the world and waiting for these migrations to happen before we ship any additional value. So over the last year we've been able to ship really meaningful improvements to creators. Just yesterday we launched Patreon video, which is the ability for creators to host video natively on Patreon. Really excited about that one. And we were able to do that while in the background also meaningfully improving our code base and moving off of old libraries and technologies onto new ones. So I’m really proud of how the team has executed on those two competing sort of objectives.
CW I use Patreon both as a creator and as a Patreon backer of people, and it is pretty impressive how smooth everything has been. I know that changes happen under the hood, just because I've talked with people who've worked there before, but as just a user, you wouldn't know that massive changes are happening and that this part has been completely rewritten or anything. It's been really smooth overall, so big kudos on that. I appreciate it.
MK I can testify to that as well. I think Patreon, even though I'm not I guess a power user of it, there's never once been where I've been like, “This is really frustrating. I wish it did this.” It all just seemed to work, which I think is quite rare.
US Yeah, thank you. You're all too kind. We do have issues like everyone else does, so we have a big bug backlog that we've went through. But yeah, we’re feeling really good about the rigor we are able to follow in these processes. One of the things we do often is what we call feature flags, so any big change or a migration is put behind a feature flag so that we can keep testing it incrementally, either with our internal engineers and support folks and in some cases even a small fraction of external users, and then watch the metrics and make sure that nothing is really degrading before we ramp up that feature flag further. So you start at 1%, 20% and so on. And often these changes are invisible to the user, so they can't tell the difference, but internally we are making a dramatic upgrade to our infrastructure. So that way of rolling things out has helped us really prevent any user-facing issues.
BP Interesting.
CW For the feature flags parts, did you roll your own and build your own system for that? I know a lot of companies do that. Or do you use a service for it?
US Yeah, so this goes into that same bucket of how when we started doing this there weren't that many services around. Now there are, but at this point we have the service built for internally managing feature flags. It's not the most full-featured, but it's sufficient for our purposes right now. But as we expand and have more and more complicated use cases we can definitely look into using some external service for it. But for now, we just have a library to do it ourselves.
CW Yeah, I think that ‘build or buy’ conversation is tough, especially when things didn't exist. You didn't have a choice. But now it's much more of a debate, like– this will save us time and effort but cost money, and figuring out that balance there.
US Yeah.
MK Just having a chat to a few people who've worked at larger companies like Microsoft and Meta and I guess Stack Overflow is one of them, as well as Patreon. They're older products, and as you mentioned, they do have a lot of legacy behind them. Technology has changed, there are a lot more convenient and easily accessible open source solutions available now, and that comes with challenges around onboarding engineers and retaining talent to a degree, and making sure that people can be productive early on in the process, because a lot of these tools and everything else, they're very specific to the company that has built them. So how do you kind of manage that, whether that's the onboarding process, whether that's facilitating growth within the engineers at Patreon? Can you talk to that at all?
US Yeah, that's a great point, Matt. And ultimately, I'm a huge believer in going with the more off the shelf open source option whenever possible because of all the advantages that you kind of alluded to. When people come on board, the last thing they want to do is learn some proprietary software that will not be applicable at the next job and doesn't really help them grow as an engineer. So generally we are committed to moving in the direction of being more and more based on open source off the shelf tools. But to the extent we have our own proprietary tools right now, we have invested in documentation around those so that we can quickly get new developers started and onboarded along with the understanding that this is not the way we want to do things going forward, so that every engineer onboarding to our team has this at the back of their mind that ultimately we plan to move off this technology. And as they are onboarding, as they're getting more familiar with that proprietary piece of technology, they can also help contribute to what that migration roadmap should look like.
BP Yeah, I'm not going to name names, but Cassidy told a story once about building an app for a streaming TV service. They went and rolled their own language and made it difficult to maintain after the person who learned that language left. So open source is the way to go.
CW Yeah. Not going to name names there.
MK Everyone's got those stories. I feel like it's always just the one engineer who has a brilliant idea and convinces senior leadership to do something and makes a huge impact and they kind of peace out and then everyone else is like, “Uhh.”
CW You're like, “Oh no. What do we do?”
US There is some value to that though, so it's not all bad. I think if you look back at the history of many of the frameworks, that's how they started because one engineer had a great idea.
CW I was going to say, tech history is riddled with that.
US Yeah. You can't completely shut off that line of innovation. My favorite story on that is they used to be a designer sitting right opposite me at Twitter and we were building these ads tools and we were writing the same CSS over and over. I was like, “I'm just going to build a toolkit for you guys so that you don't have to make these really ugly looking web pages and try to style them by hand.” And that ended up being the start of this project called Bootstrap.
CW I was going to say, was that Bootstrap?
US That was Bootstrap. So sometimes these things take on a life of their own and become big enablers for the industry. So that's also an important part of the journey.
BP Yeah. I mean, we shouldn't criticize rolling your own new thing. If we didn't, there would never be new things. For sure.
CW Innovation has to come from somewhere.
BP Absolutely. But I guess one of the most interesting lessons is when companies let folks roll their own internally, then you want to open source like a Go Lang or a React or a Bootstrap and let a community build around it. That's what's going to help it survive is creating that kind of ecosystem, not just keeping it siloed and inside of one organization.
[music plays]
BP Want to explore the latest tutorials around authentication, download code samples, and access a community of thousands? Then you need to visit the Auth0 by Okta developer center. Visit today at developer.auth0.com/stack.
[music plays]
MK I'm not sure how in-depth you can go on this, but is there anything that people should be extremely excited about in terms of Patreon's roadmap for new features for creators? Because I think we're in a really interesting state now where the economy is kind of dropping a little bit and I'm sure everyone's kind of a little bit worried around how to facilitate their continued employment or their tenure in the creative industry, and Patreon might be one of those things that helps prop up a lot of people when the ad money dries up or whatever else.
BP My side gig as a performance artist coming soon.
MK Yeah.
US Yeah. So there is lots to be excited about. We are really excited about next year and what we are building for creators and we think that what we are building has now even more of a place in the world than it had before with the move to algorithmic feeds where creators are not assured reach to their audience. Even in a world where they have amassed millions of followers, if they're going to have to recompete for that reach with every piece of content they put out, it's not a very favorable sort of set of conditions to operate in as a creator. Our product vision roadmap is all centered around this idea of giving creators control and ownership of their own audience. So you come to Patreon, you can bring over your fan base from wherever that fan base exists today. It could be YouTube, Instagram, TikTok, Snapchat, wherever that exists, often in multiple of those places. And you bring that over and that on Patreon is where you start building a direct connection to them. And we treat the users as the creators’ customers first, and our users later, which is very different from how many of the larger platforms look at their users, which is that they are those companies’ users first, and they just happen to follow the creator. So it's a very different way of looking at things. And in terms of our roadmap, once you have this central notion of direct control and ownership you're building around, a lot of things open up as things that we are interested in doing. And the crux of our roadmap is centered around this idea of content and community. So creators should be able to offer exclusive content to their fans– content that they've created, content that their fans are excited about, and have it be exclusive on Patreon so that it requires some amount of payment, either one time or direct recurring monthly payments for access to that content. And then community, which there is often a lot of excitement around creators, fans, wanting to just communicate with each other around the creator's content and we think of it as the buzz that happens before the release of a new movie where everyone is talking about what this movie is going to be like. We want to recreate that feel in the creator fan community, and we'll be building things in the product to kind of facilitate that for creators. So the content community is one of the big cruxes of our roadmap, and the other half is what we think of as creator business tools. So as a creator, you're half a creative person, but the other half is also a business person trying to run your business and make sure that you're delivering the benefits that you promised to your fans. You're figuring out how to grow your fan base, what are the promotion marketing strategies that are working best for your business so that you can double down there. You need tools to understand that, you need tools to re-engage fans who may be at the risk of churning out. So there's a lot of business tools that we can also offer to creators, which is something that we’re focusing on over the next year as well. So all in all it’s going to be a super exciting next year here for us where we provide creators with tools to grow and manage their business, and also tools to offer content community to the biggest fans.
BP That's very cool. So it's almost like you've got some Shopify-type tools coming in for people to build their own business on top of these more creator-specific tools. I've certainly seen over the last couple of weeks, as the worlds of social media and creator environments are in flux, a lot of people saying, “Hey, I'm starting a newsletter. Please follow me here.” Like, “I need to find a new way to keep in touch.” And so the issues you mentioned about migration are really interesting. How do you help people migrate their fans there? Is that a technical thing? Is that more about community focus? How do you do that?
US So we want to provide the tools for creators to be able to promote themselves on these platforms and say that there's something extra to be had here. One of the things we shipped recently was this notion of video previews, so when you host a video natively with Patreon, we will automatically generate a preview for you that you can then post to your socials. And that has a call out built into it to say, “Hey, come over to Patreon to see the full version of this video.” And that's pretty frictionless for the creator because they don't have to come up with copy or come up with new things to say. It's just kind of an always on promotion for their content. Because one of the hurdles we had seen in the past was creators felt hesitant to go back repeatedly to the fans saying, “Support me on Patreon,” because that's not really new information or you're not providing sort of new content to go with that request. So there are similarly many such tools we can build to make creators really feel good about promoting and marketing themselves on social media. And the other part of this is also that they should be really proud of what exists on Patreon. For you to promote to the whole world, “Hey, come over to Patreon,” your Patreon page should look really nice and it should put your creative work in the best light possible. So we'll be doing a lot of work over the next year also to make the creator page something that the creators can be really proud of.
BP Very cool.
CW As a comment in general, I do think that I like that there is some customizability of the creator pages and I think with any more of that artists would have a field day to be able to customize more and kind of make it look like something their own. And I think there was a time on the internet with the MySpace days and stuff where everybody was customizing everything and you could really have your little corner of the internet and then it kind of went very standardized and stuff, and I feel like I see that trend coming back of people wanting to be able to customize more and have a space on the internet. So I'm excited to hear that more of that is coming because I feel like that is definitely where the internet is going.
MK One of the things that I've heard from friends and other creators who are using Patreon as part of their revenue model, is that creating content on Patreon sometimes becomes a second job in itself. They've got whatever they're doing, whether that's tutorials or all the creative pursuits that are bringing people into Patreon and then they've got the time dedicated towards creating Patreon content as well. This is more of a high level question as opposed to an engineering question, but what kind of ways would you suggest to kind of help alleviate that problem of having all the extra head work and lead room to create Patreon-specific content as opposed to trying to more easily facilitate or automate that process?
US Right. I think our North Star here is to make it so that it's not an additional job, that it's integrated with your workflow as a creator. So you're thinking of your content creation journey as one distinct thing that spans multiple platforms versus, ‘this is what I'm creating on YouTube, this is what I'm creating for Patreon.’ And we've seen there are many creators that actually do some innovative things here that we can definitely provide better support for in the product. So that's something that we can consider. For example, we see creators do things like time windowing. So actually this is actually supported already in the product where you can make it so that your content becomes available on Patreon first, and then after X days have elapsed, it becomes more publicly available. And similarly you can do previews for example as I was talking about earlier, where you could show a segment of the content for free, but then the fuller version of the content is available behind a paywall. Other things people have tried are around the uncut version or the backstage version of the video is what is available. So I think as we continue with our journey here, we want to provide support for those journeys in the product so that it feels less like an additional thing you have to do on the side versus just one integrated thing that you do and it takes care of your socials, your Patreon, and helps you run your business with the least effort possible.
BP Very cool. You mentioned starting to create something internally that filled a need and then it went on to become Twitter Bootstrap. How did that happen from realizing that you were filling a need to realizing that it had momentum either internally or people were noticing it externally and then bringing it out to a larger community and seeing it grow on its own?
US Yeah, so the way that worked was just it ended up first proving very useful internally, so it grew from just within the ads team to the rest of the Twitter engineering team. And Twitter engineering also had a dedicated sort of open sourcing team, which was looking at what internal things could we open source, and this was definitely a strong contender given how much of a need it filled both internally and externally in the world. So that's how it went on to become one of the open source projects. Twitter has a rich array of them, but this is probably the most popular one.
BP Very cool.
[music plays]
BP All right, everybody. It is that time of the show. I can't believe this is a question, but here we go. Andy was awarded a lifeboat only 13 hours ago, helped to save a question from the dustbin of obscurity, and gave it a good answer with a positive score. “Please give me a list of all tags used on Stack Overflow.” I don't believe Andy has answered this question, but apparently he's answered it, and good for you, Andy. Enjoy your badge.
CW There you go.
BP There you go. They're a completist. All right, everybody. Thank you so much for listening. We always appreciate it. It's great to have folks on who have worked at so many companies, have so much experience to share and have helped to create things that went on to be big open source projects that so many people can avail themselves of. I'm Ben Popper. I'm the Director of Content here at Stack Overflow. You can always find me @BenPopper on Twitter as long as that site exists, I guess. And then you can email us, podcast@stackoverflow.com with questions or suggestions. And if you like the show, leave us a rating and a review. It really helps.
CW I'm Cassidy Williams. I'm the CTO over at Contenda. You can find me @Cassidoo on most things. Find me on Patreon, it's @Cassidoo there too.
MK And I'm Matt Kiernander. I'm a Developer Advocate here at Stack Overflow. You can find me online @MattKander on YouTube and Twitter. And if you want to fund $8 checkmarks on Twitter, then go check that out. No, we don't have a Patreon page for that. That was a joke.
US Alright, this is Utkarsh. You can find me on Twitter @Utkarsh. It was a joy to be here and it was great talking to you all.
BP All right, everybody. Thanks again for listening and we will talk to you soon.
CW Bye!
MK Bye!
[outro music plays]