When a company hits a period of hypergrowth, developers are in for a thrill ride. They need to start scaling their systems, moving to service architectures and clouds, and looking to solve problems others haven’t. But hypergrowth brings headaches, too, and chief among them is how to keep everyone aware of what’s going on with teams that they aren’t a part of. When Spotify ran into this hypergrowth problem, they created Backstage, an open-source framework for building developer portals. Ben and Ryan talked with Helen Gruel and Tim Hansen about the genesis of the project, keeping docs with service information, and how Backstage’s plugin ecosystem keeps engineers from getting lost among dozens of tools.
Like a lot of good tools, Backstage started as a way to stop using a spreadsheet. They knew it was something worth open-sourcing when conference attendees paid more attention to the tool than the topics of the talks.
Backstage treats docs-like-code, keeping markdown files in the same repo as the code. Down with wikis, up with pull requests!
If you want to learn more about Backstage, check out our recent webinar with Emma Indal, a web engineer at Spotify.
[intro music plays]
Ben Popper Want the best remote engineering talent? Well join over 300 companies who trust Turing.com to source, vet, match, and manage developers. With two million developers and over 100 skills, hiring high quality engineering talent has never been easier. Enjoy a no-cost two week trial at Turing.com today.
BP Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I am your host, Ben Popper, Director of Content here at Stack Overflow, joined as I often am by my wonderful colleague and collaborator, Ryan Donovan. Hey, Ryan.
Ryan Donovan Hey, everyone. How are you doing?
BP I'm a big Spotify fan, it's no secret. If you go back and look at my career when I was a writer at The Verge I did a bunch of stories about Spotify. Discover Weekly is still my favorite algorithm of all time. So I'm excited today, we're getting a chance to talk with some Spotify folks. But we're not talking about music, we're talking about developer stuff, this being the Stack Overflow Podcast. We're going to be chatting a bit with them about Backstage. It's an open platform for building developer portals. So without further ado, I'd like to welcome Tim and Helen to the show.
Tim Hansen Thank you. Glad to be here.
Helen Greul Hey, hey!
BP Hey! So nice to meet both of you. So just for our audience, if you could quickly Helen, give us a quick overview. How'd you get into the world of software and technology, and how did you land at the role you are today, and then a little bit of what it is you do day to day?
HG Yeah, I started as a software developer a bit more than 10 years ago. I used to work for industries like finance and healthcare, so pivoting to Spotify was quite a gig for me, switching to this new domain of entertainment and media basically. And then a little over five years ago I moved to Sweden to work for Spotify. I originally started as an engineering manager which I am still today. I started with the core business of Spotify, closer to the catalog where the music would go in where all of the content would be adjusted. That was me as a user of Backstage first before switching to the team who actually builds this. And then a little over two years ago I switched to the platform function within Spotify that provides all the tooling, including Backstage, to the developer community within the company. So my role today is a tech manager within the developer productivity area, working specifically on Backstage both externally and internally. That's what I do day to day. No two days are the same in a role like this, but most oftenly I find myself somewhere in the triangle of people, process, and technology. Depending on the situation, depending on the day, you could see me in one or the other corner of this triangle. Outside of work I cook a lot, I raise two kids, so that's me.
BP Awesome. Thank you for that great intro. Tim, what is your story? How'd you get into the world of software and technology? How'd you land over at Spotify, and what is it you do day to day there?
TH So I was kind of born into technology as it were. My mom and dad are both software engineers. They were back before it was cool. So growing up I always had computers around and got to toy with stuff, so kind of privileged in that way. So I've had a career over 20 years now writing software for various companies, Hewlett Packard, US government, finally landing at Spotify. And I started with Spotify in Sweden. I'm from the US, but I relocated to Sweden and started with Spotify, lived there for a year and a half before moving back to the US and now I’m based in Colorado. I've always worked in our platform organization, so always serving other developers which I think is awesome. It's a fun career to have to have users that are essentially the same as what you do day to day. You can really appreciate the struggles they're going through. When I started, my initial team at Spotify was working on Backstage, which wasn't called Backstage back then. It was the first product I worked on and I really love it and think it's a huge benefit to any company. So I'm very excited to now be working on the open source side of things and distributing that out to other companies as well.
RD So at my last job I encountered a problem like this, trying to create some sort of central space where all the developers had all their things in front of them on one single page because there were so many tools. So why did you all decide to make Backstage?
TH The origin of it was really that Spotify got into this stage of hypergrowth and we were just hiring like crazy, all these new developers were coming on and you kind of lose that institutional knowledge being widespread so everyone doesn't know all the pieces anymore or who owns what. So it started off just as a catalog of ownership, like who owns this microservice that's breaking in prod right now? Who do I contact about this? That was really the start of it. But once we had the tool in place we found it served a lot of other use cases as well. So one of the early things that we added on to Backstage was provisioning capacity so you could go and create a pool of VMs for your service, which was previously this process where you had to go and file a ticket and wait for IT and then maybe you got what you wanted. So having it all automated was this huge draw for developers and it kind of grew from there and ballooned.
RD I was going to say about the service tracking, we ended up creating just a Google Sheets to do that exact same thing. That's a huge problem within these large microservices.
TH Yeah, that's kind of where you start from initially, spreadsheet is like the go-to. I think Spotify has this culture of autonomy where when Backstage was first created it was kind of known that you couldn't create this one tool that would serve everyone perfectly and be like this platform team that owns the whole thing. So from the very start it was designed to be extensible to have these plugins where you could add functionality. So I think that was kind of key to making Backstage successful ultimately.
BP And Helen, how about you? What was your experience with the building of Backstage or some of your early experiences using it with some of your teams?
HG Yeah, I guess it was like a cure to never end in spreadsheets in my opinion, because for every company that I would start with, my boss would hand me the massive spreadsheet of 50, 40 tabs with all of the systems that my team would own and be like, “Good luck. Keep them in good shape.” And this was the first instance where it was actually something a bit more structured. It was still the early days of Backstage and we didn't even call it Backstage back then. It was System-Z when I joined five years ago, but it was at least like a structured web based representation of who owns what. And then as we saw teams were eager not only to see their software and components, but also to manage them through the same interface, that's how the idea evolved further.
BP I want to ask about the decision to make it public, but just before we get there, quickly, can you talk a little bit about the tech stack, the tools, or the frameworks? What is Backstage built of or with, and is that similar to how you build everything else at Spotify or is it different?
TH So our Backstage that we had internally that started the process out was built on Java GraphQL in the back end and was originally Angular in the front end but then converted to React sometime later. And we used Apollo Client for GraphQL communications. So it was pretty standard stack for Spotify. The thing that diverged is when we went open source we changed the back end to Node instead of Java and that was fairly controversial at Spotify.
RD Why was that controversial?
TH We tend to stick to kind of a subset of approved languages for specific purposes. So for back end generally Java’s preferred unless you have a good reason to deviate. In this case we felt like we did because it gives you this holistic development experience where you can share code between the front end and the back end. It makes it easier for someone to jump into the project because you can just run it all as a single Node service. We felt like it would aid contributions to the project having it all in one language and I feel strongly that it has.
BP Were you one of the folks who had to make the case, as you said, for why you should move away from Java?
TH I was not, fortunately. We do have a little bit of this awkward situation now where our internal Backstage is still a little bit in the old style. So we're trying to adopt the open source project ourselves and that's been a little bit of a messy process and we've had to kind of sell internally that we should be using the open source project fully ourselves and not just in some partial way. So that's been a migration that we've had to do that other companies fortunately don't have to.
RD So why did you guys decide to open it up? I understand from an external user, this was the solution to my problem at my last company, but why did you guys decide to let it loose?
BP Yeah, now you're going to have angry commenters and people asking if you’ll fix things and clients. Who needs that kind of headache?
HG Yeah, this was quite long in the making. I guess the conversation has been ongoing internally about open sourcing this since maybe 2019. And back then, we as Spotify were quite active members of the open source community as well, contributing to mostly cloud native projects. But we’ve open sourced technology before this, maybe not to the level of Backstage, but there were various projects already. But then what really set this one in motion was conversations that we had with end users in some of the cloud native end user communities as well as peer companies. We realized that the challenges that scale ups are experiencing, other big companies like Spotify are experiencing, are probably not that unique. So everyone struggles with growing their workforce, extending their teams, growing their product development capabilities and maintaining the developer productivity level essentially. And we were like, “We might be onto something here.” The biggest challenge back then was to really explain what a developer portal is, because the moment you would go, “A developer portal can solve this for you,” everybody would be like, “But what's a developer portal?” So that was the biggest challenge, just really trying to explain the concept of what it is, because we sort of created a new product niche that didn't exist back then. But then going back to 2018, a few folks went to one of the cloud conferences from Spotify and they were demoing some tool from within Backstage and it wasn't the tool itself that got the attention, but it was Backstage, the surface where we would do the demo from, and everybody was like, “But what is it?” And that probably triggered the idea of, if it's something that attracts so much interest that we're getting so many questions on, maybe it's something that we should really put out there in the open source community and see how people respond. And going back to the core philosophy of Spotify engineering and Spotify as a company, we truly believe in brain trust. And the way that Backstage was built, it was like a federated ownership model that sort of created the vast ecosystem that Backstage now is. It wasn't one team who built this, it was the brain trust of many teams that work within Spotify. So we just thought, “What if we put it out there? Imagine how many more teams, how many more companies would contribute to this.” And at the end of the day, this would benefit Spotify, this would unlock the creativity of the community. So that was the big motivation. The funny coincidence in the timing for actually putting this in the open source, it was the discussions back and forth internally within the company, and then we had Hack Week, which is a big, big deal in Spotify. This is essentially where all of the business shuts down and all of the engineers are just hacking and investigating all the new disruptive ideas. And a few folks got together to really see what Backstage could look like if we were to build this from scratch today knowing what we know now, what it would look like. And that was essentially the start of Node.js Backstage the way we know it today. And it was that week in early March when this whole thing was created and then we decided to open source it, and little did we know that on the day where it was supposed to go out, the office would be shut down because of the pandemic. It was our first day of working from home so people were freaking out like, “How are we going to open source it? We're not in the office. What are we going to do?” But it happened. It's funny to remember now.
BP That's a lot of things happening at once. I'm glad you managed to pull it off successfully.
RD Have you heard from any of the open source users about what they're doing with it and how it's affecting their lives there?
TH Yeah, we hear from open source users all the time. They kind of pop out of the woodwork which surprises us. These companies we wouldn't have imagined are using Backstage suddenly come in, they're like, “We love the tool!” As Helen was saying, when we decided to open source it there was this hemming and hawing, like, “Ah, is this going to be useful for other people?” We had kind of gotten that validation from CNCF companies. But looking back now in retrospect, of course this is something that's useful to everyone, because you need to keep track of all your software. You need to provision new things using best practices. You need to be able to find stuff. These are problems that happen at every single tech company of a significant size so it's kind of crazy to think there wasn't some kind of solution for that before and everything was tracked in spreadsheets. And we get so many companies that are using Backstage for different purposes. Some of them just use software templates to create new stuff. Some of them are only categorizing software. Some people only use API documentation, so you can kind of pick and choose the different pieces that are useful to your company and that's kind of what we encourage people to do, is to find the one most painful use case and solve for that first to get buy-in, rather than trying to tackle all the problems at once. Because that's kind of how it started at Spotify. We were just trying to track ownership and then over time it was builds and deployments and all these things and we discovered the value of bringing all this infrastructure into one place.
RD Was there a use case that surprised you?
TH Yeah, there's a couple. There's a lot of companies that use Backstage in ways that surprise us. One that comes to mind is Netflix. They use Backstage largely for the back end and they've kind of replaced the front end with their own design library and rewritten a lot of stuff. And we were like, “Really? Okay.” But there's other companies that are using it in ways we wouldn't have anticipated. They use software templates to create PRs and add infrastructure onto existing services. And we get feature requests all the time, but they’re just things that we don't use at Spotify and wouldn't have thought of but are perfect uses for Backstage.
HG Yeah, I personally find joy in hearing about the naming of Backstage, because in order to make Backstage feel like it was born within the company, people reskin it sometimes or give it different names. So at Hello Fresh for instance, they call it Hello Dev, at Zalando people call it Sunrise because this is the start of your day for developers, so the first thing that you open in the morning. So it's fun to see how your product evolves thanks to the creativity of people using it. The other thing that sits close to my heart is how people measure the effectiveness of Backstage. It’s not a question for a lot of those adopters whether they should use Backstage or not, it's more of a question of how to get the best out of it and how to really maximize the return of investment from the tool. We don't actually track per se the different ways that organizations measure the impact of Backstage, but there is a lot of appetite and interest to discover how Spotify does it so sometimes we also share this with our adopters. Internally at Spotify, going back to the roots of this initial story of how the whole thing started, it started with the onboarding, the challenges that people experienced during their onboarding. So that still is one of our North Star metrics that we track– the onboarding speed. And we measure this with a few different things. One big metric for us is the time-to-tenth PR because this is what we believe marks a successful onboarding case, the moment where an engineer is able to push their tenth PR to production. And historically when we started with Backstage the metric ballooned to being 60 days. Believe it or not, it took two whole months for someone onboarding to Spotify to be able to push their tenth PR to production. And since we started measuring, the metric went down to 20 days now or even less. So that's also something that we suggest that people use when they're in the hyper-growth period and hiring engineers like crazy. We also look at various different signals, like reviews, commits, we look at engineering satisfaction, this is a big one for us. We really want to keep an eye on the internal sentiment. We really want to be at the forefront of developer experience. So every quarter we measure engineering satisfaction with the tool and it's been steadily trending in the 85-86% of all the respondents within Spotify are genuinely happy using Backstage in their day to day. So yeah, don't get me started on metrics, I can go for hours. That's one thing I'm really, really passionate about so this is dangerous.
BP Okay, great. Well yeah, as an engineering manager I can understand that. You want to have some solid objective metrics both for productivity and for happiness. You don't want to have crummy boss-ware that's measuring keystrokes or something like that. So one question I wanted to ask, you mentioned even from the beginning you had this idea of plugins. How do the plugins work? How many of them have been created? And then obviously we want to chat a little bit about, I know there's one that works for Stack Overflow. Tell us a little bit about those plugins and plug our plugin please.
TH Yeah, like I said, Spotify is decentralized in the way we give our teams a lot of autonomy. So you can't have some huge platform team coming in and stomping on people and trying to be domain experts in everything. So we knew early on that we wanted this to be federated in a sense and give everyone the opportunity to contribute and kind of build this platform together. So Backstage was really designed that way both in the front end and back end such that we could go to teams who are domain experts in database backups or in builds or something and work with them to build the plugin that they would maintain in the future in order to surface these things in Backstage. So it was really a process of getting teams on board to this shared platform and getting them to create the things that they wanted to see. And we found that really successful. There were some hurdles in monorepo setup and making sure teams could approve their own PRs without some gatekeepers standing in the way. But once we got everything set up and a good process in place it really took off. The question was, if you're building an internal tool, why would you build it as a separate website? Why wouldn't you build it in Backstage? It's become kind of the standard way to expose any kind of infrastructure tooling you were making. And as part of that, we use Stack Overflow Enterprise at Spotify and it contains tons of Spotify-specific knowledge that wouldn't make sense on public Stack Overflow. So of course as we built out Search as part of the Backstage experience, we wanted to include those Stack Overflow questions and answers. And there's a plugin for that.
BP Yeah, that makes a lot of sense. Stack Overflow for Teams is often for sort of the proprietary knowledge base that you're not going to find in a public web search that would take you to Stack Overflow or some other forum or a YouTube video. “How to fix Spotify internal tool” is not as popular of a YouTube video as you would think. Now that you plugged our plugins are there other ones that you think are interesting or some that you can talk about that are particularly popular? What do you see folks turning to a lot across the platform?
TH I'd say first, within Spotify we have over 150 plugins. I think it's closer to 200 internal plugins now that we've created long before Backstage was open source for the most part, although there's new ones cropping up every day. And they’re anything from database control, to managing your cloud resources and costs, to huge chunks of our data pipeline workflows, ML, deployments, things like that. So we have all this power within Spotify and we're trying to get some of that into the open source. Obviously the core use cases are there and it's getting better every day, but we want to take all this stuff we built internally and make it available to other people. But as far as the open source plugins, those core use cases get a lot of attention. So there's the software catalog which kind of keeps track of ownership and lets you associate metadata with all of your software and can onboard everything and make it easily searchable. Then there's tech docs, which was a really interesting use case in Spotify. We used to have documentation scattered all over like many people do. You'd have your Confluence and then you'd have some Wiki over here that was made five years ago and then comments in the code and everything just naturally got out of date and it was hard to find things, so we had this internal effort to make tech docs, which was basically co-locate your documentation as markdown right next to your code in the same repository, and then render that nicely as HTML in Backstage. And this completely took off and replaced all of our documentation tools really organically I'd say. Developers just like writing markdown much more than authoring JIRA tickets or Confluence Wiki stuff. And that way it just stayed up to date organically and naturally, because when you're searching for code and stuff, you find this documentation in the same repo, then we built ways to make suggestions when you find things that are out of date, which automatically creates a PR and cool stuff like that. So tech docs has been a big use case. And then software templates I mentioned earlier are huge as well.
BP Very cool.
HG Just maybe a quick comment to add to that. I guess scaffolding or the software templates that Tim had mentioned, there is an interesting twist to that plugin and how it's been a way for us to instill engineering culture without being in the room with those engineers often, without taking them through the specific procedure of, “This is how software needs to be built.” It was sort of our way to instill certain standards and achieve alignment and defragmentation with that simple plugin. So once you join the company, you already have basically a recipe, a template for how software needs to look and what's the metadata that you need to be including in the components. So that alone saved us hours and hours and hours of engineering.
RD Yeah, that kind of consistency is super important at a big company.
BP Yeah. I was just going to say, it sounds from listening to both of you like Spotify has a really robust engineering culture and it’s nice to hear you talk about some of the tools and practices with a certain kind of passion, because it's clear that you found solutions to common developer problems in your own unique ways. Why don't you tell folks who are listening, if they're interested in getting involved on the open source side, making contributions, checking it out, or if they think this is something they'd want to bring into their company or organization, what's the best way to get started?
TH So Backstage.io is the open source project website. That's definitely the first starting point. In the footer there there's a link to our Discord channel that's very active. We have tons of people popping in there all the time asking questions as well as many Spotify people and people from other companies using Backstage that are very passionate about it. We also have a website at backstage.spotify.com, which has more Spotify-opinionated blog articles and such about Backstage so that's another great resource to check out.
[music plays]
BP All right, everybody. It is that time of the show. I am going to shout out the winner of a lifeboat badge: somebody who came on Stack Overflow and helped rescue a little knowledge from the dustbin of history. This is one of the funniest questions I think I've ever found. Like I said, I'm no programmer but– awarded 17 hours ago to Abuu, “I'm trying to update PHP Composer, but a problem arises.” I know that sounds like a vague question, but I'll share this in the show notes and when you go to look you'll see many problems.
RD Feels like a movie trailer.
BP Yeah. It's a setup for a PHP action film. There's a great answer in there that solves the problem, so thank you to Abuu and congrats on your lifeboat badge. All right, everybody. Thanks as always for listening. I am Ben Popper. I'm the Director of Content here at Stack Overflow. You can find me on Twitter @BenPopper. You can email us with questions or suggestions, it's just podcast@stackoverflow.com. And if you like the show, why don't you leave us a rating and a review. It really helps.
RD I'm Ryan Donovan. I edit the blog here at Stack Overflow. You can find me on Twitter @RThorDonovan, and if you have something you'd like to see us cover on the blog, please email me at pitches@stackoverflow.com.
HG Everyone, I'm Helen, Tech Manager at Spotify. You can find me on Twitter @HelenGreul and you can find me on Stack Overflow too. I'm a happy owner of a necromancy badge.
BP Ooh, the necromancy badge. That's one of my favorites to shout out. Congratulations to you.
TH And I'm Tim Hansen, Senior Engineer at Spotify. And you can find me on Twitter @Timbonicus or LinkedIn or Stack Overflow. My username is never taken so I very much enjoy it on all platforms.
BP That's great. Wow, two Stack Overflow active users in the flesh. A rare thing indeed. All right, everybody. Thank you for listening and we will talk to you soon.
[outro music plays]