On today’s episode, we’re airing a conversation recorded at the Next.JS conference. We chat with Lee Robinson, VP of Developer Experience at Vercel about the company’s vision for evolving from Webpack to Turbopack, helping customers who are cautious about migrating to a new tool, changes happening with React server components, and much more.
Webpack has been king for several years. Vercel wants folks to embrace Turbopack, but their claims about speed raised a lot of backlash after it was first announced. Lee explains why he thinks the Rust-based approach will ultimately be a big benefit to developers and how organizations who are deeply ingrained with existing tools can safely and incrementally migrate to what is, for now, a very Alpha and experimental release.
We go over the routing and rendering updates in Next.JS 13, exploring where it might offer developers more flexibility and the ability to use React server components to ship less, maybe a lot less, JavaScript. As Lee says in the episode:
“So to your point about wanting to ship less JavaScript, that was a kinda fundamental architectural decision of where we headed with the app directory. And the core of this is because it's built on React server components.
The key thing with React server components is that as your application grows in size from one component to a hundred thousand components, the amount of client-side JavaScript you send can be exactly the same. It can be constant because you can render every single component on the server.
And that's a lot different from the world of React applications today, where every new component you add for data fetching or just putting some HTML on the screen also adds additional client-side JavaScript.
So this is kind of inverting the default, back from the client to be server first. Now, of course, we still love client-side interactivity that React provides making really interactive and rich UI experiences, but the default for data fetching or just getting HTML to the browser happens from the server, and that's gonna help us reduce the amount of JavaScript.”
You can learn more about Lee on his website, LinkedIn, and Twitter. To diver deeper into his take on how Rust will impact the future of Javascript, check out a post he wrote here.
[intro music plays]
Ben Popper Amazon Web Services uses the world's most comprehensive cloud platform to solve difficult tech challenges. Learn how their expert team and inclusive culture make it happen at aws.com/careers/life-at-aws.
BP Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I am Ben Popper, live from San Francisco at the SF Jazz Center, getting ready tomorrow to attend the Next.js conference. And we were lucky enough to get a little bit of news under embargo which we're going to chat about today. I'm here with my frequent colleagues and collaborators, Ryan Donovan and Matt Kiernander. How's it going, y'all?
Matt Kiernander Very good, hello.
Ryan Donovan Good, how are you doing?
BP Good. Good to see you. So today we have a great guest, Lee Robinson, who's the VP of Developer Experience at Vercel, and we are going to be chatting about some of the changes that they are announcing, some of the new stuff they're announcing today, and how that will be impacting developers around the world. Lee, welcome to the program.
Lee Robinson Thanks for having me. Happy to be here.
BP So Lee, for folks who don't know, tell us a little bit just about how you got into the world of technology, web development, and how you landed at your current role.
LR Yeah, I have been coding for over 10 years now. For the majority of my career, I was doing kind of product development work, and then I found this fun thing called DevRel, which has been a great career change for me because it's a great intersection of all the things that I love, whether it's programming or design or writing or going to conferences and speaking and just trying to help educate and grow a community I think is super powerful and a super fun thing to be doing. So I've really enjoyed this latter part of my career. Now I'm the VP of Developer Experience at Vercel, really just trying to make the absolute best products for developers and make it as easy as possible for them to have a quick and fast development cycle and go from code on their machine to URL in the browser as quickly as possible.
BP So I think the big news that we want to get into first, and Matt, maybe you can help us lead off, was the stuff around Webpack and Turbopack. We had discussed this as a group at Stack Overflow and somebody said something to the effect of, “Webpack has been the king for seven or eight years, but clearly is technology that needed to move forward.” Matt talked about the pleasures of going to make a coffee or take a walk around the house while it was compiling, but it doesn't necessarily keep him in his flow state. So Matt, why don't you tee us off to ask a little bit about this, and then Lee, maybe you could talk about some of the technology behind it.
MK Yeah. So I've been having a look through some of the developer experience documentation of Vercel and the philosophies behind how you approach developer experience, and I'm very curious how they're kind of translated into how you built Turbopack and the process and the integrations that you went through to build out some of those features and make sure that developers were maintained in their flow state. So if you could speak to that and how Turbopack came to exist, that would be incredible.
LR Yeah. Developer experience at Vercel really rolls up in everything that we do, all the way up to really our company's mission, so trying to help developers create at the moment of inspiration. And to do that you have to remove all the barriers to entry and have the fastest iteration cycles possible. And as we were kind of just talking about, Webpack has been a really foundational tool for building the web, but when it was created, everyone was writing JavaScript tools in JavaScript, and that worked really well for a while. But now in 2022, 10 years later, we kind of have reached the limits of what is possible with some JavaScript based tools. And really the idea with Turbopack, to create a successor to Webpack, we needed to reimagine not only the architecture it was built with to make it incremental by design, but also take advantage of native tooling like Rust that can help us squeeze out those last bits of performance.
MK And were there any challenges? Web developers are very used to their JavaScript stacks. There's full stack JavaScript stacks that have existed in production for quite some time now so, making that move over to Rust, how do you anticipate that being received by developers?
LR Yeah. The move to Rust is mostly transparent for application developers. So if I'm an application developer upgrading to the latest version of Next.js, or I'm a framework author trying to integrate with Turbopack, I'm able to just not worry about the underlying architecture that Next.js uses, because from my point of view, I just write my React code and it's really fast both on my local machine and when I build for production. Now, for the actual framework team behind it building it, I think it does pose opportunities to evolve the ecosystem, obviously Webpack being a very, very large tool with a very large community. There's thousands of plugins out there that people use to kind of extend Webpack and add additional functionality. We want to make sure that the Webpack community evolves with Turbopack and make that a seamless incrementally adoptable transition, because people like to extend their compilation tools or their unification tools or their bundling tools, and we think that there's a way that we can both allow extensibility and keep the performance metrics that we want to make it fast by default.
RD When we do our survey, Rust comes out on top on most-loved languages all the time. What was all your experience working with Rust for this?
LR The Turbopack team at Vercel and the larger Turbo team which includes Turborepo, really, really likes Rust. They find it to be a very productive tool for them that enables them to really get the last mile of performance. And I think when you look at the larger picture of Next.js, we're moving towards this world where you might be able just to install Next.js as a binary and you don't even have to use NPM for n number of dependencies. It's just one kind of pre-compiled dependency. We've already started to move towards this direction because one, it allows us to vendor packages and prevent an issue where somebody hijacked a package on NPM and then it became compromised, and also it helps improve the installation time. So as we move forward towards that world I think it will take over more pieces of the experience and build it into the framework itself and probably use native tooling like Rust.
MK One of the things that we chatted about last week was that people are very married to their workflows. Webpack has been a titan of web development for a very long time. People are used to how it works. They're used to the plugins and the ecosystems, and they've got their custom built configs and all that kind of stuff. And one of the hurdles that we potentially identified was being able to take an existing project that is in Webpack and then transition that over to something like Turbopack, for instance. So can you speak to how that transition might look? Is it going to be just that if you're starting a new project you need to start with Turbopack? If you're on an existing Webpack project, can you transition that over to something with Turbopack?
LR Yeah, so for this first release, we're releasing Turbopack in Alpha, and there's a lot of stuff that's not done yet. We wanted to open source this, get it out to the community, get it in public, and have people start giving us feedback on what works and where they'd like to see the direction go. The long term vision here as we move forward in stability is to give developers the tools that they need to incrementally move over from Webpack, as well as the extensibility of a plugin system that they can use to maybe rebuild a very common Webpack plugin that they use for their custom internal setup. That's going to be kind of a major requirement if they were going to be moving off of Webpack. And we believe there's a way to do that and provide an extensible bundling platform while still having it be extremely fast. So it's something we're thinking a lot about, still pretty early, but I definitely want to make sure that the experience for developers both with their own kind of custom Webpack setups, as they try to move forward and incrementally move to this new architecture, that they can do that in a seamless way. But I also think there's a larger shift here, which is that a lot of people who were previously rolling their own Webpack configurations are now saying, “You know what? Rather than this hand rolled artisanal Webpack React application, I'm actually going to move over to Next.js, or I'm going to move over to these Meta frameworks. They're going to just manage that for me.” And that's the incremental migration I think that we're seeing a lot more often, because then not only do you do that incremental adoption and migration one time, but then in the future when Next.js 14 comes out or 15 comes out and we continue refining the underlying tools to make it the best possible developer experience, you're abstracted away from that and you don't have to go through that process.
RD I wonder how much of those kind of custom scripts were working around limitations. You're saying this is 700 times faster. Are a lot of those scripts now moot? If you have these plugins that are trying to speed up your Webpack process, are they going to even be needed anymore?
LR That's a good question. I think that, yeah, if there was a plugin that was trying to speed up your build process by caching something strategically or trying to add incrementalism into a tool that fundamentally wasn't designed for that from the start, a lot of that will be null and void or obsolete, because the architecture of Turbopack is quite a bit different actually than what Webpack started with, just because of 10 years of experience not only with Webpack itself and how it's evolved over time, but also the team at Vercel taking the expertise of Webpack as well as the expertise of Turborepo and other related tools and kind of building that together into a shared incremental engine.
BP So one of the other things that we had chatted about last week with our colleague Cassidy Williams who's on a lot, was her love for Next.js and her frustration with the amount of JavaScript that it sometimes ships and the inflexibility of the router. So when I showed her the press release under embargo it kind of spoke directly to her. I guess maybe you're getting a lot of similar feedback and making these improvements. Can you talk us a little bit through the routing and rendering updates and what you think that's going to mean for folks?
LR Yeah, I'm extremely excited about this. I've been using Next.js for a long time now, and one of the first things I gave feedback on was, “Man, it'd be great if we had a little bit more flexibility with the routing system.” And I remember the first day I joined Vercel I asked the team about it and they were like, “Oh yeah, we've thought about this a lot and it's a tough problem to solve.” And that's when I realized that Next.js, which started in 2016, since that time they've really been thinking about what's the best way to build a router for the web that's both easy to use for getting started, as well as super extensible as you move forward throughout the life cycle of your application. So the new routing system in Next.js 13, which is in the app directory, is incrementally adoptable for people to try out. And I think this is probably the most important thing, because if I'm an application developer building on Next.js, we can't just swap out the entire underlying rendering and routing system in a major version and just call it good. That's going to potentially cause a lot of churn. So we wanted to take a different approach, which is moving from the existing pages folder and the pages router over to the app directory and this new router and allowing people to upgrade to Next.js 13, use some of the new features, you don't necessarily have to use this new routing system, but as you're ready you can kind of slowly move pages over into this new world. And as you move to this new world, there's a lot of really amazing benefits and why people would want to move, but we want to give that flexibility and that control to the development teams. Because if you have hundreds or thousands of pages in your application, it's just not feasible to do all that in one swoop. So to your point about wanting to ship less JavaScript, that was a kind of fundamental architectural decision of where we headed with the app directory, and the core of this is because it's built on React server components. And the key thing with React server components is that as your application grows in size from one component to a hundred thousand components, the amount of client side JavaScript you send can be exactly the same. It can be constant because you can render every single component on the server. And that's a lot different from the world of React applications today where every new component you add for data fetching or just putting some HTML on the screen also adds additional client side JavaScript. So this is kind of inverting the default back from the client to be server-first. Now of course we still love client side interactivity that React provides to make really interactive and rich UI experiences, but the default for data fetching or just getting HTML to the browser happens from the server, and that's going to help us reduce the amount of JavaScript.
MK So you've spoken about a few pain points that Turbopack is addressing, one of which is speed, and the second of which is the amount of JavaScript being sent to the client. Were there any other pain points that you wanted to address with Turbopack that you haven't mentioned?
LR The biggest thing with Turbopack is both local iteration speed, like making updates in my editor and seeing these changes reflected in my browser immediately, as well as that we want to tackle having the fastest builds possible and making 10-millisecond builds. To get to that world it's going to require more investment from our team and right now we're solving the local iteration part, but we've built this architecture that's going to be able to kind of make iterations fast wherever that work is happening. That's a little bit different from the Next.js 13 and app directory goals, which are a little bit more focused on how can we provide a tool for developers to create the most advanced and complex UIs all while managing to ship less client side JavaScript and just have a better developer experience for offering those applications?
RD It sounds like with the router you’re offering a more sort of incremental upgrade experience where you can pick and choose features. I remember seeing years ago people on versions of Java that were like three or four points old because they didn't want to upgrade. The upgrade process would be too painful. How much of that is a concern in when you ship these new versions?
LR I mean, incremental adoption is a core developer experience philosophy for how we build Next.js and really how we ship software at Vercel. We have to be very cognizant of these large Next.js applications that exist in the world, that it's just not feasible for someone to make this jump in a short amount of time. So when we think about how do we build an incremental adoption approach into everything we do, it starts actually multiple versions ahead of when the feature is actually released. So just going back in time a little bit, the last time we did kind of a major shift in architecture for Next.js was when we moved compilation and minification to other kind of tooling pieces of what makes up Next.js. We moved those away from JavaScript-based tools to Rust-based tools as well too. And the way we do it is first we put it out in an experimental flag. We allow people to try it out, give feedback, but say, “This is not ready for the prime time yet, this is not stable by any means, but give us your feedback. Let us know how it goes.” Then we slowly iterate on it still in an opt-in fashion as developers are ready. And then finally we reach a point where this feature is stable, developers can go use it and they can expect that things are going to work, but it's not the default yet. And we make this slow transition towards finally now, for example in Next.js 13, the Rust-based minification is now finally the default after going through multiple rounds of experimental flags and stability channels, now finally getting to something that folks can still use. And there's even another layer to this too, which is that if you don't want to do that, you can still turn it off. And if you want to still use the Babble-based tooling you can still do that as well too. So it's a delicate balance of wanting to help developers get on the latest version and try out the latest features while also still respecting that they have large applications, they have a limited amount of time, they're being asked to ship XYZ features, and they don't want to spend three months managing their tooling.
MK Has there ever been a feature that as a team, as a company, you’ve thought that sometimes it's wanting to have that balance of that we want to make sure this is interoperable and people don't need to upgrade, but on the other hand, there might be another thought train being like, “This is for your own good. We are going to make this mandatory to make sure that it offers you the best experience. We know best and we'll make sure that you are following best practices.” I'm assuming that's not the way that Vercel does things.
LR So one related idea here is that Next.js provides a router and a server, and you want to be able to customize or modify the behavior of that router. And this is not only the actual rendering of pages and how you structure them in the file system and how that corresponds to addressable routes in the browser. It's also all the little configuration options like, do we have a trailing slash at the end? Should there be some kind of prefix on the route? Do we want to have internationalized routes that look at the browser and understand that my language in the browser is English and I'm in the United States and let's do the internationalized route pass that way. There's all these little things that add up to how you actually build the routes for your application, and over time we slowly added more and more of these bits of configuration options that our community asked for to kind of extend Next.js. And kind of to your point, we realized after a while that this was reaching a limit of really how far we can go with adding configuration options. We needed just to give developers the code. Just let them write whatever configuration they want and let them control every piece of that routing process. And that was actually what led to the creation of middleware in Next.js 12 where you can just write any arbitrary JavaScript to take an incoming request and then produce a response. Now in the process of doing that we had a lot of iterations on what that API looked like and got a lot of great community feedback. And the first iteration of that, I think we were pretty happy with the API design we landed on, but we also got some good community feedback like, “Hey, we feel like this part could be a little bit improved. We'd love to see this addition.” And my philosophy on it is that the customer or the developer is always right, because even though at the beginning I was like, “I don't totally understand why they want that feature,” I had to really digest and understand multiple people in the community giving this feedback, which is now leading to some further middleware improvements in Next.js 13 that we're making as well too. So I think that the key thing here is just always for us to be super plugged into the community's needs and what they're looking for out of the framework. And as long as we do right by them, I think we'll be all right.
MK So for anyone who's looking to contribute to Vercel's open source products and with the move to Rust, say for example for with Turboack, if we want to kind of extend on Turbopack's functionality, will we be doing that in Rust or will that be a JavaScript integration on top of that? Just because as Ryan has mentioned, Rust has been ranking very, very highly on the love score that we have every year, and I'm personally quite excited to dive into a Rust project to flex my skills and learn new things. So is there an opportunity here for Rust or is that going to be a JavaScript-based extension program?
LR Yeah, that's a good question. I'm not sure yet. I think in the ideal state we would allow developers to both write Rust or write JavaScript if they prefer. It's obviously hard to enable folks to still write JavaScript, or WebAssembly I guess too is possible, and then still have it be fast. So I'm not sure yet. We'll see how it progresses, but we want to make sure that it's easy for folks to kind of extend the functionality of Turbopack and really build whatever they need for their custom requirements.
[music plays]
MK And that brings us back to the end of the show. Thank you so much for listening. As always, we give a lifeboat shout out. This is an answer score of 20 or more to a question score of -3 or less that goes on to receive a score of +3 or more. Today's contributor is Rolf who answered the question, “What is @apply in CSS?” Thank you very much, Rolf, for being part of the community. I have been Matt Kiernander. I'm a Developer Advocate here at Stack Overflow. You can find me online @MattKander on YouTube and Twitter.
RD I'm Ryan Donovan. I edit the blog here at Stack Overflow; stackoverflow.blog. And if you want to find me on Twitter, I'm @RThorDonovan.
LR And I'm Lee Robinson. I'm the VP of Developer Experience at Vercel. You can find me online at leerob.io or on Twitter at @Leeerob.
BP I am Ben Popper. Find me on Twitter @BenPopper. Email us with questions or suggestions, podcast@stackoverflow.com. And if you enjoyed the show, leave us a rating and a review. It really helps. Thanks for listening everybody, and we will talk to you soon.
[outro music plays]