The Stack Overflow Podcast

When it comes to package managers, don't forget security

Episode Summary

In today’s episode, we chat about simplifying the address of your crypto wallet, static linking, the security risks you need to consider with package managers, and Paul’s paper hands.

Episode Notes

If you’re a programmer working with npm, Sara has some basic advice on best practices that will keep your codebase safe.

Today’s discussion was inspired by a blog post from Michel Gorny which you can find here.

Need to simplify the address where people can send you bitcoins? Check out, which even offers .club for your TLD.

Thanks to Tagir Valeev for answering the question: How to Split odd and even numbers and sum of both in collection using Stream. You’re our lifeboat badge winner of the week.

Episode Transcription

Paul Ford You know Naked in The Woods--whatever that show is called--just, that is technology. [Ben & Sara laugh]


Ben Popper Your team needs the most talented developers to build the most innovative products. TopTal makes it easy by unlocking the top 3% of on demand talent worldwide. Fast. Scale your team for success at

BP Hello, everybody! Welcome to the Stack Overflow Podcast a place to talk about all things programming, software, coding technology, whatever it may be. I'm Ben Popper, Director of Content here at Stack Overflow. And I'm here today with my two wonderful co-hosts. Hi Paul. Hi Sara.

Sara Chipps Hey Ben! Hey Paul! How's it going?

PF Doing good! Getting by!

SC Yeah!

BP Sara, I have this problem. I keep meeting people, you know, out at parties.

SC Are you going to a lot of parties right now?

BP Or not parties I guess. Let's say on Clubhouse, and we're talking about some new alt coin they created. And if I you know, only got a little bit of that at the start, I'd be really rich in a few years. And I say 'Oh, that's, that's great. Just send it to me. You know, I'm over at IBVB MSCEY, you know, 4G57932.' And then they forget, they forget my address. They don't bother to send it to my wallet. And I miss out on a big opportunity. So I was wondering, I was just thinking, there has to be a better way than these, you know, 26 character alphanumerics when somebody wants to send you their latest nonsense nickle, right? I mean, there has to be a better way.

SC Yeah, I'm kind of obsessed with this. This is my new this and NFTs are my new obsession. But yeah, so these domains, these ENS domains will link directly to your wallet, your crypto wallet, and it works with Ethereum based coins. So for me, so I registered SaraJo.eth, and if you send money to SaraJo.eth I get it. It's really easy. It's kind of like 'Oh, so yeah, you want to buy my car? Great. Just send, you know, half a Bitcoin to SaraJo.eth' It's pretty easy. So if anyone wants to send me money that is currently listening. SaraJo.eth.

PF They recreated DNS. But instead of DNS, it's built on destroying the entire ecology of the world.

SC No! Because this is Ethereum and the whole point of Ethereum is it's trying to become carbon neutral. Ethereum pays it down

PF Ohhhh, Ethereum's okay. 

SC Yeah, but Bitcoin's working on it too.

PF I got closer than ever. I downloaded--what is it--Gemini? 

SC Oh, yeah!

PF Yeah, 'cause I was talking to a friend, he's like 'Coinbase I have mixed feelings about Gemini is local. Depends how you feel about the Winklevoss twins.' I don't have a lot of feelings about the Winklevoss twins. So I got Gemini, and then I got bored. Same thing happened with Robinhood. And you know what I find? I just, I find a good index fund, incredibly motivating and exciting. I'm like "Look at that! Look at compound interest just working like it has since the Florentine merchants! Oh, my God!" [Ben laughs]

BP You are the exception that proves the rule. That's amazing.

PF I'm just like "I'm going to spend all week thinking about the small amount of incremental revenue that's been generated by the trading and my socially responsible index fund." I don't feel like I have any friends. I don't have any friends in modern internet world.

SC Yeah, well, you know what it is? It's the it's the feeling of like losing half your net worth and doubling your net worth in a day. You know, it's just like, woof, woof. There's a lot of serotonin. 

PF See, that's not a good feeling! That's a bad feeling, Sara! [Paul laughs] It's that feeling. This is like when I read that article about the teens and college students who are drinking like half a bottle of Benadryl, and then having like bugs all over them. They're like, "Yeah, it feels really bad. And you have bugs and you stare into the abyss and it's terrible. And I hate it. I have a real problem because I do it all the time!" Like, oh, it's like Bitcoin!

BP It's a lot of feeling, Paul. It's a lot of feeling. I did see one thing, a tweet that said, if the Winklevoss I really do hold 1% of Bitcoin. They're on their way to finally getting their revenge and being richer than Zuckerberg. So hold tight, hold tight.

PF I hate everything. I really do sometimes. [Ben laughs]

BP But Sara, you had the option here through ENS to go with So I'm happy for you, but also a little disappointed.

SC Really? I didn't even see the club option. I would have picked that definitely.

BP Yeah, they've got other options here. You can you can get your decentralized wallet, and you can go with dot club or dot art, dot cred. There's a couple of them.

SC That's nice.

BP Well, that's our requisite crypto talk for the episode. [Sara laughs]

SC We made Sara happy, let's move on.

PF Crypto Corner with SaraJo.eth! [Sara laughs]

SC I'm ready!

BP But I saw an interesting article, which I share with you all this morning about package management and the idea behind the piece, just to sort of get us started and then we can look at the topic more broadly, was about security and sort of like what the modern package manager needs to think about and how the fact is that over time, good or best practices start to get baked into what people are doing with certain, you know, Linux distribution, or, you know, some of the languages that I can, that come up all the time, Go and Rust and Python. So Paul, help people understand sort of the premise here. Or maybe take us back to where you wanted to start with just sort of like, why you think this is such a central piece of being a modern software developer?

PF Central processing units are a set of little tiny alpha--okay, so I went too far back. This is a piece by Michel Gorny. It's on So if you're a Unix aficionado, there's a lot that that URL just told you. Right? We're in Gentoo, which is a unit, it's a Linux that likes to compile, and the piece is called The Modern Packager Security Nightmare. So look, this was a huge surprise to me, as I got further and further into the world of engineering and software development. And Sara, as someone who's on the debt, you know, related to and advises the DotNet Foundation, you're gonna have opinions here, too, you might think that programming has a lot to do with writing lines of code to make the computer do things. And that is absolutely a critical part of programming, sometimes. An unbelievable amount of programming, and actually look at we've had people from Vercel and Netlify on the show, yeah, you know, there's GitHub, there's, there's sort of all these unbelievable amount of programming is about managing dependencies to deploy the code securely. And on the Unix side and increasingly, on the Windows side, there's this concept of package managers as well as inside of programming languages. And you're like, well, that's just software and blah blah blah, you know, it's just like a course it installs the things like, you know, like my Windows Installer, and no! This is the actual hardest problem in computer science is--

SC I was gonna say, maybe we could talk a little bit about what it was like before package management. So I'm sure you remember, where you'd have these huge applications with all these libraries, and things were just so slow. And that's where that whole XKCD, where 'my codes compiling with the guy's sword fighting' like, that's the whole thing. Yeah, it's just because you have so much code.

PF Well I'm 25 years old. [Sara laughs] So I have no idea what you're talking about.

SC Well, back in my day.

PF Yeah, this is correct, right. So on a web server, let's say you might set one up by compiling Apache downloading the source code to Apache and running it, and then the cost to update that binary was a lot like it was like, I gotta go into the server, I got to re install it, I gotta remember and something might break. And I gotta run, configure. And so you end up in a situation where people are individually compiling software in an ad hoc way, where everything got insecure really fast. So package manager is really from a security point of view, from a user point of view, they're really, they tend to be a little bit behind the bleeding edge, but they are making you can write one command and have a piece of software on your system. From a security point of view, it means that there's a centrally updated repository of what needs to be running. So like, I run Ubuntu Linux at home, Groovy Querrilla, if anybody has any questions, don't ask. And no, but like, I run apt get disc upgrade on a regular basis. And that goes and gets all the you know, gets me a new kernel gets everything all tidy up for me. And that is a gift, like that is I mean, it's a hell of a thing to get for free. But what it means is when when something bad happens, and people figure out how to hack into mainframe, the package security manager is the first thing that jumps in is like, okay, let's, let's first let's fix it here. And then that will affect hundreds of 1000s of systems and get them get them up to date. 

SC Yeah, it's a it's a wild world. I think. I one thing that's been really fascinating to me the past few years is learning how different companies deal with these vulnerabilities or potential vulnerabilities. Like there becomes an extra step in some places where introduced, like it used to be that you could introduce a library kind of skim it, make sure that nothing was in there that was concerning. But now you're opening up your application to developers that have no part of your application. So means you're trusting these people. And so some companies, I've seen that they put boards together of people who this sometimes this is their full time job. And sometimes it's a gathering of developers and where people propose adding these libraries or frameworks. And this board is in charge of vetting it and voting on whether it's something that can be included.

PF Which, you know, random, you know, when you're when you're a young developer coming in to or less, you know, newer career developer coming into an organization and you would like to use some tools and get some work done. That that team will break your heart. 

SC Yeah, it's a buzz kill.

PF You know, you'll talk to people, whether it be like, yeah, they're on old versions of Java. You know, and it's because they finally got it to where it needed to be to work in 2008, and just, do you remember what it used to be? See, that's the danger, right? Like you and I are just like, yeah, used to be so bad. And everyone else is like, yeah, this industry moves a billion miles a minute, why are you making me use the old stuff? Or like [Paul laughs] You don't even know!

SC Yeah, yeah, I was talking to someone the other day. And they were saying that the past month, they've been in like, 50 hours of meetings, because their company is considering upgrading Java to a new version. And they just sit there with the license, and they just go through this stuff. It's really it's really fun stuff.

PF Upgrading Java is like, like when they moved that house in San Francisco down the street three blocks.

SC It's exactly that, it's exactly that!

PF Like you got it, they actually have to, like, separate the house put on rails, and then it moves about a foot a minute. I mean, you're just you're just, it's gonna be a big deal. And you're like, well, I guess so. I guess it's hard to move a house. I couldn't do it.

SC It's hard to move a house. Other places I've talked to, they built their own. It's a little less bureaucracy, but they built their own security robots, right. So it's like, if you want to use a library, you know, run it through this tool, so that we can make sure it's okay. And the tool runs continually. And that makes a ton of sense. And it seems like a great thing to me. But the first thing I think of is like, who built the tool? How good is it? And how long before someone figures out how to get around your tool?

PF Oh yeah. Who can hack the tool? And then it's sort of like someone recently figured out, you know, the most likely names for internal packages and packages into the different software repositories. Because, you know, when people do installs, it tends to install those tools instead of the local ones.

SC Yeah, did do we talk about this at all? This was a big one.

PF No, no.

SC Yeah, was it a security researcher? I'm not sure who figured this out.

PF It was.

SC But someone figured out was NPM, that a lot of companies had packages that were local packages to them. So say I work at, you know, company XYZ, well, we have a package called XYZ Classes.

PF Yeah. Utils is always a good one.

SC Utils! Yeah, perfect. And so they figured out that and we're loading it locally, and we're using NPM. Well, they figure it out at the time that if they created a public package called XYZ Utils, and put it in NPM, then NPM would automatically go for the public version before the private version, which was exposing all these companies to his code, which good thing was his code, but who knows?

BP That's the supply chain attack. That's the same thing as the solar winds, you know, like, get it in there as one of the Yeah, the basic tools or you know, things you're going to be sent every month from some vendor, and people don't even, their eyes just glaze over. 

PF Security and package management is one of those things where, you know, it's one of those like, 'Oh, well, adults are in charge.' It's like, when you realize that you're, like, even to a doctor who's younger than you, have you had that experience yet?

SC That is very upsetting to me. Yes.

PF It is. And they're like, you're like you, you don't have any of this knowledge that I have all bundled up in this mess. You know, and they're like, 'Oh, you're gonna want to do these five things.'

SC You are a child.

PF I'm like, you're right. But I you just told me that in a voice that sounds like Chippendale. I'm not ready for that, you know, I don't want to that. That is what security is, security is just like 11 year old doctors, and you're just like, 'uh oh'. [Paul & Ben laugh] 'I'm gonna take this advice, but I'm really scared.' And they're like, "Don't worry. We get new hearts all the time." And that's package management.

SC We can't live without now is the problem.

PF That's the thing. It is absolutely bananas. How much we are dependent on goodwill. In the industry that runs every subsystem. And every piece of infrastructure in the world. Like literally, you know, Apple is worth, you know, roughly, it doesn't matter how much is worth we know. Yeah, it's a jewel beyond price, right? And Apple was dependent on the goodwill of the node package management--NPM. Ouf, almost screwed that up. NPM, which is inside of which is inside of Microsoft, which is related to GitHub, which is like, these interdependencies are tight. And frankly, there are brilliant people evaluating them trying to get them right. And so it's one of these things like, the horrible truth about our industry is that coding is actually really secondary to these incredibly boring things like package management. And, like one of the best things you can do as a developer is learn about dependency management, and, you know, code regressions and, and sort of how to deal with and manage those. And it's, you know, GitHub is starting to automate a lot of this away. The point made in the piece that we were talking about, which I think is a really good one. There. It has a great summary, which is wonderful for podcast hosts, which I'll just read a couple sentences, so static linking, dependency pinning and bundling are three bad practices that have had serious impact on the time and effort needed to eliminate vulnerabilities from production systems. Meaning, don't copy libraries all over the place and cross your fingers, like have them in one place, update them once and that way you'll get the security throughout all the systems that depend on them. And so that's, package managers do that.

SC Yeah, but that's not fun. I mean like, I'm a developer, I want to type three, I want to type six letters, and then have, now I can use underscore, wherever I want.

BP I guess, right, you were saying like, we can't go back now. And some of this was the path of least resistance and of utility, right? Like people being able to grab this stuff that other people are building, and to work with it. People expect, if you're going to get in somewhere and be useful really quickly, that you know how to do this stuff. And that you're, you know, willing to be like flexible that way, or that you're willing to be versatile that way. 

PG The big vulnerability the author is pointing out is that new programming languages like Go and Rust are really big on static linking, you don't have the libraries in the library folder, instead, you just bundle it all together into a binary, which means that, that risk point, we're actually back to where we were 20 years ago. And of course, the memory management is better, and it's garbage collected, or statically typed. So that's gonna make it all better. But then there's that one, you know, you have now 30 different binaries that might include the same TCP IP library, that would be utterly vulnerable spread across your system, and you don't know which ones it is.

SC Are you saying it's like, where we were 20 years ago, because it's like, it's almost like dealing with proprietary code, because you can't even read it. 

PF That's right, and you can't get in there. I mean, it. It's practice rather than theory, right? Like, it's all open, you could read the code, you could you know, but now we're back to like, you know, Norton 360, auditing your system and saying, looks like 'You have wind sock 3.09. And, you know, you need alpha.' And like, how are we going to get through this? And there is an answer, and it's really simple. It's take all your computers and throw them out the window into the garbage.

SC Okay, I here, I have another answer. What if it's hoard cryptocurrencies? Because in all likelihood, your system is going to be held ransom by a library that you install?

PF You know, what we are here? It's literally people going 'No more capitalism, no more communism!' You know, essentially, the horrible thing about technologies, is that centrism is the only solution. It's just sort of like, "Yeah, well, we're just gonna have to work together all the different sides in order to find--" I know, nobody wants that. It's like, "No, we're gonna have a functional reactive package installer called [inaudible]" Where you're like, cool. That's the answer. That's right. We're gonna do that. 

SC That guy wrote a great blog post. That sounds right.

PF Okay, let's, let's leave on some advice here, right? Like, you know a lot more about this than I do, right? You're, you're involved. I'm just an observer. What do people need to know about package management? Like, it seems to be a very social thing to just organize these package management tools? Like, if you're a programmer, what should you be thinking?

SC Yeah, I think the biggest thing is that, you know, we think of, we think of packages, like a natural resource, like trees, or grass, where it's just like this will always exist. And this is here to help me. And this is very nice. And I think the biggest thing that you learn is that it's actually just like a bunch of three guys and a bunch of little rubber bands, and two garbage cans. You know, like, it's just like, and it's great. There are people that put a lot of work and a lot of their lives into this. But they're people just like you. So people building these packages that you're using all the time they make mistakes, like you make mistakes, right? They have systems to try to catch those mistakes. And sometimes they have big companies behind them that do the same thing. But in the end, you can't just trust that the things people are building are always good, and always there to help you. You know what, this is actually the same thing people used to say 20 years ago, we're talking about memory garbage collection, right of like, you can't trust the garbage collector to always be working on your behalf, you have to understand what's underneath it. So I guess it sounds really similar. But I think just understanding that the people building these libraries are just human beings just like you. And sometimes they make a mistake, or sometimes they get tired, and they don't want to do it anymore. And someone gets in there. And they do mean things.

PF This is a, this is a hard thing to learn, which is that you really do assume as you're coming along in your career, that there is a secret reservoir of genius. And even if you're not, you didn't get tapped. Someone's got it. And they're over there doing exactly what you would hope they're doing. That's not the case. Like you, you'll meet people from giant companies where everybody's supposed to be a superstar and they'll be like, the smartest people in the world. And you'll be like, yeah, you actually know about three times more than me. Yeah, you're sharper, but you're not actually like, I just watched you spill ketchup on your shirt. [Ben laughs] Right? Like there she is. There he is. And you're like you're running the world you're actually right there making the thing and you know, you're not able to have functional human relationships and you might have a bad day and and yeah suddenly, all of Python is done. So don't assume malevolence, don't assume and competence. Just assume that the, the window is a little bit less wide between your skill set and their skill set. Here's what I would say, here's a good exercise for all of us, which is go look at a package management solution, a magic thing that you type in a command line, and it goes and gets all this stuff. Go figure out how it works, like go read the source code, they're actually not that complicated, tends to be like a bunch of blobs sitting on a server somewhere, and they go and they fetch it. And then they run some scripts for post install, they have some hooks, and then they're done. Right? And stay away from configure files and make files until you're ready to die inside. But the other stuff is I mean, go look at like PIP or NPM, figure out how they work. And and then you'll be horrified because you'll see human frailty and vulnerability in all of its glory. You'll also kind of that'll make you a better thinker about how you're deploying and shipping your code because again, lines of code. Meh. Doesn't matter.


BP Alright y'all, I'm gonna read us a lifeboat and we can head our separate ways. This is from five years ago, 'how to split on an even numbers and some of both in a collection using stream?' And then I guess it's #JavaStream? Yes. So 21 upvotes and accepted answer. Thank you to Tagir Valeev, appreciate it. That's our lifeboat badge winner of the week. Contribute a little knowledge and kept a question from entering the dustbin of history. 

PF Good job.

BP Good job. Thank you. Thank you Tagir. I'm Ben Popper, Director of Content here at Stack Overflow. You can always find me on Twitter @BenPopper. You can hit us up You could send me alt coins at Don't actually do that. But you could. [Ben laughs]

PF Someone's gonna register now. 

SC I promise you it's already registered. 

BP Please don't send any alt coins. [Sara laughs]

SC I'm Sara Chipps, Director of Community here at Stack Overflow. You can find me @SaraJo on GitHub, and you can send me money at SaraJo.eth, one of those is better than the other. 

BP I have to have to check with legal on this.

PF I am Paul Ford, friend of Stack Overflow. Check out my company Postlight. We are hiring, hiring, hiring. We'd love to talk to you. Think about long term sensible investment vehicles that will be good for your family based on the growth of the economy and that don't have a strong dependency on absolutely arbitrary digital infrastructure.

BP Somehow this has turned into a digital financial advice podcast. I don't know how this happened.

PF This is not Dogecoin, the opposite of Dogecoin.

BP Paul Paper Hands Ford signing off.

PF Yeah, this is not financial advice. I'm just saying. Nice and slow, nice and slow. Have faith, have faith. We're gonna get through this together.