This week we chat about the massive Solarigate hack, how attitudes towards bullying have changed over the years, and the programming projects we have in mind for the holidays.
There is a nice breakdown of the Solarigate attack here, but the most important thing to know is that just seeing the words BusinessLayer.dll is enough to make our eyes glaze over and our defenses go down.
One interesting second order effect of this intrusion is that it will be difficult to know when all malicious code and access has really been removed. It brought to mind the classic Turing Award Lecture, Reflections on Trusting Trust by Ken Thompson.
If you're trying to entertain kids over the holidays, Ben will be messing around with Roblox, which lets you create your own mini-games and has several hooks to deeper programming capabilities.
Our Lifeboat badge winner this week is Chinito, who answered the question of how you can: Set style using pure JavaScript
Paul Ford At one point, I was on a shared hosting environment and I had an mp3 of like, it was like me pretending to be an old guy talking about chimneys just to be annoying. And I named it Britney Spears Baby One More Time dot mp3 and kind of left it publicly available just to see what would happen.
Sara Chipps What happened?
PF My entire hosting account got wiped out because they found it and we're like, "Oh, yeah, this is copyright violation" and they erased everything and I, I lost all sorts of work.
[INTRO MUSIC]
Ben Popper Couchbase is a modern, multi cloud to edge, SQL friendly, JSON document database for building applications with agility, performance, and scale. If you're new to Couchbase and would like to learn more, the Couchbase Developer Portal is the best place to start loaded with tutorials, videos and documentation, as well as best practice tips, quickstart guides and community resources, including the Couchbase Developer Community Forum. Ready to get started developing on Couchbase? Visit couchbase.com/newtocouchbase.
BP Hello, good morning, everybody. And welcome to the Stack Overflow podcast. How's it going? Hi Paul, Sara.
SC I always wonder like, is it morning everywhere where people are listening to this? If it's not your morning, good whatever.
PF No, it's mandatory.
SC Oh, it has to be morning?
PF Yeah, we don't allow no yeah, we it's a new feature in the podcast player. [Ben chuckles] After 10--
SC It's not local morning.
PF No after 10, this just becomes like becomes Reply All.
SC Reply All! That's exactly what it is. [Paul chuckles]
BP When you start brewing your coffee in the morning, actually, that's how we send our newsletter, it arrives in your inbox, no matter where you are at 10am. The big, big story this week that caused a lot of media and people who don't normally talk about software to talk about software was this solar gate hack. So I saw you know, this was covered everywhere from business to politics, because its impact was so wide. And there was actually a pretty incredibly detailed blog post from Microsoft Security, breaking it down. But Paul, Sara, would love to get your thoughts on this one, it says Where it All Starts Poisoned Code Library, malicious code into solar winds dot Orion core dot business layer dot DLL. Was the door in the supply chain that caused this to spread everywhere? Have you read anything about this that you thought was interesting?
PF I mean, you know, just all you need are those three letters DLL. And you know, you know, that's where the problem starts.
SC Yeah, and if you don't know, so this is something that's fascinating and different, like something that is--when you have compiled code, often you get delivered a file that obfuscates the code inside. This is, I think, a remnant of the early world of software where people were like, if someone can read my code, they're gonna copy it, they're gonna steal my business idea, it's all gonna be over. So a DLL is generally comes along with, you know, Windows programming, dotnet programming as a final release version of a project. And the problem here is for so for all the bonuses of people not being able to see your code, people also cannot see your code. So if there's something bad in there, it's really hard for them to figure it out. And I think, to be fair, it really rarely happens. Because what would need to happen is, you know, say I'm a vendor, and I'm putting together a dll file to deliver someone that works with me or someone that had breached our security would have to insert the malicious code, you know, before we released it, and you and most places have pretty strict release policies. So it's tough to get something in there. But every so often.
PF This I mean, there's a good if you look at the when you put the link in the show notes, there's a nice chart showing what happened--I'm gonna make a loud typing noise from it, hold on. So there is a classic article, or actually, it was a lecture. Okay, so Ken Thompson was one of the progenitors of Unix and C, like he actually invented the B programming language and turned into C and co invented Go like this is as when they say Unix greybeards. They mean him. And he, he won the Turing Award, the Turing prize from the Association for Computing Machinery.
SC Wow!
PF Yeah, it's the big one. It's a good one. And his lecture, if I remember correctly, was, it's called Reflections on Trusting Trust. What it's about is like hacking at the compiler level. So that the hack, the backdoor always stays in, and how hard it is to unlock that. Right?
SC Unlock that meaning--
PF Well, then it's like, okay, we found the hack, we're going to work around it, we're going to recompile this bad boy. And then we are going to be secure, except that the compiler knows that, it knows that you're going to try to eliminate the backdoor. So it reinserts the backdoor.
SC So when it happens at the compiler level, really hard.
PF Yeah, that's right. Yeah, once you get to, and that sort of, there's a little of this going on here where they like, and this is where the state level actor part is really interesting. Because, you know, power in our industry comes from being able to go down the stack, like if you can ship a microprocessor like Apple just did, then you are a superpower in technology, right? And so like, if you can hack a microprocessor, which is what happens when you embed Unix in the firmware on your Intel devices, or whatever, then you have control over the ecosystem and so like state level folks--folks just guys hanging out. They get really excited about going further down the stack like like, you know, script kiddies, like cross site scripting stuff isn't very exciting to Russia. I mean, they'll do it, you know, you got to stay busy, you know, justify your budget. So this is like, you know, this isn't quite at the level of the processor, but boy is it it's pretty deep down, right? Like we're gonna get there inside of this DLL and it's gonna be really hard to get this thing out of there. It's an infection.
BP There was some stuff in here that, yeah, Paul, I thought was a little bit strange when they were talking about how they knew it was a state level actor. And you know, the fact that they were so sophisticated. They said that they obfuscated a lot of things, for example, not using things like keylogger and backdoor in the name of the malicious code, which to me seemed kind of obvious. Is that a pro move? [Sara laughs]
SC I like one thing I like to do is just name my code backdoor anyway. To like, freak people.
PF Yeah. Oh, you know, no, I did this once to myself. I was joking. And I named my own computer like, I named it a hash, and then Bitcoin miner on the on the lan at work, just to be silly. And then like, four months later, I was I was like, "Oh, my god I've been hacked" [Sara & Ben laugh]
BP That's right. The call's coming from inside the house.
PF Pranksters get played themselves. That's what I learned.
BP There was an interesting, like, sort of Mad Libs that they did, like you were saying to obfuscate stuff, they would generate these sort of like random strings. So they would have these are the components that could either be solar winds, wireless widgets, NPM, Apollo cloud monitoring, interface volumes and components, those people just like glaze right over. They've seen him a million times.
SC Yeah, that's true.
PF Look, I mean, I think when you what's what is state actor here, it just, I think points to like a team, right? It points to a dozen people worked on this for six months, as opposed to somebody named Cyber Lord. [Ben laughs]
SC Yeah, wearing a leather coat. A long leather coat.
BP Yeah. They were super patient. They got in. They took their time. They didn't take anything right away. And like you said, but I did hear a couple of interesting discussions, basically saying, like, to try to unwind this is next to impossible. like you'd have to, you got to scrap the whole thing and start over. Like, it's just--
PF That's right. I mean, that's true with the internet, that's true with computing.
SC If this is correct, I mean, like, this looks like something that like you'd skim over and then you just be mad at whoever wrote it, because it's just a, it's just like a try catch. There's nothing in the catch. So you know that maybe it's possibly that they did that out of ignorance. But what's more likely, is they did that to pass as code that actually went in to the [Paul laughs] one into just, every time I see that, I'm like, what lazy person put on this try catch with no catch.
PF Just catch that exception with no output. No, no response. Just like catch it.
SC If you read it, I mean, it's just says things like Orion improvement business layer. Yeah, exactly. I'm gonna yawn right through that.
PF Oh, yeah, these people are pretty good. I mean, and I think, you know, they did--that's the thing. The reason they know it's a state level actor is that it's bad enterprise code. [Ben laughs] Anyway, and the other I think the other reason you know, it's a state level actors, it doesn't the way to really do this, I think, is to sneak something in that's, that looks like it's just you trying to grab, you know, bitcoins, but actually sort of like one level down, bury the voting machine hack, or whatever you want to do, so that they don't even know why you're doing, right. Like, this is very targeted, as opposed to like, "I want Bitcoins" and so are you know, let me I'm gonna send email from your account.
SC Yeah, exactly. I'm gonna, I'm gonna--what is it when they take over your machine? Alright. Well, now that's not funny.
BP Well, I thought it was pretty fascinating. We'll include the link in the show notes, but it's definitely going to be interesting to watch as they try to unwind this because it's, they're deep deep in there now. They're looking at all our plans for nuclear weapons, all our Treasury accounts, so it's gonna be fun.
SC Our roadmap for q2.
BP Yup.
PFb Yep. That's right. [Paul laughs] Ah, we laugh and then one day you look, you know, and then one day, it's just like, really bright out the window. And you're like, "Uh oh" Yeah, it's bad. Everything's hacked everything. And when it's not like, it really kind of sucks because like, well, maybe the NSA is doing it. Or maybe Google's just looking at your stuff, or maybe Facebook's going to help itself. Or maybe it'll be the Russians like--
SC Or much worse.
PF Yeah. I mean, we're just all kind of vulnerable sacks of plump gristle at this point.
BP Oh god!
SC I'm gonna try to try to insert that into our holiday cards.
PF Let's do it. You know, bring the little birdies.
BP Wishing you well. But yeah, no, actually, there was the last little bit of this was, they were talking about why nobody has sort of, like, stop this from happening even though these cyber attacks have been going on forever. And it's because nobody wants to get into a hot war, like a real war with sending the missiles and the planes and the dying. And so like, you can kind of just keep pushing the cyber as far as it can go. Because nobody really wants to retaliate in a way, as you said, like that's meaningful, like, look out the window, and all of a sudden, you know, there's mushroom clouds. And so we kind of, we're just gonna keep playing these games forever. Like, until the day when somebody responds to a cyber attack with actual force. There's no deterrent to prevent people from just going as far as they can with this stuff,
PF I don't know if the Internet has taught us anything, is that cybering gets real, real fast. Yeah, somebody gets in there. And sees those chat logs, and now it's a divorce. So, yeah.
SC Speaking of which though, not to, not to hijack. But I've been, I've been learning so much recently, because I've been building--so we're doing moderator training for the new year. And we've been wrapping up our DNI modules. And it's very real, when you talk about the stuff that happens in cyber, you know, has a lot of real world implications. I've been reading all about so many studies that get done about cyber bullying, or online, hate speech, microaggressions. Where the studies show that people what you experienced online, long story short, TLDR, what you experienced online really affects your life, you know, from depression, to suicide levels, to different things like that. So I mean, it kind of it kind of relates to this, as we see a lot of attacks that can be targeting a group of people online or things like that. And I think those real life implications are things that people think about.
PF I feel like, I went to, my son did Taekwondo in the snow this weekend. So I took him.
SC Cool!
PF Yeah, we're at the Taekwondo class. It's at a park. It's socially distanced. And I'm watching and my son's on the he's like one of the younger kids like maybe 10 or 11 year olds are there and he's nine. And they all were just like, "Alright, man, great job!" And like, he was having a hard time with some of the moves and stuff in there. And all the kids were boosting him. And I just remember the children that I grew up with, for whatever reason, but the culture around like kids, like kids are really trained to be respectful and nice to each other in a way that I wasn't like, the children I grew up with were like "You suck." [Sara laughs]
SC Worst thing I've ever seen.
PF Yeah, like "You're garbage, don't even" and I'm just like, "Oh my God, if I'd had this, I could've like cut five years of grumpiness out of my life."
BP Yeah, no, no kids are kids are given a lot more anti bullying, training. And grownups are now all just bullies online anonymously. There's been a bit of a paradigm shift. [Paul & Ben chuckle]
PF You know, it's weird. There's this like, really bad meme, which is just, it's just like a garbage image of a woman sitting in front of the computer. And the line is "Not now honey, mom is cyber bullying the mayor." And it's one of those things where you see it, and you're like, I understand exactly what's happening here. And it shouldn't make any sense. Like we should be not here, right now.
BP I mean, I do think that it's interesting what you're saying about how like, yeah, maybe we're training kids in a different way. When I when I was growing up, I had the same experience as you like, bullying was like, it's a part of life. Like in every movie, there's a bully, and it's just something we all live with. It's like a that's life.
PF Yeah, they started, they started early. I mean, at the school, you know, we're in a liberal part of Brooklyn, but it's still it's a public school with all kinds of folks in it. And there are gender related stuff, DnI and race related stuff. Like it's just a part of the curriculum in a way that it certainly wasn't for me. And like there is this element of like, "Don't bully" you have to boost or sign saying "Don't bully" and I agree with you. When I was a kid it was just like, "You're gonna have to learn to get punched in the face sometimes, boy" You know, like, "But you're my guidance counselor!" [Ben & Sara laugh]
SC Do you think that's everywhere? Do you think that's like a Brooklyn thin? I always think about that. It's like Brooklyn tends to be--
PF Yes. Yes. Every joke applies at some level. But yeah, at the same time, though, I think that nationally, there is a focus on on making kids a little more emotionally safe and healthy than certainly there was for us, my lord.
SC That's great.
BP But Sara, you went you were homeschooled or no, what was your--I forget? You told us.
SC Yeah, I was homeschooled until high school.
PF That's the worst kind of bullying. Right?
SC Exactly! I skipped, what it was, I saved up all the bullying till high school. And then they fit it into those four years. It was really nice. [Sara laughs]
PF It's like "Hi, I don't know anybody. But I like computers." [Sara & Ben laugh]
SC Yeah!
PF Those first two years were grim. Yeah.
SC I was just thinking about this yesterday. It wasn't really a bully experience. But it was more like, oh, I'm different. I was at a slumber party with a bunch of girls. And for some reason, the question came up of like, you know, if you had one wish, would you rather be smarter or more beautiful and someone like asked me the question I looked at everyone like they were an idiot. I was like, what kind of question is this? It's like of course I would be smarter and everyone looked at me like I just like dropped a nuke. [Sara laughs] They were like "What on earth is wrong with this girl?!"
PF "Does she know where power comes from?" [Sara laughs]
SC Yeah, no, exactly.
BP Alright y'all, let's chat a little bit about what we're hoping to do over the holidays or going into 2021. My adventures in programming are gonna be game design related. My kids are super into Roblox and they have like a game studio, you can build your own game. And so we made a deal that they could they could do Roblox if they built one of their own games. So it's kind of cool. You can get it through them. And they've got API's and you can even build a little, little monetization engine. So maybe we'll make a few bucks on a new Roblox game. [Paul chuckles]
PF And then Ben doesn't come back to the podcast. He's like, "Listen, something happened, we hit it big"
BP Words on friends. I may not be coming back.
PF What about you, Sara, do you have any holiday projects?
SC So I have two. So one is my personal website, you know, the shoemaker shoes, I really got an attack it, I really got attack it. And I want to get I'm going to find an illustrator. I want to try to find Illustrator. Do some cool illustrations. And then I want to do my GitHub Read ME. This is gonna turn into a whole thing. It's already, I've already got a ton of bloat. There's too much bloat in this project already. But I want to I want to start doing some more CSS that I don't hate, which means that I'm doing Flexbox. Flexbox is what I have found to make CSS less awful.
PF Oh, yes. Flexbox is good. It is good.
SC Yeah, though, it is still tables. And we can all agree to disagree.
PF Yeah, it's a, it's an API. It's an abstraction over tables. Cool. So what is the URL? Like is it SaraChipps dot?
SC SaraChipps.com or it might be SaraJChipps.com. I remember something--
PF SaraJ.horse? [Sara laughs]
BP First up, remember the URL.
SC Yeah, SaraChipps.com. Yeah, it's like really old. It's like really old. And I haven't updated it in forever. I want to do that. And then. So there's that. And then we started doing these online workshops, a Jewelbots, which has been really fun. And so I'm building out some more curriculum, because helping kids code online is hard. Because it turns out computer like getting people's computers remotely all working in a way that can build Arduino is tough. But I think after a bunch of iterating, we're almost there. So I'm going to work on some curriculum for that, too.
BP Neat. So this is like modules and then people do it offline? Or when you say it's hard to do it online. What do you mean?
SC No, I mean, I'll getting on a Zoom call with our little Arduinos. And making some cool wearable art. We started working on those this this month. And that was really fun. Want to keep making that easier for people to join it.
BP Very cool. So when you're in a class like that, how many people usually show up? Are you the only instructor or you have other teachers with you?
SC So something that's important I've learned in programming classes is you want to have teachers assistance always because, and when someone gets stuck, they really need one on one attention. And especially when you're doing like a Zoom call, or something like that, that can mean to the whole class needs to stop. And that person can feel kind of self conscious. So like, Zoom has those Breakout Room options. So having being like, "Okay, why don't you jump into breakout room with this person? They can walk you through it." That makes it a lot easier. Yeah. So usually it's like 20, 30 kids or something like that. And it's really fun. We all talk about our favorite animals. [Ben laughs]
PF What's your favorite animal?
SC A pig. I love pigs.
PF Pigs are great.
BP Paul, what about you?
SC What about you, Paul? What are you gonna work on?
PF It's a good question. I actually, the same idea occurred to me, which is that like, my personal web page needs to actually do something because I make no sense. It's like you find a blog and five articles and my company when you Google me and LinkedIn, just people are like, what? And so a good decent representation of past work, which I have a lot, you know, I have like a spreadsheet going with things I've written and so on. So I'm thinking maybe put that online. Maybe I'll just redirect to the Google Doc, just be like "Here. the here's the spreadsheet."
SC That's great.
PF No, and then I started reading, frankly, I'm just going to use it to kind of organize my thoughts. There's two tech things I'll do which will be relevant to the podcast. One is, I started reading the 1960s era Lisp 1.5 manual, which is actually just one of those classic texts, where they're inventing everything about technology from first principles. And it's great, like I got about 15 pages in I'm like, I have to read the rest of this. This is really interesting. And so that's been fun, just to sort of like, see how the, you know, the roots of tech and how it kind of translates to really low level stuff. And then the other thing is sort of from the totally from the other direction, I want to play a little bit more like hours, not days with API's that talk directly to the database that, you know, there's there is data set with SQL lite, and then there's postgraph file with Postgres. And I want to build at least like a two hour toy app in, especially in Postgres file and play with it. We've been doing some, some work with it at work and I want to kind of get that repo and, and fool around.
SC Oh, interesting. Is that related to graph qL? Like, are there things popping up that are similar or these things have been around for a bit?
PF No, they've been around for like a bit, but just a bit. And they're, we've been using them a ton. So basically, what you do is you define your Postgres database, you get your schema all trude up, you write custom functions if you want things like search. And so when you write them in PL SQL, and then you point this thing at them postgraph file, like literally command line, and it sets up a graph qL interface to that database that is as good as a graph qL interface you could create, like, it knows all the stuff, the foreign keys become obvious relationships. And so that graph IQL interface, is just sitting there, and suddenly you can you're off to the races, programming a front end without any middleware. So I love that.
SC That's amazing. This problem existed for a while, and it's so nice to see people doing some stuff with it.
PF You just don't actually need--so we're I mean, you still need some middleware, right? Like you need, like auth is still complicated. It's, there are solutions for doing it in the database. But it's not trivial. And it's hard to wrap your head around, like going from a classic web auth based model to row level security in Postgres is kind of like what? What's happening? If you ever mess with JWTs JavaScript web tokens?
SC Yes, yes, I have, yes.
PF I find them, I find them really difficult.
SC Right. And they shouldn't be so difficult, right? Because the whole idea was for it to be less difficult.
PF There's like a JavaScript 2.5 moment where everything got a little too complicated. It was like, we're gonna have 35 build systems, and I kind of lumped them in there. And the simplifying engine has not showed up yet to be like, "don't use JWTs, use--" and I don't really know what the alternative is. So anyway, JWTs, they suck. But you know, maybe they're good. I don't know, they suck.
BP One last thing that I was gonna think about doing. And I'm actually having an interview on Wednesday is like trying to bring us a little bit in touch with the life sciences just because so many people are thinking these days about what's going on with the pandemic, and what's going on with the vaccine and stuff. So there was a really cool story about a piece of open source software built by researchers at John Hopkins, that lets you shrink the number of days for DNA sequencing. So from the 15 day operation down to three or even one. And it's got this amazing thing. It's called a portable nanopore sequencer, and it plugs into your USB drive. And then you just, like, put some stuff in there, and it'll tell you what the genetic makeup is.
PF Oh, hell, yeah. Although, you know, the number one use here is dating apps.
BP Yes. You go on your first date, you both spit into a cup, and then tells you if you're compatible long term.
PF Ohhhh, that's gonna happen. [Ben laughs]
BP But yeah, I just think it's so wild that some of this stuff, which seems like it would normally be sort of sequestered away in a lab is now open source and available with like, you know, fairly cheap peripheral that can plug into any laptop. So we'll see how that goes.
[MUSIC]
BP I'm gonna read us a quick lifeboat, and then we will say our goodbyes. Today's lifeboat goes to Chinito, awarded December 11. Set style using pure JavaScript. Okay, it's got to be pure, unadulterated.
SC Yeah. Yeah.
BP Alright. Cool. Alright, everybody.
PF Good lifeboatin'.
BP Good lifeboat lifeboatin'. I'm Ben Popper, Director of Content here at Stack Overflow. You can find me on Twitter @BenPopper. You can always email us podcast@stackoverflow.com. Have a safe and happy holidays and we'll see you in the new year.
PF Love you, love your show! Sorry, go.
SC Yeah. [Sara chuckles] I'm Sara Chipps, Director of Community here at Stack Overflow. And you cannot find me for the next two weeks.
PF That's great. I'm Paul Ford, friend of Stack Overflow. Check out my company Postlight, I'm gonna disappear as well, but in a different state.
SC Wow!
BP Very cool. Alright, let's count it down. Three, two, one.
PF Hit it!
[OUTRO MUSIC]