On today's episode we talk about Slack's breakdown of its recent outage, our first impressions of Clubhouse, and ideas for keeping Stack Overflow questions from becoming outdated.
Check out the great post from Laura Nolan, a senior engineer at Slack, breaking down their outage. Paul wants some simple command line utilities for "fix-server" and "boot-it-all-up."
Clubhouse was known early on for being popular with Silicon Valley, but it's increasingly becoming a global phenomenon. You don't have to wait for it to go public to invest, you can buy shares right now in Agora, the Chinese company powering its real time audio chat.
Got ideas for how we can version Q&A on Stack Overflow to ensure questions with accepted answers don't become outdated or obsolete? We're planning to work on this problem, so send suggestions our way.
This week's Lifeboat badge winner is Quinn, who answered the question: How to replace a string in a file using regular expressions?
Paul Ford 'Cause I just want to write command line utilities and be like, 'fix server' like I want, I want there to be a tool called fix-server or like, boot-it-all-up. [Sara chuckles]
Sara Chipps I'm sure there is someone there like 'fix server, fix server'
PF Yeah, run-- [Paul chuckles] "You gotta stop running fix server, man." "No, it worked before!"
[INTRO MUSIC]
Ben Popper Are you ready to start writing your tech story? Join an Ironhack boot camp and learn the skills you need to pursue a meaningful career in tech. Visit ironhack.com/write-your-story to find out more. Let's write your story.
BP Hello! Good morning, Sara. Good morning, Paul.
SC Good morning everyone! How's it going?
PF Good morning. I just shoveled.
SC Oh, you had to shovel? How's the snow? Y'all are deep in some snow.
PF Hopefully this is about halfway. It's probably about 10 inches so far.
BP Pretty serious.
PF It's pretty exciting. I'm very damp right now as I'm talking to you.
BP What's the accumulation in Florida there, Sara? What are you looking at there? Light dusting of sand? [Ben chuckles]
SC Just wind and some rain.
BP A gentle hammock rocking breeze? Sounds tough. [Ben laughs]
SC Yeah. I often think about whenever we say good morning, I think about how people listening to this podcast, it may be morning for them and maybe evening for them. So good whatever you got.
BP Yeah, good whatever you got, we got to bring our lived experience. But if this is how you fall asleep at night, if this relaxes you, then God bless.
PF Well if it is how they're falling asleep, we need to ASMR this up a lot more. This needs to be--[Paul speaks softly] Hey Sara, how's the programming going?
BP No, I think there's like a whole genre of music on Spotify, where you just like put it on when you go to sleep and it plays for 24 hours. You can make money that way because--
PF It's a real earner.
BP Yeah, exactly. But I had a post I wanted to share and get your thoughts. It's about Slack's big outage the day most everybody in the Western world, in the United States, came back to work when they went out. And it was kind of a funny, welcome to 2021 event. But they wrote a nice, lengthy blog post on it. And it made me think of that movie Chernobyl where it's just like, one thing happens. And in response to that thing--
SC Very similar. Very similar event.
PF Same level of crisis. No, I'm really curious to hear Sara's thoughts in this because I think she has more experience than I do. But what struck me reading this so it's Slack's outage on January 4 2021. And first of all, struck me that they opened with a Rainer Maria Rilke quote, "And now we welcome the new year full of things that have never been" So it's just like, okay, Slack, you know, partly I want to give them a medal and partly I'm like, calm down. Many people need to calm down. Um, you know, what struck me reading this is there is a real theme in cloud services based outages. And the theme is as follows. We had a great plan. But that's the theme for everything. [Paul & Sara laugh] But we had a real sense of what we have componentize things. And we've really organized our world so that when we need more services, we can kind of hit a button or the computer will hit the button for us and more services will come online, we need more storage, more EC2 instances more or whatever, until we got everything all trude up so that suddenly there's a need for 700 more machines, not a problem, just going to go rent them out of the cloud. And you know, we're looking at traffic. And if we see our CPU utilization get a little too intense, we're gonna go. And that's where it breaks, right? Because you end up building relatively complex systems that are always reliant on something else. And it's what the something else breaks, and then bottlenecks emerge that you never anticipated. And then Slack is down. Right. And that pattern seems to be really--first of all, what do you think, Sara? Was it--is this just from the cloud era? Is this something we've always lived with? Where do you come in?
SC Good question. I was actually having a great conversation about this this weekend, where so something that has really changed, and I imagine Slack, I'm actually surprised to see Slack suffering from this. I wonder if they made changes over the break. That was a suspicion everyone had this, like, did they make changes over the holidays, and now, things are breaking. But one thing that used to happen when--
PF You know, I worked I worked in [inaudible] and someone came up with the idea of the holiday sprint, which is like the most capitalist thing I've ever heard.
SC That's a nightmare.
PF Like "Are you gonna be around for the holiday sprint?" And people were like, "well I was gonna have Christmas with my family." [Sara laughs]
SC Yeah, I'm doing on a sprint at home.
PF Alright, so sorry, I cut you off.
SC No. So when it used to be you made a website and put it on a server. And then when too many people were trying to do too many things at the same time, you'd add another server, and then you would add another one and then soon you added all the servers in the data center, and then you'd have to onboard another data center. So you had that redundancy. Right?
PF Very few people have this problem. Very, very few.
SC It's a good problem to have. Yeah. But you have to think about that distributed system a lot early, pretty early, right? Because data centers are only so big. So you run into that problem pretty soon. However, one thing that we're experiencing now that everyone's starting their website on AWS and cloud services, is you don't hit that until you are very big. And you have built, you know, like, where, you know, I was talking about a platform that was limited on AWS east, right, because they've had all the servers, they just cannot provision more servers. And that is a problem you would run into pretty early on.
PF Oh, wait, wait, wait, so that you're talking to people at a platform where like, even AWS is, like, we can't help you anymore?
SC Yeah, it's like AWS East is not big enough to give you the services you need, so you're gonna need someone to distribute it.
PF You're gonna need some more clouds.
SC Yeah. And usually, you would, you would face that problem a lot earlier. But now, because this is a rare problem that people are running into, people are scaling up, and they don't need to become distributed over several data centers until they're already very big. And so there's like a lot of debt you have to pay down then. And there's a lot of things that go into designing a platform that is that big.
PF There's a lot of talk in big--first, so everybody has a fantasy about their scale, right? So especially relatively big organizations actually don't have crazy digital scale, but they just want to. And so they're like, well, we need to be multicloud. And it's like, you need to just not have servers in like the basement, like you need to calm down. Very few people have like a million transactions a minute or a second. And so like, but the fantasy is always multicloud. Like that's going to be safety. And increasingly what I see is people who are experts in this is advising you to just calm down, like just go with the cloud in front of you stop trying to architect everything so that it can run everywhere. Because that's going to be there's going to double up all the time that you need in order to build your to build your system. And you should just go with one, until it sounds like you're saying, there is a certain scale where actually no, you need to think about like the whole planet of computing resources. And what's available. And that's of course, Slack would be at that scale, right. Like that's a that's it's a foundational service that runs an enormous, you know, it's probably what the phone system used to be in 1982, it runs on slack today.
SC Yeah. And it seems like here, the problem was the call was coming from the inside of the house, right, the service, they used to provision new space was running on the same servers as the space they were running out of. So everything kind of hit that big bottleneck where they couldn't, their services couldn't move fast enough.
BP It's not like Netflix, like I know, like, for big streaming services, they build, you know, data centers, close to something to lower the latency and actually, like build nodes all over the world next to big customer centres.
PF The original use case, or one of the usuals, it was was financed, they wanted their automatic transaction platform to be literally as close to the trading computers as possible, so that they could shave time off their automatic trades.
SC That's the thing in the beginning, you don't really need to do that, you could do that right, Ben, you could start from the beginning. And to Paul's point, like start from the beginning of and, you know, make sure that you have some setup on US one east and some things in APAC, and all those different areas, making sure you have your system distributed across many, many of the clouds, but it just doesn't make any sense in the beginning.
PF First of all, they're gonna port Slack to Salesforce 360, it's all gonna run and we're not going to even have to chat anymore, we'll just move a little cards around, it'll be--that's coming. You know the thought I have as we're talking, is that the whole point of popular computing, as we all do it including programming is that it gets you away from physics, right? So it's like, oh, there's NAND gates inside your computer, I don't care, what I care is that I can type a little command and then the machine goes boop, like really like that's it for me. And I know that there are physical processes that make up the computer but you actually don't ever do unlike like, I do have one friend who occasionally will be like, you know, he'll get some work and where he'll have to factor in like the rate of rotation of the hard drive in order to him because he's doing like a real time streaming database. But other than that, most of us don't think about moving things or little lights turning on and off. And and that actually, I think what clouds do is they get you further and further away from the actual physics right just like just like programming does, and just like GUIs do, of the computer, the little lights switching on and off until they don't, like it all breaks down at a certain point in pressure and like the CPU just can't handle anymore, can't take it and that used to happen all the time, you turn on your Windows machine, and then it'd be like, I'll be there in a minute and, or, you know, something would crash. The CPU literally could kind of couldn't handle anymore in no way, like all the windows would stop moving. But we've got so many resources that we get to pretend more and more and more that there is no physical reality. And we can actually just work purely with abstractions. And, and so you know, that's what happened to Slack, Slack was like, we got it, we understand our abstractions, we've dealt with just enough physical reality, and then blammo, the network fails, right? And like the little beams of light aren't working anymore. And suddenly you have to care about that. That's hard.
SC Yeah!
BP Sara, in your conversation that you had this person that had a similar problem, or were they were discussing just kind of like, how different scaling has become over time?
SC A little bit about, you know, talking about how different scaling has become over time just talking through scaling issues that they were facing. There's an interesting corollary to this, which was what happened to Robin Hood over last week, they became so popular all of a sudden, and there were so many trades happening, that their counterparty and this is regulation, not technology, but regulation is its own kind of physics came and was like, "Listen, we just need you to show us another extra 3 billion in cash that you're just gonna hold over here. So we know you're good. In case this all goes sideways. Can you do that?" And they were like "What? No, that's 10 times the amount of money we normally keep over here."
PF Money world is so amazing, because they're just like, "Hey, according to our rules, we're gonna need this to be settled on the moon." And you're like, "Okay, but we don't have rocket ships." And it's like "Well, that's not our problem."
SC That seems to be an issue, we'll need those.
PF Better get a, get a space lawyer! No, it's this. It's in a funny way. Conceptually, it's the same thing, right? Like the abstract model in your brain. That is Robinbood's brain was like, Okay, well, there never will be a bottleneck there because we'll never be able to create that kind of network condition until a bunch of shit posters on Wall Street bets show up and create create that situation for them.
BP And then last night, of course, this all was explained if we want to connect outerspace and Robinhood with an interview between Elon Musk and the co founder of Robin Hood on Clubhouse, [Paul sighs] which is modern media. I guess that's modern media.
PF Well, yeah, no, I know. And Andreessen Horowitz is gonna start their own media company, so it's all gonna get better friends, it's all gonna be really cool.
SC Only positive things can happen.
PF Sara, are you in Clubhouse?
SC I joined this weekend actually.
PF Ugh. I can't do it, can't do it. I can't hear nerds talk. [Ben laughs]
SCI know I laugh for a long time. I only really like the sound of my own voice. [Paul laughs]
PF I knew there's an irony that we said that on this Stack Overflow Podcast. But here's the thing. I like hearing people talk about how to do stuff. But every nerd in Silicon Valley is also an economist and I just can't take it anymore.
PF Oh, I know. An economist, medical doctor. They're newly doctors, being a doctor wasn't interesting before. But every signal I get when Clubhouse explodes into the news, is a signal that I'm tired.
SC Yeah, also, this is a controversial take. It wasn't too well received. But it seems to me like Discord for olds, you know?
PF Oh, yeah. No, no, Discord is awesome. Discord is absolutely. And Discord is so not for me, like I get that. To me, like, you know, what, I noticed that there's a Garbage Day is an internet culture summary email that I like to receive. And the writer of that created a Discord community for the fans. And I'm like, this is an interesting combination, where you pay for the newsletter as a free tier and a paid tier. And then you hop on the Discord where the fans are, and you kind of see what they're talking about. And you see them kind of make the sausage a little bit and they're talking about different things that are going to be a newsletter. And I'm not gonna get too involved in that community, but I'm definitely kind of just like, cool. There's a few people I know on there, that's nice. And then, and I'm like, alright, well, this is a great paid tier, right? I got a little community, I get some stuff that not everybody else gets. And I'm supporting a writer who I think is doing really good work. And then you're going to Clubhouse and you're just like, everything I've heard about it is just like, it does sound like that sounds like Discord for olds. Yeah, it is. It's fascinating how quickly it's taken off, though.
BP Yeah, I mean, it does seem like it's a it's a good fit for like the pandemic, right. I mean, it's like, you're coming into a little salon that somebody has curated and you're having a conversation, and there's usually this feel like, Oh, you know, I would I would love to run into this person at a party or something like that. And that's impossible now.
SC I think one nightmare is like, one thing that happened to me this weekend, is you join a room, and then they ask you to be a speaker. And you're just like, ah! I don't want this, like, I just got out of the shower!
PF Chipps, you're literally talking to 1000s of people right now. You might as well chill out.
BP I mean, just it's easier, it's easier when it's not live. You definitely should be able to flip that lurker tab and like people know they can't just yeah, put you on the spot like that. That seems hard. Yeah, I agree.
PF Have you been on Ben? Have you used it?
BP No, no, I haven't. But I'm gonna search around for an invite. Is it still exclusive?
SC It is, I can send you an invite.
BP Alright Sara can send me an invite and I'll hop in there and check it out.
PF Don't send me an invite. [Ben laughs] Just saying like, I'm too--I bought a chair where I read books. Sometimes I read books about computers, my life is getting so much better recently.
BP Nice.
PF The new technology for me for 2021, my big technology prediction is called 'a chair' and use it in a chair to read a book. I'm reading it on a reMarkable 2. I mean, you know, I'm still--
BP Oh, you went there?
SC Ohhhh did you?
BP We talked about this. How's it going?
SC How is it? Should I have gotten one?
PF Okay, so number one, I have a powerful iPad that has a beautiful, responsive, colorful screen that I can zoom in and out of and it has every bit of candy and wonder and the reMarkable 2 does absolutely nothing that the iPad doesn't do. The reMarkable 2 is an epaper device that has good handwriting, nice, responsive handwriting, but it lets you read PDFs well and so on. For some reason, it just has a sweet spot where there's kind of no candy and you just sit there with a lamp and you scribble, I've been handwriting my PowerPoint for presentations. [Paul laughs] And it's just joy. It's just a little thing, it's a little overpriced happiness for, for a word nerd.
BP I like it. There has to be a little analog corner of this show where we talk about disconnecting. I love it.
PF Well, I'm reading, I'm actually reading the LISP 1.5 manual on it.
SC That sounds like a nightmare. [Paul laughs]
PF That's a classic. It's an absolute classic of computer science. And you know, it's amazing. They're just building LISP up. And it's an old PDF that I got from somewhere. I'm writing on it. I'm annotating. I write swear words around LISP. I'm living a good life here. Meanwhile, you're on Clubhouse, talking about how--
BP Also another log on the fire getting in the chair.
PF Yeah, exactly. And, you know, Sara's in a libertarian hellscape talking about--
SC No, actually, I had a very Burning Man situation this weekend where I saw this room, and I thought it was titled, like a randy title that had to do with tech. And I saw some people I knew in there and I was like, Oh, this is like one of those funny like ironic titles for a room. You know, where it's like--
PF Oh that's a danger zone in 2021. You can't count on anything. [Sara laughs]
SC I went in and then I heard the conversation. I was like, ah! Ah!
PF Yeah, no, you can't--Clubhouse like a sexually charged tech term. What are you gonna do there?
SC I thought like people, you know, you got a bar camp. And people have these racy titles for talks.
BP You gotta be careful what tent you look into. [Ben laughs]
PF That is a tricky game, right? Because it's, it's always about to rear its head. And I'm sure Clubhouse is a place where that aspect of our nerd culture can explode everywhere.
BP Sara, I wanted to throw a few things at you from the loop from last week, because I thought they were really interesting. I saw a tweet that was saying, "It's so frustrating on Stack Overflow. When you look up an answer to a question, it's a good answer. But it's from like, three versions ago or five years ago, and it just no longer applies. What are you going to do about that?" And I sent it to you and to Dez, who is on our public team. And she was like, actually, this is something we're going to work on. So I'm pretty excited about that. Tell us a little bit. It's like taking on the issue of versioning or outdated, obsolete answers, right?
SC Yeah, one of the bigger pieces of feedback we get on our site satisfaction survey is around outdated content. Something that happens is people will Google things, and they will find a question that matches their current question. But the answer, the accepted answer is now seven or eight years old, right? So it's from an earlier version of the software or the technology and it's no longer relevant. And but if you scroll down, you'll see the fifth or sixth answer at this point is something from last year, and that's actually more relevant. But sometimes that part doesn't even exist. So figuring out how to make sure answers are current is something that is on our roadmap for this year.
PF I got two fantastic questions from the Stack Exchange network. If you're ready for them.
SC Okay great, I'm excited.
BP Hit us up. Hit us up.
PF Number one, "why can I see my face in a spoon?" [Ben laughs] That is like I got incredibly baked over the weekend. And there's just no answer.
SC Is there an answer?
PF Well, that's from the that's from the physics Stack Exchange, it's tagged optics. If I hold a spoon with a concave side facing me it approximates a concave mirror if my eyes symbolized with an arrow--then its got diagrams. It is a whole thing. It singular ray the only reason I can see my face or is there something I'm missing? And people really went to town. There is a picture of a human face where we see the retina working. Light bouncing off multiple points of this spoon.
BP Yeah, this is cool. There's a very deep answer here from a new contributor with a with one hell an illustration. I hope this is was made just to answer this question.
PF Oh, that's funny because look at that face. The face doesn't have Bézier curve. It just has angles. So that is what I like to see. I like to see that in my computing. Here's the other one, and this is a special one. "How do I help my three year old daughter who is terrified of what I believe is a ghost?" [Ben laughs]
SC I mean, that's, that's a lot. That's a lot.
PF There's a lot. "She mentioned about three times something about a doll with no face nor legs just staring at mommy and daddy while we sleep and doesn't like us." [Ben laughs]
SC Okay, that's a real tough one. It seems like there's a lot of issues going on.
BP How do I stop my three all from freaking me out? [Paul laughs]
PF The best I ever had was my daughter once told me a joke. And she said--I was sitting there looking at my phone. She was sitting across the table. And she went "Daddy," she's about five. "What has a face and hands but no legs and cannot run?" or something like that. And I was like, "I don't know, honey, what?" and she said, "I don't know. But I'll tell you when I'm done eating it." And I couldn't figure out--and I looked up in absolute terror. It was the joke on the popsicle stick. [Paul & Sara laugh]
SC Oh my God, what on Earth?
PF It was, it's a clock. You know, it has a face--
SC Ohhh, okay, yeah, I'll tell you when I'm done eating it. [Paul & Sara laugh]
PF It just, it just I still I think about her face in that moment. Because she was just smiling. And she had like red all over lips 'cause it was a cherry popsicle. I'm like, "Oh, no. Oh, God!" And so children are terrifying.
BP Yes, children are terrifying. Agreed. I want to shout out one other post from Stack Overflow, I think we may have created a new meme, which is always a good thing. This is a deep dive into a security incident we had way back in May of 2019. And we can now reveal a little bit more about it. Some of it is still under investigation or whatever. But the the beautiful reveal at the end is "One does not simply break into Stack Overflow without constantly looking up how to do so on Stack Overflow."
PF Oh no.
SC That was the big thing!
BP Did you hear abou this?
SC Yeah, that was a big thing that that one of the reasons, one of the ways that we were able to identify more information about the intruder was that they kept googling how to keep going. And landing them on Stack Overflow.
PF They used Stack to break into into Stack?
SC Yeah.
PF I mean, honestly, really, there's no avoiding this situation. [Ben laughs] You have created this world.
SC Yeah, turns out.
PF That's amazing. So it's just like, that's, I mean, now you can gamify it right? Like this is now like a bad Ready Player One type thing where, just you know, how am I going to hack into into Joel's mainframe?
BP This is part of our bug bounty program for sure.
PF So where am I going to read about that? When do I get to find out?
BP It's on the blog. I'll put it in the show notes.
PF Oh, it's gonna be in the show notes, everybody!
[MUSIC]
BP Alright, y'all, I'll look up a lifeboat and we'll say our goodbyes?
SC Great!
PF Ben, I can't disagree with that.
BP You can always stop me there and say we have one more thing to talk about.
PF I think you have a fantastic plan for ending this episode. [Sara laughs]
BP Awarded January 28th to Quinn "Replace string in a file using regular expressions" So thank you, Quinn, and we'll share that answer, appreciate the lifeboat, sharing some knowledge saving the answer from the dustbin of history. Alright, everybody, I'm Ben Popper, Director of Content here at Stack Overflow. You can always find me on Twitter @BenPopper, and email us podcast@stackoverflow.com. If you love listening, please do head on over to your podcast platform of choice, leave a rating and a review, it really helps.
PF A good rating, five stars, don't do four stars just to make some point. [Ben laughs] C'mon, get int here. Show them love.
SC I'm Sara Chipps, Director of Community at Stack Overflow. And you can reach me at Sara Jo on GitHub. Probably not Clubhouse.
PF Oh we'll see. Give it time. It's a fast growing thing. My name is Paul Ford. I'm a friend of Stack Overflow. And you could check out my company Postlight and we are hiring, if you know any designers. I mean, if you're an engineer, please, but if you know any designers or product managers, we're also hiring them. We're just just growing, just it's exciting. Growth is exciting.
SC It's always great.
BP Alright. Yeah, I forgot to mention that I guess this week, you can find me on Clubhouse after Sara invites me.
PF Ah, not me. I'm gonna be reading books on my reMarkable about the history of computer science. So we'll see who wins.
SC Wouldn't that be a nice fireside chat with Paul Ford reading software manuals on a reMarkable.
PF In all seriousness, I'm thinking of setting up the camera so that I can do that. "Oh, hi. I didn't see you there." Like I'm just, it's, I really I want the chair--
BP Something I want to fall asleep to.
PF My voice will do it to you. For some reason that's--everyone around me is so tired all the time. I don't know why. Alright, everybody have a good week!
[OUTRO MUSIC]