On today’s episode, we chat with a listener, Geshan Manandhar, who has been working in the world of software engineering for two decades. He started programming in Katmandu during the days of dial up internet. Since then he’s worked across three continents, and today is a senior software engineer at Simply Wall Street. He gives his advice on how developers can change with the times and what it’s like to move into the era of serverless containers.
You can learn more about Geshan on his website or check him out on LinkedIn.
Geshan also shared the slide decks for a few of his talks on serverless and containers.
Congrats to Stack Overflow user Matthew Reed for earning a populist badge with his answer to the question: GitHub: How to do case sensitive search for the code in repository?
[intro music plays]
Ryan Donovan Welcome to the Stack Overflow Podcast, a place to talk all things software and technology. I'm Ryan Donovan, I edit the blog here at Stack Overflow, and today we're going to be talking all about the back end, the infrastructure, what powers it, how it's so complex, and how we manage it. My guest today is a listener who wrote in and said, “Hey, I would like to talk about the back end,” Geshan Manandhar, a Senior Staff Engineer at Simply Wall St. So welcome to the program, Geshan.
Geshan Manandhar Thank you, Ryan. It's great to be here.
RD Well, Geshan, at the top of the show, we like to ask our guests how you got into software and technology. What's your origin story?
GM Sure. I am a Senior Software Engineer at Simply Wall St. How it all started I think was around 2000/2001, back in Kathmandu in the primitive days of dialup internet. People didn't used to have email addresses, but then I used to have websites. So started around that with HTML, CSS, and JavaScript of that era. Basically, it progressed with the formal studies to doing C, C++, Java and these things academically and then I moved into a lot of PHP, Node, and JavaScript when I started working formally professionally.
RD And you have moved countries for jobs, is that correct?
GM Yes. So I have been working for 15+ years now across three continents. I'm originally from Nepal, so I worked in Nepal for a couple of years. Then I was in the Netherlands for a year. I didn't really work there, but I had a small internship as part of my master's. I was in Dubai for six and a half years, and I've been in Australia now for almost six years.
RD Okay, that's impressive. So let's talk back end. You've obviously worked back end for a while, and I remember the first time I heard about a back end language was PHP and you would write in your JavaScript, write in your HTML. Obviously that's not how we do things right now. What's the shift been like to watch and what is the sort of primary back end tool right now?
GM So to kind of summarize it, I started working around 2007-08, and at that time there was no concept of this back end and front end and then front end APIs calling back ends. PHP was the thing most people used and I also used it and we had everything inside the same PHP file or the same template that we used to call. So that's how things started and I guess around 2013-14 the idea of SOA and microservices and splitting stuff into back end and front end started coming in and that's where I think a clear demarcation of, “Okay, back end engineers, you do back end languages, databases, data stores and stuff, and then front end people, you guys do JavaScript and then you have many frameworks like Angular, React and stuff, and then you create a bridge between them, which is an API, which is generally a JSON API over REST,” and then that's how things kind of started. For me, I did PHP probably from 2008 to 2014-15 across two countries, and I think towards 2015-16, as I said, the idea of bringing stuff, microservices happened, and then I gradually moved into writing JavaScript at the back end, which is Vanilla Node to start with. So I think 2018-19, then I started doing more of the TypeScript thing with NestJS. And for the past three companies which have been scale ups and startups, I've been doing Node. But then the common thing with all of them is that all of them have their older apps in PHP and everything that's written new is basically written in Nest and TypeScript and Node.
RD I've seen some modern back ends just operating as full old school software– Java, Go, Scala, whatever– running on a server somewhere, talking to whatever front end code. Why do you think that shift happened from having PHP where it's sort of part of what you write on the front end to having a pretty elaborate system in the back end?
GM I think the main shift is how interactive you want your application to be and what things do you want to change when users interact. For example, for the things that you mentioned, I guess Rails is also a full stack app. You just have everything in Ruby and the whole template is just painted and served by the server. Unless you add extra JavaScript to it, there isn't by default any extra JavaScript. But then once you started having interactivity and once you need some parts of the page to be updated differently than the other parts of the page, that's where I think this extra interactivity and this need of either actually being faster or seeming to be faster for the user, all of these things came into play. And then that's where the split really started to happen where back end engineers would give you JSON and then front end engineers would consume JSON in not the language, but the framework of their choice.
RD I remember one of my first forays in JavaScript was just hiding and exposing pieces of a page, and if you serve that whole page all at once, that is just a bear. That's just a lot of code to dump on somebody's system. Do you think any of the heavier frameworks have contributed to this or are they also trying to solve the same problem?
GM I think if we oversimplify the problem, it's basically adding and removing elements from a page. That's what all of the JavaScript frameworks are doing. And even in the early-2000s, I think we used to call it AJAX or something of that sort, and then we passed XML and then get XML back, which also kind of did the same thing. You just want to update one part of the page or basically add or remove elements out of the page. And then jQuery was almost synonymous to JavaScript at one point, and I think towards the beginning of 2010-12, Angular was something that came into the picture, which was dubbed the jQuery killer at that point. And then probably a couple of years later, 2013-14, Angular also lost its light by React taking over the spotlight. So as I said, it's basically trying to add and remove elements from a page and all of them, or most of them, do the same thing. Some of the frameworks have more of a JavaScript deal to where they say, “Write everything in JavaScript and then we'll do it.” Some of them try to do two-way data binding, for example. Probably Angular still does that. For example, flavors of coffee, but everything is still coffee, and then everything is still trying to remove and add elements on the fly without trying to refresh the page, probably get smaller stuff from the server, hopefully in JSON, and then repaint it. That's what I believe most of the JavaScript frameworks do at the core.
RD That's an interesting way to simplify it. But I think the back end stuff, the responsibilities of the back end engineer are broadening to include the infrastructure it runs on. The infrastructure is getting so complicated that you have to think about where your back end pieces are located on some worldwide data center system. Do you think that's a good thing, having to think about the infrastructure, or is this just a pain?
GM I think it is a good thing to have, because if you think of how you learn and how you do things, learning things adjacent to your main depth is I think a very good thing. By that, I mean that if you are a back end engineer and you want to know about infrastructure, good thing. If you are a front end engineer and you want to know more about web performance for example, very good thing. So things like that I think kind of go hand in hand so it's good that you learn, but then I think there is also an evolution of things. For example, early-2000s or mid-2000s, there were actual roles like database administrators which probably now have gone only to the big three clouds who actually manage all of our databases these days. So with that there is a level of specialization as well as there is some level of generalization that you still need to do. As a back end engineer, I think it's a very good idea to know, learn, and understand what's happening in the background, for example. So basically the whole DevOps movement is both back end and ops teams working together. There is no throwing stuff over the wall, and then you focus on testing, deliverability, velocity, some of the shift left things, and then you have things like logging and monitoring to support that, and plus there is a very good focus on communication and collaboration. That means that if you are a software engineer and your code eventually runs on Kubernetes and you don't know anything about Kubernetes and then there is a bug, then it will be a bit difficult for you to try to figure out what's happening. Whereas even if you know some of the Kubernetes, some of the terms, some of how it works, why is it declarative, I think it will be a lot easier whenever there is an issue or anything that you want to debug to go in and then figure things out. Even if you don't have full access to Kubernetes, but at least you can kind of guess, “Okay, this is happening here,” because at times debugging is a lot of guessing and removing other things that cannot happen. So I think that's a good thing to learn.
RD It's harder to set those debug breakpoints when the software is just distributed everywhere.
GM Exactly.
RD So does that mean you support back end engineers being on the incident response on the on-call shifts?
GM That's a good one. So I have had multiple experiences of this. When I was back in Dubai at Namshi, we used to have an on-call roster and I was there for at least a couple of times, if I remember, because when I left there, we were just starting it. The philosophy behind that, I think, is you build it, you run it, so you should be on-call for your software. You should be responsible. It's not like you just made it and you threw it over the wall to the QA to test it. I don't suppose it should or would work like that. It’s your responsibility as a software engineer/ senior software engineer that you go in and you test and you verify that it's fine on production. And that also means that if there is something broken at 11 PM or 2 AM, sometimes you might be called. That's why you need to think of making it more resilient, because software is going to fail. Networks also– every request that you make does not pass always, so that means you need to think of most of the what if conditions on what if this fails here, what if this happens, so that you have a good resiliency. As well as with things like microservices where things have been kind of separate. So you being responsible for three or four of your microservices I think is very logical. That doesn't mean that you have to be on-call every other week, but depending on the size of your team, once a quarter or something like that, I think that's a very good learning. And also, while being on call, you learn about things from other teams, because if that breaks, that also at first comes to you because you are the first contact point. I think it's a balance. It's how you want to set up the on-call thing, because generally it's not only one person on-call that is the primary person, and then if that person is not responding, a secondary person, and then an escalation ladder for that as well. So it's a good thing to balance coming back to the point. I think you build it, you run it should be implemented to a point where back end engineers should generally be responsible for the software or the features they ship.
RD That is still an open question. We've had folks writing about how to set up DevOps SRE teams, and there's a lot of different topologies for how those teams can operate. Do you think that the distribution and complexity of the back end makes incident reporting and incident response harder or does it make the systems more resilient, or is it both?
GM I think back end systems, with them being distributed, is generally harder. As I pointed out, depending upon how you have set up your logging, your monitoring, your observability things, I think those are the things that basically are a light in the dark room and then it's up to you to where you want to point it and then fire and figure your things out. Generally, the issue is not solving the event or the problem, it's basically finding it out first. And most issues usually stem from someone having deployed and probably not tested or could not test or was a time sensitive thing that they couldn't follow up. That's why you don't do Friday deploys. We have recently seen the CrowdStrike thing already.
RD I know. I had three extra hours in an airport because of that.
GM Exactly. So that's the thing. It's about testing, delivery, releases. You should have the ability to do feature flags where you can be safe to just release it, I don't know, to the five people in my team. So I think those things are more important. The precaution is more important than the solution. As I said, things will fail, but then at times it might be as easy as, “Okay, someone deployed this yesterday at 4 and probably didn't find the right time to test it or thought that it would not cause anything and that's why we're having this today at 1 PM,” or something like that. So those kinds of things are easier depending upon what the release is. You could roll back, things will go back to normal, and then you can have the discussion, then you can have the postmortem of why did this happen. Of course, postmortems need to be blameless. It's always a process, never the person. So if you follow these kinds of things, I think it is complicated, but then there are things and steps to make it at least one or two steps easier. As I said, figuring out what the problem is is the bigger problem, whereas solving it might be just rolling back, or if it's a small fix and your application can or has the appetite of that downtime, just write the fix and put it on. But generally whenever something is paged at 2 AM, I don't suppose people will start writing code. They would generally restart some pods or roll back a fix.
RD You talked about how you're writing a lot of stuff for Node.js. With all this sort of overall complexity in the back end systems, the cloud microservices, is it easier to write that small piece for Node or do you have to think about the entire system when you write the one script, or you get to just write the one thing at a time?
GM I think it's a question about two things. One is how big and how small it is, which would generally depend upon the system that you're working on. So if it's a small microservice that only touches, for example, shipping carriers, then you know your blast radius. Even if you mess up stuff, you are only going to stay there. But if it's a big monolith where there are thousands of files, and probably there are cross dependencies that you don't really know of, you might have changed something and thought that it would do X, whereas another dependency might be using it to do Y. Those are the difficult scenarios where things break. In terms of Node or not, I think use the right language or right tool or framework for the task you have. The companies that I worked with started moving to Node because it was, one, getting popular, two, the mythical full stack engineer was also invoked that there is no JavaScript in the front end and then JavaScript in the back end, and probably three, it was easier to find people as well because PHP was also losing some of its popularity and Node was rising up. So I guess it's a mix of multiple things, but as I said, it's the tool for the job. If you have a very latency-sensitive thing, maybe you'll just write it in Go or Rust these days. Whereas for web applications that are used by thousands of people daily, I think Node would still serve the purpose. So I would look at it from that point of view.
RD I think Node and Deno, we talked to Ryan Dahl from Deno the other day and it’s all written in Rust. So I think there is a focus on making the thing that needs to be fast, fast, but the script itself is not spending that much time on the Node server. So you mentioned earlier that a lot of these big systems run on infrastructure as code –Kubernetes and all that– and I've read things about people complaining about complexity, how it's resume-driven development and all that. Do you think Kubernetes is a net positive for software development or infrastructure as code?
GM Let's first separate infrastructure as code and Kubernetes. Infrastructure as code is basically writing code that you execute and that would create your infrastructure in a nutshell, whereas Kubernetes is basically a very complicated system that came out of something called Borg. Borg is the thing that's used in Google, and Kubernetes is an open source version which orchestrates containers, and containers these days are generally Docker containers. There can be others, depending upon what complies with the standards, but are generally Docker containers. And as I said, it comes out of Google. It actually tries to be at Google scale which is billions of users, and that's where all of this complexity, I think, inherently comes from. So as soon as you talk about Kubernetes, if you're a startup, ask yourself the question probably two or three times before actually jumping into containers. There are and there will be multiple ways that you could run your systems without Kubernetes. I guess if you want to run Kubernetes, that means you have at least a one or two-person DevOps platform, SRE, whatever team you want to call it. If you have that, then probably you can try to go into it, or else Kubernetes is so complicated that sometimes your application would get lost in the layers of complexity that it has. It is there because it's highly flexible, configurable, extendable, and with all of this, there comes a high degree of complexity. So if you are at that scale, if you have thousands or millions of users per day, Kubernetes would be the thing to go with. If you have less than a hundred users a day, probably not.
RD Just spawn your own virtual machines.
GM Yeah. That would work fine for a hundred or less people a day.
RD So I think one of the newer buzzwords in infrastructure back end is ‘serverless.’ We've talked before about serverless containers. Do you think that's the way that infrastructure is going to move, that the backend is going to move?
GM So on that, I missed it in my intro, I am also a Google Developer Expert for Google Cloud Platform, and then I have used some of their serverless and serverless container offerings. Now coming back to it, let's divide it into two things. The first thing is serverless, and then let's go into serverless containers. So serverles I think stands on two or three main pillars. One is that you don't really manage any of your infrastructure. It's highly available and highly scalable out of the box. And three, you pay by the usage. By that I mean that if your website or your application doesn't get any traffic for a week, you pay zero. So that's basically serverless which has been much popularized with function as a service and that's kind of synonymous with serviceless but I guess a lot of other things can also be taken as serverless. For example, S3 could also qualify as serverless if I look into it from these three pillars, three points. So that's mainly serverless and these days, as I said, function as a service, which was first by Lambda and then Google has Google Cloud Functions. Azure also has Azure Functions or something like that. I think that's where the buzz of serverless came in. And the other part of serverless is being event-driven. For example, I upload an image on S3 and then one of the Lambda functions resizes it to a smaller dimension so that it can be used on my website. So that is another good example of event-driven where the event was actually uploading a file to a bucket. So those things are popularized as serverless. Now, coming into serverless containers, as I said, containers basically are synonymous to Docker these days. So think of containers in a nutshell. It's wrapping all of your dependencies, so your OS, your runtime language, in its correct version, all of your dependencies, because most of the apps will surely have dependencies, some other open source code being pulled in, and then your code is just packaged into one box, one block, and then you start moving it. There are many advantages, and one of them being that you highly minimize ‘it works on my machine’ problems because you are shipping the same container, the same artifact. If it works on your machine, it works on staging, it works in production. So exactly the same piece and saying ‘almost exactly’ because sometimes things might not work. So that's serverless and containers. Now if you combine them together, then you come into serverless containers or container as a service. So now as I've mentioned both, the reasons to use serverless containers would basically be that you have the flexibility and ease of serverless and the control of containers. For example, you want to run the latest version of Rust, but then none of the function as a service provides you. They have one or two versions earlier because they can't really catch up to the latest with serverless containers because you are using Docker containers. You just flip the number of the version, you build a container, and then you can run it. So you have those advantages. And with serverless containers, container as a service, I am more used to Google Cloud Run. So with Google Cloud Run, what you can do is you basically need to configure four things: CPU, memory, minimum instances, maximum instances. That's it. And then Google basically takes care of the scaling of all of your containers every five seconds. And if you put your minimum instances to be zero, that's serverless. So as I said, if your app doesn't get any traffic for a week, you pay nothing. So those are the advantages of serverless, then going into containers and then serverless containers coming together.
RD So what's the difference between serverless containers and basically a fully-managed Kubernetes platform?
GM So if you want a lot of flexibility, probably you'll need to go into Kubernetes. But for most of the apps that we do, most of the web apps that we do, I think serverless containers would be fine. Because in this day and age, most of the apps would by default be containerized. Now, it's the question of where do you want to run it. So if you don't have a two or three-person platforms team and you want to run it on Kubernetes, then that's a different headache that you want to take or not. And that's the thing that you just don't have anything to manage. The only thing is the four things that you're saying: per service, minimum instances, maximum instances, this amount of CPU, and this amount of memory, and then Google scales it. So even what you could do is you could say, “I want one minimum instance, so that means no cold start, and a thousand maximum instances, that means even a million people if they come into my website.” Hypothetically my page hits the first page of Hacker News or something and then 10,000 people land on my app one day, it will scale because that's how it is built. It is built for being highly available and highly scalable where you don't even have one DevOps engineer in your team. For example, I am quoting Nick from Halfbrick Studios. Halfbrick Studios makes Fruit Ninja, probably most of us have played it, and what he said was that Cloud Run eliminated the need to add an extra DevOps engineer. So it's that flexible and that good. And I guess with the AI era coming in, I think you could use things like Cloud Run Jobs to train your models and stuff. So by default, Cloud Run has the web service where you can run your web apps, and they have added the Jobs where you can run long-running jobs like cron jobs, adding data, and all of it being serverless containers is basically you don't manage anything. It's just offloading that to Google.
[music plays]
RD All right, everybody. It's that time of the show again where we shout out somebody who came on to Stack Overflow and dropped some important knowledge. Today, we're going to shout out somebody who won a Populous Badge– somebody who got the highest scoring answer that outscored an accepted answer with a score of more than 10 and by more than two times the accepted answer. So the question is, “GitHub, how to do case sensitive search for the code in repository.” The badge is awarded to Matthew Reed. So you can always update those old questions with new information and you will be rewarded for it. Well everyone, I am Ryan Dodman. I edit the blog here at Stack Overflow. If you want to reach out to us with thoughts, feedback, or show ideas, or if, like today's guest, you would like to be a guest and talk about whatever corner of the software engineering world you like, email us at podcast@stackoverflow.com. And if you like what you heard today, drop a rating and review.
GM I'm Geshan. I am on the internet at geshan.com.np. That's my blog, and you can also find me on LinkedIn. Search for my name, Geshan. Thank you.
RD All right, everyone. We'll talk to you next time.
[outro music plays]