On this sponsored episode of the podcast, we chat with COO Dan Lines and CEO Ori Keren, co-founders of LinearB, about why PRs are the chokepoint in the software development lifecycle, uncovering and automating the hidden rules of review requests, and their free tool, gitStream, that’ll find the right reviewer for your PR right now.
With companies taking a long look at developer experience, it’s time to turn that attention on the humble pull request. The folks at LinearB took a look at a million PRs — four million review cycles involving around 25,000 developers — and found that it takes about five days to get through a review and merge the code. CI/CD has done wonders getting deployments down to a day or less; maybe it’s time for continuous merge next.
On this sponsored episode of the podcast, we chat with COO Dan Lines and CEO Ori Keren, co-founders of LinearB, about why PRs are the chokepoint in the software development lifecycle, uncovering and automating the hidden rules of review requests, and their free tool, gitStream, that’ll find the right reviewer for your PR right now.
Episode notes:
So why do reviews take so long? Context switches, team leads who review everything, and the bystander effect are top contenders.
Dan and Ori hope their gitStream tool can reduce the time PRs take by automating a lot of the hidden rules for reviews. Check it out at gitstream.cm or linearb.io/dev.
Dan Lines hosts his own podcast: Dev Interrupted. Check out this episode with Stack Overflow’s very own Ben Matthews.
Connect with Dan Lines and Ori Keren on LinkedIn.
Shoutout to Rudy Velthuis for throwing a Lifeboat to the question Why should EDX be 0 before using the DIV instruction?
[intro music plays]
Ben Popper Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I am Ben Popper, Director of Content here at Stack Overflow, joined as I often am by my colleague and collaborator, Ryan Donovan. Hey, Ryan.
Ryan Donovan Howdy, Ben. How’re you doing?
BP Good. So today we are going to be talking about helping engineering leaders improve their developers’ workflow automation, their developers’ productivity. We've talked about some of this stuff on the blog before, the metrics you can use to measure it like Dora, and we know this is a big focus for a lot of technology companies, especially right now as folks are looking to get lean as headcounts or budgets are being reduced but demand may have continued to increase. So without further ado, we'd like to bring on the sponsors for this episode, the fine folks at LinearB. Dan and Ori, welcome to the Stack Overflow Podcast.
Dan Lines Thanks for having us.
Ori Keren Thanks for having us. It's great to be here.
BP So Dan, let's kick things off. For folks who don't know, how did you get into the world of software and technology, and what led you to the role you're at today?
DL Yeah, absolutely. So my name is Dan Lines. I'm one of the founders at LinearB. And in the beginning I kind of had a traditional journey studying comp sci in undergrad at Villanova University, that type of thing. I got my first job as a software engineer and eventually worked my way up through the management system to VP of engineering and all of that. But really the passion for me comes from innovation I would say, and creating cool products. That's kind of my specialty. The thing that I like most about software engineering is that I can do something on my own or with a team and create something really cool that other users could use. That's where I get my inspiration from.
BP Nice. And Ori, how about yourself?
OK Yeah, so my name is Ori Keren. I'm the co-founder and the CEO of LinearB. And I actually started at the age of 13. I had a Sinclair Spectrum, a ZX 48K RAM. But my hobby was to play games on it, but also to program in Basic. So that started to spark something for me to be in this field. I'm based in Israel in Tel Aviv, and I went through a journey. I was a developer, then a team leader, I was a VP of engineering twice in companies that grew fast and experienced all the pain points like scaling an engineering org, and I decided to try and help other people that go through the same journey of how do you do it right? How do you scale? I really connected to what Dan is saying. At the end of the day, I still go back to the age of 13 programming something in Basic, forcing my parents and my sister to look at the cool stuff I built, but now we're just doing it at scale. So building cool stuff and putting it in the hands of users is still the passion.
DL And now you have more than 48K RAM.
OK Yeah, to do stuff.
DL So that's good news.
RD Better world, better world.
BP How did you two meet and what was sort of the genesis of the company we're going to be talking about today?
DL Well we are former colleagues, so we met at a previous startup that was called CloudLock, which was cloud data security. Highly, highly successful. I had joined that company prior to Ori and at that time I was a front end software developer and started moving up through the management track. Maybe I was a manager on the US side. And since we were expanding rapidly, the executive team brought in Ori, who can speak for himself, but Ori is based in Israel, and so we were kind of doubling the size of the engineering organization. And Ori grew that office and I was reporting into Ori at that time, and then eventually we became peers at the executive levels and kind of teammates. And that's where we built our business relationship. And then also when you do that, you become friends. And after that exit that we had from that previous company, we said, “You know what? Let's combine forces here and do some good for software engineers– the area that we know best.”
RD So talking about improving the developer process, I think one of the things I found interesting is that you guys did some surveys and pinpointed one of the problem areas as the pull request. What is the problem with pull requests?
OK Yeah, so maybe I'll start with a little bit of historical background of how pull requests even made their way into mainstream enterprise development. Because when I was a developer, still a lot of these things were being done manually. So I would do a peer review sometimes, not all the time, and I think pull requests as the mechanism to do peer reviews and as the mechanism to get code merged into the main trunk, it was based on the rise of popularity of GitHub and other Git platforms that are around a lot about open source and collaboration and all of that. And kind of through open source, all those Git platforms made their way into enterprise development. And they were kind of built through collaboration of software teams that are remote with hybrid and all those situations, which is the situation for open source. So they brought a lot of the benefits of that, but also I think some slowness and other problems into the mainstream of how you develop applications, and we can dive into that.
DL Yeah, absolutely. One of the things that I say we're very fortunate to have at LinearB– so LinearB is a platform for engineering teams efficiency. But we get to analyze a lot of pull requests. That's one of the great things that come out of it. So Ryan, I think you're asking for a little bit of data. Our team looked at about a million PRs, about 4 million review cycles there, 25,000 developers involved. And what we found is that there's significant, significant room for improvement in the PR process itself. So for example, at LinearB we look at cycle time, so that's kind of that end-to-end time of, “I'm starting to code,” all the way out to deployment and everything in between. And we saw, “Okay, where is the real bottleneck here?” And when we were looking at the PR process, we're looking at days to get through that review, days and days, a little over five days on average which was kind of shocking to us when we saw it. The other thing that we saw which is great news for the industry, when you think about CI/CD, so things that happen after merge, we did see drastic improvement there. Deployment times are coming down, we're talking hours or maybe 24 hour type situations. But that middle portion of cycle time of getting someone to review my PR, getting through the review, it's highlighted in red in delays, and that's what the data said to us.
RD So is that just people sitting on a PR not doing anything or is there something else to it?
OK So in a PR there's an issuer that's kind of saying, “I need a review,” and there's the reviewer. Developers are very efficient, so once they issue a PR, probably they'll try in the first five or ten minutes to try to find somebody to review their PR, but if that doesn't happen, I'm going to move to the next task or do something else. So it's not that you don't do anything else, but the context switches between, “Okay, once somebody reviews my PR and I need to go back to it,” adds a lot of toil and a lot of tax. There's researchers that are talking, it depends on the complexity of the PR, but it's going to take me up to 30 minutes sometimes to get to the same cognitive state that I was in ready with my code when you will theoretically give me my review. So optimizing that process is something that is very interesting for us to do.
DL If you kind of just think about the nature of the PR or the DNA of the PR, you're at least interrupting one person, and you're probably interrupting two. So what does that mean? When I open a pull request, if I assigned it to Ori here, he's probably heads down coding, doing his job, doing his work, which I know he likes to do. And I'm asking, “Hey, can you stop doing that and come help me get my stuff done?” So that's one interrupt there. Now Ori knows, he thinks he's a better developer than me, so we'll go with that. I probably have bugs in my code and he's going to say, “Hey, I did do this review. Dan, your pull request was really big so it took me 30 minutes+ to review this.” Hopefully I blocked off enough time to actually make that happen, but here's a change request back to you of all the things that you need to fix. Now I'm not waiting for him to do that change request. I moved on to another piece of work, so now he's interrupting me back. And that's kind of at the most basic level just the nature of how the design is set up.
BP Yeah, a point that you're both making that really resonates with me and we talk a lot about at Stack Overflow is that context switching is a real killer for developer productivity and flow state is the place where developers feel happiest. And inherent in the PR system is, “When you get a chance, I need you to look at this,” and as you said, inherently often even in the responses is, “Well, I need you to stop what you're doing now and go back and revise some of that.” So what is a solution there? I know you had talked about their CI, their CD, continuous integration, continuous delivery. There's also continuous merge. What's your turn of the wheel that gets us past some of the inherent issues and lets people streamline this or build it into their workflow in a way that feels automated so it doesn't always interrupt them?
OK Yeah, so to describe it very well, we love data. We came from the data, we looked at the development pipeline and we said to ourselves, “If we're seeing from all this analysis that once the integration engine starts to run in CD, those are problems.” It's still hard to control the amount. Not all the companies solve it, but you have a lot of technology that helps you solve those problems. And by the way, those problems involve a human aspect. There is a human aspect there, “Okay, just making sure that release is ready to go,” but a lot of automation. And CM, continuous merge, is kind of saying, “Let's move, shift it a little bit to the left when code is ready to go.” And we just described the problem– I want the code to be merged and apply similar automations. Some of them could be fully automated like code probably in some cases can be reviewed by a bot and merged to the main trunk. By the way, some companies work in trunk-based development. That's what they do. They don't need a reviewer. And sometimes it could be semi-automated or maybe just find the most free and most relevant human resource that's available right now to do the PR. So continuous merge is all about that. There's a lot of use cases and problems that live inside it, but that's applying the same principle– let's automate as much as we can how code is making its way to the main trunk.
DL Yeah. One of the things that was really interesting about that is once we had the data saying, “Okay, this is a problem and a bottleneck,” we started to interview the teams to really understand what they were doing in the PR process. So at the highest level you might say, “Okay, we have a requirement that there's one reviewer.” Or they might say if you're in an enterprise-y financial situation, “We have to have two reviewers for every PR,” and say, “Okay, but how does it really work?” And you start peeling back the layers. Then they start telling you, “Well, sometimes if I have a PR that I think is really low risk, I'll open the PR and then I'll send a message to Ori over Slack saying, Hey, it's a low risk PR. Can you approve this? You don't even need to look at it.” It's like, “Oh, okay, that's starting to get interesting.” Then there's situations where a bot opened up a PR. Dependabot does its job, but then it's five days until someone gets to it and they're like, “Yeah, I have to set three hours aside. I go through all this Dependabot stuff one after another and I'm clicking approve and merge, approve and merge.” Other situations are, “Well, we say that we need at least one reviewer, but I don't know who should be the reviewer so I just send down to the dev team channel, ‘Is anyone free? Can someone get on this?’” And so what I started to learn is that there actually are rules behind the scenes to these PRs, but they're not codified anywhere, they're not programmed. They're happening at kind of a random rate, depending on how the developer is feeling that day and what they're trying to do, usually they get their code merged fast. And that's where we saw an opportunity to say, “Hey, you know what? What I'm learning here is that you're actually looking at the behavior of the code change and doing different things based on what changes are on the PR.” Sometimes you're going with an express lane to say, “Hey Ori, can you just push approve on this? I know it's safe. Don't even want to bug you. I'm sorry for pulling you away.” And other times you're already identifying, “I'm afraid of a security issue here. Let me get a security review.” And so what we did –this is leading into gitStream, one of the innovative products that we have at LinearB– what we did is allow teams to actually program those rules in a YAML file. And when you start programming those rules in, the outcome that you see is, “Okay, some of these PRs are moving through the pipeline much faster, but we're also increasing our quality.” And if I give a few rules that some of the platform engineers have implemented with gitStream, in some situations where they wanted to move fast, they would have a two-reviewer policy for the entire engineering org, but they would say, “You know what? In situations where it's test only changes, documentation only changes, things that we can identify as being really low risk, let's at least move that down to one reviewer or even start auto approving and putting the power back into the hands of developers.” That's when we think about developer experience. In those Dependabot situations, instead of blocking a day to say, “Let me push approve and merge, if it's a minor version change, let's approve and merge them automatically.” They started using gitStream. There's a piece of functionality where it automatically identifies an expert reviewer, so its dynamic code owner would say, “Oh, you know what, Ryan? Ryan is the person that changed this code last. He probably knows the most about this. Let's just auto assign him. He'll get the review done most efficiently.”
BP Right. This is really interesting, yeah.
RD From my experience, I've seen a lot of PRs where it’s either one person who is very busy who has to prove everything for a specific service or repo, or just the shotgun approach, and then you have everybody on a PR, you only need one reviewer, and everybody kind of sits around waiting for somebody else to do it. And I think having the kind of rules to codify it, just having that process there helps people get their ass in gear and actually approve things.
BP There's two things you said that really interest me. One is talking about where automation can fit in and maybe this one doesn't need a PR, but it needs to pass a few simple tests, and then we know we can go for it. And two, I know this one needs security. I'm going to give it that tag, and then it's automatically going to be sent to the right people. We have this thing inside of Stack Overflow Teams and it's kind of the same thing. You ask a question and if you put a certain tag in, then the subject matter expert gets notified. So instead of having to figure out who to email or what listserv or where in Slack, the right brains to the right place at the right time I think is a super useful tool. So how long have these products been in the market and how have you seen them evolve? It feels like engineering is moving more towards developer experience, platform engineering, that's what employees want at software companies and people want to keep their highly paid and hard to recruit tech staff happy. So talk to us a little bit about how it's evolved over the last year or so and how you think that fits in with the bigger trends in the industry.
DL One of the things that I really like about gitStream, I said a few rules, but the truth is it's flexible. So it's been out for a few quarters here. It is free. And what's happening is I'm learning about rules that I could never think of myself, and it has to do with that movement that we hear around platform engineering, because these are super, super talented folks that have just a developer or a coding background. They're taking gitStream and they're coding in the YAML file and adding their own rules. I'm sure it's on Stack Overflow, but what I've seen is, I think Ryan, you gave the example of the same person reviewing every single time, which is a horrible, horrible bottleneck. I've seen now a knowledge sharing rotation. “I'm going to write it so that the juniors that just joined, you have to review. Now we can actually review a little more.” And it's these platform engineers that are kind of unlocking performance, I would say improving the experience, not just with gitStream, but for everyone around the globe. Those are the people that are pushing at least our product forward.
OK Yeah, I think it's a great analysis. I can see platform engineering teams, they evolve and it kind of feels like 2022 and 2023 are the peak of that movement. They're evolving right now because the architecture became more complex. There's a lot of microservices and serverless and the stack is a lot of variants. Teams have become diverse and they work in different environments. So those platform engineering teams, they're all about, “How do we make the work more efficient?” I see them focusing mainly around developer experience that we talked about, and we can dive even more into that. A little bit about resiliency, which is shared with SRE teams you can see. And even cost now in this economy, they help there. So it's really interesting to see how this will continue to evolve in 2023. But we can see those teams, like Dan was saying, these are people that were developers, very talented ones, that understand that so much toil exists in the day-to-day of developers to complete a task end to end from, “Hey, my test is sometimes failing and sometimes passing, failing and passing. Something is wrong with it, so let's identify those and fix them.” Or the build just got too long, 20 minutes instead of the 5 minutes it used to run. And as a developer, sometimes it feels like, “Okay, we're optimizing a thing for ten minutes or even two minutes sometimes. The thing that just got two minutes longer, we want to optimize it as developers. Why? Because we know we're repeating this task a hundred times a day and we know there's a threshold that takes us out of the flow situation that you guys talked about.” So I can see these teams putting the focus on those things, making sure that developers can stay in flow. There's another super interesting angle here. Applying or putting the task on the most suitable human or the right developer at the right time, because if teams are trying to be more efficient right now, that's part of the movement that we see.
BP Right. Rather than being, “Hey, this is the day of the week you're assigned to review PRs,” or the hour of the day, it’s, “How can you make sure the person who's reviewing it is the one who's going to be able to evaluate it quickest or solve the problem best?” That's a much more intelligent approach. We're getting towards the end here so Ori and Dan, let me ask you, is there anything else you want to touch on before we go towards the outro? Otherwise, what I'll do is I'll set you up to sort of tell people where they can learn more about this and check it out and try for themselves.
DL I would just add, if you're listening to this, if you do not have that platform engineering team or that role, I would encourage you to look at the developers within your organization. I usually can spot them pretty quick with a few questions. They're usually the ones that say, “I love helping other developers succeed.” Start that group. Don't get left behind. Kind of just a shout out to the movement.
OK Yeah. I think it's a very strong point. I would add to that, you don't even have to start that group. You sometimes do it without knowing that that's what you do. They exist in every team. Like Dan is saying, if we kind of formalize a little bit better of what are the tasks that we want those teams to do, I think every engineering team can drive their efficiency much, much higher if they invest in it.
[music plays]
BP Alright everybody, it is that time of the show. We want to make sure we thank somebody from the community who came on and saved a little knowledge from the dustbin of history. Thanks to Rudy Velthuis, who came on and won a lifeboat badge for answering a question with a negative score, getting it up to a positive score of three or more, and that answer got a score of 20 or more. “Why should EDX be 0 before using the DIV instruction?” Well, Rudy can explain it to you and has helped over 12,000 people with his answer. I am Ben Popper. I'm the Director of Content here at Stack Overflow. You can always find me on Twitter @BenPopper. Email us with questions or suggestions, podcast@stackoverflow.com. And if you like the show, leave us a rating and a review. It really helps.
RD I'm Ryan Donovan. I edit the blog here at Stack Overflow– stackoverflow.blog. And you can reach out to me on Twitter @RThorDonovan.
DL So I'm Dan Lines, founder and COO of LinearB. And to check out gitStream, you can go to gitstream.cm, or linearb.io/dev. And if you want to hit me up, you can find me on LinkedIn, Dan Lines.
OK I'm Ori Keren, I'm the co-founder and CEO of LinearB. I really enjoyed this. You can find me on LinkedIn or Twitter. And like Dan said, if you are into platform engineering and gitStream, try gitstream.cm. If you are more leading an engineering group and want to try our metrics, find it at linearb.io.
BP Awesome. All right, everybody. If this rings true, if you feel like there's a platform engineer inside of you, don't be afraid. Start the group, help your friends check out gitStream. Thanks for listening, everyone, and we will talk to you soon.
[outro music plays]