On today’s episode we speak with Kohsuke Kawaguchi, who won the Google-O’Reilly Open Source award for his work on the Hudson/Jenkins project. Kohsuke began his career at Sun Microsystems. In 2010 he founded InfraDNA, which merged with Cloudbees shortly after. In 2014 he become the CTO of Cloudbees. He shares insights on the balance between community-driven open source and the need to monetize through enterprise services.
You can learn more about Kohsuke on his website.
You can read more about Jenkins here.
You can read more about Cloudbees here.
Shout to Mossmyr for contributing a question that's now part of our CI/CD Collective: Is there a way to call a Jenkins Shared Library method from another Jenkins Shared Library?
[intro music plays]
Ben Popper Hey, there. It's me, Ben Popper. Listeners of this show are obviously interested in software, so I wanted to recommend checking out the Web3 with a16z Crypto Podcast. Brought to you by venture capital firm Andreessen Horowitz, which first coined the phrase ‘software is eating the world,’ this podcast is your definitive resource on the future of the internet, from the latest trends and research to insights from top developers, scientists, and creators. The show is about more than crypto and Web3. It's for coders seeking more ownership of their work, for business leaders trying to prepare for the future today, and for others trying to understand the next innovation and tech trends. Be sure to find and subscribe to Web3 with a16z Crypto wherever you get your podcasts.
BP Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I'm your host, Ben Popper, Director of Content here at Stack Overflow, joined as I often am by my colleague and collaborator, Editor of our blog, maestro of the newsletter, Ryan Donovan. Hey, Ryan.
Ryan Donovan Good morning, Ben.
BP So I was out on sabbatical. You did a great job hosting the podcast and booking some guests. Why don't you tee us up? Who are we going to be chatting with today and what's on the docket?
RD So today we're going to be talking to Kohsuke Kawaguchi, who is the CEO of Launchable, which was just acquired by CloudBees. Kohsuke is also the creator of Jenkins, which is key to a lot of CI/CD pipelines. So today we're going to be talking about those CI/CD pipelines and how you can get DevSecOps running in your company.
BP Very cool. I did a quick search after you mentioned that and there's over almost 51,000 questions tagged with Jenkins on Stack Overflow, so quite a bit of people using it and having questions and answers. So that's very cool. Kohsuke, welcome to the show.
Kohsuke Kawaguchi Thank you very much for having me.
BP Why don't you tell folks first, how did you get into software development and programming? What was the story of getting into that world, and then what led you to creating Jenkins and coming into the role you're at today?
KK I think it was in junior high school, so it must be seventh/sixth grade. My mother had a computer. Back then, this was a time when magazines were full of people's computer programs, mostly games. I often typed it up into my computer for the sole purpose of playing the game, except the computer I had was a minor architecture, so I couldn't just type it in– I had to port them over. So that I think is how I kind of learned to program. Initially just typing one letter at a time and over time you start to understand what it means. And I kept doing it and then I was lucky, so the things I got interested in turned out to be an economically viable profession. In my college, I created the first company with my friends, and then I had a job offer from Sun Microsystems, which is sadly no more, so that brought me over to the US. And Sun was a great place for engineers. The day job wasn't too busy, so I had all the time in the world to do my “hobby programming.” So I wrote tons of open source software and then one of them just was at the right place at the right time that became Jenkins over the years.
RD Jenkins, like you said, right place, right time. It seems like it came out in 2011 or so. Is that right?
KK When I was looking at the history, the first line of commit happened in 2004. So I was just originally writing it for myself, and then next to my team, and then it was around 2007 that the rest of the world noticed that this software did exist. It was out there all along, there was just no effort to try to evangelize. And then 2011 is when Sun was bought by Oracle and then I moved to start CloudBees, so that was another marking point.
RD I think it's great that you started as something to solve your own problems. I think some of the best open source projects are. Do you have any insight or idea of why it became so prevalent in the CI/CD world?
KK To your point, I was definitely lucky. Clearly I must have spent two or three years flying solo. I was motivated enough by feedback from the team and people around me. When you have that much head start to build the value of the software, by the time the world noticed it, it does something reasonably well. And that kind of setup is actually pretty hard. Most of the time, people won't sustain enough motivation when they are not getting positive reactions for that long. In a commercial setup, you can't have a three-year project that doesn't release anything. So that, I think, was clearly one factor. Another thing is that I had a pretty painful experience contributing to some other open source projects. They had what I consider silly coding conventions or rules like, “Oh, you cannot have a method more than 20 lines.” So I have to split this seemingly logical function into two, just so that I could meet this criteria, and then you submit the path. So this was back in the day when there was no Git, so subversions, and then you have to kind of lobby the maintainers that, “Hey, this change is going to be good,” and all that stuff. So I wanted to build a project where people wouldn't have to do any of that. People are spending time on open source projects because they have ideas of something they want to do, and my job as a maintainer or the leader of the project is to get out of the way so that the people can do what they want– except I don't want to be holding all the bags, so the challenge is how to structure software both socially and architecturally so that I could encourage all the innovations and crazy ideas. And you cannot distinguish the great ideas from wacky ideas in the beginning, so I let all the flowers bloom and then only the best ones survive. So I started to form some hypotheses about how we could do just that from the early projects, so in the Jenkins project, I put effort to make that happen, and that, I think, turned out pretty well.
BP Can you just say a little bit more about that? You mentioned that the first line was in 2004. GitHub was started around 2007. What was it like in the open source world that you were working in from 2000 to 2004? It was very different from what people imagine today where they throw up a public repo quickly and people can come in and look and then contribute or fork. You had to what– communicate with people on bulletin boards and email and hash out what was going to happen? How did it work?
KK It's a crazy idea that the people would have to communicate to collaborate. This was one of the key phases in the life of open source because this was around the time that companies started taking on open source. I remember Sun and IBM were in a competitive relationship, but at one point they decided to work together, and then the vehicle in which they chose to do so was the open source project under Apache. I was sort of in the peripheral of that effort, which is why I was able to come from Japan to the US. So in some sense, the reason I got noticed was because of this– my side open source gig. So for me, open source was pretty big in the heart. So these are the times when there was very prestigious software foundations, most notably Apache Foundation and later Eclipse and some others. So people are trying to become a “committer,” which is a privileged status to be able to push the change directly. And then in order to do so, you kind of have to prove yourself that you're trustworthy, you can behave, you're one of them. So you often see people start by contributing patches, which today is like a pull request, and then gradually show you that your work is valuable, you know how this community works. There's a lot of upfront investment necessary. And then I started seeing some prestigious individuals publish open source projects that were very influential, like new web frameworks and Codehaus, another hosting platform back then that was also seen as highly prestigious. That's different from, let's say, the likes of a source board where that was the GitHub of that day. Anybody could start the project, but what that meant is too many noises. It was really an environment in which great programmers could kind of make their names out of what their individual work was. That was an exciting context for individuals.
RD One of the things we come across in open source is funding that open source. And when you originally went to CloudBees, you were building the enterprise version of Jenkins. What's it like building there as like a paid version of an open source project?
KK So CloudBees was a startup founded by Sacha Labourey. He was already a well-known figure in the open source space, and he was successful enough that he could fund the next startup. In a strange turn of events, I got in touch with him and he was trying to do originally a platform as a service, so like a Heroku for Java. And then in that process, he had the right vision that the delivery pipeline is an integral part of this platform as a service and so that's where I kind of came in. In that effort, we created the enterprise version of Jenkins. By that point, Jenkins was pretty popular. It just so happens that Sun was going down the tubes. So if that didn't happen, I was probably pretty happy working as a software engineer at Sun. It's a comfortable place and the work isn't too busy and we get all of the spotlights in the world, but alas, nothing in this world stays the same. So suddenly Oracle happened and then I started thinking, “Well maybe this software is popular enough. Perhaps I could do something about it.” So CloudBees was a great vehicle. And then as I started doing enterprise versions of Jenkins, the Jenkins community was so good that it was kind of hard to come up with things that you can charge for. We are selling to the people who can build their own stuff, so if something is too valuable then they'd be able to do so on their own. So initially it wasn't easy. I remember the sales rep was complaining to us.
BP Right, “Make it worse so we can sell some things here.”
KK It's like, “We can't sell this software for $3,000 a year. That's impossible.” Now the company literally have 60-70 a year. But off the ground, that's hard. But previously I felt like a more naive, negative view on the commercialization of open source. I think that faction probably still exists in the sense that there's something pure in this functioning model of open source, so maybe the idealism represented in the Free Software Foundation. This is all about the freedom, the movement, and then commercial interest clouds things and makes things dirty. But what I realized later as I was doing CloudBees is that if you could create a scheme in which you could bring more money, then you can use that to put more back into the project, and that's kind of exactly what CloudBees did. Engineers, they might show up as volunteers, but that only chooses certain kinds of engineers. CloudBees was able to get people like designers and doc writers.
RD Yeah, you professionalize it.
KK Yes, so it really made a key difference.
BP That's interesting to hear your thoughts evolved on that. I think we're having a similar discussion at Stack Overflow. Now we're doing licensing deals for the data and the knowledge that's been created by the community, and the argument is that we're going to put those resources back into building new features and hiring folks, and that as long as the resources are shared back into the community, then it's hopefully a virtuous cycle.
KK Exactly. There's a certain space of the industry where to make an impact you need more than just the software. You need people who know how to deploy that into a large environment. You need some people like a vendor. So information security questions, somebody is responsible and looking after all the vulnerabilities found in the software, or if people need support.
RD Right. You need hosting, all those sorts of things.
KK Exactly. I personally remember doing training myself and then those things are all useful to kind of shore up the entire ecosystem. So I made a renewed appreciation about that.
RD And then you left there to form your own company and then CloudBees acquired that company recently. What was the impetus for starting that company? What was the new problem you wanted to solve?
KK It's kind of hard to articulate all the reasons, but one of the key ones I think was that I was a CTO at CloudBees and I saw the company grow up close and I felt like maybe partly I wanted to do it myself. I felt like I learned a lot but it was from a different seat. And then the opportunity to kind of be at the driving seat, or so I thought until I realized that the CEO also has their different bosses, it’s just they’re coming from a different place.
BP Oh, you're meeting board members now. Uh-oh, okay.
KK Exactly. That was one of the motivations. The other is that I was always passionate about the developer productivity, and I thought where to drive the developer productivity next? And I started thinking that it has to be data. We have “done” the automation. So that's on par with, let's say, the manufacturing industry having all these machines on the factory floor. Well, then the next place has to be how to optimize a layout of this factory floor. Take this branch and margin practice as an example. People choose their own branch margin strategy. The smaller teams can naively do things like feature branch and then test and merge, but I saw that when the team gets bigger, they have to create a little more complex scheme. But those choices are done very arbitrarily, and then this phase of the software development changes and the project grows and they suddenly realize, “Oh, this naive branch merging can't continue.” So that felt like people were still driving the manual stick shift. “Oh, the torque is not right in this gear one, so I have to shift up,” but that felt awfully inefficient when the machine should be able to do the job easily and far better. So that was just one example of using data to define and simplify and optimize the delivery pipeline. So that became a motivation to do a new company. Also I suppose I needed a little bit of psychological distance away from the Jenkins project, otherwise I would never be able to get out of it because there's always more things to do. So it helped me to move out of CloudBees in order to do that.
BP Right. You could let go of your baby.
RD Sounds like the metaphor of moving from the manual to the automatic shifting is sort of what a lot of the CI/CD DevOps stuff is about. You're moving from manual build, manual tests, all these manual deployments to just an automated process, and now a lot of folks are adding security into that process. How does that work?
KK The vendors pitching DevSecOps have been doing that for 10 years or so. At least the way I see it, in practice that just seems to be translating to running a few tools as a part of the delivery pipeline in order to meet some compliance requirements. Originally, the term ‘continuous integration’ was this idea that Martin Fowler had where you have to be integrating more often in order to reduce the risk in a software project. But when it's adopted, all that it meant was just running build and test all the time, or that the word ‘DevOps’ was originally coined more so that these two sides of the two ends of the delivery platform should be working together. But nowadays, CI/CD just means automated pipeline. So DevSecOps seems a little bit similar. I think the aspiration that the mantra of it is a little more abstract, but in practice, it seems to be just converting that. It's undoubtedly a good practice.
BP I think what you say practically makes sense, but in the conversations Ryan and I have had, a lot of the import there is breaking down the silo, that this would get passed to security after it was built for them to understand or that they would come in once something had already been pushed, and that if you make it part of that initial process, you avoid a lot of headaches and have a more tightly integrated team. So like you said, it's not like you're coming up with something new. It's more like a change in philosophy and workflow that can be impactful.
KK Definitely. I didn't mean to imply that it wasn't useful practice. It is increasingly more and more of a reality of software engineering. And this update, dependency updates is a pain.
RD We've seen the perils of automatic updates. I got stuck in an airport for three hours because of it.
BP A lot of people missed their flight that day.
RD DevSecOps a lot of times is just sort of integrating other tools into that pipeline. Is that an okay way to do it or is there a better path forward to own that security in-house?
KK This is a great way to do it. To me, it has really highlighted the importance of certifying the changes quickly. The reason library updates are painful is because, “I guess it's okay to merge this change,” and then you press the button. I always feel like I'm taking on some level of risk. I think if people are feeling more comfortable that the changes that they're producing are good then they'd be less averse to updating the libraries and then be always on top. That to me is the mitigation to the risk from a security perspective, which is the known issues, known vulnerabilities in libraries. I think it was when the log4j issue happened, somebody published a report that half the organization was able to update that very quickly, but the other half didn’t, so there's a striking difference. You can't anticipate these things happening, but once it happens, it's a speed in which you can make a fix.
BP So what are some of the things that you are excited about that have happened over the last year that you're looking forward to, either in the open source world or at the intersection of open source and business? Are there things that are happening, whether that be part of the AI wave or completely separate from it that gets you excited, both as a programmer and as a project owner and also now as part of a business?
KK Not precisely last year, but like many people I do have to say the large language models have been super exciting. Sometimes this industry catches on to some weird things that makes me roll my eyes, like–
BP [coughing] Crypto.
KK NFTs, for example.
BP I said it. You didn't have to say it. It's okay.
KK I was looking at it like, “This is not a legitimate technology.” Blockchain is a legit technology, but NFT is not.
BP Well, that was art.
KK When you see LLMs, you can clearly see that it is a legitimate technology disruption. While some of the impact might be overhyped, it's like I say, singularity isn’t here, I don't think that's in the next five years. There's clearly so many uses and this just happened. So I was doing Launchable before that happened, so originally when we said, “Let's use data to optimize,” I was more doing numerical optimizations and things like that. So just the automatic transmission, but with LLM, it suddenly opens up a whole new way of using data to make an impact and then kind of take on the lower level human activities when it comes to software engineers, and that's been very exciting to find a place where we can use it well and then try to make an impact.
RD So I wrote an article a few years back about open source governance, and you were a benevolent dictator for life for Jenkins, perhaps still are. Do you think that governance style has its benefits? Obviously, you were a benevolent dictator. Do you think that is the better way than having a lot of cooks in the kitchen?
KK I think the key innovation in the Jenkins project that I touched upon a little bit is that it's kind of both at the same time. You described too many cooks in the kitchen. The idea is that every cook in this case, every volunteer contributor, they want to do some things. The metaphor that too many cooks overwhelms soup or something is that they only have one thing to work on, therefore they need to collaborate and then kind of compromise and do all of the painful things. So in the Jenkins project, the way we solved it is by coming up with this notion called plug-ins, it's plug-ins all the way down. And we didn't certainly invent that, but we applied that to the web app. So what that meant is that if you guys have different ideas, like if one guy wants to paint a shed in blue, the other guy wants to do it in red, go ahead. They can both do it. So I think that's really key to unlocking the innovations, because there's only so much one person can do. Collaboration is a significant friction, especially when they're talking about the global volunteer part-time community. Engineering wise, I think you can avoid forcing certain decisions, but in the social structures and as a kind of backstop, you do need some more corporate-like decision making scheme so that they are aware of the notion of benevolent dictator or some sort of condition making system, like a committee of respected village elders or whatever, is really useful. So I think it was natural that there's only certain things that the founder of the project can do because they have outsized influence. And then to have that codified, that influence is always there, whether you name it or not, and then to give it a name makes it a little easier for people to understand how this community rolls. And then the transparency, since you talked about the governance, transparency in the governance I think is actually pretty key in my mind to grow the project as opposed to people showing up to the project and sort of figuring out how this community does decision making. So I think that's beneficial. Different projects have different needs, so for a programming language, you can't do these plug-ins. So the constraints are different, but those are the things I think about.
[music plays]
BP All right, everybody. It is that time of the show. We like to shout out someone on Stack Overflow who came and contributed some knowledge. Often that's somebody who has been awarded a badge, but today I'm going to shout out our CI/CD collective where folks can go to learn more about that. And our latest question, asked 35 minutes ago, “Is there a way to call a Jenkins shared library method from another Jenkins shared library?” So somebody needs your help, the question is out there, it's part of a Collective which is a cool new thing we have on Stack Overflow. So if you've got an answer, hop on over and put it in there and earn yourself some rep. I am Ben Popper. I'm the Director of Content here at Stack Overflow. You can always find me on X @BenPopper. If you want to come on the show, if you want us to talk about a certain topic you haven't heard, or if you have just questions and suggestions, email us, podcast@stackoverflow.com. And if you enjoyed today's program, leave us a rating and a review. It really helps.
RD I'm Ryan Donovan. I edit the blog at Stack Overflow. You can find it at stackoverflow.blog. And if you want to reach out to me, you can find me on LinkedIn.
KK I'm Kohsuke Kawaguchi. I'm the creator of Jenkins. I used to be the CTO of CloudBees, and then the CEO of Launchable, and then I'm back to CloudBees. Probably the best way to find me is kohsuke.org. I'm one of these old school people who still have my own domain and all my contact information is there. Looking forward to talking to like-minded people.
BP All right, everybody. We'll put those links in the show notes. Thanks for listening, and we will talk to you soon.
[outro music plays]