Jim Highsmith, an original signatory on the Agile Manifesto, tells Ben and Ryan about what software development looked like at the time of the Apollo program, the evolution of user interface, and the meeting where “17 adventurous techies changed the world.” Plus: How a savvy shoe choice helped introduce Agile to Nike.
Jim is a pioneering software developer who was one of 17 original signatories to the Agile Manifesto.
His first engineering job was on a little NASA program you may have heard of: Project Apollo.
His latest book is Wild West to Agile: Adventures in software development evolution and revolution; get your copy here.
Find Jim on LinkedIn or his website.
Today’s Lifeboat badge winner is nCod3d for answering How can I find how many useful digits are in any given a number N?. Thanks for spreading some knowledge.
[intro music plays]
Ben Popper NEAR is the BOS: the Blockchain Operating System for an open web. Effortlessly create and distribute decentralized apps across any blockchain. Quickly build using languages you already know, like JavaScript. And with FastAuth, you get an onboarding experience that is faster and easier than Web2, no crypto required. Get started building on the BOS today at near.org/stackoverflow.
BP Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I am your host, Ben Popper, Director of Content over here at Stack Overflow, joined as I often am by my colleague and collaborator, Ryan Donovan. Ryan, I'm going to let you lead the intro today because I know you've already done a great interview and sort of set us up with this guest. So take it away, what are we going to be chatting about today, and who are we going to be chatting with?
Ryan Donovan Sure. So today we're talking with Jim Highsmith, one of the original signatories of the Agile Manifesto and the author of the book, Wild West to Agile. It talked about starting off in the early days of programming before any methodologies and then what led up to Agile and how it's had its staying power and why people love it and also complain about it.
BP Okay, yes. Before there were no methodologies, now maybe there's too many. I've heard of both sides of it. Jim, welcome to the Stack Overflow Podcast. Thank you so much for joining us.
Jim Highsmith Thank you. It's great to be here.
RD So I read a little bit of the book and I found out that you, like my mother, started at Pan Am in the ‘60’s.
JH Really? Yeah, I worked for the aerospace division, which very few people know about, and they actually ran the missile range out of Cocoa Beach, Florida. So it was different from the NASA facility and they actually had a contract with the Air Force to run the missile site, and that's where they began to shoot off the Gemini missions. And then they tested the Saturn V there a couple times before they took it up to the Merritt Island site.
BP I myself was once a technology reporter and I got into covering the commercial drone space. I went out to this amazing hangar where Google was making these balloons. It was the biggest hangar I've ever been in, and it was a reminder to me when I was there that a lot of the history of Silicon Valley goes back to the aerospace industry and to NASA and to the chips and computers and technologists who sort of convened around all that. That kind of laid the groundwork for the Intel’s of the world.
JH Yeah, I mean, when I was in school, we didn't even have a programming course, much less computer science curriculum. I was an electrical engineer. We designed circuits with transistors. It was as big as the end of your finger, and now they have 5 million transistors on something the size of a pinhead, so it was quite a bit different. And that's actually one of the things that I bring out in the book, that the technology of the time really impacted software development at the time. So software development was intertwined with the technology and also with the business need, and so the technology both enabled and restricted what you could do at the time.
RD I think there is something there. You talked about a build process taking 12 hours. That's going to change how you develop software. That's going to change how you test.
JH I mean, can you imagine somebody today sitting down hitting the enter key and getting a response 12 hours later?
BP No. People come on this podcast and they complain about a build time that takes five minutes.
RD Yeah, get a lot of ping pong.
JH Yeah. I've got a graphic in the book that shows a guy sitting at the end of this humongous computer feeding card deck in, and the other end is a printout. And that process took 12-24 hours because your stuff was batched up with everybody else's and it was a very slow processor, multiple hundreds and thousands of times slower than your phone today, and it was a very different environment. And then our testing approach was that you ran a program and if it blew up, you got a core dump of the entire core memory in hexadecimal.
RD Oh, that's rough debugging.
JH Yeah, that's what you’d debug in. You sorted through that core dump until you found your entry place and then you had to follow it, so you had to be able to read hexadecimal really well.
RD Yeah, and a lot of that software development was linked to military and government projects, right?
JH Yeah, at the time that's where they were linked. In fact, I worked on Apollo ships that went out into the Pacific Ocean to track the spacecraft as it came back, and the computers on there were military computers that they put on nuclear submarines. There was a lot going on in the aerospace business at that point in time. And in fact, early on there was some iterative development that was being done in the aerospace environment, and I mostly worked in the business environment. So one of the things I say in the book is that it is a history, not the history of software development, because it really is limited in scope to things that I had direct interaction with.
RD And the military and government is not known for its freewheeling non-conformist attitudes. You talked about the yellow shirt being a sign of nonconformity in the book. How did that change over the early years?
JH Over the early years it changed when I changed jobs looking for something that was a little less structured. So I worked for Exxon for quite a while and it was very structured, but I also learned a lot in those six years that I worked out in Houston. And I sort of rose steadily through the ranks. I got to run a big project when nobody knew anything about project management and so I kind of winged it. I had some people who had a job to do and they knew how to do it, they just needed a little guidance and help from me. And so I sort of worked my way into the era of project management. I worked for a while in a bank in Atlanta, and there was some good things about working there, but it was a little too structured for me so I moved on to something else. And finally at the end of the 70’s, I went to work for a small startup that was a one million/two million dollar a year company, and that was a big difference.
BP And so through all of those, were you still focused on software and technology, or at an Exxon or at a bank were you just project managing whatever came along– the project as it needs be, no technology necessarily involved?
JH I was always sort of involved in the IT side of it. So for example, even at Exxon as I moved up, I took a job, for example, as an accounting supervisor for a while, but it was after I'd spent two or three years as an IT developer. Of course, back in those days, we did everything from an analysis to testing. There was no waterfall, you just did it all. So then I moved into an accounting job, and then I moved into a job that was really coordinating all the software assistance development work for all the refineries in the country. And then I moved on to another job where I was a software development manager, to bring in structured techniques. So I was always involved in the IT side of it. But the other thing I did is, along the way, I got a master's degree in management. So I had both the technical background and the management background, which means I could kind of bridge the gap between the technical people and the management people early on in my career, and sort of the hallmark of my career was being able to do that.
RD As you went along, understanding both the development and the management side, you started seeing a little more structure come in, problems with the structures and how they made software. What was that pre-Agile world like when they were making software?
JH Well, let me give you an example. So this was a little later on in the ‘90’s when I had a friend that worked for Nike out in Beaverton, Oregon, and they had a project that had been going on. For 18 months, they had a requirements document and it was out of date, and so they were very unhappy. And I had been doing some work with rapid application development at the time, that was kind of a precursor to Agile. And so this friend of mine got me a meeting with the executive of IT at Nike, and the big question I had in my mind at that point was what to wear, because Nike was a real kind of laid back place and so I knew going in there with a suit and tie wasn't the thing to do. So I went in with a nice shirt, a pair of slacks, and red Nike running shoes. And the executives noticed the shoes and I got the job.
BP It's all about the shoes.
JH That's right. And so we went on to actually build a project for Nike in sort of this rapid development method using one month iterations on a six month project. The other thing that was new about this project was the fact that we were using client-server architecture for the first time. So there was a new project, there was a new technology, and one of the things that we began noticing around that era, both from working at Nike and some other places, is that what I call ‘monumental waterfall methodologies’ were not doing very well. And what had happened is, the structured techniques, the methods that they were using through the ‘80’s became subsumed into these massive sets of manuals that were what I call monumental methodologies with lots of forms and processes and checkoff points. And then the waterfall approach came in and it just reinforced all of that. And the problem we were hearing was that as business speeded up in the ‘90’s, what we kept hearing from IT managers was, “It takes too long, costs too much, it doesn't produce what I need.” And at the same time, we were going through a technology revolution in terms of the internet, object-oriented programming, GUI interfaces, and so everything changed.
RD Do you think the people before tolerated the bad interfaces because they were expert users or it let them do what they needed to do, or do you think there was no other choice for them?
JH A lot of it was no other choice, the technology of the times dictated it. I mean, during the late-70’s and all during the 80’s, we had what was called a green screen of death, which was character-based screens, and the design was to put as much data on there as possible. And so they really looked jumbled, but it was fast for people who did that day in and day out eight hours a day. One of the things that happened is we went to external users. They wouldn’t tolerate that kind of stuff so we had to make the user interfaces simpler to use, which actually, if you did that for the internal staff, it slowed them down. There's a difference between somebody who uses something eight hours a day, five days a week, and somebody who uses it intermittently. So the whole concept of interface design had to change.
BP And so how did you start to formulate some of your ideas about methodology and efficacy, you mentioned rapid application development? Was that from talking with colleagues? Did you take notes as you worked on projects and start to formulate stuff? How did you cement those ideas and then say, “All right, well now I think I understand a better approach to this. Here's how I'm going to bring that into the world and kind of make it concrete.”
JH There were sort of three stages of that which occurred for me personally during the 1990s. So I left the 1980s thoroughly involved in structured techniques, I was teaching them, I was installing them for people in methodologies, I was writing about them. And I began to get a little bit unsure that it was the right thing to do, because it wasn't the way I worked. I would work in an iterative fashion, and I remembered working early on in my career where I did everything: analysis, design, programming, testing, and I really liked that and I realized that we were taking programmers and putting them down at the end of the life cycle and really taking a lot of the energy away from what they were doing. So early in the 1990s that was kind of happening. I was thinking more, some of the evolutionary and iterative techniques were coming out– Barry Boehm and his spiral model and a couple of other ones. And I got a call from Larry Constantine. Now Larry Constantine is a guy who originated coupling and cohesion. So he's a structured guy, very well known. I got a call from him and he said he had a particular job at Amdahl, which was a big computer manufacturer at the time. And he said, “I can't take it because I'm working for IBM. Would you like to take it?” And I ended up working for them with what turned out to be a really good friend of mine for the next couple of years. And they were trying to sell a mainframe rapid development product, which may seem like an oxymoron, but their sales cycle was real long, and so what Sam wanted to do was to show the client how we could deliver value very quickly. And so we would go into a potential client of theirs, and we would get a project going, and we would do a one-month project with one-week iterations. And this was the early-90’s and we were eminently successful at it in terms of running the projects. I'm not quite sure if they were successful in going ahead and selling it, but at least we got kicked out sooner.
BP Right. Yeah, I mean, by the time they got to the end of the sales cycle, you'd already given the client the product two, three times over, so why did they need to buy it? You'd shown it to them.
JH Well, this was just a pilot kind of thing. And then in the middle-90’s I began to evolve my version of RAD, which I called Radical Software Development. And what I added to that was, I was concerned that a lot of the RAD that was going on at the time was eliminating a lot of the technical excellence and I wanted to bring the technical excellence back into the mix. So I call it my professional RAD. And then I was writing a book on radical software development, and I knew there was a missing chapter– chapter three. It was a big blank space. And chapter three was the theoretical background that I wanted, and I didn't have it and I didn't have it, and finally I started reading some stuff about complex adaptive systems theory, and I said, “That's it.” And so the idea of collaboration, the idea of emergence, really came out of that study, and I began using it and eventually I called the first book Adaptive Software Development instead of Radical Software Development. So I went through these three stages: rad to radical to adaptive during the 1990s.
RD And then sometime in the early-2000’s, what was it, 16 of you got sat down in a retreat somewhere and hashed out the Agile Manifesto. That was a single retreat, right? That was all you in a room hashing this out?
JH Yeah. But I actually got started about a year ahead of them. So what happened was– I call this the pre-manifesto period. I met Martin Fowler in New Zealand. We were both speaking at a conference down there. Seems like Martin and I have been friends for a long time and we always seem to see each other outside the US even though we both live here. And so I met him in New Zealand and we got to talking and I found out a little bit what they were doing over in the object-oriented programming world and he knew a little bit about what I was doing. And then he ended up connecting me to Kent Beck, and I sent Kent a copy of my adaptive software book and he sent me a copy of Extreme Programming. And we realized that we were on similar pages, and Kent in 2000 had a meeting of basically XP proponents. They were trying to figure out how to push XP. So Kent was there, Bob Martin was there, Ward Cunningham was there. So a number of people were there and I was one of the outsiders that Kent had invited. And so that was the genesis of getting Bob Martin interested in convening a meeting of more of a methodologist. So that's how we came up with the 17 people: some people that Bob knew, some people that I knew, some people that Martin knew, and we invited everybody that we could think of that was kind of doing a separate thing. And so that was the genesis of the meeting in Snowbird.
RD Yeah. I assume it was to develop a methodology that was a little more iterative and adaptive, that could be flexible for the shape of software to come, right?
JH Actually, no. We had no idea what we were going to do. We didn't have an agenda, we didn't have an outcome. All we knew was that we were doing things that were similar, we didn't like being called light methodologists, and we wanted to see what the similarities were. So we started the meeting actually going around the room and each of the 17 people talked about their methodology, what they did, the theories behind it, the practices they had, and we went around the room like that. And so that took about half a day and different people facilitated the meeting. Now realize, you've got 17 type-A people in this meeting, all of whom thought they were facilitating, so we finally had to make a rule. He who held the facilitation rod up got to make the rules for that part of the meeting.
RD That's right, you’ve got a talking stick.
JH So the next thing we did is to start to think about what we would like to call ourselves other than light methodologists. And so there were about 20 words that got thrown up on the board and we sorted through them and said, “We don't like this one, we don't like this one, we don't like this one.” And we finally ended up with Agile.
BP I am not as quite experienced in this world as Ryan is, but I understand that Agile has had tremendous impact on how folks work. It still comes up often in scrum calls here at Stack Overflow as we're planning out sprints. So undeniably it's played a big role in how folks have thought about working with software for many years. At the same time, I know Ryan set us up on this episode by saying that often when we do write about it on the blog or talk about it on the podcast, people come out of the woodwork to complain. So tell us from your point of view, what are some of the pros and cons, or what do you think people get or don't get about it? As it's been adopted today, why do you think it has sort of its advocates and its detractors?
JH One of the things I've done is, I've thought about this more recently, I've asked myself the question of, why did 17 adventurous techies change the world? Because you can say there's some good things that have happened and some bad things that have happened, but fundamentally, 88% of the people in the world who are doing software development at least say they're doing Agile. Now, they may not be doing it very well, but they're at least trying to do it or saying they're going to do it. You have to realize that in 2001, the light methodology community was very, very small, and we were looking at a very limited set of domains for software. And today there are many different methodologies, many different people using it, but then it was fairly small. We had no idea it was going to take off like it did. So it's interesting, people today say, “Well, you didn't do this.” And you say, “That's right. 20 years later, we didn't do that, and maybe we need to do some different stuff.” I actually think that there's several things that are important now, and I've actually put this out on a couple of blogs as a challenge. One, I think history's role is not to predict the future but to help us prepare for it, so that's one of the things that I talk about in the book. I want a little bit more knowledge of history, not just for history’s sake, but to help people think about what comes next. When I started writing the final chapter, I thought about, “Well, let me see if I can predict some things.” Then I said, “Whoa, no, no, no. Things are going too fast. I'm not going to try to predict what's going to happen at all. But what I am going to try to do is to say, if you understand the history, maybe you can be better prepared for what comes, no matter what it is.” And so I talk about the difference between methods, methodology, and mindset. So methods are things like refactoring, methodologies are things like scrum and XP and DSDM, and then mindset is really an Agile mindset. And so one of the things that I think for the future is, how do we continue to instill and install agility and adaptive leadership into our enterprises at all levels? So I think the main thing is, if you look at the rate of change in the world, it's going bigger and faster, and so what's the solution to that? It's agility, and I don't care which methodology you're using as long as you keep that as an end goal. And then the other thing that I think for a lot of the naysayers is, I understand what they're talking about. I understand how they've taken some things out of Agile and maybe pushed it to the extreme or have different ideas about it, and that's fine. But one of the things that happened with the Agile Manifesto is it inspired people. They were passionate about it. One of the things that Ward Cunningham did, which was a brilliant stroke, is when he put the manifesto up on a website, he gave a place where people could sign up and sign their own names and write about what they felt about the Agile Manifesto. Between 2001 and 2013, 15,000 people signed up, and most of it was talking about better software, so excellence in software development, and treating people like they deserve to be treated. And I think if you look at the core of what came out of that manifesto meeting, it was building better software and treating people better. I think those are the two core messages.
RD You talk in the book about being adventurous in both career and life. What do you think software-wise is the most adventurous move you've done?
JH Well, I think the most adventurous move I made had to do with when I left the safety of a big company in 1980 and took a job with a smaller company. So I went from managing a software development department to the Head of Sales and Marketing for a small startup, so that was a change in job. I went to work for a small company, a startup, when I had been working for a big company. I commuted from Atlanta to Topeka, Kansas for two and a half years. So that was a big change and I was actually in a job that I really enjoyed. And my decision came down to –and so this was kind of an adventurous decision, if you will– my decision came back down to, did I want to look back at some point in the future and regret that I hadn't taken that job? And that's what finally convinced me to make that major change and it changed the arc of my career for the better, but at the time, it was a risky move. But I didn't look at it as risky. I was too young to understand it.
RD And I know you said you didn't want to speculate about the future, but we're in a period of rapid acceleration with all the news about AI. How do you think that's going to change software development?
JH Well, I think the issues are bigger than that. If you look at things like climate change, the impact of COVID, the geopolitical situation, I think that there are some issues that hopefully some new technology will help us with in terms of building the kind of leadership we need to attack some of those issues. And so I hope that we can use things like AI in a positive way. Just like social media, I think there are always going to be consequences that were unanticipated and I think what people are concerned about today is what some of those unanticipated consequences might be and that they might be pretty large. And so I hold some of those concerns, but I also don't think you're going to stop it. It's too big, there's too much money involved. I thought it was interesting, there were some congressional hearings the other day, and some of the tech people were talking about regulation and that they would welcome some of the regulation, but Congress has proven that they can't even regulate social media, so how are they going to regulate AI? It's just a step too big, I think. So I think it's going to be interesting to see how that plays out in the next few years. One of the things that I think though is, if I look back at the ‘90’s, it was the CASE tool era– computer-aided software engineering. And these were tools to help manage all that documentation that we were gathering with the monumental methodologies. And at that point in time, I think a lot of people had in mind that we would kind of do away with developers and programmers. We wouldn't need them anymore. And I see things like AI and no-code, low-code as having that same sort of mythology in the background. And we're always going to need talented people, maybe they don't code anymore or maybe they code some or maybe they code with the assistance of AI, but basically we have to figure out a lot of stuff. I wrote an article about six or seven years ago with a couple of other colleagues at ThoughtWorks, and I had noticed that the technical stack from 2005 to 2015 had exploded. It was a huge tech stack now, and I think deciding what belongs in each place of that tech stack and looking at the various alternatives at each level is going to take people that are really good software developers. So I don't think we're going to do away with software developers, but we're going to need a different kind of talent in some of these instances.
[music plays]
BP All right, everybody. It is that time of the show. We want to remind everyone who's listening that, if you haven't already, our developer survey is coming up, so stay tuned for the results. It should be exciting. And we want to give a shout out to nCod3d. Awarded six hours ago, a Lifeboat Badge for coming on Stack Overflow and helping to save a question from the dustbin of history. “How can I find many useful digits in any given number N?” Well, if you're looking for useful digits in a given number N, we have an answer for you, and we've helped over 16,000 people, so thanks, nCod3d. I am Ben Popper. I'm the Director of Content here at Stack Overflow. You can always find me on Twitter @BenPopper. Email us with questions or suggestions, podcast@stackoverflow.com. And if you enjoy the program, why don't you leave us a rating and a review. It really helps.
RD I'm Ryan Donovan. I edit the blog here at Stack Overflow, conveniently located at stackoverflow.blog. And if you want to reach out to me on Twitter, you can find me @RThorDonovan.
JH I'm Jim Highsmith. And as I tell people, I'm semi-retired, but it's more semi than retired recently with working on my new book. So I've just written a book called Wild West to Agile, and it really is a 60 year history of software development. And it's in some ways a historical memoir in that it's got a lot of stories from my work experience in it, but it's still a history of software development. So it can be found on Amazon, InformIT. I think some of the warehouses are just getting it in stock, but it has been released and should be available hopefully pretty soon.
BP Awesome. All right, everybody. Thanks so much for listening and we will talk to you soon.
[outro music plays]