Chris Fowler, Director of Engine for Call of Duty, tells Ben and Ryan about his path from marine biology to game development, the ins and outs of game engines, and the technical feats involved in creating massively popular games like Call of Duty. Chris also explains why community feedback is so critical in game development and offers his advice for aspiring game developers.
Chris works at Sledgehammer Games (a division of Activision), which develops titles in the Call of Duty franchise. Explore their open roles here.
Want to see Chris’s engine in action? Check out COD: Modern Warfare III.
Connect with Chris on LinkedIn.
Kudos to Stack Overflow user teh.fonsi, who earned a Lifeboat badge by explaining How to execute 1 command x times in java.
[intro music plays]
Ben Popper Rapyd Cloud is a high performance managed WordPress hosting built for dynamic websites and high traffic spikes. With built-in autoscaling, 24/7 support, and top-notch security, all your hosting hassles are covered. Sign up during November and get up to eight months free. Visit Rapyd Cloud now. That's Rapyd.cloud.
BP Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I am your host, Ben Popper, Director of Content here at Stack Overflow, joined by my colleague and collaborator, Ryan Donovan.
Ryan Donovan Hello.
BP Ryan, one of the things I feel like I've said a lot, and I'll say it again because it's fun, is that whenever we bring someone on the show, we always say, “Oh, how'd you get started in software development? What brought you to coding?” 9 times out of 10 they say, “Well, I was really into games. Well, I was playing this game and I wanted to make a mod,” or whatever it may be. So for just so many different people, that has been the thing that awoke their passion and got them started on their road to learning how to code and do software, and then eventually folks end up in all kinds of positions within the industry. So I know you brought our guest today. Do you want to give us a little bit of an introduction?
RD Sure. Our guest today is Chris Fowler, Director of Engine at Activision. I reached out through a friend of a friend and he came on the show and I'm excited to talk today about how the engines work on games that everybody in the world plays.
BP Terrific. Chris, great to have you here.
Chris Fowler Thank you for having us. I'm excited. Big fan of Stack Overflow.
RD So what's your origin story?
CF My origin story is an unorthodox one. I was studying marine biology at the time and I was looking for the next surfing game coming out, because the previous surfing game was called California Games, if you ever played that. And there was a small company in Brisbane which wasn't very far from where I live, and they were making a surfing game, and I just emailed them out of the blue and said, “Hey, can I come and test your game?” And they emailed back, which was a surprise, “Yes!” So I took the train and went up there and I spent the day and I played the game and I gave them feedback and I looked around and I said, “This is a job? You get paid to do this?” and they were like, “Oh, yeah.” And I went, “Oh, this is me.” So then I just switched straight to coding at that point and discovered I could code. In my final year of my degree, I was working as QA at the studio and I did an internship for six weeks, and when I did that internship, I started as a programmer and I've been a programmer ever since. They kept me on as a programmer after that internship, and that was 25 years ago.
BP Nice.
RD Wow. So now you're Director of Engine at Activision, obviously a huge gaming company. What does that mean– the engine at a company like Activision?
CF The engine that I focus on mainly is the Call of Duty engine. So we have a bespoke engine that we have for our game which we release annually, and we've sort of amalgamated our teams and we have a single direction of where we're going with a forward plan over year after year. So you have to balance a creation of features which creates risk and change, along with innovation. And you work with so many people where we write the white papers for how to make games, and so it's an incredible place to be in terms of innovation and the ability to try and try things out new. And every time around release time, which is now, you have that moment when you realize this is a very, very popular game. It always catches you and you're always really humbled by it.
BP Is there a difference between a large software company that's like, “This is our platform,” or “This is the architecture of what we do,” or “This is the code inside of this app,” versus an engine? That word seems specific to gaming. What does engine mean and is it different from the way software might be designed inside of a firm that is supporting a social media platform or a chat app or whatever it may be?
CF In terms of games, engines tend to be sort of a more agnostic piece that multiple parts of other parts like gameplay use. So you create an engine so that you can work on PlayStation, Xbox and PC, and a gameplay engineer does one thing and it does something completely different on all of those platforms but they don't care and they don't need to interact with that.
BP Gotcha.
RD So it's a lot of rendering graphics. Does it manage any sort of networking, any sort of scripting engines, that sort of thing?
CF Yes, very much. So you'll have different scripting engines for different areas that you'll use. There's network infrastructure that's specific to every platform that you've got to adhere to. There's SDKs that they have which have interface tools when you're working with them.
RD So Call of Duty is obviously a very competitive reaction-based game. You're doing a 3D graphic-heavy engine. That's a lot of math all the time and network stuff. How do you optimize it so that people, when they're playing the game, they're getting the reactions that they want worldwide, basically?
CF So there's a lot of elements to this. At the very, very low level, most of the hardware will create super, super, super optimized commands, which are SIMD commands. Say you want to multiply two matrices. They'll create a function that you run and you give it two vectors, whatever, and it will run. It creates code that's maybe four instruction points, whereas if you were to do that manually, it'd become 40, so it's a significantly faster operation. And so especially on a console which is fixed hardware, it works very well for those situations and we utilize that heavily within our engine.
RD But for PCs or other non-fixed hardware, you have to do that for every chipset, right?
CF Yes. You have to make separate ones. You have separate APIs to go through separate different CPUs.
BP Is it an expectation after you release each new engine that then people are going to find ways to hack it or break it or find ways to take advantage of some aspect of it and then that it will have to be modified or that it will become an issue within competition that you think about when you're designing the next one?
CF You always have to think about competitive disadvantage or advantage between consoles and make sure that nobody has an advantage. A really simple example would be if you were on PlayStation 4 and say the graphics wasn't as good so the shadows were lighter, you've actually given a competitive advantage to the older generation because it's lighter so they can see more. So those are the really important things we do. We did a lot of static analysis, so we'll look at an image and we'll actually compare in our tooling to make sure that they stay equal so that there's no competitive advantage. We study input latency year on year and we always look at ways of improving that so that we capture the input from the user at the latest point possible to then process the frame and have their reaction times go, because we are a 60 FPS game, or 120 FPS if you're on the newer consoles or PC.
RD In terms of the sort of input reaction latency, I've heard of some first-person shooter games doing sort of tricks to assume that a shot will go and things will happen. Do you do any of that sort of assumptions in the movement of objects in space, or is it all just real time actuality?
CF That would be sort of client-side prediction, which the terminology I would use is ‘trust but verify,’ in that, yes, the client said it's going to do this, but we send that to the server, the server then verifies that movement, that fire, that shot, all that verifies. We'll then send that back to the user, and if everything matches up, then we just keep playing. If that doesn't match up, we wind back time and then we correct whatever the correct outcome is. So say if you were to shoot someone and they actually shot you first and it took you out and on your screen it looked like you were going to hit, we'll roll back time and then we'll roll it forward to make sure that you get the correct outcome. But in most times, because we are such a high FPS thing, you won't get that timing rollout.
BP I understand what you're saying. Nothing actually occurs on their screen until you roll back time, but just how many milliseconds are we talking about and how would you do that for people who are playing on other sides of the world?
CF This is less than a frame. So our latency, most people are at 60 ping, so we're talking one to two frames. We're not talking seconds rollback, that would be a very long time.
RD I know when people get very competitive about games that frames matter, frames count. So we talked about the graphics. I know there's a lot of ways to sort of optimize the math. There's a big load that the graphics put on computers depending on what your graphics card is. What are the ways that you optimize in the engine to sort of reduce that load for everybody, like you said, not to give a competitive advantage to somebody with a thousand dollar graphics card?
CF So it's a balancing act between fidelity and performance, and we have certain tooling in our game that we will adjust sort of scene resolution dynamically so that if the scene is really busy we'll keep your frames up by dropping the scene res. There's a whole bunch of tools that we do. There's a lot in there, probably too much that I can go into in terms of this, and on the security front, we have a very active security team.
BP It's amazing to me to think about servers on opposite sides of the world that are trying within the span of a couple of frames to understand something, rewind something, and keep that coordinated with a person on the other side of the world. Just that tiny amount of time you have to work and make all of those decisions is kind of astounding so that everybody feels like they're still playing in real time.
RD I read somebody justifying the sort of streaming gaming where the latency between you and anybody else on the internet is generally less than between your keyboard and your monitor. I don't know if that's true, but maybe you can clarify.
CF There's a certain reaction time that a user will have. The stock case is, if you've ever seen the Street Fighter battle where the reaction time of the user was 60 frames a second, they were actually hitting the button faster than we were doing wireless input latency at the time. And so that is sort of the poster child of what wireless input latency had to get to. It's there now with a lot of really, really good work from the first parties, but that was one of those situations where wireless input latency couldn't match the speed at which the button was being pressed because they're physically using a joystick that's hardwired to the machine.
RD Those fighting game folks are the frame counters I was talking about. So what else do you worry about in the game engine? What's the dividing line between the gameplay and the engine?
CF In an engine like ours, we don't serve lots of different games. So our engine's about serving the best thing we can do. We make the best Call of Duty we can because that's what we focus on, so we get less diversion. I'd say the things we focus on are features, so what are the new features we're providing, making sure that we continue with the current features and keep the speed there. It's a live title, so we're doing updates every month, so you have to keep the new content coming in, you need to keep the old content running well, and year on year, it's a nontrivial amount of work to do that.
BP You make a good point there. Ryan and I actually talked recently with some folks who work on the Unity engine, and obviously I've heard of the Unreal engine. What is the difference between making something that you know is going to be used by lots and lots of developers to make many different games versus what you're doing, which is, “We have to have the best engine inside of this one franchise.”
CF So I've worked with the Unreal Engine before and I've worked in other bespoke engines, and I think where bespoke engines really shine is pivotability in that you can be agile. You can say, “I want to fundamentally change something. I have full control of that because I wrote that.” You can pivot quite quickly and change things that are fundamental to a system, whereas if you pick up an engine that you didn't write and that you didn't own and you don't know about, to change that and then have that change from under you is a very difficult thing to resolve over time. So you could change a fundamental animation system and then they change it in an update and all your stuff is broken.
BP So you've been working on game engines for a long time now. What would you say are the most interesting evolutions or the things that you're excited about in the latest wave of stuff you've worked on?
CF Solid state drives. So they're drives that don't have a spinning disk, so there's no seek cost in reaching the content. The speed at which the data can transfer at is at an order of magnitude we haven't seen in computing. And the fundamental change of speed, seek speeds, being nullified to nothing, it changes everything. It's totally different how we think about data layout and think about how we can load. Thinking about loading straight to a GPU and not putting it in virtual memory and then throwing it into physical memory and then passing it to the GPU, going straight there, it's a possibility now. NVMes are faster than some SSD RAM. It's just amazing where we could be. So it's a very, very fun time to be in the gaming world.
RD I'm a PC gamer and I remember before SSDs, memory requirements on games were huge. It took up everything you had. That's an interesting change. And I wonder what's the best bug you've encountered? Because I've heard some crazy ones with gaming, so I'm wondering, working on this big engine, what's the things you run into?
CF I would say it's older ones, I've had some funnier ones, so on some old PlayStation and Wii ones. I did Force Unleashed back in the day, I don't know if you remember that game. That was probably one of the most fun games I ever worked on. It was making Jedi, and you could just do the craziest things we were allowed to do, it was so cool. And in that, we had this move where you could pick people up and spin them around in sort of this, I think we called it ‘maelstrom,’ and then at some point you could throw them apart and they would ragdoll. And some of the outcomes of that ragdoll were quite astonishing in terms of that body parts would get caught on a limb and then they’d keep going, and then this massive mesh is just everywhere, and you're just like, “Where's everything gone?” And then you'd have to resolve it and we had to work with the physics resolver to bring stuff back together because it was just breaking apart. And so you’d end up with these wild things. I think the funniest thing on that game was that the star texture was actually my face. They did put that little joke in there. Someone looked up closely and said, “Check this out.” I went, “What?” and they were like, “Yeah, that's your face.” I'm like, “Great, thanks.”
BP When we're talking games, it's not just your favorite bug, it's also your favorite Easter egg. That's a tradition– you’ve got to leave some Easter eggs in there.
RD So talk about some of the older games you've worked on. Before Activision, you were at smaller companies?
CF Yeah, Chrome Studios is where I started and that's where we did the Championship Surfer, Ty the Tasmanian Tiger, which weirdly enough then became Force Unleashed. And there's surprisingly little code difference between a small little platformer and a Jedi, I'll tell you that. The same code doing the same things, it just has a lightsaber now. That was really fun on that one. It's one of those ones where I actually coded the Jedi to have ML before ML was a thing. It would learn your moves, so if you did the moves more often, it would learn how to counter them better, and so you had to vary your moves. And it would learn over time if it was beating you down, so it would get dumber so that you wouldn't get people quitting.
BP Adaptive to keep you right on the edge of success.
RD That is interesting, because I think computer games have had AI of some manner for a long time. Do you think it's the same sort of AI, or was there a lot of it just sort of branching, faking it?
CF A lot of logical if-else's, I would say. It was I guess a behavior tree, so this reaction, this input, go to this reaction sort of thing.
RD Is there a favorite success or something that people would overlook besides your face on the star?
CF Coding the Jedi was super cool. I'd say my first Call of Duty and that moment when you see it on every billboard and everywhere. You see it in Times Square. You're like, “Whoa, okay.” Understanding the scope of the impact, playing the game and just seeing people. I've had the funniest time playing the game just listening to other players and the way they banter and what they get up to and just seeing that there really is this really rich community of people that just use it as a social platform. It’s just really humbling to know you're part of that.
BP That raises another interesting question which is, has there ever been a time when the feedback from users allowed you to change something interesting, or they found a bug or a side case and then you were able to then incorporate that into the engine?
CF Yes, we're always listening to the community and we're always updating. If there's a live bug and they report it, we'll check it, verify it, fix it. Depending on how we can fix it, it could be fixed within an hour, and sometimes it takes longer depending on what type of fix is required. But yes, we are very reactive to our community.
BP To the degree that you can talk about it, how big is a team working on this engine and what kind of skill sets do they vary across? You have a big team working on the engine and then there’s specialists. What are the different kinds of specialties inside of that?
CF You would have streaming specialists, you would have graphics specialists, you would have SDK specialists, people who are specific to understanding how reads work on a particular platform because they're quite different. There's packaging experts, so how we actually package the content and deliver it to people. There's content for how you own the game or the MTX and all that sort of stuff, so there's a lot of different parts. There's a large amount of speciality that we do try and generalize around with other people, but I think most game companies do the same thing where you have specialists, but you have generalists that can go in between all of them, and you want that sort of cyclic knowledge.
RD How much does the engine touch the physics and how much of a complicator is the physics to the engine?
CF So our physics engine is Havok, so we use Havok as a middleware, and it's a very, very good engine for having different layers of complexity. So you can have different stepping layers so you can have really low complexity that runs really fast, and then high complexity that runs at a different frame rate that you can be way more accurate with.
BP So you mentioned you were able to just send somebody a message, show up at their office, test the game and chart a new course. What would you recommend to folks who are listening who themselves are software developers or work in one part of development already or are just starting out of college? What do you recommend for getting into the industry you're in and for cutting your teeth and learning this kind of stuff?
CF Demos are probably the best thing. And it's not starting a demo, it's finishing a demo. Making a game is 10% start, 90% finish. So it's the edge cases of making a UI, finishing the UI, giving it settings, having a go into a game play, go through a game loop, come out of a game loop, give you feedback, give you an ability to get back in, close that loop, finish everything, complete it. Because gaming is about just keep going at it and keep using your passion to just finish what you need to do and having the grit to do that. And don't give up on rejection. So a good example would be, for my internship, when I first asked for the internship, even though I worked there, they said no. And then I went to my boss, he went to his boss, he went to his boss, and they tried it again and I was able to get in there. But I did have that. Don't let rejection mean it's a stop. Keep going and build a good team of people. If you are making it a demo, make sure you're working with a team of people. You can't do everything yourself.
[music plays]
BP All right, everybody. It is that time of the show. We want to shout out someone who came on Stack Overflow and contributed a little bit of knowledge or shared a little bit of curiosity. “How to execute 1 command X times in Java.” Thank you to teh.fonsi for coming in with a great answer on this question and earning yourself a Lifeboat. Almost 60,000 people have viewed this question and benefited from your knowledge, so congrats to teh.fonsi for figuring that one out. As always, I am Ben Popper. I'm the Director of Content here at Stack Overflow. You can always find me on X @BenPopper. If you want to come on the show, if you want to chat with me and Ryan, if you want to suggest a topic to hear more about or hear less about, you can email us, podcast@stackoverflow.com. And if you enjoyed today's program, the nicest thing you could do is leave us a rating and a review.
RD I'm Ryan Donovan. I edit the blog here at Stack Overflow. You can find it at stackoverflow.blog. And if you want to reach out to me with episode ideas, blog article ideas, hot takes, you can find me on LinkedIn.
CF I'm Chris Fowler. I'm the Director of Engine on Call of Duty. Look me up on LinkedIn, there aren't too many of us. If you want to know more about engines, I'd recommend blogs and I would recommend jumping into something that's already available and start tinkering, because the best way to learn is to make.
BP Terrific. All right, everybody. Thanks for listening, and we will talk to you soon.
[outro music plays]