Ben talks with Doug Seven, a director of software development at AWS and the GM for CodeWhisperer, an AI-powered coding companion, about his career building dev tools and how he hopes AI will give people more bandwidth for creative work.
CodeWhisperer is an AI coding companion trained on Amazon and open-source code that gives you coding suggestions in real time.
In addition to being general manager for CodeWhisperer, Doug is also the GM for Amazon CodeGuru Security, which uses machine learning to detect security policy violations and vulnerabilities.
Connect with Doug on LinkedIn.
Asked and answered: user Manodnya B won a Lifeboat badge for answering Cannot find the Start Button under CodeWhisperer in AWS Toolkit.
[intro music plays]
Ben Popper Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I'm your host, Ben Popper, Director of Content here at Stack Overflow, and today I have a very exciting guest. As you know, there are all kinds of amazing things happening in the world of software development, and one of them is that AI systems are increasingly our buddies along the way, helping to write code, to suggest things, to debug things. Today, we are lucky enough to have Doug Seven, who is the GM at Amazon for the CodeWhisperer product, on the podcast to talk about what they're doing and how he sees this whole burgeoning field of tools. So Doug, welcome to the Stack Overflow Podcast.
Doug Seven Thanks, Ben. I'm super excited to be here. If there's anything I like talking about, it is CodeWhisperer, so let's get into it.
BP We'll have to speak very quietly. So just for folks who are listening, give a quick background. How did you end up where you are as the GM of a product that's for developers by developers.
DS So I'm actually only about a year into my tenure at Amazon Web Services. I come from a little bit longer background in some other technology companies working specifically on developer tools, so having worked on various IDEs and developer productivity tools, and then having transitioned over the last number of years to building large scale cloud services. This is sort of the marriage of those two worlds, if you will, building developer productivity tools that are enabled by these large-scale AI cloud services.
BP Do you mind dating yourself a bit? Do you want to tell folks what your first IDE was or where you wrote your first line of code?
DS Oh man, it's been a long time. I grew up with a RadioShack TRS-80 computer. I stored my applications on cassette tape. It was very exciting stuff.
BP Nice. I don't remember what episode it was but we were talking to somebody at a big tech firm about databases and storage security and they said they still keep it all on tape somewhere in a vault. The important stuff still goes to tape and gets locked down inside of a mountain silo somewhere.
DS I did spend a little bit of my career as a tape librarian doing disaster recovery and backup back in the day. That was a long time ago.
BP Yeah. It's better than bit rot. They say for the long term, if you're going to put it in cold storage, you want it on tape. Very cool. So since you said you've been with Amazon for a year, does that mean that the CodeWhisperer project predates you or did you come in at its inception? Put me in relation to where you started on the project and when the project started.
DS When I joined, CodeWhisperer had already been started. In fact, AWS had already done the preview announcements back in June of 2022, so a few months before I joined they had released the preview. It was a closed preview, so you signed up and you got access to it and were able to use it. And then shortly after I joined, we took it into a public preview back in November of last year before we finally made it generally available in April of this year.
BP I think that there was a bug bounty program we did a podcast with with Amazon, and at the time they were talking to us about something called CodeGuru or bug guru and it was a similar idea, but it was for security. Are those things in the same family? Are they related? How big is the CodeWhisperer project and does it have different tentacles?
DS They're related. In addition to my role as the general manager for CodeWhisperer, I'm also the general manager for CodeGuru security, which is really a product designed around the security aspects of software development and finding and identifying security vulnerabilities in your software.
BP So from a fundamental level, as someone who started back in the Tandy RadioShack days, where do these things fit into the software development process? What is the best way to think about them? Are they autocomplete tools? Are they pair programmers? And as they become more and more capable, where should they sit in relation to the software developer, the product manager and the organization?
DS It's a fascinating evolution of these tools. Having worked on these developer productivity tools for a long time and sort of being part of the evolution from the early days of things like autocompletion where I type a class name, I hit a period, and all of a sudden I get the methods and properties that are part of that class– which is really just a dictionary lookup of what's available for that class– to now having an AI who's sort of omnipresent, if you will, or just kind of there with me, not just responding to my actions. It's not just, “Oh, I hit a period and I get the methods and properties,” but actually trying to predict what I'm doing next and suggesting to me, much like if you and I were to sit down and code an application and you're like, “Hey, Doug, why don't you try this?” That's really what it's about. This idea of an AI companion is a little bit more evolved than thinking about something like autocompletion, but that is sort of the nature of it. It's the evolution of these tools. As technology advances and as you can do more things, the tools advance with it and become bigger, better, faster, stronger.
BP So in the way that it's deployed right now for users, do they have the option to have it kind of like an autocomplete, like I start on something and then it finishes the sentence, but also there's maybe a more robust version, like you said, where I say, “Hey, I'm thinking about this feature. Can you build me maybe a rough sketch of what the backend should look like?” When people are interacting with CodeWhisperer, what are the different modalities they're in?
DS So right now there's basically two ways of interacting with CodeWhisperer. You don't necessarily make a conscious choice on how to do this. It's more about what you're doing at the time. In one way, as I'm writing code, CodeWhisperer is constantly paying attention and we have an ML classifier that runs on the client side that tries to decide when is the best time to make a suggestion for you based on what you're doing and what your likelihood of needing and wanting that suggestion is. So as you're writing code, it might suggest how to complete that section of code, or for example, as you start writing a function signature, it might suggest how you finish that function signature. It might suggest how you finish the entire function as well. And so some of that is just the context of, as I'm writing code, CodeWhisperer is trying to sort of accelerate me along with what I'm doing. The other way it works is through natural language where I write a comment that describes what my intent is, which is very common in software development where I write a comment to describe a function or a class.
BP Coming back to this part, it's going to make everything work better, faster, fix this later.
DS Yeah, exactly. And as you write the comment, CodeWhisperer understands the natural language comment that you've written and it says, “Okay.” If I say, “Oh, I want a function that does whatever,” then it goes, “Oh, I know how to do that,” and it writes that function for you. And so the idea from a productivity tool standpoint, if you think about our goal, our goal is time. We want to reduce the time it takes to do things, particularly the most mundane and boilerplate things that we want to free you up from so you can use your mental energy, your cognitive energy, on creative new kinds of things. So if I can take a small task of maybe writing a function that might take you a few minutes to do and make it a one minute task, if I could take that three or four or five minute task and make it a one-two minute task, I've saved you a bunch of time. And if I do that enough times throughout the day, that accrues to a pretty significant time savings.
BP So for the developer who's listening, when you say it's sort of familiar with what you're doing, it's familiar within an IDE, within a repo, within a branch, within your company's code base. How does it know what you're working on? And then once we answer that, the obvious next question is, how do you think about privacy and security and governance? Is it when folks are working within the AWS environment and so they have all those guarantees? When somebody is at a private institution and they have proprietary code, how do they allow CodeWhisperer to help them but feel comfortable doing that?
DS Ben, you packed a lot into that question. Let me try to break it down. So in some ways we sort of personify CodeWhisperer a little bit to try to make the analogy. So if you go get the CodeWhisperer extension, which is the AWS toolkit, that's where you find CodeWhisperer and you install it, the analogy is it's like having a developer come sit next to you and they understand 15 different programming languages and lots of different frameworks and they can help you with a bunch of different tasks and do a bunch of different things. And so immediately you're getting this productivity gain of, “Oh, I'm working on a Java app and it understands all these different frameworks and it can write code for me and help me along.” One of the announcements we recently made was for the ability to customize CodeWhisperer to the idea that if you take the analogy of the personification, then CodeWhisperer in that context would be like having a new hire developer join your team. They understand the basics of software development, they know how to write code in lots of different ways, but they don't understand your code that’s in your organization that’s private and proprietary. And so the ability to customize CodeWhisperer would be like turning that developer into a more veteran developer. Now with the customization capability, you can say, “CodeWhisperer, go look at this code that I have, get smart about this code that I have so that you can start making suggestions that align with how we write code, or the internal libraries we use, or the internal APIs we have.” And then those suggestions start to look more like the code within my organization, so now it's the difference between a new hire developer and maybe a three-year veteran developer in my organization.
BP That makes a lot of sense. And so the way to do it is within the AWS ecosystem, and so that's where things like security and governance and privacy all come in. I like the idea that you said that you can familiarize it with what you're working on, but it's sort of baked into an ecosystem that already has all those terms and services and agreements in place.
DS Yeah, it's an interesting way to talk about it. So there's two sides to that– there's CodeWhisperer as an AWS product and so it comes with all of the promises that come with working with AWS. And in fact, for CodeWhisperer in particular as our very first generative AI product, we took a really early position on things like responsible AI and what we wanted to do in how we built these tools. And so for example, one of the features of CodeWhisperer is a reference tracker, meaning if CodeWhisperer makes a suggestion and that suggestion has a high level of similarity to some of its training data, and that training data is from a licensed open source repository, we'd say, “Hey, this code matches this open source repo. It's under this particular license and here's where you can go find it,” and you can make a decision about whether or not you want to use it. That's really important in terms of how developers work and how developers work with and leverage open source projects. And so in terms of that ecosystem, I think you get all the promises and guarantees that come with an AWS product and our position on trust and security and responsibility. But CodeWhisperer itself is capable of working outside the AWS ecosystem. If you want to write code for AWS, CodeWhisperer is really good at it, but a lot of times developers have to work across different clouds or different platforms. Sometimes you're working partially on premise and partially in the cloud, and CodeWhisperer is capable of generating code for all of those environments.
BP Nice. That makes a lot of sense, and I'm glad you brought up the issue of licensing. We have OverflowAI and sort of the works and it’s a similar idea. It's going to show the links back to the questions that the synthesized answer came from, that way you can also reward the human who produced it and the code that people share on Stack Overflow is licensed in a certain way, and so you can decide if that works for you. So you mentioned that it was your first generative AI product. Under the hood, does it work in a way that's similar to other Gen AI products that we might be familiar with? Which is to say, you've trained a large language model on a lot of code, you can speak to it in natural language and it can both speak back to you in natural language and also write code? Is that how it works?
DS Yeah, that's exactly right. We've built a large language model. Coincidentally, we call it the CodeWhisperer model. It is trained on billions of lines of code. We've augmented it with best practice code for how to work with AWS so that it's really good at doing things for AWS. So that's exactly how it works. And the idea here is that we wanted to build and design a model that's capable of generating code quickly so that we could have that in IDE experience where you have really low latency, sub-second suggestion time, things like that, and to be capable across a number of different languages.
BP Yeah. I was talking to somebody the other day from Replit and they were sort of saying there is a balancing act between a big foundation model and a small model, and when you're trying to do something in the autocomplete world, you need to match the speed of thought or otherwise people will have typed past it. And so that low latency is obviously really important.
DS Exactly. There's definitely a difference from a developer's perspective of what I'm doing in the IDE and if the response time is fast enough versus if I'm having a request response type of exchange like a dialogue. I tolerate latency a lot more in a dialogue like that than I do in my IDE, so that sub-second latency time is really important.
BP Is there almost a new build time experience where you used to say, “I'm working on this, but I can't quite get this right. I'd really love a feature that does that. Can you come up with some ideas?” and if you come back in ten minutes or you come back in two hours or you come back the next day, it's built out a suggestion for you?
DS I think we're going to see a lot of evolution in the Gen AI space towards things like that, where you have this concept of AIs or agents that kind of work on your behalf where you can send them off to do long-running tasks and come back to them later. I think that's going to be a very commonplace thing in the future.
BP Cool. I guess the ability to work through a problem and iterate on it step by step seems to be one of the things that large language models actually inherently improves their performance. If they do chain of thought reasoning and they're their own critic, they tend to do better than their one shot zero-shot suggestion.
DS Correct. It's fascinating to see these things evolve and how they work and the idea that you can sort of take the results from the language model and feed it back to the language model to improve and add more context to it and the idea that you can iterate an idea or you can have it execute something and then give it feedback on the results and it goes and does it again. The capabilities of large language models are fascinating and I think we're barely scratching the surface of some of the different use cases and things we're going to have.
BP All right, I'll ask you a question that you probably won't be able to answer, but we saw Vision GPT recently and one of the really interesting things there was, “Oh, I love this website. Let me take a screenshot. Can you code me up a site that looks like this?” Or I heard an academic talking about it and they had written a diagram– “Oh, could you extract from this the math equation in a certain sort of arcane academic language?” and it's able to do it. Do you see multimodal models in the future for CodeWhisperer and Guru?
DS These ideas are ideas we experiment with in a lot of different ways. Whether or not we package those as part of CodeWhisperer or part of something else is to be determined, but I think these are areas that we're investigating constantly. We have a big research team, a big data science team, and we're always looking at the advancements in large language models and the different ways that we can use them to make AWS customers more successful. And we have a lot of great ideas that we're working on that I think will be fun to share in the future. Whether or not they're labeled CodeWhisperer or something else, I think is to be determined.
BP Right, okay. So I guess on that note, does this tie back in any way to the Bedrock products or are those separate different ideas inside of Amazon working in this space?
DS Functionally, they're very related. The CodeWhisperer team and the Bedrock team work very closely together because we're all working in the same space of these large language models and how we're operating. So there's a lot of collaboration and sharing that happens, particularly on the science side and how we're building these things and working with them.
BP You mentioned that it speaks 15 different languages and then obviously understands a lot of frameworks. Is that something that's evolving? Have you added a number of languages over the last year and do you see it multiplying from there?
DS It's definitely something that's evolving from when we first announced CodeWhisperer back in 2022 to where we are today. We started with just a few and we've added a bunch. One of the challenges that we face right now is there's a bit of a disparity between the languages in terms of their capability, and it's partially by design and related to our responsible AI approach. And so if you were looking at CodeWhisperer last week, there would be five languages that have the most support in terms of that they can do single line completions but they can also do function completions. They can give you 20-30 lines of code at a time. And then the 10 other languages really only ever gave you one line at a time. If you're working in SQL or Go or Rust you would just get one line at a time. And that was mostly us constraining the language model to have a smaller output size until we could make sure that our reference tracker was doing the right thing in terms of identifying the potential of open source code and the suggestions so that it can notify you of that. And so as we get to a point of confidence with our reference tracker capabilities with a given language, then we expand the capability of that language. So actually just this week we did a model refresh, if you will, a number of different capabilities of that model refresh, but one of them was that five more languages moved from single line completions to function completions: SQL, Kotlin, Go, Rust, and PHP. So if you were using CodeWhisperer a week ago, you were getting one line, and now today you're getting full function completions. So we're sort of always evolving and trying to expand what the product is doing. I think there's a question that we are always asking about how many languages do we need to support, given that we want to support them in the right way from a responsible AI approach. And so we've currently limited it to 15 but the model is actually capable of many more languages than the 15 that we expose, but we just want to be really sure we're doing it the right way.
BP Right. And obviously AWS and Amazon are super global companies. Is it available in multiple languages or only English for now? This was another thing I was listening to an interesting talk about, which is that it just seems like most big foundational models, and maybe that's because of the corpus on the internet, are trained mostly in English data. They can actually speak other languages, but to your point, they were saying that if you wanted to speak French, you could up its accuracy there by studying on some French data and get 10%-20% better, but the English performance would go down 1 or 2%, and is that something you want and so it's sort of a trade off there. But do you think about how you would expand this globally? Obviously Amazon has customers.
DS Oh, we're always thinking about them. We're always listening to our customers and getting feedback from them, and partially it's where do you start and where do you go to. So we started by saying it's English only, and that's more from a support standpoint. We validated that it works for English and it does all the things you want it to do in English. Very quickly as soon as we released CodeWhisperer, we saw lots of things showing up on all the various social media sites with people like, “Here is it working in Spanish.” “Here's it working in Japanese.” Because the models are very capable because when you train on such a large corpus of data, you have a lot of variety in that data and so it's really easy to discover that you support things or the model is capable of doing things maybe that you don't claim support for, which is sort of a “Yes, it can do it. It's not something we're backing.”
BP The emergent properties are the fun and the scary part. What can it do that we don't know about? Can it play chess, Doug? Do you know? Have you asked it?
DS I haven't tried to play chess, but I will say just prior to re:Invent last year, I was playing around with it and I ended up having a conversation with CodeWhisperer via comments in code. And so I would write a comment and ask a question, CodeWhisperer would answer the question, I'd ask a follow up question. So again, not a use case we designed it for, but it's fascinating what these models are capable of.
BP Is it a snarky or respectful co-worker, I'd like to know. No, I'm just kidding.
DS So far, quite respectful actually. So far.
BP Yeah, easier to work with than your average developer. Great. Well, is there anything else you'd like to touch on, whether that be something we didn't talk about specifically in terms of a value prop or something that's a little bit future looking that you want to hint at a roadmap or send somebody towards?
DS I think the thing I'm most excited about, and this is sort of more of a point in time. What I'm excited about right now today about CodeWhisperer is the evolution we're making with the model. Like I said, there's this evolution of supporting these languages in a bigger and better way, but even more so this ability to customize CodeWhisperer to understand your private code and respond in a way that aligns with how you work. Every large company that I've talked to since the day I started has been asking about this ability of how do I get CodeWhisperer to understand my code. And some of it you can get through context, but context is limited, and the idea that we can actually provide a way to customize CodeWhisperer to understand your code and respond in that way is very important and I think we're going to see some tremendous results that we've already seen just through some of our beta customers. We had Persistent Systems run sort of a cohort study where they had people using CodeWhisperer to build an application and they had people using a customized version of CodeWhisperer to build the same application, and they saw about almost a 30% productivity gain with the customized version of CodeWhisperer. So there's definitely value in doing that. It's super exciting for me.
BP It makes so much sense. At every big company it's a hairball of tech debt and architectural choices and dependencies, and if you know all those things and you're the veteran there, you're going to solve things a lot faster than if you're a new hire. So it makes a lot of sense.
DS Right. When everybody builds a wrapper around something, they build a library to abstract something, and if your code tools don't know those things then you're never going to get suggestions to use those things and you start to get more and more divergence in your code base. So the more we understand that, the more consistent that code base becomes.
BP Right. There's an interesting sort of set of ideas to work through which is, do I want this system to learn on all the code that's written inside of my company or the code I think is best? Or from all the engineers who have contributed or the 10 that I think [are best]? Because garbage, in garbage out. It's going to learn the good things and the bad quirks as well. So it's an interesting idea of when you want it to get to know your system, do you also want it to optimize maybe on top of that or give you feedback?
DS Those are the exact kinds of questions and challenges we got from our customers. And so one of the ways we did this is we said, “Well, you can point CodeWhisperer at your repo, or multiple repos, and it'll learn from all those. Good or bad, whatever is in there it's going to learn from, whether that's a GitHub repo, a GitLab repo, whatever, it's going to go look at that code and learn from it.” Or you can curate that code and say, “No, I'm going to go find the best practice code. I'm going to find the good implementations. I'm going to go get this stuff. I'm going to go put that in a secure S3 bucket and point CodeWhisperer at that and it's going to learn from that.” So it's a really great point to say, “Hey, not every line of code we've written is great, so we really need to maybe curate that a little bit.”
BP Can you say, “Can you go in and delete all the code we don't need? Let's talk tech debt here. Go in and take out all the cruft and let's make things a little more readable and a little faster, a little less memory intensive.” So that'll come someday.
DS Oh, yeah. There's a ton of places we want to go with things like supporting refactoring or application transformation or some of these different things that are these really arduous jobs that developers have to take on that aren't fun. They're not glamorous, they're not glorious, but they're necessary, and the more we can speed those up, the better.
BP I mean, you're talking to a guy who used to write SEO content for a living, so if the AI robots want to do that, be my guest.
DS Fantastic.
BP Awesome.
[music plays]
BP All right. It is that time of the show, everybody. Let's shout out a Stack Overflow question, someone who came on and helped spread a little knowledge. Manodnya B wants to know, “I couldn't find the start button under CodeWhisperer in the AWS toolkit,” but luckily somebody came on and solved the issue for them. They're using IntelliJ here. So you’ve just got to update to the latest version and the startup button will appear. Turn it on, turn it off again. Usually works for me.
DS Yeah. Definitely make sure whenever you're doing this, make sure you have the latest IDE and the latest toolkit. It's always the best practice.
BP It's always the best practice. All right, everybody. I am Ben Popper, I'm the Director of Content here at Stack Overflow. You can always find me on Twitter, or call it X, @BenPopper. You can email us questions or suggestions for the show: podcast@stackoverflow.com. And if you like the conversation, leave us a rating and a review.
DS I'm Doug Seven. I'm the General Manager for Amazon CodeWhisperer. And the best place to go check it out is our CodeWhisperer website on aws.amazon.com/CodeWhisperer.
BP Sweet, we'll put it in the show notes. Thanks for listening, and we will talk to you soon.
[outro music plays]