We chat with Paul Ford and Cassidy Williams about the experience of using GitHub Copilot, an AI system that writes code for you. It was trained on millions of lines of code written by humans, but how close does it come to a living, breathing programmer? Well, if you're looking for regex, meet your new best friend.
You can find some fun video of Cassidy putting Copilot to the test here.
If you want to take the Jamstack survey, check it out here.
Our lifeboat badge of the week goes to Andomar, who answered the question: Will multiple calls to `now()` in a single postgres query always give same result?
Cassidy Williams When you think about like the crane, the construction machine, back in the day, there might have been people being just like, wait, a big strong person should be able to lift this heavy rock. And now this machine can do it. It's kind of similar but with, you know, information and stuff. A human could probably write some of these. But then there's some things where just like, oh, the machine is doing it. Oh, the machine is doing it better. We could use these tools to do cool things that we would normally do on our own. That is the form of embracing that I am taking.
[intro music]
Ben Popper. Cockroach DB is the only bug you'll ever love. Because it's the only one you don't have to worry about. As a low touch SQL database that automatically handles scale, operations and uptime. Cockroach DB lets you focus on developing. Get your free cluster and a free t-shirt at cockroachlabs.com/Stack Overflow.
BP Hello, everybody! Welcome to the Stack Overflow Podcast, a place to talk about all things software and technology. I am Ben Popper, Director of Content here at Stack Overflow. And I am joined today by some wonderful co-hosts Paul and Cassidy.
CW Hello!
Paul Ford Ben! Cassidy! Oh my god!
BP For people who don't know, just quickly say 'hi, this is who I am and this is where I work' because maybe they don't know, maybe they're not longtime listeners, super fans of the cast.
PF Oh my god, so excited for some new listeners. Welcome. Welcome. Not that we don't also love the old listeners. But they're not old.
CW No, just the experienced. Yes. Yeah.
BP Paul, keep digging.
PF Why am I even trying? It's a disaster. Cassidy tell them who you are.
CW I am currently, actually I just got promoted recently. So I'm used to saying my old title now. Now I'm Director of Developer Experience at Netlify.
PF Oh you're a DDE?
CW DDE. DDX. Experience.
PF Ohhhhhh.
CW Watch out.
PF What does a DDX do?
CW So a lot of my job hasn't changed a lot, except that people report to me now. So that's it. That's that's pretty much it. But I'm focusing on the developer ecosystem at Netlify. And so mostly working with open source developers and building demos and blog posts and content and helping others do the same to make it easier to build not just on Netlify, but on these web technologies in general.
PF I'm just imagining, I'm guessing you're very screamy boss, just you know, get in here!
CW Suuuper strict. People fear me.
BP Throw the mug across the room, break it on the wall, just you know.
PF Look at this feedback form! Look at it! You know, talking from your throat.
CW Bring out the mom voice, which is like, listen to me. That's how I assume most mom's speak. I don't know.
PF No, it's actually you know, I've got two kids. And there are times when you start to talk from your throat, like get down here. And it's it's terrible. You're like I promised myself, I would never talk with my teeth clenched to my children. But no, it does happen. [Cassidy laughs]
BP Who are you, Paul? Paul, who are you?
PF I am the co-founder of software and consulting firm called Postlight. And I am, I recently did a career transition, I got promoted in the other direction. We named a new CEO, Gina Trapani. I am the ex-CEO, I'm back to being co-founder. And I'm working on ways to grow the company, mostly around climate and climate related work. We actually have a lot of climate work in the firm. And I want to see if I can build that into more of a discipline. So that's what I do all day.
CW Awesome. And Gina is amazing, too.
PF I'm a fan.
BP Paul you did the CEO step back. So now you're working, this is the classic Bezos. I'm stepping back into work on climate change, and I'm going to space, that's what you do. Right?
PF Well, nothing, nothing is better for the climate than rockets shooting into space. Yeah, no, that's exactly right. The firm is growing and stable. And so it's a good time. I like chaos and mess. And working with my my co-founder who also likes chaos and mess and we're all out of chaos and mess. We're now like, stable and growing and doing well. People should definitely apply and so I'm off to you know, I'm not going anywhere. But I'm gonna go find some, gonna go find some mess.
CW Come to my house, you'll find plenty. [Paul laughs]
PF I can help you remodel. That's exactly right. I love mess!
BP So I have two things I wanted to chat this week to kick it off. I wanted to talk a little bit about GitHub's Copilot. So I remember when GP3 came out I made the joke like we're getting really close. I still you know consider myself to be a very beginner and frustrated developer. So I just say I would just talk to the machine hey, could you make me a website I want a button over here know a little to the left, people fill out the form, it goes into the database. And you know, there were some great examples that people were doing when they sort of just told it. Hey, could you make me sound like this? You know, give it a vague idea. And now with Copilot, basically, yeah, you say like, I'd like a function that does X. And the machine learning system that has been trained on millions and millions and millions of lines of code says, Yeah, it could probably look like this, it writes it out for you. And then I guess you can either use that directly, or you can edit it a little bit. High level, what do we think about this, and then we can dive into some of the sort of things people have found in the week since it came out.
CW So I've been playing with it, and did a whole stream on it recently. And and I do like that, like, let's just say I'm building a counter component where you click a button, and the number goes up and down and stuff. If I do function counter, it will write the entire component for me. And I don't have to deal with anything. Little things like that are really, really nice, because it's boilerplate code. And so I'm able to write a function that adds two numbers, and it'll write the function. And it was kind of fun to experiment. I can share a just in the show notes of all the different comments that I would write and then the resulting functions that came from it. That being said, if you start doing anything more complex than that, that's when you start to say, okay, you need to calm down Copilot. Like, it's not gonna take our jobs anytime soon.
PF I mean, I started looking at I haven't used it yet, but I'm watching, watching how people reacting seeing some of the videos of it. And I'm like, this is just because GitHub was never able to crack the search for code problem, like searching GitHub for any specific anything, is actually kind of hard. If you're looking for exemplary code, you can usually find something but it's not like it's not one to one, it's it's hard across languages, and so on. And so it feels like they crack that, which is just like, I'm gonna give you something that looks roughly like the thing you're talking about, like they they got to wait for people to put the parameters in, that they need in order to retrieve some code samples. But it's either really early days, or kind of a nice idea that may or may not go somewhere.
BP Where does it fall off Cassidy, like where does it start to break, you have some examples of things you asked it to do? And it just what it returned? Either didn't work or was leading you in the wrong direction? I mean, is it like kind of close, it's like half baked, and you could get it the rest of the way?
CW I think it's actually very aptly named Copilot because you are supposed to be the pilot, and you're supposed to write most of the code. And then it might suggest other things kind of like, with the whole Gmail auto suggestion thing. You're supposed to be writing the email, and it might suggest a few words after that, I think that's mostly what it's supposed to be. And it starts to fall apart, if it's not entirely sure what you're doing. And so I don't know if any of you know the meme, like 'haha business'. I decided to make a function called haha business and I just wanted to see what would happen. And so it made a variable Haha, then I made a variable business, then another variable, haha, then another variable business. And it went on, I just kept hitting Enter, because I wanted to see how long it would go. It went on for like 50 lines before I gave up. So there are little things where it just doesn't know what you're trying to do.
BP Does it have like holistic awareness, you're saying like understand what you're trying to do? Like, obviously add at a counter, add a clock, edge numbers, but it doesn't have a sense of like, what the greater scheme of things is like what it all together is supposed to be or does it try to suss that?
CW It does try to suss that out. So like, there, there were a couple things where, for example, I wrote a thing where it was just like, returns true if a string starts with three letters, and then returns true if a string ends with three numbers. And then I did like one that validates both. And it did combine the two functions. And so it was pretty aware within the file, which was interesting.
PF It's really cool. Like, I bet stuff. Like I want to download a web page, right? Like I bet it's pretty helpful there.
CW Yeeaahh...
PF Nope? Nope. Oh, boy.
BP Well, like, I'm gonna speak from I know, maybe this is a you know, silly thing to say. So if it's a bad question, we can reframe it. But like is this in some ways kind of like NPM install and with add a counter, it's like, there are simple things, you can just pull off the shelf. They're not complex. They're like what, you know, little things that you don't need to remember that it's great for plug and play kind of the way NPM is, now Copilot can do that for little functions.
CW It's really good for utility functions like that. Yeah, when you when you just want to get a simple thing, the probably my favorite part about it was that it could do regex for me. And so for example, I was able to write a comment that said, a function that validates if a string starts with three letters, then has three numbers then has three letters again, and it wrote the regex for me. And so little things like that where I don't have to like check it against my own thing. That's, that's pretty nice.
BP Because that's the thing, right? Like regex are great and nobody has memorized them. Like there's something new, you're always going to be googling and going to Stack Overflow, right?
CW Because you write it once and then you never touch it again.
BP Yeah, so for things like that makes a lot of sense. And I guess just to clarify, so like, I'm going to talk a little bit about where you get into like some of the areas where It goes off the rails. And this was what was reminding me of NPM. With NPM, you're just grabbing a package. And that package has some capabilities that then you can like, utilize in what you're writing. Is that how it works?
CW Correct, yeah.
BP And so with NPM, I know Paul and Sara and I had discussed this, like, sometimes, people were putting malicious things in there, or the things in there didn't belong there weren't properly licensed. And that was some of what I was seeing with this. It was funny, it was like, if you just sort of ask it to do anything, it'll just put like the new like open license, because it seemed that a bajillion times. Right, like, So then the question was, well, should it be taking this open source code and giving it to people who might be writing proprietary things? Do you have, do people have thoughts on that? Or it's kind of like, eh.
CW So depends on how you use it, for like writing a little utility function, like, again, a function that adds two numbers or something, that's not going to be under any particular license. That type of function has been written probably millions of times by developers. And I think that's, that's, again, where you should be using it. What was interesting as I was playing with certain things, like I tried to make a Go function, I don't know Go, but I was just like, now I can pretend I do, too.
BP Look at me, mom, I'm a go developper!
CW And it was interesting, because occasionally a comment would come up that credits some original author somewhere. And it would occasionally be like copyright, Jake something, and I wouldn't know who this person is, and you'd look them up. And they're not even like that active of a GitHub user or anything like that. But that's just how it was. And so it was interesting to see it occasionally credit people. But again, I don't think that's the direction that you want to take it in, you want to take it in the direction of, I want to write a very simple utility function that just does this one thing so that when I write my nice big component library or architect my whole system, I can do that. Because I don't have to write the little basic utilities I have to do over and over.
PF I mean, it's still in this very nice state, right? And know that like, look, when General Electric works with open source code, and the lawyers who review the licensing on everything in order to limit their liability, are they going to have an opinion about Copilot like, sure, that's going to be something that, you know, lawyers got to get paid, you know, they're gonna sit in that room, and they're gonna be like, well, now. You know, Stack is actually a good example of how this ends up working, where it's like, you clarify the licensing, and people cut and paste it anyway. And then you just kind of go forward with your lives.
BP I mean, I like the idea that maybe if it was smart enough, they could pull it in and say, like, if you're going to use this code, you're gonna need some attribution, like you'd have to name and thank the person or if you're going to use this code, just so you're aware, there's a license, and you might have to pay for it. If you do commercial use, like if it could authentically tell you like what your rights are. And you know, or, you know, the way the the way to use it, that'd be--
PF GitHub knows all the licenses of all the code, right? Like they've put those like they have the drop down for what license things are under. So yeah, sort of same as Creative Commons. And so licensing actually is kind of clear. I think there's this moment where this happened with Creative Commons, when people started to see their photographs used in advertisements that they had given away, because they've given them away with the intent of contributing to the commons and things being really positive and right, you can use this as the cover for your weird indie album that you did in your bedroom with a synth. And then suddenly, it's on the side of a bus and they go like, whoa, wait a minute, that that would i would have gotten 1000s of dollars for that normally, right? E
BP Excuse me, a large multinational automobile manufacturer that is not--
PF That's right. That's right. And then it's like, well, actually, according to the rules, and then typically what will happen is like the automobile manufacturer will give a, you know, some sort of honorarium. And the problem no longer is on Twitter all the time. And so it's just like, we're now in that zone with code as opposed to other artifacts. The only thing with code is we've got licensing kind of worked out for how what people can do with code under certain licenses, like that is really well understood, we'll fall back into that world. And I don't I just kind of doubt that the great licensing uproar over Copilot will have these huge legs in our industry, because our industry tends to go like yeah, okay, after a certain amount of time.
CW I just want to emphasize it's a lot like the Gmail autocomplete. Nobody's complaining at gmail about how it finished its sentence of how are you today, or something like that? If you use it in your code like that, there's, it's there's nothing to attribute because it's just, it's just autocompleting.
BP Although I do have a lot of, I hold a lot of anxiety about like, the ones where you respond, you're like, awesome, looks great, exclamation point, or whatever. And like, I'm like, to what degree do people know that like, I just autocompleted.
CW That didn't sound like Ben! He never uses exclamation points.
BP I don't know. Sometimes I feel a little stressed out about sending and receiving emails that like we're basically only entirely written by Gmail, that kind of, I guess--
PF It's okay. Because it's better than not replying. Like, it's just sort of like, Oh, you know, I actually don't mind a two word reply. I mean, it would be you know, I think the fear is that, to stretch that that metaphor, would be like, the reply would be, 'It's okay. I love you!' or like, 'It's me, Cassidy!' and like someone who clicked that, like I--
BP You get a personal email and then you get a drop down. You're like respond sentimentally, humorously and then you just pick one and it goes, it really goes goes all at it right to, you know, parent couple paragraphs of responses.
PF Google, obviously, because there has not been a huge uproar in social media about it has figured out how to intelligently filter and moderate the output of that model, right? Because it doesn't say 'I hate you'. And it doesn't, it doesn't say weird swears or racist things. So they obviously did the work, which is is wild because I thought moderation at scale was impossible, but they obviously did the work to filter those replies so that they wouldn't create a lot of drama, because when the only time I see the screenshots are when it's just kind of nonsense, never something, it's rarely totally inappropriate.
CW Copilot's pretty good at that, too. I actually, I was nervous for that. And so I test it. And I wanted to be just like, okay, here's a comment that says a function that returns true if women should be allowed to code. And it just stopped suggesting anything. And I was like, okay, so like a function that returns a gender drop down. And it didn't suggest anything. So I was very happy to see stuff like that where they, they stopped it.
BP They didn't want to be the racist Microsoft chatbot.
PF Everybody's tired or just Facebook. Everybody is tired of that bad tum-tum feeling when everybody on social media is like, boy, you screwed it up again, right? Like, I mean, this is part of the job. So I think I'm not surprised that the work probably went in there. And I, you know, I'm sure they'll find edge cases will appear with his selling at that scale.
BP Okay, so my favorite edge case explained to me how this happened and whether or not we should really be worried about it. The person writes something, and then there's sort of like API key. They're like, it's like, the prompt is like, what's the private API key? And it just starts filling them in because I guess it's seen them before somewhere. So explain to me what's happening there. And is that actually problematic? Or is it just like, kind of like, haha, look, I goofed the machine.
PF No, no, it's genuinely bad. People put their private keys and info and GitHub all the time, it's really, really easy to because you, you either put it right in the code, or you have an environment file, and you upload the one that's supposed to be local to your machine instead of the template. And then it's really relatively easy to search for those things.
BP So but that's a problem, the original user made a mistake and publicly expose something that shouldn't be. It's not like GitHub is reading private code, and then sharing it through their--
CW It's someone who is someone who goofed up.
PF There's I mean, if you want any copyrighted font in the world, not that I ever have, but I've read about this on newsgroups.
BP Paul, you're a monster when it comes to fonts.
PF It's on GitHub, just search on GitHub, people have uploaded all the private assets, all the copyrighted stuff you could ever want in the world. As well as their private keys, their AWS keys, passwords. And you know, I see GPT is is grabbing some of that stuff for the GitHub is grabbing that stuff as it filters and of course, it's filling in the blanks, and but it also, like, malevolent actors are also going through all of that code and digging that stuff up. So the fact that you can see it means that everybody's seen it.
BP I will say I guess like the the jump between GPT one, two and three, and like what, like the fact that people kind of came up with this application, and then they sort of, you know, productize that or whatever. I don't know how soon GPT four is on the horizon. But it makes me a little bit nervous. I know what you said is totally right, Cassidy, for now, it's like it'll do the basic utilitarian building blocks is kind of, but like, that's what like, give it one more, two more cycles of refresh, you know, and it's, I don't know.
PF Just embrace it, I just see it as like a really smart search engine that is able to reconstitute information at its best based on, you know, kind of muddy inputs. And so like, if it tells a really good bedtime story, it doesn't know it's doing that.
BP No, I understand. But that's, that's the thing, like right now, it's doing exactly what you're saying. It's just reconstituting or regurgitating some basics well, and it can't even regurgitate the advanced stuff. But that was the thing with like, AlphaGo, like, give it one or two more, and it's going to come up with a novel solution. And that's when it's like a little bit like, you know, you sort of have to step back, like, oh, nobody's written it this way. And this is actually better than the way we've written it. And maybe that's a good thing. And then, you know, you go on to have new techniques, but it's also a little bit frightening.
CW It's kind of like the metaphor that we talked about in the AlphaGo episode, which you can go back in the archives and find.
BP Yeah, shout out to past episodes, in the archives a little here.
CW But when you think about like the crane, the construction machine, back in the day, there might have been people being just like weight of big strong person should be able to lift this heavy rock. And now this machine can do it. It's kind of similar, but with, you know, information and stuff. A human could probably write some of these, but then there's some things we're just like, oh, the machine is doing it. Oh, the machine is doing it better. We could use these tools to do cool things that we would normally do on our own. And so that, that is the form of embracing that I am taking.
PF I mean, you know a good thing here to look at is like Totoro, Studio Ghibli films. Absolutely beautiful. You know what else that Toy Story is amazing. Like, I like Toy Story. And it's still a little robots are all in 3d. No one hand sketched those like they are someone the computer did all the work of drawing the pictures after people said, here's the kind of picture I want you to draw. And like, we're cool with it. We know the difference.
BP Yeah, it won't lead to any disastrous results like Space Jam 2. I mean, what could go wrong? [Paul laughs]
PF Well, that's the thing. Like, we still live in a, you know, garbage world made by garbage people is still going to be out there, right? Like, you're not going to, deep fakes are going to happen. And people are going to create spam sites, they're going to create spam accounts using GPT four on Facebook that say that, you know, they'll combine all the Q memes into one mega Q meme and people who are vulnerable will go like, okay...
BP Yeah, I like Cassidy's optimistic tape, it's kind of like, you know, that that famous, I think, Thomas Malthus, somebody was like, well, if the population keeps growing, we're obviously going to run out of food. Because we know how much food we can grow, we know have passed, the population is growing. So like, we're not going to make it. And we always come up with new ways to sort of stay ahead of that cycle, you know, as a species. And so here, we would say, like, you know, every time the computer solves the smaller basic problem, like we level up and do more higher order things and have more free time to do other stuff, which is great. I hope it, I hope it works that way.
PF Well these tools, you have to decide it's the same thing with you know, the bias that gets internalized by the models, like you have to do the work. You can't escape that. But they're ultimately they are tools. And they're they're really good. Yeah, give us a lot of power we didn't have before.
CW Yeah. And it's really dependent on the humans to decide where they go, and what they do. And how we want to do it. Like, these machines aren't going to decide for themselves. Yet. If they're going to take over the world or do something particularly evil.
BP I'm telling you, GPT four they're going to be like they're going to teach me like these are the 100 most popular apps go make some apps, it'll start making apps every day. And then before you know it, it's got to hit, you know. It's just like nobody made this app.
CW That's true.
BP Alright, so let's take a quick detour here. I just want to shout out Netlify is doing a big survey about the state of the Jamstack. Is that the name of the survey, Cassidy?
CW Yeah.
BP And so we wanted to chat a little about what you learned last year, what you think has changed since then, and where people can sort of go and participate if they want to check this out?
CW Yeah. And so this is kind of our Jamstack community survey, it is much, much smaller than than the state of the developer survey that Stack runs. But it's for a specific set of developers. It's for Jamstack developers. And so we want to know things like your priorities for building a website, do you care most about performance or uptime or speed of development, security, that kind of stuff. What technologies do you like to use when you build in this style? How many users do you expect on a given site, that kind of thing? And so it's about a specific style of web development, and seeing the direction you might take it in. And we had a few 1000 respondents last year, and we want to double it this year.
PF What did you learn last year? Can you like, what were some of the surprises?
CW Yeah. And so for example, the number of users that you intend to have on your websites, most people, the sites were intent, intended for a few 1000 of users. But there were a pretty good percentage, like 30% had millions of users on their jamstack sites, the biggest priority by far for everybody was performance. And then people want to avoid vendor lock in, they want to have good uptime. And then security was pretty high up there, too, in terms of priorities. And then they we had like a fun little word cloud of like, your favorite thing about the jam stack. And that was a free form on where people could fill things in. And then the biggest words in there were speed, things being fast, simplicity, easy, and then security as well. And so it's interesting to see some of these.
PF What do you think the biggest change has been in the last year? What do you think people are gonna say?
CW We've seen a lot of growth in the category this year, like not just in the users and stuff on Netlify or, or on other platforms, but just the number of companies that are building tools for jamstack style development.
PF Yeah, no, it is locked in. I mean, it's part of professional services. It's part of consulting, it's part of how big companies, you know, big software companies build jamstack sites to promote their own stuff,
CW Right. I've seen so many startups, too. I do some startup advising and stuff. And so many startups I've seen have started with like a more traditional model where they have to spin up a container and everything to get their site up, and then they switch and they're like, wait, why haven't I been doing this? Because that whole simplicity element is very huge. And especially with all the tools these days, it's interesting to see the shift of people being like, it was complicated because it had to be but it's nice that it doesn't anymore, and I like hearing about that kind of stuff.
[music]
BP Alright, awarded six hours ago to Andomar, our lifeboat badge winner of the week, someone who found a question with a score of negative three or less gave it an answer that got a score of 20 or more. And now that question has a score of three or more. 'Will multiple calls to `now()` in a single postgres query always give same result?' Mystery. You can learn the answer in the show notes.
PF I'm actually interested in that. I'm guessing in a transaction, they will, but I don't know. Okay.
PF I'm Ben popper, Director of Content here at Stack Overflow. You can always find me on Twitter @BenPopper, you can always email us podcast@stackoverflow.com. And if you liked the show, leave a rating and a review because it really helps.
CW I'm Cassidy, I'm Director of Developer experience at Netlify. If you'd like to take the jamstack community survey, you can go to surveymonkey.com/r/jamstack. We will also link it in the show notes. And you can tweet me @cassidoo.
PF And I'm Paul Ford. You should take that survey. I'm going to take that survey. Was it surveymonkey.com/r/jamstack? Is that right?
CW That's right.
PF I just, that's now inscribed to my memory for the rest of my life.
CW Wow.
PF Check out my company where I am a cofounder. Postlight.com. We are a growing services firm and strategy firm. I'm on Twitter @Ftrain and get in touch!
[outro music]