Cassidy and Ceora talk with Astro creator Fred K. Schott and Cloudflare’s Brendan Irvine-Broque and Michael Hart about the intersection of open source and AI. Plus: Cassidy’s impressive robot credentials.
Cloudflare offers zero-trust security and performance tools for web and SaaS apps.
Cloudflare Workers allows devs to deploy serverless code globally to over 285 data centers around the world.
Astro is an open-source web framework built for speed. Houston is a bot that lets you chat with their docs.
Check out Confbrew, a conference session Q&A bot from Markprompt and Contenda (where Cassidy is CTO).
Connect with Brendan on LinkedIn or follow him on Twitter.
Connect with Michael on Twitter.
Connect with Fred on LinkedIn.
While you’re at it, follow Ceora and Cassidy on Twitter.
Shoutout to Lifeboat badge winner The Nail for saving if->return vs. if->else efficiency from oblivion.
[intro music plays]
Ben Popper Intuit developers are using the power of AI, data, and open source to unlock financial opportunities for millions of consumers and small businesses. Find out how you can power prosperity around the world at intuit.com/stackoverflow.
Cassidy Williams Hello everybody, and welcome to the Stack Overflow Podcast. My name is Cassidy Williams and I'm one of your regular co-hosts, along with Ceora Ford. Hey, Ceora.
Ceora Ford Hey! How are you?
CW Oh, I'm good, just vibing. I am very excited about today's episode. We have got Fred, Brendan, and Michael all here to talk about AI, open source, and more. Would you guys mind introducing yourselves and what you do?
Brendan Irvine-Broque That's great. I'm Brendan, a Product Manager for the Workers runtime at Cloudflare.
Michael Hart I am Michael. I'm a Principal Engineer on the Workers runtime at Cloudflare.
Fred Schott And I am just some open source nerd that does not belong here, but I work on a project called Astro, and we've been doing a lot of experimentation with AI and how it fits into open source.
CW Yeah, I'd love to learn more about what you are experimenting with, because none of these places where y'all work are AI companies, but it kind of speaks to the fact that AI is definitely kind of slithering into all of these different industries and making cool things, and devs are building really interesting stuff with it.
FS I can maybe start, because I think Brendan and I started talking about this a bit when we launched something called Houston. So I think as far as I know –no one has ever called me on this, but from my research I believe it to be true– we were the second open source project to ever create a chatbot for open source, so the idea of a place you could go to ask questions and get an AI-powered chatbot trained specifically on your docs. LangChain is the only project I know that did that, because obviously that's one of the big things they do, they're a big AI tool.
CW That's what they do.
FS Makes sense that they would be first, but we were honored to be second. Launched it kind of to big fanfare. We blew through our budget and took the bot down and had to scramble to get more credits from OpenAI, but really kind of cool to scratch the surface. And I think an interesting thing about our project is that we're not just like, “Oh, AI everywhere.” We actually have a lot of AI, what maybe I'd call skeptics on our team, and so we're really very carefully I think, or at least with a very kind of open mind, trying to figure out what are the problems this is solving? It's not just AI for the sake of AI. But a couple months after that, we have a couple of cool experiments that we're running, some that we've built, some that we're actually just working with cool partners on. And I've just done a lot of thinking on how projects, whether if you are an open source developer listening to this or you just care about open source, there's a couple really clear benefits and clear wins that aren't, “Move everything to AI.” They're just really cool ways that they can kind of help the problems you already have today. So that's how I came to this problem and that’s definitely my interest here.
MH What's really cool about the open source aspect is that I saw HoustonAI, maybe Brendan gave me a heads up, I can't remember, and then immediately cloned it internally at Cloudflare, called it Austin AI, and trained it on our docs. I also saw, I think it was Pete Hunt who had a blog post about doing the same thing of Dagster/Facebook fame, and it was really cool to get that gratifying feeling of, “Hang on, this just works.” It was awesome.
BB Yeah, we've been exploring a lot in this space, especially because people are coming to Cloudflare all the time trying to figure out how to build on Workers. And to me, what's really struck me is the importance of examples and how before AI, having a great example had this much leverage, it just increases exponentially now. Those are the examples that are kind of powering all these tools.
CW It reminds me of a blog post that I read by Rach Smith who works over at CodePen, and she was saying, “I don't need a fancy AI assistant. I don't need AI to do all of my problems and solve all of my problems for me. I just want really good search, and if AI can help me with really good search of anything, I'm sold on it.” And something like this I think is a perfect example of that, where you're able to ask questions and get the answers that you need based on something that exists and having that source of truth rather than something that's very hallucinations-driven based on arbitrary data.
FS So that was kind of the wild thing that we learned with that Houston project. It was realizing, “Okay, you ask Houston a question and it gives you the wrong answer. It got that wrong answer because it read your docs. It's actually pretty smart, it's good at reading, but if it gives a wrong answer, usually that means your docs are either unclear and it's having to guess, or your docs are maybe even contradictory.” It doesn't know the answer because two different things say two different things, the information's missing. And when we first launched Houston, even internally we were like, “Hey, check this out. This is something that's really cool.” It was one of those classic 3 AM hacking projects. I think our docs team, the first thought was, “Wait, are you trying to replace docs with some dumb bot?” It was actually very like, “What do you think we do here? You think a bot can replace all this time and energy that we put in, the love and care?” And realizing that, no, that's totally not a replacement. It is actually a validation of your docs. If it has a wrong answer that means your docs have a wrong answer or an unclarity or uncertainty. And actually we now totally have flipped that thinking. It is a linter for docs but conceptual, not just, “Are you formatting it correctly?” It's a conceptual linter and that has been, I think, one of the coolest mind flips for our team.
BB Yeah, this auditor that you can ask questions to.
CF Yeah, I think it's interesting how this is a great example of how AI can enhance the work that you already do. I think a lot of people think of it as, like you said, replacing whole teams, whole industries, things like that, which is very doomsday-ish and scary for a lot of people. But I think if we kind of shift away from that mindset about, “Oh, AI is going to replace developers. In 10 years, developers won't even exist,” stuff like that and look at it as a tool to help us do our work better, that is a much more realistic and more positive and probably better perspective to have. I heard you earlier, Fred, mention having this mindset and wanting to use AI as a tool and wanting to use it for good. I want to hear more about how you kind of reached that conclusion and more about that mentality.
FS I have a very narrow frame. I have been doing open source for a decade now, so I see those problems and I see that so much of our industry is built off of people working for free, and burnout is such a problem. There are worse, bigger problems in the world, but I see people who are kind of burning themselves out all the time. And our support team I think kind of suffers from that, or at least has always had that problem. Our project is big now. We have maybe 50 support questions come in a day and if you go to that channel it is chaos. It is questions coming in from people.
CW I was there this week.
FS Yes, I don't know how anyone survives. As someone who's done that, it is a torrent and it's never ending. And I'm sure customer support, other kind of service jobs, we're lucky to have a great volunteer staff, but there's real burnout there. And I think that's the kind of thing where open source and so much of what we do, there's always more work to do. I think this is one of those things where it's not going to solve every problem instantly, but this idea that it can be a really good line of defense. That's kind of been the second big insight is that support is actually a really good place to start because you're getting a user who's stuck. You're not in a hallucinated docs to someone who otherwise would've been fine and you actually send them in the wrong direction. At this point you have a user, they're stuck, they're in trouble, and they're coming to you for help. There's this really nice first line of defense that a bot like that can kind of power, which is, “Hey, listen, this might not be the perfect answer, but it's better than nothing and if you still need help after this, this can basically make it so that the people, the actual people in this channel, have less high priority things to look at. They'll get to yours actually sooner because there's this bot helping out with the easier stuff that's answered by our docs.” So I don't know if I would call that for good. I don't know if I would go that far, but burnout is just such a problem in our industry. It's really cool to see these tools cutting out some of the busy work while still letting the same people now do more with less.
MH Yeah, and I think that the questions that the users would then be asking would already be honed and shaped and a little bit more to the point rather than just open-ended.
CW Why is my code not working?
MH Yeah, exactly. You ask that to ChatGPT and then come to us.
CW My company, Contenda, we turn videos into text and summarize things and we made one, we call it Confbrew, where you can put in a playlist of videos from a conference and then you can use this other platform called Markprompt. We smooshed them together and it creates a Q&A bot based on conference talks. And it was a very similar thing too where we were thinking, how can we use this so that way it's for the good of the community. They can still ask questions about this conference, but if there's a multi-track conference and you want to go to one talk over another, you can still be able to ask questions without figuring out, “How do I hound down this speaker?” or, “How do I reach the organizers and then get to this point and then get to this point?” and it can just be a tool that saves time.
CF Yeah, this is so refreshing for me. I just kind of had that mindset, being very weary of AI and cautious about it and a little bit– I don't want to say scared.
CW It's fair to be scared though.
CF Yeah. And weary is probably the best word for it, extremely weary.
MH Yeah, there's a lot of unknowns there, isn't there?
CF Yeah. And unknown side effects and people get affected in bad ways, but it's nice to hear about AI actually being used for legitimate needs and helping people instead of hurting people, or instead of affecting people's livelihoods, it's helping them do better at their jobs. I like to hear that kind of thing. That makes me feel more like, “Okay, maybe the future with AI isn't going to be so terrible.” So that's nice.
FS Yeah. The uncertainty is also at such a whiplash level. The full circle kind of funny story started with the docs team being like, “Why are you trying to replace docs? What are you doing? AI is coming for docs next.” And then a week later they found this project which writes code based on docs. So actually, flip it. Now docs is the only type of developer anymore left. Everyone's going to write beautiful docs and then the code will be written. Every week it's like, “No, you're losing your job. Okay, no, you're actually the most important person. When the robots rise up, you're the only one that'll be left.” There's a lot of fear there for sure even if it feels not real and silly, still behind that is a lot of uncertainty about what comes next.
CF Like you're mentioning, you think, “Okay, is my future, is my career at stake because of AI?” But I think the way that you guys are building is a much more realistic way of what the future's going to look like, which is not necessarily that developers won't exist anymore, doc writers won't exist anymore. It's more so that it's going to just help us do our jobs better, which I like a whole lot better than it replacing humans doing real work.
CW It reminds me of a study that I was a part of. I think I've actually talked about it on this show before. I'm a very active Go player, I try to play every day. And when AlphaGo came out, so many players were like, “What's the point of us ever playing again? This bot will always be better than us.” But we were a part of this research group and stuff asking questions about it, and a lot of the conclusions that people were realizing was that if we use this bot as a tool rather than as this thing that we'll never achieve in our own brains, then it's something that can only make us better and play differently. And there were all of these moves that the bot was making that I had a Go coach at one point where she was saying, “Typically, this is how you would respond in this situation. This is how it was for about 400 years and then AlphaGo happened and now this is the proper response.” And it's wild how it was literally hundreds of years of knowledge changed.
MH And when you look at the graph of ELO ratings it's done this little spike now.
CW Yeah, people just got better.
MH Humans' ratings have just suddenly gotten better.
CW Yeah, because it kind of makes you think in a way that isn't just, “This is how it's been, this is how it should be.” It kind of thinks about problems in a very different way. And so I think the tool perspective is the way to think about it. And that's also kind of how you can vet products too and be just like, “Is this a product that's going to be a useful tool to help us do our jobs better or help us learn better to do something? Or is this something that's ripping off of someone?”
[music plays]
BP All right, everyone. Our annual developer survey is launching today. Last year we had insights from over 73,000 developers, insights about the most loved and loathed programming languages, how technologists learn to code, what impacts developer productivity at work, and much more. So let your voice be heard by taking our 2023 survey. The link is in the show notes. Make sure you go check it out. We want to hear from you.
[music plays]
CF So I want to hear if anyone here has more ideas or use cases for AI in a way that it'll be used as a tool to help, instead of, like Cassidy said, ripping people off.
BB I haven't seen as much about helping people write tests or testing frameworks that are really baking it in as a first-class citizen, because that seems interesting. There's been people who have built whole ways of running engineering organizations based on test driven development, and maybe I'm not paying attention to their projects, but there are other interesting things out there.
MH Given the sort of examples that we've seen of how people have maybe fed ChatGPT a schema or something and then said, “Build something off of this,” I think that those sorts of problems are ones that it actually should be quite good at. Like, “Here's some code, go generate some basic tests for me because I really can't be bothered, but I know that I need them. So do that work for me please.”
BB Or, “Let me scaffold out some tests and then show me potential examples of how I might build this.”
FS That's where I feel like maybe I'll use the autocomplete, but that was old news a year ago. Now they have, “Oh, we'll reformat your code for you. We'll add comments.” I never click any of those buttons, ever. I'm embarrassed to admit it because I think it's so powerful, but I never actually stop and ask AI to do anything like that. It's not a part of my workflow yet. There's going to be this AI thing that's just running on your code base, and instead of in your editor, it's more like it submitting PRs to your repo being like, “Hey, I found this code was confusing,” or, “Hey, this dependency, not only are we going to update the version, but we're actually going to change its usage to match the new docs.” There's all this kind of busy work.
MH Wait, you mean like Dependabot? Because man, I don't know if we want that.
FS I'm sure that's literally the reason they haven't done it yet. 20 AI-generated PRs and everyone's just like, “Oh, this is the worst.” I think there's a real cool and then risk there, which, also as the person who got into an industry of open source, that was kind of how I got my first PR. I refactored code, I fixed a dumb bug. And that's actually a fear for me– how does this affect people coming into the industry? If they can use AI as a tool, that's great, but what happens when we start to automate the use of AI even?
MH Maybe it helps though. Maybe it helps people who are a little bit like, “Oh, I want to contribute this thing to this repo but I really don't want to seem dumb. Chatbot, can you tell me if this PR would make sense or can we chat about it and you let me know if I could write it in a better way?” It may help out with things like that as well, lower that barrier to entry.
FS Yeah, there's a good chance I'm like, old man shakes fist at the clouds.
CW I want it to lower that barrier to entry. That's a question that I get very often from junior developers saying, “Is it going to replace the juniors and only seniors will get value out of it?” and that kind of stuff. And I don't think that's true, but it's still one of those things where I think we have to educate people and show people how, again, it is a tool and not a thing that you should just rely on to do all of the stuff for you. It kind of reminds me of when I started taking proper computer science classes, I was used to writing Java in Notepad and then hand compiling it and then seeing if it worked. And I remember when I started using an IDE that had autocomplete, I was like, “Well, this is cheating. Anybody who uses this won't memorize the commands.” Now I love autocomplete so deeply, but I think AI, or at least LLMs, these large language models, it is really just super good autocomplete right now that's really, really smart autocomplete. And I think if we use it as something that can help us and not just a crutch that we rely on fully and we don't think of the logic for ourselves, I think that's where it can be this valuable tool that does help us contribute to open source more, that does help us learn better, that does get rid of a lot of that busy work so you can do something more interesting.
MH We built one on Workers. I mean, it essentially does what the docs bot that we were talking about does where it can go retrieve your documents, and there are plenty of plugins out there like that can do interesting things, but something where it can be a little bit more than what it is at the moment, but in a very code-specific way and, I don't know, fetch information about the API you’re using or whatever.
FS I can't remember which book it is. Is it Pragmatic Programmer? There's got to be some comfort found in history where a developer team used to be one person coding, one person behind them reading the docs manual, another person behind them that would go run to the other room to ask someone a question. It was like the Starship control command center. And I'm sure at that point when the internet came along and Google, it's like, “Oh, but I got my start in the industry being the person who ran into the other room to ask someone a question. How will I ever break into the industry?” It's probably all going to be okay, but it's still scary.
CF Yeah, yeah. And I think there is also a lot of fear mongering, unfortunately. But what I'm gathering from this conversation is that I feel like AI is going to be much more like how we have all these different products and services out there that do the busy work of starting a project or creating a project, handle the things that we don't necessarily really want to do, but it's not necessarily totally replacing a developer. So I think that's the direction that AI is going in and like I keep saying, I'm happy to hear that, because the fear mongering can be very convincing and it can shake you up a little bit.
CW And I think it's particularly hitting us as developers now because it's so hype train-y and prevalent and stuff, but I feel like every hype train that we've seen, there's always some element of fear mongering where it's like, “Web3 is going to replace all communities. Crypto is going to replace all currencies. No-code is going to replace all developers.” There's always something to that effect.
MH Yeah, there are definitely people vested in some of this fear mongering for sure, and a little bit like, “My project is the best because it's going to replace this, this, this, and this.” And it's like, “Well, is it?”
CW I do think though that with all of this being said, again, there's so many useful applications, and again, treating it as a tool is key. One thing, speaking of Copilot, that I've been experimenting with is that new Copilot for PRs that came out. It kind of reminds me of what Fred and the docs team were talking about where it'll write a description for your PR based on the code changes, and if the description is wrong, we can be like, “Wait a minute, that's not what this code does. Oh, no.” And we've actually caught a couple things that we probably shouldn't have committed or things that we should change because the description it provides is just off enough where we have to double check ourselves. It's cool.
MH It makes that feedback loop quicker. I guess if you compare it with SEO or something like that where you're kind of like, “Okay, we want to improve our docs so that people can find the right thing,” but the feedback loop for that is massive, months or something. You maybe write things in a way that you think is better and then months down the track you find out, “Okay, all right, maybe we try this other thing.” Whereas the feedback loop now can be not exactly the same but it's still very similar where it can be like, “Okay, what does Copilot or ChatGPT think when it's given this information? Oh, hang on.”
CW It's like, “Uh-oh.”
FS Imagine just getting dunked on where it's like, “Description. I don't know what you did here. You figure it out.”
CW It has a poem generator where it writes a poem based on your code changes, just as a fun thing, which has been fun, but then at the same time we're just like, “Is it roasting us?” Sometimes it’s just like, “This is some old legacy code. Time to convert to a new mode,” and just funny little rhymes like that.
MH That was a very good robot voice.
FS Cassidy is a robot!
CW I've been caught! That was actually something where –this is totally off topic– I used to work on the Amazon Echo. I'm not going to say the name because I have one behind me that will wake up, but they had me do the Midwest accent. A lot of her listening is based on my Midwest accent where I had to read a bunch of test sentences, and so I am a robot.
CF That is so funny.
MH I saw today that Amazon has released a ChatGPT, or they're finally getting in on the fray. I was wondering what was going to take them so long. Bedrock, I think they're calling it.
CW All the companies getting in on all the hype trains.
MH Yeah. Well, I mean it's interesting to think, certainly at Cloudflare there's a bunch of stuff that we just can't use OpenAI for, and I'm sure that's true for many companies when you have internal proprietary stuff. This is obviously the opposite of the open source world, but it's really interesting to see that space blossom and it's still very much in its infancy.
FS So can we all say on the record right now on this podcast, when is Cloudflare releasing its own AI platform?
MH Well, we are releasing something but it's not quite that. I mean, we're looking at a few angles. One thing that we're very good at obviously is being close to where people are and running more lightweight sorts of things. And a lot of the time you don't need these huge large language models. I mean, with the sorts of stuff that we've been talking about so far you kind of do, but if you're just wanting to do a sales forecast or something like that and you want a little machine learning model for that, that's the sort of stuff that we're looking at providing.
BB I know that on the Stack Overflow Podcast you had Matt Butcher on pretty recently to talk about WebAssembly, and that's a space that we're spending a lot of our time focusing on. How can we do model inference in WebAssembly at our edge in interesting ways for some of these smaller models? How do we make that easier? How do we kind of uncap the limits? I think historically when people think serverless they think about the edge and they think that that's associated with just being constrained, and how do we kind of shake that notion a little bit and give you a little bit more compute power to play with?
[music plays]
CW Well, all this being said, thank you all so much for being on the show. We would like to shout out a community member for winning a Lifeboat Badge. For anybody who doesn't know, a Lifeboat Badge is someone who has an answer score of 20 or more to a question score of -3 or less that goes on to receive a score of 3 or more on Stack Overflow. And this one is awarded to The Nail for, “If->return vs. if->else efficiency.” We will link it in the show notes to see if you can learn something about that.
FS Definitely a bot, 100% a bot.
CW All that being said, I've been Cassidy. You can find me @Cassidoo. I'm CTO at Contenda.
CF And my name is Ceora Ford. I'm a Developer Advocate at Auth0 by Okta. You can find me on Twitter. My username there is @Ceeoreo_. And I also have a website/blog called Ceora.dev.
FS My name is Fred Schott. I co-created Astro, which is a framework for building websites. If you're a web developer and you haven't heard of Astro, check it out. Astro.build is the website.
BB I'm Brendan Irvine-Broque, Product Manager working on Workers. My Twitter is my last name, but that's pretty hard to spell, but check out workers.cloudflare.com.
MH My name is Michael Hart. I'm a Principal Engineer at Cloudflare, and my Twitter handle is @hichaelmart.
CW All right. Thanks, everybody. We'll see you next time. Bye!
[outro music plays]