On today’s episode we chat with Ellen Brandenberger, our Director of Product Innovation, and Jody Bailey, our CTO, about the big changes Stack Overflow put in place to create OverflowAI and the research and thinking that went into the products and features we announced this week at WeAreDevelopers.
You can learn more about OverflowAI and sign up to be an alpha tester here.
You can check out Ellen and Jody on Linkedin.
Congrats to Ben Lindsay, who was awarded a Lifeboat badge for his answer to: How can I divide each element in a tuple by a single integer?
[intro music plays]
Ben Popper Hello, everybody. Welcome back to the Stack Overflow Podcast. We have a super special episode for you today. We have two folks on from the Stack Overflow team, and we're going to talk through some of the big announcements we made yesterday at the We Are Developers Conference in Berlin, all about how Stack Overflow is approaching the brave new world of generative AI. So I want to welcome to the show two of my colleagues: Jody and Ellen. Ellen, tell the folks who you are and what it is you do here at Stack.
Ellen Brandenberger Thanks, Ben. So I'm Ellen Brandenberger. I've been at Stack for a little bit under a year. And I'm the Director of Product Innovation here at Stack Overflow, which basically means that when there are new problems to solve for developers, me and my team try to figure out how to solve them and how to bring them to all of you and to market in some cases as well.
BP And Jody, for folks who don't know, you are our CTO and have been working closely with Ellen over the last few months. Maybe give people a little bit of a picture of what we've been through to get up to the announcement we made yesterday. We talked about it a little bit on the blog, it's not a secret or anything, but we kind of went into full startups, skunkworks, try it all, see what happens mode to get to this place. So maybe give folks a little bit of background on the way things were happening inside Stack Overflow.
Jody Bailey Yeah, happy to. I'd say several months ago we all recognized that there was a unique opportunity with AI and some of the things that were happening, and we also knew that we really wanted to move quickly in terms of solving problems for both our users from the community as well as our customers. And so there was a big initiative across the organization where we essentially formed a new team, if you want to call it that, which was actually made up with several sub-teams. Collectively with Ellen and some other thought leaders in the organization, we had some broad stroke ideas of areas that we wanted to explore and deliver product and features on. And so we formed some new teams and we borrowed some existing teams and shifted priorities and essentially brought everybody under one umbrella. I think it was about 40 people all told that came together, and Ellen and I, along with a number of other folks, have been leading that collective group over the past several months to the point where we got to the most recent announcement. There was a lot of iteration there. The teams themselves did a lot of experimenting. We really tried to embrace an agile mindset and break some of the old ways of doing things and be open to getting feedback, experimenting, et cetera. And we've learned a ton as a team and grown a lot and I think we've delivered some really amazing improvements.
BP Yeah, I've been with Stack Overflow for almost five years and it was really awesome to see this process of new people joining all the time, in part because sometimes they were needed, but also sometimes they were just passionate about what's happening with this new technology, they wanted to weigh in, and week after week of sprints and coming up against a dead end and turning a different way or realizing that we had to split the teams in different directions so that we could better support staff. Let's go over some of what was announced yesterday to kind of give people the broad strokes. I would say maybe the biggest thing for folks who are listening, just your average developer or technologist, is that there are changes coming to the way search is going to work on Stack Overflow. So Ellen, you want to touch on briefly what we talked about?
EB Absolutely. So I think all the concepts that we talked about yesterday are early concepts. So as we talk through each of these items in more detail, I want us to think about each of them through that lens. So we're launching many of them in alphas across the upcoming weeks. You can go to the Stack Overflow Labs site and sign up for those alphas to participate if you're interested in many of those. But search is one of the big pieces that we really focused on. And the core problem there to me and to the teams was kind of that we have one of the largest knowledge bases on technical knowledge at Stack Overflow, validated high quality. But one of the regular pieces of feedback that we get is, “I can't find the content that I need to find,” or, “I have to search the exact keywords on the site or go to the tag and figure that out.” I think I've even seen some memes about this on the internet. So I don't think it's a new problem to folks, but I think in the context of AI and in the context of semantic search, which is not generative AI itself, but is sort of newly possible, we are able to kind of explore how we could uplevel our search in meaningful ways with just a couple changes around leveraging those new technologies. So the team experimented with combining elastic and semantic, they experimented with thinking about deploying this in more specific environments. They really were pushing the envelope in terms of figuring out which end experience would actually work best for their users. And even right now, this alpha is actually internally available on Stack Overflow for Teams for our own team, and I think Jody even earlier today was joking. He was like, “I can finally find what I need to find on Stack Overflow.” So for us, that one's a little bit less about how do we use generative AI to recommend results, which might be a good next step for us, but really leveraging the best practices in semantic to get people the knowledge and the solutions that they need, and that's really what it comes down to.
JB There's a couple of things that come to mind there. So one, absolutely with the alphas we're looking for feedback. And semantic search, like any search, we're going to continue to evolve and get better. But it also reminds me of the strategy that we had when we began. It was a crawl, walk, run approach. And we believe that, I think anyway, we’re still kind of in that crawl stage where we're rolling things out, we're getting feedback, we're continuing to improve. We have some bigger, more imaginative ideas down the road, but those are things that we're experimenting, doing research and development on, and want to get feedback from community and other members before we go down that path too far.
BP Yeah, I think one of the things also that was said by our CEO Prashanth during his keynote was that we wanted to do our take on semantic search. We wanted to maybe leverage some of what's in gen AI to provide answers or conversational answers, but always do it in a way where we retain sort of that core promise to the community, which is that people who give knowledge to us are going to get attribution, they're going to get recognition, you're going to understand what the licenses are on the code you're looking at. So very much trying to avoid the black box model of just, “Hey, I talked to the AI and it gives me an answer. Maybe it's right, maybe it's not.” The structure of the knowledge of the Q&A on Stack Overflow kind of lends itself to being able to verify what's good and what's bad and trust it. Was that something that y'all had to work through– the data structure and figuring out how we could add things like attribution?
JB Absolutely. I mean, one of the most important aspects of Stack Overflow and the content is people want to be able to trust it, and a big part of understanding and trusting it is knowing where the information came from. So especially in the search and results, we really focused on more summarization and then being able to point people back to the original articles, so attributing to articles or questions– the original content in Stack Overflow. So rather than generating information, we're looking at how we can present the information in summary that people can quickly understand, digest, and then click on the actual links to get the specific information and follow up with the person that originally answered or provided the input. So that's one of the ways that we're looking at how we help ensure that it's factual and that it's attributed.
EB Yeah. And this is a little bit more values-based, but when we started this kind of larger initiative around embedding AI technologies in our broader portfolio, that trust and that kind of bent towards attribution was one of the core values that Jody and myself and the rest of the leads on this project really tried to instill in the team. So it's not just that we're building products that should solve our community's biggest challenges, it's that those should also be embedded with core values of trust and attribution and personalization and accuracy, knowing that our end audience and that developers everywhere really value those things and that they're sort of central to providing a good solution in this space. So sort of every time we had key decision points across the projects for the teams, kind of coming back to those values and sort of saying, “Which potential option as we move forward better aligns with those values?” So keeping that top of mind was kind of key throughout.
BP Yeah. So another way we were trying to, as you point out, sort of reach developers where they're at was to introduce a Visual Studio Code plugin. So this is something that would work right in the IDE, trying to help developers find solutions without breaking sort of their flow or forcing them to context switch. Can y'all explain a little bit about what the genesis of this was and where it fits into the slew of things we announced yesterday?
EB Absolutely. So I would say it's not a new idea that Stack Overflow should build an extension to the IDE, right?
JB It's not like there are any out there, right?
EB I was going to say, it's not like our community hasn't already thought of that and built some pretty good plugins as well. Actually, one of the genesises was just that, which is our community. So we had a number of folks come to us and kind of point out that there was potentially opportunity there and there's sort of a nice organic groundswell in that area. So we took a deeper look, and then other similar products had been coming onto the market over the past 6 to 12 months. So we started digging in and did some user research that really helped us understand what is kind of the workflow for a technologist or for a developer as they're developing code, particularly in the enterprise. Because as we think about the spectrum of technologist user experience, a developer in an enterprise company looks very different from someone, Ben, like you or me or Jody, who may be hacking something together at home. So really thinking about that specific use case, we really dug in and did some research. And one of the gaps in that process for folks was sort of what is the context around the sort of experience or code that I'm developing that I'm currently lacking? So as I'm building something in real time, I have a question. How do I both discover and leverage that knowledge in a way that makes me more efficient? Enterprise leaders often describe this as ‘get back to coding,’ but from a developer perspective, it looks more like helping folks identify which solution might be best, taking those new learnings and leveraging them in real time, and then sharing those back with the rest of their Stack Overflow for Teams community and sort of saying, “This is a specific problem related to our code and our instance that maybe has a generalized solution on Stack Overflow, but benefits from that tailoring, that sort of narrowing in for our specific team and codebase,” and then documenting that and helping them kind of leverage AI to more efficiently and effectively document that solution.
BP Yeah, that's a great segue to the Stack Overflow for Teams side of the house. That's kind of like a private knowledge base within big institutions like a Microsoft or a Bloomberg where folks are asking questions, getting answers, and it's all about the proprietary code that's inside or how they set up their systems and things of that nature. So we also thought about bringing improved search over there, but also ingestion, which was not something that we kind of talked about on the public platform side. So Jody, what is knowledge ingestion and how does that work with Stack Overflow for Teams?
JB Yeah, so knowledge ingestion I think is going to be a huge advantage for our customers, especially our new customers. Similar to search, this is something that we've been trying to figure out how to solve for for a while. And we refer to it kind of as the empty stadium. It's great to be able to go to your knowledge base when there's actually knowledge in there, but when you have a brand new implementation, you need to be able to get that data in there. And we've found a lot of ways over the years to help people with that, but the thing that is true in almost every circumstance is that people already have information. They have knowledge that's been accumulated over time. And then the other thing too is that, as a technologist we recognize, or I recognize, that documentation doesn't always stay current. One of the big challenges is actually having good documentation. So the idea here is that we can help customers bring in existing content and bring it into our own format where we can actually leverage the team community, the internal organization, to help validate the content and the questions and answers as it's coming in or over time, to be able to leverage the gamification of the platform to actually help improve and ensure the content is valid. Plus, we have built-in features like content health that help us maintain the health of the information. So I really see it as a great way to help customers get existing knowledge not only in, but also improved and updated as part of that process.
BP Yeah, I love that because the cold start problem is that there's one evangelist who wants to get the documentation and the knowledge base improved, but they've got to win over a bunch of people to figure out which questions should go in there and who's going to answer them. And if somebody looks and there's no activity, they're not going to come back. Here it might pull in the question that's most frequently asked in your chat, or a bunch of documentation that is stored across different cloud services, and then all of the sudden you’re up and running and you can work more on refining it and making sure it's accurate, and like you said, figuring out what's stale and needs to be updated. So once the ingestion has happened, we've talked about meeting developers where they are with the IDE. Another one was kind of a Slack integration we did with something called Plus One. Ellen, can you explain to me a little bit about what that is?
EB Yeah, so you can think of this as the next generation of our Slack integration for Stack Overflow for Teams. So Plus One is an underlying technology which leverages generative AI to answer questions for individual teams right from Slack in real time. So the problem to solve there is not just technical, it's more broad to any organization and sort of leverages a broader knowledge base to help folks identify the right answer to a technical question to a problem at hand, and to do so in a way that's fun and engaging, but also validated. So where is it coming from, making sure we're keeping all the questions relevant to work for folks and to the construction of new technologies. So it’s really just a fun but also engaging and helpful way for folks to ask their questions right from Slack, and potentially in the future, this is something that if our customers love it, you could see it coming to Microsoft Teams or other chat-based products as a result.
BP Yeah, I love this one. We talked a second ago about ingestion, and so here it is more of that flavor I think people are familiar with of gen AI of, “I'm asking a question and I'm getting a synthesis back that's looked at all this different documentation from Confluence, GitHub, Stack Overflow, Stack Overflow Teams private and also maybe public,” and then it's trying to lead me in the right direction, maybe give me some links to questions that go deeper and cite the sources so that I can figure things out or add it back to the knowledge base if I think this is important and needs to be shown to other people.
EB Right. This is another one we've been kind of piloting internally and we've seen a lot of our internal teams really get excited about using Plus One in their own work, whether they're software developers or salespeople or marketers or otherwise. And so we're excited to see how and where our customers engage with this and what that really means for Stack Overflow for Teams in the long run.
JB I think it's another way of meeting people, learning developers, where they are. So similar to the IDE, so many of us spend a lot of time in Slack. Maybe that's a personal problem.
BP You’re not alone.
JB But it's another place where we meet people where they are. And then as Ellen said, the Prosus team have invested a lot of time and effort here and leverage multiple models to provide better solutions. So really excited to see what the feedback and response is to that and where we take it from there.
BP And then one last thing for listeners, we announced sort of an AI-focused community discussion, so if folks want to come somewhere and chat with other people to get unbiased technical resources and responses from experts, there's going to be a dedicated gen AI Stack Exchange. I know there's other ones about NLP and prompting that we've been working on, so kind of trying to open up the conversation, and for folks who are curious about this stuff as a lot of people in technology are, a place to not just ask and answer questions, but also to discuss it. Did I get that right?
EB Yeah, you absolutely did. So there's really a couple things that are launching in that sort of umbrella that you just described, Ben. One is a Stack Exchange site for –we're calling it gen AI– sort of particularly to Q&A on generative AI more broadly as well as prompt engineering as a sort of subset of that. So that site is really focused on the discussion of how to leverage generative AI in your development or how to use gen AI tools kind of as you progress as a technologist. And then we have a concept called Collectives, which has been around for a while, but recently launched one new collective in the generative AI space. And within that collective, we have a set of new features around more subjective discussions. So as we're implementing, as we're learning about these new technologies as technologists, how do we have more open discussions about the variety of ways that there might be to implement those things and start to think about how they can be best leveraged in a world that's a little bit more ambiguous than more established things. So opportunities for folks to learn and engage and discuss with our community about this new set of technologies that go beyond just products that leverage them.
BP Right. This is near and dear to my heart on the editorial front. Can't wait for people to have discussions on here and then I can just sweep that into the blog and thank the community and my job is done. We mentioned that folks can head over to the Labs page, which we'll put in the show notes, that there'll be some alpha access to certain things, hopefully by the end of August. I know there's lots of stuff that's going to be coming out over the next six months, including some fun toys that I started playing with today which you won't be hearing about right away but I'm sure you'll get the chance to experiment with later. Is there something in particular that either of you wants to shout out that you're really excited about?
EB Yeah, it's not really a feature so much as a mindset. So as Jody and I have kind of said a couple times, and Ben, I think you alluded to, this is a group that really values the community's feedback and our developers' feedback, so we're launching all of these things that we talked about today in alphas. So what we really are excited for is getting folks' feedback. That is meaningful and valuable to us as product development teams and provides an opportunity to have a conversation. We are excited to hear that feedback and take it in. We definitely can't take all of it, especially when it conflicts, but we definitely want to hear what the big themes are and work with and encourage our teams to leverage that and make things better as we kind of move towards bringing these things to broader availability.
BP Okay, you heard it here first. If you want to shape the future of Stack Overflow, we're going to have some opportunities for you. And a lot of change is coming, as it should when sort of a brand new technology emerges that is so impactful on the ecosystem.
[music plays]
BP All right, I want to say thanks to both of you for coming on. As always, let's shout out someone from the community who came and provided a little bit of knowledge. A Lifeboat Badge awarded 14 hours ago to Ben Lindsay, “How to divide each element in a tuple by a single integer.” You say tupple or toople? I say tupple. I've heard this said both ways.
JB Multiple, right? Tupple.
BP Yeah, multiple. Okay, tupple, exactly.
EB Toople?
BP I swear somebody was just asking this. “How to divide each element in a tuple by a single integer.” Thanks Ben Lindsay, for coming on and providing an answer. Congrats on your Lifeboat Badge, and you've helped over 36,000 people, so we appreciate it. As always, I am Ben Popper, Director of Content here at Stack Overflow. Find me on Twitter @BenPopper. Email us with questions or suggestions for the show, podcast@stackoverflow.com. If you like the show, leave us a rating and a review. But more importantly, head over to Labs, come to the blog and check out all the announcements, and be first in line to play with some of these new toys.
JB So I'm Jody Bailey. Probably the easiest way to reach me if you haven't before is just via LinkedIn– it's Jody Bailey at LinkedIn. And excited to get feedback and see y’all experimenting with things as we roll them out.
EB And I'm Ellen Brandenberger. Again, LinkedIn is probably best for me.
BP Awesome. All right, y'all. Thanks for coming on, and to everybody listening, we appreciate it and we'll talk to you soon.
[outro music plays]