In this episode, Alexa Montelibano and Tiago Torre, sales engineers at Stack Overflow, take you behind the scenes to show how customer feedback shapes our products, including OverflowAI. Alexa and Tiago have been working with clients to explore the three features of OverflowAI—Enhanced Search, an Auto-answer App for Slack and Microsoft Teams, and an IDE extension.
You can learn more about these three features on our Overflow AI site.
If you want to connect with Tiago, you can find him on LinkedIn. The same goes for Alexa.
A shoutout to Stack Overflow user Mahozad for earning a LifeBoat badge with their answer to the question:
How can I add Jetpack Compose & xml in the same activity?
[intro music plays]
Ben Popper Want to hear more from tech leaders shaping the future? Join Roblox founder and CEO, David Baszucki, as he sits down with the visionaries that are advancing human connection and dives deep on the innovation happening at Roblox every day. Tune in to Roblox Tech Talks at corp.roblox.com/techtalks or listen wherever you get your podcasts.
Ryan Donovan Welcome to the Stack Overflow Podcast, a place to talk all things software and technology. I'm Ryan Donovan, I edit the blog here at Stack Overflow, and I'm joined today by two of our sales engineers who have been hearing from customers about our OverflowAI launch. So welcome, Alexa Montelibano and Tiago Torre. Hey, everyone. How are you doing today?
Alexa Montelibano Doing well. Thanks for having us on.
Tiago Torre Good to be here.
RD Could you talk about your journey in technology and how you got to where you are today?
AM So for me, this is not what I studied in school. I originally studied aerospace engineering and really enjoyed a lot of the projects that I did in school and some of the problem solving. And when I graduated college, I cast a wide net of different companies that I could go to and found myself in technology, and for the last eight or so years I've been doing different roles within the technology space. I've had a couple of roles in sales of technology, some in implementation, some in support, and have just found a lot of gratification in working with customers to solve their problems through technology.
TT I actually share a lot of parallels there, Alexa. I also studied aerospace engineering.
RD Oh, a couple of rocket scientists.
TT Yeah, we have a couple of rocket scientists on the call today. And when I exited college, the reasons why I studied aerospace were not because of my love for rockets and airplanes. It was really because it was a very diverse set of engineering topics that it covered. And so I soon realized, growing up in Portugal and then exiting university there, everything surrounding aerospace was related to software anyway, so I thought I might as well go to software directly, I heard it was much better paying. So that's how I started a career in tech and one thing led to another. I did start in technical support and later on got the opportunity to move to Germany where I'm located today and support a sales team and then moved to a pre-sales role, and that's what I've been doing for the last five years.
RD All right. Obviously on the podcast we've been talking a lot about Gen AI and we're moving in that space now, too. What has been the customer reaction to Stack Overflow for Teams getting Gen AI features?
TT I think in general, we've been absolutely bombarded with enthusiasm. So it's been very well received. I think AI has been, for a big part, a buzzword for the last few months. It's all that people talk about in the tech space. But coming into this conversation with a fresh and educated take on it, which I feel is what we've brought to the table on the Stack Overflow side, just striking a good balance I feel between the human part of the conversation and the AI part of the conversation and how we meet in the middle with a community and how we marry these two worlds of humans and AI, I think has been very well received by customers. And I think the proof of that is that they've been continuously asking for more and we're finally in a place where we can show them what we've been working on and them getting their hands on it. So it's amazing to see the evolution there.
AM I would echo everything that you said, Tiago, and call out that I think in the last year, certainly AI has been a part of all of the conversations that we have with customers, whether it's how are we bringing AI into our own solution or how can we help them with AI. I think there's obviously a lot of excitement around the advancements that AI can bring to the space to productivity, but also a healthy dose of curiosity and skepticism when thinking about how do we bring this into organizations responsibly and make sure that we're adhering to their needs while also just being really conscious of it. So I think that's been a big focus as well.
RD I think that's definitely a good point to bring up. We talk a lot about hallucinations and security issues and dealing with private data, how are we addressing the pushback and the concerns that folks have?
AM It comes up in every conversation that we have with customers because it is at the forefront. I'm really proud of the approach that it seems that our product team has taken in keeping that really top of mind and focus. And I think as we talk to folks who are rolling out the product, they've made really intentional choices to make sure that we are kind of keeping that top of mind, but I think that the approach that the team has taken to incorporating things like attribution, which is making sure that people know where the knowledge is coming from and recognizing the sources of that knowledge and making sure that they're getting a lot of trust signals in the content that they're consuming regardless of what form factor they're engaging with our OverflowAI solutions, making sure that they can see things like who was the author, how many votes has this gotten, when was it written, and making sure that that attribution and that trust is built at every part of the engagement with the OverflowAI solutions.
RD So let's talk a little about the specific pieces of the product. It's OverflowAI, but at launch we have three separate things. We have Enhanced Search, we have the IDE extension, and we have the Slack Auto-Answer app. Can you talk about what problems customers feel like those are solving?
AM We can start with the Auto-Answer app for Slack. As I've been talking to a lot of customers, oftentimes they'll have some kind of help channel in their chat tools internally where people can go and post questions for their peers. And while those types of channels can be really helpful, it also requires a person, a subject matter expert, to go in and answer those questions. And because of the amount of monitoring and keeping up with that, one of my peers calls it a breeding ground for potential duplicate questions when we have these open channels like that. And so the problem that the team was looking to solve was making sure that we can take that burden off of the subject matter experts who need to go in and, even if it's just copying in a link, taking away some of that manual work to get that knowledge to the person who's seeking it. And so now with the Auto-Answer app for Slack, someone can post a question in one of these channels and automatically get a pretty instantaneous response calling on the knowledge that they've built in their internal community, which is really exciting because it means that not only are we freeing up time for those subject matter experts, but we're also making it an even faster response for those knowledge seekers who need that content and the potential for support ticket deflection is there. And so I think that's something that, as we've talked to customers and shown and discussed that use case, they get really excited about some of the time-saving value that comes with that.
RD So on a personal note, at my last job, I was handling a lot of the internal documentation efforts, and everybody asks their questions in Slack. And we had a Q&A tool, but that's not where the questions were. So always wanted something where it's like, “How can we automatically route these? How can we automate and handle these?” So I see the solution here for this one. It's kind of exciting to see this.
AM And I think even thinking about what you just said, for customers who have Stack Overflow for Teams as their Q&A tool, we know that chat continues to be a valuable tool for engaging one-on-one or in these specific scenarios where they need to collaborate. But also having the Auto-Answer app for Slack to be able to drive awareness that Stack Overflow for Teams exists is helpful for our change management too of making sure that people know that these solutions have been documented and then they can go back and refer to them later and encouraging them to reuse existing knowledge rather than starting from scratch every time.
RD Tiago, do you want to talk about the other products?
TT Sure. And I think in a way, the three of them, they're different takes and different channels and different experiences with the common goal of just getting those knowledge seekers to the internal context they need faster. And in the case of Enhanced Search and the IDE extension, not only for internal context, but also external. And I think what those two– Enhanced Search and the IDE extension, which I’ll talk a little bit about– what they have in common is that we thought about the problem that when folks are searching for knowledge internally, a lot of times they end up having to go through, let's say, five different search results from five different authors written in different levels of detail, and they first have to take that all in and kind of skim through different articles and knowledge base entries and so forth until they get the information they need to actually act on a problem and solve it. And so when we talk about enhanced search and the IDE extension, those would be two different ways of getting great summarized answers and cutting through that problem. Not having to go through all the list of results, but just getting a net new generated answer that not only summarizes the most relevant search results if you were to do a traditional search query in that sense, but they also include direct links and attribution to the actual underlying sources. So in my experience as a developer when I'm stuck and searching for that solution, I know exactly who contributed to the bits and pieces of that generated response. So whether I'm searching on Stack Overflow for Teams and using the Enhanced Search experience there, or I'm working in my IDE and using Stack Overflow for Visual Studio Code or the IDE extension in short, I'm getting that answer immediately and I'm still being able to trust or gauge the relevance and the accuracy of it because I know exactly who are the experts within my company that contributed to that knowledge, to that solution.
RD It's so important to fit within developers existing workflows. Tools die if they don't get used, and they don't get used if they're not part of a developer's workflow. So I think the Slack Auto-Answer and the IDE extension are pretty great ways to stay in front of a developer. Are there any concerns or excitement? Anything that you heard that you weren't expecting that sort of surprised you?
AM I don't know if I have an immediate answer. I think it's been really cool to talk to customers about each of the different modules and how they can benefit their organizations, just because there are use cases for each one where, as Tiago mentioned, we're bringing that experience consistently across all those different tools, and what you mentioned, Ryan, about bringing it into the workflow that people have today. I don't know that I have anything that's been an unexpected surprise, though. I think that, generally speaking, there's a lot of excitement around being able to bring this organizational knowledge into the hands of people where they work today, and then of course, incorporating some of that summarization and aggregation, the things that AI can really stand out with while also making sure that we're holding firm with our core of the community and the contributions that the humans can make that experience as well. So overall, really excited about where those conversations lead us with our customers.
TT I think we're also very privileged to have been able to have tested and proven a lot of the concepts we were working on for the last few months. And so at this stage, maybe a lot of it doesn't sound like a surprise anymore because we've been having those conversations. I would say the one piece that came across very well is a bit of what we alluded to already when I was talking in the beginning about the way we include the human elements in these AI-generated responses, just because the first experience with LLM has been always like, “I get this chunk of text that is generated for me and I really have no clue where it got it from and if I should trust it and how I am supposed to know that I can trust it.” And so building all those trust signals into it I think is definitely something that came across very well. I don't know if I would say better than I was expecting, but definitely one of the elements that resonated very well with our customers for sure.
RD So have you all played with the products yourselves? Have you used them in sort of dogfooding, champagne-drinking situations?
AM Much prefer the drinking our own champagne to the eating our own dog food, but yes, our team has had a chance to use all the modules internally for a few months. And then of course now that it's generally available, we have it in our demo systems where we get to show it to our customers as well. But it's been really great. Especially I think Enhanced Search has been used internally for several months now and I know I'm excited to now be able to share that experience with customers in their own environments, just because we've had a chance to experiment and provide feedback leading up to that point. But yes, we've had a chance to have our hands on it, and again, I think now that we're getting to the point where we get to actually show it to customers and let them see how it looks live in our demo environments, it's even more exciting because they get to feel it.
RD Can you talk about your experience with the IDE extension?
TT Sure. So Alexa and I are not developers on a day-to-day basis, but what we've worked on was not only building what we call Internal Labs, so OverflowAI Labs, which were essentially a series of exercises for our fellow SE colleagues but also colleagues in other field functions to get acquainted with the products. So naturally, being involved in building those exercises, we had to get our hands dirty with them and get experimenting and so that's been the angle of usage that we've had with the product, and now recently preparing things such as demo flows and demo scripts and thinking about that stuff. So with the IDE extension in particular, I was more involved on that module and that's why I jumped in. But I think one of the things that surprised me there was the fact that it is a conversational experience as you have it with different LLMs, but it's really impressive to see how it's aware of that. So when you're starting a conversation and then continuing it, it will actually reference or improve your prompts to make reference to previous things you said. So it's great to see how it's built in that smart way and just means it's even easier to get to that solution you're looking for.
AM I know, Tiago, you had a chance to go to the We Are Developers Conference last year when we were announcing the vision, and I think we talked about early on that the IDE extension is something that has been a resounding excitement for our customers and where they are really excited. I think whether it was from that conference and then also in some of our subsequent conversations, it seems to be one area that folks are really excited.
TT Just a last note on that, because I've heard someone describing it this way. A developer usually has the IDE just kind of always open in front of them almost at all times, and then the second window or the second tab is maybe your collaboration or chat clients whether it's MS Teams or Slack, and then the third one, that's where you start having a couple of browser tabs maybe. And you could have a Stack Overflow tab in there, but having an IDE extension just moves us to the number one place there where the developers are spending their time and attention. And so I think the most exciting part about all this is just being there with the developer as they are building amazing pieces of technology, and in the forefront of software engineering is having Stack Overflow be part of that experience for them.
RD I've seen those videos of when you finally solved the bug and closing all the tabs. It's 20 different tabs open to Stack Overflow. So it seems like we'll be saving them at least the tab closing time.
TT Yeah, exactly.
RD So what are you most excited about for this launch?
AM I think for me it's kind of what we said before, where we've had a chance to use some of this internally for the last several months and as we've been gearing up for the launch, and so I'm excited just have customers be able to see it live and to have some of those conversations where they get to have it in their hands. I think it's been really gratifying to walk them through our vision that we've been really thoughtful about, but to have something that they get to see and feel is of course the next most exciting thing. And I'm really excited, I think, especially not only for our new customers, but also for our existing customers to be able to get even more value out of the teams that they've built and the communities that they've built internally. I'm excited to see how this can continue to help enhance that experience as well.
TT For me, it would be thinking of, 6, 9, 12 months ahead, how the product will be then, because just judging by the hands-on type of mentality that we've had from our product teams as we were doing the stuff we were describing like building these internal labs and thinking about demo flows, we had questions and we had ideas and it's all been received so well and in such an agile and nimble approach from our product group that I'm sure there's great stuff down the line prepared. And I'm really keen on seeing how the product will evolve in the coming months as we get just a multitude of hands and eyes on the product and new ideas from our customers that always partner with us in building the products we're building. So that for me is the number one thing I'm looking forward to in the coming months.
RD Very exciting.
[music plays]
RD Well, it's that time again. We like to shout out somebody who came on Stack Overflow and saved a question with a little knowledge. A Lifeboat Badge was awarded two days ago to Mahozad. “Android: Jetpack Compose and XML in Activity.” So if that answers a question or you understand what that means, you can be helped along with 20,000 other people. My name is Ryan Donovan. I edit the blog here at Stack Overflow. You can find it at stackoverflow.blog. And if you want to reach out to me with article ideas, hot takes, cool tech lore, you can find me on X @RThorDonovan.
AM I'm Alexa Montelibano. I'm a Solutions Engineer here at Stack Overflow. And if you'd like to connect, I’d love to connect on LinkedIn.
TT I'm Tiago Torre. I'm a Staff Solutions Engineer here at Stack Overflow, and you can find me on LinkedIn.
RD All right, everybody. Thank you very much. Like, subscribe, and we'll see you next time.
[outro music plays]