In this episode, Ben and Ryan sit down with Inbal Shani, Chief Product Officer and Head of R&D at Twilio. They talk about how Twilio is incorporating AI into its offerings, the enormous importance of data quality in achieving high-quality responses from AI, the challenges of integrating cutting-edge AI technology into legacy systems, and how companies are turning to AI to improve developer productivity and customer engagement.
Twilio, a communication platform as a service (CPaaS), allows developers to build voice, video, and messaging capabilities into their apps. Devs can get started with their docs.
Find Inbal on LinkedIn.
Kudos to Stack Overflow user Wesos de Queso for explaining how to Prevent a toggle group from not having a toggle selected - Java FX.
AI, Twilio, Inbal Shani, machine learning, LLM, developer productivity, responsible AI, tech stack, customer engagement, conversational AI
[intro music plays]
Ben Popper Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I am Ben Popper, the director of content here at Stack Overflow, joined, as I often am, by my wonderful co-host, editor of our blog, maestro of the newsletter, Ryan Donovan. Hey, Ryan.
Ryan Donovan Hey, Ben.
BP Today we are lucky to be chatting with Inbal Shani, who is the Chief Product Officer over at Twilio. She has been working in AI for over a decade. Did a master's degree in genetic algorithms before machine learning was called ML. She joined Twilio early this year, but has done stints at GitHub, AWS, Microsoft, part of the teams that brought us some of the biggest, most well-known brands in AI, like Alexa and Copilot, and so we wanted to have her on and chat about what she's working on these days with Twilio, as well as the changing face of AI in the industry. So, without further ado, Inbal, welcome to the Stack Overflow Podcast.
Inbal Shani Thank you so much. Thank you for having me.
BP So, I guess, tell me a little bit about what drew you to this field in the first place. I'm not too familiar with genetic algorithms, so maybe a little bit of background on what got you into that world.
IS Yeah, so I, I started my career as an aerospace engineer, um, and I was an applied scientist. I was focusing on navigation and control systems, and the biggest challenge, always, when you're trying to create a, a controller is how to fine-tune that controller. And that came to the research topic: what do we do with a multi input, multi output controller? How do we optimize it to a process or a system that keeps on changing with time? And, at that time, the forefront of technology was genetic algorithm. It was the beginning of the attempt to try and find– solve optimization problem, uh, using Pareto front. So that kind of led me to the natural progression of: I'm going to do my master's degree, it's going to be in control systems, and I want to do something that is interesting, is not well known, um, that can help me figure out how to solve these big, complex problems. And I started playing with that, and, ever since then, I think ML evolved to AI, predictive AI evolved to, uh, LLMs, and, if we think, everything in between, we had, uh, neural network, we had deep learning. I've went through all these, uh, incarnations of machine learning and AI, and now recently at Twilio.
BP Nice.
RD Yeah, I saw the master's in genetic engineering, and I remember being exposed to gen– genetic engineering in college. Like you said, it, it was the top tech for, for a minute, but it's, it's kind of fallen off now. Why do you think that is?
IS I think if we're looking– I wouldn't say it's fallen off. I think it's found its niche. There, there is a time– every time a new set of, um, capabilities are coming, or a new technology is evolving, we're trying to apply it for everything, and it doesn't necessarily solve everything. So, I think, with genetic algorithms, it was very similar. We try to solve different optimization problems until it's found its niche to very complex optimization problems that require very specific, uh, set of parameters that have very specific dynamical behavior, um, and that's where it still lives. Like, it's not dead, it's just found its niche. It sames go to if we think about there was a time everything was neural network, and then everything was deep learning, and then everything was predictive AI, and now everything is LLM.
RD Mm-hmm.
IS But I think what we see, what we saw historically, and what we'll see now with LLM, that we're trying to apply LLM to solve every problem. It's not going to be able to do that successfully, and it will find its niche in, in the fullness of time.
BP Mmm. I think that's really interesting. You know, one of the funny paradoxes of LLMs is people trying to use them as search engines, um, and tune them for accuracy when, you know, they're originally meant to be dream machines that, uh, you know, create stuff based on our suggestions. So.
IS Right.
BP People are figuring out, right, where they apply and where they don't. And I think, in some cases, going back to older styles of AI or merging generative AI with much more rules-based, um, sort of classical techniques.
IS Right. I think it's interesting because you see the, um– and, and it's very true what you said about LLM, hence we see the hallucination, but it's not hallucination, it’s, it's a design factor of what it was supposed to do is really create that type of content. Predictive AI had a very different, uh, goal in mind, hence, it's very applicable to everything that is rule-based. If you think about security tools, for example, you're not going necessarily to think about, “Oh, should I use LLM to detect if the code is secure?” If there is vulnerability, you want something that is much more predictable, that is tuned to find these specific vulnerabilities in the code, versus something that is more of a, a high level search engine that can invent things.
BP Yeah. I remember somebody– I, I saw something somewhere the other day where it was like, you know, “We're a gen AI company,” and you know, “We're here to help you, like, create marketing content and optimize your supply chain,” and I thought, “Optimize your supply chain? Like, what does that have to do with gen AI?” But, um.
IS Yeah.
BP You know, that's the kind of place where a genetic algorithm might be really useful.
IS Right. I, I think that's what we do in Twilio. When we're thinking about the different problems we need to solve for our customers, we're trying to find the best method to really drive value for the customer. It's not about, “Okay, what's the latest and greatest? And let's go figure out how to apply that to any problem we have.” It's really grounding ourself in: what is the customer value? What are we trying to achieve for the customer? And then, what type of a model or rule base – or, sometimes, it's not even AI – that you're using in order to really generate that value for the customer.
RD Hmm. So do, do you think, um, you said, you know, LLMs are still finding their, their niche. Do you think there is a very good– a niche for it where it's, this is where it applies really well, as opposed to, you know, other areas.
IS I think it's, it's still to be determined. We see it kind of shining in the conversational AI.
BP Right.
IS It does a really good job in kind of mimicking a human type of response, so we see that growth in AI assistance, voice AI assistance. I think it does a really good job in translation, at least from what I have experienced. We, we had a project with OpenAI in Twilio where we connected their real time API, uh, to an IVR, and we really see, saw kind of a live translation from English to a different language. So, if you think in terms of customer support, uh, interactions, you can have your support agents everywhere in the world. You have your customers everywhere in the world. And, using the AI translation, you can see that, uh, engagement becoming much more natural, versus, let me find how to say that in your words, or needing to hire specific people in specific locations. These are kind of two great examples of seeing– I've seen it doing a great job in inventing, right? If we're looking into art, or music, or more creative elements, that's where I've seen it coming along very nicely. But I think that that's right now where I've seen more success with the LLM, versus other areas.
BP Yeah, I think another area we've seen here at Stack Overflow, and which I experience with my own family is, uh, you know, in the world of education, or research, or tutoring, you know, it's kind of this amazing universal tutor that–
IS Right.
BP –you can have a conversation with, um, and it's, uh, feels a lot more natural to get deep on a subject than running one, one web search, and then going to a site, and then running another web search, and then going to another site. You know, you can have this iterative conversation about a subject and go pretty deep.
IS Yeah. It's not very good in math though. It's still, it's still progressing. We tried. My kids are studying calculus and multivariable, and we tried to see, “Okay, how can, uh, ChatGPT solve more complex mathematical problems?” Not great.
BP Right? Yeah. Again, you know, to your point, one of the things that, you know, I, I think we're, we're going to see, and which is a emerging– you know, a mixture of experts or, you know, uh, you know, routing, uh, to different, you know, agents or toolkits, right? Like, when I ask, you know, an LLM agent, a chat agent, a math question, it should route that to a Wolfram Alpha Calculator and send me– like, you know, don't ask it to, you know, take its understanding of language and try to, try to figure out this math. You know?
IS Right.
BP That's, you're not using it right. So, let's dig in a little bit, I guess, on, yeah, like, what are some of the AI applications that you see, uh, getting the most traction within Twilio, among your customer user base? Let's start there.
IS So the, the interesting part about Twilio, Twilio has been using AI under the hood for a very long time, both for internal use cases, as well as external use case. Uh, if I'm thinking about some of our interesting use cases around security, commun– uh, secure communication. If we think about what's happening right now in the age of AI, and even before, there is a lot of, uh, spam, there is a lot of fraud, there is a lot of identity theft. Now there is even voice theft and, especially in the world of messaging, where you get all these messages from companies that pretend to be companies, but they're not really companies. So, for us, FraudGuard is a, is a tool that we have developed that its entire goal is to understand if, uh, if an SMS that was sent is legit or not. And we are running predictive AI under the hood for that. We have our own models that we have been training. We're doing the same across all, all channels. Um, so that is one example. Other example where we're using AI is for traffic optimization. So, how to detect if an information that was sent to the customer was really delivered or not. And then what's the nature of the error and kind of recommend to the customers on how to fix it. We have our Segment business, which is all around data and AI. So, if you think about a personalized identity, this is where we're using AI model to create kind of this personalization engagement with our customers, uh, as well as all the journeys mapping, which is how you should group audiences together and kind of make sure that, if it's a marketing engagement, if it's a support engagement, it gets to the right person on the right time. And, of course, everything around voice. So we're doing AI assistance. We have our IVR that is today AI-based. We are partnering with OpenAI, so that's kind of in– externally and where our customers are benefiting, but also in Twilio, we eat our own dog food. So we're using our own AI agents to power customer support. We're using our own, uh, flex contact center tools to kind of power our customer support. Um, we have Agent Copilot, which is something we offer to customers, but also use internally. Help agents become much more effective in their work. Um, and, you know, everyone is using AI for their developers. So we are using AI for developers as well. Help them be, uh, more productive. But, across the board, I think Twilio is, is very, a strong adopter of AI. It needs to be the right AI for the right use case and making sure that we have the desired outcome coming at the end of it.
RD Mm-hmm. Yeah, it's, it's interesting you bring up, uh, the AI, uh, used by developers that kind of coach in, that's something, you know, at Stack Overflow we think about a lot. We've done surveys where, you know, people are using it all the time, but they don't really trust the code output.
IS Right.
RD Do you get the sense that your developers actually get more of a, a productivity boost? Or is there a sort of, like, you know, double-checking-the-work cost to it?
IS I think that if we're thinking about what these AI assistants are helping developers to do, and the way I, I recommend developers that are asking me how to use these tools, it's kind of replacing search. If you know how to create the right prompt, you get the right information, and you get it structured, then it's ready to use. It doesn't mean you shouldn't verify it. We don't use anything that we don't verify first. That's, that's why we are developers. That's our responsibility, that even if a tool is designing a code or creating a code, we're still accountable to make sure it's the right code, and it does what it's supposed to do, and it's secure, and everything that comes with software developers. So I think if you are taking that with the right lens, yes, it does create a lot of productivity for developers. I think that it also enhances collaborations between developers because they're working on the same common base. It helps onboard new, new developers, junior developers, kind of acting as a peer programmer, uh, while you still have the more senior developers validating and checking. It does a lot of saving the work. But, if you think about it, in a general developer's day, they have two hours to write code. All the rest they're spending on everything else, like sitting in a meeting, waiting for the build, writing documentation. So, two hours of their day, they're writing code. So, if an AI tool can save 15 minutes of these two hours, it's a huge productivity gain for, for that developer and then for the company as well. So, if you put that in that lens: it's not going to replace your developer, it's going to make them more productive. Then you'll see a lot of success from developer tools,
BP Right. So you mentioned that one of the areas you're using it is to enhance agents. I wasn't sure if you meant human, like, agents on a–
IS Mm-hmm.
BP –on a call, if you meant– yeah. So talk to me a little bit about that because I, I do think that's one of the areas where I can see the utility, but I also think people understandably get worried about, you know, what happens to human jobs? So how do you apply it for customers? What are some of the results looking like? And how do you view, you know, the societal impact?
IS Yeah, I, I think when we're thinking about AI assistant, it has two elements. There is one, the automation of the engagement with the customer. So, when a customer is coming through the door, how much you can automate from the get-go, kind of shrink the funnel that is going to the human agent, or making sure that the information that is getting to the human agent is the right information. If we think about, like, without AI in the loop, what is the customer engagement look like? The first five, six minutes of the conversation is just the agent trying to figure out what's wrong with the customer. What are they here for? What are they asking us? Do we even speak the same language? Do I understand why you are here? And if you are putting an AI assistant at the fore– at the front door of that engagement, you can get a more consolidation summary of the problem. If the problem is simple, then, likely, the AI assistant can continue, follow that through, and solve the problem for the customer. Maybe it's a refund, in the retail use case. Maybe it's finding the document that I've been looking for because I'm stuck on writing this piece of code, and I don't know how to use this tool. So the simple use cases and some more advanced one we can see can be fully thrown by the AI assistant, but now comes the moment, then an escalation to a real human agent is needed. And, if at that point in time, the human agent needs to go back to the basic and start asking again all these questions, trying to understand why the customer here, we're basically not saving time.
BP Mm-hmm.
IS So, as much as we can, our Agent Copilot is summarizing all the previous engagement with the AI system, but it also stay as a Copilot to the agent through the conversation, so it's kind of listening to the conversation. Let's say the agent needs to find a document to share with the customer. Instead of going to Google, or to the knowledge base for the company, to find that specific document, the AI agent can do that on the fly, so that saves time. And then summarizing the conversation – recommendations, sending the email. There's a lot of automation that can make the agent much more effective.
BP Right.
IS Doing the meaningful work versus the busy work.
BP Mm-hmm.
RD Yeah. All the sort of, uh, transfer stuff, right? Like going over to file a ticket.
IS Right.
RD Or, you know, run the refund route, whatever it is, that, that actually does the end work for it. You could do that through an agent.
BP There's nothing worse than being on the call and getting, putting back on hold while they go to do something, and you're, you're back on the wait music, and you're like, “Oh, if I get cut off now, if I have to start from the beginning…”
IS And do you really want to talk to an agent when you're coming just for a refund?
BP Right.
IS I don't want to spend 10 minutes. I want, like, give me two clicks, and I get my refund, and I'm good to go. I don't have time.
BP Right. Yeah, no, totally. I think that makes a lot of sense. You know, I'll put in a quick plug, right? I mean, you know, at Stack Overflow we have this, you know, uh, team software that we sell, which is the knowledge base, and people are now using it as, you know, uh, superpower enterprise search with just an LLM, you know, API, and some RAG, you know? And, um, that way, all of the knowledge that's built up over time in your organization, or, especially right at a support center, where they've seen some edge case two times, but not in 10 years, you know, you want to have– be able to have quick access to that. And that is one of the things LLMs really excel at is sort of finding it, the needle in the haystack and summarizing documentation.
IS Right, but the key is what you mentioned, the data is the key. If you don't have the right data, then AI– whatever AI you are going to apply on top of that is not useful. So having the data, having the contextual data, having all this information that the AI can access, then it's become super powerful. If you don't have enough data to feed it, then it's only as good as the data that you have. And I think sometimes when we're thinking about LLM, when we're thinking about AI, we're very much focusing on the application layer. It's like, “Which LLM model should we choose? Should we use that one or that one? Should we use this application or that application?” And then, when you come to implement it for your own business and your own use case, you find out that the data that you have doesn't work well; hence, you're not getting the results you were expecting to get.
RD Yeah. I mean, we're, we're obviously pretty, uh, invested in the, the data, uh, behind the, the, uh, LLMs.
IS Yeah.
RD Do you think that– you know, there's a trend for the LLMs to gather as much data as possible to create a bigger, you know, overt-tuned model. Do you think people will start looking towards smaller, more targeted models?
IS Yeah, a hundred percent. I'm, I'm one of the first people that will say that I think LLM are really good for some use cases. I, I think about LLM as the generalist, and I think about the smaller model as the subject matter expertise. When you need to filter something, and you kind of need to find the, the theme, an LLM model is great because it has this general domain knowledge, it can understand what are you looking for. But then when it comes to actually do and, and perform an action or solve a specific problem, this is where the smaller models are much better because they're trained on solving that specific problem versus the generalist.
RD Right.
IS So it's really the decision in terms of: where do I need a generalist? Where do I need the specialist? And, and the world is in a combination of both. There's not going to be one winning over another. I think the world is really that hybrid world where we'll see maybe an uber large model that is creating that filtering and then calling to the smaller submodels to solve a specific problem, or generate a specific answer, or anything that you can think about, um, in these– and the combination of both have a very strong power.
BP Mm-hmm. Do you think about this at all in terms of traffic, data transfer, inference time? When I first got to know Twilio, it was 2010, and I was in the New York startup scene, and you know, the famous demo is, like, get on stage, and hack it together, and send a text message, and it comes out the other end, and everyone's like, “Magic!”
IS Right.
BP And now, you know, just over the last two days, you know, the two biggest makers of, of mobile operating systems have sort of pushed forward their idea of this, you know, always-on assistant, kind of similar to the, you know, the platonic ideal for now is like the agent from that movie, Her. You know, knows you, gets to know you better, understands the context, and then, you know, can, can, can be helpful to you. So, from Twilio's perspective, do you see, like, a change in the way, yeah, some of your bigger customers are sending data, or thinking about mobile, or considering, right, the cost, for example, of using a big model versus a small model? Because they can take a lot more time and, you know, cost a lot less.
IS I think, in general, when we're thinking– and, and you mentioned Twilio, the history of Twilio was the magic of the ability to really connect to every telco, most telcos that exist, without all the overhead, kind of simplifying the experience for developers that have a need to send SMS, or have a need to send email, or maybe everything around voice. But Twilio has grown so much since that point in time. Yes, we're still the largest, uh, CPaaS in, in the world today. We still are delivering the most messages. Just, just in the recent Black Friday/Cyber Monday, we sent billions and billions of emails, billions of messages. When you think about the amount of information running through the Twilio platform, it's amazing. But we took it one step further, and now we're talking about– we started in the world of communication, and then we bought Segment. And Segment brings all the data elements. So we have the communication data, now we have customer data. If you merge the two of them together, now you can deploy AI on the– on top of that. And what you create is a better engagement with customers. So customers are coming to Twilio, not just because we have the super network and the magical ability to send millions and billions of, of messages, and traffic, and information, and guarantee that the customer will get it. It's more in terms of now we can create it as a personalized engagement for you. So you get a lot of return on investment by using Twilio as that uber machine that is responsible and all the traffic that is going out of your organization towards the customers. So it comes also with trust because I'm sure that, very much like me, you have a lot of emails in your mailbox that is coming from brands that you likely never open and likely will never open. It just sits there, and if you have time and patience, you'll delete them. If not, your email is just going to grow. Our job – how we see ourselves – is to make sure that we're moving from kind of communication to connection and engagement. That's what the customers are expecting. That's what the consumers are expecting, and, and we see that new world and opportunities that are coming with AI is to really bring the contextual data and the communication together to create that next level of engagement. And that's the future of Twilio. That's what's happening right now. That's what we continue building towards.
RD I mean, like you said, you have a ton of data moving through your systems. Now you have, uh, the customer data attached to that. I wonder how you think about the sort of privacy and security aspect of all that, while still, you know, moving forward with AI and engagement initiatives.
IS Yeah, it's, it's a very good question. For us, it’s called responsible AI. So, when we're thinking about doing AI is first, which data we have and making sure we have the customer consent. But it's also making sure that when we're designing our system, we're thinking about security and privacy first. So, besides processes, we have security and privacy by design as part of the Twilio stack. We're making sure that there is the data governance. We're making sure that there is the right access control. We're making sure we're not using data we're not allowed to use without the customer consent. And, and what we've found out, which is an interesting one, that you can train a lot of the models without the need of the specific accuracy information that is running within the message. You can i– you can identify and solve a lot of the problems when it comes to understanding the customer behavior from the metadata that is in, in the o– in the– kind of the envelope of the message, in the envelope of the email, without the need to see what the customer was talking about. And, and if you're thinking about it, 80% of the use cases and 80% of the ROI is coming from that. And, and that's the, the mindset we're, we're using first is, first, we're looking to see everything that we can do without a need for a specific data, just with the data that is anonymized, it's not personalized, it's not impacting anyone, in addition to all the security measures we're putting on top of that,
BP If you don't mind me asking, to the degree that you can, can you talk a little bit about the tech stack that underlies all of this, and how you bolt on modern AI, and the ability to access lots of different models from different providers, as well as maybe some of your own, to, exactly, a tech stack that was created originally for, you know, text messages, emails, and has evolved over time? Is there, like, a, a monolith underneath of the old stuff and you're stacking it up? Or how, how's, how's it get done?
IS So Twilio has been undergoing in the past several years, uh, a stack modernization. So the monolith is going away, and we have moved to microservice architecture. Um, we're running Kubernetes, uh, right now we're running on top of AWS for the majority of our communication stack. We're also moving Segment, uh, to run on top of AWS. So if you think about our computer networking, they're very much an AWS stack, but if you're looking on top of that, then this is where sky is the limit. When we're using AI, we're trying to use best in breed, and sometimes we have a model picker, depends on the use case we have. We decide which type of models we're building ourselves, but most of the AI that we're using, it– we're not married to anyone. We're looking to see what is the best model that exists out there. Do we need to create our own? If we need to create our own, we build our own. Um, and it really depends on the use case. Uh, for example, Segment is using some models on top of, uh, AWS. Um, in the communication world, we have partnership with OpenAI. We're building partnership with other AI companies. So it really depends. But the underlying communication stack has he–, has went through a major redesign to get us to a place where we can be kind of multi-cloud when it comes to our AI decision.
BP Inbal, is there anything else you'd like me to ask? Or that you– anything we missed?
IS I think, um, you know, you started talking about Twilio and the premise of Twilio. When we think about Twilio, Twilio was built by developers for developers, and, even today, we see more than 10 million developers on top of the Twilio platform. Um, a lot of the questions that I've been asked since joining the company is: how is Twilio finding that balance between still being applicable for developers in the age of AI? What are you doing? And we spend a lot of time thinking about developer experience and how developers are going to change and shift. What does it mean being a junior developer right now in the industry in the world of AI? Are you ready to use all the tools? Because the universities are not necessarily doing a great job in preparing you for that. So what is the skill set that developers will come? And then when we're building all our experiences, what are developers going to expect to see? Are they going to expect to see some sort of a copilot to help them write code? Are they going to expect to see better documentation? Are they still want see an API? Or do they want to see something else? So when we're thinking about that shift into, from the developers, developer expectations, this is something that we keep on, on our roadmap. This something, this is part of our vision. How are we continue to make Twilio a developer platform in this world that keeps on evolving?
[music plays]
RD Yeah.
BP Alright, everybody. Thank you so much for listening. Let's shout out a user who came on Stack Overflow and contributed a little bit of knowledge or curiosity. Awarded 15 hours ago to Wesos de Queso. How do I prevent a toggle group from not having a toggle selected? This is a question in Java FX. Wesos’s answer was so good that it got more upvotes than the accepted answer, and we've helped over 8,000 people with the same question. So, appreciate it and congrats on your badge. As always, I am Ben Popper. I'm the Director of Content here at Stack Overflow. If you want to get in touch, email us podcast@stackoverflow. You can suggest a guest, you can ask a question, you can tell us what you're sick of hearing about, whatever, you know, we're open. And, uh, yeah, if you'd like today's episode, the nicest thing you could do for me: leave us a rating and a review or subscribe to hear more in the future.
RD I'm Ryan Donovan. I edit the blog here at Stack Overflow. You can find it at stackoverflow.blog. And if you want to reach out to me, you can find me on LinkedIn.
IS My name is Inbal Shani. I'm Chief Product Officer here at Twilio and Head of R&D. You can find me over at LinkedIn Inbal Shani. And if you want to learn more about Twilio and everything we do around AI, communication, and data, just hop into twilio.com. We'll have every answer for you.
BP Very cool. Alright, everybody, we'll put those links in the show notes. Thanks for listening, and we will talk to you soon.
[outro music plays]