Today’s streaming services have a vast content library, and a vastly different programming challenge than traditional television stations. Where linear TV tries to schedule content into time slots so the right demographics will see it and enjoy it, streaming services let you watch any content, any time you want. That means these service need to find a way to deliver the right content to every user, balancing what they will want with what will provide value over the long term. That’s where machine learning comes in. On this sponsored episode of the podcast, Ben and Ryan talk with Shrikant Desai and Sowmya Subramanian, two engineering leaders at Warner Bros. Discovery who shape how their ML program figures out what your next favorite show might be. We cover the tools they’ve been using to build their learning pipelines, how a viewer’s history can shape their future, and whether ML algorithms can fill a human curator’s role to surprise and delight viewers.
Our guests have done most of their ML work on AWS offerings, from AWS Personalize for their initial recommendation engine to SageMaker for model training and deployment pipeline. Now they’re building models from scratch in TensorFlow.
Want to see these recommendations in action? Check out the offerings at Discovery+ and HBOMax.
If you’re a ML/AL data scientist looking to shape the future of automated curation, check out their open roles.
Follow our guests on LinkedIn:
[intro music plays]
Ben Popper Hello, everybody. Welcome back to the Stack Overflow Podcast, a place to talk all things software and technology. I am your host, Ben Popper, Director of Content here at Stack Overflow, joined as I often am by my colleague and collaborator, Ryan Donovan. Hey, Ryan.
Ryan Donovan Hey, Ben. How are you doing today?
BP Good. We have a sponsored episode today by Warner Bros. Discovery. It's going to be an exciting one and a topic near and dear to my heart– content recommendation. I think this is something that all of us live every day. We're constantly served up suggestions for the next thing to read or listen to or watch, but how it works behind the curtain is a bit of a mystery.
RD That's right, yeah. Let's get the great and powerful Oz out here.
BP Exactly. All right, so without further ado I'd like to invite on our two guests. Sowmya and Shrikant, welcome to the Stack Overflow Podcast.
Sowmya Subramanian Thank you for having us here.
Shrikant Desai Thank you.
BP So we like to kick things off here to the degree that you're comfortable dating yourself. Sowmya, why don't you go first? Tell us a little bit about how you fell in love with computers and technology, what was the first thing you did coding on, and how did that sort of take you to where you are today?
SS Sure. Hi, I'm Sowmya. And how I got into computer science was that we had an Atari computer at home, so I'm definitely dating myself, and you could play Pacman on it with the joysticks that used to be there, and I used to play a lot of Pacman. And one day I found this Basic cartridge for the Basic programming language, and it was just lying around, nobody was using it. So I plugged it in, loaded it up, and before I knew it I was writing code and the computer was kind of talking back. You could make it do things like simple math, like seven plus two and it'll tell you nine. And I thought, “Oh my God, this is so cool!” So I don't think I even realized I was programming or what programming was. I kind of walked into it and really, really enjoyed it. And I think I was very much a logical thinker so things like flow charts and back in the day just kind of piecing together different sequences of steps and programming more and more complex things seemed extremely intuitive for me. And it was just magical to get this thing, this box that was playing Pacman to do things for me. So that was my start. And many years later I took Logo programming in high school, and again what was fascinating was using that to do some drawings. I drew a profile picture of Mahatma Gandhi who was the Freedom Fighter of India, and it came out so much nicer than myself drawing it on paper. So it was always really, really fascinating for me that you could produce things and create things and that creative aspect is what kept me going. I did computer science in undergraduate at Mount Holyoke College which is a liberal arts college, and again that intersection of technology and applied is what was fascinating. Whether it was programming a robot for obstacle detection, or we used Assembly language and I made a piano with all the features that I wish a real piano had, because I'm not a musician and I make mistakes, I really wanted to be able to delete back and restart from where I was correct. So one thing led to the next, I got into database systems as an intern at Microsoft, worked on Microsoft Access and SQL Server, went to grad school at University of Wisconsin Madison, joined Oracle for large scale database systems, decided I wanted to move into more consumer oriented things so I went to Google. So I was at Oracle for eight years, Google for 15 years and had a lot of fun, and recently joined Discovery and now Warner Bros. Discovery.
BP Terrific, that's an awesome history. I think one of the things you say that really rings true when we ask people that question, a lot of times what takes them into the world of programming is something that to them just seems fun or evokes their curiosity. Not, “I felt I needed to learn programming so I could have a successful career.” That's not often how people sort of get pulled into the space, so very cool to hear. Shrikant, how about yourself?
SD Yeah, so I fall in the category you just spoke about, because I was not a cool kid so I got introduced to computers mostly from gaming. At one point I kind of believed that computers are meant for gaming.
BP I wouldn’t argue with that. That's true, that’s true.
SD So that’s how I got introduced to computers. When I started engineering, that's when I got to know what the amazing things are that we can do with computers. That's when my curiosity kind of started triggering. And games are good, but how they are developed, how they are done, how they are coded, what happens behind the scenes, what kind of hardware, what kind of software, how they come together, how the fascinating things happen within a small box, that's when it triggered my interest in computer science. And I did my undergrad in computer science too. It was more of a breadth of coverages of different topics, algorithms, data structures, databases, those stuff. Then I started working with the fintech industry as my first career after my undergrad. There I spent a lot of time on, you can call it like a black Unix screen, where I spent time in shell scripting and databases and fine tuning C++ code. Of course the financial industry has legacy code where I got introduced to the basics of computers more than my undergrad studies. That's where I learned, “Oh, this is where we apply what we learned, and here I can apply my skills,” that's what I learned. But soon I realized that I need to study more and I need to understand more. That's when I came here for my masters where I spent a lot of time in data explorations, maybe you can call it a subdivision of databases. I spent a lot of time in data exploration courses and I did my thesis in data mining and machine learning and that's when I got introduced to a lot of machine learning and the AI sides of it. However, there was another interesting field going on in the industry at that time. It was cloud engineering. So I started exploring more on the cloud engineering side, even though I had a formal degree and studied in machine learning and AI. After spending some years in cloud engineering, the time came where I started realizing the potential and intersection of these two things. So engineering, ML and AI, how do you combine these things and build a product for the company? So that's when I started working more on the science as well as the ML/AI engineering systems point of view. I spent some time there. That has helped me to grow my career and curiosity. And at this point I feel even though after this career I don’t know anything because there is so much to learn outside. There is so much of industry and so much of domain knowledges so I think we should continue investing in that direction. I still feel like I should do my undergrad again because there is so much I did not learn when I was there at the school.
BP Right, that makes sense. Ryan, I'm going to let you dive in after this, but I guess the question I'll ask to both of you and whoever wants to take it, you mentioned your introduction and then both of you it sounded like had a bunch of time in databases and moving to cloud. Today we're going to talk a bit about sort of AI/ML and recommendations, so what's the toolkit that you're using today on the AI/ML side, and to what degree did the experiences you had in the past prepare you for those?
SD In terms of the tooling, you can see that right now that when I graduated there were not many tools available for ML and AI. It used to be more of modeling these things in Python or R or that kind of language, but the real-time systems used to run on C++ and Java kind of programming languages. But at this point there are so many frameworks like PyTorch, TensorFlow, and in terms of algorithm choices, there are multiple algorithms. You can use a lot of traditional algorithms in addition to that. The deep learning has been an area which has been tremendously changed a lot, not just from the algorithm side of it, but even from the computational point of view. At one point we had these algorithms but the systems and the capabilities and the infrastructures were not supporting those things. But today we can compute them and train models in minutes or seconds so we can solve them in seconds, like you can see in autonomous cars. When you're driving, looking at surroundings, those can make inferences and predictions in milliseconds. So that kind of computation and those things, that has helped us into the current stage. Coming back to the question, very specific to what kind of tools we use at Discovery, we are mostly an AWS shop. We started our journey through a lot of AWS services like AWS Personalize. Then we started using some of the SageMaker tools for the model training and the pipeline purposes. But in the long term we also started looking at building our own custom models for the survey, not just using the enterprise tools. And those custom models are on par with the industry. If I had to put my custom models against AWS Personalize, they're performing as good as, and in some aspects they're performing better than them. So we have built such tools in TensorFlow today and we are looking at building these machine learning pipelines end to end which can serve most of our use cases not using the generic frameworks. We are also looking at some technologies like Feast for the Feature Store. We are looking for some of the inference engines, something like Kaser, that kind of thing. We're also looking at ML flow architectures to manage experimentations and deploy the models. There are a lot of open source technologies and enterprise tools. We don't have to kind of reinvent the wheel again, but combining these two together we can build our own solutions, which is meant to solve the use cases for our company.
SS I would just add to this that in terms of tooling and infrastructure, as Shrikant said, the industry has come a really long way so the bar for using machine learning has been lowered so much over the last 10 plus years because of all the open source frameworks and technology that's out there and out of the box algorithms that you can leverage. So we use AWS Personalize a lot, and for getting started that works really, really good to solve problems like, “How do I do my homepage recommendation?” At the same time, if you want to go into it deeper and have a richer training data set, if you want to have a richer evaluation framework, if you want to have a faster turnaround time and more control over what the learning techniques and algorithms that you want to use are, then you also can build further from there and say, “How do I build my own ML infrastructure for my company or my stack, and what are things like TensorFlow and others that I could use to bootstrap and then build from?” So I think it's really getting to a place where bootstrapping and getting going is much, much simpler than it was years ago and then you can add to it more richer semantics.
RD That's interesting you talk about how it's much easier to get started with an ML project, but once you get started what then is the hard problem to solve?
SS I think there are a few. One is something as I stepped into Discovery Shrikant and I and a few others have discussed. Evaluation side by side involves how do you evaluate whether a model is doing what you want it to do and how do you measure quantifiably that the improvements you're doing are resulting in the changes and metric gains that you are looking for? That is something that is still not out of the box available and so that's one of the challenges for you to figure out for your business needs, your user needs, for the metrics that you are trying to improve, how do you continue to evaluate and iterate? So that's one. The second is the feature extraction or the metadata extraction and signals that you're using to inform your models and train your models. And again, we are barely scratching the surface and the work that we are doing today. We are using rich metadata that we have, we are getting with our assets, and is there more that we could be doing? Could we be having better video understanding and does it really make a big difference or not? If you want to add in, we've been experimenting with cowatch and metrics around that and signals from that. How do you meaningfully have high quality data and metrics that you're gathering that can funnel back into your pipeline? Because many times I think for a company that's not like a Google, the underlying metrics and data don't exist or the quality of that data doesn't exist. So you need to fix those problems first before you can apply machine learning effectively.
BP Interesting.
SD And another perspective here is, one angle is what Sowmya was telling us too, but many of the machine learning use cases, even though these tools are available in open source today, you write two lines of code in TensorFlow and your model will be ready to deploy. But when you hit the real time scenario, when you hit the business use cases, when you want to productionize that, it's a lot more than that and much more than that. Because when you say something needs to be put in production and you want to put it in front of your customer, they need to go through a number of checklists like in terms of operational ability, in terms of maintainability, in terms of scaling, in terms of all those things and how they work. So none of those things will be taught or told when these frameworks are built. So we had to take them, we had to kind of build it based on our use cases. That's something every company has to do on their own and there is no such tool or nobody will be telling us how to do that. So that's a very critical part of it. There was a Google article which basically published on the same thing too. ML code is a small piece of the bigger puzzle. There is always the features, the metadata data, I think someone touched on the data a little bit. So in a lot of tutorials you go through data is given to you. Data is already created for you. But in real time, data with high quality never exists. You always need to fight around it and get the data for your models and train those models. That's where people spend a lot of time, not in the training or not during the models.
BP Right. So I would love to hear an example from the time that the two of you have been working together or applying yourself to this project over the last year or two. What were some sort of eureka moments that were exciting? Like, “Okay, now we understand how best to refine this data to get the best result.” Or as you pointed out, “We're applying this to cowatching or repeat watching or binge watching. Now we really understand where we can have a great signal and then from there how the system can improve people's experience with the app that they're using.”
SD I can talk about one use case and Sowmya might have more too. So one of the use cases for example, we were using some features like the watch start and watch complete for every episode. So that was one of the signals we started using. We wanted to use more features like percentage watched, how much of the time each episode is watched as a signal for the machine learning algorithms. Straightforward, right? If you have that signal, today I can train the model and deploy it. That should be less than a week or maybe less than a day kind of work. But however, the challenging part was how do we get this data from the UI back to us? How can we process that data? And also the signals like when you say percentage watched, that needs to be computed from the data and aggregation systems from the backend. How do we do that? And that was not easy, and that was not set up to be in a consumable format. Even though you're consuming, we want to do real time recommendations. For example, if somebody goes to Discovery+ or HBO Max, you do some interaction, you watch some content, so that content needs to flow back and we build our features and we need to give you new recommendations based on what kind of interactions you're doing on the site. So that has to be near real time. So we don't serve the stale content on the website. So when you want it real time, that data needs to be aggregated and passed to the backend system. That ability was not there. So along with working for our algorithms, we also need to kind of work with the partner teams like a data team. We spoke to them, we asked how we can get that aggregated, how we can process it. It's terabytes and terabytes of data processed on a day to day basis because it's coming on the event stream, so how do you process that? How do you do that? That was the biggest challenging thing we faced. But at this point we got that data and we tuned into the model and we are running multiple experiments now on how much percentage watched really means somebody liked the content or somebody did not like the content so that we can make the recommendations based on that. So that was one recent use case I can talk about.
BP Yeah. Thank you, that was interesting. Sowmya, did you want to weigh in on that?
SS I can just add one more. I think without giving too much details, another aha or eureka moment was, as a company we are shifting from being very editorial and human heavy to being more machine heavy. And in the process mistakes that you make with the algorithm are costly because when you're making that culture shift happen, anytime some change you roll out doesn't quite deliver as you expected, which is going to happen, you lose credibility in the fact that the machine can really do this. And so very quickly you tend to fall back to the human side. And this has been really interesting to watch and balance, particularly coming from a very technology focused company where you start with the technology first and then you have to convince people why you need the human on top of it– that was Google and YouTube. Versus here, you are doing the complete opposite, which is you're starting with a very manual human heavy process and now you're saying, “What more could we be doing here and what is that 10x, 100x gain that we can get by leaning more into machine learning or automation?” So that is less specific about the models themselves, but more of a shift that you have to do when you're going through that technology transformation.
RD It's interesting, I'm curious if y'all would be willing to talk a little more about how machine learning can recommend content. As a consumer, I get recommended content all the time and I've heard it done through sort of nearest neighbor algorithms where they find other users like you. And then Spotify doing stuff with their Echo Nest API where they actually understand the content in some way. Do you have a mix of that? Do you rely on individual users? Do you apply some of the sort of human tagging to it?
SD So I can talk a little bit about that. So at the end of the day when you want to provide a recommendation it's basically based on the patterns, like how users spend time on the site and what kinds of things they like to do and not like to do. We understand that pattern and based on that we can recommend something to the user. One of the things is you can say that based on the history, we can recommend something. For example, Shrikant watches these kinds of shows and usually has an affinity towards these kinds of shows, let's recommend to him these kinds of shows. That's the general idea. Since you spoke about nearest neighbor algorithms, we use deep learning algorithms which are basically sequence based models. So what we do is we capture user history, what they are interested in, and what kind of interactions they have made. I just spoke about how time watched one of the features we see, like how much they watch, what they pick, where they spend time. So once we understand that then we make a recommendation based on, these are shows watched in the past, what is the probability that given a show, they watch it or not? So we can find these probabilities and rank them in order and serve it to the customer. And this is more from a recommendations point of view, and then there are a lot of other areas where we have navigational content. Basically the genre on the home page will be kind of customized based on every user, because based on our studies, we found out that there are five or six genres usually people stick to. And if that genre is on the second page, usually they have to navigate and come back so we want to avoid that. So content discovery is another big piece here, so how we can help them navigate better so that we personalize those things and say, “Okay, you are a science fiction genre guy and you watch most of the content from that, so let me put it on the first place so that you can always find it immediately when you land on the site,” and that is personalized to each user. The goal here is to make our customer’s life easy and make it simpler to use the app so that they can find the content they want to watch very easily. So that's a goal to provide these recommendations.
BP Very cool.
SS And the one other thing maybe I would add to this, coming from having done YouTube Music recommendations and YouTube Kids, I was leading those projects before. And one of the things I think we really want to make sure of, and I don't think anyone has really cracked this in the industry and that's what's so exciting and is the opportunity here. From the get go some of the design principles we are using in our algorithms and what we are monitoring is we want to ensure there's content diversity in the recommendation that is delightful for the user. So it's not diversity for the sake of diversity but to add to what the user is already stating as their intent and preference through their watch patterns. What are signals that we can infer where maybe some adjacent content might also be interesting to this user and not take them too far off while also introducing them to a richer candidate poll of content? And I think that is something when you observe what editors do, humans do, they form that web of connection, not because the metadata is telling you this, not because there's an action scene here and an action scene there. There is something more that a human is bringing to the table in connecting the dots. And I think Pandora had explored this model for music where they had humans looking through the connections and patterns and the fabric of what is connecting these different pieces of music together and then how do you bring it into the recommendation? So this is something we are definitely very actively exploring, which is what are the ways in which we infer signals that allow you to create a web that's richer than a one dimensional view that you get when you look at metadata or extract things from the content itself?
BP Yeah. I'm glad you make that point. One of my favorite stories I ever wrote as a journalist was about Discover Weekly on Spotify, and the data they use there is playlists created by people and then someone made a playlist similar to you but it has one song you didn't include. Well, sort of human taste is kind of the raw material that the algorithm can then do some pattern matching on and I've always loved the recommendations from that. I guess another thing I wanted to say there was that trust and safety are such an important part of recommendation algorithms and you make a good point that you don't want to just take people down a wormhole because they happen to watch one video. You want to have some human in the loop who is understanding why things might be interesting or appropriate for who you're recommending it to.
SS Yes, absolutely.
BP So I just wanted to ask one last question. When you are thinking about hiring and retaining technical talent now which as we all know is extremely competitive, what are the things that attract great AI/ML folks? I know in this space being able to sort of share your work or work on things that are open in some way is interesting, being able to break new ground or being able to work on really difficult challenges. So as engineering leaders within this organization how do you think about that?
SS For me the formula is not that different from a less competitive market. I think it's really, really the passion and the mission driven nature of the work I think is really important. And so when we look for whether we are hiring or retention, leaning into the vision and the opportunity of how we can impact users, how we can impact the future of streaming and consumption are at the core and heart of how you motivate both your team that is here and how do you attract the talent that wants to come and solve that problem. So very much lean into the passion and the mission and eagerness to solve that problem. And the second thing is technology transformation can be hard. It is also extremely empowering for engineers, for technologists, where it's easier sometimes to solve these problems when you're in a tech company because that is the DNA of the company, that's the DNA of everyone around you. Coming here and taking an infrastructure that may not have been built with some of this in mind and really going and having that moonshot vision and saying, “What am I going to do to get there? And I don't have 10 years to catch up with Netflix or anyone else. I don't have the luxury of time in a company like this. And what am I going to do, almost like a startup, to solve these problems with the experience that I have seen before as an individual to come up with a creative way of approaching it and in an efficient way?” So I think that is extremely fulfilling for engineers, particularly experienced engineers. And also for incoming college graduates who are looking for that greenfield opportunity and can have that kind of impact and scale. Even our interns have that.
BP Right, I think you make a good point. It's exciting to be working in the underdog position, to have that feeling, and also as you said, mission driven and to want to catch up to players and the tech space who might be ahead. I guess also if they get to see the stuff that comes out on HBO before I get to see it that would be really appealing. Do they get sneak previews? No, I'm just kidding.
SS I'm still waiting for my dog food access for HBO. I'm waiting for that too. That's my driver to be here.
BP Exactly, exactly.
[music plays]
BP All right, everybody. Thanks for listening. As always, we appreciate it. I am Ben Popper, Director of Content here at Stack Overflow. You can always find me on Twitter @BenPopper. Email us with questions or suggestions, podcast@stackoverflow.com. And if you like what you heard, leave us a rating and a review. It really helps.
RD I'm Ryan Donovan. I'm the editor of the blog here at Stack Overflow. You can find me on Twitter @RThorDonovan. And if you have a great idea for a blog post, please email me at pitches@stackoverflow.com.
SS I'm Sowmya Subramanian, EVP of Engineering at Warner Bros. Discovery leading our streaming platform efforts. And you can find me on LinkedIn and Twitter and I will try to keep up.
SD Hi, I’m Shrikant Desai, Director of Engineering. I manage most of the recommendation and personalization at Warner Bros. Discovery. Of course you can find me on LinkedIn. Please reach out if you have any questions. Thank you.
BP All right, everybody. Thanks for listening and we will talk to you soon.
[outro music plays]