This week we chat with Kelsey Hightower, a principal engineer at Google, about his history with networks, containers, and configuration management.
You can find Kelsey on Twitter here. His Github is here. His personal journey with Kubernetes is detailed in a nice piece here.
Kelsey has an interesting role at Google. He sits at the director level but is an independent contributor with no direct reports. Instead he works to help galvanize interest in particular tools and topics, driving adoption at a broad scale.
Kelsey Hightower When Kubernetes came out, most of the requests for the last four years were ''make Kubernetes work like the old stuff, I want my storage, I want the same network plugins, I want the same firewall role management,'' and a lot of enterprises kind of approach technology that where they say, ''Hey, can you make the new thing work the old way?''
Ben Popper Are you struggling to deploy cloud native applications to a hybrid cloud? Do you want to become familiar with Kubernetes and Istio? IBM Cloud has a set of free hands on training, ebooks, and an always on free tier of services to help you learn. Visit IBM.biz/StackOverflow to learn more. That's IBM.biz/Stack Overflow.
BP Hey everybody, welcome to the Stack Overflow podcast. I'm here with my wonderful cohosts Paul and Sara.
Paul Ford Hey!
Sara Chipps Hey, y'all.
PF Ohhhh my god, we're doing it again!
SC How's it goin'?
BP We have a wonderful guest this week, Kelsey Hightower. But before Sara gets introducing him, I just wanted to go over something that I think is pretty important. How to use Python and Selenium to get a lifetime supply of garlic pizza sticks. Can you walk me through the tricks that this coder used to stay in the deliciousness forever and ever?
PF I mean, this is a classic, right, this is ''I hacked the online promotion.'' [Ben laughs] You know, this is the pudding cups to get the air travel. I mean, this is you know, it's...I remember once I when I was a kid, we went across the country on Amtrak, partially due to peanut butter, like my mom just bought an enormous amount of peanut butter. [Ben laughs] And it gave us a significant discount. And that's how we saw America. Thanks, peanut butter. No, this is a good article. I mean, you know, I'm just gonna say the words that always give me comfort about our industry. Sara, you'll know them. Selenium WebDriver.
SC Yeah, Selenium WebDriver. I need a few minutes with this article in order to be able to tell you what happened. I think that somehow he used something to randomize the GUID. And just you could just keep refreshing, keep clicking that coupon. Keep the garlic sticks coming.
PF Yeah, but he wrote a little bot that got them lots of Papa John's garlic pizza sticks. It's worth reading. Just you know, if you don't know, Selenium, you should know Selenium. It's a critical piece of everything. I used it once because I couldn't get API access to something about 10 years ago, and I really wasn't supposed to be where I was. It was kind of a self project instead of an org. And you can, you can script that thing and just go to town. I downloaded terabytes of data. And then somehow they turned it off. I don't know what happened. Maybe they figured something out. But Selenium man, just go go check that out and get yourself some garlic sticks. It's also if you, if you ever want a really good time in a really like, ''Oh my God, no, what happened to the world?'' Go search for the guy who founded Papa John's his house. He has a statue of mating Eagles in his foyer so that that's just how you know anyway, but this guy got a lot of breadsticks. And that's why we use open source. Alright. Sara, who did you invite onto our program today?
SC Yeah, I'm so excited. So I have been a huge fan for a long time, finally got the chance to see him speak in person and code live at openJS last year. It was really amazing.
PF Oh, wait, code live isn't like an event. I'm like, ''oh, code live!'' No, no, actually programming in front of--
SC Living coding in front of 1000 people.
PF Literally my worst nightmare. Actually, like something I wake up from a cold sweat. And but this person does it.
SC Yeah, the whole audience is captivated. It was really amazing. And I knew it would be excellent to have him on the show. Really excited to welcome Kelsey Hightower, Principal Engineer at Google. Welcome, Kelsey!
KH Hey, thanks for having me.
PF So wait, what what does a principal engineer do all day? I always like to understand, like, what people's jobs are.
KH So in Google, they have a really great icy track. So technically, I sit at a director level with no reports. And in Google's world, I guess the higher you go up, the idea is that you can lead with a lot of influence and persuasion. So if there's something big that needs to happen, let's say industry wide. Can you lead through a little bit of thought leadership, meaning can get the world to understand what those ideas are? And then can you articulate clearly what needs to happen, whether it's a new open source project, improving existing products, and getting a lot of other engineers to understand and go out and build and ultimately ship that thing. And if you multiply that by, I guess, multiple projects at a given time, then that would be considered the type of leadership they would expect at that level.
BP And so when you say a director level, but an IC, is that something you made a conscious choice about? Because you prefer to be an IC? Or is that something that you kind of fell into due to the kind of work you were doing?
KH Yes, I've rolled into Google and I've probably managed people at almost every job I've had in the last maybe eight or nine years prior. And when I got to Google, this concept of the Distinguished Engineer and some of the fellows, and just watching those people, if you only have to manage one person, then you get to focus on some of the big picture stuff. So this is one of those situations where you still collaborate with other teams, it kind of feels like you have like the same number of meetings, one on ones are just as important. But you're not necessarily focused on a specific project, you're not specifically focused on maybe growing and cultivating your talent that's on your team, you're really focused on the ability to just say, ''Wow, what would be the things I would do if I had the time to do them?'' So that became a conscious decision for me to say, Wow, I'm finally at a company big enough to afford to be able to truly actually have executive level individual contributors and measure their impact outside of a large team in terms of being managed.
SC I've seen a lot of companies do this well, and some companies do this poorly. And I think I know a lot of engineers are excellent at what they do. And that doesn't necessarily mean they want to manage people. And so being able to have a track to distinguish those folks and recognize them for their excellence, and how they influence I think, is amazing. That's great.
BP So Kelsey, can you tell us a little bit about some of the stuff that's interesting to you right now that you're that you're diving into? And by doing some of that deeper work, hoping to maybe make an impact, or, yeah, that sort of ripple effect where other people want to get interested in that and start working on it?
KH So maybe a little retrospective about where I found success and why I'm trying to use that same model to try to replicate new areas. So if you think about configuration management, I went to work at Puppet Labs. But after spending years in the enterprise, using configuration management tools to do all the things over an extended amount of time, like three plus years of taking the enterprise from point A to point, I don't know, Z. And then taking that experience like that true hands on where the pager experience into the open source world to not only become a customer, but to actually be the person who makes the software. And I noticed when I got to Puppet is that you don't just write code all day and just ship the results, you actually interact and engage with the broader community, right? Some people will come in on the open source side, some people will be brand new, you have a chance to educate them all while balancing your kind of day job of writing code all day. But that's when I kind of understood that there's a way to influence the entire industry before you write a single line of code. First of all, you can listen to them. Stack Overflow, for example, what questions are people asking those will give you hints and clues about where your product can use improvement. And if you fast forward to Google, taken the ability to kind of do that across a wide range of things. And what are those things? I think security is a super interesting space now that we're starting to go from 7000 agents deployed on some server attempting to secure your environment. But the problem with all of those tools today, they're just all have their own configuration language, their own UI is different people in charge of them. There's no unifying way to get policy. So for me, I'm spending a lot of time and the areas, some people call service mesh, but I like to break things down a little bit lower. So there's policy. So in the open source world, there's a tool called Open Policy Agent, this is where you can actually define your policies in kind of an expressive way. And then you can actually use that policy engine to manage your cloud deployments, if your app developer do authz, once you've authenticated someone, and then there's tools like envoy that give people this whole sidecar way of thinking about networking, retry policies, routing, traffic, mirroring, traffic, all the things a developer would have to do if you're dealing with like services, or micro services, whatever you want to call it. So those areas are distinctly unique. And then the other side, there's a bit of machine learning inside of people's data centers, people love Google's machine learning capabilities. But how do you get that on a factory floor? How do you get that into a retail store? So that's one of the projects I work on amongst many.
PF Let me be a user, which I am and ask you. Because I mean, you're this makes sense to me. But I think we should go meta, and like, Here I am, one day, I say I'm gonna, I'm gonna put something up on Google Cloud run, I'm gonna have fun, it'll take a couple hours. I'll figure it out. And then I hit that hamburger menu. And that thing is big. This is a very, very large surface. And the same is true of AWS. The same is true of Azure. Like I mean, it. Clouds are enormous. And they're kind of operating systems unto themselves. And you're reaching out to people and saying, Hey, come on in here. What are the entry points now? Where do you get started with this? Because you can't just SSH into a server and run Apache and cross your fingers that it's going to work? Like it's so big.
KH Yeah. The entry point for most customers who I think are successful in the cloud is what are you trying to do first? And if you look at the way most clouds evolved, it is evolved based on what people were trying to do, right. The first service in Google was App Engine. There were no VMs, load balancers, storage, nothing, just App Engine, right? And this is for people that may not be aware of App Engine. It's kind of like Heroku, right? Give us your source code, and we'll run it for you. But then you start to attract new customers. They're like, No, I want a VM. No, I want something that looks like I have on prem or I want some of your machine learning API. So what's the entry point for most people? If it's a financial institution, they may say, ''look, the margins on processing stock trades is so low, we can no longer do it by buying, you know, 100 million dollars worth of gear, and then trying to get that back in five years, we need something a little bit more immediate.'' So for them, the entry point would be give us the biggest VMs you have, that we can turn off when we're not using that can do some machine learning using your special hardware that is cheaper than what we can do on our own. So that would be an entry point for them. Now, if your application developer and you come to me and you say, ''Look, I'm just going to be processing some HTTP requests, I need a decent back-end. Their entry point might be something like cloud runs, stick it into a container connected to a managed Postgres database, and call it good. So it's really, even for single organization, I might get 10 separate entry points, depending on what the team is trying to do. And that's why the hamburger menu is so massive, and this is why most people, if you've ever logged into Google Cloud, you notice that you can pin the things you care about, and then just hide the rest of the icons.
PF That is entirely true. But see, then I want to know that I'm like, what can you do with the TPU?
KH But you got to think about like GitHub, right? Like, if you really tried to start at the front door of GitHub, think about just slash of GitHub, whatever that is, internally, you would see hundreds of millions of like open source projects, right? And you would just get lost and paralyzed in the thought of opportunity. But the truth is, you don't need all those libraries. So what most people end up doing is you're writing Go, then you're going to be like, Oh, where's the go front door for that. And maybe it's gonna be some Go module registry, then just raw source code.
PF I think GitHub also has this advantage where like, at a fundamental level, while I use it and love it, Git is terrible. It gets really, really easy to get lost and confused. And then GitHub made that easier. And you're just like, it does that one thing and you're like, Okay, and then everything else follows. As long as they never take away that one thing, you're fine. You're just like, Alright, well, here's a whole lot of chaos and confusion, I will figure it out. But it mean, the cloud platforms, right, like you're talking about, they don't do one thing, they do 100 things. I love this world, because it just reminds me of different phases in computing, like it feels like one day, it probably will simplify in new ways. And they'll be like one mega cloud, or maybe not, I don't know, like, where do you see this world going, because you're in a position to shape it.
KH The reason why I want simplify all at once is because there are millions of companies out there, right. So one company might be ready for serverless everything, just take my app, run it and I'll make the app changes to live in that world. And there's a company that has 1000 times more money to spend. And when a customer with 1000 times more money to spend shows up and says ''I want to do things like I was doing in 1998. Where's your offering?'' [Paul laughs] You're going to figure out how to service that particular customer.
PF Because three orders of magnitude, right, like it'll it'll just isn't always does the trick.
KH So then what you're gonna see is, you know, always see a basically a computing history museum of offerings at every cloud provider, because this is just the way of the world.
SC Yeah, when they first started, there was that one big customer that they did the one off with, and now there's 25 of them.
PF Yeah 'cause has nobody is going to say, let's avoid enterprise and go for a pure consumer cloud offering. That's it, that would be a ridiculous statement.
KH Well, I tried to sum this up, whenever the enterprise, you're gonna remember mainframe is still a really good platform. I actually worked in finance and learned the mainframe. And once you understand how it super optimized for crunching numbers, is one of these things is why they still so new ones in 2020. But the thing I've noticed about maybe most companies, they usually have a ton of cash. So that means they can do things like when Kubernetes came out, most of the requests for the last four years were ''make Kubernetes work like the old stuff, I want my storage, I want the same network plugins, I want the same firewall role management,'' and a lot of enterprises kind of approach technology that where they say, ''Hey, can you make the new thing work the old way?''
SC Yeah, you know, one thing that I've observed or talked about on the show, too, is kind of what you're talking about, right? There's a new technology comes out, we no longer need the things of the past. But then some time goes on and some things break. And we realize what the value of some of those things were. So I'm sure you see a lot of people coming from a standard server environment from moving over to the cloud. Do you see a lot of difficulty with DevOps folks in the past that are trying to learn these new things? Is it barely seamless? Do you see them struggle? What have you seen?
KH I think it depends, right? So the biggest problem we've have so far in computing, right? We've been on this 40 year maturity curve, and most apps are written to the machine, right? ''Oh, I need to make this system call, specifically this system call. And that requires this specific kernel version.'' And most tools we've used, it's kind of been a gift and a curse, right? Like people are calling out to command line tools. And if you don't have a certain version of output from a specific tool, then everything breaks. And then what we start to do is those become our API's, even though there's no promise that they'll be stable, that we end up just building up all of this infrastructure, the whole world depends on on top of this very brittle thing at the bottom. So if you look at most of the kind of offerings we've had recently, they're all about decoupling the application from that low level metal, right? So VMware did it first with the whole virtual machines. And then when you get into Cloud, we kind of kept it going with Linux, but containers, or even any platform as a service that you try to upload with just one step. Don't worry about the Linux version, don't worry about the kernel version, you can only program to things that have a promise, like the ABI of the kernel, right? These are the system calls that are guaranteed to be around for a while. And if you stick to those, then I can evolve the thing at the bottom much faster than you saying, I need Red Hat 5.5, right, like, that's where we start to get into trouble. And then you kind of get stuck where you are. So this is where I see a lot of people who've automated things. So think about the concept of automation. You need to know what needs to happen at every step of a process. So you can go and automate it. But what if every step is ''alright, start this bash prompt, then call expect, then pipe the data into this version of grep with this flag, right, not dash v good new style, but dash v Solaris now, because I've been doing this for a while.'' Once that gets encoded into your fabric, you can't move anywhere, right? The first time I move you to like maybe a Linux, a different environment, everything falls apart. So that's the fundamental reason why I think a lot of people struggle, they go all in on these interfaces that aren't actually truly portable.
BP Can I jump in for a second here, I'm not as conversant in this. But it sounds like this is like the macro problem with microservices, right, which is that you're starting to depend on all of these different things that give you more flexibility allow you to scale up and down. You're not in that monolith anymore. But you create a lot more of these middle points in between them. And those become points of failure, is that what kind of what you're getting at?
KH Micro services is probably closer to like, trying to be a politician with no experience on how to govern 100 million people. [Ben & Paul laugh] So you have people that have been operating in a monolith, meaning they're the only person they live in their apartment by themselves. They were only child, they have no experience what it's like to really gain consensus with anything, right?
BP Not socialized. Not socialized.
KH Yep. They don't have to, right. You're a monolith. If everything you do, it's all in your head, and you're good to go. And then you're saying now I'm in charge of like, I don't know, 300 million people. And I'm just going to have them do things that I would do by myself. And people say, well, not everyone agrees on everything. And then what about security? How do you make sure people can identify themselves? Like why do we need ID I just have a single key to my door. What do we mean, we need different keys for everyone. So when people go into the microservices world, they don't understand the trade offs, when you go from one app to break it up into a million pieces. Number one, most people don't even know what the pieces should be, whether they should be bigger microservices, or should they be smaller micro services, people struggle with that all the time. And then once you do it, now you got to start introducing a network. And most people do not understand networking at all, like once you get past like IP addresses imports, most people just fall apart. And they assume that things will always make it to the other side and back again, and be fast. Like these are things that are fundamentally hard in networking, and in computer science in general. So when you couple that what the need to do metrics, observability, all of these disciplines that most people have never gotten around to, when everything's in one single binary, if you will. So I just think it's one of these things where it's, there's not a problem with it. Most people don't have experience ever doing this. You go to a conference, right? And you're like, ''Hey, you should be doing micro services.'' And you probably see this in your data, Stack Overflow, how to do micro services, the best ever. And then you get this comment, like the best comments on Stack Overflow are like a PhD thesis, right? Like they're 100 pages, one comment. If you're going to do microservices, let me elaborate and then it's like you've been reading for four hours, like I think I can do it now. Let me just copy and paste this code snippet at the bottom. Boom, microservices,
SC I guess that must be a common story of people with a monolith wanting to jump in the deep end. Is there a shallow end for that? You know, we have a monolith. We really want to start breaking things up. Where do you point people it's hard to get started?
KH Yeah. So when you say like the deep end, like you if can't swim in, you know, the deep end don't matter. Like you just don't need to touch the deep but you need to start you know like most pools will have like the three foot sections with the steps that lead in, I think people should like, you know, put your toe in there. If it's too cold for you, maybe this is not the right day to swim. Or you kind of put one foot in at a time and kind of get used to it, then you put another foot in. I think people should take that pragmatic step. So one would be realizing that you probably already have a services infrastructure and people are like ''Kelsey, no, we have a monolith I just told you.'' I said, ''Where's your DNS server run?'' ''Oh, that runs over here.'' It's like, ''Great. That's your first service, its called domain name service, right? Like it's over there.'' So you already have your micro service. So go and celebrate, go update LinkedIn, you've done it. [Sara & Ben laughs]
SC That's great. I'm doing it right now.
KH Right? Yeah. So you should do that. And then I think the next thing is, now you ask yourself, ''What's the next one? What should be the next service?'' And for a lot of people that may be authentication, right? The thing that creates jot tokens, the thing is, is authenticate users and passes back a session key or something like that, that you could say, all right, we could build that ourselves. Or we could use Auth0 or some of these other authentication services. So then I will tell people very concretely, that's where it's probably a good idea to set a flag, right? Do we use the auth service that's built into the bot monolith, so we can actually roll back in case the external one doesn't work, or we need more time to experiment, and maybe build the auth service out. So now you got two, and then that's going to give you enough league runway to say, Okay, how do we debug the interaction between these two services? How do we make sure that it's fast do we do caching do we do rate limiting, if you can get it right for one service? Now you have the foundation to start making this decision, and other aspects of the service or codebase.
PF But we want to go to microservices! [Paul & Sara laughs]
SC Like today! [Sara laughs]
KH Yeah, you can go today, like I will totally sell you a microservices digital transformation developer kit. [Paul laughs]
PF I read an article! I don't know, why, this sounds like work. And you have to be thoughtful and actually understand the platform you're building on top of...
KH But trust me, you should see how, how much pressure comes up off people shoulders when I tell them, they already have microservices. That DNS example, people just say, ''so you mean all this time, we've been doing microservices?'' I was like, ''yeah, no one told you?'' And then they just get so happy.
PF You know, that is a very good point, which is that there's so much sort of money and energy to be created when you brand something and say this is the right way to do it. But you leave it very ambiguous. Like you can just you can get people so confused and scared that they're doing it the wrong way. It's real, like I mean, microservices, to me has always just it just it's been an exhausting conversation in our industry, that I feel like I'm now in the 10th version of that conversation. I don't have a lot of patience for it. Like, you know, if things are working, let's start there. Right.
KH Yeah, I think it speaks to the kind of the lack of our kind of maturity, like when I watch you know, sometimes I do watch an ad, you see someone running through Central Park, and it's like, you know, this call your physician, if you have like this particular disease, like I don't have that disease, therefore, I'm not going to call my physician for that particular medication. When people see patterns or solutions to problems they don't have in tech. They're like, ''Wow, I should have that problem! Something's wrong.''
PF Yes! Oh my God. Well, you go through the airport. And you know, there's Accenture is telling you do you have, I don't know, squid five. And you're just like, ''Oh, my God, I better I better look that up.''
BP The other problem is that when it comes to tech ads there, they don't have to list all the side effects at the end. Micro services, you know, a little bit of nausea, some disorientation?
KH I think we should protest the Surgeon General do that. Like this may cause sleep deprivation, long work hours, and a failing production system.
PF Yeah, it's not going to be good. It's not going to be good. God, that would be amazing. If they had to list side effects for ads for technology, the entire the industry would melt in about 35 minutes, I think we would, that law would get passed.
BP So Kelsey, you mentioned the beginning of this call that you wake up early. And that's when you spend a little time working on things outside of your stuff, the projects interested in hacking on or open source. What's been interesting you in those areas these days?
KH I guess, you know, for me, the big goal for me is like when I see something new, I spent so many years studying the fundamentals of like things, right. So for example, when I used to manage infrastructure, things like Nginx, I won't go super deep. How does Nginx work? You know, threads versus processes, memory management, its config file, Lua modules, and you had all these things, you can do an Nginx. And then fast forward to 2020, maybe even 2016, Service Mesh, oh my god, look what you could do with this Service Mesh. You can put like this proxy in front of your app, and you can make the proxy do things and I'm staring at them, like, you know, Nginx could do a lot of that stuff. They're like, ''No, dude, this is Service Mesh. 2020 Cloud.'' [Paul laughs] I was like, ''you can run Nginx in the cloud.'' They're like, ''No, you don't get it. You just don't get it.'' So what I have to do then is like go and take these new things. So right now, again, I think Envoy Proxy is going to be transformational as Kubernetes was to containers as Linux was to Unix. I think the world and you already said today with many of the cloud providers and even some of the, you know, common ones like Cisco, and Juniper, they're all playing with Envoy as this kind of universal network control plane device, because it's extensible. It has all the properties that make a platform successful. So what I've been doing is really starting to understand is config language. So I'm writing the config by hand and to get them almost at, you know, 400 lines of YAML. For some of these examples that I'm working on.
PF That's, that's too many lines of YAML.
SC Yeah, that's, I've had nightmares with 400 lines of YAML.
PF Anything more than zero lines gets me very upset. [Sara & Ben laugh]
KH But the nice thing though, is a bit it's fully automatable, it's fully programmable, you can use it with the control plane. And this is the big difference between envoy and something like Nginx or old school Apache, there was no universal data control plane to go with those, those proxies. So I find that super interesting. And just like, what does it look like in production, and then documenting some of the patterns that we use at Google, to manage authorization across multiple services in a way that I don't think is universally understood or known.
SC This is so cool, the envoy was created at Lyft, and is now part of the cloud native computing foundation.
KH Yeah, and if you look at Amazon, Red Hat, Google, they all use Envoy in production in our cloud platforms to give you like your internal load balancer services.
PF You know, I'm looking this up, and it's become a really positive signal. And something has documentation in like the Python read the doc style. And it's got that sort of nested thing on the left, as opposed to being just like this full blast marketing site. Go, I'm looking at the Envoy docks, and I'm like, oh, yeah, okay, this is, this starts with an install, get the virus up, there's the YAML, I'm not gonna have to figure much out, oh, and then all of a sudden, you've deployed something. That is how you do it. Very exciting.
BP Kelsey, is there anything in particular you want a shout out, that could be an organization, a project, something cool you saw, you know, just something where you want to shine a light on somebody, and this is an opportunity to get in front of a few people's ears?
KH Oh, man, I guess it's so timely, you know, people should like vote. It's one of those things where I hear a lot of people are very vocal about the things that are happening to them in their real lives. And if you live in a society where we elect officials to help us try to figure that out, then I think you owe it to yourself to be a part of the process, which is voting, even at the local level. So the person running for city council, who's going to manage your property tax dollars, maybe you should have and weigh in and have a say, watch the videos and debates of those people. So I just think a lot of people could turn a lot of their, you know, social media activity into real life activities. And I think things like voting and paying attention to the details is probably the thing I would like to shine a light on.
SC That's great, super important.
BP Alright. It's time it's that time of the episode where Ben mangles some programming terminology. Every episode, we shout out a lifeboat. Somebody gets a badge for taking a question that had a negative score, giving an answer and then getting up to a score of 20 or more. So in this case, it is ''concatenate two char strings in a C program--
PF You can pronounce char like a pirate, it's okay.
BP We agreed, that I could pronounce it that way. Asked by be curious seven years ago and answered recently by DCA swell, so shout out to DCA swell. And congrats on your lifeboat badge. Alright, everybody. I am Ben Popper, Director of content here at Stack Overflow. If you want to reach us with ideas, suggestions, comments, etc. It's firstname.lastname@example.org. And you can find me on Twitter @BenPopper.
SC And I'm Sara Chipps, Director of Community here at Stack Overflow. You can find me at Sara Jo on GitHub.
KH Oh, sign offs. Like this is Kelsey Hightower, you can only find me on Twitter @KelseyHightower.
PF And this is Paul Ford, friend of Stack Overflow. I've said it before Kelsey just said it better than I could say it, but I'll say it again. Go vote. Vote. Vote. Vote.
BP Alright. Well, Kelsey, thanks again for coming on. It was really fascinating to listen to you talk about all this stuff. And yeah, I'd love to have you back sometime in the future.
KH Awesome. Thanks for having me.