Latest / The Experimentation Edge / Upwork on AI-Driven Ops at scale
Transcript
- Ashley Stirrup (00:33) Welcome Vinoj to the show. Vinoj Kumar is the vice president of engineering at Upwork, the world's leading marketplace where he leads engineering, building the platforms that connect tens of millions of businesses and freelancers globally. Thanks for joining us today, Vinoj. We're excited to explore your approach to testing and product experimentation at Upwork. Vinoj Kumar (00:53) Thanks for having me, Ashley. ⁓ As Ashley mentioned, I'm Vinoj Kumar and I lead engineering at Upwork. And before we get started, and I thought what might be fun for your listeners is my world actually sits at the intersection of infrastructure and product that doesn't get talked about much. I work on AI pipelines, example, backend systems, developer experience. And by the way, I measure success. My measure of success is always in product terms. Did search get faster? Did marketplace trust hold up? Did engineers ship better features because platform underneath them improved, so on and so forth. So experimentation at my layer looks a bit different. The feedback loops are longer and the blast radius when something goes wrong is much, much wider. I just wanted to put that out. Ashley Stirrup (01:40) that's really great context and if you're doing all that work around AI, I'm sure your world is changing all the time. Vinoj Kumar (01:47) Indeed. Ashley Stirrup (01:48) So how do you decide what to test and what not to test? Vinoj Kumar (01:53) That's a great question because honestly the idea that everything must be tested equally is a trap that kills most engineering folks, right? The way I look at it, it's not a yes or no question. It's a sliding scale. I always use one simple row, blast radius times reversibility. If we are touching something with a massive blast radius like the core search algorithm or the billing pipeline, it's hard to undo, right? We tested relentlessly. Shadow traffic, slow canneries, the works. But on the flip side, if it's an internal tool affecting a small team, we can roll it back in two minutes. Hey, to be honest, heavy testing is just a tax on our momentum. For those, my philosophy is ship it fast, watch the monitors, and fix it forward. Ashley Stirrup (02:20) Yes. Got it, that makes a lot of sense. And have there been any tests where you've had a surprising result that caused you to kind of think differently about your approach to experimentation? Vinoj Kumar (02:46) You mean to say what good looks like or yeah where everything in with things in terms of well what's good or are you referring in a different sense? Ashley Stirrup (02:55) Yeah, yeah, you know, because I would imagine in your line of business, there's kind of two levels to it. One is, is it faster? Is it more accurate or something like that? Better search. But then I would imagine there's also the, is it improving the customer experience? And like that can be much harder to predict. Vinoj Kumar (03:15) Yeah, yeah, yeah. So it's a great question. Actually, I think about this in two levels because of two customer sets like I described, right? First, my customers that are a lot of my customers in at Upwork are internal engineering teams. I hear the developers building the marketplace features, for example. For them, good means low friction, faster builds, reliable deploys, clear documentation. And when something breaks at 2 a.m., quick recovery, right? I measure that. through deploy frequency or mean time to recovery and developer satisfaction surveys and so on and so forth. On the other hand, my second customer set is indirect, but arguably more important. The freelancers and clients on the platform, they never see my infrastructure, but they feel it, right? For them, good means the platform is fast, reliable, and the app out features such as search, matching recommendations, they all deliver results instantly and also they're good, right? For example, a freelancer should be able to find the right job in minutes, not hours. A client should get quality proposals quickly. So basically the infrastructure exists to make the invisible seamless. That's the way I look about it, think about it. Ashley Stirrup (04:21) Yeah. And if you zoom out a little bit and think about all of Upwork, I would imagine that all the different teams each have their own different view on experimentation. Is that true? Do you see different approaches across the company? Vinoj Kumar (04:35) Yeah. Yeah, yeah, yeah. Yeah. It's funny that you asked that because one of the biggest lessons I've learned in that is when you look at experimentation and look at low quality and high quality, that high quality doesn't always mean high activity doesn't mean high quality, right? In fact, it can be the exact opposite. A lot of times, for example, a low quality experience disguises itself as high engagement. For example, if a user is frantically clicking like refreshing or changing their search terms five times, they aren't highly engaged. They're actually a little confused and they're kind of stuck in retry form. To find true high quality experience, I always look at success ratio. For example, if you're looking for something, how many successful outcomes did we get versus total interactions? To me, that's one good metric. At the end of the day, we want users to find what they need with fewer clicks, not more. So efficiency is the real marker for quality in that sense, if what you're asking. Ashley Stirrup (05:31) Yeah, yeah, there's a great example from Microsoft where they had a bug in their relevance engine for Bing. And so it led to poor organic search results and it meant the ads looked so much more relevant than the results. And so the ads started skyrocketing. They were getting a lot more revenue, but obviously that was, you know, a terrible outcome for their customers and not a good long-term one. so making sure you're tracking the right things is incredibly important. Vinoj Kumar (05:46) Exactly. Yeah, absolutely, absolutely. I'm nodding as you speak. Ashley Stirrup (06:05) ⁓ So, how do you distinguish between a low quality and high quality user experience? Like you mentioned one good example there, which is just tracking kind of completion rates. Are there other metrics that you use to figure out if your customers are having good experiences? Vinoj Kumar (06:21) Yeah, yeah, I actually have a great example for that. For example, a very interesting example, so to speak. We built an internal tool called Talk to Data. Basically, it's a natural language chatbot hooked up or data lake, like Looker. So anyone in the company, be it be marketing or product, could just ask questions in plain English without needing to know SQL. When it came to measuring engagement, it was tricky. Ashley Stirrup (06:44) Mm-hmm. Vinoj Kumar (06:47) At first glance, might think, hey, let's just count how many questions people are asking every day. That's raw data. But here's the catch. If a product manager asks the bot the same question five different ways, because the bot is giving bad answers, your dashboard shows high engagement. But that doesn't necessarily mean you have a frustrated product manager trying to get answers. That's a retry strong, not really success. what we did was we had to look at it completely Ashley Stirrup (07:08) Right. Yes. Vinoj Kumar (07:17) differently with different set of metrics to measure true engagement. First, we looked at explicit feedback. Hey, we track thumbs up and thumbs down ratio on the generated answers to gauge immediate quality. Okay, that's a signal. That's too bad. But we quickly realized we needed to look deeper at the friction of that interaction. So we also started tracking what we call turns to success. In other words, for example, with natural language tools, if a user asks the question and then has to correct three or four different five times saying things like, no, no, no, I meant Q3. No, no, exclude international. No, no, that's not what I meant. That's high friction, right? So even if they eventually gives a thumb thumbs up, it's very painful. So we want to optimize for that zero short or one short success meaning meaning they got exactly what they needed on the first or second try. And ultimately for this, the real proof of engagement wasn't really in chatbots dashboard. So we looked or, you know, just plain thumbs up. So what we did was we looked at the data teams Ashley Stirrup (07:51) Yes. Yes. Vinoj Kumar (08:17) JIRA backlog, when we saw the usage of talk to data going up and the number of ad hoc basic data tickets, right, for example, submitted to our analysts draw by 20%, that's when we knew the engagement was real, right? So at the end of the day, you want the ticket to the data scientists going down. Ashley Stirrup (08:30) Yeah. Yeah, I think that's such a great example because, you know, think all the AI chatbots are another example of where you have a hard time with that. If, you know, if people are asking multiple questions, is that really a good or a bad thing? And so we have a couple of the largest AI chatbot companies as customers at Growth Book. And what they're looking is daily average usage metrics. What you've got is even better than that because you can see basically further down the data supply chain and see that, they've stopped asking questions. over here because they're getting their questions answered here. And I think that's like one of the key challenges for a lot of product teams today is how do you build your product so that you get more visibility into the user experience so you can actually see if they had that positive outcome or not. Vinoj Kumar (09:19) Right. And sometimes it's not easy to measure, too, because you look at different signals. Sometimes you might have to combine them to actually ⁓ get to the right signal to measure, you know, what's good quality metric you're looking for. Yeah. Ashley Stirrup (09:31) Yeah, and so ⁓ that kind of leads into the what new or unusual metric have you used to kind of analyze new features? Vinoj Kumar (09:41) Yeah, it's good good segue. Actually, I actually love talking about this one. For example, what I find in my line of work is that the standard metrics are creating a massive silos across the team right now. I'll give you some examples, right? Your finance yelling about cloud costs. know, engineering always obsesses about latency and the data science folks only caring about, like, for example, model accuracy. And if you're only optimizing for one, you're actually ruining the other two. Right. It's always like almost like a triangle like the triangle, right? It comes at a cost to the other one. So we are actually, so in one of the experiments that we're doing here, we're actually tracking a composite metric. Let me call it cost per quality inference. I'll tell you what that is, right? Or CPQI. We are literally stitching together real-time cloud billing data, for example, performance telemetry and data science quality scores into a unified dashboard. So the idea being, ⁓ I mean, for the idea being that it forces some alignment across all these departments. It's like game changing. In other words, if so, for example, with this dashboard, if an engineer is making the system lightning fast, but it's tripling the AWS bill and degrading the AI accuracy, the CPI score is going to tank immediately. It's a composite score. So the idea is to keep the composite score high and strike the balance between quality, cost, and these trade-offs that we have to make. Ashley Stirrup (10:53) Right. Vinoj Kumar (11:00) And actually it's becoming our ultimate source of truth and how we balance infrastructure investments. Because fast is not always good, yeah. Ashley Stirrup (11:06) That makes a lot of sense. And I can just imagine there's a whole host of kind of cultural things that need to go along with that, like breaking down silos in terms of how people work together to get to these shared outcomes. Is that true? Vinoj Kumar (11:21) That's absolutely true, right? for example, when if you don't have this composite quality, then you have the silos you're optimizing for own, like I said before. And then actually one has an effect on the other. You can build something that's super fast and super good. But if your bill is super high, then your cost per product feature is going to be super high and it may not be very profitable. You have to strike that right balance. So in that sense, you have to find some sort of composite metrics to measure and move along. Ashley Stirrup (11:46) Yeah, and in your role, are you supporting data scientists across the company? Vinoj Kumar (11:52) Data engineering, there is different data science groups where I do have data engineering team that actually supports the data analytics folks. You can call it data science to some degree. Ashley Stirrup (12:02) Yeah. And so I would imagine the kind of the consumer and the, what do call them? The consultants or contractors, the people fulfilling the work, the freelancers. Yes. I would imagine that those populations, you're doing a lot of data science analytics to understand the users and how they're engaging with your platform and where the opportunities are to kind of improve things. Vinoj Kumar (12:09) Freelancers is the term. Ashley Stirrup (12:25) And so I would imagine that's very different than the kind of work you're doing on the infrastructure side to optimize that. Like you're looking at different metrics again and optimizing for different things versus the people focused on the UI and the end user experience in that regard. Vinoj Kumar (12:40) Yes, so there are several metrics from the end user experience. For example, even if you look at end user freelancers or client, what are some of the metrics that make sense? Hey, like I said, if you start with the outcome, you're looking at, is it fast enough? And I go to the website, is it fast enough? That means you look at latency on the engineering side, right? Or if you're doing searches and matches, are machine learning models providing the answers fast and accurately, right? Then accuracy comes into play. So then we start on the back end. What is the right model to build? What's the right size of the model? But again, you trade off on expensive models versus what it takes to deliver. Deliver that inference. That's why you need this combination of metric of accuracy, speed, cost. Everything goes in. Ashley Stirrup (13:22) Yeah, that's something that comes up a lot. just listened to a presentation from Kelly Hill at Khan Academy talking about optimizing an AI tutor. And they were looking at all those different dimensions, kind of the trade-offs between speed and quality and how do you make sure you don't give a student the answer, but you give them a nudge in the right direction. And so it sounds like you'd need to work pretty closely across those teams in order for you to say, OK, is the faster model better or is the more powerful model better? Is that true? Vinoj Kumar (13:55) Yeah, so again, that's a very, I mean, you can go and ask faster model better, but using some of these metrics I just described, you figure out the trade-off, the right questions to ask, right trade-offs to make, right? What does the feature cost in terms of total cost, especially in this AI-dominated world? What does it take? That goes into play as well, right? If you're infinite deep pockets, then you can build the features that's fast, that's good, that's accurate, right? It comes at a cost. Ashley Stirrup (14:15) Yeah. Yeah, there's a number of different angles. There's that cost. There's an interesting study, again, with the Bing folks ⁓ at Microsoft, where they found that every 400 milliseconds basically paid a year of an engineer's salary, every time they got their search results that much faster. Have you done any experimentation on the impact of performance on the likelihood for somebody to actually make a hire on your platform? Vinoj Kumar (14:47) I could answer, I mean, because it's got so many dimensions in there because I run the platform. Yes, there are metrics in place because the pipe, if look at the pipeline, what it takes, for example, to have a higher on the platform, you got your basic plumbing, you know, infrastructure piece, machine, you know, machine model learning, the serving piece, the R &D piece, I think it's a little bit lower complicated answer. But I can tell you that's where the outcome-based metrics comes into play, right? What you start off with, for example, one of the metrics, what does it cost to deliver a feature? What does it cost? That means you look at your inference cost, or for example, the compute cost, storage cost, because you're pulling information on a lot of stuff. So that you can translate into, for example, at Google, what we used to do is dollar, for example, dollar per search. It's what does it cost to, for an ad, right, QPS. Ashley Stirrup (15:12) Mm-hmm. Vinoj Kumar (15:41) queries per second. The faster you serve, more ad revenue that you can generate. So there's a metric you measure. So we have some similar metric to actually translate what it takes to deliver a feature-relevant job work in this case. Searches, matches, closing the contract zones. Ashley Stirrup (15:57) Yeah, makes total sense. And in a company like yours, as you're kind of basically trying to implement AI, I would imagine through every layer of your stack, whether it's your developers coding or kind of the front end experience or the back end search, for example. How do you see kind of AI evolving, you know, how you as a company, an engineering, as the broader engineering team are working together and, you know, like how do think things are going to change as AI continues to evolve? Vinoj Kumar (16:29) AI, I have to think about it little bit. So, if you look at any enterprise including Upwork, AI has an impact at every stage of it. For example, internally, how do you bring about efficiency using AI from a development perspective? What tools do you use, for example, from developer productivity? For example, if you think about it, because one of the organizations I run is developer productivity. What that means is, for example, to give you an example, If there are 600 to 1,000 engineers, let's say in each of them, and you have a lot of big pipelines that happen all the time, savings of five minutes per engineer on the build time times 1,000 could have an impact because it costs, this compute costs, engineering costs and stuff like that. The question is how do you shave off that five or 10 minutes of build time because you're running so many builds from developer experience. Then, for example, AI in this case, for example, could help you use of tools and efficiency. Ashley Stirrup (17:13) Yeah. Yeah. Vinoj Kumar (17:25) Being analyzing your build dashboard, hey, where are we building? Is there a room for optimization? For example, you start looking at each of the build stages, build stage pipelines. As an example, I'm giving you the walkthrough of what you did recently, for example. We looked at, for example, the build, we have so many thousands of pipelines that happen, so many stages per pipeline. In aggregate, look at all this pipeline and the build stages and see where you're. long pole is, or where you're spending most of the time, and optimize it. Maybe we can cache it here across the branches. One of the ways, for example, AI could help in here is in terms of this analysis of vast amount of data that's coming out in real time. As an example, you build efficiency. That's a classic example that you could do. ⁓ That's on the development side of things. Again, from a product perspective, you think about, we talk about agents a lot. Ashley Stirrup (18:05) Yeah. Yeah. Vinoj Kumar (18:18) Agents also resolve, for example, I focus a lot on the infrastructure side, like I said. For example, how do we get to use in a good client and freelancer experience? Uptime and relevancy. The site has to be up all the time. Relevancy. For that, let's say you're capturing a whole bunch of metrics and then an incident happens, how quickly can you diagnose that incident? You have volumes of data coming in through logs and metrics and data dog, cloud watch, whatnot, right? That's where AI can actually help us analyze that quickly to zero in on a problem. And the metric in watch is mean time to MTTR, mean time to resolve. On the flip side, is there a way can you predict its mean time to detection? For example, a lot of things are happening in a very vast distributed system of 800, 2000 microservices. AI can help you analyze that at volume and say, see some pattern over here, hotspot. I think this could be happening over there. Then we have our SREs look into it. Ashley Stirrup (18:55) Right. Vinoj Kumar (19:15) even before it becomes an incident, you can actually go take care of it, right? And that results in user experience. Ashley Stirrup (19:19) Yeah, yeah, that's a very powerful example. For a lot of people I talk to, I think we're seeing the bottleneck shift from development to testing because they're able to produce so much more code. Is that something you're seeing at Upwork? Vinoj Kumar (19:34) ⁓ That certainly is, but then that's certainly the case. wouldn't say burden, but then you apply AI to testing as well. For example, in this case, I mean, again, you asked about the AI world, there are multiple dimensions. For example, again, I'm going to break it down in three different pillars where I talked to. When we talk about infrastructure, you have to make sure any infrastructure changes have to be tested well because an infrastructure going down impacts the whole site, not one product silo, right? So in that case, your testing has to be rigorous because like I said, your blast radius is going to be very wide. In that case, you bring a lot of test tools. Again, you can apply a lot of AI agentic testing that cases as well. For example, most of the test cases revolve around statistical significance, A and B testing, and so on and so forth. In our case, I look into, hey, what's a blast radius? If it's infrastructure related, what tools can we employ to actually make them get tested? we could do zonal rollouts much faster. The testing when it comes to, for example, machine learning models and stuff, in this case, what you're not going to is whether they binarily fail or not, degradation over time. For example, you might be getting some good matches, but over a period of time, it starts drifting. So we need to have some framework in place to actually test it. So testing is not a question about before the product shifts, need to have like a 30, 60, 90 day periodic testing and watching so there is no drift. So the way you test is also shifting in a way compared to traditional software, right? Because now you have to kind of now you have to kind of monitor drifts in your responses and stuff like that. Ashley Stirrup (21:15) Yeah. Yeah. Yeah. Kelly at Khan Academy talked about, you know, you had the slight variations in how the user made a prompt would generate different results. But also they talked about snapshots of a model that, you know, the model was different from day to day to day. And so they would really try to lock in on one snapshot of the model and optimize for that. And then eventually shift to another snapshot. Vinoj Kumar (21:27) Hmm. Ashley Stirrup (21:42) ⁓ And that it required a lot more A-B testing once it was live because you could do so much on the eval side But you needed real users to really understand like are they more engaged as this leading to the business outcomes and with the Variability that a business user brings to the the app, you know, are you factoring that in as well? Are you accounting for that in terms of measuring is this good software or not? Vinoj Kumar (22:08) Yeah, yeah. So yeah, absolutely. Like, for example, I mentioned talk to data and testing actually is challenging. How do you know, for example, when you have new releases of talk to data, what first of all, what metric do you use? I talked about some of the metrics that we use and how do you test that? How do you test accuracy of the responses when you look at your scope of the answer? Let's say the question is about 10,000 SQL tables or whatnot, right? And the scope is very large across marketing, product monitoring, management and finance. how do you test the accuracy of the response or the correctness of the reliability of the response. So some of the techniques where you could do is you could use another LLM as a judge. You can actually use some machine learning models to actually compare results to kind of look at the validity and so on and so forth. There are some techniques you'll have to use differently than you use from traditional software testing. Ashley Stirrup (23:00) Yeah. And have you had an example of something that you made a solid investment in, then tested it and realized that while it looked good on paper, it wasn't necessarily going to perform well for the company. ⁓ And what I always think great about those is even though they feel like a loss on paper, often the learnings can lead to better investments over time. So is that a challenge that you've had to face? Vinoj Kumar (23:26) Yeah, yeah, yeah. Yeah, that's a great question actually. Yeah, there's one actually comes to mind. I'll give you an example. We actually built an edge caching system for user profiles. The goal was to actually cut the database compute load by 30 to 40%. And he said, it's a great cost saving. He's going to say, yes, $15,000, $23,000 a month because of the new caching layer, right? The team spent about six weeks on it. We staged it, it was beautiful response times drop because it's a new caching layer, right? And the team knew how to build these things from, I don't know, 200 milliseconds to about 80 and the cache hit rate was 92%, right? Seeing so right on target, we're like, hey, thumbs up, let's ship it, it works, right? But actually when we ran this in real production traffic patterns, the cracks started showing up. The reason was, for example, the problem was the data freshness. For example, the upwards marketplace is extremely dynamic. Freelancers are constantly updating their rates or portfolio pieces. And if a client, let's say if a client pulls up a cash profile that shows $75 per hour, But the freelancer changes it to $95 or something like that. Two hours ago, you've created a trust problem. Hey, I thought it was 75. See, this cashing is showing up as a trust problem, right? So during peak activity, roughly we had some percentage of cash profiles where it kind of could Ashley Stirrup (24:42) Yeah. Vinoj Kumar (24:49) potentially serve up in stale data. So we tried to fix it. So engineers say, hey, we'll try to fix it. Shorter TTS, it's called time to live, where you can invalidate the cash. Event-driven cash purchase we did. But every fix actually eroded the gates. It was very difficult to do this. These guys understood caching, but I think given the profile and stuff, we got to a point where by the time we got to a point where you could actually fix it, it was down to an acceptable 1 to 2 % hit rate. Ashley Stirrup (24:52) Yeah. Vinoj Kumar (25:18) really you started questioning whether the cash is really helping you or not, right? So at that time, and again, the cooperation complexity was so high, we decided this is not worth shipping. It happens. Ashley Stirrup (25:22) Yeah. Right. Yeah. Yeah, that's a great example of how, you know, reality brings a lot more complexity than the original design might have had in mind. ⁓ Vinoj Kumar (25:39) Yep, yep, yep. Yeah, it was a learning experience for sure. Ashley Stirrup (25:43) Yeah, yeah, I'm sure it's the kind of thing that caused you to think differently about each of your future projects and say, okay, what similar types of problems might we have with this project that we had with that one so that you can kind of think ahead and use the learnings from that project to think about what challenges you might face in the real world with new features as well. Vinoj Kumar (26:06) Yeah, so I mean it's a learning experience, but again each one is a slightly different learning experience. think what that taught us is. Experimentation is still good, right? We learned a lot and what the and the biggest learning takeaway is. Again, in this specific case it's we it it we realized we need to bring in more real world. Profiles into our testing into our test suite because it was very dynamic and stuff, so which means it requires a lot more. Ashley Stirrup (26:30) Yes. Vinoj Kumar (26:32) working with the rest of the teams, bringing those profiles, simulating them and whatnot, right? ⁓ Ashley Stirrup (26:39) Out of curiosity, ⁓ that seems like a perfect example for using feature flags so that you could roll out the new caching feature, but just apply it to a small percentage of the population and see how it performs there before you rolled it out more broadly. Vinoj Kumar (26:54) Yes, so in a way we do that. mean, pretty much a lot of companies do that. We use feature flags and stuff. The idea is that even before we get to that stage, is there a way that that's actually the final battleground, right? Even before, even there you have surprises. But even if you can get there, is there a way for us to actually build a system lot more closely? Again, it comes down to velocity of shipping, how fast we want to do. So one of the things we did was in a set of synthetic benchmarking and staging, for example, Ashley Stirrup (27:17) Yes. Right. Vinoj Kumar (27:22) all these learnings you create in staging phase one we always do is that, both in terms of scale and the profile through 10,000 requests per minute or at the cash layer for 72 hours and see how the results match. So basically create a simulated synthetic environment where you model the real world behavior very close to much earlier. Ashley Stirrup (27:51) Yeah, boy, that makes that makes a ton of sense. Vinoj Kumar (27:52) And then like two points, yeah, two points and also doing a lot more feature flag testing where you put it in production for eight days, seven days, do a lot of studies, looking to it before you decide whether you want to it out fully or what learnings can you get. Ashley Stirrup (28:07) Yeah, that makes a lot of sense. Well, we're almost at the end of our time. ⁓ If you were to kind of look back in your career and give someone ⁓ advice, let's say they're aspiring to have a role similar to yours at Upwork, what piece of advice would you give them in terms of how to run the infrastructure team the way you're running it today? Vinoj Kumar (28:28) That's a great question. I would just say come with an open mind. I mean, come with an open mind. Be bold to experiment. Don't hesitate to modernize. Don't hesitate to question the status quo because technology changes very fast. The world is changing very fast. It's not static. So it's always good to go with the open mind in terms of. I mean, you have to run the business. Don't get me wrong. You have to run the business. It needs to be stable. You don't want to stabilize. At the same time, you need to have enough room for experimentation, trying out new things so that always you're looking for a measurable outcome. Because infrastructure plays such a foundation. Nobody declares, hey, the infrastructure is good today. Nobody claps. But the new feature people clap. But you've got to make sure the infrastructure is up and running and running smooth. Ashley Stirrup (29:08) Makes a lot of sense. That's right. Yeah, yeah, I can imagine you need a whole different set of metrics to think about measure success in your role in the business versus somebody who's maybe more owning an individual product and they can measure revenue type of thing. Vinoj Kumar (29:35) Exactly, and always make the connection between the business metric to how infrastructure plays a role. that's where I see a lot of the times I see a miss because it's considered all the way invisible. But that doesn't need to be the case. Shouldn't be the case. Ashley Stirrup (29:47) Yeah. Yeah, yeah, I think that's such important advice. It's so easy to feel like, that's so far away that that's not my job. But in terms of selling the value of what you're doing internally, that's what the whole company cares about. So it's a very powerful perspective to have. Well, thank you so much for being with us today. This is Vinosh Kumar, VP of engineering at Upwork. Thank you so much. Vinoj Kumar (29:56) You're disconnected. That is correct. Thank you. Thank you for me, Ashley. It was fun chatting with you. Ashley Stirrup (30:17) It was a lot of fun.