Latest / The Unfiltered Engineer / Ep 11: Cloud Native Observability at Hyperscale feat. Monalisha Singh
Transcript
- Vishakha (Vish): Hello and welcome to the Unfiltered Engineer. My guest today is Mona Lisha Singh. Mona is a lead engineer at NetApp, where she spent over 11 years working on observability solutions for cutting edge hyperscale products in cloud native environments. She also holds a master's degree in computer engineering from UMass Lowell. Beyond her day-to-day work, she's deeply committed to mentoring the next generation of women in tech. Today, we're talking about what observability and reliability actually look like at scale and how that shifts your thinking as you move from IC to leading the work. Welcome Mona. Mona: Thank you, Vishakha. Thank you, Vishakha, for such a great introduction and hi everyone. Vishakha (Vish): I'm so excited to have you here. I absolutely love having people from different fields to come in and give their personal like experience, share their stories because every even if you have a colleague who comes in here they will have a very different experience from you. So I'm thrilled to have you here. Tell us where you're tuning in from today. Mona: So I'm based out of Boston Northwest is there is this very small town, Boxboro. and it it gets really lovely in fall time, so do do visit this area. Yeah. Vishakha (Vish): Yeah, I had been in Boston for the longest time and I absolutely love that we transitioned through four seasons. So let's dive in and I want to start with the most basic question because I want to see where you're at, what your understanding is of observability. how do you put it into words for yourself? Mona: Yeah. So observability, I think it's now very much have become a more buzzword. But many times everybody has their own definition of observability. Many people just think about observability is just like you need to just emit some metrics or some logs and that that's it. Then maybe you are done and that that that checkbox is done in software application. So I I feel that I always tell the team and whenever I get chance that it is not something that you can check box. It's something that people should reiterate, team should reiterate and also observability is not an of afterthought. You should think about it right from the beginning. It's like you f you think about how your application will fail and what would you need to understand that failure. So from the beginning, it's a day one thinking. Vishakha (Vish): I like that, because once you've started thinking from the beginning, you know actually what metrics you want to pick up on. A few weeks ago, I did an episode with a director, and his point was that's how you measure impact, and that's how you can bring in all these issues or successes to the forefront. But give us an example of what could observability look like. That's key. Mona: Mm-hmm. So, yes, so many times when you're starting small and just it's in a POC phase, it's a good idea to dump everything out. Like you want to log everything, you want to have it all the metrics that's possible that you want to make your app need to make your application better. But as you get to a point when it's beyond the POC stage and it's ready to be become a full grown product. Then you need to be more aware of okay, is this really needed? Is it something that you really would get a value out of it? Or if it is just a junk information, probably it's just gonna cost you to store it somewhere. So Those are the thinkings that you need to be really analytical about, review it, and then understand everything that is coming out of your application has to add some value. And if you talk about in terms of cloud, different clouds, you y these telemetry is not free. You are metered for everything that you're gonna emit, like everything egress, ingress is is metered. So you do want like don't wanna dump out info. You want critical errors, you want critical metrics out only. So that's that's that's my take on s telemetry. Vishakha (Vish): I think I jumped the gun. I wanted to actually ask you before even we started with observability, that how is your role involved in observability? Like what do you do day to day? And Mona: Mm. Yes. Vishakha (Vish): the whole reason that I brought you here, like Mona: Yeah. Vishakha (Vish): I was totally like speed forward, but maybe people do not know. So give us a little bit of insight on your role. Mona: Yes, yes, for sure, Vishakha, yeah. So we can step back and go a little bit more in like what is my role in the team. So I I have worked like I'm a software developer. Primary responsibility is still to write code. and observability is area that I have developed over the time. It just became my own strength over the time because I invested heavily in it and then learned and there is good resource out. there open source products and projects are there that you can leverage to gain that understanding. So it just over the time it became my one of my strengths. But core area is still writing code, still thinking about POCs, building applications, something that would bet benefit my organization and then bringing everybody seeing this value in the idea that I see it. So all that is still my role. And many times we get to mentor junior engineers also, that's also responsibility of lead in the team. Vishakha (Vish): Thank so much for giving that brief overview. The one thing that I wanted to understand, like my understanding of observability. So let's chat a little bit around what metrics do I see when I'm working. my focus is mostly full stack front end development and your focus is very cloud native. For us, the metrics is essentially like the core web vitals, which is. page load times and then the time to first byte and FCS and LCPs, Largest Contentful Paint. So give us some examples. What do those metrics look like in your environment? Mona: Yeah, yeah, thank you, Vishakha, for the context. Yes, observability at each layer will look very different and and and every layer has its own definition and every criteria that they need to meet. So that was a good example of front front end that you gave. my role is basically more on back end, so backend application that people see we are associated NetApp is associated with file systems. So we do have large file system products running out in the cloud. So mine is more on what What they should emit in case of any failure or any issue that we see. So how do we understand that data? So it's it's pretty similar, it's all about latency. That that we want to understand how much response did it like how much time did our application take to respond to any user request? Or if there was any failure, how how good was that report that came to us that d do we understand that failure or not? do we have enough data to triage it and then fix it in the future? So all that. Vishakha (Vish): if we were to baseline everything, then we're talking about saving money, saving time, and understanding the failures the fastest that we can, the errors Mona: Yeah. Vishakha (Vish): the fastest that we can, and improving upon the solution. So these are the two or three pillars that your metrics will always revolve around, like, to make the application or the system that you're working with fastest, and you're preventing errors. hopefully before production, and Mona: Yeah. Vishakha (Vish): then you're also trying to make things better, improving on the performances. It's perfect. So what's the hardest part of designing solutions for cloud native and hyperscale products that people don't see from outside? Mona: Yeah, yeah, yeah. So when we are talking about file system, like it's a basic back end that would store any customer, large enterprise data's all these customers could be anything, big airlines, banking systems, and we are talking about like petabytes and petabytes of data being stored in the back end. So that's the scale. and s application with this is scale running in cloud. So everybody knows that cloud brings this advantage of you don't o have to worry about the infra, you only worry about your application or Your product. But then that's that's also losing control. Now you are totally at the mercy of cloud. So so now this is the hardest part I feel was every time we have brought any application or any product to the cloud, you have to understand the cloud how cloud would a particular environment will behave in terms of failure if VM goes down, if network goes down, how does your application react to it? So that's something you learn when you do pr production product in cloud is that you're at the mercy of their infrastructure and you need to just build around it and make your system resilient. Vishakha (Vish): Okay, so that that makes a lot of sense because you don't have a lot of control over or I guess you have control over certain things but then if a service fails or if a solution fails, like you nothing you can do you can just wait. Mona: Yeah, yeah. Vishakha (Vish): Probably. Mona: You you're you have to make your application be reactive in such a way that it doesn't intra interrupt the customer workflows. So that's something that we need to learn how to handle. And it does and you're right, Vishakha, it doesn't ha only happen when you're going to cloud. It only also happens when cloud introduces any new feature. Now how you don't know again how how will your application react to it if something underlying has changed and you have to test it out in those environments. So yeah. Vishakha (Vish): I guess that's also where your observability comes into picture right like if you can predict up ahead like what you're measuring hopefully your metrics will do the justice like they will tell you up ahead of time that you are this is failing or that is failing, maybe whether the system is down or whether you've had a new feature that doesn't integrate well with your solution. But I can see that that becomes a very tricky part to handle. Do you have any examples where you've had to deal with, I bet you would have had to deal with Mona: Yeah. Vishakha (Vish): such kind of a situation, but how do you anticipate these failure modes? Mona: So what we do, Vishakha is like mostly when we are deploying any product, we do put some sort of proactive monitoring in place. Like our our own guard, safety guards that okay, if our system is running this hot, then we want something triggered, like we want some alert or some message needs to come to us. So those are our safety guards, and the moment one of those get triggers, somebody like in our team is watching for those alerts, either SRE or our engineering, and then we have steps to mitigate that. So it's all this is why it's all necessary to think these scenarios through when you are building the product. So how will you mitigate it? Okay, yeah, if your CPU and memory, like you're running hot, the you completely consuming it, then how what is the next step? How your application should offload it? Like, should you bring in like raise a ticket and then get extra resource added, or you should scale autonomously, the design a product like that? So these things definitely are pro. Like I I def I recommend proactive monitoring in place for these safety guards. So you understand. Okay, this these are our points where we will trip. So we need to know ahead. Vishakha (Vish): Absolutely. Can I say that I feel like you're alluding to a little bit something that I've been really digging deep these days. It's systems thinking, which Mona: Mm-hmm. Vishakha (Vish): is more like you're not thinking about your application, your solution in a silo, but you're working with multiple parts and you're trying to anticipate what could go wrong essentially in the future. Give me like Give me a brief overview of how do you think because you've had so much experience. This is probably a second nature for you to be talking Mona: Yeah. Vishakha (Vish): about systems thinking, but somebody who's watching who's still new to this area of how can I put the monitoring in place? What am I thinking about? How do I think about like the whole end to end and beyond that? I guess like when we say end to end, it makes it like, the whole system, but no, not really. Like you're talking about the other pieces too. Mona: Yeah. Vishakha (Vish): So give Mona: Mm-hmm. Vishakha (Vish): us some insight into how you've come to think about these systems and putting monitoring in place for these kinds of systems. Mona: Yeah, so my knowledge definitely has improved a lot with my with the failures I have seen and the improvements that I we we we all have to work with for our product. But na overall in general I I have learned this formula that it's okay. Many times you have to go through that cycle of understanding your product and it's not gonna happen in one one release or first release. You always will have run learning. So I what I would recommend I recommend people to understand how your system or your product behaves at this kind of scale. Put some feedback loop back in the engineering via these observability. Because we we only think about observability when things are going wrong. No, we we also want to understand how customers are using it. For example, this like web web example that you were giving. I want to understand how many times p the one particular page or option is getting clicked. So these this is also an important observability. So if this metric flows back to the engineering team that okay, this feature is being Used heavily by the customer, then they will put more effort in the testing and making it more stable. Many times everybody everything just gets the same attention when you have a distributed system kind of environment. So this just segregates it. So I would suggest in this case, observability also should be a feedback look back into the engineering. You need to understand. We call it like adoption data in our team. So you want to like if even if you're doing private preview kind of release or POC release. With some handful of customers, just put this back in, like these things in. yeah, I want to understand if they're using this feature or that feature and then how it's behaving when they do certain things. So and yeah, yeah. Vishakha (Vish): Go for it! Mona: Yes, so and this just gives understanding. So it's always better to be ready for this feedback loop and put some mechanism in. And slowly in like a couple of releases, you will understand the whole Vishakha (Vish): Yeah. Mona: system. Vishakha (Vish): You're talking a lot about being proactive with Mona: Mm. Vishakha (Vish): these things and trying to improve and also so that you can understand the system better. It's not just about making the system better. Yes, that's a piece of performance that comes with observability. But for even you to understand the product better, to understand your customers better, this is one way. Mona: Mm. Vishakha (Vish): one of the many ways to doing that and you if this kind of gives us a great segue to our next question, which is how do you know when a system is reliable versus it's just quiet and nothing's broken yet, but there is there are times when certain things you can't put Well, you can put observability on but you haven't yet realized that Mona: But yeah. Vishakha (Vish): I could have measured this or I could have prevented this. But is it actually reliable? Mona: Mm, is it a is it in the state? Yeah, it's still gonna work. Yeah. So this is actually a very good one and very tricky place. So like all the other answers, it depends. Yeah, the favorite ones that you might have seen in this this engineering world. It depends. Yeah, yeah, yeah. Vishakha (Vish): my god, it's so fun. depends. So give us a few examples to get context. Mona: Yes. Yes, so I'll give you answers a few depends. yeah. So the the depends are that many times we when you know how your product is, like you have understood it and it has been matured enough, then mm we recommend quiet being quiet in cloud or being quiet in this hyperscaler environment. Is a is a is is it o it's okay. nothing coming out is a sign of it's behaving what it is like it's just idle system, just waiting, waiting for that. And we do have health dashboards too, Vishakha. So that's another observability. we talked about proactive monitoring, then feedback loop to the engineering adoption data. Then we also have health monitoring in place. So no the your system is not generating active logs or anything, metrics or traces, but then they it's also pushing the h health back. Okay, I'm alive, I am alive, something like that, like a binary. So you need to have that in place. So you know that okay, it's an ideal system, but it's healthy. So I know it's reliable. When it's gonna come, when any request will come or anything will come, then it will respond. So have that health monitoring in place. So that's one of the depends. If you have this luxury of having health monitoring, I do recommend health monitoring to be in place. But many times if you don't if you're on a safe cost and don't want to waste on this health metric coming out of your cloud environment and then just then then you just assume that it's fine and then there will be one incident over the e e years or months that will tell you that okay, no, just because you're not seeing anything it's not fine. It this the things fall apart. So yeah, then you will go and iterate. So I I that's why I'm always recommending this process of going back and iterating and improving. so th the depend is the cost actually. If you're ready to bear the cost of health metric coming out and having a dashboard or s something up invest in that infra, then that's the best case. Otherwise you can just assume. But then ev eventually there there will be one incident that will trigger something. So then either you will put that metric or that scenario in observability or you will go to the health. So so that's how Vishakha (Vish): That's perfect. I like that that's a depends comes down to yeah, time and money. I have the more people I talk to, the more I'm convinced it's everything comes down to time and money. Like either you're saving time, you're saving money, or you are like getting more money through revenue through whatever ways. So it's those two things that eventually all the metrics would come down to like a combination of them or one single metric. We, like you talked quite a few things, right? You were talking about binary health, then we were talking about this feedback loop, and then there are other metrics that are generally there. Mona: Mm-hmm. Vishakha (Vish): Are these all, would you consider them as observability? I Mona: Yeah, yeah. So b basically yes, observability is a big, big ecosystem. it also depends on what is the goal for that product and but then these are something very basics. You want some health, you want some proactive monitoring, you want some data to understand when your system goes wrong and then feedback loop to the engineering that is adoption metric or observability in that sense. for my interest, I think I picked it up when I started a project in NetApp which which was which is a very like a textbook project breaking down a big monolithic application into smaller containers. So it was done a few years back, and that's how I got involved into this monitoring because we had to rethink about our monitoring and observability that was traditionally for with a application because that's just like one big piece. It's running then it's doing everything what it needs to, and if it goes just down, it's down. So here when the components fell apart then The challenge came, and that's when I got involved. I like this area, it gave gave me kick because it's always already innovating, like always, always, always. And it has a bunch of open source projects that are very good, like if you look at Prometheus, OpenTelemetry, all these are very good, good projects, and you can bring it in your product and at like minimal cost and but with all the benefits of the the product that it brings. So that's that d just gives me kick and then you they always are coming up with new features and stuff. So it's it just just gets me into the mood. Vishakha (Vish): I love that. you said that you want to have that feedback loop. which is not necessarily reactive. It's proactive, but it's also not an action. It's almost like observing again. But Mona: Yeah. Vishakha (Vish): based off of that, you would want to either stabilize it or remove it or make it better, whatever the case may be, wherever your metrics tell you. But this feels like, like in today's environment, in your day to day with AI and all these things that push you towards pushing new features and things like that. This almost seems like you have to be very intentional to keep that feedback loop and to keep checking it because it's not necessarily an action or a sprint item. So how do you manage to add that in your day to day and can you? Talk to us a little bit about whether you guide your team to do that or this is more individual level. Just give us some insight, how do you implement the advice day to day. Mona: Yes, yes. good good question, Vishakha. And and one of the favorite areas of mine too. So yes, we do want to keep tab on what is the recent, like what is the latest going in the industry, what people are doing and what what what is the new product or new feature coming in. But also evaluate continuously every release or every cycle that your engineering team is gonna go through. That whatever observability you have put in place so far. Is it adding any value? Did you see any cases where this was useful, where that was useful? So with the we have we evaluate that okay, this metric was useful, this log line was relevant, so we should have b better this message may may be more clear, added additional information. So we have to go through that each release cycle. And if you see something that is that hasn't added any value in past six, seven months, then it's time to just like okay, make it more. Or like what we call it, hide it or wrap it under the flag. we don't want it running all the time, but if we want it, if we can some enable some flag and then that data will come out. So then yes, yes, you you have to do that exercise. And that that exercise you can base out of your incident reports, like what were the incidents that came in, like for the product. Vishakha (Vish): Yeah, so it's, it's a bit of both, I guess, reactive and proactive ones, like with release cycles versus incidents, which would be more reactive. So that Mona: Mm-hmm. Vishakha (Vish): sums it up really well for me. it's not, I was thinking, I guess, my reason for asking that question was it seems like a lot of initiative from the team, from individuals to be able to keep looking at the feedback loop. But if this is like a ritual. in your team to evaluate post releases, then it becomes a little more easier to know what's the time when you're going to look at these metrics or these observability things Mona: Yeah, yeah. Vishakha (Vish): that you've put in place essentially. Now going back to the whole observability cloud native and hyperscale piece, what has been one of your most, I want to say nightmarish. but put any word that you like because Mona: Yeah. Vishakha (Vish): at that scale it gets multiplied, right? A failure gets multiplied exponentially, but that's my understanding. Tell me how that actually works. Mona: Yeah, so we have been through a few of those nightmares. So all this observability that I'm talking about also relies on the infrastructure that we are running on. So we have encountered a case where our component responsible for sending the health metric or health data out went itself down because it was not able to scale to the the the scale. Vishakha (Vish): my god. Mona: Yeah, yeah. Vishakha (Vish): Sorry to laugh but yeah I can totally see how that would become a nightmare. Mona: Yes, yes. So we just could not understand what's going on in that that region because the component supposed to push out the data that went itself down because it just could not manage the scale. It was dealing with. And that is that is one of our learning. But it was again reactive, like all these things I'm telling that to be proactive, proactive, but also there are learnings that will come so in that case we we had to go and like give that application more resources and that was a manual patching with Tessaris and like all the approvals in place we had to go and manually patch and then that component got up and then started pushing the health again and then we could see our health dashboards up and running fine. But it did like trigger a lot of alarms for the ten minutes that we didn't see anything. We were in the blackout mode. Vishakha (Vish): my God, so that's like, it's not only the health of the system, but it's also the health of the piece that's measuring it, right? Mona: Yeah. Yeah. And yeah. Vishakha (Vish): Give us the scale idea, what you were working with. Like when you say it couldn't handle the scale, what was it and where did it have to go to? Mona: Do yes. So so what happened here was that the the component that pushes out metric runs in a small pod and we usually limit these resources, memories and CPU what it is running on, and it cannot go beyond that. It's just done to make our product like more contained and so that one component is not hogging all the other resources. And since this is observability related component, so we don't want it to interfere with our data serving part. Like data path we call it internally. So it's ready to it's it's designed that okay, if in case of pressure we will sacrifice you, but not sacrifice user data. Like customer will get data. So yeah, yeah, trade-offs. Vishakha (Vish): priorities rate. Mona: So like all the environments we trade off. So we we did this paper calculation. now in case of emergency, customer will needs access to data, but we will compromise you. if we are crunch in the resource con crunch. So that happened one time. That resource that c pod I'm talking about monitoring, it is supposed to monitor like volumes, hundreds, few few we tested out at scale like five hundred or something. And then it c then one customer particularly came in in partic in a region and then just scaled it beyond for like 1000 because they were not just doing like they were the not resources were deploying at one go, they were doing create deletes, frequent ones. So all the create delete also created some stale data and then just it was holding a lot in memory, could not push it out on the time as fast as that component could, and then just like the the crashed and could not come back up. So we gave when when we logged in, we gave mermaly more resources, more CPU memory, it it went fine. But that was one learning for us that we then again took it back to the team and scaled that component. Okay. In case of emergency, you should not crash. You should scale to this limit. So Vishakha (Vish): my god, it's like getting greedy like the more you scale and then obviously somebody else will come in and I think that's that's not the point that here it's more so like be prepared for it and then Mona: Yeah, yeah. Vishakha (Vish): put in when you need to put in place you can always put in place so it's like anticipating these again failure modes or crash Mona: Mm-hmm. Vishakha (Vish): modes and To your point, it was essential for you to understand that this is a component that can be sacrificed, that can go down versus some other things, which makes perfect sense. think that's the part of observability that you have to, in a system, Mona: Yeah. Vishakha (Vish): you always have to make trade-offs and understand what's important for your application versus your customer versus what you would have needed for yourself to Mona: Yeah. Vishakha (Vish): bring all this together, I guess. We'll shift onto a little bit on your individual journey. So as a lead engineer, what's been the biggest shift in how you think compared to when you were an IC? And you don't have to stick to observability, you're welcome to, but give Mona: Yeah. Yeah. Vishakha (Vish): us a bit of your background, a journey into your career. Mona: Mm. Yeah, so it it it actually the change is very big. It's just not about the title or the role. The change is more of also mentally being prepared, for for that shift. And it did take me some time, although I felt feel like I got the role and I got understood the technical responsibility of the ro lead role that comes it comes with. But then I it took me some time to understand being a lead engineer, it doesn't mean that you'll be writing all the code. I just felt it was that was the biggest shift for me, Vishakalat. Like, okay, in IC you're supposed to just write code and all the output you produce is your that's that's your thing. That that's gonna be your output. But here, your output is not your code. Here, your output is as my previous manager told me that here your output is not your own code. You should make five more engineers being able to produce code with the same speed that you did before. So that's how you multiply and and it It did take me some time to understand or yeah, but then I didn't write any code here. So how is this credit mine? but then then it it i eventually started feeling more rewarding when I see my team members' contribu contributions and and and then everything. Like it did it was some commotion in the beginning but then it settled and I started Vishakha (Vish): Yeah. Mona: seeing value in everybody's work and associating it with my you can say vision or guidance. Vishakha (Vish): Yeah, absolutely because I kind of I can understand this on a smaller level. I In all of this, I guess, think that is a part about mentoring and guiding. But how is that very different from I want to say we're not necessarily teaching them how to code, but you're actually teaching them a little bit more on systems thinking. And how Mona: Mm-hmm. Vishakha (Vish): do you put that in place for your team members or any other mentees outside of your team too? How do you bring Mona: Yeah. Vishakha (Vish): them to this level of how am I going to think about this small thing that I'm working with, then there are four other pieces that you want to take care of. Mona: Yeah. Yeah. Vishakha (Vish): and how they could impact your piece of code or your deliverable essentially. Mona: Mm-hmm. Mm-hmm. Yeah, it did take some time for me also to understand that I don't need to tell them solution to the problem. I just need to n nudge the team members in the right direction. So I have started like doing it and it it it just comes with more practice and more understanding you build with person that you're working with. Sometimes some things work with one of your team member will not work with the other. It's it's very different individual qualities. Whatever you tried with this guy, it's not gonna go work with that that person. So I also have to continuously evolve myself and make It like make myself aware that okay, this is not the outcome I want, so maybe I should also change my approach. But the to the point like how do I want my team to be focused on something small that they are responsible for, but also be aware that nothing else breaks. I I tell them continuously to have active design discussions. don't do work in the silo. If you are working on something, if somebody else is gonna consume, then talk to that team continuously. Be be in sync, be ready to what their expectation is and what you are designing. So being in that kind of check, many times you'll see that okay, if there's this conversation is not going on, Vishakha, then one is one has made something that nobody else wants and they want totally different thing, which is not being worked on by anybody. So so that Vishakha (Vish): Yeah Mona: Yeah, so that sync needs that barrier has to be broken and I tell them you know you need to talk. And that conversation I only come in when there is some confusion or there is some some disagreement that they cannot resolve on their own. Otherwise I let them go through the second. Let ha be ready. Vishakha (Vish): That's yeah, that's a part of the training you have to get your stakeholder buy-in I think this is all it comes into that phase of understanding who your stakeholders are making sure you have alignment with them but I do believe that the human communication part has to happen regardless of how much Mona: Mm. Vishakha (Vish): AI is able to help you because until you tell it clearly what you need to build, it's going to just spit out rubbish, which nobody else is going to use, or it's going to put too many things that may be not required. So yeah. Mona: Yeah. Cannot be real. You know, to totally human interaction is very important for a successful product and I think Vishakha (Vish): Yeah. Mona: b they can the AI can aid, but it cannot just replace the that part. So yeah. Vishakha (Vish): I would love to wrap up this conversation with one more question how has AI helped you or has disappointed you when we talk about cloud native hyperscale observability? Mona: Yeah. So I think if I use AI at right places where I genuinely need somewhere that it's either mundane task and I need it to be done faster, generating docs, putting my idea somewhere in the notes or some confluence like a POC page, then it just does wonders. So that's amazing. And it just very handy when you are pitching a a product or a or a shift to a team. So it's just the very helpful, very and you can also ask like Like, okay, my audience is this. Give me like what kind of questions I'll get on this this new framework I'm presenting or this new idea. So it just does very well of that. And so I I like that part. But many times also if you have history and if you understand something very, very deep, a technology, then AI can I have seen AI manufacture its own answering. relate something. Like I if I give it five, six incidents, it just make it so generic, like whatever the the report I would want. It just related the data totally in a manner that I was not looking for. So then I have to again go back and tell that, okay, no, not looking for this. Looking for something very specific about this incident that I want my customers to be aware of. Vishakha (Vish): Again, it resides with you and it's so amazing that like AI makes these assumptions and you're like, where did you get that from? Like, how did you come? I said... Mona: Yeah, how did you generalize that this failure will lead to this failure will lead to this? No, it's Vishakha (Vish): Yes! Mona: it's isolated. Vishakha (Vish): my gosh, and this is so amazing that that fortunately there are people like yourself who've stayed in the system long enough to know Mona: Yep. Vishakha (Vish): that this is it needs to be a little bit more different than what you're producing and obviously the hope is that AI becomes that smart or the tools Mona: Mm-hmm. Vishakha (Vish): that you're working with become that smart that they can actually bring this context to somebody who doesn't have it and it gives them Mona: Mm-hmm. Vishakha (Vish): that insight that normally Mona: Mm-hmm. Vishakha (Vish): wouldn't have come without having a conversation with you. But I have loved this conversation, Mona. observability isn't unknown to me, but looking at the scale that you work with and how different it is, how how you have to think, it all comes down to again, systems. No matter what length or what the volume of data you're dealing with, it still comes down to think about it in a system. Who goes down? What trade-offs are we making? What decisions do we have to make? What is the judgment call? What are we keeping at stake? So I absolutely had the most wonderful time. having this conversation with you and we are at the part of the episode where I ask the same three questions to all my guests. So my first question to you is, if there was one takeaway that you'd want people to have from our conversation, what would that be? Mona: Yeah, that's that's one key takeaway which I have for me would from me will be that observability is not an afterthought. It's something that comes in day one, from day one. So Vishakha (Vish): that also pushes people to think in systems. think if you're thinking about observability from the beginning, you also can tell that primary success metrics, secondary success metrics, then there is tertiary and how can I make the system better? And it can seem overwhelming, I think in the beginning, but the Mona: Yeah. Vishakha (Vish): more you give it thought, I think the more it becomes clearer But when you think about observability into the picture, I can see that it pushes you. to design at that level that, Mona: Mm. Yeah, it's yeah. Vishakha (Vish): I'm going to affect five more teams, so let's bring them in. Mona: Yeah, yeah, yeah. It just widen up the lens for the team that okay no Vishakha (Vish): Yes! Broader. Mona: yeah, I know need to think about how the system will fail. then Vishakha (Vish): No? Mona: do I so many times you're just thinking from the observability perspective perspective, but then you br make a better design. no, I need to design for this this th this failure. Yes, I need to handle it gracefully. Vishakha (Vish): If you know these things up ahead, the implementation is just so small. Mona: Mm-hmm. Mm-hmm. Mm-hmm. Vishakha (Vish): It's not even a big deal. You can go in and make changes. If you put in place those placeholders for observability, that I need to get this data from my system here and here and here. So thank you so much for that takeaway. Second question that I have for you is there a book a show or a podcast that you've been loving or you recommend other people? Mona: Yeah, definitely I would recommend your podcast, Vishakha. It's really amazing. Vishakha (Vish): thank you. Mona: Yeah. Yeah, no, I loved it. I loved this conversation and the enthusiasm of the whole questionnaire that we went through. So it's just very good. But apart from this podcast, I Vishakha (Vish): Yeah. Mona: do have one good recommendation that I read recently. It's a very good book, Lessons in Chemistry. And I would recommend it's like basically women in tech, like how it w it's a fictional book, but it just gives you a a perspective on the struggle of a person who wanted to just push boundaries, who's just not ready to accept the the status quo of things. Vishakha (Vish): I love that. Thank you so much for sharing it. And the last question is, how do you decompress from work once your laptop is off? What do you do? Mona: Yeah. Very difficult actually these days, like in this highly competitive world that we want to like shut shut off the laptop and then just step away. But I try. I do take little trips with family occasionally. Not it doesn't have to be like a big fancy getaway, but just smaller day trips to the park or the beach or somewhere that we can all just go and relax. So I love that. Another thing I like to do is the cooking just gives me the peace of mind. Vishakha (Vish): to process all the things that you've gone through in the day. Mona: Yeah. Yeah. Yeah. Vishakha (Vish): I find that these mundane things is where you get to decompress from work Mona: Yeah. Vishakha (Vish): But thank you so much Mona it has been a pleasure having you here and I will see you in the next one. Bye! Mona: Bye bye. Thank you, Vishakha