Latest / Nutanix Weekly / Nutanix Weekly: Honey I Shrunk My Cluster (Multiple Nodes Down in RF2)

Transcript

Nutanix Weekly: Honey I Shrunk My Cluster (Multiple Nodes Down in RF2) · Nutanix Weekly

WEBVTT 1 00:00:02.270 --> 00:00:19.830 Andy Whiteside: Hello, everyone! Welcome to episode. 63 of New tennis week. Name your host, Andy White Side it's a December twelfth, 2,022 and I've got a a big crew of smart guys. Let me get them introduced here real quick, Harvey Green, my co-host from probably day one Were you on day one already I can't remember. I feel like I was. But 2 00:00:20.050 --> 00:00:24.560 Harvey Green: who knows? 3 00:00:25.180 --> 00:00:30.590 Andy Whiteside: So, Harvey, this is your first December, actually running a business running this entire Gov business. 4 00:00:30.660 --> 00:00:33.099 Andy Whiteside: you know it's getting ready to get crazy right? 5 00:00:33.140 --> 00:00:35.100 Harvey Green: Yes, 100%. 6 00:00:35.600 --> 00:00:48.029 Harvey Green: It has already done that. I don't. I don't mean to scare you, but that that means you're making money as long as the people pay their bills. Yes, that is correct. 7 00:00:48.120 --> 00:00:54.029 Harvey Green: It's always the caveat. That is always. It's kind of the way most businesses run, I think. Right? 8 00:00:54.230 --> 00:01:06.550 Andy Whiteside: Well, yeah, it is. I've been this book about about You know a retailer retailer, he retailer from back in the 2,000. You guys may value America. I don't. I didn't remember it, and I do now that this is the book. 9 00:01:06.880 --> 00:01:13.319 Andy Whiteside: But it reminds me how many companies out there run with the idea of just creating revenue because they're trying to sell themselves off as quick as they can. 10 00:01:13.490 --> 00:01:14.789 Andy Whiteside: and 11 00:01:15.300 --> 00:01:18.409 Andy Whiteside: you know. And when you have to pay your own bills, that's a different world. 12 00:01:20.380 --> 00:01:26.210 Jirah Cox: Yeah, I think you're either fundamentally in that yeah customers pay the bills model, or I guess you're running a that collecting agency. 13 00:01:26.250 --> 00:01:27.730 Jirah Cox: Those are kind of your 2 choices. 14 00:01:27.770 --> 00:01:32.209 Andy Whiteside: Well, 100%. That's the integr. Does a lot of that. We we track down a lot of payments. 15 00:01:32.480 --> 00:01:38.270 Philip Sellers: I think. Ultimately all businesses do, unless you pay up front or whatever. Philip sellers. How's it going. 16 00:01:38.690 --> 00:01:40.690 Philip Sellers: Good! How are you? Good! 17 00:01:40.740 --> 00:01:42.350 Andy Whiteside: So you are 18 00:01:42.930 --> 00:01:55.779 Andy Whiteside: coming from the customer side the Vmware of Vmware customer. Specifically, the new tanks piece is something. Have you always looked at it like longingly from afar. Or is this, you know, really your first for a into the Newtonics world? 19 00:01:56.420 --> 00:02:09.770 Philip Sellers: so I spent a day with Gyra at the Durham headquarters last month, and he made me a believer. yes, I did mispronounce that. so? 20 00:02:09.850 --> 00:02:25.049 Philip Sellers: no, you know it's it's really interesting coming from a Vm. Or background. And and looking at where new Tanks is at today. you know I knew it when it was simplicity versus mechanics, you know. The hyper converged wars back before there was V. Sand. 21 00:02:25.160 --> 00:02:34.150 Philip Sellers: all of that kind of thing. So I followed the company for a long time, and i'm really impressed with the ecosystem and services that they enable. So 22 00:02:34.160 --> 00:02:49.669 Philip Sellers: I would say, what I see about it today is it's really a platform play, and that's the investment for a customer is that you're investing in a platform that's gonna allow you to deliver your it services that you need. And that's a really cool. 23 00:02:49.920 --> 00:02:55.920 Philip Sellers: It's a cool value. Prop: When you start going out and talking to customers about where Newton is at the day. 24 00:02:56.030 --> 00:02:56.750 Yeah. 25 00:02:56.950 --> 00:02:59.750 Andy Whiteside: yeah, All on prem Colo. 26 00:02:59.970 --> 00:03:04.010 Andy Whiteside: in the cloud that matter on your terms where you want it 27 00:03:04.040 --> 00:03:05.420 Andy Whiteside: right at any point. 28 00:03:06.170 --> 00:03:08.890 Jirah Cox: Happy to have you on the 29 00:03:09.180 --> 00:03:14.939 Andy Whiteside: you've been around for quite a while as well. You're still coming back, so must be having a little bit of fun. 30 00:03:15.190 --> 00:03:16.690 Jirah Cox: Yeah, man. No, it's a blast 31 00:03:16.840 --> 00:03:23.090 Jirah Cox: you guys put up the only you canx flavored partner Podcast that i'm recording today 32 00:03:23.520 --> 00:03:27.110 Philip Sellers: this week only one this month, I hope. 33 00:03:27.180 --> 00:03:28.690 Jirah Cox: Yeah. 34 00:03:28.860 --> 00:03:36.380 Andy Whiteside: we appreciate you having having you on here, you really do validate these conversations and get a lot of really good feedback from it. 35 00:03:37.430 --> 00:03:38.830 Jirah Cox: Yeah, it's fun to do 36 00:03:39.310 --> 00:03:44.320 Andy Whiteside: Ben Rogers been you've been a customer, been a a friend of ours and multiple fronts. 37 00:03:44.350 --> 00:03:47.310 Andy Whiteside: is doing this podcast. What you thought it would be. 38 00:03:47.930 --> 00:04:00.920 Ben Rogers: it's been very interesting, you know, when I did the Citrus podcast I had 25 years of citrus under my belt. So i'm very confident, and wherever we went this is a little bit different of a ball game, and so there's been a couple of times I've showed up 39 00:04:00.930 --> 00:04:18.250 Ben Rogers: learned on the fly. You know. What I try to do is put it in a customer's perspective, and either ask questions. I think customers would want to know answers to, but might be afraid to ask or lean Our friend the Ninja gyro here, and try to get the scoop from him. As I got him on the line. Just 40 00:04:18.260 --> 00:04:36.479 Ben Rogers: you being around him is definitely a great resources. So has it been when I thought I don't I don't know if I knew what to think about it. But, man, I'm enjoying myself. And I always get every time I publish this podcast I always get positive feedback from mechanics as an organization. 41 00:04:36.490 --> 00:04:48.010 Ben Rogers: and also customers of Newton that are out there that say they learn X, Y. And Z from listening to this. You know that that's too polite to go like man. I thought there was a lot more preparation as a listener. 42 00:04:48.110 --> 00:04:57.340 Andy Whiteside: Well, he like you, said he did the citrus ones with us. So he he saw the wing in it model that still produces fruit. 43 00:04:58.760 --> 00:05:10.419 Ben Rogers: All right, so our blog let's digress on that for 1 s for those of you that are listening to this, you know Andy's philosophy on these podcasts is, you learn the subject as you log in 44 00:05:10.430 --> 00:05:24.090 Ben Rogers: to the podcast. So all of us have about 2 min to digest what we're gonna talk about it expert format. So for all of you that are listing, just kind of understand we're when we sit at the table what we're dealing with on that. 45 00:05:24.650 --> 00:05:33.840 Andy Whiteside: Yes, but like in this case the blog is the title of it's, honey. I shrunk my cluster multiple nodes down in Rf: 2 46 00:05:33.850 --> 00:05:47.499 Andy Whiteside: by what looks like Jerome Dial in would be the name if I want to try to repronounce that last name I would get. I would guess that's what I would guess, feeling I don't know where I got the th there was no th. I made that up 47 00:05:47.850 --> 00:05:50.650 Andy Whiteside: telen that makes total sense. But 48 00:05:50.700 --> 00:05:56.690 Andy Whiteside: gyro, you brought this blog, which thankfully you bring a lot of these prepared to talk about them. 49 00:05:56.850 --> 00:06:01.110 Andy Whiteside: What's the gist of what happened here, and why he wrote this? 50 00:06:01.770 --> 00:06:08.479 Jirah Cox: Yes, this is. I love this post from Jerome. It's on our community blog. By the way, where you know anybody can join as a member of the community. And 51 00:06:08.610 --> 00:06:14.100 Jirah Cox: and you know, if you want. If you've got something you want to say for sure it's a great platform for it. 52 00:06:14.330 --> 00:06:30.819 Jirah Cox: Jerome posted this this right up from from conversations he's had before of, you know. Hey, if i'm running in this case an example of 7 nodes with Rf. 2 right, which is our structure for maintaining a data protection sla of writing all data twice within a cluster. 53 00:06:30.970 --> 00:06:37.350 Jirah Cox: so that cluster. We say that it could lose one anything right, a disk or a node. What happens if it loses 2 54 00:06:37.380 --> 00:06:48.559 Jirah Cox: right? And what happens in that? In that case so great question, Great exploration. Here it's a real real world customer facing question that we get a lot actually right is like the so we plan for this much failure. But if we get more failure than that. 55 00:06:49.020 --> 00:06:59.150 Andy Whiteside: So when he says in the second paragraph, we always, when we have larger customers, we customers clusters. We always recommend Rf. 3. Is that just because 56 00:06:59.230 --> 00:07:01.689 Andy Whiteside: more is better if you have the space. 57 00:07:02.170 --> 00:07:11.700 Jirah Cox: Yeah, and I've seen some of the stats the it comes down to, you know, like anything right if you ran even just like what things just hypervisor. Only 58 00:07:11.770 --> 00:07:17.320 Jirah Cox: if you're thinking about compute availability, Well, gee! If I run a 10 node cluster or a 100 node cluster 59 00:07:17.380 --> 00:07:20.600 Jirah Cox: at some point I want to size beyond just n one 60 00:07:20.690 --> 00:07:22.879 Jirah Cox: right like I wouldn't run a 99 node 61 00:07:22.910 --> 00:07:28.219 Jirah Cox: compute cluster of any hypervisor with only n plus one availability, because my odds as a 62 00:07:28.250 --> 00:07:33.889 Jirah Cox: her practitioner, as the admin having more than one availability zone, right, a blast radius, call it 63 00:07:34.000 --> 00:07:44.680 Jirah Cox: a single node, a compute factor. down at any one time. Those odds increase astronomically beyond a certain count threshold. it always depends. But you know, usually 64 00:07:45.310 --> 00:08:03.189 Jirah Cox: let's. Let me fairly represent a lot of viewpoints here. Most Esses would tell you by by at least time, you're hitting like node 24 in a cluster. We're probably a threshold where we want to be designing for what we call R 3 right of all data written 3 times within the cluster versus all data written twice in a cluster. 65 00:08:03.200 --> 00:08:06.969 Jirah Cox: because my odds are having one node fail, and then another node 66 00:08:07.030 --> 00:08:15.830 Jirah Cox: fail after that, or maybe i'm doing maintenance. I have one down and then another. Node chooses that time to purple screen, or whatever it is. increase at that size 67 00:08:16.160 --> 00:08:21.669 Andy Whiteside: that that's kind of my that's kind of my wife's logic. I've got 4 cars, most of them older and 68 00:08:22.100 --> 00:08:31.109 Jirah Cox: chances are the more older cars I get, the more chances. One of them is we broken down? Yeah, Fantastic analogy, right? I mean, I always say hardware gonna hardware. So 69 00:08:32.330 --> 00:08:33.860 Andy Whiteside: it's gonna do what it's gonna do. 70 00:08:33.970 --> 00:08:35.090 Harvey Green: True story 71 00:08:35.159 --> 00:08:45.099 Andy Whiteside: all right. So he gets a couple of screenshots of his environment. The first one is redundancy, factor, readiness with a category of 2. The options. If I had to drop down, ability here would be 72 00:08:45.160 --> 00:08:47.860 Andy Whiteside: 2 and 3. Are there any other options? 73 00:08:48.650 --> 00:09:04.569 Jirah Cox: Basically, that's that's true for like you know 99.9, 9 lot of 9 here, most workloads. It's going to be Rf: 2 and Rf. 3 for for a very small amount of workloads where you're doing in app redundancy right, and have availability. I'm thinking of like explan, I think enough to do from thinking of 74 00:09:04.840 --> 00:09:18.059 Jirah Cox: a couple of the workloads where the application is already gonna split that right and store it 2 places. Then, of course, you can do Rf: One understanding that when you're running Rf: one, if something goes bump, and like a disk fails, or whatever it gets yanked. 75 00:09:18.110 --> 00:09:18.910 Jirah Cox: Then 76 00:09:18.940 --> 00:09:30.790 Jirah Cox: you told us you already have availability of the application little elsewhere. Right? So we don't rebuild that data. So. but yeah, for the most part, from all for common virtualization use cases, it's going to be Rf: 2 and R. 3. 77 00:09:31.250 --> 00:09:45.979 Andy Whiteside: I have to tell a little joke of myself for the longest time. I I said. Rf. Stood for right frequency, and then somebody pointed out that right starts with a W. And I was like, yeah, so, Andy, I think there is one thing that's worth mentioning here. 78 00:09:45.990 --> 00:10:05.710 Ben Rogers: and Philip kind of handed to it when he gave his introduction. This is really what creates the platform of Newtonics our ability to do this Rf. Factor. Not only is it protecting us from a from a data protection standpoint. But this also comes into play with. How quickly will the cluster recover? You know, if we've got that data spread peanut butter 79 00:10:05.720 --> 00:10:20.050 Ben Rogers: as across these nodes. If a node fails, another node can take over really quick because it's got the data on it. Also, we talked a lot about Vdi in in your podcast. And this is also what makes our Bdi the data. 80 00:10:20.060 --> 00:10:36.669 Ben Rogers: even though we're replicating the data off, we're still keeping that workload local on the node that the data is there. And again, if that node was the fail, we have the metadata to know where to pick it up. You know where we need to go next to get that. So again going back to Phillips. 81 00:10:36.680 --> 00:10:46.930 Ben Rogers: you know his mention of this is a platform. This is at the heart of this platform, and this really what makes new tanks saying when it comes to things like 82 00:10:47.010 --> 00:10:55.769 Ben Rogers: performance, replication, disaster, recovery. These are all the things we had on this idea of redundancy factor 83 00:10:57.050 --> 00:11:02.110 Jirah Cox: totally. We we live in D by Andy. I was gonna say, you know all 5 of us here right 84 00:11:02.260 --> 00:11:11.590 Jirah Cox: as technologists in the Carolinas. We're already fighting uphill battle. Right? Thank you for not like, you know, highlighting our bad spelling on top of that right? 85 00:11:12.500 --> 00:11:15.060 Andy Whiteside: Oh, you must know I went to elementary school. 86 00:11:17.040 --> 00:11:19.749 Jirah Cox: so I said, all 5 of us here right I cast a wide net. 87 00:11:21.360 --> 00:11:27.100 Andy Whiteside: So if you want to tell us what this manage Vm: high availability piece means. 88 00:11:28.250 --> 00:11:34.529 Philip Sellers: probably looking at me like you. You're not steering the screen, you idiot! Yeah, this part 89 00:11:35.520 --> 00:11:37.580 Philip Sellers: Zoom the 90 00:11:39.490 --> 00:11:40.810 Philip Sellers: hey? Go ahead, John. 91 00:11:41.080 --> 00:11:52.559 Jirah Cox: Oh, sure. So so the check box here's what channel screen there. is that ha reservation? Right. So as a as the compute layer, right as virtualization, managing the the virtual machines, right, you can 92 00:11:52.670 --> 00:12:05.920 Jirah Cox: run the high availability engine and basically 2 models out of the box. You're getting Ha! And you're getting best effort right where impacted Vms from like a hardware failure event will get restarted automatically on surviving nodes in the cluster. 93 00:12:06.100 --> 00:12:23.940 Jirah Cox: that is, of course, out of the box You get best effort, and then with this checkbox you can opt into hey? Go ahead and pre reserve memory for me. So I get guaranteed availability of my beams already. Have a pre reserved space. and then to boot back up basically you're moving from an n plus 0 Vm: memory model in front of one. 94 00:12:24.200 --> 00:12:24.890 Andy Whiteside: Yeah. 95 00:12:25.520 --> 00:12:38.079 Andy Whiteside: Yeah, I I jokingly brought that up to Philip because some of these things have been around for a little while or a while now and then mechanics platform, but the when I meet customers they assume some of this stuff is only available in the Vmware side of 96 00:12:38.130 --> 00:12:39.270 the solution. 97 00:12:39.870 --> 00:12:55.649 Philip Sellers: Yeah. And I mean this. This is essentially the same as the the Vmware. So you got a parity here. So you know. Ha! You You put in the number of hosts to fail, you know toleration level and in Vmware, and it reserves that space 98 00:12:55.710 --> 00:13:01.810 Philip Sellers: very similarly, although it's a slightly different take, I guess, here from the new tenx platform side. 99 00:13:02.180 --> 00:13:16.569 Andy Whiteside: Well, and and Phil, if that's something I would ask you, coming over from a pure Vmware world, where now you do both? Have you been surprised at how many these features that are available on the Acropolis side of things that maybe weren't available when you looked at it a while back, if ever. 100 00:13:16.980 --> 00:13:20.249 Philip Sellers: Oh, yeah, yeah, I mean it. It's 101 00:13:20.760 --> 00:13:26.009 Philip Sellers: It's pretty incredible what's been done out on the platform, and 102 00:13:26.300 --> 00:13:30.919 Philip Sellers: enabled on a HP it. It's, you know it's 103 00:13:32.180 --> 00:13:36.300 Philip Sellers: it's very comparable coming from a DM. Or background. 104 00:13:37.690 --> 00:13:46.580 Jirah Cox: So it's. I mean. I usually say it's different, but it's it. It's it's different. There's not a check box for every check. Box. You used to see it on a platform, but it's got everything you need. 105 00:13:48.240 --> 00:13:51.849 Andy Whiteside: So, Jarra, i'm gonna walk through this and kind of let you 106 00:13:52.240 --> 00:13:57.940 Andy Whiteside: go through what the customer had set up and give me the insight, and then i'll let the guys just interrupt us and 107 00:13:58.200 --> 00:14:02.939 Andy Whiteside: comment as needed. You want to hit here where he's talking about what the workload is. 108 00:14:04.190 --> 00:14:11.870 Jirah Cox: Yeah. So drone highlights he's got 30 windows vms. They're running windows. 11 zoom guy with btpm enabled. 109 00:14:12.160 --> 00:14:14.720 Jirah Cox: And of course, you know those vms are spread across 110 00:14:14.770 --> 00:14:23.979 Jirah Cox: nodes in the cluster right? So so there's some vms running on every note in the cluster roughly. What would that be? 4 and a half or so? Vms per post, on average? 111 00:14:24.470 --> 00:14:30.839 Jirah Cox: so if nothing's Maxed out on resources, he highlights these at like 25%, CPU consumption, about 40% memory consumption. 112 00:14:32.470 --> 00:14:34.109 Jirah Cox: So then, 113 00:14:34.710 --> 00:14:37.569 Andy Whiteside: How many vdi it was. Oh, 30. 114 00:14:37.970 --> 00:14:53.140 Jirah Cox: No, Yup, they're yeah, 30. Yep, 30. Vm: so then he puts on his Chaos monkey hat and decides to go crash a node. So he logs into the hardware management out of band, you know. Tells it. Hey, Power off server immediately. No warning 115 00:14:53.150 --> 00:15:01.790 Jirah Cox: no notification to like the virtualization layer. The storage layer management layer of that right? So immediately it's like power pulling the power cord. One node goes off. 116 00:15:02.510 --> 00:15:19.359 Jirah Cox: so then, of course, you know no surprise. What you expect. Right? H. A kicks in vms get restarted on surviving nodes in the cluster right? So only the impact of vms, of course, have to do anything right. Other Vm: just keep on running. so the remaining nodes power on the 117 00:15:19.560 --> 00:15:34.199 Jirah Cox: the dashboard right prison. We give a lot of pixels right there on the on the dashboard to showing the administrator cluster, state, and what's going on right? What operations are we performing? What are we recovering from? Or is everything totally just situation normal. So immediately it shows it goes into a healing state. 118 00:15:34.210 --> 00:15:39.750 Jirah Cox: It it like alerts that hey Vms are getting migrated or restarting in order to get back to a highly available state. 119 00:15:40.210 --> 00:15:56.500 Jirah Cox: and then, of course, storage rebuild also occur right? So when you lose the hardware instance right? The hypervisor instance, you're gonna lose some slice of all your user Vm. Your your customer provision Vms: and then, of course, our virtual machine as well, right. Our Cvm. Our Controller Vm. 120 00:15:56.510 --> 00:16:00.250 Jirah Cox: So we run one of on our renewable cluster so that Cbm. Goes down as well. 121 00:16:00.810 --> 00:16:07.810 Jirah Cox: It was. It was hosting some portion of the data right roughly. In this case one-seventh of all the data stored in the cluster. 122 00:16:08.020 --> 00:16:16.989 Jirah Cox: And so the other 6 surviving Cbms are going to start that rebuild to pick up the select from that that failed node, and therefore the failed Cvm. As well. 123 00:16:17.820 --> 00:16:23.390 Andy Whiteside: So gyra it didn't say or did it, whether these were persistent or non- persistent. Vdi 124 00:16:24.670 --> 00:16:28.620 Jirah Cox: Good question it didn't say, and let me think if that matters 125 00:16:28.680 --> 00:16:30.880 Jirah Cox: it really 126 00:16:31.060 --> 00:16:39.629 Jirah Cox: doesn't right? Because if it's not persistent. You're gonna get back to that pristine image. That was whatever your you know, deep freeze state that gold master is. 127 00:16:39.860 --> 00:16:46.409 Jirah Cox: If it's persistent, then that's simply just a ha event power off clean through on another node 128 00:16:46.750 --> 00:17:02.180 Philip Sellers: just may take a little longer to grab all the bits from a non persistent, because there's likely more of those. So the healing process will maybe take a little longer. But possibly, but to to chase the tangent if you're using non persistent with us, probably doing 129 00:17:02.360 --> 00:17:05.090 Jirah Cox: well. If you're doing Pvs right, then whatever network 130 00:17:05.270 --> 00:17:16.910 Jirah Cox: is gonna network. If you're if you're doing Mcs right, then you're doing different thing differences in disks right against a goldmaster snapshot, and we actually under the covers. We'll do what we call shadow clone. 131 00:17:16.980 --> 00:17:24.189 Jirah Cox: where, whenever we detect that there's one V disk in the cluster. That's doing extra duty right? It's one V disk powering multiple vms. 132 00:17:24.250 --> 00:17:26.530 Jirah Cox: We actually will cache that the 133 00:17:26.609 --> 00:17:44.220 Jirah Cox: locally on every node in the cluster, so that all those reads go back to local flash. We'll actually intercept that read operation from the hypervisor down to the V disk, and we dart it locally, even if the authority of copy was elsewhere in the cluster. So we're shortening that read path You really won't feel that I don't think 134 00:17:44.840 --> 00:17:45.610 Philip Sellers: so. 135 00:17:45.900 --> 00:17:55.769 Andy Whiteside: So, Harvey. I'm not. I'm not right to say that if you were doing non persistence, and you were using Mcs or Dsa Pvs. This checkbox where you relate with the hardware reservation 136 00:17:55.830 --> 00:17:59.940 Andy Whiteside: this this probably doesn't happen because you got to pull the machines to cover this. 137 00:17:59.990 --> 00:18:11.940 Harvey Green: Yeah, you you don't have to have that extra reservation because you do have a pool, and it doesn't matter if you lose them if you just restart more of them. The the user might 138 00:18:12.230 --> 00:18:15.100 Harvey Green: lose their session. 139 00:18:15.480 --> 00:18:19.760 Harvey Green: for you know, for a minute. But then they're able to just restart another one. 140 00:18:20.040 --> 00:18:21.430 Jirah Cox: Yeah, there's 141 00:18:21.510 --> 00:18:29.729 Jirah Cox: you're saying a model where you're doing like one to many right? You're doing lots of users per per vm. Instance not a one to one. 142 00:18:30.110 --> 00:18:35.370 Andy Whiteside: It could be one to one of machines that are always running. So let's say he had 30. 143 00:18:35.390 --> 00:18:42.680 Andy Whiteside: And truth is, he had 22 users or maybe had 30 users. He would probably have 35 to 40 up and running so he would have probably lost. 144 00:18:42.820 --> 00:18:46.390 Andy Whiteside: You know, some some of the users would have come back in and they would have had more machines. 145 00:18:46.430 --> 00:18:57.820 Andy Whiteside: the the the the hypervis, not the hypervisor, but the control plane would have recognize. Hey, i'm. I'm supposed to have 10 machines running and waiting. Something happened. I'm down to 9 or 5, or whatever I need to turn a bunch on, or whatever still up. 146 00:18:58.250 --> 00:18:58.940 Jirah Cox: Hmm. 147 00:18:59.930 --> 00:19:06.110 Andy Whiteside: So chances are good. Consultant would never check this box unless it was persistent. But who knows? I don't know what the situation is. 148 00:19:08.080 --> 00:19:13.079 Andy Whiteside: anyway? all right. So the rebuild and then, Gyro, maybe i'm here 149 00:19:13.780 --> 00:19:19.240 Andy Whiteside: am I here where it says, data, resiliency, status or no. Yeah. So so the rebuild 150 00:19:19.360 --> 00:19:27.380 Jirah Cox: it continues. And it shows so immediately. It shows, you know, my my cluster, which is built to lose one of anything at once. 151 00:19:27.430 --> 00:19:46.580 Jirah Cox: whether it's a disk or a note, or whatever. So on the dashboard right? They're going to show fault. Tolerance is 0, right? We've got a little red flag next to it. you can click on it right, and even give you the detailed view of what's currently rebuilding. And you know the the Newtonics cluster right? And really what the Cdm is doing for you all day every day. 152 00:19:46.590 --> 00:20:00.499 Jirah Cox: It's not one giant monolithic application right? Like running Cdm. Or running aos. It's got a whole bunch of micro services within the cluster that all work together right? That i'll create the the the cluster, or the ring topology 153 00:20:00.620 --> 00:20:13.369 Jirah Cox: each one can like self-elect the different leader, leaders and follower states so it'll show you what part in this screenshot. Here it's showing the Cassandra ring itself. The metadata partition right? Which is where we store our data about customer data 154 00:20:13.380 --> 00:20:26.859 Andy Whiteside: in the cluster. That's the layer that's rebuilding in this case, to back to resiliency, back to health. To where it can lose another one. Anything right? A member in that ring. So let's get back to that. Rf: that Rf: 2 stage where it's statutory covered. 155 00:20:27.160 --> 00:20:37.099 Jirah Cox: You can think of it as getting back to Rf. To across a number of measures right? Rf: 2 for user data, Rf: 2 for detects data about the cluster itself 156 00:20:38.100 --> 00:20:39.010 Andy Whiteside: and 157 00:20:39.260 --> 00:20:42.299 Andy Whiteside: Terra. What happens if it does? If there's not enough available 158 00:20:42.650 --> 00:20:44.190 Andy Whiteside: space. 159 00:20:44.220 --> 00:20:46.469 Andy Whiteside: or I guess they would have told you that before this happened 160 00:20:47.320 --> 00:20:56.609 Jirah Cox: so I mean, that's part of the planning right, is we never want any customer to be in a place where they couldn't lose their defined availability 161 00:20:56.820 --> 00:21:01.940 Jirah Cox: threshold, right like in our 2 cluster. Right you can one anything 162 00:21:02.050 --> 00:21:12.199 Jirah Cox: which includes a node? We don't want you to be within a nodes worth of filling up the cluster right if you do. And then, yeah, totally, the question is going to fill up You're gonna have a bad day. 163 00:21:12.210 --> 00:21:24.749 Jirah Cox: what it really does when you, if you really just fill it up in that you run a space, it's going to go into a read-only state to protect itself and product your data to say i'm gonna not disallow any new rights because we're just plum full, 164 00:21:24.940 --> 00:21:29.099 Jirah Cox: you know. Call support. We need to get this fixed or call, you know. 165 00:21:29.350 --> 00:21:35.680 Jirah Cox: Call ban call Harvey right call, Phil. We need to get you more space in here right so you know bigger disks other node, whatever it takes to 166 00:21:35.960 --> 00:21:44.599 Jirah Cox: to establish health there in the cluster. Often it can be like delete some old snapshots if you want it to. But yeah, if you run out of space forever, it's just gonna it's gonna go into a read-only state 167 00:21:45.690 --> 00:21:51.619 Andy Whiteside: Let me pause here, Ben Harvey, Philip, any questions, comments, takeaways. 168 00:21:53.790 --> 00:21:57.559 Harvey Green: no, I mean, I I think, that this is 169 00:21:58.040 --> 00:22:06.739 Harvey Green: definitely describing where you'll be, and you just call Phil, and he'll show up with all kinds of drives in his back pocket. He can just switch them out for you. 170 00:22:06.790 --> 00:22:12.700 Jirah Cox: You just you just you just run down to the best buy, and you know, suck some drives. 171 00:22:16.000 --> 00:22:33.739 Ben Rogers: Well, I mean one of the things we have to point out is before you got even into that state. The cluster would be screen the bloody murder. It would be going. Hey? I can't. I can't run our F 2. I'm not able to get the I'm not able to get to this compliance state need to have more memory. We have to capacity planning in the cluster. So 172 00:22:33.750 --> 00:22:47.499 Ben Rogers: even though I know we're kind of in the lab, and we're doing this. We won't see the results. I don't want any of our customers think that. Oh, there's no warning that this is coming down these boxes that these things get where they're in a UN unhealthy state. 173 00:22:47.870 --> 00:22:56.920 Jirah Cox: Totally right. But yeah, the cluster itself. We've we've taught it lots of tricks over the years. It'll email you. It can send out Snmp alerts. It can open up service now. Tickets, if you allow that. 174 00:22:57.000 --> 00:23:04.819 Jirah Cox: lots of lots of fun tricks. It's got to alert you, hey? We are entering a you know. Non resilient threshold right of the cluster state here. 175 00:23:04.950 --> 00:23:15.170 Jirah Cox: if you like, if you allow the phone home telemetry right? We call it pulse, right? So we can phone home with how healthy the cluster you're running it can ping your account team right? They'll reach out 176 00:23:15.300 --> 00:23:26.230 Jirah Cox: it. Can. there's a is a new trick that clusters learned last year actually where you can say like in this case, let's say you've got a 7 node cluster. So therefore you shouldn't write more than 6 nodes worth of data. 177 00:23:26.630 --> 00:23:30.530 Jirah Cox: You can basically tell it only show me 6 tones worth of data. 178 00:23:30.590 --> 00:23:43.749 Jirah Cox: and don't even pretend that the seventh node exists right. That is, a 100% is the 6 node line. so you can have it. Redraw all the graphs to say this is what full looks like. This is what if we're healthy, is somewhere to the left of that 179 00:23:43.780 --> 00:23:48.260 Jirah Cox: Don't don't make me track. Am I over or under my 6 node threshold 180 00:23:48.860 --> 00:23:49.600 right 181 00:23:50.500 --> 00:24:01.430 Andy Whiteside: so jar with this next sections is after 30 min. Cbm. Not reachable for 30 min. The node is being detached. In other words, it says, hey, i'm going to completely remove this guy from my stored. 182 00:24:01.870 --> 00:24:04.790 Jirah Cox: Yep. So this is what I really like a lot. So we've already 183 00:24:04.910 --> 00:24:09.249 Jirah Cox: begun the process. We've already at this point probably get close to completing the process of 184 00:24:09.270 --> 00:24:11.769 Jirah Cox: re-healing from the failure State 185 00:24:11.890 --> 00:24:16.290 Jirah Cox: what I love about mutants and the cluster design in general is it's self healing 186 00:24:16.320 --> 00:24:24.890 Jirah Cox: So we don't stay in a broken 7 node state for very long we transition to a healthy 6 node state. 6 nodes becomes the new normal. 187 00:24:24.960 --> 00:24:36.420 Jirah Cox: That's all the nodes we have in the cluster, and all the nodes we know about right. So we don't we don't stay in sort of like a degraded state. We eject the seventh note that's failed from the metadata ring right from our knowledge of what nodes exist in the cluster. 188 00:24:36.510 --> 00:24:43.130 Jirah Cox: for a lot of reasons, right one like we forget about it. There's some things we can clean up we're not waiting for him to come back online. 189 00:24:43.760 --> 00:24:46.550 Jirah Cox: another one, right? Let's say that note is down for a week 190 00:24:46.720 --> 00:24:48.210 Jirah Cox: in a week. When it comes up 191 00:24:48.330 --> 00:24:50.090 Jirah Cox: it has almost no useful data. 192 00:24:50.190 --> 00:25:04.420 Jirah Cox: so we don't want to treat it like it's You know a a prodigal node returns home right? We'll just read it like it's a new node and the rethink data over to it rather than by differences like. Oh, what's new? What's what's changed? What's not changed. 193 00:25:04.430 --> 00:25:16.889 Jirah Cox: We'll just cut bait, and then, if it comes back great, we will. We will accept it back in the cluster as a fresh node, taking that node 7 spot versus having to worry about differences. What data has changed, or what hasn't 194 00:25:17.620 --> 00:25:21.220 Andy Whiteside: we always we always heal, we always heal down to a healthy state. We don't 195 00:25:21.250 --> 00:25:23.860 Jirah Cox: stay in a degraded state whenever we can help it. 196 00:25:24.050 --> 00:25:27.409 Andy Whiteside: Yeah, that that allows you to sleep at night, not guessing what 197 00:25:27.630 --> 00:25:46.130 Ben Rogers: could be happening when not there to watch it. Guys any comments on that. Well, for for me personally, this would definitely give me peace of mind, because I've left the office several times. Where one drive that you know you had a R. 5 and one drive dropped out while you're getting the drive. Shit back to you 198 00:25:46.140 --> 00:25:54.690 Ben Rogers: those couple of hours. You're just praying to the it. Guides that don't let anything go wrong. I mean so 199 00:25:54.700 --> 00:26:19.929 Ben Rogers: for me to know that my technology would self heal itself and assume that. Oh, the units bad. We're gonna go ahead and get it out. The mix. We're gonna continue running a healthy state, and you want to bring that unit back in great. But we're going to treat that as a new unit. That's awesome, man. It definitely gives a good level of comfort that you don't have to, you know. Sit on pins and needles. Why, things are being shipped to you, or procured, or any of those things that we're all used to dealing with. 200 00:26:20.820 --> 00:26:21.570 No. 201 00:26:21.940 --> 00:26:30.079 Andy Whiteside: So, Jerome, take a step further, I believe, and now he's ready to at least know that he can take down another node and still be up and going. 202 00:26:30.410 --> 00:26:47.650 Jirah Cox: Yep. So the process here actually remains the same. So you could actually keep on crashing a node right as long as you're gonna. You can crash one at a time. As long as you are allowing for that re heal to complete, and between each node failure, so you could go from 7 to 6, 6 to 5, 5 to 4 203 00:26:48.660 --> 00:27:02.540 Jirah Cox: for down to 3, and he actually talks about, you know what if you keep on going until you only have 2 nodes left, 2 nodes right at that point. we do under the covers right? We have a mandate to ring. That is, 3 nodes at a minimum size for Rf: 2. 204 00:27:02.550 --> 00:27:21.860 Jirah Cox: That's about as that's about as failed as you can get is you could have a 3 node cluster that. Then shucks! One more node, and is down to running on 2 nodes out of 3, 1, 2 legs out of 3 on the stool. Are there? that one won't re heal because we you can't shrink down to a 2 node, healthy cluster from a larger, like 7 node cluster 205 00:27:21.870 --> 00:27:32.430 Jirah Cox: 3. No. Is the minimum there. So once you hit that free node threshold and you lose one more node you can run. You can definitely hit Rf: 2 for customer data, right? All your data written twice. In that case it's 206 00:27:32.450 --> 00:27:37.790 Jirah Cox: fully mirrored right. Everything's on node one and node 2, and Node 3 is down out of. Think 207 00:27:38.380 --> 00:27:58.159 Jirah Cox: at that point. If you lost one more node well, then, totally. You have no cluster left, right you down to one node out of 7. That's a a non survival situation for the data. Your data is safe, but it's not going to run, not going to be operable, not going to be an online state for the cluster. and at that point you do have to go, get you some spare parts and bring at least one more note the tech online from your 208 00:27:58.170 --> 00:27:59.770 Jirah Cox: one node out of 7 State 209 00:28:00.660 --> 00:28:03.659 Andy Whiteside: And tyra. Was this all doable 210 00:28:03.880 --> 00:28:06.599 Andy Whiteside: because of Yes, the magic of Newtonics, plus 211 00:28:06.810 --> 00:28:10.190 Andy Whiteside: the fact that he was running at such a low capacity to begin with. 212 00:28:10.820 --> 00:28:16.049 Jirah Cox: Totally so that's that's the real limit that most customers will hit first right unless you are only using what 213 00:28:16.260 --> 00:28:19.869 Jirah Cox: simple math would tell us. One seventh of your storage capacity. 214 00:28:20.010 --> 00:28:29.290 Jirah Cox: You'll hit that first of let's say you're You're using 3 nodes worth of storage. Then, as soon as you have to rebuild onto fewer than 3 nodes, the data doesn't fit. 215 00:28:29.400 --> 00:28:31.890 Jirah Cox: We're gonna call that cluster full. 216 00:28:32.190 --> 00:28:38.879 Jirah Cox: it's going to go into a real read-only state. And at that point you've got to you know, Certainly. Lay hands on the hardware that's failed and bring that back online. 217 00:28:39.020 --> 00:28:39.720 Yeah. 218 00:28:39.760 --> 00:28:40.470 okay. 219 00:28:41.200 --> 00:28:59.130 Jirah Cox: But yeah, you can. You can fail nose down to the I call it like the water level of the cluster. Right? If the cluster is a certain bucket, you can lose the top of the bucket and another slice of the bucket. You can keep on losing that until, however full it is and once you get that threshold, then then that's gonna fill it up and we'll go read only on that date. 220 00:29:00.370 --> 00:29:18.619 Andy Whiteside: and then the bringing this back online. It's just a matter of adding a node, adding a node, adding a node as they become healthy, and you want to reintroduce them. They come in kind of come in as a foreign object. It looks like, do they? Totally. Yup: yeah. So there's there's one click on a prison there to say, yeah, like, Admit this node back in the cluster. 221 00:29:18.630 --> 00:29:29.290 Jirah Cox: If you're unsure about it right like we. We sort of have a bit of a as a software layer. We have a little bit of a I think, a a wise and somewhat healthy mistrust of hardware health. 222 00:29:29.390 --> 00:29:36.329 Jirah Cox: So if that note's been flapping, it's been up and down it's, it's caused up enough enough enough heartburn that we ejected it from the ring. 223 00:29:36.730 --> 00:29:55.410 Jirah Cox: We're gonna make you tell us. Trust that note again rather than do that sort of fully, proactively, right? Maybe you're testing out some, you know. Flaky didn't right, or got some weird power going on in one in the cabinet in the data center. we'll let you tell us when that storm is passed, and then we'll we'll. We'll admit that node back into the ring. 224 00:29:57.090 --> 00:30:01.619 Jirah Cox: So there's some things that we think of. Some things we can't programmatically determine as software only 225 00:30:02.160 --> 00:30:11.160 Andy Whiteside: mit ctl. And so to a large degree, it's aware of that node and aware that it might come online. But it's completely mitigated it for now until you tell it, hey? I'm ready for you to reconsider 150 226 00:30:11.940 --> 00:30:13.299 Andy Whiteside: bringing this guy back in. 227 00:30:13.940 --> 00:30:15.419 Jirah Cox: Yeah, that's that's fair. 228 00:30:15.830 --> 00:30:16.500 Okay. 229 00:30:17.320 --> 00:30:19.360 Andy Whiteside: Philip, I'll go to you first. 230 00:30:19.590 --> 00:30:23.740 Andy Whiteside: Any additional questions, comments, thoughts, things you like to add. 231 00:30:23.810 --> 00:30:29.270 Philip Sellers: Yeah, I wanted to ask a little bit about exposure time. So you know, we've got this 30 min 232 00:30:29.400 --> 00:30:36.170 Philip Sellers: time out with the Cvm. Where it gets ejected out of the metadata ring. So 233 00:30:36.690 --> 00:30:44.140 Philip Sellers: you. You're really kind of sitting in an exposure, time, or or exposed state for that 30 min or 234 00:30:44.380 --> 00:30:49.550 Philip Sellers: 30 min, plus, however long it takes to continue to the rebuild. Right? 235 00:30:49.630 --> 00:30:52.400 Jirah Cox: It's a fantastic question. Actually believe it or not. You don't 236 00:30:52.480 --> 00:30:57.360 Jirah Cox: so let's as soon as you fail the node at the very next second 237 00:30:57.470 --> 00:31:00.679 Jirah Cox: we're actually starting the customer data rebuild immediately. 238 00:31:00.920 --> 00:31:08.479 Jirah Cox: and every new right as well. So if the Vm. Generates a new data onto like, let's say, at the beginning of the 6 surviving nodes in the cluster. 239 00:31:08.550 --> 00:31:16.470 Jirah Cox: We're immediately honoring that right onto 2 different nodes. Right? Maybe it was initially going to be targeted to node one and node 7 for the 2 replica copies. 240 00:31:16.510 --> 00:31:28.649 Jirah Cox: But now it'll be node, one and node 6 or node, one and node 5, or whatever it is, so we'd never accept to write as a as the platform. We never. It's at the right from the Vm. That we can't honor according to the replica factor, right? Rf: 2 or Rf. 3. 241 00:31:29.080 --> 00:31:39.450 Jirah Cox: So every data is immediately corrected, predicted that way for new data, and we immediately get the rebuild of customer data as well again immediately. So actually what it is it's like. Let's just say hypothetical for this credit scenario. 242 00:31:39.490 --> 00:31:50.460 Jirah Cox: New rights are immediately protected according to the replica factor, and let's say the rebuild finishes in 15 min. The other 15 min is simply our confidence in the note itself 243 00:31:50.480 --> 00:31:58.840 Jirah Cox: before we eject it from the minute or ring. but that's not really user-facing it Doesn't expose, risk risk or or cause exposure to your point, fellow. Great question. Fantastic question. 244 00:31:58.870 --> 00:32:17.419 Philip Sellers: And that makes a ton of sense, too, because as long as the rebuild is in place, then all your bits are protected. Yeah. Yeah. So. So there's parts of this that are totally inside baseball under the covers under the hood. I should say that we're being super transparent about what happens, and this is all in the Mechanics Bible as well. Right? 245 00:32:17.430 --> 00:32:30.989 Jirah Cox: in terms of like what? What layers of data power the system and and contribute to availability. but yeah, it's always been an operating. Thesis is, you know, we have to be absolutely paranoid about about 246 00:32:31.000 --> 00:32:43.470 Jirah Cox: customer data integrity, right? Otherwise we're just useless as a platform, right? No one should trust us. So that's always been Job. One is be a good steward of customer data, and that includes immediate rebuild with no delay timers until we start the rebuild. We never. 247 00:32:43.520 --> 00:33:00.409 Jirah Cox: We almost never Our view of the world is we never assume the hardware is gonna come back right, so we'll never delay a rebuild, hoping it will, trying to make our life easier. We'll take the harder path of sort of the rebuild now. And worst case, if that no does come back because it was just a transient power failure or a reboot. 248 00:33:00.480 --> 00:33:11.259 Jirah Cox: Well, then, worst case we've over, predicted some data, and we can do that garbage cleanup, right? But we'll never. We'll never let the customer run exposed hoping and praying that hardware comes back when it might not. 249 00:33:12.140 --> 00:33:17.899 Philip Sellers: Yeah, the other thing. And just to reiterate what you said. you know you you've got 250 00:33:18.370 --> 00:33:33.199 Philip Sellers: this concept of the local cache in every node, and with Vdi it's one of the things that really makes it a great platform for running. Vdi is having that local copy on every single one of your nodes. So especially the non persistent desktops. 251 00:33:33.370 --> 00:33:35.280 Philip Sellers: They're right there. 252 00:33:35.510 --> 00:33:44.679 Philip Sellers: you know it's it's it's never going to be faster than that. Right. It's hard. He says you're you're never going to get food faster than in your own kitchen. So 253 00:33:45.110 --> 00:33:49.220 Philip Sellers: it it's it's one of those great platform trade-offs, and and 254 00:33:49.310 --> 00:34:03.790 Philip Sellers: you said it earlier. But you know I just wanted to reiterate, re re-emphasize it because it is one of the great use, cases, and why we we like to do. Vdi on new tanks is architecturally. There's things there that help us. 255 00:34:05.950 --> 00:34:18.300 Andy Whiteside: Yeah, there's there's there that we've wanted forever. Just took new tanks showing up to make it to where that was handled at a sub under the surface layer, so we could get on with brokering connections and 256 00:34:18.750 --> 00:34:20.830 Andy Whiteside: enabling user? Experience. 257 00:34:22.960 --> 00:34:25.689 Ben Rogers: You know what the best 258 00:34:26.790 --> 00:34:39.430 Ben Rogers: best thing about. Hv: Well, one of it's the cost under the inclusion of it. But no, what are you highlighting. 259 00:34:39.550 --> 00:34:44.779 Ben Rogers: So you know we're customers, you know. Let's forget all about the prolong and all that 260 00:34:44.810 --> 00:35:00.130 Ben Rogers: we're going into. You know a little bit of a recession. Time is going to get a lot of thought a little tight. A lot of customers are looking. This is a way. Could I get my budget a little skinnier and utilize something that I already own versus trying to reinvent the will with a secondary product. 261 00:35:02.120 --> 00:35:12.999 Andy Whiteside: Yeah, I I totally get it. I totally agree, I mean in the beginning that was there was an argument that was true, but not a no brainer. But as Philip pointed a while ago, this this thing that you've 262 00:35:13.190 --> 00:35:24.020 Andy Whiteside: you guys in new tennis have created that is, this cost-effective platform that now has all these additional services that's been bolted onto it. it really expanded that story. It's not free 263 00:35:24.100 --> 00:35:29.539 Ben Rogers: now. Nobody gets it for free, but it is included in your Als license 264 00:35:30.130 --> 00:35:30.770 right? 265 00:35:31.290 --> 00:35:34.720 Jirah Cox: Totally. Yeah. Plenty of opportunity there to simplify. I mean 266 00:35:34.760 --> 00:35:35.809 Jirah Cox: I mean 267 00:35:36.140 --> 00:35:47.330 Jirah Cox: zooming out for a second. I mean Kvm is one of the most widely deployed hypervisors in the world right? The real trick we taught it here. in addition to speaking, you know, Cvm. Is for high speed. Local storage 268 00:35:47.370 --> 00:35:59.409 Jirah Cox: is is manageability right like, you know, brought it into into the family. Right? You manage it with prism. There's really not if you already know how to run new tanks and managing a. V is kind of a non issue, right? You run like any other cluster. 269 00:35:59.490 --> 00:36:02.620 Jirah Cox: so simplicity simplicity included. 270 00:36:03.730 --> 00:36:10.370 Philip Sellers: Yeah, it it's funny, Andy, You asked about the enabling Ha reservation checkbox. You know 271 00:36:10.560 --> 00:36:30.449 Philip Sellers: there's a lot here as I've explored the new tax platform. That reminds me of the early days of Vmware, where they make clustering easy. You just went in and checked the box, and that you had a cluster. And I remember doing windows. Cluster builds that took 2 weeks to to set up and configure and get all the bugs worked out prior to that. 272 00:36:30.480 --> 00:36:46.710 Philip Sellers: there's a lot of that that exists in the platform, too. simplicity. They they're They're not. Tennis is doing a good job of delivering something complex under the covers in a simplistic way, operating it in a simplistic way. 273 00:36:46.730 --> 00:36:53.059 Philip Sellers: those are things I think, that resonate with technologists when they get to to see 274 00:36:53.210 --> 00:36:56.059 Philip Sellers: what's being delivered to them. 275 00:36:56.190 --> 00:36:59.699 Philip Sellers: yeah, I I've looked at other hypervisors, one from a 276 00:37:00.640 --> 00:37:10.499 Philip Sellers: a large software company that might write operating systems, and you know it's clunky. It. It takes some 277 00:37:10.680 --> 00:37:19.239 Philip Sellers: extra steps. You know it's it's that old school clustering that you know from windows. It's. You know it's the same on hyperv, and 278 00:37:19.390 --> 00:37:26.160 Philip Sellers: there there's something to be said about the simplistic message, and and delivery that mechanics is doing here. 279 00:37:26.860 --> 00:37:37.849 Andy Whiteside: I think one of the that's Newton's has. Is it was born with the same hyper converge processes and concepts, and constructs, and from day one 280 00:37:38.240 --> 00:37:43.420 Andy Whiteside: that wasn't something that had to, you know, find its way into the solution. It was born that way. 281 00:37:43.530 --> 00:37:50.110 Andy Whiteside: and you know sometimes it's just helpful to be bored at a time when the future is there versus having to adapt to it. 282 00:37:50.860 --> 00:37:53.120 Jirah Cox: Yeah, it's pretty fair. I mean the 283 00:37:53.360 --> 00:37:57.720 Jirah Cox: the point I like to make it. It was taught to me years ago. It's like 284 00:37:57.920 --> 00:38:16.470 Jirah Cox: any system of sufficient ability to run like and solve business problems like this complexity in that right? It's just a matter of it's a design question of do we ask users to bear that complexity? Or do we buy that complexity? So they get some of that experience right there's there's complexity that is abstracting going on under the hood 285 00:38:16.540 --> 00:38:32.449 Jirah Cox: I I heard one time that you know we make. We evaluate any given Vm. Right on like 7 or 12 different metrics to determine where it's going to land right. So there's there's plenty of decisions being made under the hood, but we just don't ask users to wait into the thick of that, and make those for us right if we can. If we can abstract and we will. 286 00:38:32.870 --> 00:38:33.430 Right. 287 00:38:35.370 --> 00:38:40.269 Andy Whiteside: Well, guys, we're more or less out of time. This has been a good conversation been 288 00:38:40.520 --> 00:38:41.899 Andy Whiteside: any additional 289 00:38:41.940 --> 00:38:43.319 Andy Whiteside: thoughts, comments? 290 00:38:43.580 --> 00:38:53.000 Ben Rogers: No, I I you know I kind of go back to what they said at the beginning of the podcast man. This is a platform. It it's not just a hyper converge system anymore. 291 00:38:53.010 --> 00:39:10.749 Ben Rogers: and all the other products that we have may kind of hinge off for this idea of, you know, spreading data across the cluster day, the resiliency, all these things. So you guys want to learn more. Reach out to us. It's exciting time to be at Newton. We're taking this technology to cloud. Now. 292 00:39:10.810 --> 00:39:20.549 Ben Rogers: that's opening up some doors for so really good time to be employed for new tennis. Really good time to be running new technics, and and I look forward to what they what the future brings for us. 293 00:39:22.180 --> 00:39:24.799 Andy Whiteside: Harvey. What what did we miss? You want to cover? 294 00:39:25.070 --> 00:39:26.959 Harvey Green: well, since you 295 00:39:27.030 --> 00:39:40.090 Harvey Green: time stand and date stamp the episode at the start, i'll just remind everybody that this this week is a on a third Friday week. So we're doing our workshop on Friday from 296 00:39:40.270 --> 00:39:46.130 Harvey Green: 11 to 2 this week is mechanics database service. So 297 00:39:46.200 --> 00:39:48.009 Harvey Green: definitely jump on. 298 00:39:48.960 --> 00:39:50.640 Andy Whiteside: Formerly no one in America 299 00:39:50.820 --> 00:39:52.730 Harvey Green: formerly known as era. 300 00:39:52.950 --> 00:39:56.609 Jirah Cox: and that's that's that's a phone workshop, right? That's like 301 00:39:56.750 --> 00:40:01.490 Jirah Cox: that that goes so far beyond this kind of like Vm. Management data rebuilding. It's like 302 00:40:01.650 --> 00:40:08.479 Jirah Cox: batteries included, right like. What can the platform really do for you and help streamline a lot of day. 2 operations. 303 00:40:08.540 --> 00:40:27.529 Harvey Green: Yeah. I I will laugh at one of my friends because he always tells me I I come back with the phrase, what else can you do every time he he bring something up? this? This is one of those occasions where we just gone through. You know this entire podcast on some of the things that are underpinning the entire platform. 304 00:40:27.630 --> 00:40:33.300 Harvey Green: and then it's like, Well, what else can you do? Well, on Friday you'll see more 305 00:40:34.190 --> 00:40:36.209 where that came from. 306 00:40:38.080 --> 00:40:42.290 Philip Sellers: no, I'll be on the same workshop with 307 00:40:42.590 --> 00:40:50.670 Harvey Green: with Harvey on Friday. We'd love to see you, you know Bill's gonna do all the work this time. He's gonna sit there and relax. 308 00:40:51.620 --> 00:40:55.640 Andy Whiteside: I have no doubt it will go just fine gyra. One last chance anything. 309 00:41:01.640 --> 00:41:21.150 Jirah Cox: I shouldn't be blanking on that. Oh, well, we announced that next come in Chicago right definitely. Talk to take out the site for that talk to your account teams. Tell me you want to get some passes. and we'd love to see it in Chicago as Dot next leaves the virtual world, and goes back to the physical or something like that. The the analogy breaks down 310 00:41:21.470 --> 00:41:33.129 Andy Whiteside: well. And if you're looking for passes, if you renew with integrity, your existing licenses with their passes in. We're going to be given away, I think 20 passes passes. We I think we plan to give. 311 00:41:33.690 --> 00:41:42.400 Jirah Cox: We plan to give away both the the broncos that we're giving away as far as work has no boundaries. at the event. That's right. There you go, man. I want to be a centigrade customer. 312 00:41:42.510 --> 00:41:55.430 Harvey Green: Hey? You should be. You should be. My fear is, you wouldn't need us. But that's okay. All right, guys, we'll appreciate you guys joining and doing this with us, and we'll do it again in a week or 2. 313 00:41:56.560 --> 00:41:59.020 Jirah Cox: Sounds good, All right, thanks, Everybody.