Latest / Oxide and Friends / Grown-up ZFS Data Corruption Bug

Transcript

Grown-up ZFS Data Corruption Bug · Oxide and Friends

• 0:00
  Bryan Cantrill: Alright. Well, this is and, you know, Adam, I was I I like your title for this. Like, because I was like, is. No. Because I wonder, like, is it a ZFS data corruption bug old enough to vote?
• 0:11
  I mean, that is that's very, you know, that's Very wordy. Very, very wordy. Then And you got like, you know, you kinda wanna play into adulthood because it is 18 years old, but I just love it. You know, it's all grown up. Our little our little baby data corruption bug is all grown up and it's gone into the world.
• 0:29
  Adam Leventhal: Yeah. And definitely no question about whether we try it as an adult.
• 0:33
  Bryan Cantrill: No. No question whether we try this as adult. I mean, like, look, we probably would have I mean, if this had been a 16 year old CFS data data corruption, we probably would tried it as an adult. Are you
• 0:41
  Adam Leventhal: right? Exactly.
• 0:41
  Bryan Cantrill: But yes. Yeah.
• 0:42
  For sure. But but now we don't even have to have that conversation. Now it's like this Precisely. This bug is is going away for the rest of its life. So but that's you know what?
• 0:51
  Let's not go too far with that analogy. As we as as many analogies, I wish I had stopped several several elaborations ago.
• 0:58
  Adam Leventhal: Sounds right.
• 0:59
  Bryan Cantrill: Okay. So this because this is the the great thing about this bug is it includes a bonus data corruption bug that we debugged in trying to debug this one. So Alan is here. Alan, this one starts tell us how this one starts. Because I think this one starts with you noticing, that we we had our, our pod just bring the chime early.
• 1:27
  We had our podcast episode on crucible, which may have been as recently as a month ago, as long ago as five years ago. I really don't know.
• 1:34
  Alan Hanson: Somewhere in that range.
• 1:35
  Bryan Cantrill: Somewhere in that range.
• 1:37
  Adam Leventhal: Well well, I I would just note on that crucible one. I think we, you know, I think Alan and James were a little bit reluctant to
• 1:42
  Bryan Cantrill: come on the show. And I think it
• 1:45
  Adam Leventhal: was like, look. Don't don't worry. Nobody's gonna listen to it anyway. It's fine. And it was, for a long time, our most listened to show.
• 1:51
  So sorry that we misled you on that one.
• 1:53
  Alan Hanson: Even if you But,
• 1:54
  Adam Leventhal: yeah, it was a minute ago.
• 1:55
  Alan Hanson: Listens out. If you
• 1:56
  Bryan Cantrill: If you take your mom your mom did, like, relisten to it quite a bit. She was clearly looking for something. So if we
• 2:02
  Alan Hanson: take finding it.
• 2:03
  Bryan Cantrill: Yeah. Yes. Just not finding it. Exactly. Yeah.
• 2:07
  No. It was a it was a it was a popular one. Well, no. I think the the reason to not wanna talk about it is the reason you never wanna talk about a system that's responsible for data data correctness. Integrity.
• 2:19
  Data integrity because you don't wanna taunt the gods. You don't want the you the listener that you are worried about in that podcast is gods. The gods tuning in and being like, oh, that's a pretty fancy system you got there. It it sounds like you're pretty proud of it. That sounds like it always gets the data correct.
• 2:36
  Well, maybe it doesn't, pal. Yeah. So that that is the reason that I think we're all very superstitious about talking about this stuff. And hey, we shot our mouth off and here we are. So Mhmm.
• 2:48
  Alan, I'm sorry. Clearly, data corruption bug is is is a direct responsibility of us convincing you to be on the podcast. So I'm really sorry for that. But so how does this thing start? Because you noticed that we've got some data corruption on crucible volumes in our dog food rack.
• 3:05
  Is that correct?
• 3:05
  Alan Hanson: Yeah. Think well, I think Angela might have actually flagged flagged the issue first and then I went and looked and said Yeah, we are failing to decrypt a block which was very puzzling to all of us because we're like that how, what, a lot of confusion. And yeah, I mean you can sort of read the beginning of the issue but certainly there's a lot of false there's a lot of confusion going on.
• 3:40
  Bryan Cantrill: What are the kind of the initial so we always encrypt by default. Yeah. So a failure to decrypt is alarming because that that can indicate well, it can indicate data corruption.
• 3:55
  Alan Hanson: Yeah. And we hadn't seen like this particular panic trip in, I don't know, three and a half years or something. Since we were still developing the code. So it very confusing and like how could we get here, what's going on. I think it took a day or so before we really analyzed the problem and figured out that we had this on this one extent, we had 32 k that was all zeros.
• 4:30
  Bryan Cantrill: I I do like this this issue, which we've linked, is itself literature. And I do love the the line casting around in desperation. I find some detritus, which feels like, I mean, I I the the that just feels like a a one line poem of for for debugging data corruption. So yeah. So we what are you able to what what do you find in the detritus?
• 4:55
  Alan Hanson: I'd have to I need to bring up the issue too so I can reference my actual statement.
• 5:03
  Bryan Cantrill: I mean,
• 5:03
  Adam Leventhal: it's And while you do that, Alan, I would just quick aside, and I hate to be the senator from context,
• 5:08
  but
• 5:08
  Bryan Cantrill: I would like to advise you that you're under oath. Is that where you're going with this?
• 5:11
  Adam Leventhal: That's right. No. I was gonna say, we are talking about the oxide storage service
• 5:17
  Bryan Cantrill: Yes.
• 5:17
  Adam Leventhal: Which provides virtual disks to virtual machines. We talked about it a bunch on that episode, but just, you know, for teeny bit of context, it is has ZFSs, some underpinnings. We store data in triplicate on three different distinct servers and then provide a a redundant, you know, virtual disk to these virtual machines.
• 5:41
  Bryan Cantrill: That's right. And we had actually so, I mean, Alan, you had said we had not seen this since we were developing Crucible. Yeah. But in fact, a very important change had been made to the the surrounding system. In that, and queue episode, much more recent episode, we had, Dave had done and team had done this terrific work on update, and we were now updating a system while it was while it was more in motion.
• 6:12
  We were not parking the rack any longer to update the system which meant there was gonna be more rebooting of sleds while there was effectively activity.
• 6:21
  Alan Hanson: Yeah. I think this was this is definitely a key part of what the eventual issue is that we hadn't we had done a lot of testing of restarting processes and verifying things that way but we hadn't done a lot of like power off the system, power back on testing. And I think that was sort of a new a new area for us.
• 6:44
  Bryan Cantrill: But that is probably not top of mind when you're seeing data corruption. Although maybe or maybe it was. I don't know. And the the as you were casting about in desperation, finding detritus.
• 6:54
  Alan Hanson: Well, I I think, yeah, in the beginning, there's a lot of there's a lot of desperation. There's a lot of casting around. Part of the the issue is is that we're we're doing a lot of things here and it's not clear which one is the one that makes a difference.
• 7:08
  Bryan Cantrill: Right.
• 7:08
  Alan Hanson: So we're like, here's some things that there's lots of things that changed. One of them was the update. We pretty sure it wasn't update related directly but still not sure.
• 7:20
  Bryan Cantrill: Right.
• 7:20
  Alan Hanson: There's always a lack of confidence in the code you wrote that's the first place you go to because it's gotta be
• 7:27
  Bryan Cantrill: It's gotta be me.
• 7:28
  Alan Hanson: It's gotta be me. Right. So you always you know suspect your own stuff. And then there were were some we weren't sure if the live repair was part of the prop. And I think one of our challenges here was that we didn't we didn't know that there was data corruption until we tried to actually read the block.
• 7:50
  Okay. So something happened at some point in the past and now down the road we try to read the block and then there's a problem. So initially we were like well something has happened at some point and we've we've got this corruption, but we don't know what the timeline is. We don't know the start date yet.
• 8:12
  Adam Leventhal: And which is the insidious factor of data corruption? Data corruption both at rest and even in memory because the you're separated at such a distance from discovery from when the thing occurred. And obviously, like, data at rest corruption is is much worse. But in all cases, you're like, all I know is I found it here. There's, you know, the way where it was introduced, who knows?
• 8:38
  Bryan Cantrill: Yeah. Yeah. I mean, and it was introduced sometime in the distant past and you often have at first little data other than the corruption itself. And the corruption itself can be important. And what what are we able to determine about the the kind of the contour of the corruption, if anything?
• 8:57
  Alan Hanson: Well, we we know that it's 32 k of zeros, and it's right at the start of the file. Okay. And so far, it took us a little bit to find that initially. And then once we had that sort of data point, we were able to sort of build a verify tool. Matt, you know, came up with a plan and so we could go and then verify and we went and verified every single downstairs that we had on dog food to look to see if this was anywhere else.
• 9:30
  Bryan Cantrill: Okay. And what did we find?
• 9:31
  Alan Hanson: So on the the the first time we looked, we only found the one failure case.
• 9:38
  Bryan Cantrill: Okay. And so I mean, whether that's of is that a relief or not? That's that's kind of like, I mean, is what it is.
• 9:44
  Alan Hanson: It is. Yeah. I mean, well, I it it's good because you don't wanna find it. But then if if there's a problem, you wanna find it and you wanna find it as I don't know.
• 9:56
  Adam Leventhal: Yeah. Yeah. If it was happening everywhere, in some ways, like, obviously, that's much worse.
• 10:01
  Bryan Cantrill: Yes. But it also means,
• 10:02
  Adam Leventhal: like, it's probably gonna be findable.
• 10:03
  Bryan Cantrill: Right. That's right.
• 10:04
  Adam Leventhal: Right. It's like, if I have one ant, it's like, am I gonna follow it everywhere? If I have a billion ants, like, I probably can find where they're going.
• 10:11
  Bryan Cantrill: Oh, that's I'm gonna try that next when when we when the ants come back here in the wet season. I'm gonna I'm gonna I'm remind my gonna remind Bridget, like, hey. Hey. Hey. At least we have a million ants.
• 10:20
  Adam Leventhal: Glasses. Glasses. Our glass is a thousand percent full. Okay.
• 10:25
  Bryan Cantrill: Can you imagine how difficult this would be if we were just a single ant? I'm gonna try that. Oh, that's I like that one.
• 10:31
  Adam Leventhal: I'll I'll work on that metaphor.
• 10:32
  Bryan Cantrill: Yeah. That that's right.
• 10:35
  Alan Hanson: Yeah. So this and I unfortunately, this also gave us other escape routes that maybe we'll get to later but only having one case of this that sort of gave us the option of like, what if something else outside of crucible corrupted the file by accident. Right. So we sort of that was sort of like a escape hatch we had in the back of our mind, but I don't wanna get too far ahead.
• 11:04
  Bryan Cantrill: Yeah. Yeah. And is that a and this yeah. That that just also is a kind of a grip of terror too. Because something outside of crucible is corrupting and it's like, I mean, worse.
• 11:13
  I mean, really.
• 11:14
  Alan Hanson: Worse worse than me. For
• 11:17
  Bryan Cantrill: us. Maybe better for worse for somebody else. Exactly. Yeah.
• 11:21
  Alan Hanson: Well, if if there was like some if we were running some program that was like trying to find problems and it I I don't know. I don't know. It it was desperation.
• 11:31
  Bryan Cantrill: Desperation. Okay. So we find it in one this is the only spot basically. The the the place we found that and again, is what it is. Some solace, but we found it in the one spot.
• 11:42
  Alan Hanson: And I think I found the desperation quote. And that was actually part of the challenge here was that we were doing all these rebooting of of sleds, which was we weren't able to get the logs on the sled right before the the reboot. So we'd sort of lose some of our history.
• 12:03
  Bryan Cantrill: Right.
• 12:03
  Alan Hanson: So we didn't know we knew, well, here are some of the things that happened, but there's some stuff that's in this sort of behind the the moon, the dark side of the moon. We don't know what goes on there. Dave went in and added some additional stuff that would help us sort of pick up the logs after a sled had been rebooted, you know, in a surprise manner that later on would help us. Oh, interesting. Okay.
• 12:32
  Bryan Cantrill: One
• 12:32
  Matt Keeter 2: of the other things we did at this point is started stuffing more data into spare bytes in the crucible extent files where we had, like, 48 byte slots for the encryption context, and we were using, like, 42 bytes of those for actual data. And so we just stuck other information in the remaining six bytes, so that we could then look at that after the system came back up. And so in this case, was, like, the specific flush number that had last been seen by the system.
• 12:58
  Bryan Cantrill: Yeah. Interesting. And I feel we've talked about this a lot on our most vexing bugs, but I think this is an extremely important technique that you're talking about, Matt. This is a a half measure. This is not fixing the bug, obviously.
• 13:11
  In fact, this is presuming the existence of the bug, which is a very reasonable assumption. And then asking yourself the question, what additional information would we like to have when we next see this? And I actually think that this is not only is this very technically important because it assures that you will get more information. I actually think that this is very psychologically important because now you're rooting for the bug to occur again. And which, of course, you always are rooting for the bug, but you're also like also I I don't know.
• 13:42
  Maybe this is my own my own psychological terror that I'm gripped by, but it's like
• 13:45
  Alan Hanson: Wait. It gives you something to do too. Like you're taking action. You're having some agency in the
• 13:50
  Bryan Cantrill: I do think it's, yeah, it's really important. Having some agency, right? Exactly. Like this is, the bug is no longer doing this to you. You are doing it to the bug, which is very important.
• 13:58
  So yeah, so we add, and so Matt, you're describing, so that was the actual so describe the flush number a little bit and why that was something that we actually wanted to have in there.
• 14:09
  Matt Keeter 2: Yeah. So the way that Crucible works is it's block oriented and each block has both the, like, raw data that's being stored, which is encrypted, and then some amount of encryption metadata, which is like the nonce and things that we need to verify that the encryption decrypted successfully. And those are stored in two separate places. So And we have a particular little dance that we use to make sure that there is always at least one set of valid encryption metadata for every block. So in this case, we have two what we call slots.
• 14:36
  We write encryption metadata to one slot, then write the block data. And that way, if we lose power halfway through this, the encryption metadata from the other slot is still valid, we can decrypt the block. This is all, like in theory, there was actually a bug related to this that we found, while debugging this other issue. But those slots are just a fixed location on disk, and we weren't using all of it. And so the flush number is whenever we flush the system whenever the the VM sends a flush, we increment the flush number internally, and then we write that, to each extent file.
• 15:09
  And so having the flush number gives us some idea of ordering in the system where we can see, like, okay. This was flushed or this block was flushed at a particular flush number, and this block was written at a particular different flush number, so we know the order in which they landed on disk.
• 15:23
  Bryan Cantrill: Right. Right. So we get that absolute ordering, which is which it it is, potentially hugely important, or it would be it would be a very important additional piece of data, about this problem. And then, Matt, when is it in here that we that we discover the the bonus data corruption bug?
• 15:44
  Matt Keeter 2: So the bonus data corruption bug, I think, was mostly thanks to Justin, who set up a little Jepsen cluster with Crucible and, like, wrote a hacky adapter to be able to run Crucible with the Jepsen style of, like, randomly killing the network and randomly restarting, systems. And he started noticing different looking data corruption and different looking failures to decrypt in his system, which was obviously another bad sign, but it was different shapes. So it was not like the first 32 k of data being zeros. It was that, like, specific blocks or specific contexts were failing to decrypt, but, like, scattered randomly through the file instead of always at the beginning.
• 16:24
  Bryan Cantrill: And then so okay. So then now we have got the bug that we've got because in that bug, he's we've got a decent amount of reproducibility on. And so we get we've actually got that one maybe in a position to be able to debug.
• 16:37
  Matt Keeter 2: Yeah. And so that one, I was, actually out of the factory, theoretically working at the factory, but in practice, just working on data corruption because the factory was in good shape. And that ended up being us, misusing or relying on Crucible's ordering guarantees or relying on CFS's ordering guarantees in a way that was not quite correct. So there's a specific ticket. Let's see if I can find the specific link to this.
• 17:03
  Bryan Cantrill: And Matt, is now a good time to bring up the what Matt Aaron's we have taken internally to calling the affidavit? Matt, do you know Matt Aaron's, do you know that we call this the affidavit? Is this like because this is such an expression that we use, I would say frequently, but we may not have
• 17:21
  Matt Ahrens: No. I haven't heard that.
• 17:22
  Bryan Cantrill: Okay. Well, you're the one who signed the affidavit. I think it's maybe it's important for you to be aware of the affidavit.
• 17:28
  Matt Ahrens: Uh-oh.
• 17:29
  Bryan Cantrill: Yeah. No. No. No. No.
• 17:30
  No. No. Uh-oh. No.
• 17:32
  Alan Hanson: I mean, just because we have it printed on the wall. But we exactly.
• 17:35
  Adam Leventhal: Look. You guys are gonna make him start bringing a lawyer to every one of our text message conversations. Okay?
• 17:40
  Bryan Cantrill: No. In fact, I don't want him to bring a lawyer. That is actually you don't need to bring a lawyer in here. We don't need the lawyer's just gonna slow everything down.
• 17:46
  Matt Ahrens: When you say that, it makes it sound like I need a
• 17:48
  Bryan Cantrill: lawyer. That's fine.
• 17:49
  Adam Leventhal: Oh, no. No. No.
• 17:50
  Matt Keeter 2: It's not No. As it as it turns out, I I was the one that needed the lawyer here, so
• 17:53
  Bryan Cantrill: you're fine.
• 17:55
  Matt Keeter 2: But so the the affidavit was way back in the past. Crucible stored its encryption metadata in SQLite, which was, like, very guaranteed to land on disk and very guaranteed to have strong ordering and everything, but also was flushing to disk, like, all the time, because that's what it does. And so we switched, a couple years ago to what we called a raw file format where the encryption data and the block data were stored in the same file. And this is where I was talking about how we write one set of encryption data, then the block data. And if we lose power in in a situation where things are half written, the stuff is still allowed to decrypt.
• 18:29
  And I think Adam went to you, Matt, and asked, like, what guarantees do we have about the actual order in which CFS writes land on disk? And you wrote a very thoughtful write up about this, explaining that things would mostly land on the disk in order with a few subtle exceptions, and we took that as gospel. And it turns out that we were violating one of those subtle exceptions, in this second bug that we found while investigating the first bug.
• 18:54
  Matt Ahrens: Okay. So I was one for two in my, affidavit.
• 18:58
  Bryan Cantrill: No. No. No.
• 18:59
  Andy Fiddaman: I think that the affidavit
• 19:00
  Adam Leventhal: was correct.
• 19:00
  Bryan Cantrill: No. The affidavit is correct. Yep. The the no. The the we were we we misimplemented very very subtly the the affidavit, the the the terms of the affidavit.
• 19:13
  Matt Keeter 2: And so
• 19:13
  Matt Ahrens: But the the bug that the the ZFS bug that you guys found seems like
• 19:18
  Bryan Cantrill: That would that one that that one
• 19:20
  Matt Ahrens: That's wrong. Right? Like Yeah.
• 19:21
  Bryan Cantrill: Yeah. That is
• 19:22
  Matt Ahrens: that is not adhering to the affidavit.
• 19:24
  Bryan Cantrill: That is correct. So so in But as
• 19:28
  Adam Leventhal: your lawyer, Matt, I would just say in OpenZFS at the time of writing the affidavit, it may have already been corrected. So, we'll we'll just submit that to the jury. But but please continue, Matt.
• 19:40
  Matt Keeter 2: Yeah. So the ordering issue was that ZFS writes will land on disk in order unless you have written the same region more than once. Because when ZFS goes to persist writes to the ZIL, it looks at the current state of data. So if you imagine writing offset zero to a certain value and then writing offset zero to another value later on, when it's persisting, it will get the later data when it persists the first write to the disk. This is all very hard to explain while just, like, gesturing mildly at my computer.
• 20:09
  Bryan Cantrill: I think we got you. We we can we can hear your gesturing. You're you're gesticulating is coming through in the audio. So I think it's a it's and the yeah. That's what everybody's pointing out in the chat.
• 20:20
  Like, isn't affidavits don't really have terms or sworn statements? And yes. This is a sworn statement from Matt about the way ZFS works. And so we that's that's exactly what this is.
• 20:29
  Matt Ahrens: Even Yeah. I think that it's maybe confusing for folks because you would think that when you do operations, they either get logged or they don't get logged. Like, I'm doing some writes, and hopefully you're logging them. Right? Like, you do a write command, and then it gets logged.
• 20:46
  And then you do another write command, and then that gets logged later, right? But the delayed logging in DFS allows for improved performance in a lot of cases, but also can introduce these strange behaviors, like where you're saying you do a write and then you do a later write, and then the logging pulls in the later data instead of the earlier data. It's all because of this delayed logging, the potential for delayed logging in ZFS, which I think is kind of maybe the root of all evil in at least in Azil.
• 21:21
  Bryan Cantrill: Well, I mean, performance is the root of all evil. So by just by extension, this is this is this is very close to the root of all evil, but the the need to make highly performing systems. The or actually maybe or correct systems. Because as Alan is fond of pointing out, it's like, if we can eliminate the correctness constraint, I can make this thing really, really fast. So, you know, between the two of them, there are there are two gnarly constraints in terms of correctness and performance.
• 21:46
  Okay. So Matt, we find a bonus bug and we
• 21:53
  Alan Hanson: We're able to reproduce it.
• 21:54
  Bryan Cantrill: We're able to reproduce on the
• 21:56
  Alan Hanson: racklet. So where it's like, okay, yeah. We and Matt has a has an idea on where this is. Let's add this this extra sink in at this at this particular step.
• 22:11
  Bryan Cantrill: And is as I recall, was the fix to that, like, actually fewer syncs? Am I remembering that correctly, Matt? Or
• 22:18
  Matt Keeter 2: No. It was more f syncs.
• 22:19
  Bryan Cantrill: It was more f syncs.
• 22:20
  Matt Keeter 2: We basically had to add an f sync. Yeah. Yeah. Of course. You know?
• 22:24
  The cause of the cause of every it's either fewer or more f syncs. We'll fix everything. So we fixed this bug, but I I think I remember at the time telling people, like, I do not understand how this bug could cause 32 k of zeros to show up at the beginning of the file. So, like, I was still waiting for the other shoe to drop at this point, because I was like, you know, my mental model of the bug perfectly predicts the behavior that we've seen on the the Jepsen test and everything, but it really does not predict this data corruption.
• 22:49
  Bryan Cantrill: Which does have a very specific signature in terms of that that the 32 k being zeros. And maybe actually, at some point, maybe now is the time, Matt Arons, to because it reminds me a lot of the if I if memory serves the last 64 bytes being corrupted due to the is it the Silicon Image controller? Am I remembering that correctly, Matt? Do you remember this? No.
• 23:19
  Oh, do you I really? Am I okay. Now I'm now I'm worried that this is dementia. Because this is the this is the the data corruption bug that you and Bonwik had early. And you're like, this the last 64 bytes of a block, I think was stale, if I remember correctly.
• 23:44
  The
• 23:46
  Matt Ahrens: Yeah, I mean, we're going back like, you know, more than eighteen years.
• 23:49
  Bryan Cantrill: More than eighteen years. No, no, this is definitely more than eighteen years. This is way more than eighteen years. This would be This is definitely in the 20 plus category. And this is early enough in ZFS that you and Jeff are like, this is a ZFS data corruption bug.
• 24:07
  Like we've got data corruption somewhere in ZFS. This was at the time when people are starting to run it on their desktops and so on. So it's like, are running it kind of experimentally, but you two believe that this has got to be a bug in ZFS. And you're going back over and over and over and over ZFS and not actually finding anything. And then I can't God, I was really hoping you would remember because now someone observed that this was only happening on a particular DMA controller.
• 24:44
  And, but it's like a very And again, I think it's Silicon Image. And maybe is that Am I One, do I have that correct? Two, am I mispronouncing it? Because now I'm like, now I Is that that supposed to be Silicon Image? I
• 24:58
  Matt Ahrens: I think it's Silicon Image.
• 25:00
  Bryan Cantrill: It's Silicon Image. So why am I getting it like I don't remember. Why am making it fancier with Ahmadge? What am I doing over here? Am I just but, know, I as we know, I I mispronounce many things, including the word image, apparently.
• 25:10
  Although I knew I do know the word image. I'd like to say that. I just thought that they were, I don't know. The so is is any of this coming is is this are we still, like, still waters on this one? Is there anything that is rippling?
• 25:22
  Yeah.
• 25:22
  Matt Ahrens: I don't remember this at all. That was, like, I mean, I was probably, like, you know, 23 years old or something. Yeah.
• 25:28
  Bryan Cantrill: No.
• 25:28
  Matt Ahrens: Yes. I'm, like, twice that old now. So
• 25:32
  Bryan Cantrill: Adam, I know that you may
• 25:35
  Adam Leventhal: not about childhood memories. Like, this is Matt's first memory.
• 25:38
  Matt Ahrens: I know this is Yeah. Pretty much. Pretty much. Also, probably, you know, there were probably a lot of bugs in the first, like, year or two of ZFS, and this is one that rose to the level of, like, people outside of the ZFS team noticing it or, like, hearing the story. But I've all it's probably lost in the, you know, tsunami of early early No.
• 26:01
  No. It's it's
• 26:02
  Adam Leventhal: much easier. Now we're dipping into the realm of trauma. Like, Matt has erased this memory from his
• 26:07
  Bryan Cantrill: I know. I know. I I I I said at some point, I'm gonna but, Adam, do you remember this? Am I I I mean, I am now
• 26:13
  Adam Leventhal: I can say with certainty you're making this up. You're making this up for attention or for content, but, no. Nobody else remembers
• 26:21
  Bryan Cantrill: this. I think we don't know why you're making it up exactly. The only thing we could say for certain is that this is completely fabricated. No. The the so this but this DMA controller had the so it it it was always on the same DMA controller, but it was like, no, no, that's impossible because this DMA controller is like the This is a very popular DMA controller.
• 26:41
  And there were literally millions of them in the field. And ultimately, it was found when the DMA satisfied certain VA constraints, it would simply not write the last 64 bytes. And it was wholly on the controller or on the HBA. This was I just remember, apparently I'm the only one that remembers this, or I implanted this in my own memory as something that was convenient, just because there was this moment, Matt, certainly I had the realization of like, there is rampant data corruption in the world that only ZFS is detecting. And it was just a real eye opener about But again, this story is definitely having less resonance because I'm only the way to the fact that it's completely fabricated may may make this slightly less impactful, unfortunately.
• 27:38
  Alan Hanson: But not completely.
• 27:39
  Bryan Cantrill: I mean Yeah. Yeah. Thanks. Thanks for meanwhile, Adam's like, don't jump in there and save them. Let them drown.
• 27:44
  Let him drown.
• 27:45
  Adam Leventhal: Well, just to just throw a small life preserver, I would say that Matt Matt and I definitely got came to the same conclusion about data corruption being everywhere when we were at Delphix, where we were running ZFS on top of EMC arrays. And they would upgrade the EMC array, and we would detect all this data corruption. And customers would come back and say, well, none of our Windows boxes, you know, even noticed. Like, yes. Yes.
• 28:12
  Yes. You're welcome.
• 28:13
  Bryan Cantrill: Exactly. That's right. Right. Exactly. Well, so the the the the moral of that story slash fabrication was that the the the particular pattern of corruption, it does really matter.
• 28:23
  And the fact that you the fact that we had this first 32 k being zeros was a very specific fingerprint. Matt, as you say, the bonus data corruption bug did not match that fingerprint and it just felt but but the second one was still out there.
• 28:36
  Alan Hanson: Well, this was this was actually my when I told my wife it was still in the first one. Like, she didn't know. Everything was on the table in the beginning. Yeah. It's us, is it us, you know, where is it?
• 28:45
  So but finding this the first one that Matt found, I was trying really hard to convince myself this okay this could be it or there could be some other rogue agent that had done something to do this 32 k. Like, I don't know. I don't know. But maybe maybe squinting really hard.
• 29:09
  Bryan Cantrill: And but so so we are I mean, we fixed that bug, Matt. You get you get the fix in for that, which is great. And we continue to to test the get back to test in this release.
• 29:20
  Alan Hanson: Yeah. And now that we sort of had this scanner that Matt had made for us, I was going out and scanning dog food you know like every other day I was scanning it and because it wasn't we didn't really have it completely polished. Took a couple hours to go dig through all the logs and make sure that oh, that's just a false positive because we're doing IO to the to the volume and things like that. So like we had it we had sort of I was gathering data on to look for this corruption while we were still doing updates in the background.
• 29:55
  Bryan Cantrill: Yeah. Okay.
• 29:55
  Adam Leventhal: And dog food, it should be said, is our the instance of Oxide, the Oxide rack that we use internally, and we we do our our dog fooding, our our eating our own dog food, that is to say, of the next release.
• 30:08
  Bryan Cantrill: Yes. And if you are actually in the Oxide office, dog food actually lives in the Oxide office and kinda looks the part. It's got you know, don't look at it's it's, you know, it's taken some licks. That that is definitely not production anything, but it has been it's been invaluable for us to to be able to. So we are continuing to do I mean, in in part because we really wanna test this update functionality, which has the side effect of bouncing boxes when they're doing real work.
• 30:37
  Alan Hanson: Mhmm.
• 30:39
  Bryan Cantrill: Alright. So what's the what's the next? We've bonus data corruption bug fixed and all of us thinking that like, wow. Well, it didn't exactly match the fingerprint, but it was a data corruption bug. We actually knew, like, we definitely found a data corruption bug, which is very important, and we're very grateful to have that fixed.
• 30:56
  Matt, great going on that, great going by Justin to actually take this Jepsen like approach to testing it. What was the next kind of the next turn here?
• 31:05
  Alan Hanson: So I think we had we did the first update of dog food to the new bits and we found a bunch of problems and there was there was some panic for sure but then we had sort of looked at all of the where they were and said, okay, this was the old code running as we're updating to the new code and it was the problems we're seeing on the on the old code. So it it sort of made sense. Right. We could look at it and say, yeah. Okay.
• 31:38
  That we do see a bunch of these zeros, it's all the old code where the bug actually happened during the update process. And once things had got to the new code, it looked okay. And so this was like I think Halloween weekend and I was like, I'm gonna continue to do my background scans and I'm gonna do three more updates to dog food. Yeah. So like we had the one the first one to the new bits bunch of problems.
• 32:06
  I did another one no problems, another one no problems, another one no problems. So was like okay I think I think we got it because I've done a bunch of updates now which was our sort of reproducer.
• 32:17
  Bryan Cantrill: Right. Right.
• 32:18
  Alan Hanson: And everything was looking okay. So I was like, alright, I think, you know, we'll let this one go into the wild. And then Angela did another update and we had another instance that then panicked with this this 32 k zeros problem. And I think I I sent
• 32:45
  Matt Keeter 2: Throughout this whole process, we've been posting, like, DEFCON pictures of various DEFCON levels in our group chat. So I think this brought us back to DEFCON zero.
• 32:54
  Alan Hanson: Yeah. And I think and it like, I was so sad at first that I only DM'd Matt directly. I didn't like put it in the storage channel. So I was like I think found another one and you know. So we started messaging back and forth and it was like, Yeah, this does look bad.
• 33:17
  And then there was I had to go back to the product huddle that week and be like, you know, that bug that we said we fixed and we had stopped the release for and now we're like on a go. It's it's not fixed.
• 33:34
  Bryan Cantrill: We're back to well, which again, I mean, the the given the mismatch in fingerprint, there is some solace in this because like, there is this little piece that didn't add up from the previous bug. Like, just did not we we it was really hard to talk our way into that what feels like an extremely distinctive fingerprint. This 32 k of zeros just sounds just feels very unusual. Yeah. Yeah, Matt, how was it what was your was your feeling?
• 34:02
  What was your feeling when when we, we hit this again?
• 34:08
  Matt Keeter 2: Concern, I think was what I posted in all capital letters to Alan. Concern. Right. Right. But also the feeling like, yeah.
• 34:17
  You know? Okay. Sure. I I I kind of I was I was waiting for this shoe to drop.
• 34:20
  Bryan Cantrill: Right. Right. Right. Right. Right.
• 34:21
  Right. We were kind of like waiting for it and like, okay. So we still have it. We we we fixed a bug. We know we fixed a bug.
• 34:28
  We fixed an important bug. We'd like no. We've got total solace in that. It was not the as it turns out, was not this bug, was not the bug that caused us to go search for this. So now what?
• 34:39
  How many corrupt blocks do you have, Alan, at this point?
• 34:44
  Alan Hanson: I don't know if it was just one or if I had multiple. I I think there may have been multiple cases this in this time where we had yeah. I think there was I think there was a a couple that were bad.
• 35:02
  Bryan Cantrill: Bolded concern in all caps.
• 35:05
  Alan Hanson: Yes. So what because we had a little more logging now and I had been doing these regular scans. I knew that like on October 30 everything was good. And on November 1, we had a problem.
• 35:21
  Bryan Cantrill: Right. Right. So I
• 35:21
  Alan Hanson: had like a
• 35:22
  Bryan Cantrill: Yeah. Time That's actually really important.
• 35:25
  Andy Fiddaman: Yeah.
• 35:25
  Alan Hanson: And so then from that and from the logging, the new logging stuff we had and sort of knowing a little better what we're looking for I was able to like walk all the way forward from the good time to the bad time and then once I had the sort of end state I then walked backwards with the like you know here's the truth table at the end, here's the truth table at the beginning and figure out where about where we got a corruption and a scary thing about this was that because this corruption was sort of hidden from crucible at one point crucible had repaired and we have three we have a three way mirror and we had one of the legs of the mirror that had the corruption and two that didn't. But the IO is still going on. We don't even know that there's corruption out there. And then at some point we say, these three these three mirrors aren't in sync. Let me make them all the same again and we copy the corruption from one of the mirrors to the other mirror.
• 36:25
  Right. That was like, you know, the infection is spreading.
• 36:29
  Bryan Cantrill: Right. Right. It was yes. It was the opposite of repair.
• 36:32
  Alan Hanson: The opposite of repair. Yeah. So but I was able to sort of walk backwards, figure out, okay. I'm pretty sure this problem happened during during a power cycle of a specific downstairs when it was doing repair of one of the one of the extent files.
• 36:55
  Bryan Cantrill: Okay. So that's a that's a really important development.
• 36:58
  Alan Hanson: Yeah. So like to know, okay, here here is the condition that we think caused it because that enabled me to go over to our sort of
• 37:07
  Bryan Cantrill: So this is repeat this. This is a power cycle while we are repairing. And repairing is not I mean, it's not an abnormal mode of the system in that we expect there to be some repair whenever a sled disappears and reappears, there'll be some some repair happening. Yeah. So and we expect sleds to come and go, but it's also, like, not, you know, in in kind of the default operation, we kind of expect sleds to stay up.
• 37:41
  Yeah. And so we this is a slightly unusual operation to be engaged in a repair and then to have the lose power while doing the repair. I mean, we've we've got a a more unusual situation, which might explain why we don't see it as much as we would otherwise. You could explain why you'd be able to do a bunch of updates and not see it.
• 38:04
  Alan Hanson: Yeah. And I think it was the double thing here that really was the trigger was we need to be in this either repair or reconciliation and then lose power.
• 38:17
  Bryan Cantrill: Okay. So that's an important detail. What do you Then what do you do with this detail? Because right now you head over to one of our racklets and if I recall correctly, start pounding on them.
• 38:29
  Alan Hanson: And start Yeah. So we do stuff to make sure that the discs are all busy so that we know that on a power loss there's gonna have to be some repair that goes on. And then I just tried to keep narrowing the operations that still would show the the failure. Right. And it does feel good when you're able to do it frequently because that really gives you hope that you're gonna you're gonna get to the end.
• 38:58
  Bryan Cantrill: Absolutely. Yeah. Well, and then so then we, at some point here, you are able to reproduce this now on the Rocklitz. Yeah. And you are able to reproduce it way more than once.
• 39:12
  Yeah. This is the which is great news. Yep. I mean, you were looking a little down, but I thought it was great news. Yeah.
• 39:20
  Because, this is I mean, we we now be are beginning to get a corner on this thing. And I think the thing that that was both most vexing and most encouraging is all of this corruption had this same 32 k of zeros fingerprint.
• 39:36
  Alan Hanson: Yep. Yep. That was really Yeah. That was the happy part of the dark part of, I guess.
• 39:44
  Bryan Cantrill: That was the happy part of the dark part. Also it's just weird. It's just like 32 ks is just like not a number sloshing around in a lot of places.
• 39:50
  Alan Hanson: We had some ideas, Matt had some great bad ideas about changing certain ZFS tunables to see if maybe this 32 k was something that we could you know, we were looking for anything that was 32.
• 40:05
  Bryan Cantrill: Anything that would I mean, I love that. I love it is a great bad idea because anything that I can do to like, can I turn a knob and just change this change the problem? That way at least I know that like, okay, these things are related somehow. And we can begin to because on the one hand, we've got a cordon around it and that we can reproduce it. But on the other hand, it's like it requires you know, we are, this only happens when we actually power off, hard power off a sled.
• 40:30
  So it's like, which is not exactly an easy failure mode to go debug because mean, the system is, you know,
• 40:36
  Alan Hanson: it's off. And I think James I mean both James and Matt and myself we were all I mean there were lots of sort of false trails that we had attempted that didn't lead to anything and there's lots of stuff going on where we're like we did the bug and you can see like the knowing the result, it's entertaining to go back and read
• 40:57
  Bryan Cantrill: it and
• 40:58
  Alan Hanson: be like, well, that's that's never gonna work.
• 41:00
  Bryan Cantrill: Right.
• 41:00
  Alan Hanson: Oh, that you should've you should've went that way but you didn't. You know? And you sort of see all the the directions we wandering around there in the dark.
• 41:10
  Bryan Cantrill: Yeah, I mean, always think that all those directions end up being important because one, they allow you to better understand the system. I mean, I'm gonna go explore this aspect of the system. You come up with more confidence or in the case of Justin and kicking the snot out of one aspect of the system, you discover something else, which actually becomes germane. And or you do develop kind of the better tooling along the way.
• 41:35
  Alan Hanson: Yeah, there were certainly improved tooling that we came out of this with where we can better inspect the sort of metadata and and crucible and and things like that that we're definitely gonna keep going forward. Know?
• 41:47
  Bryan Cantrill: So Matt is is saying in the chat that so ZFS immediate right size is 32 k. So let's change that to six let's hey. Why not? Change that to 16 k? And the but, Matt, that does not I'd a great bad idea.
• 42:03
  Truly great bad idea. I I I a terrific hypothesis, but no.
• 42:07
  Matt Keeter 2: I just found this by ripgrepping through, like, the entire ZFS subdirectory for any for that number showing up anywhere.
• 42:14
  Bryan Cantrill: Would you I mean
• 42:15
  Adam Leventhal: That is would a sign of inspiration.
• 42:16
  Matt Keeter 2: Yeah. What what Yeah. Yeah. It's not a good sign.
• 42:18
  Bryan Cantrill: Yeah. But, you you know, you know, I think it's that's like it is. It's a great idea. And, and importantly, like, you can do the experiment, and you can really and you can then really disconfirm it. Like, okay.
• 42:30
  This is actually, like, we we
• 42:32
  Adam Leventhal: 100%. It is both a sign of desperation and a great idea because it's one of these, like, hypotheses that is really easy to
• 42:40
  Bryan Cantrill: Easy to test.
• 42:40
  Adam Leventhal: And explore.
• 42:41
  Bryan Cantrill: Yeah. Easy to test, easy to invalidate. Like, you're good. Not that one. Back to rip grab.
• 42:47
  And okay, so within what is the next kind of, the next sequence of events? Are we at that Friday yet?
• 42:54
  Alan Hanson: I think this is about, we're at that Friday, yeah.
• 42:56
  Bryan Cantrill: Yeah, and where, because there's a moment when the because I was talking with you and Josh in particular about the prospects of this actually being a because our assumption has been this is a crucible bug. Yeah. Just in general. The Like, we we had
• 43:17
  Alan Hanson: Robert denies knowledge of this but I had gone to him originally with my here's what we need to do when to make sure we're repairing extents on an unsafe shutdown and we work together and came up with the sort of step by step copy the file, f sync the file, f sync the directory, move the file, f like this you know seven or eight f syncs around this whole repair process so that we could know where we are if interrupted at any point and recover from that if interrupted at any point. And I I felt like we had come up with a a good solid solution. Adam, I went back and looked and you had actually reviewed this code and gave it your, you know, blessing. So I was like
• 44:00
  Andy Fiddaman: No.
• 44:02
  Bryan Cantrill: Yes. Okay. Yeah. We actually we have a signed statement from you saying I I had a Leventhal in perpetuity.
• 44:09
  Adam Leventhal: Mhmm. Yeah. It's a deepfake.
• 44:13
  Bryan Cantrill: It is in this case.
• 44:14
  Alan Hanson: This was pretty deep fake. So, yeah, this is we're on the Friday now and I'm like, I I don't I don't doesn't seem I really can't find anything with with crucible that's wrong. And it's really kind of feels like this might be something in in ZFS with the Zilog, but that how? I I didn't understand how. Like, we're not you know, there are other people who use ZFS and like
• 44:42
  Bryan Cantrill: There are other people who use ZFS. Yeah.
• 44:44
  Alan Hanson: I was like, just I don't know but I don't know what to look for. Like, who you know, what what can we do to try to figure out?
• 44:54
  Bryan Cantrill: Yeah. And I think that the, and I definitely remember there's a little bit of like, well, can't possibly be a bug in CFS. And I think it's like, the fact that something has been used for a long time, even with a terrific track record, think does not, it may make it unlikely. And you should never, mean, it should never be one's assumption that this is, but it's like, it's like chip bugs. It's like these things do exist.
• 45:22
  And and, you know, Matt Arons, I know you're like it's like, These things definitely exist, by the way.
• 45:30
  Matt Ahrens: Yeah. And I would add that, like, for sure, ZFS is used by a lot of people in a lot of different ways, but it's probably not you know, the lot those people are not pulling the power plug and then relying on, you know, strict ordering of a very careful system, carefully designed application that's using it. There's probably not a lot of other people that are doing what you're doing. There were enough that it was found in OpenZFS, ZFS on Linux, and fixed a few years earlier, but still it existed there. You know, it existed in ZFS for, like, fifteen years.
• 46:08
  So
• 46:10
  Bryan Cantrill: Yeah. And I think I think that you just can never really you you have to be just very, very careful about appealing to the longevity of a thing. Mean, again, it's like it's all the reason in the world to say, this should not be our first assumption, but we should we we just we need to follow where the the data sends us and wherever that might be, even if it sends us much deeper into the system.
• 46:35
  Alan Hanson: But I I think next is the the sort of breakthrough is James and what he he discovered for us.
• 46:43
  Bryan Cantrill: Yeah. So James, why why don't you describe your breakthrough? Because this was very important.
• 46:48
  James McMahon: Excuse me. We had known that it was a power off during the this repair process that that seemed to be the problem. So, we were talking about it, and it seemed we said it seemed like if there was a if this was true, that this was a problem, we should be able to reproduce this with a small program, not crucible, not the whole, you know, control plane and everything set up. So I had been doing I'd been working on something else, had been doing a little bit of this type of testing in a VM, in a Propolis VM, just running a program and terminating the Propolis hypervisor to simulate this power off, and saw the issue with a very small program. So simulating the part of the repair where we take something that we had copied from a remote downstairs into our current directory, and then copying that over the extent, you control like you terminate the Prepla server and then that simulated or reproduced the problem, you know, nothing else required.
• 47:49
  So that certainly was a surprising result there. We originally thought it was something from there, we thought it was something in the Rust standard library, but we then reproduced it again with just a very simple c program. At that point, basically, everything was pointing to some sort of an issue, with ZFS.
• 48:09
  Bryan Cantrill: Yeah. So in both so I I and I wanna get to that program in a second. But so, James, when you, reproduce were you surprised that you were able to reproduce this so easily in Rust in because, I mean, this is a this is a huge lurch forward because now we are not powering off sleds. Now we are bouncing VMs. We are and we've got we've got Crucible out of the picture, which is really great for from Alan.
• 48:36
  I mean, yeah.
• 48:37
  Alan Hanson: Well, my wife, do you know? Don't worry. Don't worry.
• 48:40
  Bryan Cantrill: Don't Don't don't worry. I'm not sure about everyone else. I think I may be the only one at the company. We may get rid of everybody else, but they, the no. The I mean, that was huge to get to get Crucible out of the picture.
• 48:52
  But now we've got Rust very much in the picture, and there is a period of time where, I mean, it's like, okay. This is a maybe this is a very weird Rust standard library. So it's like, you know, let's go into how this thing how complicated can it possibly be? The actual path that, like, writes to because what and James, you wanna describe it just a little bit. Know you linked to it in the chat, but kind of like because there are a bunch of operations that are happening where we are we're we're renaming we're moving copying files, renaming files, and so on.
• 49:29
  Do you wanna describe that just a little Because I think that that's really important here.
• 49:34
  James McMahon: So the the Rust program writes out a file. So this would simulate existing data in a downstairs. So it writes out an extent file. The second step would be create this dot copy file, write a whole bunch of random data there, completely different, simulating a file that we had pulled via the repair API from a remote downstairs. And then the third step was to use standard fscopy to overwrite from the copy file to the extent file.
• 50:06
  The fourth step was an fsync. And then it would keep doing this over and over, ticking up, incrementing the extents that it was working on, very much like our repair operation would do.
• 50:19
  Bryan Cantrill: And so this is great news because now this thing is much smaller, but it's looking like Ross is very much it has a possibility here. So Dan then realized, like, okay, I'm gonna try to actually reproduce this in C. And I I just kinda saw this go flying by in the chat. I don't know. For whatever reason, I'm like, I don't know.
• 50:40
  It feels like a long shot. But it which is which it shouldn't, because Dan is then able to, like, pretty quickly reproduce this at in a c program. And if he if we'd not been able to reproduce it in c, that wouldn't have necessarily That that wouldn't necessarily been a hugely valuable data point. That wouldn't have That would not have pointed the finger definitively at Rust.
• 51:02
  Andy Fiddaman: Mhmm.
• 51:03
  Bryan Cantrill: The fact that we could reproduce it and see meant that like, okay, there's now a whole bunch of software that's not in the loop here. Yep. That now we are really down to to to not very much in terms of like, this is actually this is very clearly now a ZFS bug. This is the this is crucible. Not only can crucible go home, all of Rust can go home.
• 51:26
  Yeah. So this is this is this is good news. And then, you know, this is one of the advantages of having a global team is, you know, sometimes when it's time for, some of the members of the team to go to bed, other team other members of the team are just getting up. So, Andy, at some point, you kinda come in here to, I think, your Monday morning and, and having seen all this having gone on.
• 51:54
  Andy Fiddaman: Yes. Try. Came in Monday morning with my time zone advantage, and there was a lot of chat to catch up from over the weekend. But, you know, nice place to be in. We've got a a small c program that reproduces this, and James has proven that it reproduces.
• 52:08
  So I take that and I spin up a VM and I reproduce it. And then I have to start looking at the operating system. What I did is I I gave it a dedicated ZFS pool on a dedicated disk so that I keep it all separate from the operating system and sort of streamline the tracing I was doing because it ZFS gets a bit noisy if you start running Dtrace on it. And I started I had a bit of suspicion that this has something to do with, write caching or something to do with the ZIL which is a ZFS intent log where transactions get queued up before they're written out. And those are used and replayed if we have to recover from a power failure.
• 52:48
  So what I do is I run a loop to reproduce this and every time that I reboot the machine I snapshot the disk so I've got it in its state before it comes back up. And after I reproduce it, I can import that pool into a different VM and have a look at it and actually dump out the VIL. And then we had a I think by then, that was very quick description, but by then, other people had started to wake up on the East Coast. We got on a call and I posted this data I collected. And I think it was Matt that very quickly came in and said, in this trace of records, I can see that we are doing four eight k writes, then a truncate, and then a whole slew of writes after it.
• 53:28
  And wouldn't that explain why we have 32 k
• 53:32
  Bryan Cantrill: of zeros at the start of the file?
• 53:36
  Andy Fiddaman: That's gonna be And then
• 53:37
  Bryan Cantrill: That's gonna be a great moment. Yeah. That just feels like blood in the water at this point. Yeah.
• 53:44
  Andy Fiddaman: So at that point, the the hypothesis is that when we replay the ZIL after booting after a a power cycle, a a hard reboot, we're playing the transactions in the wrong order. All the transactions are in the ZIL in the wrong order. And at that point, you reach for DTrace and some speculative tracing and start watching you gotta use speculative tracing occasionally.
• 54:09
  Bryan Cantrill: Know, I don't know if you're
• 54:10
  Andy Fiddaman: just saying That
• 54:10
  Bryan Cantrill: was a benefit or not, but I just you know, I let's just say that it
• 54:15
  Andy Fiddaman: That's true.
• 54:15
  Bryan Cantrill: I did. I I detractor once said that speculative tracing has never been used for anything, and I'd like to I every time we use speculative tracing on a bug, I I I think of that particular detractor. So anyway, sorry. Go on, Andy.
• 54:27
  Andy Fiddaman: Oh, I'll be entirely honest. It's it's maybe twice a year I use it, but when I use it, it's useful.
• 54:31
  Bryan Cantrill: That's right. Yeah. Exactly.
• 54:33
  Andy Fiddaman: Yeah. I start logging the calls into the the functions that put entries into the ZIL and we run the reproducer, and I can correlate a reproduce bug happening in conjunction with the entries being written correctly. So the reproducer that's in c, which is linked to from the Lumos ticket is actually part of the Lumos ticket. It's presumably being pasted a few times in chat. It creates two random files and then it copies one over the other and it does that by opening the destination file with a truncate flag.
• 55:08
  So the very first write to that file has been truncated and then you get all your data going. And it writes the data in eight k chunks. And I can see that we end up with this corruption, but all of the entries went into the ZIL or were queued into the ZIL log in the correct order, but they ended up on disk in the wrong order. And that was interesting. It showed up.
• 55:30
  Bryan Cantrill: That's a that's that's an important detail. And
• 55:36
  Andy Fiddaman: there's a lot of detrace involved here, but at the end, I realize that some of the some of the right transactions are being committed to the ZIL as if they'd been synchronous rights to the file. And when you commit the ZIL to disk, you commit all synchronous rights first, and then you do all of the asynchronous rights afterwards. And that explains why they're being reordered. But the question was why are these rights going in asynchronous? They weren't synchronous rights, but they were put into the ZIL as if they had been.
• 56:09
  Bryan Cantrill: Right.
• 56:12
  Andy Fiddaman: And this comes back to an optimization that was put into Solaris in 2007.
• 56:20
  Bryan Cantrill: 2007.
• 56:22
  Andy Fiddaman: 2007. Yeah. When Sun were doing
• 56:24
  Bryan Cantrill: July 2000. Exactly. I know. I I want I both both stared at this guy. It's July 2007 is when this thing comes in.
• 56:31
  07/24/2007.
• 56:34
  Andy Fiddaman: Yeah. So this is when Sun were bringing up their new CoolThread service, the t one, the Spark t one, which were low power chips that had a lot of threads. So I think they had a up to eight cores, but each core could have six threads or something like that. That's something like 32 threads plus those early cool threads. They weren't particularly fast.
• 56:53
  They'll have lots of limitations, like one FPU for every core. That's all I think.
• 56:59
  Adam Leventhal: Yeah. That a very serious limitation of a turnout.
• 57:02
  Bryan Cantrill: Well, I mean, come on. Who actually does floating point operations?
• 57:05
  Adam Leventhal: Oh, mean, the other oh, okay. Oh, Everyone all the time. Everyone all
• 57:08
  Bryan Cantrill: the Yeah.
• 57:11
  Andy Fiddaman: And at the time, I was consulting for someone that was filling data centers with Sunblade six thousands and t sixty three forty servers. And I had a lot of lots of work parallelizing things, basically, trying to but, yeah, some were were doing this, and they were running my I'm reconstructing from what I can read in the Open Solaris book, which is the public information we've got, but it looks like they were running a whole load of benchmarks for different workloads. And they realized that when they did a workload that was supposed to simulate a busy mail server, which is lots of async rights and the old f sync, there was an extremely hot lock in the ZIL code that was causing f sync to be slower than it should be.
• 57:52
  Bryan Cantrill: In And it was this was coming out of the Niagara work. So I actually okay. That's interesting. I thought this was actually this was coming out of the supporting CFS in the field, but that's that's actually interesting that it was coming out of a out of more of a synthetic benchmark.
• 58:07
  Andy Fiddaman: Yeah. I mean, the ticket is is all the information I've got, and it talks about some evaluation that was done, but I don't have the data. So Yeah. Interesting. They were seeing slow f syncs in a very, very hot lock.
• 58:19
  And their solution appears to have been when a thread initiates an f sync, let's log the next four writes that that thread does into the ZIL as if they've been synchronous. And it takes a lot less work to insert it as if it was a synchronous write than if it was asynchronous because asynchronous write just goes into the global transaction group synchronous list as opposed to having to find the object, find the right list, chuck it on the end. That's what I think is going on.
• 58:48
  Bryan Cantrill: Yeah. Interesting. And
• 58:50
  Andy Fiddaman: yeah. I don't know why it's I don't know why it's four, but if you write your data in eight k chunks, four is going to explain 32 k very nicely.
• 58:58
  Adam Leventhal: I appreciate, Andy, that you're describing this in the most generous fashion. And even as you do so, you're like, that doesn't sound right. That sounds fishy.
• 59:08
  Bryan Cantrill: Well, also whenever there's something like this where you, like, have this 32 k, where could this possibly be coming from? And then you discover it's like, oh, it's four times a day. It's like, there's your answer, Fishbowl. But that I mean, I don't know, Adam. Do you does that I I I often think of that where they they they they have determined that, that Homer is mister Sparkle and why that has happened.
• 59:33
  Adam Leventhal: Yes. Yeah. But even Andy's just saying it like, hey. This this really complex, you know, primitive around synchronous semantics across which people reason yeah. A barrier across which people reason.
• 59:50
  What if we just, like, I don't know. Like, we're loosey goosey about that. What if that what if we just do that instead?
• 59:57
  Bryan Cantrill: Lucy goosey, though, in kind of a strange way. You know what I mean? Like, it feels does it it it kind of has, like, a feel of, no. No. Like, I'm making this thing synchronous.
• 1:00:06
  Like, making something synchronous is always, like, I don't know, always feels better to be synchronous. Isn't that better? Is that feels better.
• 1:00:13
  Andy Fiddaman: Of course, if you if your system doesn't fail, it's fine because the data goes to disk in the right order and this stuff and the ZIL is never used.
• 1:00:20
  Bryan Cantrill: Yeah. Exactly. If it's just pieces of fail, why do have the ZIL at all? I mean, it's
• 1:00:23
  Adam Leventhal: like Shut up, nerd. Right.
• 1:00:28
  Andy Fiddaman: Yeah. And the only thing we have in the issue is something like, as as was suggested in the evaluation, I'm going to hack up the this suggestion. That's also a very positive result. And indeed, that they did get f sync down significantly. But, of course, unsafe at any speed from what they've done.
• 1:00:48
  Alan Hanson: As a sort of callback, part of the part of the problem here was that we were f syncing. We had crossed the barrier into too many f syncs, and that put us into this window that would trigger this bug.
• 1:01:03
  Bryan Cantrill: Right. Right.
• 1:01:05
  Alan Hanson: Like we had talked before about
• 1:01:06
  Bryan Cantrill: f syncs
• 1:01:07
  Andy Fiddaman: followed very quickly by a lot of rights. Yeah.
• 1:01:11
  Bryan Cantrill: And Matt, Aaron, do you now do do you recall anything? Mean, this is this is now truly I mean, obviously, eighteen years ago. And did this Yeah. Did this issue ring any bells for you? I mean, is this or or or maybe more recently because the OpenZ I mean, this had been removed from OpenZFS, but honestly, not very long ago.
• 1:01:34
  So this is like a And
• 1:01:35
  Adam Leventhal: kind of subtly, I like, in the OpenZFS repo, it was sort of a, oh, by the way, we're gonna remove this thing that might cause data corruption. But, like Yeah. I was sort of sus like, expected data corruption to be, like, the above Yeah. Like, the Yeah. Not not paragraph eight subsection three.
• 1:01:58
  Andy Fiddaman: I I think it was act first I think it was first removed because they were having trouble with the TSD, the the the Fed specific data implementation on Linux or FreeBSD, whichever one they were bringing in. I think it was removed, then it was put back because it had a performance effect. And then it was finally taken out in 2023 with comments that said, you know, we need to guarantee the order of rights.
• 1:02:21
  Matt Ahrens: Yeah I think the original bug I went back and looked at I mean I don't remember this particular where you know the original change that introduced this bug in Sclaris back in 2007. I don't remember this particular change, but I definitely remember that era of ZFS is lower than other things, and we need to run lots of benchmarks, and we need to make lots of heuristic tweaks. And for sure, at the time, I remember being know like holding my breath for something to go wrong with messing with the inner workings of this sensitive you know the bill is very sensitive both to performance and correctness. And and, you know, it's not like the easiest to understand subsystem. Yeah.
• 1:03:21
  Yes. So I love
• 1:03:22
  Adam Leventhal: I love this description of of this very politic description of your esteemed colleagues all volunteering to assist with performance work
• 1:03:31
  Matt Keeter 2: Yes.
• 1:03:32
  Adam Leventhal: Leaving you to work worry about the correctness work.
• 1:03:34
  Bryan Cantrill: That's right.
• 1:03:36
  Matt Ahrens: Yeah. That's the it's kinda what it felt like at the time.
• 1:03:40
  Bryan Cantrill: My job is to make the system faster. Your job is Yeah.
• 1:03:46
  Matt Ahrens: And and I mean, the the bug report from back then, it has a lot of documentation about how they made the system faster. The the lock is not as contended. The benchmark goes faster, which is great. Yes. But yeah, I mean, that you know, we were trying to get ZFS, like, to be more widely accepted.
• 1:04:05
  Performance is a big part of that. When you're competing with, like, simpler systems Yeah. Like UFS as well as Linux file systems like EXT. So it made sense to do this work, but I think it was was known to be scary, and it just took a couple decades for us to figure out why.
• 1:04:28
  Bryan Cantrill: And then in terms of when this work I mean, because as you say, it it was removed in OpenCFS, what, like, years ago. Was Yeah. It feels like that might've been though as part of other work where someone realized like, oh, and also like, think this might have possibility for data corruption. Or do you know, or was there some, a similar experience where this was actually debugged from first principles?
• 1:04:52
  Matt Ahrens: I don't know. I mean, the person who fixed that, Alexander, he didn't say in his bug report, like, if they had customers that hid this or whatever. It kind of seems like he noticed a bunch of code that wasn't needed and then kept digging around and then was like, that's weird. That's weird too. Gee, this is really not great.
• 1:05:15
  And fixed all that stuff, which is wonderful. Yeah. But, yeah, like, he he probably buried the lead with, like you know, you're, like, five paragraph thin. He's like, yeah. We cannot randomly jump between sync and async, unless we want data corruption due to write reordering.
• 1:05:31
  Yeah. That that is true. But I thought that the analysis Andy did was great. And they're right up in the bug report, the the most bug report that you did, Andy, was really wonderful and evocative and kinda told the whole story of where the bug came from and how it shows up. So I I really appreciated that.
• 1:05:56
  Andy Fiddaman: Well, then Andy One really nice.
• 1:05:58
  Bryan Cantrill: Yeah. Go ahead.
• 1:05:59
  Andy Fiddaman: I'll say one really nice thing that the developer did for us, whoever integrated this, they made it attunable. That number four was not fixed. It was in a global variable. Yes. And I assume oh, I'm getting here.
• 1:06:11
  I gotta assume. I assume they want they did the number four would arrive that by playing with it a bit. Yeah.
• 1:06:17
  Matt Ahrens: For sure. A 100%. He tried a bunch of values Oh my god. To see what would perform best on that benchmark on that hardware twenty years ago, and that was the best value. Nobody And ever looked at it since.
• 1:06:27
  Bryan Cantrill: Oh my god. I know. And what what
• 1:06:30
  Adam Leventhal: in defense of that so obviously a problem.
• 1:06:32
  Bryan Cantrill: I think
• 1:06:32
  Andy Fiddaman: Oh, go. Yeah.
• 1:06:33
  Bryan Cantrill: Go on.
• 1:06:33
  Adam Leventhal: That, like Wait.
• 1:06:34
  Matt Ahrens: What else can you do?
• 1:06:35
  Adam Leventhal: Well, there there are numbers like that that persist, I think, in ZFS, like 200 Metaslabs per
• 1:06:40
  Matt Ahrens: Oh, yeah.
• 1:06:41
  Adam Leventhal: Disk where you're like That sounds like it worked really well on Jeff Bonwick's desktop machine in 2002 running at 5,400 RPM or whatever. I mean, I don't know whether that number should be higher or lower or whether 200 is, like, some universal constant. But, like, constants, you know, happen.
• 1:06:59
  Bryan Cantrill: I I Adam, I love the I love your in other entree, like, in defense of this, because, actually, I wanna go actually, I wanna go after this other thing that southern this other I actually Yeah. Right.
• 1:07:10
  Adam Leventhal: Let me sideswipe Jeff for no reason.
• 1:07:12
  Bryan Cantrill: Yes. Sideswipe Jeff. Exactly. But yeah. I mean, you're I mean, you're right.
• 1:07:17
  I mean, ultimately, when you have these things I mean, our objective is to make all these things auto tuning. You can't do that for everything. But this was also a I mean, the tunable, I think, kind of indicated that this was, that there were some heuristics here. That there were this was this was not a, and and that number, yeah, again, comes from, as you say, twenty year a twenty year old machine, one that is that probably does not bear much doesn't bear much resemblance in many ways to the machines we run today. So but, Andy, this is in many ways the best possible thing.
• 1:07:52
  The fact that it is tunable, one, means we can very quickly I mean, just to this one feels like we've got a lot more supporting evidence around than we did around the the 32 k tunable that Matt had found. This is feels a little like this is a little more on more solid footing than ripcrep. But it similarly allows us to explore this by tuning this thing down to zero. And let's see what happens. I think that was the that was your next move.
• 1:08:19
  Right?
• 1:08:21
  Andy Fiddaman: Yes. And one of our colleagues, Justin, had to set up in his lab, and I think over the overnight, he did 120 boot loops or something with it set with it set to various values and could prove that when it was set to zero, it the the bug never manifested at all. So we're pretty confident by then. It that is not a complete fix because there are other ordering problems in Deville, but we weren't hitting them with the workload we're running. But they have now been fixed, but we didn't need to necessarily fix those to get the release over.
• 1:08:50
  Bryan Cantrill: Yeah. And we'd from a I it's a a strict, like, the because we were holding the release for this for for obvious reasons. And, you know, I think we we knew that we had we had a cordon around this thing, blood in the water felt like we were going to make, we're just going to, it's one these things where like, we're just going to know a lot more in a couple of hours. So let's see where And we I mean, I was definitely concerned that like, yeah, we're going to know a lot more in a couple of hours. And what we're gonna have is a really dicey fix where we're gonna have a fix that itself is not gonna be low risk.
• 1:09:26
  I mean, is always the concern you have with something like this. Yeah. Where we, yep, we actually know what it is, but now we need to go do surgery on the ZIL and then ship it to customers. Like what can go wrong? And I mean, that's obviously, we would do what we would have to do, but you just kind of hope God, don't know.
• 1:09:44
  I well, we're just gonna know more. So let's go know more. And then like, oh, sweet Jesus, there's a tunable we can set to zero. And just turn off this behavior was I mean, the you could not ask Yeah. For a a quicker resolution from that perspective or a lower risk fix in particular.
• 1:10:05
  Felt like the risk of turning this to zero, there was some potential risk to performance, but- But not. Not a regression. A regression. Adam, I'm sure you've turned Alan's perspective on performance regressions. If you are transitioning the system from incorrect to correct, it's not a performance regression.
• 1:10:29
  It can't be because That's right.
• 1:10:31
  Adam Leventhal: That's who's right.
• 1:10:32
  Bryan Cantrill: No. Be because the the the the the system, as as Alan has said many times over, if correctness is not a constraint, I can make this thing really, really fast.
• 1:10:42
  Adam Leventhal: Ergo. Right?
• 1:10:43
  Bryan Cantrill: Ergo, it is not a regression. Right.
• 1:10:45
  Adam Leventhal: Yeah. I like that logic.
• 1:10:47
  Bryan Cantrill: I I love that logic. I I think whenever I I mentioned this to Robert, Robert's like, well, I'm glad that you feel that way. It's like, By the way, the system is Okay, that's great. So I think that we
• 1:11:01
  Adam Leventhal: Right now do Spectre and Meltdown.
• 1:11:02
  Bryan Cantrill: Yeah, I know. A 100%. Exactly. No, no. It's like, I'm glad you feel that way.
• 1:11:06
  Our customers may not see it the same way, obviously performance is very important as well. But it does give you, again, a release perspective, it's like, no, we're going to set this tunable to zero. And now we've got a lot of confidence in actually shipping this thing out. Andy, that's not like tuning it to zero is not the fix that we actually want. We, I think, want to eliminate this entirely or that's the fix you ended up doing.
• 1:11:35
  You then have further analysis because, okay, there is a performance issue here. And so let's go understand that. And then you did some Could you describe some of the follow-up work you did to actually look at the performance of this thing? And actually going back to some of this, all right, let's go back to this original file bench profile, and let's see how this thing actually does.
• 1:11:57
  Andy Fiddaman: Yeah. So, yeah, we obviously wanted to fix it properly. So I went through and wrote up a fix and went through all of the ZIL logic and tried to fix all of the things I thought were problems. And then I went and found the same commit in OpenZFS, and it was pretty much the same. They they'd fixed the same things as we had.
• 1:12:13
  Yeah. And a further optimization around the the ZFS volume where we used to a CAT build transaction per object. And, course, there's only one object in the ZFS volume, so you can actually get a bit more speed there if you just treat everything asynchronous when it goes into the VIL. So we did I pulled that in as well.
• 1:12:29
  Bryan Cantrill: Oh, it should. Yeah.
• 1:12:31
  Andy Fiddaman: And got that up for review in upstream Lumos, and then I did some yeah. I went and found FileBench and managed to get it to compile and managed to get it to disable ASLR and all the other things you need to do and ran some benchmarks on old and new. I was actually on a Gimlet I went for just because there was one on the bench. Yeah. And I gotta find the right word that isn't degradation, but the difference I saw.
• 1:12:56
  Bryan Cantrill: Yeah. Was
• 1:12:58
  Andy Fiddaman: No more than sort of 2% change in F Sync performance. So
• 1:13:04
  Bryan Cantrill: F Sync performance is it. On this benchmark that is is designed or this benchmark, which exhibited this problem.
• 1:13:12
  Andy Fiddaman: So Yeah. And it's actually a benchmark that's pretty close to crucible repair. It's a lot of async rights and a lot of ever syncs scattered in the mix. I didn't spend the time to actually do one that looks more like Griselbul, but we could do that. But at this point, we know it's not a massive blow to performance, and it is now correct and it wasn't before.
• 1:13:33
  So that is the route we're taking. But I think we can do some further work in the future on this lock that is highly contended. Yes. Yeah. But we'll do it properly.
• 1:13:42
  Bryan Cantrill: We'll do it properly. Yeah. Yeah. I think that there there was some other kind of approach that we wanna go take. And I think also, I mean, this is from the performance of our system.
• 1:13:50
  This is a crucible repair operation. Mhmm. That operation is itself asynchronous. Right? I mean, the the it it's like if the performance were pathological on that, that could actually be that that you could have ways that that would degradate system performance.
• 1:14:07
  But it's this is a a code path that very much needs to be correct. All of crucible needs to be correct, but this one especially so. And this is not is not, there are other paths in crucible that are the hot paths for performance.
• 1:14:24
  Alan Hanson: Yeah, and this isn't one of
• 1:14:25
  Adam Leventhal: them.
• 1:14:25
  Bryan Cantrill: This is not one of them. Is that fair to say,
• 1:14:28
  Alan Hanson: Yeah, yep.
• 1:14:33
  Bryan Cantrill: Mean, this ended up being, We went from Friday feeling Oh, I don't know. I mean, I guess you felt kind of despairing. I felt like excited that we had this thing actually reproducible. So you know, we're gonna get this. It may take us weeks, but we will get this thing this point.
• 1:14:50
  Yep. But I mean, as we've had, as we've seen so many times before, and as we saw, ironically on the bug that was happening kind of concurrently with the first round of this that we had Dave and Co on talking about, there was a baton being passed here as people are trying I mean, obviously, Justin and Matt, but then also James, you being able to reproduce it simply, Dan being able to produce it even more simply and see kind of setting it up for Andy to really be able to, again, put a corner around it and really cinch it tight. Was a lot of fun to see.
• 1:15:32
  Alan Hanson: It's a lot of fun now looking back. Yeah.
• 1:15:34
  Bryan Cantrill: It's yeah. I mean, it is I mean, on the on the one hand, it's like, God, it is it is great content. It is such great content. On the other, it's like, maybe you we don't need this much. We don't need to do that.
• 1:15:46
  We don't need Hey, gods. We don't need any I don't know.
• 1:15:50
  Alan Hanson: Some content, but maybe Yeah.
• 1:15:51
  Bryan Cantrill: Yeah. Maybe we don't. We're okay on this actually. We've kind of like I mean, people are like, Jesus Christ, another data corruption episode. How many of these things are you gonna have?
• 1:15:59
  I mean, you know, let's get But I really was It was terrific. And I think there are, again, are a bunch of, I feel, lessons here. One of them, I mean, Matt, just in terms of your half measure that I think was important. Being able to When you see a pathology like this, instead of closing your eyes and pretending that it didn't happen, which is Which first of all, you can want to do. That's okay to want that.
• 1:16:29
  Just that the you actually need to go do is really try to to actually go hit the system harder where you think it might be more more vulnerable.
• 1:16:41
  Alan Hanson: And if you don't like, in the beginning, we didn't know what it was that would be hitting it harder. So what do we need to do to get that information? Like, okay. We we can add more logging here. We can put some debug data in here.
• 1:16:55
  Like, if if you don't know what it is to do to reproduce it, then go do what you need to do to get the information that will help you, you know.
• 1:17:05
  Bryan Cantrill: Right. Exactly. To to actually go to and and the information, the tooling that you need to actually get this stuff nailed. But Matt Arons, you should know that the affidavit holds. And it continues to be very valuable to that.
• 1:17:27
  And I also very appreciative of having, even though we've kind of got different CFS implementations out there, very helpful to be able to compare notes across those implementations for sure.
• 1:17:44
  Matt Ahrens: Yeah. Sure thing. I found that email that I think you're referring to, from October 2023, which was just a few days before the fix was integrated into, OpenZFS. So I don't quite have that excuse for not, you know, code reviewing and understanding all of the ZIL before I sent Adam the affidavit, but glad it was helpful.
• 1:18:06
  Bryan Cantrill: Yeah. It it remains helpful. I I it it was extremely helpful because it was again, allowed us to make this very important transition in the system that Matt described earlier. So, it it has been very helpful. And again, terrific work all around.
• 1:18:24
  Alan Hanson: Truly a team effort. Like everybody from all directions throwing in little pieces that move sort of the whole process along.
• 1:18:35
  Bryan Cantrill: It's a very very important aspect of this. Data corruption needs to be shared suffering. It it really is is is very important. And I also when when debugging problems like this, I I think they're very paralyzable in the sense that people can try things out. I mean, when when like, for example, when Dan was trying out writing this and see, it's like, that's fine.
• 1:18:53
  Like, yeah, go down that route. If it yields something, that's great. If it doesn't yield something, that's fine because lots of other routes didn't yield anything. And then, Oh, wow, it yields something. Great.
• 1:19:01
  Okay. That kind of lurches us in that direction. Yep. Well, awesome. Thank you everybody.
• 1:19:08
  Andy, thanks for your terrific analysis in the Lumos bug. We'll link in the show notes, but 17734 is the Lumos bug that that goes into the actual details and a really terrific work on all of this. And thank you everyone. Matt Arons, thank you again the affidavit and for all of your great help from afar. And now I really need to go figure out if I'm hallucinating this DMA issue from twenty plus years ago.
• 1:19:42
  I is Silicon image. Silicon image.
• 1:19:45
  Matt Ahrens: It sounds very truthy.
• 1:19:47
  Bryan Cantrill: It's it does sound truthy. It sounds truthy, which is what makes it so plausible that I have told myself some elaborate lie. I I well, anyway, I'm not I'll go do my do my homework on that.
• 1:19:56
  Adam Leventhal: Someone said you you mentioned it in a in a talk like years and years ago.
• 1:19:59
  Bryan Cantrill: Oh, I for sure have mentioned it in talks like that. Sorry. If that's the standard for like, I definitely know I'm
• 1:20:04
  Adam Leventhal: You've not accepted yourself. Oh, yeah.
• 1:20:06
  Bryan Cantrill: I mean, for sure on that one. This is what has me concerned. It's an implanted memory because it's extreme I mean, it's a it's a it's a great yard, you know. It's like Yeah. Maybe the thing is like, I even remember whose office I was in when this thing got I mean, it's like, if this thing's implanted, it's elaborate and it's really I I I mean
• 1:20:25
  Alan Hanson: Do you really wanna know? Like, maybe maybe you don't.
• 1:20:27
  Bryan Cantrill: You know, maybe I don't. Like, what what you know, who's to say
• 1:20:30
  Adam Leventhal: Just let it be.
• 1:20:31
  Bryan Cantrill: Yeah. Let it be. It's it it could have happened, and that's the point, really. Yeah. No.
• 1:20:36
  I definitely have told this one plenty. This this true or not this truthy story, I've told plenty. It's a parable.
• 1:20:44
  Adam Leventhal: It's a parable. It's got a it's got a whole lesson to it.
• 1:20:47
  Bryan Cantrill: It's a parable. Awesome. Well, thank you very much everyone. And, yeah, thanks. I know I know what I will be giving thanks for at my Thanksgiving table.
• 1:20:57
  Correct data. Yep. There we go. Awesome. Thanks everyone.
• 1:21:02
  See you next time.