ZFS corruption related to snapshots post-2.0.x upgrade #12014
Comments
jgoerzen
added
Status: Triage Needed
|
Two other interesting tidbits... When I do the reboot after this issue occurs, the mounting of the individual zfs datasets is S L O W. Several seconds each, and that normally just flies by. After scrubbing, it is back to normal speed of mounting. The datasets that have snapshot issues vary with each one. Sometimes it's just one, sometimes many. But var is almost always included. (Though its parent, which has almost no activity ever, also is from time to time, so that's odd.) |
|
Same symptoms here, more or less. See also issue #11688. |
|
I also have the symptom with the corrupted snapshots, without kernel panics so far. So far it only affected my Debian system with Linux 5.10 and zfs 2.0.3 (I've turned the server off for today, I can check the exact versions tomorrow). Also, while the system has the 2.0.3 zfs utils + module, the pool is still left on 0.8.6 format. I wasn't able to execute On the corrupted system, after I got the mail from ZED, I manually ran a scrub at first, after which the I've rebooted the server into an Ubuntu 20.10 live with zfs 0.8.4-1ubuntu11 The errors didn't seem to affect the data on the zvols (all 4 affected snapshots are of zvols). The zvols are used as disks for VMs with ext4 on them. I have two other Ubuntu 21.04 based Systems with zfs-2.0.2-1ubuntu5 which are not affected until now. However, they have their pools already upgraded to 2. All are snapshotted with sanoid and have the datasets encrypted.
EDIT:
EDIT 2:
|
|
I'm seeing this too, on Ubuntu 21.04, also using zfs encryption I have znapzend running, and it makes a lot of snapshots. Sometimes, some of them are bad, and can't be used (for example, attempting to send them to a replica destination fails). I now use the In the most recent case (this morning) I had something like 4300 errors (many more than I'd seen previously). There are no block-level errors (read/write/cksum). They're cleared after destroying the affected snapshots and scrubbing (and maybe a reboot, depending on .. day?) Warning! Speculation below:
|
|
@jgoerzen Can you
In my case, #11688 (which you already reference), I've discovered that rebooting "heals" the snapshot -- at least using the patchset I mentioned there |
|
I'll be glad to. Unfortunately, I rebooted the machine yesterday, so I expect it will be about a week before the problem recurs. It is interesting to see the discussion today in #11688. The unique factor about the machine that doesn't work for me is that I have encryption enabled. It wouldn't surprise me to see the same thing here, but I will of course wait for it to recur and let you know. |
|
Hello @aerusso, The problem recurred over the weekend and I noticed it this morning. Unfortunately, the incident that caused it had already expired out of the It should be noted that my hourly snap/send stuff runs at 17 minutes past the hour, so that may explain this timestamp correlation. zpool status reported: Unfortunately I forgot to attempt to do a So I think that answers the question. After a reboot but before a scrub, the |
|
I have similar symptoms, on an encrypted single-ssd ubuntu 21.04 boot pool, using stock zfs from ubuntu's repos. Deleting the affected snapshots and scrubbing previously cleared the errors, but on reoccurence, repeated scrubbing (without deleting them) caused a deadlock. My system has ECC memory, so it's probably not RAM related.
|
|
@cbreak-black Was there a system restart between the occurrence of the corrupted snapshot and the problems? Restarting has "fixed" this symptom for me (though you will need to scrub twice for the message to disappear, I believe). I have a suspicion that this may be a version of #10737 , which has an MR under way there. The behavior I am experiencing could be explained by that bug (syncoid starts many I'm holding off on trying to bisect this issue (at least) until testing that MR. (And all the above is conjecture!) |
|
@aerusso No, without a restart I got into the scrub-hang, and had to restart hard. Afterwards, the scrub finished, and several of the errors vanished. The rest of the errors vanished after deleting the snapshots and scrubbing again. |
|
Can I join the club too? #10019 |
|
@InsanePrawn I can't seem to find commit 4d5b4a33d in any repository I know of (and neither can github, apparently, either). However, in your report you say this was a "recent git master" and the commit I'm currently betting on being guilty is da92d5c, which was committed in November of the previous year, so I can't use your data point to rule out my theory! Also, it sounds like you didn't have any good way to reproduce the error --- however, you were using a test pool. Compared to my reproduction strategy (which is just, turn my computer on and browse the web, check mail, etc.) it might be easier to narrow in on a test case (or might have been easier a year and a half ago, when this was all fresh). Anyway, if you have any scripts or ideas of what you were doing that caused this besides "snapshots being created and deleted every couple minutes", it might be useful too. (I already tried lots of snapshot creations and deletions during fio on several datasets in a VM). |
|
Yeah, idk why I didn't go look for the commit in my issue - luckily for us, that server (and pool; it does say yolo, but it's my private server's root pool. it's just that i won't cry much if it breaks; originally due to then unreleased crypto) and the git repo on it still exist. Looks like 4d5b4a33d was two systemd-generator commits by me after 610eec4 |
|
FWIW the dataset the issue appeared on was an empty filesystem (maybe a single small file inside) dataset that had snapshots (without actual fs activity) taken in quick intervals (somewhere between 30s and 5m intervals) in parallel with a few (5-15) other similarly empty datasets. The pool is a raidz2 on 3.5" spinning SATA disks. Edit: Turns out the dataset also still exists, the defective snapshot however does not anymore. I doubt that's helpful? |
|
@InsanePrawn Does running the zrepl workload reproduce the bug on 2.0.5 (or another recent release?) I don't think the snapshot is terribly important --- unless you're able to really dig into it with zdb (which I have not developed sufficient expertise to do). Rather, I think it's the workload, hardware setup, and (possibly, but I don't understand the mechanism at all) the dataset itself. Encryption also is a common theme, but that might just affect the presentation (i.e., there's no MAC to fail in the unencrypted, unauthenticated, case). Getting at |
|
I've since added redundancy to my pool (it's now a mirror with two devices), and disabled autotrim. The snapshot corruption still happens. Still don't know what is causing it. And I also don't know if the corruption happens when creating the snapshot, and only later gets discovered (when I try to zfs send the snapshots), or if snapshots get corrupted some time in between creation and sending. |
|
@cbreak-black Can you enable the all-debug.sh ZEDlet, and put the temporary directory somewhere permanent (i.e., not the default of This will get the output of I'll repeat this here: if anyone gets me a reliable reproducer on a new pool, I have no doubt we'll be able to solve this in short order. |
|
Just mentioning here that we saw this on TrueNAS 12.0-U5 with OpenZFS 2.0.5 as well -- see #11688 (comment) for our story. |
|
Since I don't see anyone mentioning it here yet, #11679 contains a number of stories about the ARC getting confused when encryption is involved and, in a very similar looking illumos bug linked from there, eating data at least once. |
|
@gamanakis Nope, I'm not using raw (-w). |
|
it's present in v2.1.1 as well: |
|
@aerusso you wrote that da92d5c may be the cause of this issue. My workstation at work panics after a couple of days and I need to reset it. Could you provide a branch of 2.1.1 with this commit reverted (as revert causes merge conflicts I can't fix myself) so I could test if the machine no longer crashes? |
|
@phreaker0 Unfortunately, the bug that da92d5c introduced (#10737) was fixed by #12299, which I believe is present in all maintained branches now. It does not fix #11688, (which I suspect is the same as this bug). I'm currently running 0.8.6 on Linux 5.4.y, and am hoping to wait out this bug (I don't have a lot of time right now, or for the foreseeable future). But, If you have a reliable reproducer (or a whole lot of time) you could bisect while running 5.4 (or some other pre-5.10 kernel). I can help anyone who wants to do that. If we can find the guilty commit, I have no doubt this can be resolved. |
|
Are you still seeing any errors with scrub tripping assertions, or is this "just" the snapshots reporting errors? If the snapshots are always from things that were sent+received, could you possibly look at which snapshots this happened with, and then we can try to see what differs on both sides? |
|
For us its always the sending side that's (falsely) reporting the corruption, there is no way to compare anything as the snapshots aren't even sent out. Rebooting the system allows sending the snapshots fine again, so I guess its some sort of corruption that happens in memory only and isn't an actual corruption of the underlying data. Scrubbing without rebooting does not solve the issue either. Scrubs don't destroy or corrupt anything further though, its just that the only thing fixing this is a reboot. |
The mentioned issue never causes scrub errors as corruption happens during read or omitted read specifically. |
|
@rincebrain I still have the issue with zfs 2.2.2 The "corrupted" snaphost will be reported/triggered during a "zfs send" the log will display something like: A reboot will make the snapshot accessible again. Doing twice a scrub will clear the error reported by zpool status. I noticed that syncoid is doing a I have added a |
|
As my colleague @tjikkun hoped unfortunately #15526 does not resolve the issue. The 'corrupt' snapshot indeed never gets send after the corrupt snapshots starting to exist. Don't know if the sending is related to the cause of this issue. Although ZFS starts to notice the corruption during a ZFS send attempt. Errors during ZFS send attempts of the corrupt snapshot: |
Unfortunately, the above attempt failed, I had the bug happening his morning. I had a look at /proc/spl/kstat/zfs/dbgmsg, and among lot of lines, I found the following, starting to be reported at the time of the "zfs send" operation:
|
|
On the same staging server the issue came back again after 9 days. We are unable to directly reproduce the behaviour but if we leave the system running with a ZFS version higher than 0.8.6 it occurs with encrypted snapshots after a few days or weeks. We are unable to read all the data from the local corrupted snaphot as ZFS is unable to mount it: The following logs in After a while we are again able to read the data in the snapshot content. But nothing particular seems to be logged in The behaviour that a snapshot gets noticed as corrupted as far as we can pinpoint happens during or somewhere arround a ZFS send to external backup server. Rebooting the server and performing two scrubs again fixes the errors. I added additional journal and |
|
Please rename this issue to include the term "encryption" |
|
A small note FWIW, I have a system (laptop) that used to produce this issue reasonably regularly (noted somewhere way above). The issue persisted through changes of OS (ubuntu -> nixos), obviously changes of zfs version over time, and a change of the internal SSD (but not the pool, the swap was a mirror and detach). Anyway, none of this is conclusive, and of course I'm tempting fate by posting this, but I think the change that made the difference was of replication target. I use znapzend to take and send snapshots, to two destinations. In the original setup, one of the destinations was a remote server, the other was a second pool on USB disks (with LUKS) that I would attach from time to time. I reconfigured that a while ago to use two different remote servers, and around the same time switched to using raw sends. I'm pretty sure I haven't seen the problem since. |
|
Man, I wish all of you can make confirmation that problem only occurs with encrypted pool (native or LUKS). Little bit nervous for my system |
|
I've been running 2.2.6 on a machine that was exhibiting the snapshot corruption issue for just over a week now, and haven't had a reoccurrence yet, but too early to be certain. |
|
Message ID: ***@***.***>
I think I've probably mentioned this before. I experience "permanent
errors" when I do full pool (recursive) backups of my laptop which is
running Debian bookworm with root on ZFS (using rpool/bpool split.) Over
several years *all* permanent errors have "rolled off" as older snapshots
are cleared up. I leave full pool backups disabled and try them once in a
while to see if the problem persists. Several months ago it did.
I perform backups of my home filesystem (excluding things like ~/Downloads)
and have not seen any corruption.
I have a desktop that performs full pool backups to a locally connected HDD
and have never seen corruption on that.
At present my laptop is running 2.2.5 (with pending update to 2.2.6 on next
reboot) on Debian Stable and I use syncoid for backing up.
***@***.***:~$ zfs --version
zfs-2.2.6-1~bpo12+1
zfs-kmod-2.2.5-1~bpo12+1
***@***.***:~$
I am not nervous but I also have multiple backups including off-site.
…--
Beautiful Sunny Winfield
|
|
I can confirm that 2.2.6 does not fix this snapshot corruption for me (or indeed the kernel panics and hanging zfs processes on the receiving end). |
|
@scratchings I want to make sure your errors weren't caused by RAM bitflips. If you disassemble your computer, remove dust, and assemble it again, does the issue disappear? A lot of dust tends to cause errors in random computer parts. Sometimes, dust causes errors in RAM or GPU. |
|
Also can confirm that with ZFS 2.2.6 the problem unfortunately still exists. On a staging system an encrypted snapshot showed up as 'corrupt' after the snapshot was created. Although rebooting the server makes the data in the snapshot available again. We are still unable to reproduce it. Although writing more data to the filesystem maybe seems to help in the issue to arise. Errors during ZFS send attempts of the corrupt snapshot: You are unable to access the 'corrupt' data inside the snapshot after it happens. All the other snapshots before and after are fine. After a reboot however I can access the data in the snapshot again. So the data isn't actually corrupt but fine. Performing two scrubs after the reboot also makes the error go away and makes So it is a really annoying bug as ZFS sees data as corrupt even as it actually isn't. Just to reiterate: ZFS 0.8.6 doesn't have this issue. |
|
@J0riz If you remove dust from the computer case, does the issue go away? Dust causes electrostatic shock which flips bits in RAM. Random RAM errors can cause software errors. After thoroughly cleaning my computer case, it seems I stopped seeing various errors. |
|
@amano-kenji This may have been the root cause of an issue you were having, but it is unlikely that RAM errors would cause this problem in 2.x, but not cause them in 0.8.y. Also, it sounds like you may have been experiencing multiple issues (and I'd expect that people reporting here aren't experiencing a cluster of weird/buggy behavior). I do recommend everyone seeing these problems run memtest to rule out RAM issues, though. That's far more reliable than just cleaning out dust (though it won't help allergies! :-) . |
|
Although we are getting off track: All our (staging) systems are running in a TIer 3+ datacenter. All air is filtered and I never saw dust in our datacenter. Just to reiterate: ZFS 0.8.6 doesn't have this issue with encrypted snapshots showing up with 'permanent errors'. Prompt: Please go back on topic. |
|
I just want to follow up that I USED to have these issues, mainly brought on from bring an older pool to a newer ZFS where some mess was created because something in 2 didn't like something from an earlier version. I usually fixed my issue by either purging the bad snapshot that triggered the corruption using the various ZFS kernel flags that suppress errors, and performing a scrub, or simply copying the data out if I can't purge the bad snapshot, and then purging and recreating the impacted dataset. It has not resulted in and data corruption/loss for me, and the ZFS pool has been humming along without issue since. So my suspicion is that users still impacted are coming in from older versions of ZFS, and rather than devs trying to figure out every nuance for everyone, which is probably almost impossible, it's probably just easier to recreate the faulty dataset. For clarity, I'm using encrypted datasets |
I've been running ZFS with native encryption on a laptop I purchased about 5 years ago. I'm not positive about the version of ZFS I started with but it might have been 0.7.x. At some point I began seeing these "permanent errors" in snapshots and suspected that the NVME SSD was beginning to fail. I purchased a Samsung 980 PRO to replace it, thinking that would likely be the most reliable device I could purchase at the time. The issues continued with the replacement. I found that if I disabled the full pool backups I had been running, the errors eventually went away as the snapshots were deleted. I have not seen this issue with the "normal" backups that just include my user files. I've re-enabled the full pool backup occasionally to see if the problem persists. It has and when found, I disable that backup and the "permanent errors" eventually go away. On 2024-09-23 I reinstated the full pool backups to see if "permanent errors" returned. There is now a single permanent error in the pool. The last time I did this, the pool reported over 100 errors before I disabled this backup. I'm cautiously optimistic that the fixes reported in 2.2.5, 2.2.6 have reduced but not fully eliminated the error. Creation information for the pool is |
|
I'm also running 2.2.6 and still have the same issues with my encrypted pools on several servers. It typically happens after the server is running for a couple of days and then errors will shown on the backup replication of my ssd root pool to my hdd storage pool. This is the way I do my hourly replication currently: The script for destroying the metadata errors in the affected snapshots (if those aren't cleared the zfs replication won't work because of I/O errors): If the replication fails I will get the permanents errors. An erroneous replications looks like this: Running two scrubs will clear the existing errors (if the affected snapshots were destroyed before). But once errors starts showing the will only increase from here and only a reboot will make sure that errors won't show for some time. So far I don't experienced any "real" data errors beside losing the affected snapshots.
I think I already tried that, but I probably will do it again to check if this may work. |
|
My suspicion would be that something is unsafely using some piece of metadata in the encrypted dataset being sent, at the same time as something else goes to use it, and you're getting a spurious decrypt/decompress/checksum error from that inconsistent state, and then it goes away on subsequent recheck. But that's just a guess, it's not like I have one locally that reproduces it. It'd be useful to look at the zpool events messages to see exactly what object produced the error so we can inspect and try to reproduce it, probably. |
It's actually pretty well established when these issues first appeared, which makes their longevity even more puzzling/concerning. Native encryption was rolled out with v0.8.0 (May 2019 - 0.8.x is probably what you started on) and the corruption issues first appeared in 2.0.0 (Late 2020). They've gotten better over time, but it's still been years that these issues have persisted in some form or other. |
|
It's nobody's responsibility to fix it. There's not like, assigned areas of the project where it's so-and-so's job to fix this if it breaks, and the original company that contributed the project A) doesn't seem to hit these and B) appears to have stopped contributing some time ago. The rest of the companies that use OpenZFS, to my knowledge, mostly don't use native encryption, so that leaves random volunteers or any exceptions, to fix these. That's not intended as an indictment of anyone involved, it's just a statement of "nobody's responsible for making sure it gets done, so it doesn't get done if nobody is responsible for it and not enough people are in the set of {able, motivated by encountering it} to have it happen organically" |
and others
System information
Describe the problem you're observing
Since upgrading to 2.0.x and enabling crypto, every week or so, I start to have issues with my zfs send/receive-based backups. Upon investigating, I will see output like this:
Of note, the
<0xeb51>is sometimes a snapshot name; if Izfs destroythe snapshot, it is replaced by this tag.Bug #11688 implies that zfs destroy on the snapshot and then a scrub will fix it. For me, it did not. If I run a scrub without rebooting after seeing this kind of
zpool statusoutput, I get the following in very short order, and the scrub (and eventually much of the system) hangs:However I want to stress that this backtrace is not the original cause of the problem, and it only appears if I do a scrub without first rebooting.
After that panic, the scrub stalled -- and a second error appeared:
I have found the solution to this issue is to reboot into single-user mode and run a scrub. Sometimes it takes several scrubs, maybe even with some reboots in between, but eventually it will clear up the issue. If I reboot before scrubbing, I do not get the panic or the hung scrub.
I run this same version of ZoL on two other machines, one of which runs this same kernel version. What is unique about this machine?
I made a significant effort to rule out hardware issues, including running several memory tests and the built-in Dell diagnostics. I believe I have rules that out.
Describe how to reproduce the problem
I can't at will. I have to wait for a spell.
Include any warning/errors/backtraces from the system logs
See above
Potentially related bugs
arc_buf_destroyis in silent corruption for thousands files gives input/output error but cannot be detected with scrub - at least for openzfs 2.0.0 #11443. The behavior described there has some parallels to what I observe. I am uncertain from the discussion what that means for this.The text was updated successfully, but these errors were encountered: