incorrect/corrupted data after receiving a snapshot, but fine after remounting

[digitalsignalperson]

I just had this strange occurrence while testing a new replication setup. Both systems are arch linux with zfs 2.2.3. The send/receive was of the form ssh root@x zfs send -p -I @snap1 xpool/xxxx@snap2 | zfs recv xpool/xxxx

After receiving an incremental snapshot, comparing the md5sum of a file on both the source and destination system showed they were different. It was a text file so I used diff to compare the files. On the system that received the incremental snapshot, the file contents did not include the changes from the latest snapshot, and the end of the file contained a bunch of null bytes. The diff output showed this as ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

Then I unmounted and remounted the dataset in question, and now the files were equivalent.

So what happened? I haven't been able to reproduce this, as normally after receiving the incremental snapshot, both sides have the exact same data and I can confirm (e.g. md5sum). It seems like something strange happened where ZFS did not properly mount the data from the incremental snapshot, yet zfs mount shows the dataset is mounted.

I wish I had tried: was the dataset still writable? Would I have corrupted something if I had tried writing?

I'm not sure if this is enough info for a bug report (can't reproduce). Anyone know any similar issues? Or what to look out for, or anything I can do post-hoc to investigate this? I'm also wondering how can I detect this next time, other than having some process to manually check the hashes of each file changed in the incremental snapshot.

One thing that is interesting about zfs recv, is if I have a file open and cannot do zfs umount xpool/xxxx ("cannot unmount '/mnt/xxxx': pool or dataset is busy"), I can still receive an incremental snapshot to this dataset and all the data is updated seamlessly. Except for this one time it seemed to get mangled. zpool history -il shows each recv having a sequence like

receive xpool/xxxx/%recv (33755)
finish receiving xpool/xxxx/%recv (33755) snap=tmp_2024.03.13.12.27.06
clone swap xpool/xxxx/%recv (33755) parent=0029
snapshot xpool/xxxx@tmp_2024.03.13.12.27.06 (32977)
destroy xpool/xxxx/%recv (33755) (bptree, mintxg=43740)

where the clone swap thing I think is where that seamless update would be.

[rincebrain]

I believe the fundamental problem at hand is that there's not really a great way to tell everybody that their concept of the underlying file is no longer valid when playing games like this.

"In theory", the way this happens is that you do a new mount in place of the old one, anyone walking things by path gets the new one, old fd holders see their old version until they stop, whatever.

But that turns out to break a lot of people's expectations, as mounting over a live mountpoint usually does. So we play games where we pretend the old references are all still valid, but then you might get different results for what you just read 5 seconds ago versus now, or your OS's cached version of a file might no longer be correct.

Saying this makes me wonder if on Linux, triggering -o remount whenever we do this might minimize how bad this can be, but I don't know if remount can trigger EBUSY, it's just a thought.