Incremental send stream requires -L (--large-block), to match previous receive

[user2718281]

I encountered something very similar to what was described in this question two years ago. However, it wasn't answered and I couldn't find an answer anywhere else either.

I'm using ZFS 2.1.11 on Debian 12. All the pools were created using this version as well. This is what's happening:

I have three pools. I can send incremental snapshots from pool1 to pool2 and pool3, as well as between pool2 and pool3. But when I try to send one to pool1, I get this message:

cannot receive incremental stream: incremental send stream requires -L (--large-block), to match previous receive.

If I send it with the suggested -L flag, I get the same message and the snapshot still isn't received. The recordsize property is 1M for the datasets in all three pools.

These are the large_blocks properties:

pool1     feature@large_blocks           active                         local
pool2     feature@large_blocks           active                         local
pool3     feature@large_blocks           enabled                        local

The only thing that comes to my mind which sets them apart is that pool1 is using 512e drives, and pool2 and pool3 are using 4Kn drives. But they're all using ashift=12 so I think that shouldn't matter.

I can provide additional info if needed.

Any help would be appreciated.

[rincebrain]

If I had to guess, you did the initial send without -L to pool3, and so it chopped up the blocks into smaller ones, so now -i is failing with -L because the problem is the reverse of what it's saying - that you're trying to change the block size of an already written file.

But you could figure that out more clearly by checking a file above 1M on the same dataset on pool2 and pool3 with zdb to see what sizes the records are.

I'm not really sure why you're assuming ashift is anywhere in the story.

[user2718281]

pool1 was created first, and then I sent the dataset to pool2 and pool3. I sent it without -L. Did I need to use -L in the initial send (non-incremental) because the dataset used recordsize=1M? I thought zfs send | zfs recv doesn't need any additional options if the sending/receiving datasets use the same recordsize.

Could you give me an example of how to check it with zdb?

Thanks!

[rincebrain]

Really, I suppose just stat could tell you, probably.

$ stat /test/CentOS-Stream-8-x86_64-20210603-boot.iso
  File: /test/CentOS-Stream-8-x86_64-20210603-boot.iso
  Size: 759169024       Blocks: 1459666    IO Block: 131072 regular file
Device: aeh/174d        Inode: 589834      Links: 1

Where it says "IO Block" there is going to be the recordsize of the particular file.

And no, if you don't use -L, then zfs send will chop up any records over 128k into 128k-sized chunks on send, in order to let you keep using zfs send when the destination didn't support those.

So you'd probably want to resend everything having this problem from pool1 to pool2 and pool3 with zfs send -L, along with any other flags you were using, if that wasn't your goal.