How to understand space usage of zvols on RAIDZ?

[hg2ecz]

System information

zfsonlinux 0.8.6 and 2.0.2
--- | ---
Distribution Name | Debian
Distribution Version | 10.8
Linux Kernel | 5.9.0-0.bpo.5-amd64
Architecture | x86_64
ZFS Version | 2.0.2-1 bpo10+1 and 0.8.6
SPL Version | 2.0.2-1 bpo10+1 and 0.8.6

Describe the problem you're observing

I want to create a 24TB zvol, but I can to create maximum 15TB.
zfs create -V 24TB data/datavol # out of space !!!
zfs create -V 16TB data/datavol # out of space !!!
zfs create -V 15TB data/datavol # maximum size what I can create on 24TB dataset

# zpool list # raidz (5 x 6,7TB disk)
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
data  33,4T  5,85M  33,4T        -         -     0%     0%  1.00x    ONLINE  -

# zfs list
NAME                USED  AVAIL     REFER  MOUNTPOINT
data                  24,5T  1,38T      153K  /data
data/datavol      24,5T  25,9T     89,5K  -

Describe how to reproduce the problem

So I have 25TB dataset (raidz from 5x6,7TB) and I can't create greater than 15 TB zvol.

But I make a raid5 with mdadm
mdadm -C -l5 -n5 /dev/md5 /dev/sda5 /dev/sdb5 /dev/sdc5 /dev/sdd5 /dev/sde5
zpool create data /dev/md5 # without raidz on Linux software raid5
zfs create -V 25TB data/testvol # this work fine

 zpool list
 NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
 data  26.7T   632K  26.7T        -         -     0%     0%  1.00x    ONLINE  -

 zfs list
 NAME               USED  AVAIL     REFER  MOUNTPOINT
 data             25.8T   104G       96K  /data
 data2/datavol  25.8T  25.9T       56K  -

Include any warning/errors/backtraces from the system logs

[ahrens]

In general, the following would be helpful to understand the space usage here:

before you create any volumes:
zpool status
zpool list -v
zfs list
zdb data | grep ashift

then:
zfs create -V 15TB data/datavol
zpool list -v
zfs list

All that said, this is probably working as designed, assuming 4k sectors (ashift=12). Each 8K (2-sector) block uses a single sector (4k) of parity. So 15TB of 8KB blocks (default volblocksize=8k) takes up at least 22.5TB space on disk (including parity). You will use less parity by decreasing the sector size (not practical without changing your hardware) or increasing the block size (e.g. volblocksize=32k, or the default recordsize=128k for filesystems).

[hg2ecz]

Many thanks. It's not a well-known option. :)
zfs create -V 25TB data/datavol -o volblocksize=32k # it's good (and I have +1 TB space)

 # zdb data | grep ashift
            ashift: 12
            ashift: 12

# zpool list
data      33.4T  1.40M  33.4T        -         -     0%     0%  1.00x    ONLINE  -
    raidz1   33.4T  1.40M  33.4T        -         -     0%  0.00%      -  ONLINE  

# zfs list
data                   25.2T   672G      153K  /data
data/datavol       25.2T  25.9T     89.5K  -

Size: 27 487 791 MB  # Ok