Incorrect space accounting with ZFS?

[alpha754293]

I am trying to do a really simple experiment with ZFS where I create a file using the command dd if=/dev/zero of=disk.1 bs=4096 count=16384 so that it would create the requiste 64 MiB minimum size for the device to be used with ZFS.

ls -l
total 262144
-rw-r--r-- 1 proxmox proxmox 67108864 Oct 10 18:20 disk.1

So that is the correct size.

And then I create a simple test ZFS pool test, using the command:

zpool create test /home/proxmox/disk.1

That was successful.

zpool status
  pool: test
 state: ONLINE
config:

        NAME                    STATE     READ WRITE CKSUM
        test                    ONLINE       0     0     0
          /home/proxmox/disk.1  ONLINE       0     0     0

errors: No known data errors

zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
test    48M   105K  47.9M        -         -     5%     0%  1.00x    ONLINE  -

zfs list
NAME   USED  AVAIL     REFER  MOUNTPOINT
test   105K  23.9M       24K  /test

Why does the SIZE and AVAIL differ by almost 50%?

And then why does the SIZE differ from the actual size by almost 25%? (i.e. it should show 64 MiB, but instead, shows it as being 48 MiB)

Does the metadata really take up 16 MiB on a 64 MiB file that is used as the vdev?

zpool -p get all
NAME  PROPERTY                       VALUE                          SOURCE
test  size                           50331648                       -
test  capacity                       0                              -
test  altroot                        -                              default
test  health                         ONLINE                         -
test  guid                           9790107242169098281            -
test  version                        -                              default
test  bootfs                         -                              default
test  delegation                     on                             default
test  autoreplace                    off                            default
test  cachefile                      -                              default
test  failmode                       wait                           default
test  listsnapshots                  off                            default
test  autoexpand                     off                            default
test  dedupratio                     1.00                           -
test  free                           50224128                       -
test  allocated                      107520                         -
test  readonly                       off                            -
test  ashift                         0                              default
test  comment                        -                              default
test  expandsize                     -                              -
test  freeing                        0                              -
test  fragmentation                  5                              -
test  leaked                         0                              -
test  multihost                      off                            default
test  checkpoint                     -                              -
test  load_guid                      7583279236417683052            -
test  autotrim                       off                            default
test  compatibility                  off                            default
test  feature@async_destroy          enabled                        local
test  feature@empty_bpobj            enabled                        local
test  feature@lz4_compress           active                         local
test  feature@multi_vdev_crash_dump  enabled                        local
test  feature@spacemap_histogram     active                         local
test  feature@enabled_txg            active                         local
test  feature@hole_birth             active                         local
test  feature@extensible_dataset     active                         local
test  feature@embedded_data          active                         local
test  feature@bookmarks              enabled                        local
test  feature@filesystem_limits      enabled                        local
test  feature@large_blocks           enabled                        local
test  feature@large_dnode            enabled                        local
test  feature@sha512                 enabled                        local
test  feature@skein                  enabled                        local
test  feature@edonr                  enabled                        local
test  feature@userobj_accounting     active                         local
test  feature@encryption             enabled                        local
test  feature@project_quota          active                         local
test  feature@device_removal         enabled                        local
test  feature@obsolete_counts        enabled                        local
test  feature@zpool_checkpoint       enabled                        local
test  feature@spacemap_v2            active                         local
test  feature@allocation_classes     enabled                        local
test  feature@resilver_defer         enabled                        local
test  feature@bookmark_v2            enabled                        local
test  feature@redaction_bookmarks    enabled                        local
test  feature@redacted_datasets      enabled                        local
test  feature@bookmark_written       enabled                        local
test  feature@log_spacemap           active                         local
test  feature@livelist               enabled                        local
test  feature@device_rebuild         enabled                        local
test  feature@zstd_compress          enabled                        local
test  feature@draid                  enabled                        local

zfs get -p all
NAME  PROPERTY              VALUE                  SOURCE
test  type                  filesystem             -
test  creation              1728598852             -
test  used                  107520                 -
test  available             25058304               -
test  referenced            24576                  -
test  compressratio         1.00                   -
test  mounted               yes                    -
test  quota                 0                      default
test  reservation           0                      default
test  recordsize            131072                 default
test  mountpoint            /test                  default
test  sharenfs              off                    default
test  checksum              on                     default
test  compression           off                    default
test  atime                 on                     default
test  devices               on                     default
test  exec                  on                     default
test  setuid                on                     default
test  readonly              off                    default
test  zoned                 off                    default
test  snapdir               hidden                 default
test  aclmode               discard                default
test  aclinherit            restricted             default
test  createtxg             1                      -
test  canmount              on                     default
test  xattr                 on                     default
test  copies                1                      default
test  version               5                      -
test  utf8only              off                    -
test  normalization         none                   -
test  casesensitivity       sensitive              -
test  vscan                 off                    default
test  nbmand                off                    default
test  sharesmb              off                    default
test  refquota              0                      default
test  refreservation        0                      default
test  guid                  11570211743856140487   -
test  primarycache          all                    default
test  secondarycache        all                    default
test  usedbysnapshots       0                      -
test  usedbydataset         24576                  -
test  usedbychildren        82944                  -
test  usedbyrefreservation  0                      -
test  logbias               latency                default
test  objsetid              54                     -
test  dedup                 off                    default
test  mlslabel              none                   default
test  sync                  standard               default
test  dnodesize             legacy                 default
test  refcompressratio      1.00                   -
test  written               24576                  -
test  logicalused           39936                  -
test  logicalreferenced     12288                  -
test  volmode               default                default
test  filesystem_limit      18446744073709551615   default
test  snapshot_limit        18446744073709551615   default
test  filesystem_count      18446744073709551615   default
test  snapshot_count        18446744073709551615   default
test  snapdev               hidden                 default
test  acltype               off                    default
test  context               none                   default
test  fscontext             none                   default
test  defcontext            none                   default
test  rootcontext           none                   default
test  relatime              off                    default
test  redundant_metadata    all                    default
test  overlay               on                     default
test  encryption            off                    default
test  keylocation           none                   default
test  keyformat             none                   default
test  pbkdf2iters           0                      default
test  special_small_blocks  0                      default

Based on what I can see here, I am struggling to understand why the SIZE and AVAIL is so far off from where I expected it to be.

Can someone please educate me about why I am seeing the different sizes that I am seeing, and how to interpret these results?

Your help in educating me is greatly appreciated.

Thank you.

edit This is with ZFS version 2.1.15. I've deployed this via Proxmox 7.4-19 as a Proxmox VM, mostly because I was able to deploy this really, really quickly, to test this. This isn't going to be used for anything other than this simple test.

zfs --version
zfs-2.1.15-pve1
zfs-kmod-2.1.15-pve1

[thulle]

For one I know that zfs reserves some space for itself so it can still continue working if it runs out of space. You can search for old issues about not being able to remove files after filling the pool for example, where there was no space to write the new transaction with the file removal or something like that.

[alpha754293]

Thank you.

Yeah - I read about that in the ZFS Administration Guide, but a quick, cursory check on that via the zpool get -p all and zfs get -p all commands show that the current reservations are all zero.

I am also aware of the general, best practices recommendation not to fill the pool > 80% (although I've filled it ~90% before but I didn't very too much further into that. At 90%, it's starting to REALLY push the limits, until this temporary excusion > 80% was eliminated (as I was shuffling data around).

But for an empty, brand new pool, to reserve this much space for itself (if that's what it is doing) -- seems quite excessive, as a percentage of the total space.

[IvanVolosyuk]

There can be fixed size reserved. You can create zpool backed on sparse file, just 'truncate -s 1T disk.2' and create zpool there. I bet you will get more reasonable percent of metadata :)

…

On Fri, Oct 11, 2024, 11:00 alpha754293 ***@***.***> wrote: Thank you. Yeah - I read about that in the ZFS Administration Guide <https://docs.oracle.com/cd/E19120-01/open.solaris/817-2271/gbchp/index.html>, but a quick, cursory check on that via the zpool get -p all and zfs get -p all commands show that the current reservations are all zero. I am also aware of the general, best practices recommendation not to fill the pool > 80% (although I've filled it ~90% before but I didn't very too much further into that. At 90%, it's starting to REALLY push the limits, until this temporary excusion > 80% was eliminated (as I was shuffling data around). But for an empty, brand new pool, to reserve this much space for itself (if that's what it is doing) -- seems quite excessive, as a percentage of the total space. — Reply to this email directly, view it on GitHub <#16634 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXQ6HM5XIOQRZ5Y3RVQCOLZ24ILTAVCNFSM6AAAAABPXYMZY2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTAOJQHE3DEMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[alpha754293]

Thank you for your help.

But that still doesn't really answer why I am "missing" 16M (from a 64M "pool"), right off the bat.

My thinking is that if I create a 64M file, and I am using it as a "stripe" vdev (single drive/file vdev), then I would expect that the size to be reported by zpool list to be the 64M that I gave it, via said 64M file.

Unfortunately, that's not what the results show.

So there must be something wrong in terms of my thinking about the theory of operation.

The fact that there is 16M "missing" -- this is the ZFS/math experiment that I am trying to learn more about.

The pool isn't going to be used for actual storing of any data.

I only created the Proxmox VM with 4 GB of disk space because that's enough for the 3.4 GB it takes, for the Proxmox installation.

So, given it 1T won't really be an option.

I am trying to learn more about how the math in ZFS works.

Your help in this endeavour is greatly appreciated.

[no-usernames-left]

4 GB and 64 MB are worlds apart, but as mentioned 64 MB is absolutely unrealistic.

Create a 4 GB pool and a 1 TB pool and see what you get for both.

[IvanVolosyuk]

You can still create 1T sparse file on 4GB disk and run 'zpool create' on it. Sparse file will not allocate that much space on your original filesystem.

# truncate -s 1T disk
# zpool create temp disk
cannot use '/dev/disk': must be a block device or regular file
# zpool create temp $PWD/disk
# zpool list temp
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
temp  1016G   104K  1016G        -         -     0%     0%  1.00x    ONLINE  -
# zpool export temp
# ls -l disk
-rw-r--r-- 1 root root 1099511627776 Oct 14 16:12 disk
# du -hs disk
225K    disk

So, 1T sparse file with zpool uses just 225K on my system with ZFS. In your case it may be slightly larger. I once created a 100 petabytes disk for virtual windows when doing experiments and I obviously don't have that much storage in my system. That's the power of sparse files.

[alpha754293]

Ahh...okay. Thank you.

But if you created the 1 TiB sparse file, shouldn't you expect the SIZE field from zpool list to show 1T?

Instead, it only shows 1016G, which means that you're still losing 8G, right off the bat.

Where did that 8G go?

If I create a file that is 1024G (e.g. 1T), when I ask/check it via zpool list, I would expect that the SIZE field to report back 1T. But it doesn't.

Why? This is what I don't understand.

[alpha754293]

4 GB and 64 MB are worlds apart, but as mentioned 64 MB is absolutely unrealistic.

Create a 4 GB pool and a 1 TB pool and see what you get for both.

The size of the pool isn't the issue here.

The fact that I start off with a number, and expect the same number to be returned when I ask it "how big is my pool that I just created with a file of x size?" should return "x size" back to me.

Whether it's 4G or 1T, the starting size doesn't matter.

The fact that the starting size does not equal the SIZE as reported by zpool list -- to me, that's the problem.

The output doesn't match my E(X).

If I start off with 67,108,864 apples, and I say "put it into a great big bin", when I check how many apples are in the bin, I should expect 67,108,864. But that's not the case here.

I dump 67,108,864 applies into the "bin" and when I check how many apples are in the bin, the bin reports back 50,331,648 apples.

Somehow, I LOST 16,777,216 apples between what I started off with, and after dumping that them into the bin.

How is that possible?

That doesn't make any sense to me.

Again, it doesn't matter I'm starting with 67,108,864 apples or whether I'm starting with 4,294,967,296 apples or whether I'm starting with 1,099,511,627,776. When I check how many apples are in the bin, I should be getting what I started with, as the answer.

That's not what's happening here, and I don't really understand why.

Your help in trying to explain why "apples" are missing (why SIZE from zpool list < filesize) is greatly appreciated.

[robn]

You are indeed seeing a reserve, known internally as "slop". This is space that OpenZFS hangs on to to make sure it always has some working space. The detail of the math is in comments in the code but the tl;dr is:

• ~3.2% of the pool's total capacity
• maximum 128GiB
• minimum 128MiB
• but if the pool is < 128MiB, then it's half the pool

So you're seeing that last case: the ~48M pool is < ~128M, so ~24M is reserved for the slop.

[alpha754293]

I see the math and I would agree with the math IF and ONLY IF the pool is 48M, then sure -- I can see and understand that the result for zfs list would end up being 24M.

But that's not the case here.

I created a 64 MiB file (which is the minimum required for a vdev).

So, shouldn't the size that is reported by zpool list be 64M and then therefore; per the math above, 64M pool size < 128M, and therefore; 1/2 of the space would be reserved for the slop which would be 32M.

So, shouldn't the correct math (for a 64 MiB vdev)/result for zpool status -- it should show the size is 64M.

And then the AVAIL size for zfs list should be 32M, no?

This is the part that I don't understand -- why, with a 64M file, that the SIZE output for zpool list -- why that isn't 64M?

64M - 48M = 16M.

I have no idea where that 16 MiB went.

The math above describes the math for 64M - 16M (which is "missing") = 48M --> and then divided by 2, for the slop.

But that doesn't explain where that 16M went in the first place.

I appreciate you educating me about this.

I am looking to learn more about where the "missing" 16M went.

Thank you.