Do I still need 20% free space to keep performance?

[Reset816]

There are lots of discussions before but I don't know if they are still applicable for the newest version of zfs.
I just care about random read performance. Really need some advice.

[Reset816]

I am trying to upgrade my school's open source software mirror site, which doesn't care write performance.
Don't have enough money to buy more disk and ram.

[rincebrain]

There are various cutoffs where ZFS switches to different strategies to try and minimize performance loss when the ratio of available space is low enough - other filesystems are not really any different, they just don't advertise it as much.

AFAIK the cutoff for ZFS right now where it actually deliberately does something different is something like 96% full - or, more specifically, a 96% full spacemap will result in going from first-fit to best-fit allocation to try and minimize the need to use gang blocks if it can't fit things "correctly".

(Well, it also prefers other metaslabs at 70% free space fragmentation or more, and won't allocate at all if the metaslab groups are at 95% or more until they all are, but the above is the one I suspect most people might notice as a more subtle effect.)

What kind of performance numbers are you looking to maintain, with what storage configuration?

[Reset816]

Now I got three 10TB HHD and three 18TB HHD and 110G RAM. I want to reach 10GB/s read, which is the maximum bandwidth of my network. I know that's seems impossible, but I still want to reach as much performance as I can.
One possible configuration is raidz of three 10TB HHD and raidz of three 18TB HHD.
And another possible configuration is btrfs 10 of these 6 disks.
It is hard to make a decision for me and I have stuck here for days, looking up many references.
Maybe the ZFS ARC is much better than LRU in this mirror site scenario.

[rincebrain]

I mean, ZFS can do striped mirrors, it's not like it only supports stripes or parity raid.

10 GB/s is more throughput than all of your disks can do together - I'd estimate them as maybe totaling 250 MB/s * 6 = 1500 MB/s max, maybe, in peak conditions? And that's not a ZFS or btrfs or w/e limit, that's just raw reading/writing the disks.

Is your networking actually 10 GB/s, or 10 Gb/s? Because the latter is both far more common and far easier to achieve.

If your workload is highly cacheable/prefetchable, you might be able to get some good results with ARC beyond what the raw disks will do for uncached IO.

You said you're providing an open source software mirror - that seems like you're more likely to care about bursty workloads that aren't necessarily cached than about the same few bits of data being hit over and over again, unless almost all of your load is from the same few things, which IME is only going to be true if e.g. there's a new Ubuntu release or something and your traffic is temporarily dominated by that.

[Reset816]

I mean, ZFS can do striped mirrors, it's not like it only supports stripes or parity raid.

10 GB/s is more throughput than all of your disks can do together - I'd estimate them as maybe totaling 250 MB/s * 6 = 1500 MB/s max, maybe, in peak conditions? And that's not a ZFS or btrfs or w/e limit, that's just raw reading/writing the disks.

Is your networking actually 10 GB/s, or 10 Gb/s? Because the latter is both far more common and far easier to achieve.

If your workload is highly cacheable/prefetchable, you might be able to get some good results with ARC beyond what the raw disks will do for uncached IO.

You said you're providing an open source software mirror - that seems like you're more likely to care about bursty workloads that aren't necessarily cached than about the same few bits of data being hit over and over again, unless almost all of your load is from the same few things, which IME is only going to be true if e.g. there's a new Ubuntu release or something and your traffic is temporarily dominated by that.

Oh it's 10 Gb/s. Sorry I made a mistake.

[Reset816]

I mean, ZFS can do striped mirrors, it's not like it only supports stripes or parity raid.

10 GB/s is more throughput than all of your disks can do together - I'd estimate them as maybe totaling 250 MB/s * 6 = 1500 MB/s max, maybe, in peak conditions? And that's not a ZFS or btrfs or w/e limit, that's just raw reading/writing the disks.

Is your networking actually 10 GB/s, or 10 Gb/s? Because the latter is both far more common and far easier to achieve.

If your workload is highly cacheable/prefetchable, you might be able to get some good results with ARC beyond what the raw disks will do for uncached IO.

You said you're providing an open source software mirror - that seems like you're more likely to care about bursty workloads that aren't necessarily cached than about the same few bits of data being hit over and over again, unless almost all of your load is from the same few things, which IME is only going to be true if e.g. there's a new Ubuntu release or something and your traffic is temporarily dominated by that.

Besides, I noticed another mirror site their ARC hit rate reach 80% with 300GB RAM.
I guess cache most of the popular software can provide most of bandwidth. In the short term, people download/update more or less the same softwares.

[ams-tschoening]

I am trying to upgrade my school's open source software mirror site, which doesn't care write performance.
Don't have enough money to buy more disk and ram.

You didn't tell anything about what your users actually demand, only that YOU want to have things as fast as possible. Do your users really care if they e.g. need to wait an additional minute downloading some Linux ISO?

Additionally, you didn't mention how much data of which type you are actually storing, how old things at least need to be, how much free space you have right now etc. Remember that ZFS supports different compression algorithms and depending on your kind of data and CPU power and stuff, it might be worth it applying the highest possible compression level to data, possibly resulting in X % of your pool staying free anyway already.

One possible configuration is raidz of three 10TB HHD and raidz of three 18TB HHD.
And another possible configuration is btrfs 10 of these 6 disks.

If (random) reads are your main focus, you might as well consider a 3-way mirror. Things heavily depend on how much storage space you actually need, how much you might benefit from compression and stuff. There are some discussions about a 3-way mirror performing best for reads, while others don't seem to see those benefits. Things might depend on OS and stuff as well, but might be worth a try for your actual setup.

https://forums.freebsd.org/threads/zfs-read-performance-of-mirrored-vdevs.59879/#post-343931

Don't have enough money to buy more disk and ram.

You might want to consider using some ZRAM-device as well, which makes some parts of your RAM becoming compressed. Though I'm not sure if this isn't already the case for the ZFS ARC when applying compression to data storage.

https://en.wikipedia.org/wiki/Zram
https://github.com/systemd/zram-generator
https://en.wikipedia.org/wiki/Zswap

[rincebrain]

ZFS ARC will keep things compressed in RAM if they're compressed on disk, though sometimes it will keep uncompressed copies around for a bit when, say, you read it.

[dvaezazizi]

This post further signifies that the documentation/behavior of ZFS needs to specify/notify the end user of expected performance changes when disk space is utilized at "high" percentages.

The other option is to reserve space at the time of vol creation so that performance degradation isn't an issue. It's a bit dangerous on the enterprise side if a vol gets 100% b/c some sysop forgot to reserve space back in the day.

[robszy]

Real problem here is permament performance degradation when you cross mark (96 in today's version) even after free up space to have this 20 % free because structure of file system cannot change so it cannot be defragmented and algorithm to seek for free space is not so good when fs is very fragmented - real issue is fragmentation of free space which cannot be fixed

Only way to fix is send | receive which could be problem in production.

Documentation of this is very important