NVMe drive layout considerations with multi-socket CPU's

[stovakor]

Let's assume I have the following hardware configuration.
A 2U server with 24x 2.5" U.2 NVMe slots.
Let's assume the PCIe lane configuration is such that the first 12 slots of the server map directly to CPU socket 1, and the remaining 12 slots map directly to CPU socket 2.

https://infohub.delltechnologies.com/en-US/l/nvme-and-i-o-topologies-for-next-generation-dell-poweredge-servers-1/poweredge-r7525-populated-with-24-nvme-drives-4/

Let's assume my objective is to extract the maximum performance from the hardware. Is it better to have all drives <12 mapping to one CPU or the other? Or would there be performance benefits (or other considerations) splitting the drives between different CPU sockets?

We can hopefully ignore any mirror vs RAID-Z discussions because the crux of the question surrounds optimal drive configuration at the hardware layer.

[amotin]

At this time ZFS still has no NUMA awareness outside of what underlying OS can do. So unless you are going to manually manage some CPU affinity for all relevant individual threads to one domain, I would make topology symmetrical. But then your application or networking may not get balanced, etc. so it all may end up out of reasonable control.

[stovakor]

So to summarize, the interlinks between CPU sockets (be them AMD or Intel) are still currently faster than NVMe Gen4 or even potentially Gen5 PCIe such that there would not be any performance issues with a byte being read from a drive mapped to CPU2 that is requested by a network card or other application mapped to CPU1?

[amotin]

On a way from SSD through ZFS to NIC data will be copied in memory about 3-5 times. It will be a much bigger problem for the interlinks than single read/write from SSD itself.

[Ramalama2]

To begin with, for 24x Nvme (u.2/u.3/e1) SSDs, no one that wants performance would even thinking of using ZFS.
It makes literally no sense, since you will loose around 90% of raw performance.

EDIT: Just to avoid further stupid discussions:
The main issue, that you'll run into, will be your memory speed, since every block needs to get copied 3x to the memory.
IOPS will get too complicated but Speed is an very easy example, so if one drive can do 5GB/s, and you use an Raid10 Array, which is the only usable Raid-Level, to get Performance/Speed/IOPS out of 24 u.2 drives:
Read: 24x5 = 120GB/s x3 = 360GB/s
Write: 12x5 = 60GB/s x3 = 180GB/s

Theory how it could work:
Forget your interconnect, interconnect is to slow, so you can do only 1 Socket system and you need at least 360GB/s of memory speed.
Since one DDR5 Module (Sigle Channel) has around 36-40GB/s, you need at least 360/36 = 10 channels.
So the only Platform that can do this is Genoa (12 Channels)

But thats only memory, there are so many other factors, even checksumming will bottleneck.
Until there is direct_io support and some sort of better acceleration like using AVX512/SSE3/4/4.1 or something like ZIA, there is anyway no sense to use that much nvme drives with zfs, even if you have the memory bandwidth.

I have 2x such Genoa Systems, with maxed out memory channels and tryed ZFS with 8x Micron 7450 Max in Raid10 on each server.
The Performance compared to basically any other filesystem is utterly bad. In the end we had to use a proprietary FS, because there is simply not really an alternative to ZFS.

Lets hope things changes, because if not ZFS will get only a filesystem for HDD's and Homelabbers, maybe Bcachefs might be a solution, but first benchmarks shows that its even slower as zfs and there seems to be still bugs.

Don't want to attack anyone who loves zfs, i love it myself, its just the reality because we move more and more to faster/more affordable nvme's. (Data center U.3 ssds aren't that expensive anymore)
Cheers

[stovakor]

ESXI does not fit my use case due to 64TB LUN limitations. I need volumes
larger than 64TB.
I don't "need" a hypervisor per se, but am open to it if the abstraction
layer between the NVMe storage and the OS would yield better performance.
We only have the 1 node though. How will proxmox present the multiple NVMe
drives as a single volume any better than ZFS?

[Ramalama2]

Which FS/Blocksystem has a limit of 64TB, i don't even think that something like ext4 has such a small limit.
What i know is, that lvm for example has no limits at all, just what the kernel can handle, so lets take straight LVM for example, because i think people don't understand how similar it is to ZFS, sure it misses a lot of features, but the basics are there, and if you need performance its the only free way.

• First we need to correct one thing, thats in the head of almost everyone that loves ZFS,
  bitswap/checksums that ZFS provides for Data integrity, isn't that important on enterprise NVME Drives, because they have built-in ECC.
  Sure ZFS will surely provide a good feel and probably even some benefit in rare cases, but its simply not important enough on enterprise ssds, that you choose ZFS because of that.
• Second, ZFS Provides the ability to take unlimited snapshots, as backup replacement for example, but most companies have a backup server and even proxmox provides their own PBS Solution.
  So i would say, the deciding factor here is, probably Volume Shadow Copy? Because there you need a ton of snapshots, otherwise in a normal VM environment you simply don't need more as 1/2 Snapshots of the VM.
  Thats the main weakpoint of LVM, because it gets slower the more snapshots you have, but please take into account that you also start with a much higher speed/iops compared to ZFS.
• Third Compression/Deduplication, LVM Provideswith plugins Compression/Deduplication but i never tested that and would stay away from that, i simply don't like plugins or something not heavily tested. ZFS Compression with LZ4 or ZSTD, is almost an perfect top notch solution where it makes no sense to not use it. ZSTD on ZFS for HDD Pools is even better as LZ4 in my opinion, but on nvmes clearly LZ4, because nvmes are simply too fast. Deduplication on ZFS, is absolutely amazing on a Backup Server with Hard-Drives, but an absolute no go with everything flash based, even sata ssds. With deduplication you can loose around 2/3 of your performance, if not even more.
• Datasets on ZFS, exactly the same thing on LVM is called LV (Logical Volume)

Now the ground explanation:
PVs = Physical Volumes, its either real Disks or Partitions/Volumes
VGs = Volume Groups, it's like simply a bunch of disks dedicated for LVM, its comparable with a ZFS Pool, but without the Raid.
LVs = Logical Volumes, you can compare it simply pools and datasets on ZFS.

LVM is just a bit different in understanding, because the actual raid gets created in LV level, not in VG(Pool) level, which is at first a bit confusing, but its actually the same.
In LVM2, the RAID level configuration is applied at the Logical Volume (LV) level, not at the Volume Group (VG) level. This means each logical volume within a volume group can have its own distinct RAID configuration, independent of other logical volumes in the same group.

So imagine it, as you can define a lot of disks to the Volume Group and you can afterwards create a ton of Datasets that have all its own Raid levels.
You can even create cache-pools or writecache only pools, similar to ZIL/Special vdev on ZFS, but again, its very limited featurewise, even barely comparable. But its not the point, i just want to imagine, that its similar to ZFS actually.

So only one thing is left, you can use lvm2 bare, or you can use lvm2-thin dataset on top of lvm2, the bare approach supports simply clusterization, so access from different nodes to same Blocksystem, lvm2-thin supports on other hand Snapshotting and some other features but no Clusterization.
Its actually exactly the same with ZFS, just that ZFS supports Snapshots in any condition, because ZFS does snapshots on bare Blocksystem but the ZFS-Filesystem doesn't support Clusterization.
Dunno how i can explain that better, since ZFS is somewhat a mix between Blocksystem and Filesystem, its both, so its harder to explain.

The only real Downside of LVM, that it is not a Filesystem, like ZFS is both, LVM is only a Blocksystem.
That means in Short, in a example of Proxmox (because it seems like you are familiar with)
VM's can run bare on LVM2 or LVM2-Thin, because they use qcow and raw, but LXC-Containers or any access on the Proxmox host itself (samba server or whatever else), cannot run on LVM, because those need a filesystem.

So you can simply use bare LVM for VM's for the maximal performance and create a dataset and format it with ext4 for example for everything else that needs a Filesystem. Or you can create multiple LVM2-Thin Datasets for example that are not size limited and act exactly the same way as ZFS Datasets and format it with whatever filesystem you like.
The datasets will take only the space they need for the files inside them, which in the end is your Big Raid pool i believe?

Cheers

[Ramalama2]

BTW, 16x64GB will be slower as 8x64 or 8x128.
Not so much slower, but most Platforms i seen do downclock the memory if you use 2 Dimms per Channel.
Often from 4800 to 4200, or 5200 to 4200 if in 2D Config.
But i also cannot tell you how much the downclock slows down the bandwidth, never tested.

But keep that in mind, read your Mainboard or Chassis Manufacturer manual/specs something, its mentioned there usually somewhere.

And yeah i know that 128GB DDR-5 ECC Modules are insanely stupid more expensive as 64GB, but its your decision.
Cheers

[stovakor]

This is great information. Our workload will be 50+TB of large (mostly static) non-compressible media files. Ranging from 10-40GB per file. Once the media is generated we need to retain it indefinitely. So our storage will only grow and grow.

As far as the 64TB limit, that is a well known VMWare limitation.
https://en.wikipedia.org/wiki/VMware_VMFS
https://configmax.esp.vmware.com/guest?vmwareproduct=vSphere&release=vSphere%207.0&categories=2-0

So to summarize, we can use LVM to combine the disks in place of RAIDZ concepts from ZFS which would provide much better performance. We must be mindful though that we will need to put a file system on top of that (ext4, etc.).

[Ramalama2]

Continue below, i quoted, it made something weird xD

[Ramalama2]

This is great information. Our workload will be 50+TB of large (mostly static) non-compressible media files. Ranging from 10-40GB per file. Once the media is generated we need to retain it indefinitely. So our storage will only grow and grow.

As far as the 64TB limit, that is a well known VMWare limitation. https://en.wikipedia.org/wiki/VMware_VMFS https://configmax.esp.vmware.com/guest?vmwareproduct=vSphere&release=vSphere%207.0&categories=2-0

So to summarize, we can use LVM to combine the disks in place of RAIDZ concepts from ZFS which would provide much better performance. We must be mindful though that we will need to put a file system on top of that (ext4, etc.).

But tbh, if your Workload is such a big static Files, why do you need 24x NVME's?
I talked always about max performance and meant mostly iops performance.

But your workload doesnt need performance at all, wouldn't it be a better solution to save the money and use ZFS with HDD's + a really fast mirror a bigger special vdev (like 3,8TB) in case you need still something that needs a lot of iops or a VM or something?

If i would be you, i would buy instead some big HDD's, like 12x22TB use them in Raid10, or a stripe of 2x Raid-Z2 with a mirror of special vdev like i suggested above, just to make sure that all sorts of Workloads are fast.
Then i would set a recordsize of 1mb, so smaller files/Blocks as 1mb goes to that 3,8TB fast vdev.

That gives you 130TB of usable Space, which is plenty and you can still do 1,2GB/s write and 2,4GB/s speed with Raid10.
Which will fully saturate a 10GBe Connection with ease.
And you can save on the Memory either, no need to populate all dimm channels or thinking about 2dimms per channel or not etc...

Don't missinterprete my writing, but i readed you initial Request as, 24xNVME and ZFS, which leaded me to all this.
People that want 24xNVME Drives in a Server, do this for 99% of the time for the speed, and im Talking about speeds of at least 50GB/s, where you either use this as SAN with a 400GBe Network Card or Locally on the Server itself.

Cheers

[stovakor]

HDD's are out. We're future proofing this solution. We understand it's overkill now, but don't want to keep building new hardware every 24-36 months. I want something that is going to be able to keep up with workloads 5, 7, 9 years from now. The MTBF of SSD's is in the millions of hours. And for a mostly read intensive workload, the data's long term integrity and availability is what is drawing us to SSD's.

And we're not starting with 24x NVMe, that's just our final build out capability. The server will be capable of supporting 24x NVMe drives with full bandwidth x4 lane performance (see first post for links for more info).
Assuming ZFS we were initially targeting 6x or 8x 30.76TB NVMe drives in RAID-Z1 (possibly Z2), then growing by 6 or 8 drives every few years.

[Ramalama2]

Then use ZFS, and forget LVM or anything else, because then till then, ZFS will have ZIA and direct_io + some other features which will make it suitable for SSDS i believe.

However, i hope you know that adding drives will force you to rebuild the ZFS pool, not a big deal if you can copy the data to something else and add it back, but just to mention.

[amotin]

While not exactly a direct-io, if massive data caching is not required due to very fast SSDs, setting primarycache=metadata on modern ZFS should allow to avoid a number of memory copies for both reads and writes. The 5 copies I mentioned was the total maximum, including network services. ZFS itself may be down to about 2, depending on settings and situation. It is a price to pay for functionality. Direct-IO should help in some cases though.

Rebuilding pool should not be necessary, don't exaggerate. With SSDs not being a performance bottleneck, perfect data distribution should not matter that much to bother offloading the data, unless the pool was too fragmented due to overflow before expansion. Improved space vs speed vdev balance for imbalanced cases like that is on my plans list.

[stovakor]

Apologies for the ignorance, but I thought ZFS allowed expansion of the pool by adding additional vdev's. So if we started with a 6x30TB RAID-Z1, we could (down the road) add another 6x30TB RAID-Z1 to the existing pool without requiring rebuilds or data migrations.

[amotin]

Sure. You can do it. The only downside will be that old data will stay on old vdev(s), while significant part of new writes will go to the new ones. It may create some performance artifacts. But as I have told, in case of fast SSDs it should not be a huge deal.

[stovakor]

Is there a resliver or rebalance method to redistribute the data evenly across the new expanded pool without having to migrate data off , start over, and migrate data back?

[amotin]

Not within ZFS itself. I've heard people using some scripts just copying data in place, causing rebalance that way.

[Ramalama2]

@amotin you are right, primarycache=metadata is indeed a must have thing on extremely fast ssds, i think it should be default for anything can do above 10GB/s speeds of the nvme pool, below that its an personal preference/opinion/etc...

about the pool expansion, ugh, i know that works, but thats somehow confusing to me/weird, because usually i would expect from an stripe of 2x Raid-Z1, twice the performance of one, but if you expand it to 2 afterwards you will get sometime the performance of 2 and sometimes of one, which is a bit weird simply.
i mean we know that anyway, just need somehow to add my 5 cents, dunno xD