High Memory Usage for Zpool with multiple datasets.

[nikhil-adhau]

I am observing a significant memory usage issue in my ZFS setup that I hope to get some insights on. Specifically, I have around 3,000 datasets, and I'm noticing an additional 4.4 GB of memory usage, alongside 2.2 GB being used by the ARC.

Datasets Count	Total Memory Usage (in mb)	ARC size (in mb)
0	4729	192
100	4823	263
200	4974	334
500	5547	544
1000	6180	883
2000	7651	1536
3000	9156	2258

Setup Details:
ZFS version: 2.2
OS: Rocky Linux 8.9

Why does ZFS require such a high amount of memory for managing datasets, especially with no data present in them?
Are there specific configurations or properties I should consider adjusting to reduce memory overhead?
Is there a general rule of thumb for memory usage per dataset that I should be aware of?

Any insights or recommendations would be greatly appreciated!

[amotin]

I don't have a rule for number of datasets, but IIRC in TrueNAS we have a rule of warning users about having more than 10K of snapshots before management becomes difficult. It is difficult to say what exactly consumes the memory, since part might be kernel outside of ZFS, part may be ZFS memory that is held by kernel references (that may depend not only on number of datasets, but also on number of files and ongoing activity) and finally part may be normal voluntary caching. I've recently updated arc_summary tool in ZFS 2.3 to make it more clear what ARC size consist of. You might look into in on 2.2 also, but it was less focused for that purpose there.