What is a good value for zfs_arc_max?

[victornoel]

I'm using a computer with 32G memory and I'm thinking that using the default value for zfs_arc_max (half of the memory if I correctly understood) is a bit over the top for so much memory (especially because I run processes that can potentially need a lot of memory).

Basically, the hypothesis here is that the best value for zfs_arc_max should be dependent on my ZFS installation (number of pools, dataset, size of disks, etc) and not a function of the size of my memory.

I couldn't find anywhere any recommended "good" values for having a functioning system. Ideally, I would like to know what is a minimum good value, maybe depending on the kind of ZFS layout I have.

In my case, for example, I have one pool with one disk of ~1TB and 2 datasets, one for root and one for home, with encryption for a typical dev desktop usage.

[IvanVolosyuk]

You can probably estimate your working set and set zfs_arc_max accordingly. I'm using 64G ram: 16G (VM's explicit hugepages), 64G zram (with my scripts to spill to disk compressed pages over 4G), 16G min 32G max ARC. My working set is usually ~20G, but when I compile stuff it can go to >40G. I set zfs_arc_min to 16G to force usage of zram and don't break L2ARC in memory headers + MFU. ZFS will actually shrinks ARC when more memory needed for system, thus I also use: ```echo 50 >/sys/module/zfs/parameters/zfs_arc_pc_percent``` in order to apply some memory pressure to the rest of the system and use zram more actively.

[victornoel]

This completely expanded my horizons about what I can do :) Thanks a lot!

So to be sure I understand:

• what you call the working set is how much memory the computer is using in practice
• you have 64G physical memory
• you setup a 64G zram (which will of course use much less than 64G since it's compressed)
  • and also it will spill to disk if it actually uses more than 4G, or is it spilling 4G+ pages to disk?
  • how do you configure this spilling to disk?
  • I'm discovering zram actually and a quick search didn't help me, sorry if it's something already well documented somewhere else
• you use zfs_arc_min with a kinda high value and zfs_arc_pc_percent to ensure that ZFS does not try too much to release memory, since the zram is here to be used as most as possible
• and so one of the conclusions is that actually zfs_arc_max is not really interesting to reduce in this kind of scenario.

Is this correct?

On my side, I should try to evaluate what is my working memory, and then adjust all the rest (incl. zfs_arc_max if I don't go the zram way) to leaving enough memory space for it. Would that be the idea?

Thanks again!

[IvanVolosyuk]

Yes, you understood correctly.
I use 5.10.x kernel which don't have advanced zram idle marking, so when number of compressed pages in memory exceeds 4G I do:

echo idle >/sys/block/zram0/writeback
echo all >/sys/block/zram0/idle

I have written a simple c++ program to do just that. This can be done in a shell script as well. You can have read about the writeback at https://docs.kernel.org/admin-guide/blockdev/zram.html

ZFS will actually releases memory to system on pressure and will work reasonably well by default. You may want to limit the maximum size of ARC if you need large amount of memory very fast and ARC shrinking speed cannot keep up with amount of memory requested.