Is Data Written to a Pool Also Written into the ARC?

[Haravikk]

As the title says, I'm curious whether any data being written to a pool is written into the ARC, or does the ARC only operate upon data that is being read from a pool?

Basically I'm trying to confirm some behaviour I've been observing – my home setup consists of three pools, one is my working pool (zdata), one is a direct attached backup pool (zbackup), and the third is my NAS (zbackupds). Basically what I do for my backups is I send regularly from zdata to zbackup, then periodically send from zbackup to zbackupds (when it runs, this occurs immediately after the zdata -> zbackup send/receive).

But I've noticed that the disk activity on zbackup during each step is broadly the same, except that it flips from writing (zdata -> zbackup) to reading (zbackup -> zbackupds) which seems to suggest that nothing (or very little) is being pulled from ARC, and it's all coming from the disk(s).

As a result I'm interested in optimising that step somehow, as most of my backups tend not to be that large (usually in the order of 1-3gb) so could be buffered in RAM, but the fact that the frequency of the backups is different makes that tricky (I can't just retain the output of zfs send and repeat it). I could alternatively change the procedure so both backups are from zdata, which isn't impossible, but partly why I set up this "chained" approach is I may eventually move zbackup to a NAS as well.

It seems like it would be beneficial in my case if the first backup (to zbackup) populated the ARC with the new data, as it would mean that the bulk of the data going to zbackupds could be fetched from ARC, since that step runs immediately afterwards.

TL;DR
Basically the questions I'm interested in are:

• Does any written data currently get stored in ARC (i.e- without reading it afterwards)?
• If yes:
• • What are the ways to tune this, can cause ZFS to put more/everything in there for some/all datasets/pools?
• If not:
• • Is there any particular reason why not? I know it could result in wasted space (ARC entries that get purged without being used) but that seems like a configuration problem (users who know they need it could opt-in).
• • What would be the difficulties in implementing this?

[amotin]

With exception of Direct I/O added in ZFS 2.3 and which has to be requested explicitly, all normal I/O in ZFS, including writes, goes through ARC. But ARC is called "Adaptive" for a reason. It tries to balance between MRU and MFU policies based on the workload it experiences over time to prevent massive wipe out by large sequential operations. I suppose the backup is not the only workload of your system, so you may take a look on arc_summary output to see how the ARC capacity is distributed. Also send/receive is not exactly a general-purpose workload. IIRC that code is intentionally optimized to not use ARC when it does not include required data and so may be not very helpful. But it may need a deeper look specifically for sending and receiving. At least send, I think, while uses data already available in ARC, does not populate it.

[Haravikk]

I suppose it would really be the zfs receive side that I'm interested in, since that's the part doing the actual writing of the received data – if ZFS is already caching writes in ARC normally and zfs receive is bypassing that somewhere, what do you think the complexity would be in allowing some kind of override, e.g- an option on zfs receive to allow writes to hit ARC?

[no-usernames-left]

If your backups are 1-3 GB then why are you worried about them being cached? The network will likely be the bottleneck, not the disk I/O.

[Haravikk]

I'm not worried about speed, it's more just the fact that the backups (from zbackup to zbackupds) are being read right back off the disks that they were literally just written to, rather than coming out of ARC, so it means the disks get twice the amount of activity (1-3gb of read activity following 1-3gb of write activity, rather than just the latter).

There's nothing functionally wrong as such, it just struck me as odd that none of the data seemed to be in ARC having just been received – I doubt the extra wear is too much to worry about, but it is still extra, which means those disks in the "middle" pool spend less time idle etc.

[amotin]

Yes, receive mostly bypasses ARC for data. I knew its implementation of "lightweight write" bypasses dbuf cache, but on a quick code refresh I see that it also bypasses ARC, calling ZIO pipeline directly, similar to Direct I/O. It allows to avoid unnecessary memory copies and other overheads to improve throughput, plus not wipe ARC possibly used by other workloads. It is believed not very typical for backups to be accessed just after write, so it is reasonable optimization.

[mschilli87]

Makes sense IMHO. The only way to 'improve' this that I can see would be to make it user tunable. But I have a feeling the extra complexity to maintain two implementations of receive might be a bit too high price to pay for this niche use case.

[Haravikk]

If it were to be implemented, an option on zfs receive could make sense? This would allow easy enabling of it only in cases where you know the data will be used sooner after receiving, as that may not be all the time, or you may want to only use it when you know the send size isn't too large (for my own case a 1-3gb send is fine, but if I'm doing a 12gb+ one that would exceed the size of my current primary ARC). A tuneable would enable it for all pools/datasets I think, and run the risk of being left enabled if you only need it temporarily.

Granted my "backup chaining" case is a bit unusual, but I wonder if there are others where it would make sense? Only thing I've thought of off-hand would be some kind of data-processing setup where you have one system generating data and then using send/receive to copy it to other systems that consume it – not sure if anyone currently does that using ZFS though. A similar case would be a system setup for failover (where the clones are read-only until the primary system fails), but that doesn't guarantee immediate reading of new data (unless it has to be loaded in somehow by running software to be indexed etc.).