L2ARC is failing to rebuild even though there is plenty of memory

[FallingSnow]

I've just noticed my L2ARC is failing to rebuild on reboot. I have plenty of memory still available. Any idea why it's telling me there is low memory?

Dmesg:

...
[May 4 20:22] NOTICE: System running low on memory, aborting L2ARC rebuild.
[  +0.001074] NOTICE: System running low on memory, aborting L2ARC rebuild.

$ free -h
               total        used        free      shared  buff/cache   available
Mem:            62Gi        26Gi        35Gi        16Mi       391Mi        35Gi
Swap:           62Gi          0B        62Gi

$ arc_summary
...
ARC size (current):                                    49.7 %   15.9 GiB
        Target size (adaptive):                        49.7 %   15.9 GiB
        Min size (hard limit):                          6.1 %    1.9 GiB
        Max size (high water):                           16:1   32.0 GiB

[amotin]

Because by default L2ARC is not allowed to use more than 33% of ARC for its headers.

[rincebrain]

Shouldn't you not be able to get into that situation, then, without a chicken-and-egg problem?

e.g. how did a system which limited you to 33% of ARC for L2ARC headers produce an L2ARC that cannot rebuild without taking up more than 33% of ARC, unless ARC size changed or the rebuild overhead is nontrivially huge and counted against it?

[amotin]

One possible explanation is that L2ARC evictions are not logged, so rebuild may allocate headers for already delete data and so go over the limit.

[FallingSnow]

This makes sense given my headers are taking up 33% of my ARC.

L2ARC size (adaptive):                                         985.2 GiB
        Compressed:                                    95.8 %  944.0 GiB
        Header size:                                    1.1 %   10.5 GiB

[rincebrain]

n.b. the code has a comment suggesting you can try offlining and onlining the l2arc device to make it trigger rebuilding again after changing the tunables controlling when it refuses to rebuild.

[amotin]

Deleted data that were evicted from L2ARC before reboot but resurrected from L2ARC log by rebuild.

[rincebrain]

So not actual garbage data, just data that wasn't useful to persist but wasn't otherwise stopped from doing so?

It seems like we could conceivably overwrite those headers to preclude that, no? If it's a bother to anyone?

[amotin]

As I mentioned above, L2ARC log includes new writes, but not early evictions. We could probably add those, but it would still be a bit weird, because since the log is written and read sequentially, we would first rebuild those evicted headers and then evicted them again. And after some L2ARC overwrites we would get some eviction records for headers we never rebuilt. It would be fun in other words. :)

[rincebrain]

I guess I'm wondering why we couldn't reach backward and just scribble over the headers in question, unless that would invalidate every entry thereafter?

[amotin]

All L2ARC is designed for sequential writes.