availability-distribution: potential memory leak

[sandreim]

This is a nice pattern of increased CPU usage over time. I didn't look much at it, but it might be some iteration of a collection that doesn't get cleaned up.

https://grafana.teleport.parity.io/goto/9zQcAE9IR?orgId=1

CC @alindima

[alindima]

I looked into this.

The subsystem itself is not doing any work in the main loop, as it spawns separate tasks for everything.

The CPU usage for the tasks that the subsystem spawns (pov-receiver, chunk-receiver, pov-fetcher, chunk-fetcher) all look normal.

I discovered the culprit: Jaeger spans.
We hold a local hashmap of spans:

polkadot-sdk/polkadot/node/network/availability-distribution/src/lib.rs

Line 107 in ebcbca3

let mut spans: HashMap<Hash, jaeger::PerLeafSpan> = HashMap::new();

which adds a new entry on every leaf activation, but only gets cleaned up when a block is finalized:

polkadot-sdk/polkadot/node/network/availability-distribution/src/lib.rs

Line 175 in ebcbca3

FromOrchestra::Signal(OverseerSignal::BlockFinalized(hash, _)) => {

However, when a block gets finalized, we only get notified about the block with the largest height (the leaf), so we never clean up spans for relay parents that got finalized in a batch (since Grandpa finalizes chains, not blocks). Therefore, the size of the hashmap is almost always strictly increasing.

Now, to fix this there are a couple of options:

1. Remove Jaeger altoghether: Remove jaeger #4995
2. Fix this somehow. Simplest idea is to also store the block number for each span and when getting a finalized block, also trim all spans with lower block heights. Another option is to modify the overseer to also supply the active leaves chain that have been deactivated by this finalization.

[alindima]

Here's a PR that fixes the issue #5321