Prometheus gauge volatility for system/transient scope

[hivenet-mathieu-lacage]

Hi,

Recently, I started to collect prometheus metrics for a couple of libp2p nodes based on v0.27.3. I am trying to make sense of the ressource manager metrics, specifically, libp2p_rcmgr_streams, libp2p_rcmgr_peer_streams_count and libp2p_rcmgr_previous_peer_streams_count.

Interestingly, all libp2p_rcmgr_streams{scope=~"system|transient", instance="XXXXX"} graphics look similarly spiky:

Which is surprising to me: this graphic really looks like the gauge is being reset intermitently to zero. I got curious so I plotted this query:

sum by (scope)(libp2p_rcmgr_streams{instance="XXXX",dir="inbound"})

with this result:

where we see the sum of protocol gauges being mostly exactly equal to the system gauge, except when it is reset to zero. Has anyone already reported similar problems with the streams gauge ? Am I missing something obvious that explains the observed behavior ?

On a related note, I expected this to be true:

sum by (instance) (libp2p_rcmgr_streams{scope="protocol",dir="inbound"}) + sum by (instance) (libp2p_rcmgr_streams{scope="transient",dir="inbound"}) = sum by (instance) (libp2p_rcmgr_streams{scope="system",dir="inbound"})

Is this a reasonable expectation ?

Mathieu