connectionpool_inPool gauge not correcly reflecting connections in pool as it never decreases

[jongunnip]

We are logging the connectionpool_inPool metric every minute to track open connections. Unfortunately, this metric appears to increase with each new connection created and does not decrease when the connection is closed as the metric only increases over time. We can use netstat to see that there are nowhere near as many open connections as reported by connectionpool_inPool.

The connectionpool_inUse metric seems to be accurate.

Here is the code we use to register the metric colleciton in our Bootstrap.java:

// codahale metrics registry
var codahaleRegistry = new MetricRegistry();
bind(MetricRegistry.class).toInstance(codahaleRegistry);
Registry spectatorRegistry = new MetricsRegistry(Clock.SYSTEM, codahaleRegistry);
bind(Registry.class).toInstance(spectatorRegistry);

We are using Zuul 2.4.0. We saw the same behavior in 2.3.0. I attempted to look at the code that tracks these metrics but the increment/decrement logic for the connection tracking was hard for me to follow as it appeared to be sprinkled across many methods.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[jongunnip]

I'm still interested in whether anyone has ideas about how to fix the connectionpool_inPool metrics. We are starting to implement connection limits and having better insight into connection use is important in being able to tweak and monitor connection behavior.

[jguerra]

I don't see the same issue in our internal inUse metrics. The counter should be decremented by DefaultClientChannelManager.release() which will be called when a pooled connection is no longer in use

[jongunnip]

Sorry, @jguerra. I had a typo in my coment yesterday. It should have said inPool and not inUse. I edited the comment to fix that. The inUse metrics are working well. It is the inPool ones (as mentioned in the issue title) that are not reporting properly.