Fix monitoring metrics for individual collectors

[ananya-mallik-ps]

Fix monitoring metrics

Problem

Currently, each time a metrics endpoint is hit (with collect parameter e.g., /metrics?collect=pubsub.googleapis.com/topic), a new collector is created. This causes the monitoring metrics (stackdriver_monitoring_api_calls_total, stackdriver_monitoring_scrapes_total etc.) to reset on each scrape.

Solution

This PR:

• Add collector caching in the handler using a map
• Key the cache using project ID + filters combination
• Reuse collectors to maintain metric state across scrapes

Testing

• Tested build locally, with one of our GCP project for different collector(s)
• Deployed in one of our cluster and tested with multiple collector (~ 110+ different metrics-type-prefixes)
• Verified metrics accumulate properly across scrapes for each collector
• Memory usage remains stable over time and is similar to previous version

[ananya-mallik-ps]

@SuperQ @kgeckhart

[kgeckhart]

I think this is awesome and allows aggregate deltas + descriptor cache to be compatible with collect time filters (resolving #315).

Since both of those features prune their storage via collector usage we should probably add a collector cache TTL that at least matches the largest TTL if either feature is in use. This way if a scrape config changes the collect filters the exporter will eventually clean itself up without needing to be restarted.

[ananya-mallik-ps]

Since both of those features prune their storage via collector usage we should probably add a collector cache TTL that at least matches the largest TTL if either feature is in use. This way if a scrape config changes the collect filters the exporter will eventually clean itself up without needing to be restarted.

@kgeckhart Added default collector cache TTL to be 2 hours (open to suggestion) and overriding it with largest of all 3 TTLs.

[kgeckhart]

Sorry I think I didn't fully explain the purpose of the TTL and how it relates to the descriptor cache/aggregate deltas.

There's three use cases,

1. A Collector is consistently being used -> safe to keep forever
2. A Collector is no longer being used with the descriptor cache and/or aggregate deltas enabled -> Remove after greatest TTL between the two features
   • Both features add memory overhead through caching but prune themselves when the collector is in use
   • In the case of aggregate deltas, this could be a very large amount of memory because every metric is kept around to enable the delta aggregations
   • Removing the collector after the TTL will drop these caches that would otherwise be left around until the exporter is restarted
3. A Collector is no longer being used and descriptor cache/aggregate deltas is disabled -> Remove after the default 2 hour TTL (overhead of these collectors should negligible)

I left some more direct comments for the implementation.

[ananya-mallik-ps]

@kgeckhart : Added suggested changes for descriptor cache/aggregate deltas use case.