I suspect that it is possible to simply add new rules that aggregate metrics at a lower frequency. That is, I don't think we need a second instance.

https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/

I would also prefer a lower frequency logging approach than having a duplicate instance.

The reason I suggested a second Prometheus is because the retention policy is instance wide and not per metric, see

However, this has been a while ago. Maybe newer version of Prometheus allow for per metric data retention. To explore.

So yes we can lower the polling frequency, but if we can not increase the retention duration, then we still do not have long term stats, which is our ultimate goal.

Also, I suggested a second instance of Prometheus on a different machine. So it's not really "duplicated" because it's not the same role:

• it can still provide stats even if the real PAVICS host is down (hardware failure, data corruption, ...)
• it can aggregate all other PAVICS hosts (staging, tests, ...)

No, you're right, there is one retention policy per instance: https://stackoverflow.com/questions/69630832/how-to-store-data-in-prometheus-with-different-retention-time-per-job-or-targets#:~:text=Prometheus%20doesn't%20support%20multiple,configs%20and%20distinct%20retention%20periods.

Ok. If it is a limitation of Prometheus, then let's try with a second one.

Post on solutions to this problem, which in the jargon seems to be known as "downsampling":
https://last9.io/blog/downsampling-aggregating-metrics-in-prometheus-practical-strategies-to-manage-cardinality-and-query-performance/

Instead of a second prometheus to scrape the first we could also use one of the other technologies they recommend for longterm storage (https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage). Both Thanos and M3 seem to be recommended elsewhere.

We could create recording rules for the metrics we care about storing in prometheus and then use an external tool to store those specific metrics over a longer term and query just those metrics over larger time-scales as needed.

I've been playing around with Thanos. One issue with Thanos is that is stores all metrics from a prometheus instance which can end up being a very large amount of data (even if the data is compacted somewhat: https://thanos.io/tip/thanos/quick-tutorial.md/#compactor).

As discussed here there are potential ways to store only those metrics that we care about which would reduce the amount of additional disk space needed to store the data.

It looks like we're likely going to need to introduce a second prometheus instance even if we also use Thanos so that we can select which metrics we store long term.