Stopping export of metrics after some time of work

[romaadereyko]

Container is working, but prometheus don't see metrics after 1 week of working. Logs are stopping at this moment. this problem repeat on 7 different machines with different parameters

[jvstein]

@romaadereyko I need more information to help.

Are you able to view the metrics page in your browser or via curl? This will depend on how you are running the exporter, but typically something like curl http://localhost:9332/metrics/ should return the values.

If that works, but Prometheus can't return the metrics, then it's likely that Prometheus isn't scraping the exporter. The exporter does all of it's work at request time, so you won't see anything in the logs until a request is made.

[romaadereyko]

Yep, curl view all metrics and they are change after every request. I’ll try to screenshot this Всего наилучшего С уважением Адерейко Р.М. +375 29 3 800 787 А1 +375 33 3 800 787 МТС Пн, 2 дек. 2024 г. в 22:08, Jeff Stein ***@***.***>:

…

@romaadereyko <https://github.com/romaadereyko> I need more information to help. Are you able to view the metrics page in your browser or via curl? This will depend on how you are running the exporter, but typically something like curl http://localhost:9332/metrics/ should return the values. If that works, but Prometheus can't return the metrics, then it's likely that Prometheus isn't scraping the exporter. The exporter does all of it's work at request time, so you won't see anything in the logs until a request is made. — Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADI354QDRKLT2QAMZOR3WUT2DSVZLAVCNFSM6AAAAABS2UWN6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJSGUYDAMRXGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jvstein]

@romaadereyko Oh, so the metrics are collected successfully for a week and then they stop?

[romaadereyko]

Yes, it does. Вт, 3 дек. 2024 г. в 21:30, Jeff Stein ***@***.***>:

…

@romaadereyko <https://github.com/romaadereyko> Oh, so the metrics are collected successfully for a week and then they stop? — Reply to this email directly, view it on GitHub <#33 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADI354Q7T65TPUQVYQTFFID2DX2D7AVCNFSM6AAAAABS2UWN6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJVGMYDCMRYG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jvstein]

Does the exporter still respond with updated values via curl when they stop in Prometheus? Does the exporter return an error? How long does the request take?

If the exporter still responds, I would start looking at:

• The logs of your prometheus installation, maybe with --log.level=debug.
• The prometheus up and scrape_duration_seconds metrics from the prometheus server itself. A up == 0 would indicate problems contacting the exporter and a high scrape_duration_seconds might indicate a timeout when collecting metrics.

[romaadereyko]

No, exporter respond with not updated metrics
I'll setup debug level & I'll see metrics when container wouldn't respond with updated metrics

[romaadereyko]

Exporter respond with not updated metrics. After restart exporter responds with updated metrics
Exception.json

[jvstein]

The Connection reset by peer means something is interrupting the connection during the request.

Do you have a reverse proxy in front of the exporter? Is there a configurable timeout on it? The timestamps look close, but I don't know about how that's all logged.

I would recommend trying a different version of Python to see if it's reproducible. Either way, this seems like a defect in the runtime environment or the wsgiref web server and not something I can address here.