[EPIC] Telemetry tracking issue

[liamsi]

Telemetry

We've started the telemetry integration process as outlined in #901, and this issue gathers all the sub-issues and/or PRs related to the integration/implementation process.

Base(d)

Scaffolding for further integration

• Decide on a metrics solution (e.g. Prometheus) and create lightweight spec #663
• feat(cmd): initial integration of OpenTelemetry with OTLP exporter #907
• cmd: OTLP TLS connection #934
• cmd: shutdown for OTel exporters #935
• cmd: add version of the Node with other metadata to OTel exporters attributes #936
• node: runtime switching for telemetry and profiling #937
• [Feature Request]: UX for telemetry endpoint #1003

Metrics Integration Spots

Actual code points for us to monitor/measure/observe/etc

• eds/retriever: Cover with metrics #639
• Monitoring for node.Store #98
• fraud: cover fraud package with metrics #999
• [Feature Request]: metrics for PayForData and last sample requested #1040
• Uptime metric

Tracing Integration Spots

• ipld: consider tracing for GetShares
• share: simple trace for Light (besides Full) Availability

[liamsi]

Key metrics of the top off my head:

• measure sampling time
• measure BlockSync time (per block and in total from height h, where h could be 0/genesis) for full nodes
• header sync time
• ....