Monitoring and Observability

[gorkem]

Goal

Enable reporting on the overall health of the portal. Provide monitoring for crucial system metrics. Introduce tooling to help understand and debug issues in production. Enable data that help trace, understand, and diagnose problems in production environments.

Some of the areas of work are

• Metrics (Prometheus)
• Logging
• Health Checks
• (Distributed) Tracing (moved to separate epic Support for distributed tracing #632)

Acceptance criteria

• Users can enable their Backstage instance to output metrics in a format that can be pulled in by Prometheus.
• User can enable structured logging and can send them to log repositories
• Health Checks for the backstage instance is available
• Users can optionally enable Tracing (moved to separated epic Support for distributed tracing #632)
• Documentation is available

Requirements

• Plugin
• Test plan
• Installation documentation
• End user documentation

Issues in Epic

• Enable Prometheus metrics output #230
• Integration with Openshift Logging #500
• Enable Health Checks #565
• Users can optionally enable Tracing
  • [SPIKE] Investigate Distributed Tracing #582

Notes

Additional context

[kadel]

A few notes from the discussion with Christophe:

• The important part is monitoring. Ideally, it should integrate with OpenShift monitoring, but users should also be able to use their own Prometheus instance.
• If there is no time, tracing and logging can wait after Milestone 4
• Similarly to monitoring tracing should integrate with OpenShift tracing (Jeager and OpenTelemetry)
• Logging should also integrate with OpenShift logging.

Issues in Epic

• Enable Prometheus metrics output #230
• add config option to optionally create ServiceMonitor resource helm-backstage#96

[rtaniwa]

@kadel assuming the spike is complete, is this epic now complete (all acceptance criteria met)? Do we have documentation that the doc team can use to seed their content?

[kadel]

@kadel assuming the spike is complete, is this epic now complete (all acceptance criteria met)? Do we have documentation that the doc team can use to seed their content?

Except for the tracing all acceptance criteria a met. Upstream documentation is in Getting started guide (https://github.com/janus-idp/backstage-showcase/blob/main/showcase-docs/getting-started.md)

I kept this open to remind myself that I needed to create a new epic to capture the tracing part, which we will do later.

[kadel]

I've created a separate epic (#632) for tracking Tracing related work, which we will do later.