[SPIKE] Investigate Distributed Tracing

[kadel]

User story:
As a Backstage user, I want to be able to optionally enable distributed tracing in my instance and connect it to my distributed tracing stack so that I can debug and trace issues in a production environment.

Goal: Investigate what are our options and what is effort required to enable distributed tracing for Backstage and integrate it into the OpenShift Distributed tracing stack.

[jesuino]

For tracing OpenTelemetry is likely the technology and standard to be used. The tracing must be done in the application using OpenTelemetry APIs and later exported to some tool, such as Jaegar or Openshift Distributed Tracing.

I found that there are some OpenTelemetry in place in Backstage. It was added by these PRs:

OpenTelemetry was introduced a while ago in Backstage, but only for metrics:
backstage/backstage#15145

Later some actual tracing was added (1.17.0 release):
backstage/backstage#17534

However, it seems that tracing is not exported, so it is not exposed for comsuption. Here are some refs:

Backstage general Observability is discussed on the following EPIC:
backstage/backstage#1519

The epic 14518 is specific for tracing collector and tracing in general:
backstage/backstage#14518

Notice that epic 14518 is stale, so no timeline to when this will be implemented.

--

Conclusion: tracing export seems to be not available today in backstage. A possible solution could adding the tracing collector on midstream to confirm that the exported tracing information can be consumed by Openshift Distributed Tracing or Jaegar.

[rtaniwa]

@jasperchui @kadel if this is done, can I close this spike? Are there follow-on issues we should be opening that might track future work for in this space?

[kadel]

I'm going to close this and create a new epic to track future tracing related work.

[kadel]

I'm going to close this and create a new epic to track future tracing related work.

#632