Update OTel for better tracing

[NeuralFlux]

linked to biothings/biothings_explorer#857

Note

requires new environment variable OTEL_EXPORTER_OTLP_ENDPOINT

[tokebe]

Is it at all possible to not require a new environment variable through exporter config? A new environment variable would require changes with ITRB.

[NeuralFlux]

@tokebe I didn't know that. I created a new var because I did not find any reference to the old one jaeger-otel-agent.sri. Main question is: does that var include the protocol or only has the hostname?

[tokebe]

The old variables were JAEGER_HOST and JAEGER_PORT, containing only the hostname and port, respectively. IIRC we don't actually rely on the environment variables and just fall back to the default, but want the ability to set them if they change in the future.

It looks like you can configure the exporter instance where it takes a url which is the combined host:port/endpoint. You should be fine just using string formatting and the existing variables/their defaults to create that.

[NeuralFlux]

The new exporter uses HTTP over UDP, which was used by the old exporter. So is it okay to change the default port and endpoint to 4318 and /v1/traces respectively? Jaeger ref

[tokebe]

Makes sense to me. I'll double check with the team in charge of the production jaeger instance.