Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add OpenTelemetry logging #303

Open
bgins opened this issue Aug 23, 2024 · 0 comments
Open

Add OpenTelemetry logging #303

bgins opened this issue Aug 23, 2024 · 0 comments

Comments

@bgins
Copy link
Contributor

bgins commented Aug 23, 2024

We have recently started adding OpenTelemetry support to our code base with traces. In addition to traces, we should add support for OpenTelemetry logs.

Add logging

Adding OpenTelemetry logs will require a couple of changes:

  • Add an OpenTelemetry logger provider
  • Replace or update the service logger implementation to emit OpenTelemetry formatted logs

We should preserve logging to stdout and file-based logging for non-interactive processes, such as when we run a service in a Docker container.

Logs should be associated with a trace where possible. Let's research the mechanism for doing this and start by associating logs with the run_job trace.

Export logs

We should export logs for services we operate to our telemetry infrastructure. Services where we should collect include the solver, job creator, and resource provider on Testnet and Devnet. We should avoid collecting logs from resource providers and CLI/onchain job originators that we do not operate.

Let's use Fluent Bit to collect logs and send them to our telemetry infrastructure. We could use Fluent Bit's OpenTelemetry input and send logs directly to Fluent Bit running alongside our service. Alternatively, we could use the Tail input to tail file-based logs.

If we use the OpenTelemetry input, we should add an option to enable/disable sending logs directly to the Fluent Bit process.

Note that it should be possible to configure Fluent Bit to send logs to telemetry infrastructure and write the logs to a file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

3 participants