You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We have recently started adding OpenTelemetry support to our code base with traces. In addition to traces, we should add support for OpenTelemetry logs.
Add logging
Adding OpenTelemetry logs will require a couple of changes:
Add an OpenTelemetry logger provider
Replace or update the service logger implementation to emit OpenTelemetry formatted logs
We should preserve logging to stdout and file-based logging for non-interactive processes, such as when we run a service in a Docker container.
Logs should be associated with a trace where possible. Let's research the mechanism for doing this and start by associating logs with the run_job trace.
Export logs
We should export logs for services we operate to our telemetry infrastructure. Services where we should collect include the solver, job creator, and resource provider on Testnet and Devnet. We should avoid collecting logs from resource providers and CLI/onchain job originators that we do not operate.
Let's use Fluent Bit to collect logs and send them to our telemetry infrastructure. We could use Fluent Bit's OpenTelemetry input and send logs directly to Fluent Bit running alongside our service. Alternatively, we could use the Tail input to tail file-based logs.
If we use the OpenTelemetry input, we should add an option to enable/disable sending logs directly to the Fluent Bit process.
Note that it should be possible to configure Fluent Bit to send logs to telemetry infrastructure and write the logs to a file.
The text was updated successfully, but these errors were encountered:
We have recently started adding OpenTelemetry support to our code base with traces. In addition to traces, we should add support for OpenTelemetry logs.
Add logging
Adding OpenTelemetry logs will require a couple of changes:
We should preserve logging to
stdout
and file-based logging for non-interactive processes, such as when we run a service in a Docker container.Logs should be associated with a trace where possible. Let's research the mechanism for doing this and start by associating logs with the
run_job
trace.Export logs
We should export logs for services we operate to our telemetry infrastructure. Services where we should collect include the solver, job creator, and resource provider on Testnet and Devnet. We should avoid collecting logs from resource providers and CLI/onchain job originators that we do not operate.
Let's use Fluent Bit to collect logs and send them to our telemetry infrastructure. We could use Fluent Bit's OpenTelemetry input and send logs directly to Fluent Bit running alongside our service. Alternatively, we could use the Tail input to tail file-based logs.
If we use the OpenTelemetry input, we should add an option to enable/disable sending logs directly to the Fluent Bit process.
Note that it should be possible to configure Fluent Bit to send logs to telemetry infrastructure and write the logs to a file.
The text was updated successfully, but these errors were encountered: