What information is logged when otel.javaagent.debug is enabled?

[rkennedy-mode]

We have a service utilizing the Java agent, which is logging some root spans that appear to be disassociated from the actual trace they ought to be a part of. We're wondering how best to debug this and noticed the otel.javaagent.debug property mentioned in the Troubleshooting section of the repository README:

To turn on the agent's internal debug logging:

-Dotel.javaagent.debug=true

The documentation doesn't indicate, however, what sort of information may be found when debugging is enabled. So we're not sure if enabling this is likely to help us troubleshoot these detached spans (e.g. is there any hope that we might be able to get a stack trace associated with the start of the span to understand what cases are slipping past the instrumentation?).

For those curious, we see the following span names and associated instrumentation libraries in these detached spans:

name	library.name	library.version
S3.PutObject	io.opentelemetry.javaagent.aws-sdk-1.11	1.1.0
DynamoDBv2.UpdateItem	io.opentelemetry.javaagent.aws-sdk-1.11	1.1.0
HTTP POST	io.opentelemetry.javaagent.http-url-connection	1.1.0
HTTP GET	io.opentelemetry.javaagent.http-url-connection	1.1.0

[jkwatson]

This list is probably not comprehensive, but it's what I remember off the top of my head:

1. The logging span exporter is turned on, so you should see any created spans logged out.
2. The agent keeps some internal metrics about a few things (spans whose instrumentation has been suppressed, and a couple of other things). These metrics will get logged out periodically
3. Details of what instrumentation has been applied and what hasn't (this is very noisy and will make up most of the debug logs, IIRC)

[mateuszrzeszutek]

We have some context propagation debug logs, but that only happens when a SERVER span encounters a previous span in the current context. Maybe we should do a similar thing for INTERNAL/CLIENT spans that do not have a parent...

@rkennedy-mode are you able to reproduce that issue? Would it be possible to extract an example app that simulates this problem?

[rkennedy-mode]

We're not presently able to reproduce, which is why we're looking for additional debugging information. Having a stack trace taken at the time the span was created would potentially provide some benefit in understanding where these AWS and Bugsnag SDK calls are originating to see if we can figure out why there was no enclosing Trace available at the time of their creation.

The only useful information I see in these is the thread name. In the case of the DynamoDB client it's dynamodb-executor (presumably an ExecutorService internal to the DynamoDB client), which implies to me that some bit of asynchronous processing isn't properly inheriting trace details. Alternatively it implies that we have a code path to the DynamoDB client that lives outside of the other instrumentation mechanisms. This is a Dropwizard application and it's HTTP entry points are being picked up by both the Jetty and JAX-RS instrumentation. So this seems unlikely.