Spans for long running tasks - can we send span events out as they happen?

[adriangb]

Spans fall short when modeling long-running events because they can't be sent until the event finishes. Child spans do get sent, but they lack any information (attributes, start time, etc.) from their parent. Nonetheless, this is still useful because at least you get some information. Unfortunately, this is not true for span events: if you add an event to a span it won't get sent until the span completes. This means that span events are in some ways inferior to child spans or just emitting an out-of-band log with correlation.

It would be really interesting if we could greedily send out span events as they occur. Further, if we could have some sort of semantic convention around an event for "a span was started but is still ongoing" then we could also improve the situation where you can get no information on a long-running span until it completes.

[jack-berg]

We currently have an experimental event API which sends events as log records. There are several open questions related to the event API, but one of them is how they interact with span events? My opinion is that after we work out some of the other issues with the event API and get it stable, that we should deprecate the span event API and encourage folks to directly use the event API. Users would then be able to configure the SDK on whether events recorded in the context of a span are attached to the span as span events when exported, or exported independently as log events, with trace context fields tying back to the related span.

We're working on spinning up an event working group to discuss these questions and more.

[adriangb]

Nice, thanks for the reply!

It sounds like what you are saying is that:

1. Exporting these sort of "span started" events via logs/events makes more sense than as spans.
2. In the future there may be a specific event for this (assuming something like name=span-started,domain=opentelemetry).
3. The SDKs may be configured so that the existing methods on spans use this new event (which would be emitted immediately) instead of appending it to the span and waiting until that span is emitted.

Is this interpretation correct?

[adriangb]

Quick question for you @jack-berg: is there anything in the spec stopping collectors, exporters, etc. from doing something like dropping an event but not the corresponding span or the span but not the corresponding event? If not that seems like it would be an immediate problem in the face of something like sampling.

[jack-berg]

No there is not. There have been discussions about introducing a log sampler concept to parallel trace samplers. If we did such a thing, it would be natural to:

• Introduce a built in sampler which only records log records if the corresponding span is sampled
• Retain all log records by default (for backwards compatibility and to align with user expectations)

Even with the presence of such a sampler, OpenTelemetry gives you ample ability to shoot yourself in the foot. It will always be possible to produce log records where the associated span isn't sampled, or spans where the associated log records are not sampled. This is fine - things should work sensibly by default but artificially restricting this type assumes to much about use cases and would ultimately frustrate users.