[receiver/datadog] Span Obfuscation

[lopes-felipe]

Description

Datadog Agent's span processor has an obfuscation pipeline that—among other things—obfuscates sensitive data, cleans up span resource names, and extracts particular components into specific span metadata.

This PR implements the same behavior into the datadogreceiver, but disabled by default for retro compatibility with existing users.

Link to tracking issue

Fixes

Testing

Documentation

[MovieStoreGuy]

@lopes-felipe,

Have you checked to see if any of the existing processors would provide this functionality?

I am hesitant to allow adding this into the receiver when it could already be achieved with existing functionality. There is also the added concern that this component is effectively bundling the datadog agent into the collector which doesn't sound ideal.

[lopes-felipe]

@lopes-felipe,

Have you checked to see if any of the existing processors would provide this functionality?

I am hesitant to allow adding this into the receiver when it could already be achieved with existing functionality. There is also the added concern that this component is effectively bundling the datadog agent into the collector which doesn't sound ideal.

Hey @MovieStoreGuy,

Thank you for the review. Yes, I have checked all the existing processors, and none of them have the capability to clean or/and obfuscate the span data in such a context-aware/structured way as it's done here.

For instance, like this simple test demonstrates, it takes a SQL query with comments, aliases, and repetitions (ref), and turn it into a flat, un-aliased, un-commented, deduplicated string (ref), something that is just not possible with any other processor currently available.

I would be more than happy to propose something like this for an existing processor such as Span Processor though, but I guess we could first prove the value of such a focused processing approach as this in a more narrowed-scope component such as Datadog Receiver here.

Finally, regarding bundling the Datadog Agent into the Collector, please correct me if I'm wrong here, but I don't think this particular feature implementation is moving this (very reasonable) concern any further, considering what has already taken place by the Datadog Exporter, and for the very own Datadog Receiver. Still, again, I might be missing something here.

Thank you again for the thoughtful review :)

[lopes-felipe]

Hey @MovieStoreGuy, would you mind taking another look at this, considering now my previous comment? Thank you :)

[MovieStoreGuy]

Hey @lopes-felipe ,

Thank you for the review. Yes, I have checked all the existing processors, and none of them have the capability to clean or/and obfuscate the span data in such a context-aware/structured way as it's done here.

Have you looked at https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/redactionprocessor ?

I am trying to conscious of is the Datadog receiver doing too much and tightly coupling a receivers role and a processors role. This historical made for a bad experience and has introduced performance issues in those components.

Realistically I would expect this work to be done within the above redaction processor with configuration defined.

[MovieStoreGuy]

Removing this from the root package will break some tests and generated code. It is expected to be defined here.

[lopes-felipe]

Hey @MovieStoreGuy,

I am trying to conscious of is the Datadog receiver doing too much and tightly coupling a receivers role and a processors role. This historical made for a bad experience and has introduced performance issues in those components.

I see, and that definitely makes sense. Considering that, I have put up this issue to discuss this feature in a more broad, vendor-agnostic approach to the Span Processor.

I think we can close this now. Thanks.