OTLP/gRPC or OTLP/HTTP sink

[rhys-evans]

Hi

I would like to send OTLP data (in particular trace data, for my use case) to any OTLP compatible backend.

Would this be supported by an HTTP sink "extension" ?

Ideally for me this would take batched trace data from S3 eg

OTEL Collector example: (not complete)

    receivers:
      otlp:
        protocols:
          http:
            endpoint: 0.0.0.0:4318
            cors:
              allowed_origins:
                - "http://*"
                - "https://*"

    exporters:
      awss3:
        s3uploader:
            region: us-west-2
            s3_bucket: "tempo-traces-bucket"
            s3_prefix: 'metric'
            s3_partition: 'minute'

    processors:
      batch:
        send_batch_size: 10000
        timeout: 30s
      resource:
        attributes:
        - key: service.instance.id
          from_attribute: k8s.pod.uid
          action: insert
      memory_limiter:
        check_interval: 5s
        limit_mib: 200

    service:
      pipelines:
        traces:
          processors: [memory_limiter, resource, batch]
          exporters: [awss3, spanmetrics]

Dataprepper pipeline config example idea:

traces-pipeline:
  workers: 2
  delay: "100"
  source:
    s3:
        notification_type: sqs
        compression: automatic
        sqs:
            queue_url: "https://sqs.eu-west-1.amazonaws.com/<account_id>/queue"
        aws:
            region: "us-west-2"
  buffer:
    bounded_blocking:
      buffer_size: 4096
      batch_size: 512
  processor:
    - parse_json:
  sink:
    - otlp:
        endpoint: http://otlp-backend

Thoughts ?

There seems to have been a attempt (I have not tested it) to do this in logstash here https://github.com/paulgrav/logstash-output-opentelemetry/blob/main/README.md

[KarstenSchnitter]

The main challenge will be to map events to the proper format. OpenTelemetry uses a complex data models for each of the supported signals: logs, metrics and traces. The resource linked by @rhys-evans only covers logs, which are currently the easiest format to generate. It is missing the instrumentation scope entirely, too.

An OpenTelemetry sink most likely requires complex configuration on field mappings and aggregations. Specification of the intended signal type is required too, since OpenTelemetry uses different endpoints for each one of those. Switching between gRPC and http/protobuf transport is another feature and requires implementation of two (not so) different clients. The configuration might look similar to the OpenTelemetry Collector:

sink:
  - otlp/logs:
    network:
      endpoint: otlp-backend
      tls:
        cert: cert.pem
        key: key.pem
    resource:
      service.name: field_key
    instrumentation:
      scope: field_key
    attribute:
      name: field_key
    body: field_key

However, there will be a lot more details. E.g. all (resource) attributes and the body can be of several types: string, integer, hash,...
This is a challenge for parsing arbitrary data.

I wonder, whether this feature is not already provided in a more mature form by the OpenTelemetry Collector.

[rhys-evans]

Hi, thanks for the response.

There are a few things I was looking to cover off with this usecase, and they seem to be mostly non technical.

1. Use of 1 tool to ingest our logs and traces, as it would be nice to consolidate down to one tool for this (easier learning curve, support etc) (our logs go to OpenSearch, our traces need to go elsewhere (for reasons))
2. The otel-collector is missing the s3 event notification -> sqs workflow, however this may be easier to implement there
3. I was going to look at the http sink, documented here https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/http-sink , but that doesn't seem to have made it to prime time yet. But will spend some time looking how to simulate it

So our work flow is teams collect their own traces, logs etc, they then "process" them and store them in S3 / GCS, these S3 objects (which may contain multiple events per type) are then ingested, normalised and made visible to other consumers centrally. These central tools (storage and visualisation) may change over time but the desire is to align with the opentelemetry standards moving forward, as this seems to be the direction of travel of the "general" industry at present

But it does pose the question of where dataprepper is going, is it aligned only to the opensearch ecosystem, primarily to the opensearch ecosystem or as a general ingestion and transformation tool to be used across multiple products.

Note I am not saying 1 tool should cover all usecases, just trying to understand

Thanks

[dlvenable]

@rhys-evans ,

This is an interesting proposal. I don't think we'd have a general gRPC sink for the reasons @KarstenSchnitter expressed. We could possibly have a sink for OTel and maybe you configure which model or models it supports.

You are correct that the http sink is not currently ready and would need some work to get going. I also think it would not work for this particular OTel use-case because it would be attempting to send more generic HTTP messages rather than the OTel messages.

As far as where Data Prepper is going, I think we are focused more on working within the OpenSearch ecosystem and ingestion into OpenSearch. But, we believe that operating with tangential systems is also important. For example, being able to write logs to S3 as well is important. And we'd like to support Prometheus because many users run OpenSearch and Prometheus together. Similarly, working within the OTel ecosystem aligns with this. I'd be happy to accept PRs that provide some of these behaviors you've requested.

[KarstenSchnitter]

I am happy to assist with PRs providing an OpenTelemetr< sink. Additionally, I can see a use-cases for exporting data from OpenSearch to OpenTelemetry.