New component: Apache Druid Receiver

[yuanlihan]

The purpose and use-cases of the new component

To be able to stream metrics, logs and alerts directly from the HTTP Emitter of Apache Druid.

Example configuration for the component

receivers:
  apachedruid:
    endpoint: 0.0.0.0:9000
    metrics_path: '/services/collector/metrics'
    logs_path: '/services/collector/logs'

Telemetry data types supported

Metrics and Logs.

Is this a vendor-specific component?

• This is a vendor-specific component
• If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

@yuanlihan

Sponsor (optional)

@atoulme

Additional context

Apache Druid is a high performance, real-time analytics database that delivers sub-second queries on streaming and batch data at scale and under load.

It'll be great to integrate Apache Druid with OpenTelemetry ecosystem. I'm ready to submit the first PR of the new component if the proposal looks good to the team!

[yuanlihan]

Hmm, I found a similar issue #8466 but it was closed.

Hi @dmitryax 👋, it'll be really great if you're still interested in sponsoring this component.

[crobert-1]

Removing needs triage based on the fact that this was already an accepted component, just not implemented. Details of this receiver still need discussed.

[atoulme]

What metrics would you expose? What logs? Would the collector scrape metrics or would there be a mechanism in place to receive them?

[yuanlihan]

@atoulme Thanks for your input!

The metrics to be exposed will be https://druid.apache.org/docs/latest/operations/metrics and the logs will include request logs and alert logs, see https://druid.apache.org/docs/latest/operations/request-logging and https://druid.apache.org/docs/latest/operations/alerts.

Would the collector scrape metrics or would there be a mechanism in place to receive them?

Good point. In this case, I think we'll need a new receiver component which includes HTTP listeners to receive metrics and logs emitted by Druid, see more about HTTP emitter module.

[atoulme]

The metric names don't match semantic conventions of OpenTelemetry, in particular the use of /. It would be good to identify a subset you want to start with. OK for HTTP, we have other components that open a HTTP port and receive data.

[yuanlihan]

The metric names don't match semantic conventions of OpenTelemetry, in particular the use of /.

That's true. I tried script to batch converting them to OpenTelemetry metrics to get the metadata.yaml needed, see also https://github.com/apache/druid/blob/master/docs/operations/metrics.md.

[yuanlihan]

Hmm, I found a similar issue #8466 but it was closed.

Hi @dmitryax 👋, it'll be really great if you're still interested in sponsoring this component.

Gentle bump @dmitryax

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[atoulme]

I can sponsor this receiver.

[yuanlihan]

@atoulme Great! I'm working on the first PR, and hopefully, it'll be ready for review this week or early next week.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.

[naman47vyas]

Wonder how to implement a druid receiver. Basic high level steps would include:

1. Make an HTTP server in the receiver as druid emits the metrics unlike many other receivers where we query some status variables. This server would listen to metrics emitted by druid.
2. We would need some metadata about the metrics that need scrapping. Druid's metric page does have the metadata info but, information regarding the attributes(dimensions in druid's terminology), units and type of metrics is absent.
3. Once we have the Druid's metrics and its corresponding metadata info, we would need to translate the metrics from druid's format to otel format.
4. Export the metrics.

I would love to carry forward the work for this issue. Though I might need some guidance as well. Looking for sponsors as of now.

[atoulme]

I am still available to sponsor.

[naman47vyas]

Cool, I do have a very hacky solution already in place, I'll start the development promptly on the proper solution.

[atoulme]

Please make sure you read CONTRIBUTING.md. The first PR is a simple skeleton. The existing PR that wasn't merged was almost there.

[markdavidmc0]

Is anyone still working on this?
I might be interested in supporting depending on what's in progress.