How to configure Prometheus exporter in Kubernetes?

[g0loob]

Hello 👋

Components

exporter/prometheusexporter

Problem

I can't see metrics from the Collector in the Prometheus UI. I suspect that my configuration is wrong, making Prometheus unable to scrape them, but I can't figure out what's wrong.

Setup

I'm using:

• OpenTelemetry Collector Helm chart
• kube-prometheus-stack Helm chart
• Jaeger Helm chart

to create a simple example of receiving OTLP telemetry data and export it to Jaeger and Prometheus.

I have a following override of values.yaml file for the Collector:

mode: "deployment"

fullnameOverride: "otel-collector"

presets:
  kubernetesAttributes:
    enabled: true

config:
  receivers:
    jaeger: null
    zipkin: null
  
  exporters:
    debug:
      verbosity: detailed
    
    otlp/jaeger:
      endpoint: jaeger-collector.default.svc.cluster.local:14250
      tls:
        insecure: true

    prometheus:
      endpoint: ${env:MY_POD_IP}:8889
  
  service:
    pipelines:
      traces:
        receivers:
          - otlp
        processors:
          - memory_limiter
          - batch
        exporters:
          - otlp/jaeger
      
      metrics:
        receivers:
          - otlp
          - prometheus
        processors:
          - memory_limiter
          - batch
        exporters:
          - debug
          - prometheus
      
      logs: null

ports:
  jaeger-compact:
    enabled: false
  jaeger-thrift:
    enabled: false
  jaeger-grpc:
    enabled: false
  zipkin:
    enabled: false
  metrics:
    enabled: true
    containerPort: 8889
    servicePort: 8889
    protocol: TCP

resources:
  limits:
    cpu: 2
    memory: 4Gi

ingress:
  enabled: false

serviceMonitor:
  enabled: true

Expectation

Once, the metrics port (8888) and the ServiceMonitor are enabled, I expected that Prometheus will be abled to scrape Collector's metrics. Unfortunately, I can't find any metric from the debug exporter in the Prometheus UI.

Additional information

• I'm avoiding using remote write receiver as it's an experimental feature.
• I'm avoiding using OTLP receiver as it's an experimental feature.
• When I port-forward metrics port 8888, I'm not able to reach localhost:8888/metrics endpoint and get Prometheus metrics.

---

Could you provide an example how to configure Collector with Prometheus? Or point out the mistakes in my configuration?

I managed to do it with "plain" Docker containers, but I'm having hard times configuring Kubernetes resources.

[g0loob]

The problem was that I was unaware that Prometheus CRD has a ServiceMonitor selector. By default, it only looked for ServiceMonitors which had release: prometheus label. After changing the configuration of ServiceMonitor to:

serviceMonitor:
  enabled: true

  extraLabels:
    release: prometheus

I was able to see the metrics in the UI.

[song2park]

Hi i'm using same structure.
What chart version are you using?

For me, it gives an error saying error decoding 'exporters': unknown type: "prometheus" for id: "prometheus" (valid values: [nop otlp otlphttp file loadbalancing otelarrow debug]) when trying to set prometheus in exporters

opentelemetry-collector-k8s docker image only start from 0.98.0.

[g0loob]

Hi, sorry but I don't remember. Maybe 0.90.0 (whichever was latest in February). It was just for PoC, it's not something we frequently use.
Also, I used Helm charts, not Docker images i.e. opentelemetry-collector-k8s.

Perhaps they changed something in the meantime.

[song2park]

Thank you for the reply.
I'm using Helm chart too
think you're right they changed something to do with pipelines and the previous images older than 0.98.0 not available atm.

I solved it by replacing image for the chart from opentelemetry-collector-k8s to opentelemetry-collector-contrib as I don't need collect all of k8s resources but my own apps in the cluster.