Job label of loki prometheus metrics

[kimxogus]

I have installed loki distributed using helm chart with service monitor, and grafana agent (as a daemonset) with service monitor config below.

      // servicemonitor
      prometheus.operator.servicemonitors "default" {
        forward_to = [prometheus.remote_write.default.receiver]
        rule {
          action        = "keep"
          regex         = env("HOSTNAME")
          source_labels = ["__meta_kubernetes_pod_node_name"]
        }
        selector {
          match_expression {
            key      = "release"
            operator = "In"
            values   = ["prometheus"]
          }
        }
      }

I found that loki metrics' job label is different between prometheus operator and grafana agent.

• From prometheus

{__name__="loki_build_info", branch="HEAD", cluster="CLUSTER_NAME", component="distributor", goarch="amd64", goos="linux", goversion="go1.21.3", instance="IP:3100", job="monitoring/distributor", namespace="monitoring", pod="loki-distributed-distributor-...", prometheus="...", prometheus_replica="...", revision="a17308db6", tags="netgo", version="2.9.2"}

• From grafana agent

{__name__="loki_build_info", branch="HEAD", cluster="CLUSTER_NAME", container="distributor", endpoint="http", goarch="amd64", goos="linux", goversion="go1.21.3", instance="IP:3100", job="loki-distributed-distributor", namespace="monitoring", pod="loki-distributed-distributor-xxx", revision="a17308db6", service="loki-distributed-distributor", tags="netgo", version="2.9.2"}

Is there a missing feature in grafana agent? or am i missing some config?

[rfratto]

It looks like our translation of the ServiceMonitor to labels is different than the version of Prometheus Operator you're using. We used to be in sync, but we might've drifted since then.

cc @captncraig

[kimxogus]

Grafana loki's official grafana dashboard is based on $namespace/{shorten job name} format.
I think this should be in sync for dashboard compatibilities.
https://github.com/grafana/loki/blob/main/production/loki-mixin-compiled/dashboards/loki-operational.json#L665