discovery.kubernetes does not provide any targets on OpenShift

[m4d53]

Hi there,

we're currently trying to grab the logs of our pods on OpenShift over the K8S api with the grafana agent in flow mode and thats where our issues begin: discovery.kubernetes lists no targets.

Here is our code:

config.river:

logging {
    level  = "debug"
    format = "logfmt"
    write_to = [loki.write.xyz.receiver]
}

discovery.kubernetes "pods" {
    role = "pod"
}

discovery.kubernetes "services" {
    role = "service"
}

loki.write "xyz" {
    endpoint {
        url = "http://xyz-loki:3100/loki/api/v1/push"
    }
}

loki.source.kubernetes "pods" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [loki.write.xyz.receiver]
  clustering {
    enabled = false
  }
}

serviceaccount.yaml:

kind: ServiceAccount
apiVersion: v1
metadata:
  name: xyz-monitoring
---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: xyz-prometheus-monitor
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: xyz-app-monitoring-role
rules:
  - verbs:
      - get
      - watch
      - list
    apiGroups:
      - ''
    resources:
      - pods
      - pods/log
      - services
      - routes
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: xyz-app-monitoring
subjects:
  - kind: ServiceAccount
    name: xyz-prometheus-monitor
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: xyz-app-monitoring-role

deployment.yaml:

kind: Deployment
apiVersion: apps/v1
metadata:
  name: xyz-grafana-agent
  labels:
    app: xyz-grafana-agent
    app.kubernetes.io/part-of: monitoring
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  selector:
    matchLabels:
      app: xyz-grafana-agent
  template:
    metadata:
      labels:
        app: xyz-grafana-agent
      annotations:
        xyz.justiz.de/type: 'monitoring'
    spec:
      serviceAccount: xyz-monitoring
      imagePullSecrets:
      - name: image-pull-secret
      containers:
      - name: xyz-grafana-agent
        image: 'atosjustiz.azurecr.io/infra-tools/grafana-agent'
        env:
        - name: AGENT_MODE
          value: flow
        - name: LOKI_HOST
          value: xyz-loki:3100
        command:
        - /bin/grafana-agent
        - run
        - --server.http.listen-addr=0.0.0.0:12345
        - /etc/agent/config.river
        - --disable-reporting
        ports:
        - name: http
          containerPort: 12345
          protocol: TCP
        readinessProbe:
          httpGet:
            path: /-/ready
            port: 12345
          initialDelaySeconds: 10
          timeoutSeconds: 1
        imagePullPolicy: Always
        volumeMounts:
        - name: config
          mountPath: /etc/agent
        - name: data-agent
          mountPath: /data-agent
      volumes:
      - name: config
        configMap:
          name: grafana-agent-flow
      - name: data-agent
        emptyDir: {}
      securityContext:
        runAsNonRoot: true

Where are we failing? :(

[m4d53]

We found the issue at the service account level, the service account for the pod was forgotten at the binding level.