Pods metrics endpoint

[SSvilen]

By default the pod metrics are getting scraped from the following kubelet endpoint:
/metrics/cadvisor

Which is more or less empty on windows machines.
I could workaroung that problem by creating new Service Monitor, which scrapes the mertics from the '/metrics/resource' endpoint

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/name: kubelet-windows
    k8s-app: kubelet-windows
  name: kubelet-windows
  namespace: openshift-monitoring
spec:
  endpoints:
    - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      bearerTokenSecret:
        key: ""
      honorLabels: true
      honorTimestamps: false
      interval: 30s
      path: /metrics/resource
      port: https-metrics
      relabelings:
        - sourceLabels:
            - __metrics_path__
          replacement: "/metrics/cadvisor"
          targetLabel: metrics_path
        - replacement: dummy
          targetLabel: image
      scheme: https
      scrapeTimeout: 30s
      tlsConfig:
        ca: {}
        caFile: /etc/prometheus/configmaps/kubelet-serving-ca-bundle/ca-bundle.crt
        cert: {}
  jobLabel: job
  targetLabels:
    - job
    - metrics_path
  namespaceSelector:
    matchNames:
      - kube-system
  selector:
    matchLabels:
      k8s-app: kubelet-windows

Now the CPU and memory consumption are visible in OpenShift Console.
Or is there another (built-in) way to achive that?

[aravindhp]

@SSvilen please provide your cluster details. Is this the on the same cluster? What does oc adm top pod return? Please provide a console screenshot.

[SSvilen]

@aravindhp ,

sorry, I believe I didn't describe the problem correctly. The top alway worked, because of the metrics server.
The problem is the representation in the console, more specifically the deployment metrics dashboard.

For the pod CPU metrics I modified the cluster-monitoring-operator-prometheus-rules Prometheus rule to the following:

 - expr: sum(rate(container_cpu_usage_seconds_total{container="",pod!=""}[5m])
        or rate(pod_cpu_usage_seconds_total{pod!=""}[5m])) BY (pod, namespace)

Now I have some metrics in the Console:

For the memory it seems not that trivial, because under Windows the metric for the whole pod is called:

pod_memory_working_set_bytes

So the problem is not critical, but it would be nice if at some point the same kind of features are available for both Linux und Windows Pods.

[mansikulkarni96]

@SSvilen thanks for opening the issue.
You are right, the pod metrics for Linux come from cAdvisor. However, we do not get same metrics for Windows, hence we plan to get them from the Windows exporter running on the node. This is something that we have in our pipeline and we plan to add support for displaying pod graphs in console soon.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[MattPOlson]

This still seems to be an issue, is there a plan to fix this in a future release? I was able to get metrics by creating the new service monitor as describe above and creating the following two new rules in windows-prometheus-k8s-rules PrometheusRule instance

    - expr: >-
        sum(rate(container_cpu_usage_seconds_total{pod!=""}[5m])) BY (pod,
        namespace)
      record: 'pod:container_cpu_usage:sum'
    - expr: |
        pod_memory_working_set_bytes
      record: container_memory_working_set_bytes

[mansikulkarni96]

@MattPOlson yes there is a plan to add support for pod metrics it is being tracked in JIRA here https://issues.redhat.com/browse/WINC-568

[sebsoto]

/reopen
So we can track this on Github as well

[openshift-ci]

@sebsoto: Reopened this issue.

In response to this:

/reopen
So we can track this on Github as well

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sebsoto]

/remove-lifecycle rotten

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[sebsoto]

/remove-lifecycle stale

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[alinaryan]

/remove-lifecycle rotten

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.