[Feature] _nodes/stats/os support for cgroup v2

[jondryk]

Describe the bug

After upgrading to GKE version 1.26 (view changes), cgroup v2 has become the default, resulting in _nodes/stats/os not returning cgroup metrics.

Related component

Other

To Reproduce

1. Configure k8s cluster with cgroup v2 enabled
2. Spin up opensearch-k8s-operator and OpenSearch cluster by using guide (link)
3. Get information from _node/stats API, by using OpenSearch Dashboard (Dev Tools): GET _node/stats/os
4. Get response without cgroup section:

{
  "_nodes": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "cluster_name": "quickstart",
  "nodes": {
    "cZANEenxTXms8yIFkb9_fQ": {
      "timestamp": 1711552940692,
      "name": "quickstart",
      "transport_address": "10.201.8.2:9300",
      "host": "quickstart",
      "ip": "10.201.8.2:9300",
      "roles": [
        "cluster_manager"
      ],
      "attributes": {
        "shard_indexing_pressure_enabled": "true"
      },
      "os": {
        "timestamp": 1711552940692,
        "cpu": {
          "percent": 36,
          "load_average": {
            "1m": 1.28,
            "5m": 1.17,
            "15m": 1.08
          }
        },
        "mem": {
          "total_in_bytes": 2147483648,
          "free_in_bytes": 1716224,
          "used_in_bytes": 2145767424,
          "free_percent": 0,
          "used_percent": 100
        },
        "swap": {
          "total_in_bytes": 0,
          "free_in_bytes": 0,
          "used_in_bytes": 0
        }
      }
    },

Expected behavior

Should retrieve response with cgroup (documentation) from GET _node/stats/os. Example response:

{
  "_nodes": {
    "total": 6,
    "successful": 6,
    "failed": 0
  },
  "cluster_name": "quickstart",
  "nodes": {
    "NBKECfQPSaSUXrVzUx_ykw": {
      "timestamp": 1711558029057,
      "name": "quickstart",
      "transport_address": "10.241.8.11:9300",
      "host": "10.241.8.11",
      "ip": "10.241.8.11:9300",
      "roles": [
        "master"
      ],
      "os": {
        "timestamp": 1711558029057,
        "cpu": {
          "percent": 5,
          "load_average": {
            "1m": 0.83,
            "5m": 0.76,
            "15m": 0.7
          }
        },
        "mem": {
          "total_in_bytes": 4294967296,
          "adjusted_total_in_bytes": 4294967296,
          "free_in_bytes": 1336111104,
          "used_in_bytes": 2958856192,
          "free_percent": 31,
          "used_percent": 69
        },
        "swap": {
          "total_in_bytes": 0,
          "free_in_bytes": 0,
          "used_in_bytes": 0
        },
        "cgroup": {
          "cpuacct": {
            "control_group": "/",
            "usage_nanos": 255341381158
          },
          "cpu": {
            "control_group": "/",
            "cfs_period_micros": 100000,
            "cfs_quota_micros": 200000,
            "stat": {
              "number_of_elapsed_periods": 28171495,
              "number_of_times_throttled": 433,
              "time_throttled_nanos": 12260969
            }
          },
          "memory": {
            "control_group": "/",
            "limit_in_bytes": "4294967296",
            "usage_in_bytes": "2958856192"
          }
        }
      }
    }
  }
}

Additional Details

Plugins
prometheus-exporter-plugin-for-opensearch

Host/Environment (please complete the following information):
OS:

• GKE version: 1.27
• node version: 1.27.11-gke.1062000
• image type: UBUNTU_CONTAINERD

OpenSearch:

• opensearch-k8s-operator version: 2.4.0
• opensearch-cluster version: 2.11.1

Additional context

[peternied]

[Triage - attendees 1 2 3 4 5 6]
@jondryk We don't believe that CGroupV2 is supported - we love to see a pull request to add this support

[peternied]

[Triage - attendees 1 2 3 4 5 6]

@jondryk Note; when making a pull request please do not use or reference code that is not Apache 2.0 license (or similar license)