node_exporter_metrics complains about "read error, check permissions: /devices/system/cpu/cpu[0-9]*"

[daudo]

Describe the bug
I am trying to use the node_exporter_metrics plugin to retrieve some host metrics (and subsequently send them to opensearch, but that doesn't matter here).

In order to do so, I have the following docker compose configuration:

services:
[...]
  fluent-bit:
[...]
    image: fluent/fluent-bit:latest
    container_name: fluent-bit
    volumes:
      - "./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro"
      - "./parsers.conf:/fluent-bit/etc/parsers.conf:ro"
      - "./journal_transform.lua:/fluent-bit/scripts/journal_transform.lua:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"  # this is for docker
      - /var/log:/host/var/log:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /run/log/journal:/run/log/journal:ro
      - /etc/machine-id:/etc/machine-id:ro
    environment:
[...]
      path.procfs: /host/proc
      path.sysfs: /host/sysfs
    ports:
      - "2020:2020"  # Fluent Bit metrics endpoint
    privileged: true # just to be sure v1
    user: root # just to be sure v2

the relevant part of the fluent bit conf looks like this:

[...]
# Host Metrics Collection
[INPUT]
    Name            node_exporter_metrics
    Tag             node.metrics
    Scrape_Interval 10
    path.procfs     /host/procfs
    path.sysfs      /host/sysfs
    # Available metrics groups
    Metrics         cpu,memory,network,disk,diskio,process
[...]

... but no CPU metrics are being collected.

Instead, after starting up fluent bit, I see log messages like below:

[2024/11/06 22:37:47] [ info] [fluent bit] version=3.1.10, commit=e28f447995, pid=1
[2024/11/06 22:37:47] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/11/06 22:37:47] [ info] [cmetrics] version=0.9.7
[2024/11/06 22:37:47] [ info] [ctraces ] version=0.5.6
[2024/11/06 22:37:47] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] initializing
[2024/11/06 22:37:47] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] storage_strategy='memory' (memory only)
[2024/11/06 22:37:47] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.procfs = /host/procfs
[2024/11/06 22:37:47] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.sysfs  = /host/sysfs
[2024/11/06 22:37:47] [error] [input:node_exporter_metrics:node_exporter_metrics.0] read error, check permissions: /devices/system/cpu/cpu[0-9]*
[2024/11/06 22:37:47] [error] [/src/fluent-bit/plugins/in_node_exporter_metrics/ne_utils.c:170 errno=2] No such file or directory
[2024/11/06 22:37:47] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] thread instance initialized
[2024/11/06 22:37:47] [ info] [input:docker_events:docker_events.1] initializing
[2024/11/06 22:37:47] [ info] [input:docker_events:docker_events.1] storage_strategy='memory' (memory only)
[2024/11/06 22:37:47] [ info] [input:docker_events:docker_events.1] listening for events on /var/run/docker.sock
[2024/11/06 22:37:47] [ info] [input:docker:docker.2] initializing
[2024/11/06 22:37:47] [ info] [input:docker:docker.2] storage_strategy='memory' (memory only)
[2024/11/06 22:37:47] [ info] [input:docker:docker.2] Detected cgroups v2
[2024/11/06 22:37:47] [ info] [input:systemd:systemd.3] initializing
[2024/11/06 22:37:47] [ info] [input:systemd:systemd.3] storage_strategy='memory' (memory only)
[2024/11/06 22:37:47] [ warn] [input:systemd:systemd.3] seek_cursor failed
[2024/11/06 22:37:47] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/11/06 22:37:47] [ info] [sp] stream processor started
[2024/11/06 22:37:57] [error] [input:node_exporter_metrics:node_exporter_metrics.0] read error, check permissions: /devices/system/cpu/cpu[0-9]*
[2024/11/06 22:37:57] [error] [/src/fluent-bit/plugins/in_node_exporter_metrics/ne_utils.c:170 errno=2] No such file or directory
[2024/11/06 22:38:07] [error] [input:node_exporter_metrics:node_exporter_metrics.0] read error, check permissions: /devices/system/cpu/cpu[0-9]*
[2024/11/06 22:38:07] [error] [/src/fluent-bit/plugins/in_node_exporter_metrics/ne_utils.c:170 errno=2] No such file or directory
[2024/11/06 22:38:17] [error] [input:node_exporter_metrics:node_exporter_metrics.0] read error, check permissions: /devices/system/cpu/cpu[0-9]*
[2024/11/06 22:38:17] [error] [/src/fluent-bit/plugins/in_node_exporter_metrics/ne_utils.c:170 errno=2] No such file or directory
[2024/11/06 22:38:27] [error] [input:node_exporter_metrics:node_exporter_metrics.0] read error, check permissions: /devices/system/cpu/cpu[0-9]*
[2024/11/06 22:38:27] [error] [/src/fluent-bit/plugins/in_node_exporter_metrics/ne_utils.c:170 errno=2] No such file or directory
[2024/11/06 22:38:37] [error] [input:node_exporter_metrics:node_exporter_metrics.0] read error, check permissions: /devices/system/cpu/cpu[0-9]*
[2024/11/06 22:38:37] [error] [/src/fluent-bit/plugins/in_node_exporter_metrics/ne_utils.c:170 errno=2] No such file or directory
[2024/11/06 22:38:47] [error] [input:node_exporter_metrics:node_exporter_metrics.0] read error, check permissions: /devices/system/cpu/cpu[0-9]*
[...]

Expected behavior
I'd expect the CPU metrics getting collected without an error in the logs.

Your Environment

• Version used: 3.1.10 fluent/fluent-bit:latest from docker hub
• Environment name and version (e.g. Kubernetes? What version?): docker 27.3.1
• Operating System and version: image from docker hub, debian 12.7 as the host system

Additional context
What puzzles me is the path the log message contains: /devices/system/cpu/cpu[0-9]*

Because this path does not exist obviously, it either has to be /sys/devices/system/cpu/cpu[0-9]* or in my case
/host/sys/devices/system/cpu/cpu[0-9]*
But maybe the log message just doesn't print the full path, I don't fully understand the code here.