thermal_zone collector stuck on Jetson Orin Nano

[gouthamve]

Host operating system: output of uname -a

Linux jetson 5.15.136-tegra #1 SMP PREEMPT Wed Apr 24 19:36:48 PDT 2024 aarch64 aarch64 aarch64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.8.1 (branch: HEAD, revision: 400c3979931613db930ea035f39ce7b377cdbb5b)
  build user:       root@7afbff271a3f
  build date:       20240521-18:36:53
  go version:       go1.22.3
  platform:         linux/arm64
  tags:             unknown

node_exporter command line flags

./node_exporter

node_exporter log output

Expand logs

ts=2024-07-07T12:37:02.195Z caller=node_exporter.go:193 level=info msg="Starting node_exporter" version="(version=1.8.1, branch=HEAD, revision=400c3979931613db930ea035f39ce7b377cdbb5b)"
ts=2024-07-07T12:37:02.195Z caller=node_exporter.go:194 level=info msg="Build context" build_context="(go=go1.22.3, platform=linux/arm64, user=root@7afbff271a3f, date=20240521-18:36:53, tags=unknown)"
ts=2024-07-07T12:37:02.196Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/)
ts=2024-07-07T12:37:02.197Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$
ts=2024-07-07T12:37:02.197Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(z?ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:111 level=info msg="Enabled collectors"
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=arp
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=bcache
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=bonding
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=btrfs
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=conntrack
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=cpu
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=cpufreq
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=diskstats
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=dmi
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=edac
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=entropy
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=fibrechannel
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=filefd
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=filesystem
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=hwmon
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=infiniband
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=ipvs
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=loadavg
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=mdadm
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=meminfo
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=netclass
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=netdev
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=netstat
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=nfs
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=nfsd
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=nvme
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=os
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=powersupplyclass
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=pressure
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=rapl
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=schedstat
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=selinux
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=sockstat
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=softnet
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=stat
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=tapestats
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=textfile
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=thermal_zone
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=time
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=timex
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=udp_queues
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=uname
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=vmstat
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=watchdog
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=xfs
ts=2024-07-07T12:37:02.198Z caller=node_exporter.go:118 level=info collector=zfs
ts=2024-07-07T12:37:02.199Z caller=tls_config.go:313 level=info msg="Listening on" address=[::]:9100
ts=2024-07-07T12:37:02.199Z caller=tls_config.go:316 level=info msg="TLS is disabled." http2=false address=[::]:9100

Are you running node_exporter in Docker?

No

What did you do that produced an error?

Scraped it. And the scrape was stuck. Even after a couple of minutes the scrape didn't succeed. I narrowed it down to thermal_zone collector.

Running ./node_exporter --no-collector.thermal_zone makes the scrapes work again.

How can I debug this further?

[SuperQ]

thermal_zone comes from prometheus/procfs. It walks files in /sys/class/thermal.

[erik-fauna]

Same issue appears when using the docker image. Trying to curl the metrics hangs unless you add the --no-collector.thermal_zone flag to command.

Using go version go1.22.5