Problem initializing the sensor using the latest version

[jungersa]

Hello,
I'm using the docker image powerapi/hwpc-sensor with the latest version which is currently version 1.4.0, but it can't seem to work as the sensors doesn't get initialized correctly, which is odd as I didn't change the configuration and the version hasn't changed since the last time it worked for me two/three weeks ago (start of November):

Here are the logs from the docker image:

I: 24-11-24 17:15:15 build: version unknown (rev: unknown)
I: 24-11-24 17:15:15 uname: Linux 6.11.9-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 17 Nov 2024 16:06:17 +0000 x86_64
I: 24-11-24 17:15:15 pmu: found ix86arch 'Intel X86 architectural PMU' having 7 events, 7 counters (4 general, 3 fixed)
I: 24-11-24 17:15:15 pmu: found perf 'perf_events generic PMU' having 82 events, 0 counters (0 general, 0 fixed)
I: 24-11-24 17:15:15 pmu: found rapl 'Intel RAPL' having 4 events, 3 counters (0 general, 3 fixed)
I: 24-11-24 17:15:15 pmu: found perf_raw 'perf_events raw PMU' having 1 events, 0 counters (0 general, 0 fixed)
I: 24-11-24 17:15:15 pmu: found skl 'Intel Skylake' having 84 events, 11 counters (8 general, 3 fixed)
I: 24-11-24 17:15:15 pmu: found intel_msr 'Intel MSR' having 6 events, 6 counters (0 general, 6 fixed)
I: 24-11-24 17:15:15 sensor: configuration is valid, starting monitoring...
I: 24-11-24 17:15:15 perf: monitoring actor started
I: 24-11-24 17:15:15 perf</system.slice/system-gpg\x2dagent\x2dbrowser.slice>: monitoring actor started
I: 24-11-24 17:15:15 perf</system.slice/system-modprobe.slice>: monitoring actor started
I: 24-11-24 17:15:15 perf</system.slice/upower.service>: monitoring actor started
I: 24-11-24 17:15:15 perf</system.slice/rtkit-daemon.service>: monitoring actor started
I: 24-11-24 17:15:15 perf</dev-mqueue.mount>: monitoring actor started
I: 24-11-24 17:15:15 perf</system.slice/system-gpg\x2dagent\x2dextra.slice>: monitoring actor started
I: 24-11-24 17:15:15 perf</system.slice/wpa_supplicant.service>: monitoring actor started
I: 24-11-24 17:15:15 perf</dev-hugepages.mount>: monitoring actor started
E: 24-11-24 17:15:15 perf</user.slice/user-1000.slice/[email protected]/app.slice/dconf.service>: failed opening perf event for group=core cpu=0 event=CPU_CLK_UNHALTED:THREAD_P errno=24
E: 24-11-24 17:15:15 perf</user.slice/user-1000.slice/[email protected]/session.slice/pulseaudio.service>: failed opening perf event for group=core cpu=0 event=CPU_CLK_UNHALTED:REF_P errno=24
E: 24-11-24 17:15:15 perf</init.scope>: failed opening perf event for group=core cpu=11 event=LLC_MISSES errno=24
E: 24-11-24 17:15:15 perf</system.slice/docker.socket>: failed opening perf event for group=core cpu=8 event=LLC_MISSES errno=24
E: 24-11-24 17:15:15 perf</system.slice/docker-7a6413a977c0c5245038a24708c11496af92e75f29f657335ab5886fe754061b.scope>: failed opening perf event for group=core cpu=6 event=CPU_CLK_UNHALTED:REF_P errno=24
E: 24-11-24 17:15:15 perf</system.slice/systemd-timesyncd.service>: failed opening perf event for group=core cpu=5 event=INSTRUCTIONS_RETIRED errno=24
I: 24-11-24 17:15:15 perf</system.slice/containerd.service>: monitoring actor started
E: 24-11-24 17:15:15 perf</system.slice/systemd-timesyncd.service>: failed to setup perf for group=core pkg=0 cpu=5
E: 24-11-24 17:15:15 perf</system.slice/system-gpg\x2dagent\x2dssh.slice>: failed opening perf event for group=core cpu=11 event=CPU_CLK_UNHALTED:THREAD_P errno=24
E: 24-11-24 17:15:15 perf</system.slice/sddm.service>: failed opening perf event for group=core cpu=5 event=CPU_CLK_UNHALTED:REF_P errno=24
E: 24-11-24 17:15:15 perf</system.slice/docker.service>: failed opening perf event for group=core cpu=6 event=CPU_CLK_UNHALTED:REF_P errno=24
E: 24-11-24 17:15:15 perf</system.slice/sddm.service>: failed to setup perf for group=core pkg=0 cpu=5
E: 24-11-24 17:15:15 perf</system.slice/docker.service>: failed to setup perf for group=core pkg=0 cpu=6
E: 24-11-24 17:15:15 perf</system.slice/systemd-udevd.service/udev>: failed opening perf event for group=core cpu=10 event=CPU_CLK_UNHALTED:THREAD_P errno=24
E: 24-11-24 17:15:15 perf</sys-kernel-config.mount>: failed opening perf event for group=core cpu=1 event=CPU_CLK_UNHALTED:REF_P errno=24
E: 24-11-24 17:15:15 perf</system.slice/systemd-udevd.service/udev>: failed to setup perf for group=core pkg=0 cpu=10
E: 24-11-24 17:15:15 perf</sys-kernel-config.mount>: failed to setup perf for group=core pkg=0 cpu=1
E: 24-11-24 17:15:15 perf</init.scope>: failed to setup perf for group=core pkg=0 cpu=11
E: 24-11-24 17:15:15 perf</system.slice/NetworkManager.service>: failed opening perf event for group=core cpu=11 event=CPU_CLK_UNHALTED:THREAD_P errno=24
E: 24-11-24 17:15:15 perf</system.slice/docker-7a6413a977c0c5245038a24708c11496af92e75f29f657335ab5886fe754061b.scope>: failed to setup perf for group=core pkg=0 cpu=6
I: 24-11-24 17:15:15 perf</user.slice/user-1000.slice/session-2.scope>: monitoring actor started
E: 24-11-24 17:15:15 perf</system.slice/NetworkManager.service>: failed to setup perf for group=core pkg=0 cpu=11
E: 24-11-24 17:15:15 perf</system.slice/system-gpg\x2dagent\x2dssh.slice>: failed to setup perf for group=core pkg=0 cpu=11
E: 24-11-24 17:15:15 perf</user.slice/user-1000.slice/[email protected]/app.slice/dconf.service>: failed to setup perf for group=core pkg=0 cpu=0
E: 24-11-24 17:15:15 perf</system.slice/docker.socket>: failed to setup perf for group=core pkg=0 cpu=8
E: 24-11-24 17:15:15 perf</system.slice/system-gpg\x2dagent\x2dssh.slice>: cannot initialize perf monitoring
E: 24-11-24 17:15:15 perf</system.slice/docker-ae3f478e4d9e2730c608ac0062ff9a20c709f9d32c4705838fb41eb0cc069939.scope>: failed opening perf event for group=core cpu=6 event=INSTRUCTIONS_RETIRED errno=24
E: 24-11-24 17:15:15 perf</system.slice/sddm.service>: cannot initialize perf monitoring
E: 24-11-24 17:15:15 perf</user.slice/user-1000.slice/[email protected]/session.slice/pulseaudio.service>: failed to setup perf for group=core pkg=0 cpu=0
E: 24-11-24 17:15:15 perf</system.slice/systemd-udevd.service/udev>: cannot initialize perf monitoring
E: 24-11-24 17:15:15 perf</system.slice/NetworkManager.service>: cannot initialize perf monitoring
E: 24-11-24 17:15:15 perf</sys-kernel-config.mount>: cannot initialize perf monitoring
E: 24-11-24 17:15:15 perf</init.scope>: cannot initialize perf monitoring
E: 24-11-24 17:15:15 perf</system.slice/docker.socket>: cannot initialize perf monitoring
E: 24-11-24 17:15:15 perf</system.slice/docker-7a6413a977c0c5245038a24708c11496af92e75f29f657335ab5886fe754061b.scope>: cannot initialize perf monitoring
E: 24-11-24 17:15:15 perf</user.slice/user-1000.slice/[email protected]/app.slice/dconf.service>: cannot initialize perf monitoring
E: 24-11-24 17:15:15 perf</system.slice/systemd-timesyncd.service>: cannot initialize perf monitoring
E: 24-11-24 17:15:15 perf</user.slice/user-1000.slice/[email protected]/session.slice/pulseaudio.service>: cannot initialize perf monitoring
E: 24-11-24 17:15:15 perf</system.slice/docker-ae3f478e4d9e2730c608ac0062ff9a20c709f9d32c4705838fb41eb0cc069939.scope>: failed to setup perf for group=core pkg=0 cpu=6
E: 24-11-24 17:15:15 perf</system.slice/docker.service>: cannot initialize perf monitoring
E: 24-11-24 17:15:15 perf</system.slice/docker-ae3f478e4d9e2730c608ac0062ff9a20c709f9d32c4705838fb41eb0cc069939.scope>: cannot initialize perf monitoring

Here is the sensor configuration used:

{
"name": "sensor",
"verbose": true,
"frequency": 1000,
"output": {
"type": "mongodb",
"uri": "mongodb://127.0.0.1",
"database": "db_sensor",
"collection": "report"
},
"system": {
"rapl": {
"events": [
"RAPL_ENERGY_PKG"
],
"monitoring_type": "MONITOR_ONE_CPU_PER_SOCKET"
},
"msr": {
"events": [
"TSC",
"APERF",
"MPERF"
]
}
},
"container": {
"core": {
"events": [
"CPU_CLK_UNHALTED:REF_P",
"CPU_CLK_UNHALTED:THREAD_P",
"LLC_MISSES",
"INSTRUCTIONS_RETIRED"
]
}
}
}

I looked at #63, and tried changing the event configuration but didn't fix the problem and the found solution wasn't the issue for me as I'm using cgroups v2 and not v1.

Thanks in advance for your help

[gfieni]

Hello,
This looks like a problem with the open files ulimit. (errno 24/EMFILE : Too many open files)
As you said, the sensor and its configuration did not change, it is likely caused by a package update.

Could you give me the result of the systemctl show docker |grep LimitNOFILE command ?
You can also try to set the ulimit when starting the container with the --ulimit nofile=262144:262144 parameter and see if you still have this error.

[jungersa]

Thank you when setting the ulimit in the commands with the --ulimit nofile=262144:262144 parameter it works as expected, as for the output of the command systemctl show docker |grep LimitNOFILE :

LimitNOFILE=524288
LimitNOFILESoft=1024

[gfieni]

Thanks for the information.
We will improve the documentation to prevent this kind of problems in the future.