You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
In #1795, we introduced the ability for kernels that support the 1M BPF program limit to raise certain tunables used to restrict program size. Many upstream distros and companies backport features, which means that the kernel version is somewhat meaningless (older kernels could have the feature you are checking for). We knew this was a possibility, but thought it was safe to assume that any version > 5.1 would be guaranteed to have the functionality.
Linux 6.10 introduced BPF tokens, which are a newer way to allow less privileged processes to access BPF functionality. This changes the BPF permission model and can even cause programs to fall back to the lower, 4096 instruction limit. Therefore, it's possible that a process with less permissions can be forced to use a lower limit. This causes PEMs with this reduced permission to fail to start since our tunables to reduce program size will still be increased despite the kernel forcing us to use 4096 instructions.
This was detected while fixing #2035, where our adhoc 6.10 qemu builds failed with instruction limit errors (as seen below):
$ bazel run -c dbg src/stirling/source_connectors/socket_tracer:dns_trace_bpf_test_qemu_interactive
bash-5.2# src/stirling/source_connectors/socket_tracer/dns_trace_bpf_test
I20241009 14:21:07.577044 135 socket_trace_connector.cc:468] Kernel version greater than V5.1 detected (6.11.1), raised loop limit to 882 and chunk limit to 84
I20241009 14:21:07.578644 135 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.11.1
I20241009 14:21:07.578760 135 linux_headers.cc:381] Detected kernel release (uname -r): 6.11.1
I20241009 14:21:07.580492 135 linux_headers.cc:202] Using Linux headers from: /lib/modules/6.11.1/build and /lib/modules/6.11.1/source.
I20241009 14:21:07.585541 135 bcc_wrapper.cc:166] Initializing BPF program ...
I20241009 14:22:06.109444 135 scoped_timer.h:48] Timer(init_bpf_program) : 58.52 s
bpf: Argument list too long. Program too large (18400 insns), at most 4096 insns
./src/stirling/source_connectors/socket_tracer/testing/socket_trace_bpf_test_fixture.h:54: Failure
Value of: IsOK(::px::StatusAdapter(source_->Init()))
Actual: false (Internal : Failed to load syscall__probe_ret_writev: -1)
Expected: true
I20241009 14:22:11.929546 135 container_runner.cc:53] podman rm -f dns_server_52108331944 &>/dev/null
[ FAILED ] DNSTraceTest.Capture (87555 ms)
[----------] 1 test from DNSTraceTest (87556 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (87557 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] DNSTraceTest.Capture
1 FAILED TEST
I20241009 14:22:12.809012 135 env.cc:51] Shutting down
We should change this logic to based on a dynamic probe of the system. bpftool is able to identify if a kernel is capable of the higher instruction limit by inserting a test program above the 4096 limit. This would address this bug and properly identify any backported kernels that should have the higher limits applied.
The text was updated successfully, but these errors were encountered:
I have a crude test on this branch that inserts > 4096 BPF asm instructions. This test works on < 5.2 kernels, but it appears to pass on the 6.11 kernel that is later scope to 4096 instructions when the socket tracer program is loaded.
More investigation needs to be done to see how this dynamic probe can trigger the 6.11 instruction limit restriction.
Describe the bug
In #1795, we introduced the ability for kernels that support the 1M BPF program limit to raise certain tunables used to restrict program size. Many upstream distros and companies backport features, which means that the kernel version is somewhat meaningless (older kernels could have the feature you are checking for). We knew this was a possibility, but thought it was safe to assume that any version > 5.1 would be guaranteed to have the functionality.
Linux 6.10 introduced BPF tokens, which are a newer way to allow less privileged processes to access BPF functionality. This changes the BPF permission model and can even cause programs to fall back to the lower, 4096 instruction limit. Therefore, it's possible that a process with less permissions can be forced to use a lower limit. This causes PEMs with this reduced permission to fail to start since our tunables to reduce program size will still be increased despite the kernel forcing us to use 4096 instructions.
This was detected while fixing #2035, where our adhoc 6.10 qemu builds failed with instruction limit errors (as seen below):
We should change this logic to based on a dynamic probe of the system.
bpftool
is able to identify if a kernel is capable of the higher instruction limit by inserting a test program above the 4096 limit. This would address this bug and properly identify any backported kernels that should have the higher limits applied.The text was updated successfully, but these errors were encountered: