Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamically probe system for availability of larger (1M) BPF instruction limit #2040

Open
ddelnano opened this issue Oct 9, 2024 · 1 comment
Labels
area/datacollector Issues related to Stirling (datacollector)

Comments

@ddelnano
Copy link
Member

ddelnano commented Oct 9, 2024

Describe the bug
In #1795, we introduced the ability for kernels that support the 1M BPF program limit to raise certain tunables used to restrict program size. Many upstream distros and companies backport features, which means that the kernel version is somewhat meaningless (older kernels could have the feature you are checking for). We knew this was a possibility, but thought it was safe to assume that any version > 5.1 would be guaranteed to have the functionality.

Linux 6.10 introduced BPF tokens, which are a newer way to allow less privileged processes to access BPF functionality. This changes the BPF permission model and can even cause programs to fall back to the lower, 4096 instruction limit. Therefore, it's possible that a process with less permissions can be forced to use a lower limit. This causes PEMs with this reduced permission to fail to start since our tunables to reduce program size will still be increased despite the kernel forcing us to use 4096 instructions.

This was detected while fixing #2035, where our adhoc 6.10 qemu builds failed with instruction limit errors (as seen below):

$ bazel run -c dbg src/stirling/source_connectors/socket_tracer:dns_trace_bpf_test_qemu_interactive
bash-5.2# src/stirling/source_connectors/socket_tracer/dns_trace_bpf_test
I20241009 14:21:07.577044   135 socket_trace_connector.cc:468] Kernel version greater than V5.1 detected (6.11.1), raised loop limit to 882 and chunk limit to 84
I20241009 14:21:07.578644   135 kernel_version.cc:82] Obtained Linux version string from `uname`: 6.11.1
I20241009 14:21:07.578760   135 linux_headers.cc:381] Detected kernel release (uname -r): 6.11.1
I20241009 14:21:07.580492   135 linux_headers.cc:202] Using Linux headers from: /lib/modules/6.11.1/build and /lib/modules/6.11.1/source.
I20241009 14:21:07.585541   135 bcc_wrapper.cc:166] Initializing BPF program ...
I20241009 14:22:06.109444   135 scoped_timer.h:48] Timer(init_bpf_program) : 58.52 s
bpf: Argument list too long. Program  too large (18400 insns), at most 4096 insns

./src/stirling/source_connectors/socket_tracer/testing/socket_trace_bpf_test_fixture.h:54: Failure
Value of: IsOK(::px::StatusAdapter(source_->Init()))
  Actual: false (Internal : Failed to load syscall__probe_ret_writev: -1)
Expected: true
I20241009 14:22:11.929546   135 container_runner.cc:53] podman rm -f dns_server_52108331944 &>/dev/null
[  FAILED  ] DNSTraceTest.Capture (87555 ms)
[----------] 1 test from DNSTraceTest (87556 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (87557 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DNSTraceTest.Capture

 1 FAILED TEST
I20241009 14:22:12.809012   135 env.cc:51] Shutting down

We should change this logic to based on a dynamic probe of the system. bpftool is able to identify if a kernel is capable of the higher instruction limit by inserting a test program above the 4096 limit. This would address this bug and properly identify any backported kernels that should have the higher limits applied.

@ddelnano
Copy link
Member Author

I have a crude test on this branch that inserts > 4096 BPF asm instructions. This test works on < 5.2 kernels, but it appears to pass on the 6.11 kernel that is later scope to 4096 instructions when the socket tracer program is loaded.

More investigation needs to be done to see how this dynamic probe can trigger the 6.11 instruction limit restriction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/datacollector Issues related to Stirling (datacollector)
Projects
None yet
Development

No branches or pull requests

1 participant