From 2de28b19d8d1d6fb8c1ae368eb81b88ff492a102 Mon Sep 17 00:00:00 2001 From: Andrei Fedotov Date: Wed, 3 Apr 2024 20:28:53 +0300 Subject: [PATCH] Docs for userStacktrace Post action parameter Signed-off-by: Andrey Fedotov --- .../docs/concepts/tracing-policy/selectors.md | 101 +++++++++++++----- docs/content/en/docs/reference/grpc-api.md | 6 +- docs/data/tetragon_flags.yaml | 3 + 3 files changed, 80 insertions(+), 30 deletions(-) diff --git a/docs/content/en/docs/concepts/tracing-policy/selectors.md b/docs/content/en/docs/concepts/tracing-policy/selectors.md index f1a8ee74483..209111573b8 100644 --- a/docs/content/en/docs/concepts/tracing-policy/selectors.md +++ b/docs/content/en/docs/concepts/tracing-policy/selectors.md @@ -905,9 +905,10 @@ can be expanded to all processes by specifying the same with the value "global". #### Stack traces -`Post` takes the `stackTrace` parameter, when turned to `true` (by default to +`Post` takes the `kernelStackTrace` parameter, when turned to `true` (by default to `false`) it enables dump of the kernel stack trace to the hook point in kprobes -events. For example, the following kprobe hook can be used to retrieve the +events. To dump user space stack trace set `userStackTrace` parameter to `true`. +For example, the following kprobe hook can be used to retrieve the kernel stack to `kfree_skb_reason`, the function called in the kernel to drop kernel socket buffers. @@ -917,37 +918,41 @@ kprobes: selectors: - matchActions: - action: Post - stackTrace: true + kernelStackTrace: true + userStackTrace: true ``` {{< caution >}} -By default Tetragon does not expose the kernel addresses, you need to enable -the flag `--expose-kernel-addresses` to get the addresses along the rest. +By default Tetragon does not expose the linear addresses from kernel space or +user space, you need to enable the flag `--expose-stack-addresses` to get the +addresses along the rest. Note that the Tetragon agent is using its privilege to read the kernel symbols and their address. Being able to retrieve kernel symbols address can be used to break kernel address space layout randomization (KASLR) so only privileged users should be able to enable this feature and read events containing stack traces. +The same thing we can say about retrieving address for user mode processes. +Stack trace addresses can be used to bypass address space layout randomization (ASLR). {{< /caution >}} -Once loaded, events created from this policy will contain a new `stack_trace` +Once loaded, events created from this policy will contain a new `kernel_stack_trace` field on the `process_kprobe` event with an output similar to: ``` { - "address": "18446670264256120896", - "offset": "272", - "symbol": "skb_release_head_state" + "address": "18446744072119856613", + "offset": "5", + "symbol": "kfree_skb_reason" }, { - "address": "18446670264257081512", - "offset": "88", - "symbol": "ip_protocol_deliver_rcu" + "address": "18446744072119769755", + "offset": "107", + "symbol": "__sys_connect_file" }, { - "address": "18446670264257082152", - "offset": "88", - "symbol": "ip_local_deliver_finish" + "address": "18446744072119769989", + "offset": "181", + "symbol": "__sys_connect" }, [...] ``` @@ -955,29 +960,69 @@ field on the `process_kprobe` event with an output similar to: The "address" is the kernel function address, "offset" is the offset into the native instruction for the function and "symbol" is the function symbol name. +User mode stack trace is contained in `user_stack_trace` field on the +`process_kprobe` event and looks like: + +``` +{ + "address": "140498967885099", + "offset": "1209643", + "symbol": "__connect", + "module": "/usr/lib/x86_64-linux-gnu/libc.so.6" +}, +{ + "address": "140498968021470", + "offset": "1346014", + "symbol": "inet_pton", + "module": "/usr/lib/x86_64-linux-gnu/libc.so.6" +}, +{ + "address": "140498971185511", + "offset": "106855", + "module": "/usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0" +}, +``` +The "address" is the function address, "offset" is the function offset from the +beginning of the binary module. "module" is the absolute path of the binary file +to which address belongs. "symbol" is the function symbol name. "symbol" may be missing +if the binary file is stripped. + +{{< note >}} +Information from `procfs (/proc//maps)` is used to symbolize user +stack trace addresses. Stack trace addresses extraction and symbolizing are async. +It might happen that process is terminated and the `/proc//maps` file will be +not existed at user stack trace symbolization step. In such case user stack traces +for very short living process might be not collected. + +For Linux kernels before 5.15 user stack traces may be incomplete (some stack +traces entries may be missed). +{{< /note >}} + This output can be enhanced in a more human friendly using the `tetra getevents -o compact` command. Indeed, by default, it will print the stack trace along the compact output of the event similarly to this: ``` -❓ syscall /usr/bin/curl kfree_skb_reason - 0xffffbcdee5bf4840: skb_release_head_state+0x110 - 0xffffbcdee5be3678: __sys_connect_file+0x88 - 0xffffbcdee5be376c: __sys_connect+0xbc - 0xffffbcdee5be37bc: __arm64_sys_connect+0x28 - 0xffffbcdee4dfcd68: invoke_syscall+0x78 - 0xffffbcdee4dfcf70: el0_svc_common.constprop.0+0x180 - 0xffffbcdee4dfcfa4: do_el0_svc+0x30 - 0xffffbcdee5e861e8: el0_svc+0x48 - 0xffffbcdee5e881b4: el0t_64_sync_handler+0xa4 - 0xffffbcdee4de1e38: el0t_64_sync+0x1a4 +❓ syscall /usr/bin/curl kfree_skb_reason +Kernel: + 0xffffffffa13f2de5: kfree_skb_reason+0x5 + 0xffffffffa13dda9b: __sys_connect_file+0x6b + 0xffffffffa13ddb85: __sys_connect+0xb5 + 0xffffffffa13ddbd8: __x64_sys_connect+0x18 + 0xffffffffa1714bd8: do_syscall_64+0x58 + 0xffffffffa18000e6: entry_SYSCALL_64_after_hwframe+0x6e +User space: + 0x7f878cf2752b: __connect (/usr/lib/x86_64-linux-gnu/libc.so.6+0x12752b) + 0x7f878cf489de: inet_pton (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1489de) + 0x7f878d1b6167: (/usr/lib/x86_64-linux-gnu/libcurl.so.4.7.0+0x1a167) ``` -The printing format is `"0x%x: %s+0x%x", address, symbol, offset`. +The printing format for kernel stack trace is `"0x%x: %s+0x%x", address, symbol, offset`. +The printing format for user stack trace is `"0x%x: %s (%s+0x%x)", address, symbol, module, offset`. {{< note >}} Compact output will display missing addresses as `0x0`, see the above note on -`--expose-kernel-addresses` for more info. +`--expose-stack-addresses` for more info. {{< /note >}} ### NoPost action diff --git a/docs/content/en/docs/reference/grpc-api.md b/docs/content/en/docs/reference/grpc-api.md index dc79d02fb4d..8b4c68313d1 100644 --- a/docs/content/en/docs/reference/grpc-api.md +++ b/docs/content/en/docs/reference/grpc-api.md @@ -528,11 +528,12 @@ https://github.com/opencontainers/runtime-spec/blob/main/config.md#createcontain | args | [KprobeArgument](#tetragon-KprobeArgument) | repeated | Arguments definition of the observed kprobe. | | return | [KprobeArgument](#tetragon-KprobeArgument) | | Return value definition of the observed kprobe. | | action | [KprobeAction](#tetragon-KprobeAction) | | Action performed when the kprobe matched. | -| stack_trace | [StackTraceEntry](#tetragon-StackTraceEntry) | repeated | Kernel stack trace to the call. | +| kernel_stack_trace | [StackTraceEntry](#tetragon-StackTraceEntry) | repeated | Kernel stack trace to the call. | | policy_name | [string](#string) | | Name of the Tracing Policy that created that kprobe. | | return_action | [KprobeAction](#tetragon-KprobeAction) | | Action performed when the return kprobe executed. | | message | [string](#string) | | Short message of the Tracing Policy to inform users what is going on. | | tags | [string](#string) | repeated | Tags of the Tracing Policy to categorize the event. | +| user_stack_trace | [StackTraceEntry](#tetragon-StackTraceEntry) | repeated | User-mode stack trace to the call. | @@ -595,9 +596,10 @@ RuntimeHookRequest synchronously propagates information to the agent about run-t | Field | Type | Label | Description | | ----- | ---- | ----- | ----------- | -| address | [uint64](#uint64) | | address is the kernel function address. | +| address | [uint64](#uint64) | | linear address of the function in kernel or user space. | | offset | [uint64](#uint64) | | offset is the offset into the native instructions for the function. | | symbol | [string](#string) | | symbol is the symbol name of the function. | +| module | [string](#string) | | module path for user space addresses. | diff --git a/docs/data/tetragon_flags.yaml b/docs/data/tetragon_flags.yaml index 22fb29a9873..6ad1e43bb7f 100644 --- a/docs/data/tetragon_flags.yaml +++ b/docs/data/tetragon_flags.yaml @@ -97,6 +97,9 @@ options: - name: expose-kernel-addresses default_value: "false" usage: Expose real kernel addresses in events stack traces + - name: expose-stack-addresses + default_value: "false" + usage: Expose real linear addresses in events stack traces - name: field-filters usage: Field filters for event exports - name: force-large-progs