bootstrap

Demo BPF applications

Bootstrap

bootstrap is an example of a simple (but realistic) BPF application. It tracks process starts (exec() family of syscalls, to be precise) and exits and emits data about filename, PID and parent PID, as well as exit status and duration of the process life. With -d <min-duration-ms> you can specify minimum duration of the process to log. In such mode process start (technically, exec()) events are not output (see example output below).

bootstrap was created in the similar spirit as libbpf-tools from BCC package, but is designed to be more stand-alone and with simpler Makefile to simplify adoption to user's particular needs. It demonstrates the use of typical BPF features:

• cooperating BPF programs (tracepoint handlers for process exec and exit events, in this particular case);
• BPF map for maintaining the state;
• BPF ring buffer for sending data to user-space;
• global variables for application behavior parameterization.
• it utilizes BPF CO-RE and vmlinux.h to read extra process information from kernel's struct task_struct.

bootstrap is intended to be the starting point for your own BPF application, with things like BPF CO-RE and vmlinux.h, consuming BPF ring buffer data, command line arguments parsing, graceful Ctrl-C handling, etc. all taken care of for you, which are crucial but mundane tasks that are no fun, but necessary to be able to do anything useful. Just copy/paste and do simple renaming to get yourself started.

Here's an example output:

$ sudo sudo ./wasm-bpf bootstrap.wasm -h
BPF bootstrap demo application.

It traces process start and exits and shows associated 
information (filename, process duration, PID and PPID, etc).

USAGE: ./bootstrap [-d <min-duration-ms>] -v
$ sudo ./wasm-bpf bootstrap.wasm
TIME     EVENT COMM             PID     PPID    FILENAME/EXIT CODE
18:57:58 EXEC  sed              74911   74910   /usr/bin/sed
18:57:58 EXIT  sed              74911   74910   [0] (2ms)
18:57:58 EXIT  cat              74912   74910   [0] (0ms)
18:57:58 EXEC  cat              74913   74910   /usr/bin/cat
18:57:59 EXIT  cat              74913   74910   [0] (0ms)
18:57:59 EXEC  cat              74914   74910   /usr/bin/cat
18:57:59 EXIT  cat              74914   74910   [0] (0ms)
18:57:59 EXEC  cat              74915   74910   /usr/bin/cat
18:57:59 EXIT  cat              74915   74910   [0] (1ms)
18:57:59 EXEC  sleep            74916   74910   /usr/bin/sleep

The original c code is from libbpf-bootstrap.

the compile process of the bootstrap.wasm

We can provide a similar developing experience as the libbpf-bootstrap development. Just run make to build the wasm binary:

make

This would invoke the following steps:

• build the BPF program using clang and llvm-strip to strip debug info:

  clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -I../../third_party/vmlinux/x86/ -idirafter /usr/lib/llvm-15/lib/clang/15.0.2/include -idirafter /usr/local/include -idirafter /usr/include/x86_64-linux-gnu -idirafter /usr/include -c bootstrap.bpf.c -o bootstrap.bpf.o

  The kernel part of the BPF program is exactly the same as the libbpf(Or any other style can be compiled by clang. The BCC style can be compiled in this way once the bcc to libbpf converter is completed.

• generate the C header file from the BPF program:

  ../../third_party/bpftool/src/bpftool gen skeleton -j bootstrap.bpf.o > bootstrap.skel.h

  The C skel would include a skeleton for the BPF program to operate the BPF object, and control the life cycle of the BPF program, for example:

    struct bootstrap_bpf {
        struct bpf_object_skeleton *skeleton;
        struct bpf_object *obj;
        struct {
            struct bpf_map *exec_start;
            struct bpf_map *rb;
            struct bpf_map *rodata;
        } maps;
        struct {
            struct bpf_program *handle_exec;
            struct bpf_program *handle_exit;
        } progs;
        struct bootstrap_bpf__rodata {
            unsigned long long min_duration_ns;
        } *rodata;
        struct bootstrap_bpf__bss {
            uint64_t /* pointer */ name_ptr;
        } *bss;
    };

  Because the struct layout of the host (or eBPF side) may be different from the struct layout of the target (Wasm side), all pointer will be converted to integers base on the host's pointer size. For example, the name_ptr is a pointer to the name field in the struct exec_start_t struct. Besides, the padding bytes will be explicitly added to the struct to make sure the struct layout is the same as the target side, for example, using char __pad0[4];. This is generated by our modified bpftool tool for Wasm.

• build user space wasm code

  /opt/wasi-sdk/bin/clang -O2 --sysroot=/opt/wasi-sdk/share/wasi-sysroot -Wl,--allow-undefined -o bootstrap.wasm bootstrap.c

  A wasi-sdk is required to build the wasm binary. You can use the emcc toolchain to build the wasm binary as well, the command should be similar.

  You can run the following command to install the wasi-sdk:

  wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-17/wasi-sdk-17.0-linux.tar.gz
  tar -zxf wasi-sdk-17.0-linux.tar.gz
  sudo mkdir -p /opt/wasi-sdk/ && sudo mv wasi-sdk-17.0/* /opt/wasi-sdk/

  Since the struct layout of the host (or eBPF side) may be different from the struct layout of the target (Wasm side), you need to take care of the struct layout shared between the user space code. You can use the ecc and our wasm-bpftool for generating the C header file for the user space code:

   ecc bootstrap.h --header-only
   ../../third_party/bpftool/src/bpftool btf dump file bootstrap.bpf.o format c -j > bootstrap.wasm.h

  The ecc compiler in eunomia-bpf will use libclang and llvm to find all struct definitions in the header file, and automatically add more btf info to the ebpf object. The original clang may not always generate enough btf info for the wasm-bpf tool to generate the correct C header file.

  Note: This process and tools is not always required, you can do that mannually. You can mannually write all event structs definitions with __attribute__((packed)) to avoid padding bytes, and convert all pointer to correct integers between the host and the wasm side. All types must be defined the same size and layout as the host side in wasm as well. This would be easy for simple events, but it would be hard for complex programs, so we create the wasm specific bpftool to generate the C header file for the user space code contains all type defines and correct struct layout from the BTF info.

  We have create a specical POC tool outside of bpftool for generate C structs serialization-free bindings between eBPF/host side and Wasm, you can find it in c-struct-bindgen. More details about how to deal with the struct layout issue can be found in the README of the c-struct-bindgen tool.

  The libbpf API for wasm program is provided as an header only library, you can find it in libbpf-wasm.h. The wasm program can use the libbpf API and syscall to operate the BPF object, for example:

  /* Load and verify BPF application */
  skel = bootstrap_bpf__open();
  /* Parameterize BPF code with minimum duration parameter */
  skel->rodata->min_duration_ns = env.min_duration_ms * 1000000ULL;
  /* Load & verify BPF programs */
  err = bootstrap_bpf__load(skel);
  /* Attach tracepoints */
  err = bootstrap_bpf__attach(skel);

  The rodata section is used to store the global variables in the BPF program, and the bss section is used to store the global variables in the user space code, which will be memory mapped to the correct offset at the bpftool gen skeleton time, so libelf library is not required to be compiled in Wasm and the runtime can still dynamically load and operate the BPF object.

  The wasm side C code will be a slightly different from the native libbpf code, but it would provide the most ability from the eBPF side, for example, polling from the ring buffer or perf buffer, accessing the map from both the Wasm side and eBPF side, loading, attaching and detaching BPF programs, etc. It can support a lare number of eBPF program types and maps, covering the use cases of most eBPF programs from tracing, networking, security, etc.

  Because some feature is missing in wasm side, for example, the signal handler is not support yet(2023/2), the original C code cannot be compiled to wasm directly, you need to modify the code slightly to make it work. We would try our best to make the wasm side libbpf API as close as possible to the native libbpf API, so maybe the user space code can be compiled to wasm directly in the future. More language bindings(Rust, Go, etc...) for wasm side bpf API will also be provided soon.

  The polling API would be a wrapped for both ring buffer and perf buffer, and the user space code can use the same API to poll events from either ring buffer or perf buffer, depends on the type specified in the BPF program. For example, a ring buffer polling for a map defined as BPF_MAP_TYPE_RINGBUF:

  struct {
      __uint(type, BPF_MAP_TYPE_RINGBUF);
      __uint(max_entries, 256 * 1024);
  } rb SEC(".maps");

  You can use the following code to poll events from the ring buffer:

  rb = bpf_buffer__open(skel->maps.rb, handle_event, NULL);
  /* Process events */
  printf("%-8s %-5s %-16s %-7s %-7s %s\n", "TIME", "EVENT", "COMM", "PID",
         "PPID", "FILENAME/EXIT CODE");
  while (!exiting) {
      // poll buffer
      err = bpf_buffer__poll(rb, 100 /* timeout, ms */);

  No serialization overhead is required for the ring buffer polling. The bpf_buffer__poll API will call the handle_event function to process the event data in the ring buffer.

  The runtime is build on the top of libbpf CO-RE(Compile Once – Run Everywhere) API to load bpf object into the kernel, so a wasm-bpf program will not rely on the kernel version where it's built, and can be run on any kernel version with BPF CO-RE support.

  The size of bootstrap.wasm would be only ~90K. It would be very easy to distributed over the network and can dynamically deploy, load and run on another machine in less than 100ms. No kernel header, LLVM, clang dependency is required at the runtime, and don't need to do the heavy compilation work!

  For a more complex example, you can find the runqlat program in the examples directory.