Initial support for .so compartments

Implement support for compartments to be gives as dynamic libraries
(i.e., `so` files). This includes looking for library dependencies, and
loading those in the same compartment as the given user file. These
libraries are expected to be found in the path given by the environment
variable `COMP_LIBRARY_PATH`.

What this means is that we now also add a "local" `libc` to each
compartment that needs it (likely all?). Thus, we can greatly reduce the
number of intercepts we require. Perhaps we can overhaul the mechanism
entirely to only intercept hard-coded functions we know we need (e.g.,
allocator calls).

The support for static binaries has not been removed completely, but
likely has been broken due to this overhaul. Whether to leave as is,
to remove completely, or to add support for it remains to be decided.

Further small changes:
* Rework testing infrastructure to account for `so` compartments
* Replace `false` `assert`s and `exit`s with `err`s
* Replace all `pread`s with `do_pread` wrappers
* Rename intercept functions from `manager_` prefix to `intercepted_`
* Slight `README` update
* Replace `intptr_t` usage with `void*` where appopriate
* Fixed a bug where we would find symbols in dependency libraries with a
  value of 0x0 to be eagerly relocated. Assumption is these symbols are
  meant to be relocated in the dependency library itself. We now simply
  filter out 0x0 addresses for symbols to eagerly relocate

.run_cheri_qemu_and_test.py

@@ -24,9 +24,12 @@ def run_tests(qemu: boot_cheribsd.QemuCheriBSDInstance, args: argparse.Namespace
         boot_cheribsd.set_ld_library_path_with_sysroot(qemu)
     boot_cheribsd.info("Running tests for CHERI-ELF-compartments")
 
+    # Test environment setup
+    subprocess.run(["./tests/init_test.py"], check = True)
+
     # Run command on host to test the executed client
     os.chdir(f"{args.build_dir}/build")
-    subprocess.run(["ctest", "--output-on-failure"], check=True)
+    subprocess.run(["ctest", "--output-on-failure"], check = True)
     return True
 
 if __name__ == '__main__':

CMakeLists.txt

@@ -2,7 +2,7 @@ cmake_minimum_required(VERSION 3.16)
 project(CHERI_ELF_Compartments LANGUAGES C ASM)
 
 # Set global compilation options
-set(CMAKE_COMPILE_WARNING_AS_ERROR ON)
+add_compile_options(-pedantic -Wno-gnu-binary-literal -Wno-language-extension-token -Werror)
 
 # Set useful directory variables
 set(TEST_DIR ${CMAKE_SOURCE_DIR}/tests)
@@ -12,12 +12,12 @@ set(INCLUDE_DIR ${CMAKE_SOURCE_DIR}/include)
 # Build lua
 set(LUA_DIR ${CMAKE_SOURCE_DIR}/third-party/lua)
 set(LUA_INSTALL_DIR ${LUA_DIR})
-set(LUA_LIB_PATH ${LUA_DIR}/liblua.a)
+set(LUA_LIB_PATH ${LUA_DIR}/liblua.so)
 set(LUA_INCLUDE_DIR ${LUA_DIR})
 
 add_custom_command(
     OUTPUT ${LUA_LIB_PATH}
-    COMMAND make a
+    COMMAND make
     WORKING_DIRECTORY ${LUA_DIR}
 )
 

README.md

@@ -24,18 +24,15 @@ ELF binary. Each compartment will be restricted to its own memory chunk, and
 should not be able to access (i.e., load or store) any memory outside the chunk
 determined by the manager.
 
-Compartments are loaded from a file on disk, in the form of a pre-compiled
-static ELF binary. We prefer static binaries, as this ensures (most) of the
-compartment functionality is self contained (further discussion on exceptions
-[in this Section](#function-interception)). Compartments are then allocated a
-chunk of the manager process' memory, and the executable code of the
-compartment is loaded inside that chunk. Additional memory is reserved within
-the chunk for the compartment's stack and heap. A capability is defined around
-the allocated chunk. Whenever we transition to a given compartment, we load its
-respective capability inside the DDC, essentially limiting memory loads and
-stores within the compartment's allocated memory region. Once the compartment
-is loaded into memory, we are able to call functions defined in the binary from
-the manager.
+Compartments are loaded from a file on disk, in the form of a pre-compiled ELF
+binary. Compartments are then allocated a chunk of the manager process' memory,
+and the executable code of the compartment is loaded inside that chunk.
+Additional memory is reserved within the chunk for the compartment's stack and
+heap. A capability is defined around the allocated chunk. Whenever we
+transition to a given compartment, we load its respective capability inside the
+DDC, essentially limiting memory loads and stores within the compartment's
+allocated memory region. Once the compartment is loaded into memory, we are
+able to call functions defined in the binary from the manager.
 
 There is one exception when a compartment may access memory outside its
 designated region: if it is passed a capability by another compartment, then
@@ -45,6 +42,26 @@ useful for sharing more complex data across compartments, but doing so
 essentially removes the security over the shared region for the original owner
 compartment, so must be done so sparsely.
 
+## Structure overview
+
+The project is split into the following components (subject to change):
+* `manager` - this exposes the main API that users are expected to use. It
+  offers functions to initialize a compartment from a given ELF binary, execute
+  a compartment, and general compartment management features. It currently does
+  not support deleting compartments.
+* `compartment` - this is mainly to do with compartment internals, and reading
+  ELF data for the given input file, managing memory, and various other
+  interesting bits we do to ensure code can function when
+  DDC-compartmentalized.
+* `intercept` - a feature to automatically intercept functions within
+  compartments that need to be executed within a higher-trust level. To our
+  knowledge, these are functions that might call into vDSO[^1][^2], or perform
+  system calls. Other situations might be added here as we explore more.
+* `mem_mng` - we implement a simple bump allocator to ensure that internal
+  compartment memory allocations are done within the compartment, in the area
+  specifically marked to be used as heap. This is the implementation of the
+  allocator
+
 ### Executing a compartment
 
 TODO
@@ -71,6 +88,8 @@ function, then transitioning back into the compartment.
 Current limitations (as this is a work in progress, some of these are planned
 to be addressed):
 * single compartment;
-* the user-code must be compiled with `--static` and `--image-base` set to some
-  pre-determined variable;
 * only 3 arguments supported for argument passing, not floats or pointers.
+* did not check for support for capabilities within compartments
+
+[^1]: https://en.wikipedia.org/wiki/VDSO
+[^2]: https://lwn.net/Articles/446528/

include/compartment.h

@@ -3,8 +3,12 @@
 
 #include <assert.h>
 #include <elf.h>
+#include <err.h>
+#include <errno.h>
 #include <fcntl.h>
+#include <fts.h>
 #include <stdbool.h>
+#include <stddef.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
@@ -15,8 +19,6 @@
 
 #include "cheriintrin.h"
 
-#include "manager.h"
-
 // Morello `gcc` defines `ptraddr_t` in `stddef.h`, while `clang` does so in
 // `stdint.h`
 // TODO are there any other diverging definition files?
@@ -27,17 +29,13 @@
 #define align_down(x, align)    __builtin_align_down(x, align)
 #define align_up(x, align)      __builtin_align_up(x, align)
 
-// Maximum number of allowed segments per loaded ELF file
-// TODO rethink number/make it a parameter
-#define SEG_MAX_COUNT 20
-
 // TODO once things stabilize, recheck if all struct members are required
 // currently there's quite a bit of redundancy to make things easier to think
 // about
 
-struct func_intercept;
 void compartment_transition_out();
-int64_t comp_exec_in(void*, void* __capability, void*, void*, size_t);
+int64_t comp_exec_in(void*, void* __capability, void*, void*, size_t,
+                     void* __capability);
 void comp_exec_out();
 
 // Declare built-in function for cache synchronization:
@@ -51,8 +49,14 @@ extern void __clear_cache(void*, void*);
 // Number of instructions required by the transition function
 #define COMP_TRANS_FN_INSTR_CNT 4
 
+extern void* __capability sealed_redirect_cap;
+extern void* __capability comp_return_caps[2];
+
 /* For a function to be intercepted, information required to insert the
  * redirect code and perform redirection
+ *
+ * TODO recheck this is properly used, or re-design into a more light-weight
+ * approach with pre-given transition capabilities
  */
 struct intercept_patch
 {
@@ -73,41 +77,62 @@ struct intercept_patch
 #error Expecting 64-bit Arm Morello platform
 #endif
 
-/* Struct representing configuration data for one entry point; this is just
- * information that we expect to appear in the compartment, as given by its
- * compartment configuration file
- */
-struct ConfigEntryPoint
-{
-    const char* name;
-    size_t arg_count;
-    char** args_type;
-};
-
 /* Struct representing a valid entry point to a compartment
  */
 struct entry_point
 {
     const char* fn_name;
-    uintptr_t fn_addr;
-    size_t arg_count;
-    char** arg_types;
+    void* fn_addr;
 };
 
 /* Struct representing one segment of an ELF binary.
  *
  * TODO expand */
 struct SegmentMap
 {
-    uintptr_t mem_bot;
-    uintptr_t mem_top;
+    void* mem_bot;
+    void* mem_top;
     size_t offset;
-    size_t correction;
+    ptrdiff_t correction;
     size_t mem_sz;
     size_t file_sz;
     int prot_flags;
 };
 
+/* Struct representing an eager relocation to be made at map-time - instead of
+ * lazily looking up function addresses once a function is called at runtime,
+ * via PLT/GOT, we update the expected addresses eagerly once the code is
+ * mapped into memory, via `comp_map`
+ */
+struct CompRelaMapping
+{
+    char* rela_name;
+    void* rela_address; // address of relocation in compartment
+    void* target_func_address; // address of actual function
+};
+
+/* Struct representing a symbol entry of a dependency library
+ */
+struct LibDependencySymbol
+{
+    char* sym_name;
+    intptr_t sym_offset;
+};
+
+/* Struct representing a library dependency for one of our given compartments
+ */
+struct LibDependency
+{
+    char* lib_name;
+    char* lib_path;
+    size_t lib_segs_count;
+    size_t lib_segs_size;
+    void* lib_mem_base;
+    struct SegmentMap** lib_segs;
+    size_t lib_syms_count;
+    struct LibDependencySymbol* lib_syms;
+};
+
 /* Struct representing ELF data necessary to load and eventually execute a
  * compartment
  */
@@ -119,65 +144,67 @@ struct Compartment
     Elf64_Half elf_type;
     // Execution info
     Elf64_Half phdr;
-    Elf64_Half phentsize;
-    Elf64_Half phnum;
     void* __capability ddc;
     // ELF data
-    size_t size;
-    uintptr_t base;
+    size_t size; // size of compartment in memory
+    void* base; // address where to load compartment
     size_t entry_point_count;
-    struct entry_point** comp_fns; // TODO
-    uintptr_t* relas;
-    size_t relas_cnt;
+    struct entry_point** comp_fns;
+    void* mem_top;
     bool mapped;
     bool mapped_full;
     // Segments data
-    struct SegmentMap* segs[SEG_MAX_COUNT]; // TODO
+    struct SegmentMap** segs;
     size_t seg_count;
     size_t segs_size;
     // Scratch memory
-    uintptr_t scratch_mem_base;
+    void* scratch_mem_base;
     size_t scratch_mem_size;
     size_t scratch_mem_alloc;
 
     size_t scratch_mem_heap_size;
-    uintptr_t scratch_mem_stack_top;
+    void* scratch_mem_stack_top;
     size_t scratch_mem_stack_size;
-    uintptr_t stack_pointer;
+    void* stack_pointer;
     struct mem_alloc* alloc_head;
 
-    uintptr_t manager_caps;
+    void* manager_caps;
     size_t max_manager_caps_count;
     size_t active_manager_caps_count;
 
-    uintptr_t mng_trans_fn;
+    void* mng_trans_fn;
     size_t mng_trans_fn_sz;
+
+    // Only for shared object compartments
+    size_t rela_maps_count;
+    struct CompRelaMapping* rela_maps;
+    size_t lib_deps_count;
+    struct LibDependency** lib_deps;
+
     // Hardware info - maybe move
     size_t page_size;
+
     // Misc
     short curr_intercept_count;
-    struct intercept_patch patches[INTERCEPT_FUNC_COUNT];
+    struct intercept_patch* intercept_patches;
 };
 
-extern struct Compartment** comps;
-
 int entry_point_cmp(const void*, const void*);
 struct Compartment* comp_init();
-struct Compartment* comp_from_elf(char*, struct ConfigEntryPoint*, size_t);
-void comp_register_ddc(struct Compartment*);
-void comp_add_intercept(struct Compartment*, uintptr_t, struct func_intercept);
+struct Compartment* comp_from_elf(char*, char**, size_t, char**, void**, size_t, void*);
+void comp_add_intercept(struct Compartment*, uintptr_t, uintptr_t);
 void comp_stack_push(struct Compartment*, const void*, size_t);
 void comp_map(struct Compartment*);
 void comp_map_full(struct Compartment*);
 int64_t comp_exec(struct Compartment*, char*, void*, size_t);
 void comp_clean(struct Compartment*);
 
-void log_new_comp(struct Compartment*);
 struct Compartment* find_comp(struct Compartment*);
 
-void setup_stack(struct Compartment*);
-
-void comp_print(struct Compartment*);
-void segmap_print(struct SegmentMap*);
+static ssize_t do_pread(int, void*, size_t, off_t);
+static Elf64_Sym* find_symbols(const char**, size_t, bool, Elf64_Sym*, char*, size_t);
+static char* find_in_dir(const char*, char*);
+static void init_comp_scratch_mem(struct Compartment*);
+static void init_lib_dep_info(struct LibDependency*, struct Compartment*);
 
 #endif // _COMPARTMENT_H