Skip to content

Unikraft Hackathon

Jump to bottom
Costin Lupu edited this page Oct 4, 2019 · 23 revisions

Repository contents

This repository contains a bunch of scripts and config files for helping out when building, running and debugging Unikraft VMs.

  • configs/ - configuration files for the Unikraft build and xl
  • http-server-example/ - an HTTP server example which may be used for the last proposed application
  • patches/ - examples of helper patches
  • scripts/ - helper scripts for running the VMs

Testing Infrastructure

In order to run Unikraft, you would need one machine for KVM and one for Xen.

  • For Xen use ssh xen-guest@<host>
  • For KVM use ssh kvm-guest@<host>

Please ask the hackathon staff for host addresses.

Cloning Unikraft repositories

A Unikraft VM is based on the Unikraft core, its external libraries and the application code. A common project setup would group the applications and libraries in directories on the same level as the Unikraft core tree:

my-unikraft-project/
├── apps/
│   ├── daytime/
│   ├── echo-tcp/
│   ├── echo-udp/
│   └── helloworld/
├── libs/
│   ├── lwip/
│   └── newlib/
└── unikraft/

You can use the unikraft-clone-repos.sh script in order to create the starting project structure. The script clones the core repository, newlib, lwip and the helloworld application.

$ unikraft-clone-repos.sh my-unikraft-project

We will be using the following repos:

Configuring

After you cloned the repos, go to the helloworld application and run make menuconfig for running the configuration menu. Unikraft uses the same configuration system as the Linux kernel (Kconfig). We will build Unikraft images for Xen and KVM, so first step would be to go to Platform Configuration menu and make the following changes:

  • select Xen guest image
    • select Emergency console for debug output
  • select KVM guest Next you will need to configure the logging level, so you should go to Library Configuration menu and for the internal ukdebug library the following changes need to be made:
  • change Debug message level to have the Show all types of debug messages value
  • change Message redirection to have the Kernel messages on debug output value

Building

In order for an application or library to work with Unikraft you need to provide at least the following three files:

  • Makefile: Used to specify where the main Unikraft repo is with respect to the application's repo, as well as repos for any external libraries the application needs to use.
  • Makefile.uk: The main Makefile used to specify which sources to build (and optionally where to fetch them from), include paths, flags and any application-specific targets.
  • Config.uk: A Kconfig-like snippet used to populate Unikraft's menu with application-specific options. You can find further information here.
my-first-app/
├── build/        # generated by the build
├── Config.uk
├── main.c
├── Makefile
└── Makefile.uk

Run make to generate the Unikraft images which will be put in the build/ subdirectory in the application directory.

Running

KVM

Please make sure you have qemu-system installed on your Linux system. A simple Qemu command for running would look like this:

$ qemu-system-x86_64 -enable-kvm -nographic -device isa-debug-exit -kernel <path-to-kvm-kernel>

You can use and extend the start-qemu.sh script for running Qemu when more complex setups are required (e.g. running with several different devices). For example, if you want to use a network device you should also give the interface name and MAC address as parameters. At the end of the command you can add custom Qemu parameters:

$ start-qemu.sh <path-to-kvm-kernel> qemuif0 aa:bb:cc:00:00:01 -append "my application parameters"

Networking setup

When using network devices, one should setup the bridge which will be used for connecting the VM network interfaces before running the VM. You can use the setup-bridge.sh script to create a bridge named qemubr and to configure the host IP on 10.0.0.1. Feel free to change the values in config.sh script if other configuration is desired.

$ setup-bridge.sh start

DHCP setup

There is a lightweight script to help setup a DHCP server on the bridge interface. It is written with Ubuntu/Debian in mind and configures the isc-dhcp-server package with a basic setup. Feel free to change IP ranges and other config parameters to suit your setup and use-case.

Xen

The toolstack for VM management on Xen is xl. The command for running VMs is:

$ xl create xl.conf

You have an example of xl configuration file in this repo. All you have to change is the kernel path and the VM name. You can check the output generated by the VM by running the following command and searching for the VM identifier to locate its output.

$ xl dmesg | less

You can use xl for checking the list of the current instantiated VMs by running the xl list command. However, your newly created VM will not be listed because it just prints a message and exits. So, lets create the VM again, but this time leaving it in paused state right after creation to be able to see it in the VM list.

$ xl create -p xl.conf     # -p leaves the VM in paused state
$ xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  3071    10     r-----     100.9
my-unikraft-vm                              20     8     1     --p---       0.0
$ xl unpause 20            # unpause the VM

Debugging

Debugging a VM involves the same steps as when debugging an application remotely. With support from both hypervisor and toolstack (Qemu on KVM, xl on Xen), a GDB server attaches to the VM which is in the paused state after creation. Next, the GDB debugger must connect to the GDB server and unpause the VM using the continue command. Here you can find detailed information about the debugging steps. Please note that on KVM there are some issues.

Enabling debug symbols

Before debugging, we must ensure that the debug symbols are generated when building Unikraft. For this you have to run make menuconfig again and go to the Build Options menu. The following need to be changed:

  • Optimization level: No optimizations
  • Debugging information:
    • Debug information level: Level 3
  • unselect Strip final image

KVM

Unfortunately, we encountered different behaviors when debugging with GDB, depending on the GDB versions that were used. A fast hackish solution would be to add a busy waiting loop in the code which checks the value of guarding variable that would be reset after connecting with GDB to the VM (e.g. the patch patches/0001-kvm-debug-guard.patch in this repo).

Starting applications

In order to understand the challenges raised by development and integration of new applications and libraries, we propose the following three applications. A good approach would be to first write the application for Linux and then trying to port it for Unikraft using the helloworld application as an example of how to write the Config and Makefiles.

Dependencies

The helloworld application uses a very minimalistic libc implementation called nolibc, which is an internal Unikraft library. For the next applications, you will be using an alternative libc implementation called Newlib, which is an external Unikraft library. More info about Newlib on Wikipedia.

The applications will also need a networking stack implementation in order to enable proper connectivity. For this you will use lwip, also an external Unikraft library. More info about lwip on Wikipedia.

You will need to change the Makefile of your application in order to provide the paths for the required libraries:

diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -1,6 +1,6 @@
 UK_ROOT ?= $(PWD)/../../unikraft
 UK_LIBS ?= $(PWD)/../../libs
-LIBS :=
+LIBS := $(UK_LIBS)/newlib:$(UK_LIBS)/lwip

 all:
        @make -C $(UK_ROOT) A=$(PWD) L=$(LIBS)

Configuration

For the development of the proposed application, we will be next focusing only on KVM. So, first step will be to disable the Xen guest in the configuration menu.
Then, from Platform Configurations -> KVM Guest configuration entry, enable the Virtio Network Driver.
Next, go to the Library Configuration menu and make sure that newlib and lwip are selected. Also, check that a scheduler is selected (the only available one at this point is the Cooperative Round-Robin scheduler ukschedcoop).
After that, go to the configuration menu for lwip and leave only the following selected:

  • IPv4
  • Calculate checksum on sending
  • UDP support
  • TCP support
  • ICMP support
  • DHCP client
  • Socket API

Save your changes and run make again. If you encounter difficulties in configuring the image, you can use our example in the repo and simply copy it in the application directory:

apps/myapp$ cp <path-to-this-repo>/configs/unikraft-build/config-kvm-network .config

Applications

1. TCP server

A simple TCP server running on port 13 which sends a "Hello World!" message on each connection. Testing command:

$ nc <VM server IP> 13
Hello World!

2. Echo UDP/TCP client & server

A server which either prints or sends back the same message it receives from another Unikraft VM acting as client. Testing scenario example:

# Starting the server
$ start-qemu.sh <path-to-kvm-kernel> qemuif-server aa:bb:cc:00:00:01 -append "app=server" 

# Starting the client
$ start-qemu.sh <path-to-kvm-kernel> qemuif-client aa:bb:cc:00:00:02 -append "app=client msg='Hello World'"

3. HTTP server

A HTTP server that replies with simple messages. You may use the example in http-server-example/http_reply_once.c as a starting point. In http-server-example/http-parser/samples/test_get_request_path.c you may find an example on how to use the HTTP parser.

# Starting the server
$ start-qemu.sh <path-to-kvm-kernel> qemuif aa:bb:cc:00:00:01 -append "port=8080" 

# Sending a HTTP request
$ wget http://<VM server IP>:8080
Clone this wiki locally