Implement VirtIO GPU and input devices

[shengwen-tw]

Currently semu does not support graphics rendering.

I think VirtIO GPU with 2D mode might be a good starting point to go.
Later we can then emulate mouse and keyboard for more sophisticated features.

[jserv]

homebrew-qemu-virgl illustrates how virtio-gpu can be implemented via VirGL.

[shengwen-tw]

Side note:
For implementing VirtIO GPU with Linux 6.x kernel, we should refer to VIRTIO Spec. 1.2 (Draft 01 released on 09 May 2022) instead of earlier versions (i.e., VIRTIO Spec. 1.1).

The register 0xac, 0xb0, 0xb4, 0xb8, 0xbc and 0xc0 are only documented since v1.2.

Update:
The definition of virtio_gpu_ctrl_hdr also varies from v1.1 to v1.2.

[shengwen-tw]

By observation, when the Linux kernel loads virtio-gpu module, it reads register 0xac (SHMSel), 0xb0 (SHMLenLow), 0xb4 (SHMLenHigh), 0xb8 (SHMBaseLow), and 0xbc (SHMBaseHigh) for a shared memory space. (Those registers tell the address and length of a shared memory specified with an ID.)

Seems like shared memory is meant to be a device by design, and may be registered via the device tree (I'm not quite sure).

Eventually, I found that QEMU just passes -1 to register 0xb0 (SHMLenLow) and 0xb4 (SHMLenHigh), which just works for me now.

[shengwen-tw]

embear illustrates a method for accessing DRM Framebuffer to display an image.

The article first shows that to test whether a display is working or not, we can use:

$ dd if=/dev/urandom of=/dev/fb0

For showing an image on the display, the author demonstrated a method that involves using libdrm.

• drm-framebuffer

[shengwen-tw]

drm-test provides several utilities for testing GPU, which can be enabled via Buildroot.

After loading virtio-gpu into the kernel, we can inspect the device using:

$ modetest -M virtio_gpu

...
Connectors:
id	encoder	status		name		size (mm)	modes	encoders
34	0	connected	Virtual-1      	0x0		35	35

...
 #18 1024x768 119.99 1024 1072 1104 1184 768 771 775 813 115500 flags: phsync, nvsync; type: driver
...

Next, type the following command to set up the drm buffer:

$ modetest -M virtio_gpu -s 34:1024x768

setting mode 1024x768-119.99Hz on connectors 34, crtc 33

In addition, the following command can be used to show the virtio-gpu status:

$ cat /sys/devices/platform/soc/f4300000.virtio/drm/card0/card0-Virtual-1/enabled

enabled

Before setting up the drm buffer, the output is disabled.

[shengwen-tw]

psun3x's commit to the acrn-hypervisor is a good reference of understanding how to handle the virtio-gpu 2D command sequence.

What worth mentioning are:

1. It utilizes Pixman for low-level pixel manipulation, which is useful for dealing with format such as B8G8R8X8.
2. Linked list object is good for managing 2D resources of virtio-gpu; since the scanout memory is dynamically allocated (and deallocated) by the guest OS.

Update: The complete source code can be found at here.

[shengwen-tw]

The article "Image Stride" by Microsoft illustrates the concept of scan-line, padding, and stride, which is helpful for understanding the code of acrn-hypervisor.

[shengwen-tw]

References on SDL programming:

1. Max Slater's article demonstrates the C++ code for rendering texture (i.e., image) with SDL.
2. SDL_CreateRGBSurfaceFrom can be used to convert image data from 1D array (i.e., char img[width * height * channels]) to SDL_Surface (i.e., SDL's data type of image).
3. SDL_CreateRGBSurfaceWithFormatFrom is similar to SDL_CreateRGBSurfaceFrom but can take care of pitch (i.e., stride) and can be assigned with format like SDL_PIXELFORMAT_RGBA8888 instead of bitmask.
4. Stackoverflow: Rendering pixels from array of RGB values in SDL 1.2?

[shengwen-tw]

Fluxbox and Blackbox are two Lightweight Window Managers suitable for demonstrating virtio-gpu of the semu.

[shengwen-tw]

semu is now able to launch X Window System (X11) for basic rendering.

However, semu is a minimalist system emulator and does not plan to enable X11 due to its unfavorable large footprint, this thread serves as a record for setting up X11.

To enable X11, turn on the following options in the Buildroot with make menuconfig:

+BR2_INSTALL_LIBSTDCPP=y
+BR2_PACKAGE_XORG7=y
+BR2_PACKAGE_XSERVER_XORG_SERVER=y
+BR2_PACKAGE_XSERVER_XORG_SERVER_MODULAR=y
+BR2_PACKAGE_XAPP_XINIT=y
+BR2_PACKAGE_XTERM=y
+BR2_PACKAGE_XAPP_XCLOCK=y
+BR2_PACKAGE_XAPP_TWM=y

To launch X11, initiate semu then type the following commands:

$ mkdir mnt
$ mount /dev/vda mnt
$ sh mnt/run.sh
$ startx

The user should expect to observe the following result:

Reference: How to install X11 on my own Linux Buildroot system?

[shengwen-tw]

The following guild demonstrates the cross-compilation procedure of the DirectFB2 as it is currently not integrated by Buildroot.

1. DirectFB2 Cross Compilation:

# Clone DirectFB2 under semu
cd semu/
git clone https://github.com/directfb2/DirectFB2.git
cd DirectFB2/

# Set up Buildroot RISC-V toolchain's path
export PATH=$PATH:~/workspace/ncku/semu/buildroot/output/host/bin

# Set up Buildroot output directory's path
export BUILDROOT_OUT="/home/shengwen/workspace/ncku/semu/buildroot/output/"

# Apply patch to DirectFB2
wget https://gist.githubusercontent.com/shengwen-tw/e5dffd8aefd7d2019cfbb4b7f1048ef3/raw/b079443aefb965742a6c6227fe4d0cf831d87cb1/riscv-directfb2.patch
git apply riscv-directfb2.patch

# Download cross-compilation file
wget https://gist.githubusercontent.com/shengwen-tw/098da8c41ba7fbb9162ddaa4ff62b29e/raw/5ab962990d19a8bd1a8f378ef9b0b0ef1c5fb36a/riscv-cross-file

# Build and install DirectFB2
meson --cross-file riscv-cross-file build/riscv
meson compile -C build/riscv
DESTDIR=$BUILDROOT_OUT/host/riscv32-buildroot-linux-gnu/sysroot meson install -C build/riscv

2. DirectFB-examples Cross Compilation

# Clone DirectFB-examples under semu
cd semu/
git clone https://github.com/directfb2/DirectFB-examples.git
cd DirectFB-examples/

# Download cross-compilation file
wget https://gist.githubusercontent.com/shengwen-tw/098da8c41ba7fbb9162ddaa4ff62b29e/raw/5ab962990d19a8bd1a8f378ef9b0b0ef1c5fb36a/riscv-cross-file

# Build DirectFB-examples
meson --cross-file riscv-cross-file build/riscv
meson compile -C build/riscv

See also:

• DirectFB2 Cross File for RISC-V
• DirectFB2 Patch for RISC-V

References:

• Back to DirectFB! - FOSDEM 2022
• Resolve __NR_futex undefined error
• Meson Cross File Documentation

[jserv]

• DirectFB2 Cross File for RISC-V
• DirectFB2 Patch for RISC-V

Did you submit patches to buildroot project?

[shengwen-tw]

• DirectFB2 Cross File for RISC-V
• DirectFB2 Patch for RISC-V

Did you submit patches to buildroot project?

I followed DirectFB2's official guide to build it with meson, but leveraged the toolchain and rootfs created by Buildroot (i.e., there is no patch to Buildroot involves here). Note that the sysroot by Buildroot provides necessary system headers for cross compilation.

For the patch to the DirectFB2, so far I have no idea why cross compilation will complain the futex syscall (i.e., __NR_futex) to be undefined without the patch. I saw someone else also encountered a similar problem and their solution resolved the error I had here.

[jserv]

I tend to merge #34 if the proposed changes can be cleaned up. Meanwhile, we shall consider to implement virtio-input. cleverca22/mini-rv32ima implements virtio-input (and even virtio-snd!) in a minimalist way, and it would be great both virtio-gpu and virtio-input can work with DirectFB2.

[shengwen-tw]

A pull request to resolve the compilation error of DirectFB2 for RV32 is created.
Alias __NR_futex with __NR_futex_time64 for RV32 platform #150

[shengwen-tw]

The virtio-input specifies the driver to populate the input event buffers for the device to write:

struct virtio_input_event { 
  le16 type;  /* Keyboard or mouse */
  le16 code;  /* Key or movement */
  le32 value;
};

The type and code information for the Linux can be acquired from
https://github.com/torvalds/linux/blob/master/include/uapi/linux/input-event-codes.h

[jserv]

A simple frame-buffer for Linux is described as a frame-buffer setup by firmware or the bootloader, assuming the display hardware is already configured to scan out from the memory specified by the reg property. The relevant configuration is CONFIG_FB_SIMPLE. RISCVBox demonstrates how to straightforwardly set up its ramfb to enable a simple framebuffer device to function with Linux.

I am considering to integrate simple framebuffer prior to the VirtIO GPU device for the sake of minimal changes required for graphics enablement.

[shengwen-tw]

Understanding evdev is a good article for understanding event handling (i.e., /dev/input/eventX) in the Linux kernel.

Additionally, as a side note for EV_ABS:

The kernel only sends events when the value changes, so even if the actual hardware keeps sending events, you may never see them in the output if the value remains the same. In other words, holding a finger perfectly still on a touchpad creates plenty of hardware events, but you won't see anything coming out of the event node.

[shengwen-tw]

Record a sample code for reading Linux input events in C:
https://gist.github.com/shengwen-tw/964b88c402867c76bd7553b3c8c48c99

Meanwhile, lsinput is also convenient for listing all input events and can be installed using:

sudo apt install input-utils

[shengwen-tw]

kernel.org provides an introduction to event codes of the Linux kernel:
https://www.kernel.org/doc/html/v6.2/input/event-codes.html#guidelines

[shengwen-tw]

Virtqueues and virtio ring: How the data travels by Eugenio Pérez Martín explains the concept of Available ring, Used ring, and Descriptor table of the VirtIO.

[shengwen-tw]

The key codes of the SDL2 can be found at:
https://github.com/libsdl-org/SDL/blob/SDL2/include/SDL_keycode.h

[shengwen-tw]

The header file of the virglrender, which contains references to the usable API, can be found at:
https://gitlab.freedesktop.org/virgl/virglrenderer/-/blob/main/src/virglrenderer.h

[shengwen-tw]

EGL

EGL is an interface between Khronos rendering APIs (such as OpenGL, OpenGL ES or OpenVG) and the underlying native platform windowing system.

Reference: https://en.wikipedia.org/wiki/EGL_(API)

[shengwen-tw]

GLX

GLX (initialism for "OpenGL Extension to the X Window System") is an extension to the X Window System core protocol providing an interface between OpenGL and the X Window System as well as extensions to OpenGL itself.

Reference: https://en.wikipedia.org/wiki/GLX

[shengwen-tw]

GLES

OpenGL for Embedded Systems (OpenGL ES or GLES) is a subset[2] of the OpenGL computer graphics rendering application programming interface (API) for rendering 2D and 3D computer graphics such as those used by video games, typically hardware-accelerated using a graphics processing unit (GPU).

Reference: https://en.wikipedia.org/wiki/OpenGL_ES

[shengwen-tw]

crosvm (The Chrome OS Virtual Machine Monitor) is a Rust-implemented virtual machine monitor (VMM) based on Linux’s KVM hypervisor, with a focus on simplicity, security, and speed.

It provides a reference for implementing virtio-gpu with 3D rendering mode as some operations are not well documented in the Virtual I/O Device Committee Specification:
https://github.com/google/crosvm/blob/main/devices/src/virtio/gpu/protocol.rs

[shengwen-tw]

Since VirGL is not well documented even in the latest VirtIO spec, we collect some useful materials here:

1. https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg02136.html
2. https://elixir.bootlin.com/qemu/v8.0.4/source/contrib/vhost-user-gpu/virgl.c
3. https://github.com/Keenuts/virtio-gpu-documentation/blob/master/src/virtio-gpu.md

[shengwen-tw]

kmscube is a little demonstration program for how to drive bare metal graphics without a compositor like X11, wayland or similar, using DRM/KMS (kernel mode setting), GBM (graphics buffer manager) and EGL for rendering content using OpenGL or OpenGL ES.