Skip to content

Linux VM Kernel Debugging

Jump to bottom Edit New page
Ross Philipson edited this page Feb 7, 2015 · 47 revisions

These are some handy steps for doing Linux live kernel debugging on the OpenXT platform. OpenXT makes this readily doable by using serial over IP within QEMU to connect the remote debugger GDB to the kernel debugger KGDB. Of course everyone has their favorite distro but for this we will stick to just one – Debian. Most of the steps should be the same with other flavors aside from the package management bits and specifics of rebuilding the kernel sources.

Beforehand

Here are some things to setup up first and to keep in mind while going through this:

  • This is based on using Debian Wheezy HVMs. That means apt package manager and .deb package files.
  • Throughout this guide:
  • target is the VM that is being kernel debugged
  • host is the debugger, the VM where GDB is remotely connecting to the target.
  • For simplicity, it is assumed both the host and target are the same OS or distro. The paths and users on both are the same. Also password-less SSH login and sudo setup are assumed.
  • A basic set of development tools is needed on the target to build the kernel - apt-get install build-essential should be sufficient (note sometimes dpkg-dev needs to be installed manually).

So to get started, install OpenXT and create 2 Debian Wheezy HVMs. For debugging to work, SELinux and stubdoms need to be turned off. Stubdoms are disabled in the Advanced tabs for the VMs. To disable SELinux, run a terminal, use nr to log into the admin role then run rw to make the rootfs read-write. Next edit /etc/selinux/config and set SELINUX=permissive and save. Reboot.

Building the Kernel

The KGDB debugger components need to be enabled in the target kernel. This requires building a custom kernel. On the target, get the kernel source package - in this case for Wheezy:

$ sudo apt-get install linux-source-3.2

This drops off a tarball /usr/srclinux-source-3.2.tar.bz2. Make a directory called ~/kernel and extract the tarball there. Change to ~/kernel/linux-source-3.2 that has the kernel sources.

Next the kernel sources need a configuration file. The simplest thing is to start with the one for the currently installed Wheezy kernel. Copy /boot/config-3.2.0-4-amd64 as .config in the current sources dir (note the current config file might have a different name). These are the setting that should be enabled/disabled in the .config:

# CONFIG_DEBUG_RODATA is not set
CONFIG_DEBUG_INFO=y
CONFIG_FRAME_POINTER=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y

Briefly, these are the base set of kernel debug features that need to be modified. CONFIG_DEBUG_RODATA marks the .text section RO and prevent KGDB from emitting break-point instruction in the code. CONFIG_DEBUG_INFO enables debug information/symbolic data in the kernel image. CONFIG_FRAME_POINTER preserves stack frame pointers makes stack back-tracing and changing frames easier. The last two enable the KGDB debugger extensions using a serial console.

Though you can edit .config, it is usually done using one of the editing interfaces. Using make menuconfig, the setting are here:

  • "Kernel hacking"
  • "Compile the kernel with debug info"
  • "Compile the kernel with frame pointers"
  • "KGDB: kernel debugger --->"
  • "KGDB: use kgdb over the serial console"
  • "Write protect kernel read-only data structures"

Other debugging features may be enabled/disabled at this point too. It is also recommended that a local version suffix is used for the kernel name to help identify the build. Under "General setup" enter a suffix in "Local version" (.e.g. "-mykgdb"). Once all this is done, save the configuration and:

$ make deb-pkg

This will produce 3 packages in the ~/kernel dir:

  • linux-image-3.2.65-kgdb_3.2.65-kgdb-1_amd64.deb
  • linux-headers-3.2.65-kgdb_3.2.65-kgdb-1_amd64.deb
  • linux-libc-dev_3.2.65-kgdb-1_amd64.deb

Installing the Kernel

This is straight forward, simply install the .deb packages:

$ sudo dpkg -i linux-image-3.2.65-kgdb_3.2.65-kgdb-1_amd64.deb
$ sudo dpkg -i linux-headers-3.2.65-kgdb_3.2.65-kgdb-1_amd64.deb

This will create a several files in /boot that can be identified by the local version suffix used (the kernel image, copy of the config, initramfs and system map) and install the modules to /lib/modules as expected. The grub boot loader config will also be updated. The headers will end up under /usr/src.

The new kernel will need a couple of kernel command line parameters to tell KGDB how to connect over serial. Edit /etc/default/grub and set GRUB_CMDLINE_LINUX="kgdboc=ttyS0,115200 kgdbcon". Note if there were already values set, the KGDB ones can be appended. Then update grub and reboot:

$ sudo update-grub; reboot

Setup the Debugger

In the host VM, first install the debugger with apt-get install gdb. Next the host needs to have the built kernel sources from the target for gdb to work on. The best way to get this (and to synchronize further changes) is to just rsync it. Create a ~/kernel dir and sync up the sources and build output:

$ rsync myuser@mytarget:~/kernel/linux-source-3.2 ~/kernel

The host is now ready to do some debugging stuffs. The last step is to connect the emulated serial ports from the two VMs.

Connecting Serial Ports

The emulated serial ports in QEMU can be setup to pipe their communications over TCP/IP - this is how the two VMs will be connected. This could be done either way but in this example the emulated serial port for the host VM will be the listening end and the target will connect. The next steps need to be done in a terminal in dom0 as root. Run this to setup the host VM:

$ xec-vm -n <host-vm-name> set extra-xenvm "serial=tcp::4545,server,nowait"

And this to setup the target:

$ xec-vm -n <target-vm-name> set extra-xenvm "serial=tcp:0.0.0.0:4545"

The port is arbitrary but 4545 works just fine. Now, since the host is the listening side, it needs to be started first, then the target. It could be reversed if need be. Start them both up.

Remote Kernel Debugging

Everything is in place to do some actual debugging. To ready the target, the kernel execution must be halted and ready to receive a connection from the debugger this is done two ways. One is to add the kgdbwait parameter to the kernel command line in addition to the parameter that were added earlier. More on that parameter can be found in the KGDB documentation. The second which will be used here is to use the magix of sysrq. In the target VM, open a terminal as root and:

$ echo g > /proc/sysrq-trigger

The VM will freeze and the kernel is now waiting for a debugger connection. Over in the host VM, open a terminal as root and cd to the location of the kernel sources (e.g. /home/myuser/kernel/linux-source-3.2). Start the debugger:

$ gdb ./vmlinux

This will load the kernel image and symbols. At the gdb command prompt, connect to the remote target VM:

(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0

Unless something is wrong, gdb should now be connected and in control of the remote kernel. Run a command like bt to see a stack back-trace of where the kernel is waiting for gdb. Refer to the gdb documentation for further commands. To let the kernel resume in the target, type c. The sysrq-trigger can then be used to break to the debugger again.

Modifications can be made to the kernel image on the target, the changes can be rsync'ed to the host. Then the systems can be rebooted and kernel debugging can be resumed.

Add a custom footer
Add a custom sidebar
Clone this wiki locally