Week 3: Setting up Linux kernel Test Bench

My task for this week is to compile the remoteproc framework, and setup its test bench. After analyzing the framework’s source code, I figured that it cannot be compiled as a modulable framework/driver. This means that I’ll have to compile the whole kernel in order to test the framework.

To compile a kernel, I’ll just have to go into the root of a kernel source file, and then run make menuconfig (requires the ncurses library) and select the components that we need for our kernel. As expected, I couldn’t choose to make the remoteproc framework as modularized. In fact the framework will be automatically built-in the kernel if I choose to include a platform-specific remoteproc driver. As I haven’t started my driver development yet, I included the remoteproc driver by selecting the ste modem’s remoteproc driver (platform specific). After configuring the kernel build information, I started to compile by simply runnning make. For now, everything seems to be compiling well, and I can see the remoteproc framework’s .o files generated by the compiler.

A normal compiling process of a basic Linux kernel takes a lot of time. Depending on the system used, it can easily goes as long as 1-2 hours. But there is a useful tool to reduce this wait time by sharing the compiling loads on more than one PC in a network. The tool is called distcc and every PC in a network must have its service (called distccd daemon) running in order to make the load sharing possible. My experience with this tool is not utterly successful because I didn’t experience a significantly reduced compilation time (about 40+ mins). Maybe because the time I was compiling, there were not many running machines in our network. After running the distcc monitoring too distccmon-gnome, I found that the compilation was well distributed between two IP addresses, with each IP address having several slot, so the problem should be with my kernel configuration. Maybe I included too many drivers in it because I wasn’t too sure which driver is really needed for my system. I will try to strip as many drivers as I can from my kernel next time to cut the compilation time.

Command to compile using distcc: make -j8 CC=distcc. The “-j8” option means that 8 threads of the compiling process will be created and shared between different machines. We can perfectly change the number of thread that we want by changing this option to -j4, -j10 and etc.

I have two options available to test the kernel. Either I replace my current Linux distribution’s kernel with the one that I just compiled, or I setup a virtual machine to run the kernel. The latter option seems to be the better one because I will not have to worry about ruining my current system if anything goes wrong. I can just reconfigure the virtual machine to make it run again.

For virtualization, there are two most used tools or methods: Virtualbox and KVM. Virtualbox is normally easier to use because it has an easy GUI, and even beginner can setup a virtual machine in no time. KVM on the other hand is more advanced and faster too because it is a kernel based virtual machine. KVM also offers the possibility to load our own kernel instead of the one installed with the linux distribution which I find very interesting as I’ll be changing my kernel quite often as the development goes. The downside is that it is not as easy as Virtualbox to use because at default, there is no GUI offered. But users can install a utility called Virtual Machine Manager (VMM) to have GUI for KVM and thus make it easier to use. But it is always better to just use the command lines because pure geek doesn’t like fancy GUI, and only use commands. Finally the KVM is chosen to be used as our test bench because it offers the possibility to load a custom kernel.

Here is a little tutorial on how to setup a KVM virtual machine:

1. Create a virtual hard drive to install our filesystem:
   qemu-img create -f qcow2 our_hdd_name.qcow2 {size_in_gigabyte}G
2. Run the kvm with a CD image to install a basic linux system:
   kvm -cdrom cd_image.iso our_hdd_name.qcow2
3. Load our virtual machine with the installed hard drive:
   kvm our_hdd_name.qcow2
4. Load our virtual machine with a custom kernel (bzImage type):
   kvm -kernel custom_kernel -initrd initramfs.gz our_hdd_name.qcow2

Note that if we want to load a custom kernel, we have to include the initrd because otherwise the kernel will say “kernel panic” because it couldn’t find any filesystem connected to it. By including the initrd (or initramfs), we create a temporary filesystem to be used with the kernel. For now, I’m still figuring the structure and way to create an initrd image.

Update: I successfully compiled the remoteproc framework without having to compile the whole kernel. By examining the Kbuild’s module documentation, in order to build external modules, we must have a prebuilt kernel available that contains the configuration and header files used in the build. My mistake was that I was trying to compile the module against the system’s kernel. I therefore modified my Makefile to change the kernel source by replacing:

make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules
make -C <path_to_newer_kernel_src> M=$PWD modules

I now have the module file (remoteproc.ko) ready to be inserted at any time. Might try to insert it using to a basic linux system run in Virtualbox (I’m not using KVM this time because with Virtualbox, I can easily share a folder between the host and a guest and that means the remoteproc module that I compiled in host environment can easily be passed to the guest OS).