Week 4: Setting up the test bench continued

It is almost unbelievable that it has been a month since I joined Aeste. This week has been my least productive week so far because I’d been trying to resolve non-important but work-hindering problems. During week 3, I managed to compile the remoteproc framework independently and the last thing that I needed to do is to just insert the module into any working kernel.

Firstly, I tried to insert into a debian distribution. Unfortunately the debian’s kernel is quite old (3.2.0) and my module was compiled against a later version (3.9.8), so it didn’t work. I received “invalid module format” error while trying to insert. I tried to upgrade the kernel version but I got a dependency error while trying to install the linux-header files.

And then I tried arch linux on my virtualbox because it will install you the latest kernel, so I’ll get the proper linux kernel headers installed to be able to insert my module. I then shared a folder with the host OS that contains the compiled remoteproc module. Even after having a system which uses the same kernel as my module’s I still couldn’t insmod the module. If I compile the module inside the virtual system (against the system’s kernel), I’ll get “unknown symbol in module” error when trying to insert it using insmod. the output from dmesg:

[ 415.595227] remoteproc: Unknown symbol vring_transport_features (err 0)
[ 415.595324] remoteproc: Unknown symbol vring_interrupt (err 0)
[ 415.595361] remoteproc: Unknown symbol vring_new_virtqueue (err 0)
[ 415.595374] remoteproc: Unknown symbol unregister_virtio_device (err 0)
[ 415.595427] remoteproc: Unknown symbol register_virtio_device (err 0)
[ 415.595439] remoteproc: Unknown symbol vring_del_virtqueue (err 0)

This message is giving me a hint that the virtio framework is not installed on the system! I inserted the virtio.ko and virtio_ring.ko, and now I am able to insert the the remoteproc framework! The module insertion is verified by running lsmod command.

Before I succeeded inserting the remoteproc into my arch linux using virtualbox, I tried to use the kvm as my test platform because I thought that it is not possible to compile just the framework without compiling the whole kernel because of the errors that I encountered while trying to insert the remoteproc module. I wasted my time trying to make my initramfs work with my kernel, wasted my time trying to install arch linux on kvm because it ran very slow with arch linux installation!

Because it is too slow to install a linux distribution, I decided to only run kvm using external kernel and external initrd or initramfs file (which will be my test file system). I’ll just include a compiled remoteproc module into my initramfs and try to insert it after booting up the system (which didn’t work because of unknown reasons).

Here is the right step to test a module on a virtual machine:

1. Compile a kernel according to our needs (eg: 3.5.0-x86_66)
2. Compile our module against the kernel that we compiled earlier (explained in my week 3 post)
3. Insert the compiled module into our guest OS which runs exactly the same kernel version as the one that we used in step 1
4. The module should now be inserted without any error

In the meantime, I still continue to study the framework as my side job because I still couldn’t determine the proper interface for the framework. It seems that this framework is targeted to a system that has a side processor which is connected to the IOMMU. So, this means that the side processor will have its own device address and maybe connected through a specific bus such as AGP, PCI and etc. From what I understand (by reading the source code) all the information regarding the processor will be included in its firmware, so the framework will just have to extract the firmware and allocate the necessary memory for the processor. It may seems impossible to implement this through ethernet, but in computing, nearly everything is possible. We will just have to create a virtual interface that will be connected to the IOMMU. This interface will be an interconnect between the ethernet and the iommu.

In conclusion, this week has been pretty much all about switching between different distributions, different virtual platforms, and different bits (from i386 to x86_64) just to insert the module. I’m hoping to be able to have a sturdy test system and to be able to determine the interface of the remoteproc framework so that I can start developing a driver, which is my real task at Aeste!

An important lesson learned this week: Always define carefully all the different components that you will use for your test system! Make sure you use the same kernel version (between the host OS and the one you use to compile your module), same bit compatibility (64 or 32 bits)! Otherwise everything will mix up and cause different kind of errors that you won’t be able to understand!

p/s: an older kernel version may be use for development purpose because using nobody’s using the latest kernel, so the driver that I’ll be developing might not be compatible with user system.