Compiling for aarch64/riscv64 from qemu user mode chroot

2021-01-28

This is a novel approach that I've been experimenting with lately, to do some legwork towards supporting architectures for which I either don't have physical hardware or else the hardware is limited in capability so much as to make native compilation impractical.

Qemu, in addition to running full virtual machines, also has what is known as "user mode emulation". In a nutshell, when using qemu in this way one is able to run binaries for another architecture without creating virtual hardware or a virtual kernel. In this case, qemu just acts as a translation layer, translating instructions designed for one ISA to instructions that can be executed on the host processor. By not running a virtual kernel on virtual hardware, there is actually a huge savings of system resources.

Setting this up, while not trivial, is surprisingly straightforward. First, a suitable root filesystem must be created. Qemu must be compiled statically, and the static binary copied into the chroot filesystem. Then, by using binfmt-misc, one can register this qemu executable as the program interpreter for binaries of the guest architecture. In this way it is actually possible to set up a functional chroot on a well spec'ed X86_64 machine running Risc-V binaries. There is a bit more to getting this to work, but suffice to say that it does in fact work, within the normal limitations of a chroot.

While running foreign binaries does incur some performance penalty, this method is actually proving to be a significant improvement over, say, compiling for aarch64 on a Raspberry Pi 4, as I have four times the memory and a significantly faster processor with twice the number of cores available. And it has the benefit of not requiring a cross compiler. Instead, one is running a native compiler via an emulation layer. But this eliminates many of the pain points of cross compiling. It's a neat trick.

So what does this mean for HitchHiker? It means that work is now well underway towards running HitchHiker on a BeagleV when they become widely available later this year. I have been eagerly waiting to dip toes into Risc-V as soon as an affordable option was available, and it looks like said option is now on the horizon.

Tags for this post: