Running containers on the 'wrong' architecture.

Published: 2022-08-04 00:08
Last Updated: 2022-08-17 12:32
Author(s): Enimihil
Tags: #buildah #containers #emulation #linux #podman #qemu #skopeo

Containers are pretty useful, like a chroot that's been given extra powers, to isolate networking, user ids, pids, and all sorts of other things. And without the overhead of a whole virtual machine. However, that does limit them to running software that would actually run on your actual machine.

I use a ppc64le machine as a daily driver, so lots of things are 'awkard', if they assume that x86_64 is the only thing that exists. However, there is a tool that can emulate other CPUs and run programs in a different instruction set: qemu. You don't—it turns out—even need to emulate an entire hardware platform, or need special virtualization-specific support (that's just for speed). A qemu user emulation tool can run binaries for a foreign architecture (but otherwise compatible with your OS kernel) by emulating the CPU and translating system calls.

User emulation is already a lot like a container, and can make things like certain kinds of cross compiling or development work easier (as well as making it possible to run proprietary software on your architecture of choice).

On Gentoo, you need to have app-emulation/qemu set up to include the architectures you want to emulate in the QEMU_USER_TARGETS use_expand variable. (And for using this within containers, you want the +static-user useflag set as well).

Then, assuming you have a C program like this:

hello.c (Source)

#include <stdio.h>

int main() {
    printf("%s", "Hello from main()\n");
    return 0;
}

And you compile it for a different native and non-native architectures

$ gcc -o hello-ppc64le-dyn hello.c
$ gcc -o hello-ppc64le --static  hello.c
$ x86_64-multilib-linux-gnu-gcc -o hello-x86_64-dyn hello.c
$ x86_64-multilib-linux-gnu-gcc -o hello-x86_64 --static hello.c

We can then run the two native ones, and see the expected output.

$ ./hello-ppc64le-dyn
Hello from main()
$ ./hello-ppc64le
Hello from main()
$ ./hello-x86_64
bash: ./hello-x86_64: cannot execute binary file: Exec format error
$ ./hello-x86_64-dyn
bash: ./hello-x86_64-dyn: cannot execute binary file: Exec format error

The qemu-x86_64 program can run these, however.

$ qemu-x86_64 ./hello-x86_64
Hello from main()
$ qemu-x86_64 ./hello-x86_64-dyn
Hello from main()

(Assuming the cross compilation setup has the right environment for the dynamically linked version)

The Linux kernel has a feature that allows configuring an interpreter for these binaries, in much the same way that you can embed a #! (shebang) line into a script: binfmt_misc.

This can be configured through various mechanisms on different distros, but on Gentoo the openrc init system supplies a /etc/init.d/binfmt service to enable that mapping by dropping a file into /etc/binfmt.d/.

Mine looks like this:

qemu-user.conf (Source)

#:qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-aarch64:OCF
#:qemu-aarch64_be:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-aarch64_be:OCF
#:qemu-alpha:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x26\x90:\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-alpha:OCF
#:qemu-arm:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-arm:OCF
#:qemu-armeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x28:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-armeb:OCF
#:qemu-hexagon:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xa4\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-hexagon:OCF
#:qemu-hppa:M::\x7f\x45\x4c\x46\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x0f:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-hppa:OCF
:qemu-i386:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x03\x00:\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-i386:OCF
:qemu-i486:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x06\x00:\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-i386:OCF
#:qemu-m68k:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x04:\xff\xff\xff\xff\xff\xff\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-m68k:OCF
#:qemu-microblaze:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xba\xab:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-microblaze:OCF
#:qemu-microblazeel:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xab\xba:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-microblazeel:OCF
#:qemu-mips:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mips:OCF
#:qemu-mips64:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mips64:OCF
#:qemu-mips64el:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mips64el:OCF
#:qemu-mipsel:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipsel:OCF
#:qemu-mipsn32:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-mipsn32:OCF
#:qemu-mipsn32el:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x08\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-mipsn32el:OCF
#:qemu-or1k:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x5c:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-or1k:OCF
:qemu-ppc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x14:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc:OCF
:qemu-ppc64:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x15:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-ppc64:OCF
#:qemu-riscv32:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-riscv32:OCF
#:qemu-riscv64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xf3\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-riscv64:OCF
#:qemu-s390x:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x16:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-s390x:OCF
#:qemu-sh4:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-sh4:OCF
#:qemu-sh4eb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2a:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sh4eb:OCF
#:qemu-sparc:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x02:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sparc:OCF
#:qemu-sparc32plus:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x12:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sparc32plus:OCF
#:qemu-sparc64:M::\x7fELF\x02\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x2b:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-sparc64:OCF
:qemu-x86_64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00:\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-x86_64:OCF
#:qemu-xtensa:M::\x7fELF\x01\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x5e\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/usr/bin/qemu-xtensa:OCF
#:qemu-xtensaeb:M::\x7fELF\x01\x02\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x5e:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff:/usr/bin/qemu-xtensaeb:OCF

So with that file in place and /etc/init.d/binfmt start (run as root, and possibly enabled as a boot service).

We can then just run the non-native executables 'normally':

$ ./hello-x86_64
Hello from main()
$ ./hello-x86_64-dyn
Hello from main()

This is pretty cool, and useful on its own, but you quickly have to figure out how to install dependencies and deal with a project's build system to really use it for much other than toy examples.

But we can use the static binaries directly in a container, if we want.

Containerfile.x86_64 (Source)

FROM scratch
COPY hello-x86_64 /
CMD [ "/hello-x86_64" ]

Then executing:

$ $ buildah bud --platform=linux/amd64 -t localhost/hello:x86_64 Containerfile.x86_64
STEP 1/3: FROM scratch
STEP 2/3: COPY hello-x86_64 /
STEP 3/3: CMD [ "/hello-x86_64" ]
COMMIT localhost/hello:x86_64
Getting image source signatures
Copying blob 0efe8d85f660 skipped: already exists
Copying config 6326d28447 done
Writing manifest to image destination
Storing signatures
--> 6326d284477
[Warning] one or more build args were not consumed: [TARGETARCH TARGETOS TARGETPLATFORM]
Successfully tagged localhost/hello:x86_64
6326d284477b07a5e60adbda3a8861f7ffabf4e49a55fd24706c143dd2622b95

We then get our container built, and can run it:

$ podman run --rm -it localhost/hello:x86_64
Hello from main()

And that works great, but if we try the dynamically linked version:

Containerfile.x86_64-dyn (Source)

FROM scratch
COPY hello-x86_64-dyn /
CMD [ "/hello-x86_64-dyn" ]

$ $ buildah bud --platform=linux/amd64 -t localhost/hello:x86_64 Containerfile.x86_64
STEP 1/3: FROM scratch
STEP 2/3: COPY hello-x86_64-dyn /
STEP 3/3: CMD [ "/hello-x86_64-dyn" ]
COMMIT localhost/hello:x86_64-dyn
Getting image source signatures
Copying blob 14266541c9b2 done
Copying config 6e4c2c5dfb done
Writing manifest to image destination
Storing signatures
--> 6e4c2c5dfbd
[Warning] one or more build args were not consumed: [TARGETARCH TARGETOS TARGETPLATFORM]
Successfully tagged localhost/hello:x86_64-dyn
6e4c2c5dfbd8eb283d51b4929229561fd10a1df4439d97646d6b3a29355ea95e

$ podman run --rm -it localhost/hello:x86_64-dyn
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory

We get something different. The dynamic executable requires the dynamic linker within the container, and we only put the executable file there.

But we are able to execute the binaries through qemu-x86_64 just fine, even within the container, whose own root filesystem doesn't have the emulator to suport our native CPU.

But the dynamic linker would work if a whole chroot-style image was present in the container.

Doing a simple podman run --rm -it docker.io/library/ubuntu:latest will pull an appropriate image for the architecture we're running on, and we can see that if we use skopeo to inspect the image locally:

$ podman run --arch ppc64le --rm -it docker.io/library/ubuntu:latest
trying to pull docker.io/library/ubuntu:latest...
getting image source signatures
copying blob 6f8c2fc0a4f9 skipped: already exists
copying config fb133e10be done
writing manifest to image destination
storing signatures
root@6e00a22706a8:/# uname -m
ppc64le
root@6e00a22706a8:/#
exit
$ skopeo inspect containers-storage:docker.io/library/ubuntu
{
    "name": "docker.io/library/ubuntu",
    "digest": "sha256:4e5d0032fffc670be1788b945476b5997b29b50aec6e84af7a76e8fb030c4326",
    "repotags": [],
    "created": "2022-08-02t01:31:14.74904496z",
    "dockerversion": "20.10.12",
    "labels": null,
    "architecture": "ppc64le",
    "os": "linux",
    "layers": [
        "sha256:6f8c2fc0a4f976c1c0bd1c0e14022b3ed8b7c83cdb247c83016052c3678aaf28"
    ],
    "env": [
        "path=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}

But we can choose a non-native architecture, and assuming we configured the binfmt correctly and the image is available:

$ podman run --arch amd64 --rm -it docker.io/library/ubuntu:latest
Trying to pull docker.io/library/ubuntu:latest...
Getting image source signatures
Copying blob d19f32bd9e41 skipped: already exists
Copying config df5de72bdb done
Writing manifest to image destination
Storing signatures
root@159ea7cdac2c:/# uname -m
x86_64
root@159ea7cdac2c:/#
exit
$ skopeo inspect containers-storage:docker.io/library/ubuntu
{
    "Name": "docker.io/library/ubuntu",
    "Digest": "sha256:42ba2dfce475de1113d55602d40af18415897167d47c2045ec7b6d9746ff148f",
    "RepoTags": [],
    "Created": "2022-08-02T01:30:56.165288114Z",
    "DockerVersion": "20.10.12",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:d19f32bd9e4106d487f1a703fc2f09c8edadd92db4405d477978e8e466ab290d"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}

Then we can see we were running the container with the non-native architecture just fine.

Using that container to build software, and to package for non-native architectures can all be done locally.