Smartango.com

Building multi arch docker image from linux host

Key concepts

Container manager.There is an actor, a daemon, or something that pull and run the right image when instructucted to do that.
That means, you can use docker, docker swarm mode, kubernetes, k3s, or whatever daemon to monitor or just run container
based on image taken from some place, and that daemon need to understand what to pull.

Architecture: every machine has its own instruction set, device, etc. and its own binary format.
This is “architecture” in this context (the context of cross building a docker image).

Binary format, binfmt in linux. There is a lot of things on this topic, and too advanced.
Anyway, Linux kernel support a kind of pluggable executor based on magic character, characters used
to recognize the architecture, those magics are bounded with executor. Typically /proc fs or /sys fs is used
to setup and inspect kernel runtime setup, so the current binfmt status can be inspected in /proc/sys/fs/binfmt_misc/*
More infos in lwn article: https://lwn.net/Articles/630727/

Docker registry is a service that run somewhere and keep track of images and theirs tags.

Image manifest. This was a new concept to me, tag is not enough for describing an image, an image is identified by
its manifest, that is a json structure that keep reference to the real image(s) tag(s), its architecture, and other infos.

So the manifest, someway, overlap the concept of tag for image, but in fact that is a trick: a default manifest
with the same tag is created and associated to the unique image tag, when manifest is not treated explicitly.

So manifest has a tag, like each image has a tag, but manifest refers to more images.

Deal explicitly with manifest

Docker CLI, the command line interface used to communicate with dockerd, is not intended to deal with manifest,
I do not know why, but it can do it only enabling “experimental” feature, by editing ~/.docker/config.json with

{"experimental": "enabled"}

Then is possible to use it

Commands:

  • docker manifest inspect --verbose TAG
  • docker manifest create MANIFESTTAG [IMAGETAG1 [IMAGETAG2 [IMAGETAG3 […]]]]
  • docker manifest push MANIFESTTAG

inspect is to inspect, create is to create, push is to push the created manifest into the remote registry.

There are limitations here, for example one can create a manifest tag and push it to the registry, but if
the manifest refers to image tags that are not already pushed in the registry the service will complain about it
(or at least I suppose, never tried)

Real building multiarch

What I used to understand it as of few days ago, a Dockerfile refer to main image in FROM section.
But today my idea about it become more complex and I am a bit confused.
Maybe it refers to a Manifest? and the image is automatically selected during the docker build
phase from a set of image, based on matching architecture?

Anyway, one can explicetely refer to a specific architecture by its name, for example using alpine distro
it is a prefix: ‘amd64/’, ‘arm32v7/’, ‘arm64v8/’, …

In Dockerfile is possible to define arguments and their defaults:

ARG ARCH=
FROM ${ARCH}alpine

and give values to arguments in command line during docker build:

docker build -t IMAGETAGNAME --build-args ARCH=arm32v7/

at this point the docker daemon try to build it based on image referred in FROM: arm32v7/alpine

This image contains a filesystem with binary in a format specific to arm32v7 architecture, so you have:

standard_init_linux.go:211: exec user process caused “exec format error”

it means there is no magic matched, so docker tried to execute it directly, but by

docker run --rm --privileged docker/binfmt:a7996909642ee92942dcd6cff44b9b95f08dad64

the folder /proc/sys/fs/binfmt_misc is populated, i.e.:

cat /proc/sys/fs/binfmt_misc/qemu-arm

enabled
interpreter /usr/bin/qemu-arm
flags: OCF
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff

and now again

docker build -t IMAGETAGNAME –build-args ARCH=arm32v7/

will build the image in a qemu-arm machine runned magically for that job.

Then the steps are:

  1. built image tag for each architecture
  2. push image tag for each architecture
  3. create a manifest naming it with a tag and referring to all architecture created in 1. and pushed in 2.
  4. push the manifest created in 3. to the docker registry

There is nothing complex or special, but one just need to understand the concept behind the words: image tag, manifest tag, manifest, architecture, registry, binfmt, crossbuild, …

References

I found it hard to understand concepts, my learning path (I spent 5/6 hours) was to check first this document
https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way/

From there I needed to understand github actions, and github secrets

  1. github actions

An action is defined by

i. a trigger (on: push: branch: master, for example)
ii. and a job, made of: name, environment, steps

i.e. https://github.com/danielecr/uuid-provider/blob/main/.github/workflows/image.yml

  1. github secrets

It is possible to define variable binded with the repo, those are secrets and environment variable

It is possible to access those variables from github action, i.e. by ${{ secrets.MYSECRET }}

But still I was confused about it.

I googled more, and continuosly found reference to windows platform, and everywhere try to sell that
buildx as “easy way”, with result to be more confused.

I finally found https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/
still a lot of advertising for buildx script, but reference to https://hub.docker.com/r/docker/binfmt/tags?page=1&ordering=last_updated
and by guessing it https://github.com/docker/binfmt
and suggestion to use the newer https://github.com/linuxkit/linuxkit (but I still need to see it)

Just for curiosity, the only graphical description of build environment I found was in https://www.stereolabs.com/docs/docker/building-arm-container-on-x86/ but it give little/no infos about binfmt, and it is a content specific to jetbrain device.

Posted

in

by

daniele

Tags: