Building Multi-Arch Docker Images on Github Actions

I recently needed to create multi-arch images so that I could spin up containers on my arm-based laptop. Github Actions only runs on x64-86 VMs, which can make building images for other architectures tricky.

Regular old Docker Build

name: Docker
on:
  push:
    branches: ['main']
jobs:
  docker:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3
      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}
      - name: Build and push
        uses: docker/build-push-action@v3
        with:
          push: true
          tags: user/app:latest

With the above Github Actions file, we could build and push to Docker Hub easily. Now let's check out how we can use buildx to build for multiple architectures.

QEMU and Buildx

QEMU is a generic and open source machine emulator and virtualizer. When used as a machine emulator, QEMU can run OSes and programs made for one machine (e.g. an ARM board) on a different machine (e.g. your own PC). By using dynamic translation, it achieves very good performance.

Docker buildx is a Docker plugin that extends the building ability of images by using the BuildKit builder. It allows us to build images for different platforms and architectures.

With a combination of QEMU and buildx, which are conveniently also available as Github Actions, we can easily get our images built on emulated OSes and push onto the docker registry with multi-platform support!

Add the following after the registry checkout on our Github Actions workflow file:

- name: Set up QEMU
  uses: docker/setup-qemu-action@v2
  with:
    platforms: 'arm64,arm'
- name: Set up Docker buildx
  uses: docker/setup-buildx-action@v2

Note: The QEMU setup step has to be put before the buildx step!

By default, setup-qemu-action will create many VMs, most of which you might not care about. Make sure to put in the platforms you actually want to build for in the platforms: field to limit this to only ones you actually need.

Arch-specific Dockerfile instructions

The docker-build-push-action will provide the --platform flag to docker buildx. This will set the following build ARGs automatically:

TARGETPLATFORM - platform of the build result. Eg linux/amd64, linux/arm/v7, windows/amd64.
TARGETOS - OS component of TARGETPLATFORM
TARGETARCH - architecture component of TARGETPLATFORM
TARGETVARIANT - variant component of TARGETPLATFORM
BUILDPLATFORM - platform of the node performing the build.
BUILDOS - OS component of BUILDPLATFORM
BUILDARCH - architecture component of BUILDPLATFORM
BUILDVARIANT - variant component of BUILDPLATFORM

To use these ARGs in your Dockerfiles, you need to add them like the following example grabbed from Docker docs:

FROM alpine
ARG TARGETPLATFORM
RUN echo "I'm building for $TARGETPLATFORM"

End result

As a result, we can build and push multi-arch images by emulating other architectures on Github's x86_64 VMs using QEMU and building with docker buildx. We can even do arch-specific actions by using the ARGs automatically set by buildx.

Closing notes

Although this is convenient, no land without stones, or meat without bones.

There is one big and one relatively small problem this might cause for you.

1. Super long build times

Your builds might be taking very long after enabling multi-arch builds. This seems to be a known problem with emulating, but you could use some tricks to make less emulated code run while building.

In my case, I was building a Go application inside a build image, then copying that onto a target image in a two-step Dockerfile like so:

FROM golang:1.20-alpine as builder
 
WORKDIR /
 
RUN go env -w GOPRIVATE=github.com/myorganization
 
ARG GITHUB_TOKEN
RUN echo "machine github.com login github password $GITHUB_TOKEN" >> /root/.netrc
 
COPY . .
RUN go build "-ldflags=-s -w" -o /my-application cmd/main.go
 
 
FROM alpine:3.8
 
COPY --from=builder /my-application /app/my-app
 
WORKDIR /app
CMD /app/my-app

Instead of running the go build inside the Dockerfile, I switched to building directly inside the Github Actions using GOOS and GOARCH variables to specify building for a specific arch:

- name: Set up Go 1.20
  uses: actions/setup-go@v1
  with:
    go-version: 1.20
 
- name: Setup netrc
  run: |
    echo "machine github.com login github password ${{ secrets.ACCESS_TOKEN }}" > ~/.netrc
- name: Build application
  run: |
    go env -w GOPRIVATE=github.com/myorganization
    GOOS=linux GOARCH=amd64 go build "-ldflags=-s -w" -o build/my-app-amd64 cmd/main.go
    GOOS=linux GOARCH=arm64 go build "-ldflags=-s -w" -o build/my-app-arm64 cmd/main.go

And then copying those builds into the container using the buildx ARGs:

FROM alpine:3.8
 
ARG TARGETARCH
COPY build/my-app-${TARGETARCH} /app/my-app
 
WORKDIR /app
CMD /app/my-app

In the end, this lowered the whole execution time from ~6m to ~1,5m. I've heard people lower theirs down from ~1h to ~15m as well.

Pushing to AWS ECR creates multiple images

If you're pushing to AWS ECR, you might realize one push is creating many untagged images of type Image and one image of type Image Index containing all the tags, which looks like this:

This is actually normal, since pushing a multi-arch image really is just pushing untagged images for each arch and using an Image Index to redirect the user to the correct architecture image when pulling. The arch-specific images are untagged so that they cannot be easily discovered and pulled, and instead the Image Index that handles this is reached.

It looks like AWS might update the UI to make this look less cluttered in the future.