Skip to content

build: docker runtime build is not reproducible #133

Open

Description

@f321x

The current docker build of the type2-runtime is not reproducible, which means compiling it on different machines or at different times produces binaries that are not byte-for-byte similar (different hash).
This is an issue for downstream projects trying to distribute appimages in a reproducible way so users can easily verify the appimage actually contains the source code at a specific commit by building the appimage themselves and comparing it against the distributed appimage.

Currently i apply the following patch on the type2-runtime repo at commit 5e7217b7cfeecee1491c2d251e355c3cf8ba6e4d to make the build more reproducible.
Some parts might be overkill but it would be great if the type2-runtime project could consider reproducibility in their development process and incorporate some trivial changes like pinning versions in the docker image to make it simpler to use for downstream projects.

From 0c54d91dd1d33235ae97566600e692edfb613642 Mon Sep 17 00:00:00 2001
From: f321x <f@x.com>
Date: Thu, 10 Jul 2025 17:45:20 +0200
Subject: [PATCH] make docker build reproducible

attempts to make the docker build more reproducible by:
* pinning the docker image (alpine:3.21) to a hash
* version pinning the apk packages in the dockerfile
* setting TZ, LC_ALL and SOURCE_DATE_EPOCH in the container
* only building single threaded (make -j1)
* use a fixed build directory in `build-runtime.sh` instead of mktemp
* prevent linker from adding build id (-Wl,--build-id=none)
* replace absolute build paths in debug info with relative paths
  (-fdebug-prefix-map=$(PWD)=.)
* replace absolute paths in all compiler output with relative paths
  (-ffile-prefix-map=$(PWD)=.)
* stop adding gnu-debuglink to runtime binary
---
 scripts/build-runtime.sh               | 18 +++++++++++----
 scripts/common/install-dependencies.sh |  2 +-
 scripts/docker/Dockerfile              | 32 ++++++++++++++++++++++----
 src/runtime/Makefile                   |  2 +-
 4 files changed, 42 insertions(+), 12 deletions(-)

diff --git a/scripts/build-runtime.sh b/scripts/build-runtime.sh
index 3ce3b91..e11f082 100755
--- a/scripts/build-runtime.sh
+++ b/scripts/build-runtime.sh
@@ -8,8 +8,10 @@ set -euo pipefail
 out_dir="$(readlink -f "$(pwd)")"/out
 mkdir -p "$out_dir"

-# we create a temporary build directory
-build_dir="$(mktemp -d -t type2-runtime-build-XXXXXX)"
+# we create a temporary build directory with a fixed name for reproducibility
+build_dir="$(readlink -f "$(pwd)")"/build-runtime-temp
+rm -rf "$build_dir"
+mkdir -p "$build_dir"

 # since the plain ol' Makefile doesn't support out-of-source builds at all, we need to copy all the files
 cp -R src "$build_dir"/
@@ -17,13 +19,14 @@ cp -R src "$build_dir"/
 pushd "$build_dir"

 pushd src/runtime/
-make -j"$(nproc)" runtime
+make -j1 runtime

 file runtime

 objcopy --only-keep-debug runtime runtime.debug

-strip --strip-debug --strip-unneeded runtime
+# strip --strip-debug --strip-unneeded runtime
+strip --strip-all runtime

 ls -lh runtime runtime.debug

@@ -50,7 +53,7 @@ fi
 mv runtime runtime-"$architecture"
 mv runtime.debug runtime-"$architecture".debug

-objcopy --add-gnu-debuglink runtime-"$architecture".debug runtime-"$architecture"
+# objcopy --add-gnu-debuglink runtime-"$architecture".debug runtime-"$architecture"

 # "classic" magic bytes which cannot be embedded with compiler magic, always do AFTER strip
 # needs to be done after calls to objcopy, strip etc.
@@ -61,3 +64,8 @@ cp runtime-"$architecture" "$out_dir"/
 cp runtime-"$architecture".debug "$out_dir"/

 ls -al "$out_dir"
+
+# cleanup
+popd  # return to build_dir
+popd  # return to original working directory
+rm -rf "$build_dir"
diff --git a/scripts/common/install-dependencies.sh b/scripts/common/install-dependencies.sh
index 0e21cdb..5237079 100755
--- a/scripts/common/install-dependencies.sh
+++ b/scripts/common/install-dependencies.sh
@@ -39,7 +39,7 @@ tar xf 0.5.2.tar.gz
 pushd squashfuse-*/
 ./autogen.sh
 ./configure LDFLAGS="-static"
-make -j"$(nproc)"
+make -j1
 make install
 /usr/bin/install -c -m 644 ./*.h '/usr/local/include/squashfuse'
 popd
diff --git a/scripts/docker/Dockerfile b/scripts/docker/Dockerfile
index 07b6533..fba9c6e 100644
--- a/scripts/docker/Dockerfile
+++ b/scripts/docker/Dockerfile
@@ -1,13 +1,35 @@
-FROM alpine:3.21
+FROM alpine:3.21@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c

 # includes dependencies from https://git.alpinelinux.org/aports/tree/main/fuse3/APKBUILD
 RUN apk add --no-cache \
-    bash alpine-sdk util-linux strace file autoconf automake libtool xz \
-    eudev-dev gettext-dev linux-headers meson \
-    zstd-dev zstd-static zlib-dev zlib-static clang musl-dev mimalloc-dev
+    bash=5.2.37-r0 \
+    alpine-sdk=1.1-r0 \
+    util-linux=2.40.4-r1 \
+    strace=6.12-r0 \
+    file=5.46-r2 \
+    autoconf=2.72-r0 \
+    automake=1.17-r0 \
+    libtool=2.4.7-r3 \
+    xz=5.6.3-r1 \
+    eudev-dev=3.2.14-r5 \
+    gettext-dev=0.22.5-r0 \
+    linux-headers=6.6-r1 \
+    meson=1.6.1-r0 \
+    zstd-dev=1.5.6-r2 \
+    zstd-static=1.5.6-r2 \
+    zlib-dev=1.3.1-r2 \
+    zlib-static=1.3.1-r2 \
+    clang19=19.1.4-r0 \
+    musl-dev=1.2.5-r9 \
+    mimalloc2-dev=2.1.7-r0

 COPY scripts/common/install-dependencies.sh /tmp/scripts/common/install-dependencies.sh
 COPY patches/ /tmp/patches/

+# Set environment variables for reproducible build
+ENV SOURCE_DATE_EPOCH=1640995200
+ENV TZ=UTC
+ENV LC_ALL=C
+
 WORKDIR /tmp
-RUN bash scripts/common/install-dependencies.sh
+RUN bash scripts/common/install-dependencies.sh
\ No newline at end of file
diff --git a/src/runtime/Makefile b/src/runtime/Makefile
index 9fd4165..3a3cbaa 100644
--- a/src/runtime/Makefile
+++ b/src/runtime/Makefile
@@ -1,6 +1,6 @@
 GIT_COMMIT := $(shell cat version)
 CC            = clang
-CFLAGS        = -std=gnu99 -Os -D_FILE_OFFSET_BITS=64 -DGIT_COMMIT=\"$(GIT_COMMIT)\" -T data_sections.ld -ffunction-sections -fdata-sections -Wl,--gc-sections -static -Wall -Werror -static-pie
+CFLAGS        = -std=gnu99 -Os -D_FILE_OFFSET_BITS=64 -DGIT_COMMIT=\"$(GIT_COMMIT)\" -T data_sections.ld -ffunction-sections -fdata-sections -Wl,--gc-sections -Wl,--build-id=none -static -Wall -Werror -static-pie -fdebug-prefix-map=$(PWD)=. -ffile-prefix-map=$(PWD)=.
 LIBS          = -lsquashfuse -lsquashfuse_ll -lzstd -lz -lfuse3 -lmimalloc

 all: runtime
--
2.50.0

Activity

  1. lalten commented on Sep 24, 2025

    @lalten

    Big +1 for this. Also the "continuous build" tag is very much the opposite of where modern build systems are headed: hermetic, deterministic and reproducible builds, provenance attestations, immutable releases, ...
    It would be great to modernize the approach here to make the project more suitable for consumption by more modern systems

  2. TheAssassin commented on Nov 8, 2025

    @TheAssassin
    Member

    Please send a PR so that we can discuss these suggestions one by one.

  3. lalten commented on Nov 8, 2025

    @lalten

    Here's something that I tried: lalten#9

    It's using Bazel as build system and compiles code and dependencies fully from source using a hermetic clang toolchain.

    It does not support using mimalloc and muslc at the same time
    Only amd64 and aarch64 builds are supported

    I could set up debug fission in the future if need be

    The runtime binaries are quite a bit smaller (440kB/465kB) than the "official" ones (915kB/922kB), probably due to compiling the entire dependency tree with optimizations on and being able to strip symbols that aren't needed.

    Note: I have not run extensive tests using these runtimes yet.

  4. probonopd commented on Nov 8, 2025

    @probonopd
    Member

    Thanks @lalten. I know you want to help and I appreciate it. However,...

    It's using Bazel as build system

    ...I don't know anything about Bazel, and don't have the time to learn it.

    My experience with all of these tools are that initially things work great, but after some time things start to change and break, and then no one knows what to do. Usually those "in the know" tell one to update things, which usually results in even more breakages, fixing which is the least favorite way to spend my free time.

    This is why, to quote from the README:

    Do NOT add a complicated "build system" (like autotools, CMake,...) other than the existing simple Makefile and bash

    I would like to keep it this way so this project will still build just fine for decades to come without maintenance/updates needed on the build system.

    Can it be ported to just Makefiles and POSIX shell?

  5. lalten commented on Nov 8, 2025

    @lalten

    Thanks @lalten. I know you want to help and I appreciate it. However,...

    It's using Bazel as build system

    ...I don't know anything about Bazel, and don't have the time to learn it.

    Bazel can be daunting with its internal complexity, but you don't need to become a Bazel expert just to use it.

    This is why, to quote from the README:

    Do NOT add a complicated "build system" (like autotools, CMake,...) other than the existing simple Makefile and bash

    Maybe it's time to rethink that 🤷 To be honest I think the Bazel setup in that fork PR is much simpler than the current system. I can see how that might not be obvious when you haven't used Bazel before. But even just the number of lines (41 BUILD.bazel, 19 MODULE.bazel) is smaller than the current one.

    Can it be ported to just Makefiles and POSIX shell?

    Definitely not. Unless you want to wrap the existing build in a shell wrapper with a single line: bazel build --compilation_mode opt //:runtimes

  6. lalten commented on Nov 8, 2025

    @lalten

    While it would be definitely nice to land the Bazel build in this repo that wasn't actually my original intention. Like others (#92 (comment)) I need the type-2 runtime as a guaranteed never-changing build artifact for https://github.com/lalten/rules_appimage.
    Currently the rules_appimage Bazel ruleset depends on a random "release" that I manually did on my type2-runtime fork in 2022. My intention is to get that under control and hermetically and deterministically build by depend on the sources, as is idiomatic with Bazel.
    lalten#9 is my approach at achieveing that, since you declared in probonopd/static-tools#41 (comment) that you have no interest in providing stable releases

  7. lalten commented on Nov 8, 2025

    @lalten

    If you want to try the Bazel build, the only thing you need is bazelisk to provide the bazel command. All other dependencies are fully managed. So basically:

    $ git clone -b bazel https://github.com/lalten/type2-runtime.git
    $ cd type-2 runtime
    $ bazel build //:runtimes

    These commands will work on Linux, Macos, and probably also Windows (untested). The resulting cross-compiled binaries (aarch64 and amd64) should be byte identical.

  8. probonopd commented on Nov 8, 2025

    @probonopd
    Member

    To be honest I think the Bazel setup in that fork PR is much simpler than the current system. I can see how that might not be obvious when you haven't used Bazel before. But even just the number of lines (41 BUILD.bazel, 19 MODULE.bazel) is smaller than the current one.

    Yes, that's how it always starts. Fewer LOC = some third party tool does "magic".
    And those third party tools, unlike Makefiles and POSIX shells, keep changing.

    Which is almost guaranteed to break in the coming decades if we don't keep updating stuff all the time.

    (Please don't take it personally @lalten but it is a recurring pattern that I already had foreseen when I wrote that line in the README.)

  9. TheAssassin commented on Nov 8, 2025

    @TheAssassin
    Member

    @lalten it's pointless to argue here from my experience. I personally don't think CMake is complex in any way. @probonopd's position won't change. I've tried. This makes working on this codebase and its build toolchain extremely annoying, time consuming and unattractive.

  10. Beethoven-n commented on Dec 2, 2025

    @Beethoven-n

    you should probably just use make or shell for this. the build tooling for appimages is already relatively complex, you don't need to add a dependency just because that's what you know

  11. probonopd commented on Dec 4, 2025

    @probonopd
    Member

    Yes, POSIX sh is welcome in this repo.

  12. TheAssassin commented on Oct 19, 2026

    @TheAssassin
    Member

    Should be improved by #159. The concept should generally also work for other paths. Stripping everything just makes debugging harder for no good reason.

  13. f321x commented on Oct 20, 2026

    @f321x
    ContributorAuthor

    Would you be willing to review a PR introducing a Docker build chain based on a Debian base image (instead of the current Alpine one)?

    Currently we patch the Alpine packages (as in the patch above) to a version pin, however this is a maintenance burden as the Alpine repository doesn't serve older pinned packages. So before most releases (every couple month) we have to update the patch to pin the recent versions. This also makes it harder for builders to reproduce our older releases.

    With a Debian base image we could pin the apt repository (like here and here) which is more stable.
    This could either be an alternative to the Alpine based one or a full replacement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions