Reproducible Builds and Citadel

March 22, 2025 · reproducible-builds, supply-chain, security

When you install a binary, you rely on the people and machines that built it. You usually cannot see the build server, and you cannot tell from the binary alone whether it matches the published source.

A binary can be altered at several points: in the source repository, on the build machine, or in the toolchain. Without reproducible builds, checking for this means either accepting the vendor’s statement or reverse engineering the artifact.

How reproducibility works

A build is reproducible when the same source, built with the same toolchain and the same settings, produces bit-identical output on any machine.

If that holds, a release can be checked by people outside the project:

  • Someone rebuilds the published source and compares the hash of their output with the hash of the official artifact.
  • An auditor who reviewed a given source revision can confirm that a release was built from that revision.
  • If a build server is compromised, or a vendor is pressured into shipping a modified binary, the official artifact no longer matches independent rebuilds. The mismatch shows up as soon as someone compares hashes.

What it requires

Most build systems leak details of the build environment into the output. To get identical results, the build has to control them:

  • Timestamps. Embedded build dates are fixed to a known value.
  • Paths. Absolute paths of the build directory are removed or normalized.
  • Ordering and locale. File ordering in archives and locale settings are fixed so they do not change between machines.
  • Toolchain. Compiler versions and build flags are pinned as part of the build definition.

Limits

Reproducibility shows that a binary was built from a given source. It does not show that the source is safe. Malicious code builds reproducibly too, so source review is still needed.

Citadel

Citadel is built from source with OpenEmbedded, and the base OS ships as one signed image. The source will be published. Releases will include reproducible-build checksums and a signed manifest, so anyone can rebuild an image and compare it with the one we ship.

Later posts will cover the build pipeline in more detail, including how the checksums on the release page are produced.