Docker 29 made my bot restart on every deploy

My deploy loop is deliberately boring: a systemd timer pulls a repository every five minutes, computes a fingerprint, and runs docker compose up -d --build only when something changed. Compose, in turn, is supposed to recreate only the containers whose configuration or image changed. So when I pushed a README-only commit and watched the Telegram bot restart, something was wrong.

What I saw

docker compose up -d --build printed Container voyager-golu-1 Recreate even though the bot’s source had not changed. The stt and caddy containers next to it stayed Running. The bot came back within a second — it tolerates restarts well — but “every deploy bounces production” is exactly the kind of thing a pull-based loop must not do.

Narrowing it down

First suspect: the build context. If anything under the bot’s checkout changed between builds, COPY . . would invalidate the cache and produce a new image. But the repository was clean, .dockerignore excluded .git, and find -newer showed nothing touched.

Second suspect: the build itself. Two back-to-back docker compose build runs with no changes produced two different image IDs — while docker image inspect showed the layers and the config were byte-identical and the image’s Created timestamp never moved. Every build step was CACHED. So the ID was changing without anything inside the image changing.

That can only happen if the ID is not the image config digest anymore.

The reproduction is three lines, and it is worth keeping around:

for i in 1 2; do docker compose build -q golu; docker images golu:latest --format '{{.ID}}'; done
docker image inspect golu:latest --format '{{.RootFS.Layers}}'   # identical both times

Two different IDs, one set of layers.

The cause

Docker 29 uses the containerd image store by default. With it, the “image ID” Compose compares is the digest of the top-level manifest list that buildx exports — and by default buildx attaches a provenance attestation manifest to that list. The attestation carries build metadata, including timestamps, so it is different on every build; the manifest list digest therefore changes on every build; Compose sees a new image; the container is recreated.

Nothing was wrong with the cache. The ID was simply measuring something that included “when did you run this build”.

The fix

Disable the default attestations for the build:

export BUILDX_NO_DEFAULT_ATTESTATIONS=1
docker compose up -d --build

With attestations off, the exported artifact is just the image, and its ID is stable as long as the layers are. Two builds in a row now yield the same ID, and a forced redeploy reports Container voyager-golu-1 Running instead of Recreate.

Two footnotes from the investigation:

• Compose’s build.provenance: false looked like the right knob, but in the version I had it did not change the outcome. The environment variable did, so that is what lives in the deploy script, with a comment explaining why.
• If you want attestations for supply-chain reasons, the alternative is to stop comparing by image ID — for example, tag builds with the source commit and let Compose recreate when the tag changes. I did not need the attestations on a single-host setup, so I turned them off.

The lesson

“Only what changed should restart” is a property you have to test, not assume. The cheapest test is the one I should have run on day one: deploy twice with no changes and read the output. Anything that says Recreate the second time is a bug.