The bind mount that pinned an inode

A small one, but it bit hard enough to be worth writing down.

The setup

Caddy runs as a container with its configuration bind-mounted from the repository:

volumes:
  - ./caddy/Caddyfile:/etc/caddy/Caddyfile:ro

The deploy script pulls the repository and, because Compose cannot see the contents of a bind mount changing, follows docker compose up -d with a graceful caddy reload. I added a respond /healthz 200 line, pushed, watched the deploy succeed and the reload report no errors — and /healthz still returned 404.

What was actually happening

Inside the container, /etc/caddy/Caddyfile did not contain the new line. The file on the host did.

The reason is how git writes files. When it updates a tracked file it does not edit it in place; it writes a new file and renames it over the old path. Rename gives the path a new inode. A bind mount of a single file, however, is attached to the inode that existed when the container started. After the pull, the host path pointed at the new inode while the container was still looking at the old one — which was unchanged, and now unlinked from any path on the host.

caddy reload did exactly what it was told: it re-read the file it could see, found it unchanged, and logged “config is unchanged”.

How to see it

The check that settles it takes ten seconds. Compare what the container sees with what the host has, and compare inode numbers:

docker compose exec caddy grep -c healthz /etc/caddy/Caddyfile   # 0 — the container's copy
grep -c healthz caddy/Caddyfile                                  # 1 — the host's copy
stat -c %i caddy/Caddyfile                                       # a new inode after every git pull

If the host’s inode changes on each pull and the container keeps serving the old content, you are looking at this bug, not at a reload problem. It is easy to chase the wrong thing here: the reload returns success, the logs are clean, and docker compose up -d reports the container as unchanged — which, from Compose’s point of view, it is.

The fix

Mount the directory, not the file:

volumes:
  - ./caddy:/etc/caddy:ro

A directory bind mount is attached to the directory’s inode; files inside it are looked up by name on every access, so a rename inside the directory is visible immediately. After this change the same reload picked up the new Caddyfile at once.

The general rule

If a process or a tool replaces files by rename — git, most editors, atomic-write helpers, anything using mv over a path — a single-file bind mount will go stale. Mount the parent directory and point the application at the path inside it. This holds for Docker, for Kubernetes subPath mounts, and for anything else that resolves the mount target once.

The reload step was still right to keep. Compose recreates a container only when its configuration changes; a file that changed underneath a bind mount is invisible to it. Directory mount plus explicit reload is the combination that makes configuration changes land without restarting the edge.