field noteLinux · Filesystems · Updates

Flush before rename: updates that survive a power cut

An atomic rename is only atomic for the name. Without a flush, a power cut can leave a renamed file with the wrong contents.

Write to a temporary file, then rename it over the real one: the textbook way to update a file atomically. Readers see the old file or the new one, never half of each. It has a gap, though, and an update tool I reviewed fell into it.

The gap

The rename is recorded in the filesystem's metadata, but the new file's data can still be sitting in memory, waiting to be written out. Some filesystems are configured to delay that for minutes. If the power goes in that window, the name can survive while the data does not, and after a reboot the file is empty or holds the wrong contents. Until then the system keeps running happily from memory, so it looks healthy.

The fix

Flush the data to disk before the rename, then flush the directory so the rename itself is on disk too.

tmp=$(mktemp "$dest.XXXX")
cp new.conf "$tmp"
sync "$tmp"                # data on disk first
mv -f "$tmp" "$dest"        # atomic swap of the name
sync "$(dirname "$dest")"   # and the directory entry
The same order applies in any language: fsync the file, rename, fsync the directory.

What I took from it

  • "Atomic" and "durable" are different promises. Updates need both.
  • Test updates by cutting the power, not by reasoning about them: the bug only shows after a reboot.