The finance team noticed before I did: one night’s summary was about 60% of what it should have been, and the file’s mtime was two minutes later than every other night.

The job was a small Python script that pulled the day’s orders, aggregated them, and wrote summary.csv. It ran from cron every night at 02:00 and normally took about forty seconds. That night it took eleven minutes, because a DNS lookup in the middle of the run had no timeout and the resolver was having a bad night. At 02:15, while the first copy was still waiting, cron started the next scheduled run.

Nothing in the script prevented that. Cron does not care what is already running; it starts what the calendar says to start. So two processes read the same database, both wrote summary.csv, and the one that finished second won. Its data was partial.

The first fix is one line in the crontab:

*/15 * * * * flock -n /var/lock/billing.lock /usr/local/bin/billing-summary >> /var/log/billing.log 2>&1

flock -n takes an exclusive lock on the file and exits immediately with status 1 if somebody else holds it. The lock is held by an open file descriptor, so it is released when the process exits — including when it is killed, which is the part that matters. No stale lock files to clean up, no PID files to get wrong.

The second fix is in the script. A job that writes a file should write a new file and rename it:

import os, tempfile

fd, tmp = tempfile.mkstemp(dir="/var/lib/billing", prefix="summary.", suffix=".tmp")
with os.fdopen(fd, "w") as f:
    write_rows(f)
os.replace(tmp, "/var/lib/billing/summary.csv")

os.replace is atomic on the same filesystem. A reader either sees the old file or the new one, never a half-written one. This does not prevent two concurrent runs, but it makes each run’s output internally consistent, which is what turned a duplicate run into a wrong number.

The third fix is a timeout, because a job that can hang for eleven minutes will eventually hang for eleven hours:

timeout --signal=TERM --kill-after=30 600 /usr/local/bin/billing-summary

Three things I check now when a scheduled job can overlap:

  • Does the scheduler know about the previous run? Cron does not. systemd timers can: Type=oneshot plus OnCalendar= will not start a unit that is already active. If the job is important enough to page someone, a timer is usually the better home for it.
  • Is the output written atomically? A lock protects against overlap, not against a crash halfway through a write. Both are needed.
  • Is there more than one host running the same crontab? During a migration we had two machines with the same image for three days. flock on a local file would not have helped there; the lock has to live somewhere shared, or the job has to be idempotent for the day. I add hostname to the first log line of every scheduled job now.

The log line that would have saved me an hour:

2026-02-03 02:00:01 start pid=4811 host=db-01
2026-02-03 02:15:00 start pid=4902 host=db-01
2026-02-03 02:16:44 done  pid=4811 rows=18422
2026-02-03 02:16:51 done  pid=4902 rows=11093

Two starts, two different PIDs, same host. It is obvious once it is written down. It was invisible for a week because the script only logged on success.