Skip to content

Software Engineering

1 post with the tag “Software Engineering”

GitHub Actions Sleep-Loop Bug: Years of Failed Runs and the Cost to Developers

A four-line Bash function in the GitHub Actions runner has caused failed runs and idle machines for years. The function was meant to pause execution briefly. On busy runners it can spin forever, hold a CPU core at 100%, and leave a job running long after it should have ended. One developer reported 5,135 hours of billed idle time on a single runner.

The runner codebase started around 2016. Early commits show Windows developers borrowing a Stack Overflow trick from 16 years earlier: use ping to simulate a delay where sleep is not available. That pattern became a top-level function called safe_sleep:

Terminal window
if [ $? -eq 4 ]; then
sleep 5 || ping -n 6 127.0.0.1 > nul || (for i in `seq 1 5000`; do echo >&5; done)
fi

The chain tries sleep first, then ping for about 4 seconds, then 5,000 echo writes to /dev/null. It worked, at the cost of CPU time.

In 2022 the code changed to a tighter loop:

Terminal window
start=$SECONDS
while [ $((SECONDS - start)) -ne ${1?} ]; do :; done

The intent: read Bash’s SECONDS variable, which increments every second, and loop until the target is reached. The flaw shows up under load. If scheduling makes SECONDS jump from 4 to 6, the comparison -ne 5 is never false, so the loop never exits. With no sleep inside the loop, it pegs one CPU core at 100%, half of a standard 2-vCPU runner, and starves other tasks on the machine.

One developer measured a single runner idling for 5,135 hours. At GitHub’s $0.08 per vCPU-minute rate, that is about $2,400 in billed time for one machine. Projects felt the effect too. Zigg moved to Codeberg, citing “inexcusable bugs” and what it called “vibe scheduling” after Microsoft’s AI pivot, with job prioritization that stalled even main-branch commits.

A fix surfaced in 2024: use -le instead of -ne, so the loop stops once the target is reached or passed:

Terminal window
while [ $((SECONDS - start)) -le ${1?} ]; do :; done

A pull request with this change was proposed in February 2022. It was auto-closed after a month and merged about 1.5 years later, after public complaints. Other regressions followed the same pattern. A later refactor replaced a plain Object.getOwnPropertyNames call with nested loops and redundant if statements, and file hashing broke as a result:

function getKeys(obj) {
return Object.getOwnPropertyNames(obj);
}

The fix was small and had been available for years.

The runner is a shared codebase with a long review backlog, and the sleep function sits in a busy path where changes carry risk. Matt Lad of Antithesis summed up the engineering assessment: this is not peak engineering. The platform bills per minute, so a looping sleep is not harmless. Teams that depend on Actions should audit runner logs for jobs that run far past their expected time, and pin runner versions that contain the fix.