Finding the slow half of a CI pipeline

2 April 2026

Build times grow the way weight does: nobody notices the day it happens. By the time someone complains, the pipeline is twelve minutes long and no one can say which part is new.

Split it in two first

Before optimising anything, separate total time into two buckets:

• Things you control — compilation, tests, linting, image builds.
• Things you don't — runner queue time, image pulls, dependency downloads, artifact upload.

This matters because the two have completely different fixes, and people routinely spend a week shaving thirty seconds off a test suite while four minutes go to a cold dependency cache.

Cheap instrumentation

You don't need a tracing system. Wrapping each step with a timestamp and dumping it at the end gets you most of the way:

step() {
  local name="$1"; shift
  local start=$(date +%s)
  "$@"
  echo "TIMING ${name} $(( $(date +%s) - start ))s" >> timings.txt
}

Then print timings.txt as the last step. After a week you'll have enough runs to see variance, which is more informative than any single build.

What usually turns up

In roughly the order I've found them:

1. A cache key that includes something volatile, so it never hits.
2. Docker layers invalidated by copying the whole source tree before installing dependencies.
3. Tests that sleep. Always more than you think.
4. A single test file that takes longer than the other two hundred combined.

None of these require clever tooling to find. They require measuring before guessing, which is the part that gets skipped.