Developer tools
Find out why CI is failing
Read a red build down to the actual assertion or gate that failed, and say whether it is this change, the branch it sits on, or the pipeline.
By Toolspoke
Skill procedure
Most time lost to a red build is spent guessing from the summary. The summary names a job. The cause is in the last twenty lines of that job's log, and reading them first is almost always faster than any theory.
Steps
- List the check runs on the head commit and note which failed and which were skipped. A skipped job that should have run is its own answer.
- Fetch the failing job's log and read the end of it. Not the middle, not the first error — the last twenty lines, which is where the runner prints what it decided. A suite can print hundreds of red lines and still have failed on something else entirely.
- Classify what you find, because the three have different fixes:
- An assertion: a test disagrees with the code. This is the change's, and the fix is in the diff.
- A gate: coverage, lint, typecheck, a size budget. The tests all passed and the build is still red. Read the threshold and the measured number, and fix the specific shortfall rather than the whole file.
- The pipeline: checkout, install, a missing service, the runner disappearing. Nothing to do with the change.
- Check whether it is yours at all. Look at the same job on the base branch's latest commit. Red there too means the branch inherited it, and the fix belongs to whoever broke the base — say so rather than absorbing it.
- Reproduce it locally before pushing a fix, with the same command the job runs. A local suite that skips what CI runs will agree with you and be wrong; check the run's summary line for skipped files before trusting a green local run.
- Push one fix and watch it. Speculative pushes cost a full cycle each and make the history unreadable.
Never
- Skip, disable or quarantine a failing test to get to green.
- Push an empty commit, or close and reopen, to "kick" CI.
- Call a failure flaky without evidence: the same job passing on the identical commit, or a failure before any test body ran.