When CI Went Dark: Gating Merges Locally and the Queue Nobody Measured
On October 5 every GitHub Actions job in our relay codebase failed before a runner was assigned. We gated merges on local checks for a day, brought CI back on a self-hosted pool, and then measured where the time went: jobs that ran for seconds were waiting tens of minutes for a runner.
At about 03:49 UTC on Monday, October 5, every GitHub Actions job in our relay
codebase started failing. Not failing tests: failing before a runner was
assigned, with zero steps executed. Pull requests, pushes to main, all of
it. GitHub's status page showed Actions as operational. The most likely cause
was an Actions minutes or spending limit on the organisation, but we never
confirmed it.
Our merge rule at the time was strict and simple: CI must pass on the exact head commit. With no CI, that rule blocked every merge. The repository was eight days old, coding agents were opening pull requests around the clock, and finished work was piling up behind a check that could not run.
This post covers the three days that followed. We replaced CI with a local gate for about a day, brought it back on a self-hosted runner pool with a new structure, and then did the thing we should have done first: measured where the time in a CI run actually went. The answer was not "running tests." It was waiting for a machine to run them on. Along the way we found runs that nobody was waiting for, still burning runner time, and a small trick that cancels them for free.
Day one: a merge gate with no CI
The first decision was what not to do. We considered two obvious options and took neither.
| Option | Why we did not take it |
|---|---|
| Raise the spending limit and keep the CI gate | Not chosen at the time; the cause was unconfirmed, and we wanted a path that did not depend on it |
| Pause all merges until CI returns | Rejected: it blocks finished work, and with agents producing a steady stream of pull requests the backlog would only grow |
| Disable CI and gate on local checks | Chosen, recorded in a decision record the same morning |
We disabled the workflows at the repository level, left the workflow files in place with a header comment pointing to the decision, and wrote down exactly what a merge now required.
A pull request may merge only when all of these hold:
- Its exact head contains the latest
main. No merging a branch that was tested against an older base. - The full local suite has passed on that head. Not on an earlier commit, not on a subset.
- Its body lists which checks ran. The record of verification travels with the change.
The full local suite, per module:
| Module | Checks |
|---|---|
| local-relay | Test policy, go vet, the full Go test suite against a test database, web tests and the rich-content inventory check when web assets change, contract drift tests |
| hosted-relay | Build, format check, test policy, unit tests, and the PostgreSQL integration suite against a throwaway database |
Each condition closes a specific gap. The first prevents the classic "both
branches were green, the merge is red" problem: if your head contains the
latest main, the code you tested is the code that lands. The second removes
the temptation to run "the relevant tests." The third turns an honour system
into an auditable one: a reviewer, human or agent, can see what was claimed.
Nine pull requests merged under the local gate before CI came back. That is the good news. The honest news is what the gate cannot do.
| What CI gives you | What the local gate gave us |
|---|---|
| An independent machine runs the checks | The author's machine runs them, often the author's agent |
| A result nobody can edit | A list in the PR body that the author writes |
| The same environment every time | Whatever is installed locally |
| A record tied to a commit SHA | A claim tied to a commit SHA |
| Enforcement by branch protection | Enforcement by convention |
A local gate is a promise, not a proof. It works for a day when the people (and agents) making the promise are following rules they wrote down that morning. We would not run it for a month. We also wrote down when to revisit it: when a dedicated runner setup works reliably, or GitHub-hosted runners come back.
Day two: CI returns, restructured
On October 6 we moved the workflows onto a shared pool of self-hosted
runners, labelled [self-hosted, 4g-mem], and turned CI back on. The same
day we restructured it. Before, one workflow ran everything. After, it looked
like this:
Three pieces, each with a reason:
- One change-detection job. It decides which modules a change touches. Only affected modules run; unchanged ones skip.
- Per-module reusable workflows. Each module owns its workflow file, with its own shell, working directory and environment defaults. The repo owner chose module maintainability over a single sequential job.
- An aggregate gate. One final job checks that change detection succeeded and that every selected module passed on the exact PR head. Branch protection requires only this one check. The gate runs on every pull request, including documentation-only ones, so there is always a result.
Why the aggregate gate? Because required checks in GitHub are matched by name, and a workflow that skips some modules produces a different set of names depending on what changed. One stable name that always reports is easier to protect than a list that changes shape.
This design was correct for correctness. It was about to be wrong for time.
Day three: the queue nobody measured
By October 7, pull requests were taking tens of minutes to get a result, even tiny ones. The intuitive diagnosis was "the test suites are slow." We did not trust intuition, so we pulled the latest 100 workflow runs and their job records from GitHub's Actions API and split each job's time into two parts:
- Queue time: from when the job was created to when a runner started it.
- Execution time: from when it started to when it completed.
The sample was a busy afternoon: 82 PR runs, 14 main push runs and four
closed-PR workflows, across 39 branches. Skipped jobs, and cancelled jobs
that never got a runner, were left out of the timings.
| Job | Samples | Median wait | Max wait | Median execution |
|---|---|---|---|---|
| Change detection | 44 | 20m 02s | 85m 44s | 17 s |
| Final gate | 14 | 28m 22s | 89m 26s | 11.5 s |
| Local Relay | 26 | 22m 41s | 93m 11s | 20m 43s |
| Hosted Relay | 9 | 30m 43s | 66m 36s | 5m 29s |
| Tools | 2 | 27m 33s | 39m 48s | 2m 19s |
Read the first two rows again. A job that runs for 17 seconds waited a median of 20 minutes to start. The gate, which runs for 11 seconds, waited 28. The audit's own summary put it plainly: "Observed queue delays substantially exceeded execution time for short change detection and gate jobs."
Three completed runs show what that means for a person waiting on a pull request:
| Run | Waiting for runners | Executing | Total |
|---|---|---|---|
| Documentation only | 34m 17s | 30 s | 34m 49s |
| No module selected | 64m 03s | 30 s | 64m 38s |
| A Hosted Relay change | 67m 46s | 6m 14s | 74m 03s |
The documentation-only run is the clearest. Every module skipped. Change detection waited 6m 49s for a runner, finished in seconds, and then the gate waited another 27m 28s for a runner of its own. Thirty seconds of work, half an hour of waiting.
Why the queue was that long
Four causes, and they multiplied each other.
- One pool for everything. Every job asked for the same runner label. A 17-second control job had no reserved capacity, so it queued behind 20-minute test jobs. At the moment we snapshotted, all four runner name prefixes we could see were busy with module tests while other jobs queued.
- Serial stage barriers. Detect, then modules, then gate. Each stage needs a fresh runner assignment. A docs-only PR needs two; a module change needs at least three on its critical path. The waits add up even when the work is trivial. The structure we added the day before for correctness had tripled the number of times a PR stood in line.
- Long occupancy. The Local Relay test step in one inspected run lasted 22m 24s. Its logs showed 400.9 s in ordinary store tests, 200.3 s in a registry backup test and 150.2 s in a stream backup test, run in sequence with race detection on. That was real test demand, holding a runner for twenty minutes at a time.
- High arrival rate, and work nobody wanted. Every PR push started a run, with no path filter at the event level. Separate PRs are separate concurrency groups, so they compete instead of replacing each other. And 67 of the 100 sampled runs ended cancelled. Eleven jobs that had already been assigned a runner before cancellation used 117m 34s of runner time between them.
Measure the queue separately from the run. Most CI dashboards show total
duration, which hides the split. The Actions API gives you created_at,
started_at and completed_at for every job. Queue time is the first gap,
execution the second. If you have never computed both, you do not know what
your CI is slow at.
Orphaned runs, and a workflow that does nothing
The fourth cause hid a specific bug. Our PR runs and our main push runs used
different concurrency groups, which is normal: you do not want a push to
main to cancel a PR's run. But it meant that when a PR merged or closed,
its in-flight CI kept going. Nothing was waiting for that result. The run
still held a runner, or still sat in the queue ahead of someone who was
waiting.
The fix we shipped on October 7 is a workflow that does nothing.
| Setting | Value | Why |
|---|---|---|
| Trigger | Pull request closed | Fires on merge or close |
| Concurrency group | ci-<PR number>, the same group the PR's CI uses | Entering the group is what does the work |
cancel-in-progress | true | Cancels whatever is running or pending in that group |
| Permissions | None | It makes no API calls |
| The job | One step that prints a line | It exists only so the workflow is valid |
Here is the mechanism. GitHub Actions allows at most one running workflow in
a concurrency group, and with cancel-in-progress set, a new run in the
group cancels the one already executing.
So when the closed-PR workflow is created, it joins the PR's group and GitHub
cancels the PR's CI run. That happens when the run is admitted to the group,
before the no-op job even gets a runner. The cancellation does not wait in
the queue that caused the problem.
It replaced an earlier version that polled the API to find and cancel runs. The new one needs no token, no write permissions, no checkout and no script.
It has real limits, and we wrote them into the decision record:
- Event ordering is not guaranteed. GitHub's docs say so directly: "Since the actual start time of a job or run may vary, ordering is not guaranteed." A CI run created after the close is processed can arrive later and escape the cancellation.
- It does not retry or confirm. If cancellation fails, nothing notices.
- The no-op job still queues. One bookkeeping run in the audit queued for 26m 12s and ran for 12 s. That is not the cancellation latency (the cancellation already happened) but it is still a job in the shared pool.
- It only works from the default branch. A workflow triggered by PR
close has to exist on
mainbefore it takes effect.
What we tried, rejected and kept
The audit pointed at fixes. Not all of them survived contact with the repo owner, and the reasons are worth recording.
| Idea | Outcome | Reason |
|---|---|---|
| One job, with each module's steps as composite actions | Rejected | Composite actions do not support action-level shell, working-directory and environment defaults, and merging check steps to configure them once hurt readability |
| Remove the aggregate gate, require each module's check directly | Explored, then rejected | A skipped reusable workflow reports a different check name from one that ran, so branch protection would have to know the workflow structure; that coupling was the problem the gate solved |
| Reserve separate capacity for change detection and the gate | Proposed, deferred | The repo owner explicitly deferred runner-pool changes |
| Cancel closed PRs' runs with a no-op workflow | Shipped | No permissions, no API calls, cancels before queueing |
| Shrink the slowest test fixtures | Shipped | Less demand on the pool without changing the structure |
The last row is the kind of fix that holds up. One registry backup test created 4,002 sessions to prove a backup header could exceed a 1 MiB bound. The new version uses 1,200 sessions with longer (still legal) IDs and asserts directly that the header crosses the same bound. Setup and restore rows dropped by 70%, and the test still proves the same thing. We measured it only on a development machine, so we do not claim a CI number for it yet.
The trade we made, knowingly: keeping per-module workflows and the aggregate gate keeps the extra runner assignments and their queueing risk. We chose maintainable workflows and a stable required check over the fastest possible schedule, and recorded that the scheduling problem remains open.
Where the time goes in a CI run
If you take one thing from this post, take this taxonomy. Each row is a different problem with a different fix, and total duration hides all of them.
| Where time goes | How to see it | Typical fix |
|---|---|---|
| Queue wait | started_at minus created_at, per job | Reserve capacity for short control jobs; fewer stage boundaries |
| Stage barriers | Count runner assignments on the critical path | Fewer stages, or let control logic run without a runner |
| Execution | completed_at minus started_at | Profile the slowest tests; fix fixtures and waits, not coverage |
| Superseded work | Cancelled runs that had a runner | Concurrency groups that replace stale runs |
| Orphaned work | Runs still going after their PR closed | Cancel on close |
| Infrastructure | Setup steps, cache restores, failed extractions | Fix separately; rarely the big number |
How to run this audit on your own CI
It took us an afternoon. You can do it with the GitHub CLI and a spreadsheet.
- Pull a sample. List your latest 100 workflow runs, then each run's jobs. A busy period tells you more than a quiet one.
- Split each job's time. Queue is start minus creation. Execution is completion minus start. Exclude skipped jobs and jobs cancelled before they got a runner.
- Group by job name. Report median and maximum for each, side by side.
- Find three exemplar runs. A trivial change, a no-op, a normal change. Their waiting-versus-executing split is the story you will tell your team.
- Count cancelled runs and the runner minutes they used before cancellation.
- Write it down with its limits. Ours noted that the fetches were sequential snapshots, that queue time includes provisioning and dispatch, and that it was a busy-period sample, not a service level.
What this does not fix
- The queue itself. On October 7 we understood it; we had not solved it. Control jobs still shared one pool with long test jobs, and each PR still needed several runner assignments.
- Capacity. We could not see the organisation's runner inventory with the credentials we had, so we could not prove a hard capacity limit. The contention diagnosis is strongly supported, not proven.
- Demand. Agents running in parallel push often. More pushes means more runs. Cancelling stale ones helps; it does not reduce the number of changes.
- The local gate's trust gap. Listing checks in a PR body is a claim. When CI came back, it came back as the source of truth.
- Late-arriving runs. The cancel-on-close trick can miss a run that arrives after the close event.
What it cost
A day of merges on the honour system. A self-hosted pool that we now operate: machines, their labels, their caches and their failures. A CI structure with more moving parts than one workflow. And an audit's worth of engineering time that produced a clear diagnosis and only partial fixes, which is the normal shape of real infrastructure work.
The breakdown
- When CI cannot run, do not pause merges and do not merge blind. Write a
local gate with three conditions: head contains latest
main, full suite on that head, checks listed in the PR body. - Treat the local gate as temporary, and write down when you will revisit it.
- Change detection, per-module workflows and one aggregate gate give you selective runs and a stable required check.
- Every stage boundary is another wait for a runner. Count them.
- Measure queue time and execution time separately, per job. Short jobs waiting long is the signature of a shared, saturated pool.
- Give PR runs and
mainruns different concurrency groups, then cancel a PR's group when it closes, with a no-op workflow that needs no permissions. - Shrink the slowest fixtures before you add machines.
A CI run is not slow because its tests are slow until you have measured the part where nothing is running at all.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.