Skip to main content

When CI Went Dark: Gating Merges Locally and the Queue Nobody Measured

On October 5 every GitHub Actions job in our relay codebase failed before a runner was assigned. We gated merges on local checks for a day, brought CI back on a self-hosted pool, and then measured where the time went: jobs that ran for seconds were waiting tens of minutes for a runner.

CatalEx Engineering · Published October 8, 2026 · 09:00 UTC

At about 03:49 UTC on Monday, October 5, every GitHub Actions job in our relay codebase started failing. Not failing tests: failing before a runner was assigned, with zero steps executed. Pull requests, pushes to main, all of it. GitHub's status page showed Actions as operational. The most likely cause was an Actions minutes or spending limit on the organisation, but we never confirmed it.

Our merge rule at the time was strict and simple: CI must pass on the exact head commit. With no CI, that rule blocked every merge. The repository was eight days old, coding agents were opening pull requests around the clock, and finished work was piling up behind a check that could not run.

This post covers the three days that followed. We replaced CI with a local gate for about a day, brought it back on a self-hosted runner pool with a new structure, and then did the thing we should have done first: measured where the time in a CI run actually went. The answer was not "running tests." It was waiting for a machine to run them on. Along the way we found runs that nobody was waiting for, still burning runner time, and a small trick that cancels them for free.

34m 17sa docs-only PR waiting for runners
30 sthe same PR actually executing
67 of 100sampled runs ended cancelled
117m 34srunner time burned by 11 cancelled jobs

Day one: a merge gate with no CI​

The first decision was what not to do. We considered two obvious options and took neither.

OptionWhy we did not take it
Raise the spending limit and keep the CI gateNot chosen at the time; the cause was unconfirmed, and we wanted a path that did not depend on it
Pause all merges until CI returnsRejected: it blocks finished work, and with agents producing a steady stream of pull requests the backlog would only grow
Disable CI and gate on local checksChosen, recorded in a decision record the same morning

We disabled the workflows at the repository level, left the workflow files in place with a header comment pointing to the decision, and wrote down exactly what a merge now required.

docs/decisions/2026-10-05-ci-disabled-local-gate.md · Merge conditions

A pull request may merge only when all of these hold:

  1. Its exact head contains the latest main. No merging a branch that was tested against an older base.
  2. The full local suite has passed on that head. Not on an earlier commit, not on a subset.
  3. Its body lists which checks ran. The record of verification travels with the change.

The full local suite, per module:

ModuleChecks
local-relayTest policy, go vet, the full Go test suite against a test database, web tests and the rich-content inventory check when web assets change, contract drift tests
hosted-relayBuild, format check, test policy, unit tests, and the PostgreSQL integration suite against a throwaway database

Each condition closes a specific gap. The first prevents the classic "both branches were green, the merge is red" problem: if your head contains the latest main, the code you tested is the code that lands. The second removes the temptation to run "the relevant tests." The third turns an honour system into an auditable one: a reviewer, human or agent, can see what was claimed.

Nine pull requests merged under the local gate before CI came back. That is the good news. The honest news is what the gate cannot do.

What CI gives youWhat the local gate gave us
An independent machine runs the checksThe author's machine runs them, often the author's agent
A result nobody can editA list in the PR body that the author writes
The same environment every timeWhatever is installed locally
A record tied to a commit SHAA claim tied to a commit SHA
Enforcement by branch protectionEnforcement by convention

A local gate is a promise, not a proof. It works for a day when the people (and agents) making the promise are following rules they wrote down that morning. We would not run it for a month. We also wrote down when to revisit it: when a dedicated runner setup works reliably, or GitHub-hosted runners come back.

Day two: CI returns, restructured​

On October 6 we moved the workflows onto a shared pool of self-hosted runners, labelled [self-hosted, 4g-mem], and turned CI back on. The same day we restructured it. Before, one workflow ran everything. After, it looked like this:

Three pieces, each with a reason:

  • One change-detection job. It decides which modules a change touches. Only affected modules run; unchanged ones skip.
  • Per-module reusable workflows. Each module owns its workflow file, with its own shell, working directory and environment defaults. The repo owner chose module maintainability over a single sequential job.
  • An aggregate gate. One final job checks that change detection succeeded and that every selected module passed on the exact PR head. Branch protection requires only this one check. The gate runs on every pull request, including documentation-only ones, so there is always a result.

Why the aggregate gate? Because required checks in GitHub are matched by name, and a workflow that skips some modules produces a different set of names depending on what changed. One stable name that always reports is easier to protect than a list that changes shape.

This design was correct for correctness. It was about to be wrong for time.

Day three: the queue nobody measured​

By October 7, pull requests were taking tens of minutes to get a result, even tiny ones. The intuitive diagnosis was "the test suites are slow." We did not trust intuition, so we pulled the latest 100 workflow runs and their job records from GitHub's Actions API and split each job's time into two parts:

  • Queue time: from when the job was created to when a runner started it.
  • Execution time: from when it started to when it completed.

The sample was a busy afternoon: 82 PR runs, 14 main push runs and four closed-PR workflows, across 39 branches. Skipped jobs, and cancelled jobs that never got a runner, were left out of the timings.

JobSamplesMedian waitMax waitMedian execution
Change detection4420m 02s85m 44s17 s
Final gate1428m 22s89m 26s11.5 s
Local Relay2622m 41s93m 11s20m 43s
Hosted Relay930m 43s66m 36s5m 29s
Tools227m 33s39m 48s2m 19s

Read the first two rows again. A job that runs for 17 seconds waited a median of 20 minutes to start. The gate, which runs for 11 seconds, waited 28. The audit's own summary put it plainly: "Observed queue delays substantially exceeded execution time for short change detection and gate jobs."

Three completed runs show what that means for a person waiting on a pull request:

RunWaiting for runnersExecutingTotal
Documentation only34m 17s30 s34m 49s
No module selected64m 03s30 s64m 38s
A Hosted Relay change67m 46s6m 14s74m 03s

The documentation-only run is the clearest. Every module skipped. Change detection waited 6m 49s for a runner, finished in seconds, and then the gate waited another 27m 28s for a runner of its own. Thirty seconds of work, half an hour of waiting.

Why the queue was that long​

Four causes, and they multiplied each other.

  1. One pool for everything. Every job asked for the same runner label. A 17-second control job had no reserved capacity, so it queued behind 20-minute test jobs. At the moment we snapshotted, all four runner name prefixes we could see were busy with module tests while other jobs queued.
  2. Serial stage barriers. Detect, then modules, then gate. Each stage needs a fresh runner assignment. A docs-only PR needs two; a module change needs at least three on its critical path. The waits add up even when the work is trivial. The structure we added the day before for correctness had tripled the number of times a PR stood in line.
  3. Long occupancy. The Local Relay test step in one inspected run lasted 22m 24s. Its logs showed 400.9 s in ordinary store tests, 200.3 s in a registry backup test and 150.2 s in a stream backup test, run in sequence with race detection on. That was real test demand, holding a runner for twenty minutes at a time.
  4. High arrival rate, and work nobody wanted. Every PR push started a run, with no path filter at the event level. Separate PRs are separate concurrency groups, so they compete instead of replacing each other. And 67 of the 100 sampled runs ended cancelled. Eleven jobs that had already been assigned a runner before cancellation used 117m 34s of runner time between them.

Measure the queue separately from the run. Most CI dashboards show total duration, which hides the split. The Actions API gives you created_at, started_at and completed_at for every job. Queue time is the first gap, execution the second. If you have never computed both, you do not know what your CI is slow at.

Orphaned runs, and a workflow that does nothing​

The fourth cause hid a specific bug. Our PR runs and our main push runs used different concurrency groups, which is normal: you do not want a push to main to cancel a PR's run. But it meant that when a PR merged or closed, its in-flight CI kept going. Nothing was waiting for that result. The run still held a runner, or still sat in the queue ahead of someone who was waiting.

The fix we shipped on October 7 is a workflow that does nothing.

.github/workflows/cancel-closed-pr-ci.yml · What it declares
SettingValueWhy
TriggerPull request closedFires on merge or close
Concurrency groupci-<PR number>, the same group the PR's CI usesEntering the group is what does the work
cancel-in-progresstrueCancels whatever is running or pending in that group
PermissionsNoneIt makes no API calls
The jobOne step that prints a lineIt exists only so the workflow is valid

Here is the mechanism. GitHub Actions allows at most one running workflow in a concurrency group, and with cancel-in-progress set, a new run in the group cancels the one already executing. So when the closed-PR workflow is created, it joins the PR's group and GitHub cancels the PR's CI run. That happens when the run is admitted to the group, before the no-op job even gets a runner. The cancellation does not wait in the queue that caused the problem.

It replaced an earlier version that polled the API to find and cancel runs. The new one needs no token, no write permissions, no checkout and no script.

It has real limits, and we wrote them into the decision record:

  • Event ordering is not guaranteed. GitHub's docs say so directly: "Since the actual start time of a job or run may vary, ordering is not guaranteed." A CI run created after the close is processed can arrive later and escape the cancellation.
  • It does not retry or confirm. If cancellation fails, nothing notices.
  • The no-op job still queues. One bookkeeping run in the audit queued for 26m 12s and ran for 12 s. That is not the cancellation latency (the cancellation already happened) but it is still a job in the shared pool.
  • It only works from the default branch. A workflow triggered by PR close has to exist on main before it takes effect.

What we tried, rejected and kept​

The audit pointed at fixes. Not all of them survived contact with the repo owner, and the reasons are worth recording.

IdeaOutcomeReason
One job, with each module's steps as composite actionsRejectedComposite actions do not support action-level shell, working-directory and environment defaults, and merging check steps to configure them once hurt readability
Remove the aggregate gate, require each module's check directlyExplored, then rejectedA skipped reusable workflow reports a different check name from one that ran, so branch protection would have to know the workflow structure; that coupling was the problem the gate solved
Reserve separate capacity for change detection and the gateProposed, deferredThe repo owner explicitly deferred runner-pool changes
Cancel closed PRs' runs with a no-op workflowShippedNo permissions, no API calls, cancels before queueing
Shrink the slowest test fixturesShippedLess demand on the pool without changing the structure

The last row is the kind of fix that holds up. One registry backup test created 4,002 sessions to prove a backup header could exceed a 1 MiB bound. The new version uses 1,200 sessions with longer (still legal) IDs and asserts directly that the header crosses the same bound. Setup and restore rows dropped by 70%, and the test still proves the same thing. We measured it only on a development machine, so we do not claim a CI number for it yet.

The trade we made, knowingly: keeping per-module workflows and the aggregate gate keeps the extra runner assignments and their queueing risk. We chose maintainable workflows and a stable required check over the fastest possible schedule, and recorded that the scheduling problem remains open.

Where the time goes in a CI run​

If you take one thing from this post, take this taxonomy. Each row is a different problem with a different fix, and total duration hides all of them.

Where time goesHow to see itTypical fix
Queue waitstarted_at minus created_at, per jobReserve capacity for short control jobs; fewer stage boundaries
Stage barriersCount runner assignments on the critical pathFewer stages, or let control logic run without a runner
Executioncompleted_at minus started_atProfile the slowest tests; fix fixtures and waits, not coverage
Superseded workCancelled runs that had a runnerConcurrency groups that replace stale runs
Orphaned workRuns still going after their PR closedCancel on close
InfrastructureSetup steps, cache restores, failed extractionsFix separately; rarely the big number

How to run this audit on your own CI​

It took us an afternoon. You can do it with the GitHub CLI and a spreadsheet.

  1. Pull a sample. List your latest 100 workflow runs, then each run's jobs. A busy period tells you more than a quiet one.
  2. Split each job's time. Queue is start minus creation. Execution is completion minus start. Exclude skipped jobs and jobs cancelled before they got a runner.
  3. Group by job name. Report median and maximum for each, side by side.
  4. Find three exemplar runs. A trivial change, a no-op, a normal change. Their waiting-versus-executing split is the story you will tell your team.
  5. Count cancelled runs and the runner minutes they used before cancellation.
  6. Write it down with its limits. Ours noted that the fetches were sequential snapshots, that queue time includes provisioning and dispatch, and that it was a busy-period sample, not a service level.

What this does not fix​

  • The queue itself. On October 7 we understood it; we had not solved it. Control jobs still shared one pool with long test jobs, and each PR still needed several runner assignments.
  • Capacity. We could not see the organisation's runner inventory with the credentials we had, so we could not prove a hard capacity limit. The contention diagnosis is strongly supported, not proven.
  • Demand. Agents running in parallel push often. More pushes means more runs. Cancelling stale ones helps; it does not reduce the number of changes.
  • The local gate's trust gap. Listing checks in a PR body is a claim. When CI came back, it came back as the source of truth.
  • Late-arriving runs. The cancel-on-close trick can miss a run that arrives after the close event.

What it cost​

A day of merges on the honour system. A self-hosted pool that we now operate: machines, their labels, their caches and their failures. A CI structure with more moving parts than one workflow. And an audit's worth of engineering time that produced a clear diagnosis and only partial fixes, which is the normal shape of real infrastructure work.

The breakdown​

  1. When CI cannot run, do not pause merges and do not merge blind. Write a local gate with three conditions: head contains latest main, full suite on that head, checks listed in the PR body.
  2. Treat the local gate as temporary, and write down when you will revisit it.
  3. Change detection, per-module workflows and one aggregate gate give you selective runs and a stable required check.
  4. Every stage boundary is another wait for a runner. Count them.
  5. Measure queue time and execution time separately, per job. Short jobs waiting long is the signature of a shared, saturated pool.
  6. Give PR runs and main runs different concurrency groups, then cancel a PR's group when it closes, with a no-op workflow that needs no permissions.
  7. Shrink the slowest fixtures before you add machines.

A CI run is not slow because its tests are slow until you have measured the part where nothing is running at all.


Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.