Skip to main content

CI at Agent Speed: Selecting Tests by What They Read, a Ten-Minute Ceiling, and Batching Main

When coding agents merge about 57 pull requests a day, CI runs out of runners before it runs out of tests. How we select tests by imports and recorded file reads, why hand-written skip rules and coverage-based narrowing failed, how the merge gate enforces a ten-minute ceiling, and how main now runs in batches with an agent bisecting failures.

CatalEx Engineering · Published November 18, 2026 · 09:00 UTC

On November 13 a pull request in our relay codebase passed CI, merged, and broke the production deploy. Nothing in CI was red. The deploy tests had not run at all: a hand-written rule had decided the change could not affect them. The rule listed the files those tests depended on, and it was missing one kind. The deploy tests read the Go route registrations to check them against an allowlist. The pull request changed a route registration.

That afternoon we audited every hand-written skip rule in CI. Every one of them missed files its tests read.

This is how our CI works now, after six weeks of coding agents opening almost all of our pull requests. Three problems arrived together. Which tests may a pull request skip, and on what evidence? How fast must CI be, and who enforces it? And when main goes red at nine merges an hour, how do you find the commit that did it?

The load​

~57merged PRs a day, 30 days to Nov 9
563 of 586PRs with coding-agent attribution, Oct 30 to Nov 9
~3xgrowth in CI runner demand, early Oct to Nov
~22self-hosted runners, 4 GB each

Our CI uses about 375,000 runner-minutes a month, all on a shared pool of self-hosted machines with 4 GB of memory and often one vCPU. The growth came from parallel agent sessions: each one pushes, and a pull request is pushed a median of three times before it merges. In one week, 61 percent of pull requests had a run cancelled by a newer push.

Hosted plans do not fix this: our demand is about 125 times what the entry plans of GitHub Actions or CircleCI include. So the lever is runner-minutes per change, and the hard constraint is that saving them must never let a defect through.

Why the obvious fixes failed​

We tried, in order, most of what a team reaches for.

ApproachWhat happenedVerdict
Run everything on every pushLocal Relay's full suite on every pull request added 28 percent job-minutes per PR and made no run shorter, because waiting for runners set the run timesRejected while runners queue
Hand-written skip rulesThe audit found every rule missed reads. One missed route registrations and a production deploy brokeBanned
Narrow by coverage: run only tests that executed the changed linesUnsound, and a replay showed it saved about 2 percentDropped
A hermetic build system such as BazelIts sandbox restricts writes, not reads, so it gives no stronger read guarantee than a complete trace, at the cost of remodelling every moduleRejected

Two of these look correct, so here is the detail.

Coverage-based narrowing​

The idea is attractive. main records which test packages executed which lines. When a pull request changes a line, run only the tests that executed it. We ran this from November 5 to 13. A review found four ways it was wrong:

  • A change outside a function body (a constant, type, struct tag or init function) changes tests that never execute a line of that file.
  • A case condition is evaluated without its clause's coverage block running, so the tests that evaluate it were not selected.
  • Merges since the map was recorded add call paths it does not know.
  • Some tests execute code only in some runs. Across 88 maps from two days, 17 executions vanished between runs of unchanged files, for no reason a selector could detect.

We then built the sound version, which narrows only changes inside function bodies and distrusts tests changed since the map, and replayed it over 41 merged pull requests:

SelectionPackagesTest-secondsLongest shard, mean / max
Line-level narrowing (what we had)1,50117,03370 s / 193 s
Sound function-level narrowing1,72318,39575 s / 193 s
Imports and recorded reads only1,75618,72376 s / 193 s

The sound version saves about 2 percent of test-seconds, with the same longest shard. Wall time is set by the longest shard, so the saving would not shorten a single run. We removed it.

Bazel​

A hermetic build graph promises that every test declares its inputs, so selection becomes a graph query. Bazel's own documentation says its sandbox "doesn't hide the host environment in any way" and that processes "can freely access all files on the file system"; the Linux sandbox makes the filesystem outside the sandbox read-only, which stops writes, not reads (Bazel sandboxing). A test that opens an absolute path reads an undeclared file unnoticed. Its read guarantee rests on the same assumption as a trace: every read is visible. Migrating would also mean remodelling 48 TestMain files and 52 test files that start subprocesses. We kept the trace.

How selection works now​

The rule, accepted on November 17 as RFC043, is narrow on purpose:

A pull request head is merged only after every test that depends on its change has passed on that head. A test depends on the change through the import graph or through a read in the read record.

There are two sources of evidence, and an audit that keeps the second honest.

  1. Imports. A Go package is selected if one of its files changed or its test binary depends on a package whose files changed, in any module of the repository. Vitest files are selected through their import graph.
  2. Recorded reads. Every Go test runs with Go's test action log turned on, which records the files a test opens, stats and lists. After each passing main run, a script reduces the logs to the files each package read outside its own directory and merges them into a cached map. This has run since October 17.
  3. The nightly audit. The action log cannot see reads in init, in TestMain before the tests start, or by a subprocess. So every night a workflow runs each module's whole suite under strace -f, following every process the test binaries start, and adds what it finds to the read record. As of this week, nobody declares a Go test's reads by hand.
  4. Freshness. If the last successful audit is more than 48 hours old, the pull request runs every package of each module it starts. The record tolerates one missed night, not two.
  5. The fallbacks. A change to CI machinery, test settings or anything the selector cannot parse selects everything. So does a selector crash. An error in the read record may only add tests, never remove them.

Suites outside Go follow a stricter rule: they either run on every push, or their runner enforces the declared inputs while the tests run. Our Vitest apps fail a test file at runtime if it reads a file outside the app that is not declared. Suites that take seconds, such as the tooling Python tests and the deploy gates, simply run on every pull request. That costs about 2 to 3 extra runner-minutes per pull request and no wall time, because they run beside longer jobs.

Selection never uses failure history, predicted outcomes, test duration or a time budget. Those work elsewhere; we excluded them on purpose.

What could still drop out silently​

Selection is one way to miss a test. Not wiring it into CI at all is another, and with agents it is the more common one. An agent writing a pull request believes CI runs the suite it just changed. Our own research found a pull request whose description said browser specs were "updated by reading the source, not run locally; CI runs them". No workflow ran them.

Since October 17 an inventory job runs on every event and fails if anything could silently drop out: a module not wired into CI, a package with no selectable tests, a build tag never tested, or a test file no runner owns.

The escape budget​

The invariant permits exactly one gap: a read that no record has seen yet. A failure on main that such a gap hid is a selection escape. Each one is recorded as a defect, names the pull request, the skipped test and the read that hid it, and its read enters the record before the next audit completes.

SettingValue
TargetAt most 0.1 percent of merged pull requests over 30 days
SuspensionAt 2 or more escapes over that rate, selection stops and every PR runs full module suites
RestorationAutomatic, when the 30-day rate is back within target
VisibilityEvery PR run's summary says when selection is suspended
Manual overrideA full-ci label forces a full run; every run uploads what it actually ran

For example, two escapes in 1,600 merges is 0.125 percent, and selection suspends itself. A single escape never does: its gap closes and the rate is reported.

How this compares​

ApproachWho uses itWhat it may skip
Build-graph dependencyGoogle presubmit, UberTests outside the dependency graph
Recorded traces, full run as safety netShopifyTests whose recorded call graph misses the change; Shopify reports 99.94 percent recall
Prediction from historyMetaAbout two thirds of dependent tests, still catching over 99.9 percent of regressions
A time budgetShopify, proposedWhatever is left when time runs out

We take only the first two rows, plus the budget. The repo owner's words when accepting it: "OK to relax, but I'd like to be super selective."

Making each run shorter without running less​

Selection reduces runner-minutes. Wall time needed separate work, done under a rule from November 13: slow CI is fixed at its cause, never by running fewer tests.

Caches. Pull requests read the runners' shared Go caches but only main writes them, so a pull request cannot poison or evict them. Go's build cache entries depend on the source path, so pull request checkouts now point the module cache at main's path and hit main's compiled packages. Pull request suites compile without linking test binaries.

Sharding. A planner packs test packages longest-first into shards, using durations that main records. It starts a new shard for every 120 seconds of estimated wall time, up to 4 for Hosted Relay and 3 for Local Relay. Each shard keeps its coverage profile and a final job unions them before the changed-line coverage gate. If two shards planned from different duration maps, the run fails instead of silently dropping the tests between them.

The planner cannot split a package. Hosted Relay's root integration package was about 155 of the module's 375 test-seconds, so its shard always finished last. On November 17 it was split across 4 shards of its own.

Tiers. Since late on November 17, draft pushes run checks and selected unit tests. The heavy tier (end-to-end suites, the PostgreSQL leg and that root integration package) runs when a pull request is marked ready, and a required merge-ready job holds the merge until it passes on that exact head.

The ten-minute ceiling​

9.9 minmedian PR CI run, Nov 11 to 13
14.2 minp90 of the same 56 runs
48%of those runs over 10 minutes

On November 13 the team agreed that all CI for a push finishes in under ten minutes, from the start of the run until its last job ends. The same day, a pull request added a job that pushed its run to about 13 minutes, and nothing flagged it.

A rule with no check is a wish, but a strict check would have blocked almost every merge. Sample runs measured for the decision took 16 minutes 58 seconds, 13 minutes 8 seconds, 14 minutes 25 seconds for a documentation-only change, and 4 minutes 43 seconds. Most of the overrun came from slow runners and waits between jobs, not from the pull request.

So the merge gate refuses any run of ten minutes or more unless the pull request description carries two lines:

Pull request description · required over 10 minutes
  • CI time: <minutes> min: at least the run's measured whole minutes, so a stale, smaller number fails.
  • Follow-up: TD-<n>: a tech-debt entry that exists and records the overrun.

The refusal names the three slowest jobs. The limit is configurable downward only: it cannot be raised above 600 seconds without changing the team agreement.

No overrun passes silently. Its author measures it, states it and links the debt. The ceiling is measured on the run the merge gate judges: the latest run of the head while the pull request was ready, not an earlier, faster draft run.

Red main​

Phase one: cancelled runs hid the culprit​

main ran one CI run at a time, and a newer merge cancelled the run that was waiting. Of 196 main runs from November 9 to 12, 83 were cancelled, and main was red 25 percent of the time. A red run had tested several merges at once and could not say which one broke it. A red main also stopped nothing: deploys took the head without waiting.

We modelled the textbook fix, refusing merges while main is red. Over the same window, 28 percent of pull requests would have waited, with a median of 25 minutes and a p90 of 170, adding 16 minutes to the mean time to merge. The repo owner refused it: people already avoid CI because of waits like that.

Phase two: a run per commit, gating releases​

On November 12 every main commit got its own uncancellable run, and the release tool refused to ship a commit whose own run had not passed. A red main blocked shipping, not merging. It cost about 22 extra full runs a day, about 1,800 runner-minutes.

It lasted five days. At nine merges an hour, per-commit runs consumed 900 to 1,100 of the pool's roughly 1,320 runner-minutes an hour. A complete main run costs about 102 runner-minutes. Pull request jobs queued for up to 13 minutes each, and full main runs stretched from 24 minutes when quiet to 40 to 57.

Phase three: batches, with an agent bisecting​

On November 17 main moved to batches. GitHub Actions already has the mechanism. A concurrency group holds at most one running and one pending run, and a newly queued run replaces the pending one (GitHub docs). With cancel-in-progress off, a single main group batches greedily: the executing run finishes, and the next run tests the newest head, covering every commit that landed meanwhile. Chromium's LUCI scheduler names the same policy GREEDY_BATCHING, with a default of one concurrent invocation (config.proto), and Google's TAP batches post-submit changes and splits a failing batch to find the culprit (Software Engineering at Google, ch. 23).

The parts that make batching safe:

  • Bisect only what failed. A probe reruns only the failed jobs on one commit.
  • A strict culprit rule. A commit is the culprit only if its probe fails twice and its parent's probe passes twice. Fail then pass on the same commit is classified flaky. This stops one flaky test from blaming an innocent pull request.
  • Ship the newest commit that passed, and show how far behind the head it is.
  • A built-in way back. If probe runner-minutes exceed the minutes batching saved, or the median batch passes 8 commits, an agent must open a pull request that returns main to one run per commit.

The health loop​

Every failed main run opens or updates a Red main: CI run <id> issue. It lists the failed tests, the culprit candidates (pull requests merged since the last green run) and any selection escapes. An agent reruns the failed jobs once. A pass labels the issue flaky. Otherwise the agent classifies it as regression, infra, or main_only (a check pull requests do not run). An unclassified run counts as a regression. A regression gets a fix within one working day or a revert, and the revert goes through the normal merge gate. No step waits for a person, and a red main blocks no other merge.

A worked example: one change, end to end​

Here is how the pieces compose for a single agent change to a Hosted Relay store package, as the system is designed to treat it. It is a composite, not one recorded run.

StepWhat happensWhy
Draft pushChecks run in full. Unit tests run for the store package, every package whose test binary imports it, and every package whose read record lists a changed fileImports plus reads
Same pushA migration test reads a schema file the change edited. It runs, because main's logs recorded that readRecorded reads
Marked readyA new run on the same head adds the heavy tier: end-to-end and PostgreSQL tests the change selectsTiers
Run takes 11 minutesThe merge gate refuses until the description states CI time: 11 min and a tech-debt linkTen-minute ceiling
MergeLands while a main batch is running; it joins the next batchGreedy batching
Batch failsThe agent probes only the failed job, names the culprit by the two-and-two rule, opens the issueCulprit finding
Failing test had been skipped on the PRRecorded as a selection escape; the missing read enters the recordEscape budget

Selection keeps draft runs cheap, tiers move expensive tests to the run that matters, the gate keeps that run honest about time, and batching keeps main from eating the pool.

How we measure it​

MetricTargetWhere it is reported
main first-attempt pass rate, 7 days95 percent; a pass on rerun counts as failed and flakyWeekly health report
Failed main runs per classRepair the leading class first while below targetWeekly health report
Selection escapes, 30 daysAt most 0.1 percent of mergesWeekly health report
PR CI wall time on the judged runUnder 10 minutesMerge gate, per pull request
Batch size, probe minutes, minutes saved, PR queue timeBack off if probes exceed savings or median batch over 8Weekly health report

Where it stands today​

~71%main first-attempt pass rate, Nov 14 to 17
95%target
none yetwall-time measurement after tiers
  • main passes about 71 percent of first attempts. The target is 95. With that rate, many batches will be red, and each red batch costs a bisection. The backoff rule exists because batching may not pay off at this pass rate.
  • No measured wall time after the tiers. The tiers landed late on November 17. The plan predicts about 12 minutes for a ready run. We have not measured it.
  • Batching is less than a day old. We have no batch sizes or probe costs yet.
  • Some suites still run in no CI job. Local Relay's native, stack and provider end-to-end suites, and the browser and desktop end-to-end suites, are tracked as debt.
  • Two pull requests can pass alone and fail together. We rejected a merge queue to keep the infrastructure simple. The full main run catches it after merge.

What it costs​

  • Machinery. A read recorder, a nightly trace, an inventory guard, a shard planner, a timed merge gate, a health workflow and a bisecting agent. Together they are a system someone must own.
  • The permitted gap is real. A read no record has seen yet can hide a failure until main catches it. The budget bounds the damage; it does not remove it.
  • Releases lag the head. Under batching, the newest shippable commit can be several commits behind main.
  • Process friction. Authors mark a pull request ready once, since each ready push costs a heavy-tier run.

The deeper version, as a recap​

  1. At agent speed the constraint is runner-minutes per change.
  2. Skip a test only on evidence: the import graph and recorded reads.
  3. Record reads on main, audit them nightly with a full trace, and stop trusting the record after 48 hours.
  4. No hand-written skip rules, from people or agents.
  5. Coverage-based narrowing is unsound at the edges and saved us about 2 percent. Measure before you build it.
  6. Budget the gap at 0.1 percent of merges, with automatic suspension.
  7. Shorten runs at the cause: caches written only by main, shards planned from real durations, heavy tests on the run that merges.
  8. Enforce the ceiling in the merge gate; an overrun needs its time and its debt written down.
  9. Batch main, bisect only failed jobs, and name a culprit only on two failures against two passes.
  10. Publish the pass rate, even when it is 71 percent.

CI that keeps up with agents is not the CI that runs the fewest tests. It is the CI that knows exactly why it skipped each one.


Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.