Most Slow Tests Were Not Slow
Our slowest CI job spent most of its time waiting, not computing. This is how we measured it, the day of about forty speed commits that followed, the signals that tell a wait from real work, the ideas we tried and dropped, and the rule we now apply to running tests on a second database engine.
In late October the PostgreSQL job for Hosted Relay, the hosted gateway in our relay codebase, took between 10 and 18 minutes. Its biggest package was an integration suite of 2,474 subtests. In a local reproduction with CI's database settings, those subtests added up to 511 seconds. The suite runs four at a time, so 511 divided by 4 is about 128 seconds, which matched the measured wall time of 130 seconds almost exactly.
The surprise was the CPU. Over those 130 seconds the test process used under half a CPU. The suite was not slow because it did a lot of work. It was slow because each test spent most of its life waiting for something, and four waiting tests at a time is still mostly waiting.
That finding reorganised how we approach test speed. On November 8 we spent a day on it and merged about forty commits that made tests faster, and the method went into a guide whose first line is the title of this post:
Most slow tests were not slow: they waited on a production default, started a process or a connection per test, or exposed a product bug.
This is a playbook. It starts with how to tell a wait from work, then goes situation by situation through what we found and what we did, including the ideas that did not work.
Waiting looks like work from the outside
A CI job reports wall time. Wall time cannot tell you whether a test was busy or idle. Two measurements together can:
- Wall time against CPU time. If a package takes 130 seconds and uses 30 seconds of CPU, roughly three quarters of it is waiting.
- Sum of subtest time against the parallelism cap. If the sum divided by the parallel limit equals the wall time, the cap is setting the duration. Raising parallelism would help, but only because you would be waiting on more things at once. Removing the waits helps more.
The loop stops when the next cause is under about 5 percent of the package. Two steps in it are easy to skip and expensive to skip.
Confirm each candidate alone. Ranking tests by wall time in a parallel run credits each test with its neighbours' time. A test can look slow because it shared a core with something slow.
Do not shard first. Sharding a suite over more runners, adding runners, or lowering parallelism all leave the waits in place. Sharding is a backstop for a suite that is already lean.
The signals
These are the signals from the guide, the ones we now read before touching a slow test:
| Signal | What it usually means |
|---|---|
| Durations are round: 1.0, 2.0, 5.0 seconds | A production interval is being waited out. Often a missing wake-up in the product, not the test |
| CPU samples far below wall time in a profile | A timer. A goroutine dump taken mid-test shows what it is parked on |
| A Vitest file takes about as long on a laptop as on CI | Waiting. A uniform 8 to 13 times ratio means CPU |
| Subtests of about 0.1 seconds each, but a package of many seconds | Each case pays a fixed wait or setup cost |
Round numbers deserve their own sentence. Code rarely takes exactly two seconds by accident. When a test's duration sits on a whole number, or on a multiple of one, it is almost always sitting out a configured interval.
Situation: the test inherits production pacing
What we found. Our Google Chat connector paces itself as production requires: one write per second per space, and two seconds between status edits. The integration fixture built the connector with those production defaults. Each Chat delivery in a test took 830 to 875 milliseconds. The 157 Chat subtests averaged 0.86 seconds against 0.15 for everything else, and made up 27 percent of all subtest time.
What we did. The connector already accepted a clock and a sleep function. The service simply gave tests no way to set them before its workers started. We added that, and the fixture now runs Chat pacing on a virtual clock whose sleep moves time forward instead of waiting.
| Scope | Before | After |
|---|---|---|
| Google Chat tests only | 34.4 s | 12.9 s |
| The whole integration package | 130 s | 101 s |
On November 8 the same treatment went to every Chat test stack, to Slack deadlines, and to retry delays and budgets in Hosted Relay. In Local Relay, the test stacks stopped inheriting production reconnect, gateway, handshake and probe intervals. The guide's rule: set each interval for the test, in the one helper every stack uses, and keep a sensible value. A 200 millisecond probe, not the 10 millisecond minimum, because one command then sat out its own 5 second control timeout instead.
Situation: the slow test is a product bug
This is the case that makes the whole exercise worth doing.
What we found. Once Chat pacing no longer dominated, tests still showed
gaps of 0.8 to 1.0 seconds between steps. The delivery worker polls an idle
outbox once a second. A reply queued while the outbox was idle waited for the
next poll. In tests that was an annoyance. In production it meant every
agent reply waited up to a second after an idle period, because a commit
path used tx.Commit().Error, which skipped the after-commit hook that was
supposed to wake the worker.
What we did. Fixed the product: delivery and dispatch now wake on every commit, and a lint now refuses the raw commit call in favour of a helper that runs the after-commit wakes. The guide records the lesson under the heading wake-ups are product code:
Setting the poll to 100 ms made the tests fast and hid it.
That is the trap. The test-only fix, a shorter poll interval in the fixture, was available, quick, and would have made the suite green and fast. It would also have hidden a one-second latency from every user.
The same day turned up another. A dashboard test of a long chat was slow because projecting a replayed conversation scanned every message per record, which made replaying N records cubic. That was fixed in the product too, with a lookup done once per projection.
Situation: the same expensive read, repeated
What we found. Our statement-failure sweep tests run an operation of N statements N+1 times, failing a different statement each time, and read a 15-table ledger before and after every attempt. That makes them grow with the square of the statement count, and they were 20 percent of subtest time. The "before" ledger cannot change between attempts, because each attempt asserts that it left nothing behind.
What we did. Read the "before" ledger once. Every assertion stayed. The sweep tests went from 11.8 to 8.5 seconds.
Situation: a fresh database per test
What we found. Hosted Relay's PostgreSQL integration tests created and dropped a database per test, copied from a migrated template. Each one cost 42 milliseconds of wall time, which came to 73 seconds across 1,745 fixtures, or 14 percent of the package. Measured by CPU it was worse. PostgreSQL used 1.3 cores against the test process's 0.56. On a runner with about two CPUs, the database's setup work set the pace.
We looked at how large, long-lived suites isolate their databases before choosing. Rails and Django give each worker process its own database and usually roll back a transaction per test (Rails testing guide, Django testing tools). Rollback does not fit our service: it commits, runs after-commit work, holds an advisory lock on a dedicated connection and runs workers on other connections. One shared database does not fit either: the service takes a per-database ownership lock and its workers claim every due row.
What we did. One database per parallel slot, reset between tests by deleting every row with foreign keys deferred, and sequences restored to the template's positions. Each slot runs one test at a time, so every conflict above only ever sees one test.
| Package | Wall time before | After | PostgreSQL CPU before | After |
|---|---|---|---|---|
| Integration suite | 228 s | 106 s | 293 s | 119 s |
| Service package | 75 s | 45 s | 77 s | 29 s |
Those are local measurements with CI's server flags. On November 8 the same
idea reached SQLite fixtures: lend reset slots and keep connection pools. Two
details from that change are worth stealing. Reset with
DELETE FROM t WHERE true, never a bare DELETE, which in SQLite rewrites
every index page. And never ATTACH, which re-parses the schema.
Situation: setup that never changes
Several fixes on November 8 were the same idea in different clothes: work done per test whose result cannot change during the run.
| Per-test cost | Fix |
|---|---|
| Account fixtures hashed passwords with bcrypt at cost 10 | Use bcrypt's minimum cost for fixtures, through an option applied after config validation |
| Each migration test walked every migration from an empty database | Build a migrated template once per migration revision and copy it; start a migration test at its own schema |
| Each test built a git repository from scratch | Build one template repository and copy it, skipping lock files |
| A readiness loop spawned a process on every poll | Poll in process; spawn once |
| A fixture far bigger than the limit it proves | Size it just over the limit |
| Journals and state files fsynced on every commit | A RAM directory such as /dev/shm for test journals, and fewer commits |
bcrypt is a good example of a cost that is a feature in production and pure waste in a fixture. Each step of its cost parameter doubles the work, and Go's package defines a default of 10 and a minimum of 4 (golang.org/x/crypto/bcrypt). A fixture that creates an account per test pays that difference on every test, to protect a password nobody will ever attack.
Situation: the frontend suite imports the world
The dashboard's web suite had its own version of the problem. The fixes on November 8 and the week before:
- happy-dom instead of a full DOM for suites that only need one, and no DOM at all for tests that do not render.
- Fake delays in Vitest instead of real waits, and fake
setIntervalalone for polls. - Per-icon imports. Importing the icon library's barrel file in every test file paid for every icon, every time.
- Mount less. Production stopped rendering closed dialogs and hidden tab panels, and a test picker offered three options instead of hundreds. Both made tests faster and the product lighter.
Two rules we wrote down because we nearly broke them
Never shrink a must-succeed timeout. An operation that must succeed finishes seconds inside its timeout, because a CI runner can have a single CPU. During the speed work someone cut the Slack test constants. The local run got faster, and both SQLite CI shards failed. The fix for a slow deadline test is to fake the clock the bound reads, not to move the bound.
Measure on CI, not on a laptop. In the October investigation the CI runner was about 2.6 times slower per test than a laptop for the database suite. Across our suites, CI runners are 4 to 13 times slower than a developer machine and CPU-bound, and a Mac misleads in both directions: a file sync there is a full fsync, SQLite's wasm build uses more memory, and every fresh executable is scanned on first launch. Runner noise is 15 to 100 percent between runs. One CI run therefore cannot prove a gain under 20 percent. Compare shares inside a single run, or repeat the run.
To make that possible, every Go job now writes a summary with the 25 slowest tests and 10 slowest packages, plus a per-test duration artifact. You cannot rank what you cannot see.
What we tried and dropped
The guide keeps a table of ideas that were measured and did not pay. It says "do not retry without new evidence." This is the part of a speed effort that usually gets lost, and it is the part that saves the next person a day.
| Idea | Why not |
|---|---|
Vitest isolate: false | Module-level state leaks between files. It is the only thing that removes the 95 ms happy-dom import per file, and we still refuse it |
| Vitest thread pool, dependency optimizer, Node compile cache, a bigger heap semi-space | No gain; the optimizer was slower |
| Narrower Testing Library queries, accessibility checks off | Slower, or 6 percent faster at the cost of 10 broken tests. The real cost is React rendering |
GORM PrepareStmt | No CPU gain, and it breaks across schema resets |
| A shared store fixture template | 4 percent CPU for 330 call-site edits |
-covermode=set | No consistent gain in a CI A/B |
Narrow -coverpkg | Breaks tests that re-execute their own binary |
core.fsync=none for test git repositories | Within noise |
A faster sleep stub for a polling script | The loop polled about 20 times more, and each poll forks |
| Merging tests into tables for speed | Saves about nothing; each case keeps its fixture |
| Faster runners | Refused. Fix the tests |
The second database engine
The last change of that week was a policy, not an optimisation.
Our relay modules run on SQLite by default and PostgreSQL in production. Every
suite ran on SQLite. PostgreSQL also ran each module's whole set of database
packages on every push to main. Local Relay's PostgreSQL step took about 22
minutes, and almost every test in it had already passed on SQLite minutes
earlier.
The repo owner's view, recorded in the decision of November 8, is that CI tests check that specific parts are logically correct. They are not a way to run the whole codebase on every engine. Running the same test on both engines is a red flag unless the second run proves something SQLite cannot.
So now SQLite runs every suite, integration included. PostgreSQL runs only
tests in *_postgres_test.go files or behind a postgres build tag, plus one
pass per module that applies the PostgreSQL migrations up, down and up again.
A pull request starts the PostgreSQL leg only when it touches PostgreSQL code,
the driver or files that start the test database. A full PostgreSQL run of
everything still exists as a target a person can run by hand.
The decision states its own residual risk, and so should we. Engine-agnostic code whose behaviour differs on PostgreSQL (SQL dialect, row order, NULL handling, collation, type coercion) is no longer exercised there. A defect of that kind reaches production unless a PostgreSQL-tagged test covers it. The mitigation is a review rule: code whose behaviour depends on the engine gets a PostgreSQL test. The decision also names its revisit trigger: a production or staging defect that a PostgreSQL run of an engine-agnostic test would have caught.
What this does not fix
- Real compute. When CPU really is the cost, as in React rendering, these techniques stop helping. The dropped-ideas table is mostly that boundary.
- Shared runners. Our runners are slow and noisy. Faster tests make each job shorter. They do nothing about the time a job waits for a runner.
- Engine-specific bugs. The PostgreSQL policy trades about 22 minutes on
mainfor a known blind spot, which review has to cover. - The next slow test. New tests get written against production defaults every day. Our conventions ask that a new test over about 2 seconds in the CI summary explains why in its pull request. That is a judgement check, not a gate.
What it costs
- Seams in product code. Virtual clocks need injection points: a clock option on the connector, deadlines and tickers passed as dependencies. Production code grows parameters that only tests set.
- Fixture complexity. Reset slots, templates and kept pools are more code than "create a fresh database". They need their own tests, and a check, whenever a slot is handed back, that no connection is left behind.
- A day of focus. About forty commits in one day is a lot of change in shared test helpers, landing while other work continued.
The playbook
| If you see | It is probably | Do this |
|---|---|---|
| Package sum of subtests ÷ parallel cap ≈ wall time, low CPU | Tests waiting, not working | Find the waits before touching parallelism |
| Round durations: 1, 2, 5 seconds | A production interval waited out | Look for a missing wake-up in the product first, then set the interval for the test |
| A shorter poll in the fixture would fix it | A product latency bug, possibly | Fix the wake-up. Do not hide it |
| A must-succeed timeout makes a test slow | A deadline on a real clock | Fake the clock the bound reads. Never shrink the bound |
| Every subtest pays the same 40 ms | Per-test setup | Build once, copy or reset per test |
| Fixture uses a production-cost hash or KDF | Deliberate cost in the wrong place | Minimum cost for fixtures only |
| Vitest file as slow locally as on CI | Real waits or heavy imports | Fake delays, per-icon imports, lighter DOM |
| A gain of 10 percent on one CI run | Noise, probably | Compare shares within a run, or repeat it |
| The same test runs on two database engines | Duplicate verification | Keep the second engine for what only it can prove |
A test that takes a second is rarely doing a second of work. Find what it is waiting for. Sometimes it is your users, too.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.