Flaky Is a Defect: Fake Clocks and the End of Wall-Clock Asserts
A test that fails one run in twenty is telling you something true about timing, either in the test or in the product. These are the rules we adopted in one week to treat intermittent failures as defects, the lint that bans wall-clock upper bounds, and the mechanics behind each flaky pattern we removed.
A Go test in our relay codebase checks that a command writes a secret only to the file descriptor it is given. The test opens a pipe, passes the write end's descriptor number on the command line, runs the command, and reads the pipe. It passed almost every time. It was also one of four test races we closed in a single pull request on October 15.
The test was wrong in a way you cannot see by reading it quickly. The test held the
pipe as an *os.File and passed its number with f.Fd(). Nothing used the
file object after that line, so the garbage collector was free to finalize
it, and an *os.File finalizer closes its descriptor. Sometimes that happened
before the command wrote. Sometimes it happened after the command had closed
the number itself, by which time a parallel test might have been handed the
same number for something else.
Rerunning that job would have turned it green. That is the problem with flaky tests: the cheapest fix is always available, and it is always wrong. With coding agents opening most of our pull requests, the cheapest fix is also the one most likely to be taken, many times a day, without anyone deciding to take it.
Between October 12 and 15 we wrote down three sets of rules and one lint. This post walks through them, and explains the mechanism behind each flaky pattern, because a rule you do not understand is one you will argue your way around.
A rerun is a sample, not a verdict
The first rule went into our coding conventions on October 12:
A test that fails intermittently, in CI or locally, is also a defect. Investigate its first failure before rerunning a job to pass ... A rerun that passes is not evidence that nothing is wrong.
Think about what a rerun measures. A test that fails 5 percent of the time passes a rerun 95 percent of the time. The rerun has not told you the failure was noise. It has drawn a second sample from the same distribution and got the likely answer. The first failure is the only evidence you have, and the rerun throws it away.
The failure is real information about one of two things. Either the test depends on an ordering or a timing it does not control, or the product does. You find out which only by making the failure happen on purpose.
The rule names the ways to do that:
| Method | What it changes | Good for |
|---|---|---|
go test -count=N | Runs the test N times in one process | Low-probability races; state leaking between runs |
go test -cpu=1,2,4 | Runs with different GOMAXPROCS | Orderings that only appear with real parallelism, or only without it |
| A loaded machine | Starves the scheduler, as a shared CI runner does | Timing assumptions; anything with a deadline |
| A temporary delay in the suspected window | Widens a race from microseconds to milliseconds | Confirming which interleaving it is |
The last row has research behind it. In their study of flaky tests in open source projects, Luo, Hariri, Eloussi and Marinov found that the top three causes were asynchronous waits (74 of 161 classified fixes, 45 percent), concurrency (20 percent) and test order dependency (12 percent), and that many asynchronous-wait flakes can be made to fail reliably just by changing the time delays around them (An Empirical Analysis of Flaky Tests, FSE 2014). Most of what follows falls into those three buckets.
Pattern one: the wall-clock upper bound
The flaky assertion we found most often looked like this in words: start a timer, do the thing, assert that less than some number of milliseconds passed. It reads like a performance check. It is actually a bet on the scheduler.
Why it flakes
The elapsed time of an operation on a real clock includes everything the machine did meanwhile: other test packages running in parallel, the garbage collector, the operating system descheduling the process. Our CI runners are shared self-hosted machines with 4 GB of memory and often a single vCPU. On one of those, a 50 millisecond operation can take 400 milliseconds because four other test binaries wanted the same core.
Now ask what the bound protects against. If the code is correct, the bound can
only ever produce a false failure. If the code is broken and hangs, the test
hangs too, and go test's own timeout fails it. The second case does not need
the bound. So the conventions now say:
A real-time bound only stops a test that is already failing; it never decides whether correct behavior passes.
A lower bound is different, and still allowed. "This timeout must not fire before 2 seconds" cannot flake, because a slow machine only makes the measured time longer.
The lint
On October 13 the repository's own test-lint tool gained a rule. It rejects
testify assertions that put an upper bound on elapsed wall-clock time, meaning
time.Since, time.Now().Sub, or a variable holding either. It runs on every
module and blocks CI.
When we first ran it, it found 15 such assertions: 11 in Local Relay, 2 in
Hosted Relay, and one each in the shared gateway library and the CLI client.
Three of the 15 were inside a synctest bubble, where time is fake and an
exact bound is deterministic, so the rule exempts code inside synctest.Test.
The other twelve were reworked so that behaviour, not timing, decides the
result. There turned out to be four ways to do that:
| Situation | Rework |
|---|---|
| The test is fully in memory | Move it into synctest and assert the exact fake-time bound |
| The bound told apart two timeouts, a short one and a long one | Make the other timeout outlast any test run, so a regression hangs until go test fails it |
| The waiting party would complete anyway, just late | Release it late on purpose (a lock released, or a decision sent, five seconds in), so a regression produces a wrong result instead of a slow pass |
| Another step bound or a never-arriving result already guards the test | Delete the bound; check a refreshed timestamp against a captured lower bound |
The third row is the clever one. A test that says "the waiter must return quickly because the lock is free" can instead hold the lock for five seconds and assert that the waiter returned the right value. If the waiter is wrongly blocked on the lock, it returns after five seconds with a result that proves it waited. The assertion is about correctness. Time only makes the bug visible.
How fake time actually works
testing/synctest, stable since Go 1.25, runs a test inside a bubble with its
own clock. The clock starts at midnight UTC on January 1, 2000. It does not
tick. It moves only when every goroutine in the bubble is durably blocked,
meaning it can only be woken by another goroutine in the same bubble. Then the
clock jumps straight to the next timer that could wake one of them
(package docs,
Go blog).
That definition has edges, and our conventions name them:
- A goroutine waiting on a mutex, on real socket I/O or in a system call is not durably blocked. The bubble cannot know whether something outside will wake it.
- So an idle bubble does not prove that a subprocess or a socket peer has
finished.
synctestmakes in-memory timing deterministic. It does not make external I/O deterministic. - For protocol tests, serve over
net.Pipeinstead of a loopback socket, so both ends are in-memory channels the bubble understands.
Pattern two: the effect before the state
The October 12 conventions added a section that is about product code, not tests, because the flaky test was reporting a real ordering bug:
Record the state that another party may act on before making the effect visible to it.
Why it flakes
Take a typical case. A goroutine handles a command. It sends a receipt to the peer, and then it updates its own record to say the command is done. The peer reacts to the receipt immediately and asks for status. Most of the time the goroutine has already updated its record by the time the question arrives. Under load, sometimes it has not, and the peer sees "not done" for a command it was just told is done.
The fix is in the product: update the state first, under the same lock its readers take. If the effect then fails, recover through the failure path that already exists, such as closing the connection so a later one resends.
How to test it
The test rule that goes with it, added the same day: test concurrent code at its interleavings. When one goroutine makes an effect visible and another party may act on it at once, hold that effect and assert the state at that point. For a connection, that means leaving a frame unacknowledged, so the sender is stuck exactly in the window, and asserting what a reader sees right then. A test that passes because the scheduler happened to run things in the usual order has not covered the interleaving. It has only failed to find it.
Pattern three: readiness that is not readiness
On October 15 one pull request closed four test races across Local and Hosted Relay and added rules for each. Three of them are worth understanding.
A fake child process is ready only when it says so
A development stack starts several helper processes. A test starts the stack, then stops it, and checks that each child cleaned up. Sometimes a child left its state behind.
The mechanism: a newly started process takes a moment to install its signal
handlers. Until it does, a stop signal takes the default action, which for
SIGTERM is to die immediately. A test that sends the stop signal one
millisecond too early kills the child before it can clean up or record that
it finished.
The rule: have the child report readiness after installing its handlers, for example by answering on its listener or writing to a pipe, and stop it only after that report. Code that starts several processes waits for each one's own readiness. One process being ready never stands in for another. In the product code, the fix was that the stack now waits for the archive service to answer HTTP before starting the gateway that depends on it.
Never pass the descriptor of a file the test keeps
This is the opening story. The rule says a descriptor handed to code that will
close it belongs to that code alone. Never pass f.Fd() of an *os.File the
test keeps, because once f is unreachable its finalizer may close the number
at any time. Instead, duplicate the descriptor while holding
syscall.ForkLock, mark the duplicate close-on-exec so a child forked by a
parallel test cannot inherit it, close the original, and hand over the
duplicate. Or pass the file to a child process through exec.Cmd.ExtraFiles.
A final message must be complete on its own
An HTTP handler that flushes a reply without a declared length, then closes
the connection, has sent the body bytes but not the end of the chunked body.
Go's server writes that terminator only after the handler returns. A close in
between cuts it off, and the client sees a truncated response once in a
while. The rule: give such a reply a Content-Length, or close only after the
handler has returned.
The taxonomy
| Pattern | Why it flakes, mechanically | How to make it fail on purpose | Fix | Automated check |
|---|---|---|---|---|
| Wall-clock upper bound | Elapsed time includes scheduler and GC stalls on a shared runner | Run on a loaded machine, -cpu=1 | synctest, or let behaviour decide | test-lint rule since Oct 13 |
| Real sleep or poll to wait for work | The work usually finishes in time, until it doesn't | Load; shorten the sleep | Injected clock or completion signal | Convention; reviewers |
| Effect before state | A peer reacts between the send and the update | Delay after the send | Update state first, under the lock | Test that holds the effect |
| Early stop of a child | Default signal action before handlers exist | Stop immediately after start | Child reports readiness after handlers | Convention; reviewers |
f.Fd() of a kept file | Finalizer closes the number; a parallel test reuses it | -count with GC pressure | Close-on-exec duplicate or ExtraFiles | Convention; reviewers |
| Unterminated final reply | Chunked end written after the handler returns | Close right after flush | Content-Length, or close after return | Convention; reviewers |
| Platform paths | A temp root under /private exists only on macOS | Run on Linux CI | t.TempDir() or /tmp | test-lint rule since Oct 12 |
Only two rows are fully automated. The others are judgement rules that live in the conventions every agent reads before writing tests, and that reviewers check. We would like more of them to be lints. Most are hard to detect statically without a lot of false positives.
Why this matters more with agents
Three things change when most pull requests come from coding agents.
The rerun is the path of least resistance. An agent whose job is to get a pull request green will rerun a failed job if nothing tells it not to, and it will be right most of the time. The rule has to be written where the agent reads it, with the reason, so it treats a first failure as the start of an investigation.
Agents copy patterns. One elapsed-time assertion in a test file becomes five when an agent writes the neighbouring tests in the same style. A lint stops the pattern from spreading. A comment in review does not.
Volume turns rare into daily. A test that flakes once in two hundred runs is a curiosity at ten runs a day. If parallel agent sessions push hundreds of runs a day, it fails every day, and every failure costs someone a rerun or an investigation.
How to measure it
- Reruns that turned red into green. Count them. Each one is an unreproduced defect report that someone chose not to read.
- First-attempt pass rate on
main. A rerun that passes still counts as a failure for this number. - Stress runs for new timing-sensitive tests.
-countand-cpuover the tests a change adds, on a loaded machine. We do this by hand today; it is a candidate for an automated job. - The nightly race detector. Since October 8 it runs nightly instead of on pull requests, and race failures do not block merges. Each one is still a defect to fix, not a flake to rerun.
What this does not fix
- Real I/O stays real.
synctestcovers in-memory timing. Sockets, subprocesses, databases and the filesystem still race, and the rules for them are judgement, not tooling. - Most of the taxonomy is unenforced. Five of seven rows rely on someone reading the conventions. An agent that does not load them, or a reviewer in a hurry, lets the pattern back in.
- The bugs a stress run cannot reach. Some interleavings need a specific
load shape that neither
-cpunor a delay produces. Those still surface after merge. - The rules are new. They were written between October 12 and 15. We do
not yet have the numbers to say how much they reduce flaky failures on
main, and we will not claim it until we do.
What it costs
- Rework. Twelve timing assertions were rewritten, and every one needed a different trick from the table above.
- Slower merges when a flake blocks. If you may not rerun until you have reproduced, a flaky failure holds a pull request for the length of an investigation.
- Some product changes. Readiness reporting and state-before-effect ordering touched production code, not just tests.
- A lint with an exemption. The
synctestexemption is correct, and it is also a place a determined author can hide a real-time bound by mistake.
The checklist
Copy this into the conventions your agents read.
- An intermittent failure is a defect. Read the first failure before any rerun.
- Reproduce it on purpose:
-count,-cpu, a loaded machine, or a temporary delay in the suspected window. - No upper bound on elapsed wall-clock time outside
synctest. Enforce it with a lint. - Lower bounds are fine. A timeout that must not fire early cannot flake.
- In-memory timing goes in
synctestor behind an injected clock. Real I/O gets a generous guard, not a sub-second deadline. - Update state before making an effect visible, under the readers' lock.
- Test concurrent code at its interleavings: hold the effect, then assert.
- A fake child is ready only after it reports readiness, after installing its signal handlers. Each process reports its own.
- Never pass
f.Fd()of a file the test keeps. Pass a close-on-exec duplicate or useExtraFiles. - A final message before a close carries its own length.
- A rerun that passes counts as a failure in your pass-rate metric.
A flaky test is a bug report with a low delivery rate. The rerun button is how you decline to read it.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.