Skip to main content

Flaky Is a Defect: Fake Clocks and the End of Wall-Clock Asserts

A test that fails one run in twenty is telling you something true about timing, either in the test or in the product. These are the rules we adopted in one week to treat intermittent failures as defects, the lint that bans wall-clock upper bounds, and the mechanics behind each flaky pattern we removed.

CatalEx Engineering · Published October 23, 2026 · 09:00 UTC

A Go test in our relay codebase checks that a command writes a secret only to the file descriptor it is given. The test opens a pipe, passes the write end's descriptor number on the command line, runs the command, and reads the pipe. It passed almost every time. It was also one of four test races we closed in a single pull request on October 15.

The test was wrong in a way you cannot see by reading it quickly. The test held the pipe as an *os.File and passed its number with f.Fd(). Nothing used the file object after that line, so the garbage collector was free to finalize it, and an *os.File finalizer closes its descriptor. Sometimes that happened before the command wrote. Sometimes it happened after the command had closed the number itself, by which time a parallel test might have been handed the same number for something else.

Rerunning that job would have turned it green. That is the problem with flaky tests: the cheapest fix is always available, and it is always wrong. With coding agents opening most of our pull requests, the cheapest fix is also the one most likely to be taken, many times a day, without anyone deciding to take it.

Between October 12 and 15 we wrote down three sets of rules and one lint. This post walks through them, and explains the mechanism behind each flaky pattern, because a rule you do not understand is one you will argue your way around.

A rerun is a sample, not a verdict​

The first rule went into our coding conventions on October 12:

A test that fails intermittently, in CI or locally, is also a defect. Investigate its first failure before rerunning a job to pass ... A rerun that passes is not evidence that nothing is wrong.

Think about what a rerun measures. A test that fails 5 percent of the time passes a rerun 95 percent of the time. The rerun has not told you the failure was noise. It has drawn a second sample from the same distribution and got the likely answer. The first failure is the only evidence you have, and the rerun throws it away.

The failure is real information about one of two things. Either the test depends on an ordering or a timing it does not control, or the product does. You find out which only by making the failure happen on purpose.

The rule names the ways to do that:

MethodWhat it changesGood for
go test -count=NRuns the test N times in one processLow-probability races; state leaking between runs
go test -cpu=1,2,4Runs with different GOMAXPROCSOrderings that only appear with real parallelism, or only without it
A loaded machineStarves the scheduler, as a shared CI runner doesTiming assumptions; anything with a deadline
A temporary delay in the suspected windowWidens a race from microseconds to millisecondsConfirming which interleaving it is

The last row has research behind it. In their study of flaky tests in open source projects, Luo, Hariri, Eloussi and Marinov found that the top three causes were asynchronous waits (74 of 161 classified fixes, 45 percent), concurrency (20 percent) and test order dependency (12 percent), and that many asynchronous-wait flakes can be made to fail reliably just by changing the time delays around them (An Empirical Analysis of Flaky Tests, FSE 2014). Most of what follows falls into those three buckets.

Pattern one: the wall-clock upper bound​

The flaky assertion we found most often looked like this in words: start a timer, do the thing, assert that less than some number of milliseconds passed. It reads like a performance check. It is actually a bet on the scheduler.

Why it flakes​

The elapsed time of an operation on a real clock includes everything the machine did meanwhile: other test packages running in parallel, the garbage collector, the operating system descheduling the process. Our CI runners are shared self-hosted machines with 4 GB of memory and often a single vCPU. On one of those, a 50 millisecond operation can take 400 milliseconds because four other test binaries wanted the same core.

Now ask what the bound protects against. If the code is correct, the bound can only ever produce a false failure. If the code is broken and hangs, the test hangs too, and go test's own timeout fails it. The second case does not need the bound. So the conventions now say:

A real-time bound only stops a test that is already failing; it never decides whether correct behavior passes.

A lower bound is different, and still allowed. "This timeout must not fire before 2 seconds" cannot flake, because a slow machine only makes the measured time longer.

The lint​

On October 13 the repository's own test-lint tool gained a rule. It rejects testify assertions that put an upper bound on elapsed wall-clock time, meaning time.Since, time.Now().Sub, or a variable holding either. It runs on every module and blocks CI.

When we first ran it, it found 15 such assertions: 11 in Local Relay, 2 in Hosted Relay, and one each in the shared gateway library and the CLI client. Three of the 15 were inside a synctest bubble, where time is fake and an exact bound is deterministic, so the rule exempts code inside synctest.Test. The other twelve were reworked so that behaviour, not timing, decides the result. There turned out to be four ways to do that:

SituationRework
The test is fully in memoryMove it into synctest and assert the exact fake-time bound
The bound told apart two timeouts, a short one and a long oneMake the other timeout outlast any test run, so a regression hangs until go test fails it
The waiting party would complete anyway, just lateRelease it late on purpose (a lock released, or a decision sent, five seconds in), so a regression produces a wrong result instead of a slow pass
Another step bound or a never-arriving result already guards the testDelete the bound; check a refreshed timestamp against a captured lower bound

The third row is the clever one. A test that says "the waiter must return quickly because the lock is free" can instead hold the lock for five seconds and assert that the waiter returned the right value. If the waiter is wrongly blocked on the lock, it returns after five seconds with a result that proves it waited. The assertion is about correctness. Time only makes the bug visible.

How fake time actually works​

testing/synctest, stable since Go 1.25, runs a test inside a bubble with its own clock. The clock starts at midnight UTC on January 1, 2000. It does not tick. It moves only when every goroutine in the bubble is durably blocked, meaning it can only be woken by another goroutine in the same bubble. Then the clock jumps straight to the next timer that could wake one of them (package docs, Go blog).

That definition has edges, and our conventions name them:

  • A goroutine waiting on a mutex, on real socket I/O or in a system call is not durably blocked. The bubble cannot know whether something outside will wake it.
  • So an idle bubble does not prove that a subprocess or a socket peer has finished. synctest makes in-memory timing deterministic. It does not make external I/O deterministic.
  • For protocol tests, serve over net.Pipe instead of a loopback socket, so both ends are in-memory channels the bubble understands.

Pattern two: the effect before the state​

The October 12 conventions added a section that is about product code, not tests, because the flaky test was reporting a real ordering bug:

Record the state that another party may act on before making the effect visible to it.

Why it flakes​

Take a typical case. A goroutine handles a command. It sends a receipt to the peer, and then it updates its own record to say the command is done. The peer reacts to the receipt immediately and asks for status. Most of the time the goroutine has already updated its record by the time the question arrives. Under load, sometimes it has not, and the peer sees "not done" for a command it was just told is done.

The fix is in the product: update the state first, under the same lock its readers take. If the effect then fails, recover through the failure path that already exists, such as closing the connection so a later one resends.

How to test it​

The test rule that goes with it, added the same day: test concurrent code at its interleavings. When one goroutine makes an effect visible and another party may act on it at once, hold that effect and assert the state at that point. For a connection, that means leaving a frame unacknowledged, so the sender is stuck exactly in the window, and asserting what a reader sees right then. A test that passes because the scheduler happened to run things in the usual order has not covered the interleaving. It has only failed to find it.

Pattern three: readiness that is not readiness​

On October 15 one pull request closed four test races across Local and Hosted Relay and added rules for each. Three of them are worth understanding.

A fake child process is ready only when it says so​

A development stack starts several helper processes. A test starts the stack, then stops it, and checks that each child cleaned up. Sometimes a child left its state behind.

The mechanism: a newly started process takes a moment to install its signal handlers. Until it does, a stop signal takes the default action, which for SIGTERM is to die immediately. A test that sends the stop signal one millisecond too early kills the child before it can clean up or record that it finished.

The rule: have the child report readiness after installing its handlers, for example by answering on its listener or writing to a pipe, and stop it only after that report. Code that starts several processes waits for each one's own readiness. One process being ready never stands in for another. In the product code, the fix was that the stack now waits for the archive service to answer HTTP before starting the gateway that depends on it.

Never pass the descriptor of a file the test keeps​

This is the opening story. The rule says a descriptor handed to code that will close it belongs to that code alone. Never pass f.Fd() of an *os.File the test keeps, because once f is unreachable its finalizer may close the number at any time. Instead, duplicate the descriptor while holding syscall.ForkLock, mark the duplicate close-on-exec so a child forked by a parallel test cannot inherit it, close the original, and hand over the duplicate. Or pass the file to a child process through exec.Cmd.ExtraFiles.

A final message must be complete on its own​

An HTTP handler that flushes a reply without a declared length, then closes the connection, has sent the body bytes but not the end of the chunked body. Go's server writes that terminator only after the handler returns. A close in between cuts it off, and the client sees a truncated response once in a while. The rule: give such a reply a Content-Length, or close only after the handler has returned.

The taxonomy​

PatternWhy it flakes, mechanicallyHow to make it fail on purposeFixAutomated check
Wall-clock upper boundElapsed time includes scheduler and GC stalls on a shared runnerRun on a loaded machine, -cpu=1synctest, or let behaviour decidetest-lint rule since Oct 13
Real sleep or poll to wait for workThe work usually finishes in time, until it doesn'tLoad; shorten the sleepInjected clock or completion signalConvention; reviewers
Effect before stateA peer reacts between the send and the updateDelay after the sendUpdate state first, under the lockTest that holds the effect
Early stop of a childDefault signal action before handlers existStop immediately after startChild reports readiness after handlersConvention; reviewers
f.Fd() of a kept fileFinalizer closes the number; a parallel test reuses it-count with GC pressureClose-on-exec duplicate or ExtraFilesConvention; reviewers
Unterminated final replyChunked end written after the handler returnsClose right after flushContent-Length, or close after returnConvention; reviewers
Platform pathsA temp root under /private exists only on macOSRun on Linux CIt.TempDir() or /tmptest-lint rule since Oct 12

Only two rows are fully automated. The others are judgement rules that live in the conventions every agent reads before writing tests, and that reviewers check. We would like more of them to be lints. Most are hard to detect statically without a lot of false positives.

Why this matters more with agents​

Three things change when most pull requests come from coding agents.

The rerun is the path of least resistance. An agent whose job is to get a pull request green will rerun a failed job if nothing tells it not to, and it will be right most of the time. The rule has to be written where the agent reads it, with the reason, so it treats a first failure as the start of an investigation.

Agents copy patterns. One elapsed-time assertion in a test file becomes five when an agent writes the neighbouring tests in the same style. A lint stops the pattern from spreading. A comment in review does not.

Volume turns rare into daily. A test that flakes once in two hundred runs is a curiosity at ten runs a day. If parallel agent sessions push hundreds of runs a day, it fails every day, and every failure costs someone a rerun or an investigation.

How to measure it​

  • Reruns that turned red into green. Count them. Each one is an unreproduced defect report that someone chose not to read.
  • First-attempt pass rate on main. A rerun that passes still counts as a failure for this number.
  • Stress runs for new timing-sensitive tests. -count and -cpu over the tests a change adds, on a loaded machine. We do this by hand today; it is a candidate for an automated job.
  • The nightly race detector. Since October 8 it runs nightly instead of on pull requests, and race failures do not block merges. Each one is still a defect to fix, not a flake to rerun.

What this does not fix​

  • Real I/O stays real. synctest covers in-memory timing. Sockets, subprocesses, databases and the filesystem still race, and the rules for them are judgement, not tooling.
  • Most of the taxonomy is unenforced. Five of seven rows rely on someone reading the conventions. An agent that does not load them, or a reviewer in a hurry, lets the pattern back in.
  • The bugs a stress run cannot reach. Some interleavings need a specific load shape that neither -cpu nor a delay produces. Those still surface after merge.
  • The rules are new. They were written between October 12 and 15. We do not yet have the numbers to say how much they reduce flaky failures on main, and we will not claim it until we do.

What it costs​

  • Rework. Twelve timing assertions were rewritten, and every one needed a different trick from the table above.
  • Slower merges when a flake blocks. If you may not rerun until you have reproduced, a flaky failure holds a pull request for the length of an investigation.
  • Some product changes. Readiness reporting and state-before-effect ordering touched production code, not just tests.
  • A lint with an exemption. The synctest exemption is correct, and it is also a place a determined author can hide a real-time bound by mistake.

The checklist​

Copy this into the conventions your agents read.

  • An intermittent failure is a defect. Read the first failure before any rerun.
  • Reproduce it on purpose: -count, -cpu, a loaded machine, or a temporary delay in the suspected window.
  • No upper bound on elapsed wall-clock time outside synctest. Enforce it with a lint.
  • Lower bounds are fine. A timeout that must not fire early cannot flake.
  • In-memory timing goes in synctest or behind an injected clock. Real I/O gets a generous guard, not a sub-second deadline.
  • Update state before making an effect visible, under the readers' lock.
  • Test concurrent code at its interleavings: hold the effect, then assert.
  • A fake child is ready only after it reports readiness, after installing its signal handlers. Each process reports its own.
  • Never pass f.Fd() of a file the test keeps. Pass a close-on-exec duplicate or use ExtraFiles.
  • A final message before a close carries its own length.
  • A rerun that passes counts as a failure in your pass-rate metric.

A flaky test is a bug report with a low delivery rate. The rerun button is how you decline to read it.


Written by CatalEx Engineering. We build the AI operating layer for AI-native companies: one platform to build, deploy, and run AI agents in production. More at catalex.co.