Cron Is Not a Scheduler: Running Agents on Time at Scale
"Run this agent every morning at 9" sounds like a one-line feature. There's a cron expression, there's a job table, there's a thing that reads the table. Ship it Friday.
Cron is the easy 5%. The other 95% is not firing twice, not losing a fire, not holding a database transaction open across a task that runs for minutes, and deciding whose 9am you actually mean. A cron expression is a trigger. A scheduler is everything you build around it.
This is a tour of that everything — the design of the scheduling layer under CatalEx, and the reasoning behind the parts that look strange until you've been bitten.
The shape of the stack
A managed cron trigger fires a tick endpoint every minute. The jobs service scans the database for anything due, and dispatches each due job through a managed task queue. The queue calls back into the owning service's handler, which does the actual work.
Four moving parts, three network hops, and every one of them is a place where the same job can fire twice or not at all. Everything below is about closing those gaps.
At the top of the stack, a retry is worse than a miss
The cron trigger is configured with zero retries. Deliberately. The comment in the code is the whole argument:
The next tick will re-evaluate any jobs that were due; re-trying a failed tick risks double-dispatch against the same window.
This inverts the instinct you've built everywhere else in your system. Down in the leaves, retry is free and correct. At the root of a scheduling stack, retry is a duplicate-fire generator. The tick is not a request that must succeed; it's a sample of a clock that will be sampled again in sixty seconds. A missed tick costs you a job running one minute late. A retried tick that partially succeeded costs you a job that ran twice — and "ran twice" for an agent means two emails sent, two tickets filed, two rows written into someone's CRM.
Fail-open on the way down. Fail-closed on the way up.
Four layers against double-fire
Zero retries on the tick is a policy, not a mechanism. Below it sit four mechanisms, each one assuming the layer above it failed.
| # | Mechanism | What it stops |
|---|---|---|
| 0 | Zero retries on the tick | Two ticks evaluating the same due window |
| 1 | FOR UPDATE SKIP LOCKED on the due-job scan | Two dispatcher instances claiming the same row |
| 2 | next_run advanced in the same committed transaction as the execution row | The job still looking due after it's been claimed |
| 3 | Task-name dedup — the enqueue passes execution_id as the queue's task name | Queue redelivery inside the dedup window |
| 4 | Idempotency claim at the handler — insert ON CONFLICT DO NOTHING on execution_id | Everything the first four missed |
Layer 1 is why you can run more than one dispatcher. SELECT … FOR UPDATE SKIP LOCKED on the due-job scan means concurrent dispatcher instances never
double-dispatch the same job — the second instance doesn't block on the locked
row, it walks past it and takes the next one. Without SKIP LOCKED you get
either a convoy (everyone blocks behind row one) or a single-instance
scheduler, which is a single point of failure with extra steps.
Layer 2 is the one people skip. next_run is advanced in the same
committed transaction that writes the execution row. The moment that commit
lands, the job is no longer due — before the dispatch has even been
attempted. The scan can't pick it up again on the next tick, because as far as
the query is concerned, the window has already passed.
Layer 3 costs one argument. Every enqueue passes the execution id as the queue's task name. A queue that redelivers the same task name inside its dedup window drops it. This is free protection you get by naming a thing carefully.
Layer 4 is the backstop, and its reasoning is the elegant bit. The handler,
before doing any work at all, inserts a claim row keyed on execution_id with
ON CONFLICT DO NOTHING. If the insert claimed nothing, someone else already
has this execution, and the handler returns {"skipped": "duplicate execution"} and goes home.
Why this is safe. A legitimate retry mints a NEW execution_id, so the SAME execution_id arriving twice is always a duplicate delivery, never a lost retry.
The usual objection to handler-side idempotency is that you'll swallow a real retry. You won't — but only because retries are minted upstream as new executions, so identity and attempt are never conflated. Idempotency keys work when the key means this delivery, not this task. Get that distinction wrong and your dedup layer starts eating work.
Three rules about transaction boundaries
These are design rules, not scars. Each one exists because the alternative has a specific, predictable failure mode.
Never hold a transaction across the dispatch. A downstream agent run takes minutes. Wrap the dispatch in the transaction that read the due jobs and you're holding an idle-in-transaction connection for the duration — long enough to trip the database's idle-in-transaction timeout, at which point the connection dies mid-update and you get to find out what your code does about that. Hence the split: read due jobs, commit, then dispatch.
Commit before you dispatch — in that order. Dispatch first and the callback opens a fresh transaction that cannot see your uncommitted execution row. The handler looks up an execution that doesn't exist, and here's the vicious part: it still returns 200. The queue sees success, never retries, and the work vanishes with no error anywhere. Silent loss is worse than loud failure, and the ordering is the only thing between you and it.
Report failures in-band. A malformed job returns HTTP 200 with
status="failed" in the body. It's tempting to return a 5xx — it feels honest.
But the tick's transport layer would read that as a transport error and apply
transport retry semantics to a permanently broken job, re-dispatching a payload
that will never parse. Status belongs in the body so the tick can apply its own
retry policy. HTTP status codes describe the conversation, not the work.
Whose 9am?
Now the part that looks like a bug and isn't.
Cron parsing is a standard 5-field expression, always UTC. There is no timezone column. Not "we haven't added one yet" — the API contract explicitly forbids timezone-bearing fields on the wire. The database stores a UTC cron and nothing else.
All local↔UTC math lives in the browser, because the browser is the only place that knows the user's local zone. The server never guesses, never infers from a header, never stores a stale zone that was true when the user signed up in a city they've since left. There's one authority for "your 9am" and it's the machine sitting in front of the user.
There's no interval primitive either. "Every N hours" compiles in the browser
into ${minute} */${n} * * *, with the divisors restricted to
[1,2,3,4,6,8,12] — the clean divisors of 24. That restriction is doing real
work: */5 hours doesn't tile 24 evenly, so it silently skips at the midnight
wraparound. You'd get 0,5,10,15,20 and then a four-hour gap, forever, and
nobody would notice for weeks. Restricting the input is cheaper than explaining
the arithmetic.
Alongside the cron sits a display descriptor — frequency, time_of_day, days_of_week, interval_hours — stored purely so the UI can round-trip losslessly. Rendering a UTC cron through a timezone-naive formatter prints UTC numbers styled as wall-clock time, which is how you end up telling a user their job runs at 2pm when it runs at 9am.
Weekly schedules roll each weekday independently through UTC, then dedupe. A local Monday 11pm legitimately lands on Tuesday UTC while the rest of the week doesn't move. The naive version — shift the time, then shift every day by +1 — is wrong for exactly the days that cross midnight, which is the only case where it matters.
The honest tradeoff: because only a UTC cron is stored, schedules do not follow DST. A 9am local schedule becomes 8am or 10am local when the offset changes. Nothing recomputes crons on DST boundaries. This is a real limitation and we'd rather say it plainly than let you discover it in March. The alternative isn't free: following DST means storing the user's zone, which means recomputing every affected cron whenever a government changes its rules, and owning the ambiguity of schedules that land in the hour that doesn't exist or the hour that happens twice. UTC-only buys a schema with one interpretation. That's the trade, made with open eyes.
The unglamorous rest
The remaining details are the ones that decide whether the thing survives a year.
Execution timeout is a per-job timeout_minutes, constrained at the database
between 1 and 15. A stale sweep marks anything past its timeout as failed and
retries it — which is the only place a retry is minted, and it mints a new
execution id, which is what makes layer 4 sound.
Primary keys are UUIDv7 — time-ordered, so inserts land at the end of the index
instead of scattering across it. Hot-path indexes are partial: the index on
next_run carries WHERE status = 'active', because the dispatcher never asks
about inactive jobs and there's no reason to make it page past them.
A duplicate-schedule guard allows one schedule per agent per cron expression, enforced at the application layer, returning 409. Distinct agents may share a cron — the constraint is against a user double-clicking their way into two identical jobs, not against 9am being popular.
Orphan protection runs both ways. Deleting an agent purges its schedules; and if a run finds its agent already gone, the handler purges the schedules itself. Otherwise an orphaned schedule fires every tick forever, doing nothing, and nobody notices because nothing is failing.
And a run that suspends — waiting on a human approval — ends without a terminal frame. The execution still counts as completed, and records a status breadcrumb instead. The dispatch did its job; a retry would double-run the agent. The scheduler's contract is "the work was handed off correctly," not "the work finished."
That's the shape of it. One cron expression at the top, and underneath it four layers of dedup, three rules about where a transaction may not go, a timezone model with a stated limitation, and a fistful of indexes. The cron was the easy part. It's always the easy part.
Written by CatalEx Engineering. We build the AI operating layer for AI-native companies — one platform to build, deploy, and run AI agents in production. More at catalex.co.