Skip to content

Commit 9f10cf2

Browse files
committed
fix(selfhost): phase-align the cron scheduler to wall-clock boundaries
Cloudflare's own */2 * * * * cron trigger fires exactly on wall-clock 2-minute boundaries (:00, :02, :04, ...), which every minute-gated job in enqueueScheduledJobs (minute % 10 === 0, minute === 0, minute % 30 === 0 -- all even) depends on to ever run. The self-host entrypoint's plain setInterval instead ticked every CRON_INTERVAL_MS from whatever moment the container booted, with no relation to wall-clock boundaries -- and since the interval evenly divides an hour, that locks every tick to a FIXED minute parity for the container's entire lifetime. A container booting in an odd minute then ticks ONLY on odd minutes forever, so refresh-registry, ops-alerts, sweep-watchdog, backfill-registered-repos, and both reconciliation sweeps silently NEVER fire. Confirmed live on edge-nl-01: the app container booted at :49 (odd) and ran 3+ hours of on-schedule ~2-minute ticks with zero occurrences of any minute-gated job, while the unconditional every-tick sweep ran normally the whole time. Phase-align the first tick to the next true wall-clock boundary (computed from epoch, itself minute-aligned) with a one-shot setTimeout, then hand off to setInterval from that aligned moment.
1 parent b63fa69 commit 9f10cf2

3 files changed

Lines changed: 65 additions & 3 deletions

File tree

src/selfhost/cron-alignment.ts

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
/** Milliseconds from `nowMs` until the next wall-clock boundary of `intervalMs`, so a self-host `setTimeout`
2+
* can phase-align its first tick to the same instants Cloudflare's own cron trigger would fire on (e.g. the
3+
* every-2-minutes trigger fires exactly at :00, :02, :04, … UTC). Computed against epoch -- itself minute-aligned --
4+
* rather than the caller's own boot time, since `nowMs % intervalMs` only lands on true minute boundaries
5+
* (matching what `enqueueScheduledJobs`'s `getUTCMinutes()`-based gates check) when measured from a fixed,
6+
* minute-aligned origin; measuring from an arbitrary boot moment would just reproduce the exact bug this
7+
* exists to fix (see server.ts's cron setup). Exactly on a boundary already (`nowMs % intervalMs === 0`)
8+
* waits a FULL intervalMs rather than firing immediately, matching `setInterval`'s own "no immediate first
9+
* fire" semantics the caller is replacing. */
10+
export function delayToNextWallClockBoundaryMs(nowMs: number, intervalMs: number): number {
11+
const msIntoCycle = nowMs % intervalMs;
12+
return msIntoCycle === 0 ? intervalMs : intervalMs - msIntoCycle;
13+
}

src/server.ts

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ import {
6060
import { clockSkewSampleAgeSeconds, clockSkewSecondsSample } from "./selfhost/clock-skew";
6161
import { d1DatabaseSizeBytesSample, d1SignalSnapshotsRowsPerKeySample, d1TableRowCountSamples, isD1SizeProbeEnabled, runD1SizeProbe } from "./selfhost/d1-size-probe";
6262
import { gauge, gaugeVector, incr, observe, renderMetrics, setSelfHostedMetricsMode } from "./selfhost/metrics";
63+
import { delayToNextWallClockBoundaryMs } from "./selfhost/cron-alignment";
6364
import { runSelfHostMigrations } from "./selfhost/migrate";
6465
import { createPgAdapter, tuneGithubRateLimitObservationsAutovacuum, widenGithubIdColumnsToBigint } from "./selfhost/pg-adapter";
6566
import { createPgQueue } from "./selfhost/pg-queue";
@@ -1090,10 +1091,23 @@ async function main(): Promise<void> {
10901091

10911092
backend.queue.start();
10921093

1093-
// Cron — loopover ticks ~every 2 minutes; drive the SAME scheduled handler.
1094+
// Cron — loopover ticks ~every 2 minutes; drive the SAME scheduled handler. Cloudflare's own `*/2 * * * *`
1095+
// trigger fires exactly on wall-clock 2-minute boundaries (:00, :02, :04, …), which is what
1096+
// enqueueScheduledJobs's minute-gated jobs (`minute % 10 === 0`, `minute === 0`, `minute % 30 === 0` — all
1097+
// even) rely on to ever run. A plain `setInterval(fn, intervalMs)` instead ticks every intervalMs FROM
1098+
// WHATEVER MOMENT THE CONTAINER BOOTED, with no relation to wall-clock boundaries — and since intervalMs
1099+
// evenly divides an hour, that locks the tick's minute value to a FIXED parity for the container's entire
1100+
// lifetime. A container that happens to boot in an odd minute then ticks ONLY on odd minutes forever, so
1101+
// every minute-gated job above silently NEVER fires — confirmed live on edge-nl-01 (booted at an odd
1102+
// minute: 3+ hours of ~2-min ticks with zero refresh-registry/ops-alerts/sweep-watchdog/reconciliation
1103+
// dispatches, while the unconditional every-tick sweep ran normally). Phase-align the FIRST tick to the
1104+
// next true wall-clock boundary — computed from epoch, which is itself minute-aligned, so `Date.now() %
1105+
// intervalMs` lands on the same boundaries Cloudflare's cron would for any intervalMs that evenly divides
1106+
// an hour (the default 120_000 included) — with a one-shot setTimeout, then hand off to setInterval from
1107+
// that aligned moment so every subsequent tick keeps landing on those boundaries.
10941108
const intervalMs = Number(process.env.CRON_INTERVAL_MS ?? 120_000);
10951109
/* v8 ignore start -- self-host entrypoint timers start a live server; monitor semantics are covered in selfhost tests. */
1096-
const cron = setInterval(() => {
1110+
const runCronTick = (): void => {
10971111
const controller = {
10981112
scheduledTime: Date.now(),
10991113
cron: "*/2 * * * *",
@@ -1110,7 +1124,11 @@ async function main(): Promise<void> {
11101124
}),
11111125
),
11121126
);
1113-
}, intervalMs);
1127+
};
1128+
let cron: NodeJS.Timeout = setTimeout(() => {
1129+
runCronTick();
1130+
cron = setInterval(runCronTick, intervalMs);
1131+
}, delayToNextWallClockBoundaryMs(Date.now(), intervalMs));
11141132
/* v8 ignore stop */
11151133

11161134
// Orb fleet-telemetry export — ALWAYS ON (the fleet-calibration contract of self-hosting). Self-gates
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
import { describe, expect, it } from "vitest";
2+
import { delayToNextWallClockBoundaryMs } from "../../src/selfhost/cron-alignment";
3+
4+
const TWO_MINUTES_MS = 120_000;
5+
6+
describe("delayToNextWallClockBoundaryMs (self-host cron phase alignment)", () => {
7+
it("returns the delay to the next boundary when booting mid-cycle", () => {
8+
// 2026-07-21T20:49:04.768Z -- the exact odd-minute boot moment observed live on edge-nl-01, which
9+
// (with a plain, unaligned setInterval) locked every subsequent tick to odd minutes forever.
10+
const bootMs = Date.parse("2026-07-21T20:49:04.768Z");
11+
const delay = delayToNextWallClockBoundaryMs(bootMs, TWO_MINUTES_MS);
12+
const firstTickMs = bootMs + delay;
13+
expect(new Date(firstTickMs).getUTCMinutes() % 2).toBe(0);
14+
expect(delay).toBeGreaterThan(0);
15+
expect(delay).toBeLessThanOrEqual(TWO_MINUTES_MS);
16+
});
17+
18+
it("waits a full interval when already exactly on a boundary, matching setInterval's no-immediate-fire semantics", () => {
19+
const onBoundaryMs = Date.parse("2026-07-21T20:50:00.000Z");
20+
expect(delayToNextWallClockBoundaryMs(onBoundaryMs, TWO_MINUTES_MS)).toBe(TWO_MINUTES_MS);
21+
});
22+
23+
it("aligns to a minute-10/30 boundary regardless of which second within the minute it boots", () => {
24+
const bootMs = Date.parse("2026-07-21T20:57:43.219Z");
25+
const delay = delayToNextWallClockBoundaryMs(bootMs, TWO_MINUTES_MS);
26+
const firstTick = new Date(bootMs + delay);
27+
expect(firstTick.getUTCMinutes()).toBe(58);
28+
expect(firstTick.getUTCSeconds()).toBe(0);
29+
expect(firstTick.getUTCMilliseconds()).toBe(0);
30+
});
31+
});

0 commit comments

Comments
 (0)