fix: invert shouldRefill() condition in v1 scheduler rate limiter#4379
fix: invert shouldRefill() condition in v1 scheduler rate limiter#4379CoderSufiyan wants to merge 2 commits into
Conversation
The shouldRefill() method in the v1 scheduler rate limiter had an inverted boolean condition. It returned true when nextRefillAt was still in the future (window still open, no refill needed) and false when nextRefillAt was in the past (window closed, refill needed). This caused two issues: 1. Constant write-lock churn in use() — shouldRefill() returned true almost always (when window was open), triggering unnecessary flushToDatabase() calls that immediately no-opped. 2. Spurious rate-limit rejections after window rollover — shouldRefill() returned false, so use() never triggered the on-demand refresh. Flipping the condition with '!' aligns shouldRefill() with the flushToDatabase() guard, which correctly no-ops while nextRefillAt.After(now). Fixes hatchet-dev#4346
|
@CoderSufiyan is attempting to deploy a commit to the Hatchet Team on Vercel. A member of the Team first needs to authorize it. |
STiFLeR7
left a comment
There was a problem hiding this comment.
Verified the fix against the sibling guard in flushToDatabase (line ~293: if r.nextRefillAt.After(time.Now().UTC()) { return nil }) — the two now agree: use() calls flushToDatabase exactly when nextRefillAt has actually passed, instead of the inverted original which called it while a refill was not yet due and stopped calling it once one became due. No race between the two separate time.Now() calls either, since time only moves forward between them and nextRefillAt is fixed — so the two checks can't disagree.
One gap: this ships with no test. Given shouldRefill/nextRefillAt is guarded by nextRefillAtMu and sits on a concurrency-sensitive path, a focused unit test asserting shouldRefill() is false while nextRefillAt is in the future and true once it's passed (matching flushToDatabase's polarity) would make this much faster to merge with confidence and guard against the exact inversion class of bug regressing later. The original issue reporter (#4346) even offered to include one — might be worth pulling that in before merge.
- TestRateLimiter_ShouldRefill covers nil/future/past nextRefillAt - TestRateLimiter_UseRefreshesFromDbOnExpiredWindow verifies that use() triggers flushToDatabase when the refill window has passed and a depleted rate limit is refreshed from the database
|
Good call on the tests. I've added:
All 13 rate limit tests pass. |
Description
shouldRefill()in the v1 scheduler rate limiter had its condition inverted. It returnedtruewhennextRefillAtwas still in the future (window still open, no refill needed) — the opposite of when a refill is actually due.The bug
pkg/scheduling/v1/rate_limit.go:182:This returns
truewhile the refill time is still in the future. Compare with the guard insideflushToDatabase()(line 292), which no-ops whilenextRefillAt.After(now)and only does real work once the boundary has passed. The two are meant to agree;shouldRefill()got the negation wrong.Impact
Both failure modes land in
use()at line 118:After a window rolls over:
shouldRefill()reportsfalse, souse()never triggers the on-demand refresh. Tasks are evaluated against depleted counts from the previous window and rejected until the 1sloopFlushticker fires. Up to ~1s of spurious rate-limit rejections per window rollover, per tenant.Before the window rolls over (the common case):
shouldRefill()reportstrue, so everyuse()call runsflushToDatabase(), which takes three write locks (unflushedMu,dbRateLimitsMu,nextRefillAtMu) and immediately returns doing nothing.use()sits on the per-task scheduling path — constant write-lock churn for zero work.Fix
This aligns
shouldRefill()with theflushToDatabase()guard: returnstruewhennow >= nextRefillAt(refill is due),falsewhile the window is still open.Fixes #4346