Skip to content

fix--rate-limit-upsert-deadlock#4392

Open
grutt wants to merge 1 commit into
mainfrom
fix--rate-limit-upsert-deadlock
Open

fix--rate-limit-upsert-deadlock#4392
grutt wants to merge 1 commit into
mainfrom
fix--rate-limit-upsert-deadlock

Conversation

@grutt

@grutt grutt commented Jul 9, 2026

Copy link
Copy Markdown
Member

Description

The queue loop re-upserted dynamic rate limit definitions on every tick without the per-tenant advisory lock that UpdateRateLimits uses, causing frequent deadlock detected (40P01) errors against the RateLimit table. Upserts now follow the same lock discipline and skip no-op writes entirely.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

What's Changed

  • Acquire the per-tenant advisory lock before UpsertRateLimitsBulk, matching the lock ordering in UpdateRateLimits
  • Run the upsert on the caller's optimistic transaction when present; otherwise open a short dedicated transaction
  • Cache the last-written (limitValue, window) per key (5 min TTL) and skip upserts for unchanged definitions, removing per-tick write churn

Checklist

Changes have been:

  • Tested (unit, integration, or manually with steps specified)
  • Linted and formatted
  • Documented (where applicable)
  • Added to CHANGELOG (where applicable) -- see Keep a Changelog

🤖 AI Disclosure
  • I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.

  • Details: Cursor (Claude) used to diagnose the deadlock.

@vercel

vercel Bot commented Jul 9, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hatchet-docs Ready Ready Preview, Comment Jul 9, 2026 7:23pm

Request Review

@github-actions github-actions Bot added the engine Related to the core Hatchet engine label Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine Related to the core Hatchet engine

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant