Add idempotency key reclaim + tests

[colonelpanic8]

Summary

• Add optional idempotency key field to workflow trigger APIs/clients
• Implement idempotency key reclaim on terminal status with configurable TTL and recheck interval
• Track last_denied_at and clean up keys on terminal status updates
• Add migration for last_denied_at plus expanded idempotency tests (including concurrency/mixed-key cases)
• Update flake dev shell with lint tooling

Testing

• CGO_ENABLED=0 go test ./pkg/repository -run Idempotency -count=1
• CGO_ENABLED=0 go test ./... -count=1
• nix develop -c golangci-lint run ./... --config .golangci.yml --new-from-rev=origin/main

Notes

• Requires migration 20260205120000_v1_0_76.sql before deploying server changes.

[vercel]

@colonelpanic8 is attempting to deploy a commit to the Hatchet Team on Vercel.

A member of the Team first needs to authorize it.

[promptless-for-oss]

📝 Documentation updates detected!

New suggestion: Document idempotency keys feature for workflow triggers

---

Tip: Leave inline comments with @Promptless on suggestion diffs in the Promptless dashboard for targeted refinements 💬

[Copilot]

Pull request overview

This PR extends workflow triggering to support idempotency keys end-to-end, adds server-side “reclaim” behavior for keys stuck on terminal workflow runs (with configurable TTL/recheck throttling), and expands repository tests/migrations to support the new last_denied_at tracking.

Changes:

• Add optional idempotency_key/idempotencyKey to workflow trigger APIs and SDK clients (Go/TS/Python) and propagate it to trigger options.
• Implement idempotency key reclaim + deny throttling (last_denied_at) in the Go repository, plus controller-side cleanup on terminal status updates.
• Add migration/schema updates and expand repository tests (including concurrency and mixed-key scenarios); add Nix dev shell tooling.

Reviewed changes

Copilot reviewed 50 out of 51 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
sql/schema/v1-core.sql	Adds last_denied_at column to idempotency key table for deny throttling.
cmd/hatchet-migrate/migrate/migrations/20260205120000_v1_0_76.sql	Migration to add/drop last_denied_at.
api-contracts/workflows/workflows.proto	Adds idempotency_key to v0 trigger request.
api-contracts/v1/workflows.proto	Adds idempotency_key to v1 trigger request.
pkg/repository/trigger.go	Implements key creation/claiming + reclaim-on-terminal + duplicate handling plumbing.
pkg/repository/shared.go / pkg/repository/repository.go / pkg/config/*	Threads new TTL/recheck config through repository construction and server config.
pkg/repository/sqlcv1/*	Adds new SQL queries for OLAP status reads, terminal fallback, and idempotency key operations/fields.
internal/services/controllers/olap/*	Deletes idempotency keys on terminal task/DAG status updates.
internal/services/admin/*	Accepts idempotency key in admin trigger APIs; maps duplicate-key to AlreadyExists.
sdks/typescript/*	Adds idempotencyKey in TS types/clients and regenerates protos; improves protoc generation script.
sdks/python/*	Adds idempotency_key plumbing in client and regenerates protos/stubs.
pkg/repository/idempotency_trigger_test.go / pkg/repository/*test*	Adds extensive idempotency tests and improves Postgres test harness reuse.
flake.nix / flake.lock	Adds Nix dev shell with Go/protoc/lint tooling.

Comments suppressed due to low confidence (1)

pkg/repository/trigger.go:2180

• idempotencyKeyToExternalIds is a map keyed only by idempotency key, so if the incoming opts slice contains the same idempotency key more than once, earlier entries are silently overwritten. Later, those overwritten opts will be skipped because their (key, externalId) pair won’t be present in keyClaimantPairToWasClaimed, but the key also won’t appear in duplicateKeys/denyUpdateKeys, so the caller gets a partial trigger with no explicit signal. Consider detecting duplicate idempotency keys within the same request and either (a) returning a deterministic error, or (b) including them in duplicateKeys (and logging/updating last_denied_at) so the behavior is observable.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[promptless-for-oss]

📝 Documentation updates detected!

New suggestion: Document idempotency keys feature for workflow triggers

---

Tip: Enable auto-create PR in your Docs Collection to publish suggestions automatically 🤖

[gregfurman]

Hey @colonelpanic8 👋 Apologies for the delay here. We're currently in the middle of a release week but should have more capacity next week to give this a review

[mrkaye97]

hi @colonelpanic8 - closing this as folded into #4045. feel free to take a look there and let me know if you have thoughts!

[colonelpanic8]

hi @colonelpanic8 - closing this as folded into #4045. feel free to take a look there and let me know if you have thoughts!

cool. how similar is the interface to what I had originally?

do you know when this will get merged?

[mrkaye97]

hi @colonelpanic8 - closing this as folded into #4045. feel free to take a look there and let me know if you have thoughts!

cool. how similar is the interface to what I had originally?

do you know when this will get merged?

It's somewhat similar, but there are important differences. One big one is that idempotency is configured on the task / workflow directly via a CEL expression that uses the input or additional metadata, as opposed to being passed at runtime (as this wouldn't work for events, webhooks, etc.). Feel free to take a look through the docs and examples there! Hoping to merge in the next day or two.

[colonelpanic8]

@mrkaye97 Thanks for pointing me to #4045. I took a detailed look at the current implementation, and I’m pretty disappointed that this PR was closed as “folded into” #4045, because the behavior in #4045 does not actually subsume the behavior I implemented here.

The distinction is key derivation versus key lifetime. #4045 adds a useful workflow-level mechanism for deriving a key from a CEL expression, but it implements fixed-window deduplication:

• the key is claimed with expires_at = now + ttl;
• an existing claim becomes claimable only when expires_at <= now;
• cleanup only removes expired keys;
• IdempotencyConfig exposes only expression and ttl;
• terminal task/DAG status updates do not release the claim;
• there is no per-trigger release, bypass, or lifetime override.

What I need is active-execution idempotency:

1. Reject another trigger with the same key while the claiming workflow run is non-terminal.
2. Release/reclaim that key when the run reaches COMPLETED, FAILED, or CANCELLED.
3. Keep a TTL only as a safety net for an orphaned/stuck claim or a missed terminal update.

Those semantics cannot be reproduced by choosing a different fixed TTL. For example, suppose run A claims key K at 00:00, runs for two minutes, and completes at 00:02. A trigger at 00:01 should be rejected, but a trigger at 00:03 should be allowed immediately. With a 24-hour TTL, #4045 rejects the legitimate 00:03 trigger for almost a day. With a one-minute TTL, it can admit a duplicate while A is still running. No fixed TTL provides “for the lifetime of the active execution.”

This matters concretely for Railbird. A streaming ProcessVideo run can intentionally finish/suspend and later needs to resume the same processing record with the same logical key. Failure and recovery paths likewise need to enqueue again as soon as the previous run is terminal, while still preventing concurrent duplicate execution. Making the application manufacture a new nonce or infer Hatchet’s run state is not equivalent: it moves lifecycle coordination out of Hatchet and introduces opportunities for both lost retries and duplicate runs.

There are things I like in #4045: workflow-level CEL expressions, broader trigger-path coverage, and returning the existing run ID on collision are all improvements. But those are orthogonal to whether a claim is fixed-TTL or released on terminal status. I don’t think it is accurate to describe #2950 as folded into #4045 while the central lifecycle behavior is absent.

After reviewing #4045’s current structure, this also does not look architecturally difficult to support as an opt-in policy. One possible design would be:

• extend the workflow idempotency config with a lifecycle policy, e.g. FIXED_TTL versus UNTIL_TERMINAL_WITH_TTL_FALLBACK (or a release_on_terminal flag);
• persist that policy with the claim;
• when a standalone task or DAG reaches a terminal status, delete the claim only when tenant, key, and claimed_by_external_id still match;
• on a collision, optionally recheck the claimant’s terminal status and reclaim atomically, covering lag or failure in asynchronous terminal cleanup;
• retain the TTL as the fallback;
• test task and DAG completion/failure/cancellation plus the race where an old run terminates after an expired key has already been claimed by a newer run.

The terminal task/DAG hooks used by this PR still exist, and #4045 already records the claimant external ID and propagates the evaluated idempotency key onto runs, so the two designs appear compatible rather than mutually exclusive. The SQL must condition deletion on the claimant ID so an old run cannot delete a newer claim after TTL-based reuse.

Would you be open to preserving this as an idempotency lifecycle option in #4045 or accepting a focused follow-up PR that layers terminal release/reclaim onto the new expression-based design? This behavior is the reason I implemented #2950, and fixed-window deduplication does not meet that requirement.