fix(cloud): drop tasks past payload store retention

[grutt]

Description

Resolves the FIXME in the dispatcher's populateTaskData: when bulk input retrieval fails, the dispatcher previously requeued every task in the batch regardless of whether the error was transient (e.g. an S3 rate limit) or permanent (the payload no longer exists in the external store). Tasks whose offloaded payloads are permanently gone would loop forever through assign/requeue, never reaching a terminal state. Because these zombie tasks also hold GROUP_ROUND_ROBIN concurrency slots, they can block all other work behind the same concurrency keys and build up a large backlog for the tenant.

This PR introduces a sentinel error, repository.ErrPayloadNotFound. When the dispatcher sees this error from a bulk retrieval, it retries retrieval per task and splits the batch: tasks whose payloads are permanently missing are failed non-retryably with a descriptive error message.

A companion change in the cloud S3 external store wraps NoSuchKey errors on index file fetches (and external IDs missing from an index file) with the new sentinel; it depends on this PR being released first.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Checklist

Changes have been:

• Tested (unit, integration, or manually with steps specified)
• Linted and formatted
• Documented (where applicable)
• [] Added to CHANGELOG (where applicable) -- see Keep a Changelog

---

🤖 AI Disclosure

• I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.

• Details: Claude (Cursor agent) was used to investigate the production incident that motivated this change

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hatchet-docs	Ready	Preview, Comment	Jul 10, 2026 7:44pm

[github-actions]

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=true). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

[github-actions]

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=false). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

[mrkaye97]

do we do this kind of thing concurrently in other places? just checking to make sure we're being consistent, not sure off the top of my head

[mrkaye97]

this sketches me out a little - feels odd to be returning both a result and an error

[github-actions]

Benchmark results

goos: linux
goarch: amd64
pkg: github.com/hatchet-dev/hatchet/pkg/scheduling/v1
cpu: AMD Ryzen 9 7950X3D 16-Core Processor          
              │ /tmp/old.txt │         /tmp/new.txt          │
              │    sec/op    │    sec/op     vs base         │
RateLimiter-8    49.50µ ± 6%   57.57µ ± 21%  ~ (p=0.132 n=6)

              │ /tmp/old.txt │         /tmp/new.txt          │
              │     B/op     │     B/op      vs base         │
RateLimiter-8   137.7Ki ± 0%   137.7Ki ± 0%  ~ (p=0.630 n=6)

              │ /tmp/old.txt │          /tmp/new.txt          │
              │  allocs/op   │  allocs/op   vs base           │
RateLimiter-8    1.022k ± 0%   1.022k ± 0%  ~ (p=1.000 n=6) ¹
¹ all samples are equal

_{Compared against main (0511964)}