Skip to content

fix(engine): populate external ids for durable task children inside of the existing trigger tx#4344

Merged
mrkaye97 merged 1 commit into
mainfrom
mk/fix-durable-orphaned-child-bug
Jul 2, 2026
Merged

fix(engine): populate external ids for durable task children inside of the existing trigger tx#4344
mrkaye97 merged 1 commit into
mainfrom
mk/fix-durable-orphaned-child-bug

Conversation

@mrkaye97

@mrkaye97 mrkaye97 commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Description

Fixes an issue where we'd populate external ids (which also had the side effect of creating task events) in one tx, commit, and then begin a second tx to spawn the children. this meant that if we populated, committed, and then the second tx failed, we wouldn't roll back, and we'd get into a stuck state where we'd just hang because there would be a ghost id in the list of runs to skip, which would result in:

failed to get or create event log entries: expected to find log entry for skipped child task external id

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Checklist

Changes have been:

  • Tested (unit, integration, or manually with steps specified)
  • Linted and formatted
  • Documented (where applicable)
  • Added to CHANGELOG (where applicable) -- see Keep a Changelog

🤖 AI Disclosure
  • I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.
  • Details: Used Claude Code to help diagnose / narrow down the problem, wrote the fix myself

@vercel

vercel Bot commented Jul 2, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
hatchet-docs Ready Ready Preview, Comment Jul 2, 2026 4:43pm

Request Review

@github-actions github-actions Bot added the engine Related to the core Hatchet engine label Jul 2, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes a transactional consistency bug in durable task child-run triggering by ensuring external IDs (and the associated task events) are populated within the same database transaction as durable RUN event ingestion, preventing “ghost” skipped-child IDs when later steps fail and roll back.

Changes:

  • Wrap PopulateExternalIdsForWorkflow in an explicit transaction and introduce a tx-aware internal helper.
  • Thread the existing ingestion transaction into child external-id generation so task events and ingestion outcomes commit/rollback together.
  • Remove the pre-ingestion external-id population call from the dispatcher path, relying on ingestion to do it atomically.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
pkg/repository/ids.go Introduces tx-scoped external-id population and passes the ingestion tx through child ID generation.
pkg/repository/durable_events.go Populates external IDs inside IngestDurableTaskEvent’s existing transaction for RUN events.
internal/services/dispatcher/server_v1.go Removes redundant pre-ingestion external-id population so ingestion is the single atomic path.

Comment on lines +1067 to +1069
if populateErr := r.populateExternalIdsForWorkflow(ctx, tx, tenantId, opts.TriggerRuns.TriggerOpts); populateErr != nil {
return nil, fmt.Errorf("failed to populate external ids for workflow: %w", populateErr)
}
@mrkaye97 mrkaye97 merged commit 728b3ff into main Jul 2, 2026
56 checks passed
@mrkaye97 mrkaye97 deleted the mk/fix-durable-orphaned-child-bug branch July 2, 2026 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine Related to the core Hatchet engine

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants