Skip to content
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@

### Highlights

- Tasks can declare a slot cost, so a task that needs more memory or CPU consumes more than one worker slot. See [Task Slot Cost](https://docs.hatchet.run/v1/advanced-assignment/slot-cost).
- Fixes timeouts in `UserSession` cleanup jobs by adding an index via a new `hatchet-migrate` migration.

## [0.90.13] - 2026-06-29
Expand Down Expand Up @@ -596,7 +597,7 @@
- Cleanup old workers via daily gocron by @mnafees in [#3663](https://github.com/hatchet-dev/hatchet/pull/3663)

### Removed
- Remove "account" dropdown', add notifications dropdown by @TehShrike in [#3365](https://github.com/hatchet-dev/hatchet/pull/3365)

Check warning on line 600 in CHANGELOG.md

View workflow job for this annotation

GitHub Actions / spelling

"Teh" should be "The".


## [0.83.33] - 2026-04-21
Expand Down Expand Up @@ -826,7 +827,7 @@

### Changed
- Move OTel tables to OLAP repo by @mnafees in [#3369](https://github.com/hatchet-dev/hatchet/pull/3369)
- Don't flicker the runs table when refetching by @TehShrike in [#3367](https://github.com/hatchet-dev/hatchet/pull/3367)

Check warning on line 830 in CHANGELOG.md

View workflow job for this annotation

GitHub Actions / spelling

"Teh" should be "The".


## [0.82.0] - 2026-03-23
Expand All @@ -853,7 +854,7 @@
- Add latency plots to load test by @juliusgeo in [#3259](https://github.com/hatchet-dev/hatchet/pull/3259)

### Changed
- Only return non-archived tenants in TenantMember queries by @TehShrike in [#3317](https://github.com/hatchet-dev/hatchet/pull/3317)

Check warning on line 857 in CHANGELOG.md

View workflow job for this annotation

GitHub Actions / spelling

"Teh" should be "The".


## [0.81.0] - 2026-03-18
Expand All @@ -877,9 +878,9 @@
## [0.80.8] - 2026-03-18

### Changed
- Guarantee that organization tenants will always be an array by @TehShrike in [#3316](https://github.com/hatchet-dev/hatchet/pull/3316)

Check warning on line 881 in CHANGELOG.md

View workflow job for this annotation

GitHub Actions / spelling

"Teh" should be "The".
- New organizations+tenants screen by @TehShrike in [#3198](https://github.com/hatchet-dev/hatchet/pull/3198)

Check warning on line 882 in CHANGELOG.md

View workflow job for this annotation

GitHub Actions / spelling

"Teh" should be "The".
- Improved invitation accept screen by @TehShrike in [#3151](https://github.com/hatchet-dev/hatchet/pull/3151)

Check warning on line 883 in CHANGELOG.md

View workflow job for this annotation

GitHub Actions / spelling

"Teh" should be "The".

### Removed
- Remove dispatch backlog, replace with timeout lock acquisition by @juliusgeo in [#3290](https://github.com/hatchet-dev/hatchet/pull/3290)
Expand Down Expand Up @@ -911,7 +912,7 @@
## [0.79.44] - 2026-03-15

### Added
- Add support for additional RBAC configruation via YAML configuration by @grutt in [#3285](https://github.com/hatchet-dev/hatchet/pull/3285)

Check warning on line 915 in CHANGELOG.md

View workflow job for this annotation

GitHub Actions / spelling

"configruation" should be "configurations" or "configuration".

### Changed
- Gracefully handle empty bulk scheduled deletes by @avirajkhare00 in [#3279](https://github.com/hatchet-dev/hatchet/pull/3279)
Expand Down Expand Up @@ -1045,7 +1046,7 @@
## [0.79.11] - 2026-02-27

### Added
- New "create organization" and "create tenant" interfaces by @TehShrike in [#3068](https://github.com/hatchet-dev/hatchet/pull/3068)

Check warning on line 1049 in CHANGELOG.md

View workflow job for this annotation

GitHub Actions / spelling

"Teh" should be "The".

### Fixed
- Modals should appear above the mobile sidebar by @TehShrike in [#3114](https://github.com/hatchet-dev/hatchet/pull/3114)
Expand Down
42 changes: 42 additions & 0 deletions examples/go/slot-cost/main.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

26 changes: 26 additions & 0 deletions examples/python/slot_cost/worker.py

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 19 additions & 0 deletions examples/typescript/slot_cost/workflow.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions frontend/docs/pages/reference/changelog/python.mdx
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
{/* AUTOGENERATED — do not edit. Run `task sync-changelog` to regenerate from sdks/python/CHANGELOG.md */}

## v1.34.0 - 2026-07-09

### Added

- Added `slot_cost` to the `hatchet.task` and `workflow.task` decorators, so a task that needs more memory or CPU can consume more than one worker slot and a worker runs fewer of them at once. On older engines it has no effect. See [Task Slot Cost](https://docs.hatchet.run/v1/advanced-assignment/slot-cost).

## v1.33.18 - 2026-07-08

### Fixed
Expand Down
6 changes: 6 additions & 0 deletions frontend/docs/pages/reference/changelog/typescript.mdx
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
{/* AUTOGENERATED — do not edit. Run `task sync-changelog` to regenerate from sdks/typescript/CHANGELOG.md */}

## v1.25.0 - 2026-07-09

### Added

- Added `slotCost` to task options, so a task that needs more memory or CPU can consume more than one worker slot and a worker runs fewer of them at once. Durable tasks ignore it, and on older engines it has no effect. See [Task Slot Cost](https://docs.hatchet.run/v1/advanced-assignment/slot-cost).

## v1.24.3 - 2026-06-17

### Removed
Expand Down
1 change: 1 addition & 0 deletions frontend/docs/pages/v1/advanced-assignment/_meta.js
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,6 @@ export default {
index: { display: "hidden" },
"sticky-assignment": "Sticky Assignment",
"worker-affinity": "Worker Affinity",
"slot-cost": "Task Slot Cost",
"manual-slot-release": "Manual Slot Release",
};
3 changes: 2 additions & 1 deletion frontend/docs/pages/v1/advanced-assignment/index.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Advanced Assignment
description: Sticky assignment, worker affinity, and manual slot release.
description: Sticky assignment, worker affinity, task slot cost, and manual slot release.
---

# Advanced Assignment
Expand All @@ -9,4 +9,5 @@ Fine-tune how tasks are assigned to workers.

- [Sticky Assignment](/v1/advanced-assignment/sticky-assignment) — Pin tasks to specific workers
- [Worker Affinity](/v1/advanced-assignment/worker-affinity) — Prefer specific workers
- [Task Slot Cost](/v1/advanced-assignment/slot-cost) — Set how many worker slots a task consumes
- [Manual Slot Release](/v1/advanced-assignment/manual-slot-release) — Control when slots free up
44 changes: 44 additions & 0 deletions frontend/docs/pages/v1/advanced-assignment/slot-cost.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
import { snippets } from "@/lib/generated/snippets";
import { Snippet } from "@/components/code";
import { Callout, Tabs } from "nextra/components";
import UniversalTabs from "@/components/UniversalTabs";

# Task Slot Cost

Every worker has a fixed number of slots that limit how many tasks it runs at once, set with the `slots` option on the worker and defaulting to 100. By default a task consumes one slot while it runs. Slot cost lets a task consume more than one slot, so a task that needs more memory or CPU takes up more of a worker's capacity and the worker runs fewer of them at the same time.

## Setting slot cost

Set the slot cost when you define a task. A cost of 5 means the task reserves five slots while it runs.

<UniversalTabs items={["Python", "Typescript", "Go"]}>
<Tabs.Tab title="Python">
<Snippet src={snippets.python.slot_cost.worker.slot_cost} />
</Tabs.Tab>
<Tabs.Tab title="Typescript">
<Snippet src={snippets.typescript.slot_cost.workflow.slot_cost} />
</Tabs.Tab>
<Tabs.Tab title="Go">
<Snippet src={snippets.go.slot_cost.main.slot_cost} />
</Tabs.Tab>
</UniversalTabs>

On a worker with 100 slots, the omega task at cost 5 and the weenie task at cost 1 draw from the same 100 slots. The worker runs at most 20 omega tasks, or 100 weenie tasks, or any mix whose costs sum to 100.

## How the reservation works

Slot capacity is local to a single worker. A task's slot cost is charged against the one worker that runs it, and a reservation cannot span two workers. A task with cost 5 needs a single worker with 5 free slots. It cannot take 3 slots from one worker and 2 from another, even when the worker pool has capacity in total.

Set the slot count on each worker to at least the largest slot cost you use. If a task's cost is greater than every worker's slot count, the task can never be scheduled. It waits in the queue until its [schedule timeout](../timeouts.mdx) and is then cancelled.

<Callout type="info">
Changing a task's slot cost changes its workflow version, so the next
registration creates a new version. Leaving the cost unset, or setting it to
1, keeps the task at one slot and does not change the version.
</Callout>

## Slot cost and concurrency limits

Slot cost and [concurrency limits](../concurrency.mdx) solve different problems. A concurrency limit caps how many runs of a key execute at once, and with a static key the limit applies across the whole worker pool. Slot cost does not limit the number of runs. It changes how much of a worker one run consumes.

If you want at most five heavy runs at once, use a concurrency limit. If you want each heavy run to take five times the worker capacity of a light run, use slot cost. The two can be combined on the same task.
7 changes: 7 additions & 0 deletions frontend/docs/pages/v1/concurrency.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,13 @@ You should primarily use concurrency control when you need to ensure fair access

Concurrency control also lets you limit the number of runs for a workflow globally, if you use a static CEL expression, such as `'global'`. This and [rate limiting](./rate-limits.mdx) are the recommended mechanisms for setting per-workflow throughput limits.

<Callout type="info">
Concurrency limits how many runs happen at once. To instead control how much
of a worker's capacity a single run consumes, so a heavy task takes up more
slots than a light one, see [Task Slot
Cost](./advanced-assignment/slot-cost.mdx).
</Callout>

## Available Strategies:

- [**Group Round Robin**](#group-round-robin) queues incoming task and workflow runs and only dispatches them to workers and triggers them once an available slot is open.
Expand Down
2 changes: 2 additions & 0 deletions frontend/docs/pages/v1/faq.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ If your workers are resource starved, there are basically two options:
1. Reduce the slot count, so the worker runs less work concurrently. This is a blunt instrument, in the sense that it doesn't let you _tune_ resources to the needs of the workload running on the worker. For instance, if you're using 100% of your memory but only 10% of your CPU, reducing the slot count will likely help the worker stay online, but you'll be significantly under-utilizing CPU. In this case, you can:
2. Reconfigure the specs of the machine the worker is running on. For instance, in the example above, you might be able to migrate from a CPU-optimized machine to a memory-optimized one, which will give you more efficient resource utilization across the board.

If only some of your tasks are heavy, [task slot cost](/v1/advanced-assignment/slot-cost) lets those tasks consume more than one slot, so the slot count can stay tuned for the light ones.

On the other hand, if your workers are underutilizing resources, your options are:

1. Increase the number of slots on them so they can pick up more work. This is especially helpful for heavily I/O bound tasks, which generally are spending most of their time waiting.
Expand Down
2 changes: 1 addition & 1 deletion frontend/docs/pages/v1/troubleshooting/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ If tasks remain in the `QUEUED` state and never move to `RUNNING`:

1. **No workers registered for the task** — check the Workers tab in the dashboard and confirm a worker is registered that handles the task name. If you recently renamed a task, make sure the worker has been restarted with the updated code.

2. **All worker slots are full** — if every slot is occupied by other tasks, new tasks will wait in the queue. Check worker utilization in the dashboard or increase the [slot count](/v1/workers#slots).
2. **All worker slots are full** — if every slot is occupied by other tasks, new tasks will wait in the queue. Check worker utilization in the dashboard or increase the [slot count](/v1/workers#slots). A task with a [slot cost](/v1/advanced-assignment/slot-cost) higher than every worker's slot count can never be scheduled, and is cancelled when its schedule timeout is reached.

3. **Concurrency or rate limit is blocking** — if you've configured [concurrency limits](/v1/concurrency) or [rate limits](/v1/rate-limits), tasks may be held back intentionally. Review your configuration.

Expand Down
2 changes: 2 additions & 0 deletions frontend/docs/pages/v1/workers.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -144,4 +144,6 @@ To set the slot count, pass the `slots` option when declaring the worker:

The default slot count for workers in Hatchet is 100. In many cases, leaving the default as-is will be perfectly fine, especially when first getting set up with Hatchet.

By default each running task consumes one slot. A task can be configured to consume more than one, so a task that needs more memory or CPU takes up more of a worker's capacity. See [Task Slot Cost](./advanced-assignment/slot-cost.mdx).

<Keywords keywords="worker, slots, registration, concurrency, horizontal scaling, autoscaling" />
4 changes: 4 additions & 0 deletions pkg/client/create/task.go
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,10 @@ type WorkflowTask[I, O any] struct {
Parents []NamedTask

DefaultPriority *int32

// (optional) SlotCost is the number of default worker slots this task consumes. Defaults to one.
// Durable tasks ignore it.
SlotCost *int32
}

type WorkflowOnFailureTask[I, O any] struct {
Expand Down
84 changes: 84 additions & 0 deletions pkg/repository/slot_cost_writepath_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
//go:build !e2e && !load && !rampup && !integration

package repository

import (
"context"
"testing"

"github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)

// The trigger v1_step_slot_request_insert_trigger inserts a one-slot row ({default: 1}, or
// {durable: 1} for a durable step) on Step insert. This checks that a registered slot cost
// overwrites that row and persists.
func TestSlotCostPersistsThroughWritePath(t *testing.T) {
pool, cleanup := setupPostgresWithMigration(t)
defer cleanup()

ctx := context.Background()
repo := newWorkflowTestRepository(pool)

// The overwrite only matters while this trigger exists, so require it to be present.
requireTriggerExists(ctx, t, pool, "v1_step_slot_request_insert_trigger")

desc := "slot-cost-writepath"
opts := &CreateWorkflowVersionOpts{
Name: "slot-cost-writepath",
Description: &desc,
Tasks: []CreateStepOpts{
{ReadableId: "heavy", Action: "integration:heavy", SlotRequests: map[string]int32{SlotTypeDefault: 5}},
{ReadableId: "light", Action: "integration:light", SlotRequests: map[string]int32{SlotTypeDefault: 1}},
{ReadableId: "plain", Action: "integration:plain"},
{ReadableId: "dur", Action: "integration:dur", IsDurable: true},
},
}

_, err := repo.PutWorkflowVersion(ctx, internalTenantId, opts)
require.NoError(t, err)

got := readSlotRequestsByStep(ctx, t, pool)

assert.Equal(t, map[string]int32{SlotTypeDefault: 5}, got["heavy"], "explicit slot cost 5 should persist")
assert.Equal(t, map[string]int32{SlotTypeDefault: 1}, got["light"], "explicit slot cost 1 should persist")
assert.Equal(t, map[string]int32{SlotTypeDefault: 1}, got["plain"], "no explicit cost stays at one default slot")
assert.Equal(t, map[string]int32{SlotTypeDurable: 1}, got["dur"], "durable default is unchanged")
}

func requireTriggerExists(ctx context.Context, t *testing.T, pool *pgxpool.Pool, name string) {
t.Helper()

var exists bool
err := pool.QueryRow(ctx, `SELECT EXISTS (SELECT 1 FROM pg_trigger WHERE tgname = $1)`, name).Scan(&exists)
require.NoError(t, err)
require.True(t, exists, "test premise requires the compatibility trigger %q to be present", name)
}

func readSlotRequestsByStep(ctx context.Context, t *testing.T, pool *pgxpool.Pool) map[string]map[string]int32 {
t.Helper()

rows, err := pool.Query(ctx, `
SELECT s."readableId", r.slot_type, r.units
FROM v1_step_slot_request r
JOIN "Step" s ON s.id = r.step_id
WHERE r.tenant_id = $1
`, internalTenantId)
require.NoError(t, err)
defer rows.Close()

out := map[string]map[string]int32{}
for rows.Next() {
var readableID, slotType string
var units int32
require.NoError(t, rows.Scan(&readableID, &slotType, &units))
if out[readableID] == nil {
out[readableID] = map[string]int32{}
}
out[readableID][slotType] = units
}
require.NoError(t, rows.Err())

return out
}
7 changes: 5 additions & 2 deletions pkg/repository/sqlcv1/workflows.sql
Original file line number Diff line number Diff line change
Expand Up @@ -322,8 +322,11 @@ SELECT
unnest(@units::integer[]),
CURRENT_TIMESTAMP,
CURRENT_TIMESTAMP
-- NOTE: ON CONFLICT can be removed after the 0_76_d migration is run to remove insert triggers added in 0_76
ON CONFLICT (tenant_id, step_id, slot_type) DO NOTHING;
-- The trigger v1_step_slot_request_insert_trigger writes a {default: 1} (or {durable: 1}) row on
-- Step insert, so DO UPDATE overwrites it with the requested units instead of leaving the default.
-- The conflict handling is only here because of that trigger.
ON CONFLICT (tenant_id, step_id, slot_type) DO UPDATE
SET units = EXCLUDED.units, updated_at = CURRENT_TIMESTAMP;

-- name: AddStepParents :exec
INSERT INTO "_StepOrder" ("A", "B")
Expand Down
7 changes: 5 additions & 2 deletions pkg/repository/sqlcv1/workflows.sql.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading