Skip to content

Quick start never creates or mentions a WorkerPool, so the first Task fails with ResourceExhausted #437

Description

@ilaigold

Summary

AX runs each Task as a Substrate actor, and Substrate needs a WorkerPool with a free worker in the Task's atespace. Nothing in the quick start creates or mentions one:

  • make deploy does not create one.
  • The README and docs/ do not mention one.
  • Substrate's hack/install-ate.sh --deploy-ate-system does not create one either; only its demos do.

So on a fresh install, the first Task fails:

Phase:  Failed
Ready   False   ActorResumeFailed   resuming actor default/validate2: rpc error: code = ResourceExhausted desc = no free workers available

Because of #367, the Task then stays Failed even after a pool is added.

Reproduced on

  • AX v0.3.1 (e70162a) with Agent Substrate v0.2.0, on kind (Docker Desktop, arm64), 2026-09-28.
  • Earlier, AX v0.3.0 on GKE Standard 1.36.3-gke.1767000.

main (ac23328) still has no WorkerPool manifest under deploy/ or examples/, and the README and docs/ do not mention one.

Workaround

With this pool in the Task's atespace, a new Task ran and ax ssh worked. It is adapted from Substrate's demos/counter/counter.yaml.tmpl and applied with ko apply from a Substrate checkout:

apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
  name: ax-pool
  namespace: default   # the Task's atespace
spec:
  replicas: 2
  workerImage: ko://github.com/agent-substrate/substrate/cmd/ateom-gvisor
  template:
    nodeSelector:
      ate.dev/substrate-version: "v0.2.0"   # must match the label the install puts on the nodes
    resources:
      limits: { cpu: "1", memory: 1Gi }
      requests: { cpu: 250m, memory: 1Gi }

Suggestion

  • Add a minimal WorkerPool step between "Deploy the control plane" and "Run your first task", or have make deploy create a small default pool.
  • Mention the ate.dev/substrate-version node label. A pool whose nodeSelector does not match it never schedules its workers, and the Task fails with the same ResourceExhausted.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions