Add basic per-actor external volume flow by msau42 · Pull Request #405 · agent-substrate/substrate

Michelle Au (msau42) · 2026-07-08T01:01:24Z

Initial part of #232.

Extends the ActorTemplate API to add support for per-actor external volumes and adds control plane and atelet hooks following the Actor lifecycle.

Callouts to volume operations are abstracted with a Volume interface. Right now, only a mock volume plugin for testing has been implemented, but we will add CSI support next.

Tests pass
Appropriate changes to documentation are included in the PR - Not going to update documentation until we add CSI support.

Michelle Au (msau42) · 2026-07-08T15:46:21Z

+	Capacity resource.Quantity `json:"capacity" protobuf:"bytes,1,opt,name=capacity"`
+	// storageClassName refers to the StorageClass to create the volume from.
+	// +required
+	StorageClassName string `json:"storageClassName" protobuf:"bytes,2,opt,name=storageClassName"`


open for discussion: can we reference k8s objects or should we create our own api?

I don't think there's a cleaner way. In theory we could have an ateapi level api which just acts as an indirect, but I don't see a reason at this moment.

Michelle Au (msau42) · 2026-07-08T15:48:09Z

  map<string, string> match_labels = 1;
 }

+message ExternalVolume {


discuss: should ExternalVolume be its own standalone object rather than embedded fields into the Actor?

Right now ExternalVolume is tied to an Actor but we are also thinking about the scenario where you want to share a volume across multiple actors.

I vote 👍

We all know how tricky volumes can be in k8s, with RWO, RWX, RO, mounts on the same node being virtually RWX but not ACTUALLY RWX, etc. If we make this a first-class resource, do we need to track the same sorts of things (e.g. NFS can be RWX, blockdev can't)?

Yes, we will need to add the notion of access modes and enforce that shared volumes can only be RWX or ROX.

Michelle Au (msau42) · 2026-07-08T15:56:14Z

 	}
 	err = s.persistence.CreateActor(ctx, actor)
 	if err != nil {
+		// Cleanup created volumes if DB write fails


Discuss: In general, how should we approach errors when an operation only partially succeeds? Should we rollback all the partially completed operations? What happens if the rollback fails?

Or we leave the system in the partial state and require users to cleanup by calling Delete? Even then, if deletion fails and Create is called again, we could still have partially leftover resources from a previous invocation.

My current thoughts:

If there is a pair of "Create" and "Delete" operations, then we can leave the partial state and clean it up on the Delete. This does imply we will have to add some "Creating" and possibly "Deleting" state to Actor before we start creating resources for it. I already have marked that as a todo.

For operations like "Resume", there isn't a corresponding delete operation if the resume failed. It's possible resume could be called on a different node. That means we could have partial state left on the first node. In those cases we can try to rollback the state as much as possible but that would still be a best effort operation. This has a few implications:

We probably need to have some process on the node that monitors oprhaned resources and tries to clean them up and/or alert

We need to make sure these orphaned resources don't get reused if a new actor with the same name gets created. This potentially implies that we should have some sort of actor uuid and be using that in our actor directories.

Tim Hockin (thockin)

I didn't get too far into the reading of this yet, but wanted to send a couple questions before EOD

Tim Hockin (thockin) · 2026-07-09T00:19:39Z

+	Capacity resource.Quantity `json:"capacity" protobuf:"bytes,1,opt,name=capacity"`
+	// storageClassName refers to the StorageClass to create the volume from.
+	// +required
+	StorageClassName string `json:"storageClassName" protobuf:"bytes,2,opt,name=storageClassName"`


I don't think there's a cleaner way. In theory we could have an ateapi level api which just acts as an indirect, but I don't see a reason at this moment.

Tim Hockin (thockin) · 2026-07-09T00:27:07Z

+
+// VolumePlugin abstracts storage operations.
+type VolumePlugin interface {
+	CreateVolume(ctx context.Context, name string, capacity string, storageClass string) (volumeID string, err error)


Do you see this as "kubernetes PVC is the plugin", or "CSI is the plugin", or "NFS is the plugin" ?

What I really want to understand is what credentials are granted to the API server to create and destroy volumes, and what additional layers are in between. E.g. if I need to create a million k8s PVCs and PVs, that's trouble, but if I need to give ateapi binaries the ability to delete EBS volumes in the same context as k8s, that's also trouble.

My thought is that substrate will directly call CSI grpc APIs (so ateapi needs to be authorized to communicate with CSI drivers). We will not be creating PVC objects, although we will need to have a substrate API reference to the volume IDs (which I have added in this PR).

Michelle Au (msau42) · 2026-07-09T01:09:45Z

 			Type:        "bind",
-			Source:      ateompath.DurableDirVolumeMountPoint(atespace, actorID, vm.GetName()),
+			Source:      srcPath,
+			Options:     []string{"bind", "rw"},


double check this. durabledir didn't set any options.

Tim Hockin (thockin)

I read this. It seems broadly correct. I am still worried about permissions, but not enough to block this.

I swear I had another comment, and it was important, but I forgot what it was. Oh well, Friday. :)

Tim Hockin (thockin) · 2026-07-10T15:30:46Z

+	// for each actor. The volume only lives as long as the actor and is deleted
+	// when the actor is deleted.
+	// +optional
+	ExternalVolumeTemplate *ExternalVolumeTemplate `json:"externalVolumeTemplate,omitempty" protobuf:"bytes,3,opt,name=externalVolumeTemplate"`


Do we need the word "Template" in here? All of these are templates

Open to naming, I am trying to distinguish between volumes that will trigger dynamic provisioning, and volumes that won't. So for shared volumes, for instance, I am thinking we could have something like:

volumes: externalVolumeTemplate: - name: my-own-volume storageClassName: class1 externalVolume: - name: shared-volume volumeName: someExistingVolume (in the same atespace) # This volume is already created and populated with data through other means

Eitan Yarmush (EItanya) · 2026-07-12T19:58:51Z

+	// for each actor. The volume only lives as long as the actor and is deleted
+	// when the actor is deleted.
+	// +optional
+	ExternalVolumeTemplate *ExternalVolumeTemplate `json:"externalVolumeTemplate,omitempty" protobuf:"bytes,3,opt,name=externalVolumeTemplate"`


What exactly does External mean here, External to substrate? If we are just wrapping a StorageClass anyway should we name it PersistentVolumeSource to be closer to the source material?

It's not REALLY a PV in k8s, it's just reusing k8s storage classes

Open to naming :)

But agree with Tim that this doesn't have quite the same semantics as PVs in k8s (and even the "persistent" part in k8s is misleading sometimes). At least with the per-actor volumes, the volume is definitely not persistent in the traditional sense (it only lives as long as the actor).

By "External", I'm trying to distinguish with the sandbox-backed filesystems, like rootfs and durabledir. The volumes in this case are backed by storage systems external to substrate (ie nfs, remote block, pfs, distributed fs, etc).

Eitan Yarmush (EItanya) · 2026-07-12T19:59:53Z

  map<string, string> match_labels = 1;
 }

+message ExternalVolume {


I vote 👍

This adds a new externalVolumeTemplate where users specify the capacity, storage driver and storage properties they would like via StorageClass. When an actor is created, we will also provision a volume for the actor according to the template. The volume is deleted when the actor is deleted.

Adds external volumes to the Actor object and integrates control plane Actor workflows with the corresponding volume operations: CreateActor -> CreateVolume DeleteActor -> DeleteVolume ResumeActor -> AttachVolume Pause/Suspend Actor -> DetachVolume

Volume operations are integrated into the actor workflows: - Run/Restore: MountVolume - Checkpoint: UnmountVolume Volumes are mounted under the actor directory and passed as bind mounts to OCI.

This allows for different storage backends to be supported. Right now it only contains the mock volume plugin, which is only intended for testing purposes right now.

It makes the file counter path configurable so we can test it against different volume types. And adds an optional file verification step to test the presence of a file on an existing volume.

Benjamin Elder (BenTheElder)

[AI First Pass Experiment]

Benjamin Elder (BenTheElder) · 2026-07-22T22:58:52Z

+	return mounted
+}
+
+func actorVolumeID(ref *ateapipb.ObjectRef, volumeName string) string {


🟡 🤖 The hyphen join is ambiguous: atespace, actor name, and volume name may all contain hyphens, so distinct actors can produce the same ID (actor a in atespace x-y vs actor y-a in atespace x). This string is the idempotency name passed to CreateVolume, and CSI dedupes by name — colliding actors would silently share one storage volume. The existing TODO about actor UUID would fix it; an unambiguous encoding of the three parts would too.

Benjamin Elder (BenTheElder) · 2026-07-22T22:58:52Z

+	}
+
+	// Delete associated volumes (TODO: best effort?)
+	s.deleteActorVolumes(ctx, req.GetActor(), actor.GetActorVolumes())


🟡 🤖 Two issues with volume deletion here. Volumes are never detached first — for an actor deleted while running or paused, CSI DeleteVolume on an attached volume fails. And since failures are only logged before the record is deleted below, the volume leaks with no record pointing at it. Consider detaching first (reuse detachActorVolumes) and deleting the record only after volume deletion succeeds, or persisting a deletion intent for background retry.

This is an interesting question. While the state machine here (#119) does allow delete from SUSPENDED, PAUSED and CRASHED states, I believe for pause/crash, there's probably still some cleanup that has to happen from atelet, and that workflow is currently missing today. Dmitry Berkovich (@dberkov) do we plan to add this workflow?

Benjamin Elder (BenTheElder) · 2026-07-22T22:58:52Z

+	// for each actor. The volume only lives as long as the actor and is deleted
+	// when the actor is deleted.
+	// +optional
+	ExternalVolumeTemplate *ExternalVolumeTemplate `json:"externalVolumeTemplate,omitempty"`


🟡 🤖 externalVolumeTemplate needs the same microvm gate DurableDir has (the CEL rule below only checks durableDir). Nothing in the microvm path handles VOLUME_TYPE_EXTERNAL: atelet writes a bind mount whose source is a host path that doesn't exist inside the guest (only the bundle rootfs is shared over virtio-fs), so container creation fails at runtime instead of admission.

Michelle Au (msau42) commented Jul 8, 2026

View reviewed changes

Michelle Au (msau42) force-pushed the mock-mount-dev branch from 3a4a327 to ccc7a2d Compare July 8, 2026 17:05

Michelle Au (msau42) changed the title ~~Add basic external volume flow~~ Add basic per-actor external volume flow Jul 8, 2026

Michelle Au (msau42) force-pushed the mock-mount-dev branch 2 times, most recently from c2286ea to 96fa815 Compare July 8, 2026 22:50

Haven Xia (HavenXia) reviewed Jul 9, 2026

View reviewed changes

Comment thread cmd/ateapi/internal/controlapi/volumes.go Outdated

Tim Hockin (thockin) reviewed Jul 9, 2026

View reviewed changes

Michelle Au (msau42) commented Jul 9, 2026

View reviewed changes

Zheng Lammerts (zlammerts-svg) added this to Substrate Milestones (go/substrate-milestones) Jul 9, 2026

github-project-automation Bot moved this to Todo in Substrate Milestones (go/substrate-milestones) Jul 9, 2026

Zheng Lammerts (zlammerts-svg) removed this from Substrate Milestones (go/substrate-milestones) Jul 9, 2026

Michelle Au (msau42) force-pushed the mock-mount-dev branch 4 times, most recently from 4890a34 to ffc96b8 Compare July 10, 2026 18:02

Julian Gutierrez Oschmann (juli4n) reviewed Jul 10, 2026

View reviewed changes

Comment thread pkg/api/v1alpha1/actortemplate_types.go Outdated

Tim Hockin (thockin) reviewed Jul 11, 2026

View reviewed changes

Eitan Yarmush (EItanya) reviewed Jul 12, 2026

View reviewed changes

Michelle Au (msau42) force-pushed the mock-mount-dev branch from ffc96b8 to 97b9861 Compare July 22, 2026 05:10

Michelle Au (msau42) force-pushed the mock-mount-dev branch from 97b9861 to 33fbae5 Compare July 22, 2026 21:22

ateapi handling of external volumes.

3aa87ee

Adds external volumes to the Actor object and integrates control plane Actor workflows with the corresponding volume operations: CreateActor -> CreateVolume DeleteActor -> DeleteVolume ResumeActor -> AttachVolume Pause/Suspend Actor -> DetachVolume

Michelle Au (msau42) force-pushed the mock-mount-dev branch from 33fbae5 to ec684cf Compare July 22, 2026 21:46

Dmitry Berkovich (dberkov) self-requested a review July 22, 2026 22:13

Michelle Au (msau42) added 4 commits July 22, 2026 22:46

atelet handling of external volumes.

bf75edb

Volume operations are integrated into the actor workflows: - Run/Restore: MountVolume - Checkpoint: UnmountVolume Volumes are mounted under the actor directory and passed as bind mounts to OCI.

Allow ateom to see host mounts

4c1dda5

Volume plugin interface and mock volume implementation.

1e444e7

This allows for different storage backends to be supported. Right now it only contains the mock volume plugin, which is only intended for testing purposes right now.

Update counter demo with external volume using mock volume plugin.

90ad7f1

It makes the file counter path configurable so we can test it against different volume types. And adds an optional file verification step to test the presence of a file on an existing volume.

Michelle Au (msau42) force-pushed the mock-mount-dev branch from ec684cf to 90ad7f1 Compare July 22, 2026 22:46

Benjamin Elder (BenTheElder) reviewed Jul 22, 2026

View reviewed changes

Uh oh!

Conversation

Michelle Au (msau42) commented Jul 8, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michelle Au (msau42) Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Tim Hockin (thockin) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Tim Hockin (thockin) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michelle Au (msau42) Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Michelle Au (msau42) Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Michelle Au (msau42) Jul 8, 2026 •

edited

Loading

Michelle Au (msau42) Jul 21, 2026 •

edited

Loading

Michelle Au (msau42) Jul 23, 2026 •

edited

Loading