[Question] Why retain container semantics inside a microVM sandbox?

[flyflypeng]

Background

CubeSandbox already uses a microVM as its isolation boundary. However, container-oriented concepts still appear throughout the internal architecture:

• CubeShim bridges containerd Shim v2 with the sandbox VM lifecycle.
• The guest-agent protocol exposes operations such as CreateContainer, StartContainer, and RemoveContainer.
• The control-plane model retains PodSandbox and container-oriented concepts, including cases where a type=sandbox container represents the sandbox itself.

This architecture is understandable for a Kubernetes/CRI runtime. Kubernetes requires the PodSandbox and Container abstractions, and VM-backed runtimes such as Kata Containers need to map those concepts onto a virtual machine.

For an agent sandbox platform, however, the actual isolation and lifecycle unit is the microVM sandbox. Even if CubeAPI and the SDKs expose a Sandbox to users, much of the internal lifecycle still appears to be modeled as containers inside that sandbox.

Platforms such as E2B and AgentENV appear to place the micro-sandbox itself closer to the center of both the user-facing and internal domain models. This comparison concerns the abstraction boundary rather than suggesting that their implementations are otherwise identical.

Open question

What are the architectural reasons for retaining container semantics inside CubeSandbox after the sandbox has already become a microVM?

More specifically:

1. Is the container layer primarily required for containerd/CRI and Kubernetes ecosystem compatibility?
2. Is supporting multiple independently managed containers inside one microVM an intentional product capability or long-term direction?
3. Which responsibilities still require a container identity and lifecycle rather than a sandbox-level workload or process abstraction?
4. Is the container model considered part of CubeSandbox's core domain, or is it mainly a compatibility layer inherited from its Kubernetes/runtime lineage?
5. Would the architecture still retain container concepts if Kubernetes and CRI compatibility were not requirements?

Possible architectural direction

Would it be cleaner to make a microVM-native Sandbox the core abstraction and move Kubernetes/container semantics into an optional adapter layer?

This would not necessarily mean dropping OCI image support. OCI could remain an image and workload packaging format without requiring Container to remain a first-class lifecycle or identity domain throughout the system.

A microVM-native core model could make the following boundaries easier to understand:

• isolation boundary;
• lifecycle and ownership;
• snapshot and restore boundary;
• API identity;
• resource accounting;
• failure handling and cleanup.

It could also prevent a general-purpose agent sandbox from remaining coupled to the secure-container abstraction historically used to integrate VM runtimes with Kubernetes.

The purpose of this issue is to understand the intended abstraction boundary and architectural direction before proposing any concrete refactoring.

[cubesandboxbot]

Thanks for the detailed architectural question — it's clearly written and follows the Question template.

---

This issue might be related to existing issues. Please check:

• [Feature Request] Add pure MicroVM model support while retaining MicroVM container compatibility #179: "[Feature Request] Add pure MicroVM model support while retaining MicroVM container compatibility" (closed) — raises the same container-vs-microVM domain model question, comparing CubeSandbox's MicroVM container model with E2B's pure MicroVM model and Kata container ecosystem compatibility.
• [Feature Request] Refactor control plane API architecture: unify protocol definitions, eliminate redundant types, centralize validation #1023: "[Feature Request] Refactor control plane API architecture: unify protocol definitions, eliminate redundant types, centralize validation" (open) — a related architecture discussion about restructuring the control plane, though focused on protocol/type redundancy rather than container semantics.

[cubesandboxbot]

---

This issue might be related to an existing issue. Please check:

• [Feature Request] Add pure MicroVM model support while retaining MicroVM container compatibility #179: [Feature Request] Add pure MicroVM model support while retaining MicroVM container compatibility — also discusses the container-vs-pure-microVM abstraction boundary and references E2B's pure-microVM model. It is closed, but its discussion may be relevant context for your question.

[cubesandboxbot]

This issue might be a duplicate of existing issues. Please check:

• [Feature Request] Add pure MicroVM model support while retaining MicroVM container compatibility #179: [Feature Request] "Add pure MicroVM model support while retaining MicroVM container compatibility" — a closed feature request covering the same architectural topic: whether CubeSandbox should keep its container layer inside the microVM (for K8s/Kata ecosystem compatibility) or move toward a pure microVM-native model. It makes the same E2B comparison and discusses heterogeneous OS / LibOS support. Maintainers responded there that this direction is not currently on the roadmap, so that prior discussion is worth referencing before this question goes further.

[kinwin-ustc]

This is a legacy architecture design that is compatible with the container ecosystem and use cases, and can also introduce a single VM with multiple containers (a common approach for Kubernetes pods).

In today's agent-based scenarios, we can consider a single MicroVM as a single container with a completely identical lifecycle. Retaining the container within the VM is merely a way of presenting resources. However, it seems that providing sandbox abstraction through containers within the VM still offers some advantages. We have more room for environment preparation and business logic extension (outside the container's namespace). These actions are completely invisible to the agent itself, providing further security and logical isolation. We can also leverage the capabilities provided by cgroups/namespaces to assist us in many tasks. Without this layer, this logic might need to be performed in envd, completely visible to the agent within the sandbox. Furthermore, the sandbox directly interacting with virtualized hardware and channels would introduce a greater security burden. In fact, the overhead outside the container is minimal. The VM rootfs is highly streamlined; the VM only starts a cube-agent (as the OS's first process) to prepare the container's runtime environment and then launch the container. This overhead is negligible.

Therefore, we currently retain this design and believe it will not have any impact. However, if we need to support non-Linux sandbox environments in the future, this design may need to be re-evaluated.

[flyflypeng]

I agree with your point that the MicroVM + container combination provides better extensibility and security. For example, if we add observability-related processes inside the MicroVM, it would be better to isolate them from the sandbox environment exposed to the Agent, so that they are mutually invisible.

However, from the perspective of the Agent Sandbox use case and its users, what users interact with are the management APIs for sandboxes and templates, along with the in-sandbox data-plane Process and Filesystem APIs exposed through envd. The original container-oriented Shim v2 interface could be deprecated; there is no need to preserve compatibility with it.

Also, while analyzing the CubeSandbox source code related to cold-start MicroVM snapshots, I found a design that does not seem very elegant. When Cubelet creates a cold-start MicroVM snapshot, it records the containerID in the snapshot metadata. Suppose the container ID in the current MicroVM is A. Later, when a sandbox is created from that snapshot, the CreateTask API inserts a new container ID, B. This then requires a series of ID mappings and translations, with cube-agent reconnecting to and restoring the original container instance A.

To me, this is an architectural issue caused by retaining the container concept in the management components. A cleaner approach would be to let cube-agent start an embedded container inside the MicroVM based on an OCI Spec, while avoiding the management of any container concept outside the MicroVM.

Therefore, my suggestion is that, in the long term, the management components on the data-plane node could be refactored. The current cubelet + containerd + containerd-shimv2 stack could be consolidated into a management component centered on MicroVM sandboxes.

[kinwin-ustc]

To me, this is an architectural issue caused by retaining the container concept in the management components. A cleaner approach would be to let cube-agent start an embedded container inside the MicroVM based on an OCI Spec, while avoiding the management of any container concept outside the MicroVM.

I agree with your point, and that's what I hope to achieve as well. The current situation has many historical reasons, and we will gradually resolve these historical issues in the future.