feat(sdk): add sandbox failover and reload - #39
Merged
Merged
Conversation
Expose a small synchronous checkpoint API in akernel-sdk with immutable checkpoint identities, restore-to-new-sandbox semantics, and explicit list and delete operations. Keep snapshot lifetime manual and default checkpoint timeouts to 180 seconds without adding TTL policy to the SDK. Adapt the default YuanRong sandbox backend, document the public contract, and add unit and integration coverage. Extend standalone GitHub CI to run checkpoint and restore scenarios for runsc and Firecracker on KVM runners. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Advance sandboxd to the latest mainline checkpoint implementation and pin YuanRong to the coordinated reusable snapshot integration. This keeps the AKernel build on the current StartRequest-based restore contract instead of the obsolete implementation used by the earlier prototype. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Enable YuanRong's snapshot data plane in both node startup modes and provide a stable checkpoint staging directory. Advance sandboxd and YuanRong to the implementations validated with the SDK checkpoint contract for runsc and Firecracker. Document the standalone storage and manual checkpoint lifecycle so operators do not confuse node-local staging with an SDK TTL or automatic expiration. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Remove the static /etc/resolv.conf mount from standalone and Helm base OCI configuration. Sandboxd already prepares the resolver file for every sandbox, using the host resolver without an ACL and its managed resolver when an ACL is enabled, so carrying both mounts creates duplicate OCI destinations during checkpoint restore. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Advance sandboxd and YuanRong together so the runtime capability response, guest handoff endpoints, FunctionSystem environment configuration, and RRT restore protocol use one compatible contract in AKernel builds. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Chamberlain1998
pushed a commit
to Chamberlain1998/AKernel
that referenced
this pull request
Aug 26, 2026
Merge AKernel PR inclusionAI#39 with the local reload and failover API so both lifecycle surfaces remain available. Preserve the synchronized YuanRong snapshot implementation and advance sandboxd to b35d309, which adds the restore mount override fix on top of the PR checkpoint integration. Signed-off-by: ChamberlainJI <jichenglin1@huawei.com>
tianyuzhou95
marked this pull request as ready for review
August 27, 2026 09:20
Upgrade the bundled YuanRong core, RRT, and default sandbox SDK to 0.10.1rc1 so AKernel builds use the released checkpoint/restore implementation. Keep the optional actor-based SDK on its independently tested 0.9.9 release. Advance sandboxd to the merged cooperative checkpoint implementation and pin the matching YuanRong source tag. Document the Harbor Deep-SWE manual acceptance flow used to validate process and container continuity. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Move the default sandbox bridge from Docker's standard 172.17.0.0/16 range to 10.88.0.0/16 across Helm and cloud deployment templates. Update resolver rewriting to use the matching bridge address. This avoids deterministic overlap when AKernel itself or a workload runs Docker while preserving one consistent network default across deployment methods. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Expose a restore timeout in the public Sandbox API and pass it to the default YuanRong backend. Default restores to 300 seconds so large Firecracker artifacts can be transferred and verified without a fixed one-minute client deadline. Document and test the timeout contract while keeping resource overrides out of the initial restore API. Remove the heavyweight Deep-SWE procedure from the repository because it remains a test-node-only acceptance workload. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Advance sandboxd to the Firecracker guest-root and transient checkpoint memory fixes. Advance YuanRong to the coordinated lifecycle deadline and heartbeat-fencing changes used by the SDK restore timeout. Keeping the two revisions together ensures AKernel builds one compatible checkpoint and restore chain for long-running Firecracker workloads. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Use the released 0.10.1rc3 core, sandbox SDK, and RRT artifacts while keeping the optional actor SDK on 0.9.9. Verify the official release assets and allow an unreleased RRT binary to be supplied with a matching checksum for testing. Switch the bundled node to YuanRong's local-only snapshot mode and keep checkpoint state on the persistent AKernel data mount. This places artifacts on the same operator-managed filesystem as sandboxd writable storage without requiring the control plane to rebuild YuanRong. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Advance sandboxd to upstream main with Firecracker incremental checkpoint support and the merged restore isolation fixes. Keep the submodule on a publicly fetchable upstream revision while the writeback follow-up is reviewed independently. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
tianyuzhou95
force-pushed
the
albert/cr
branch
from
August 30, 2026 17:19
a1e0788 to
b683f03
Compare
Advance sandboxd to the revision that charges Firecracker snapshot page cache outside the sandbox cgroup on unified cgroup v2. The revision also uses a Full first baseline and avoids synchronous hashes of large local checkpoint components. This pin records the sandboxd build validated by the standalone runsc, Firecracker, and Harbor with Deep-SWE checkpoint/restore tests. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
tianyuzhou95
force-pushed
the
albert/cr
branch
from
August 30, 2026 17:20
b683f03 to
43d3328
Compare
Advance sandboxd to the tested Firecracker checkpoint implementation that starts memory artifact writeback in a bounded background worker. Periodic checkpoints can now overlap physical XFS I/O with guest work while retaining the existing buffered checkpoint durability contract. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Advance sandboxd to the existing PR revision validated with same-node runsc and Firecracker checkpoint reloads. The revision restores gVisor's separate uncompressed page files correctly and retains the tested Firecracker checkpoint optimizations. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Expose same-node automatic failover and explicit reload while keeping recovery-point creation inside the sandbox workload. Remove the public checkpoint catalog so recovery points continue to follow the source sandbox lifecycle. Configure the internal RRT socket at /run/akernel/rrt.sock and exercise runsc and Firecracker reloads in standalone CI, including restored state and network connectivity. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
tianyuzhou95
changed the title
Extend the standalone reload integration test with an SDK-side HTTP service. Verify that both runsc and Firecracker can reach the reverse tunnel before checkpoint and after the tunnel reconnects to the restored physical sandbox. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
tianyuzhou95
force-pushed
the
albert/cr
branch
from
August 31, 2026 05:05
f720f83 to
e19c1fa
Compare
rcyw
requested changes
Aug 31, 2026
Collaborator
There was a problem hiding this comment.
Two AKernel-level contract issues remain: reload() must clearly define what a False result represents and align its implementation, documentation, and tests, while the public Sandbox.id contract must be consistent across backends and documentation. The sandboxd pin now points to the merged upstream commit, but the YuanRong submodule revision must still be available from its declared upstream. Details are inline.
|
|
||
| `Sandbox(failover=True)` opts into same-node recovery of the same logical | ||
| sandbox after its physical runtime fails. `sandbox.reload()` requests the same | ||
| rollback explicitly and returns `False` when no usable local anonymous |
Collaborator
There was a problem hiding this comment.
The documentation defines False narrowly as meaning that no usable checkpoint exists, while the pinned openyuanrong-sandbox==0.10.1rc3 also returns False for every SandboxError, including transport, server, authentication, and response failures. Please define the intended reload() contract and align the dependency behavior, documentation, and tests. If False is reserved for a missing checkpoint, other failures should propagate; if reload is intentionally best-effort, the documentation should state that False may also represent an operational failure.
Advance the sandboxd submodule to the upstream merge of the checkpoint restore hardening and Firecracker performance work. This replaces the temporary personal-branch revision with the stable public mainline commit while preserving the implementation validated by the AKernel end-to-end tests. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
tianyuzhou95
force-pushed
the
albert/cr
branch
from
August 31, 2026 09:19
f90ccfa to
f0598de
Compare
Keep the standalone checkpoint reload gate on runsc so the regular CI job does not require KVM or build the deprecated bundled Python profile. Install curl in the default RRT sandbox and use it for the internal Unix-socket checkpoint request, reverse-tunnel probe, and restored network check. Update the maintained example and SDK documentation to follow the same RRT workflow. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Treat a supported backend reload as a best-effort operation and return false when the rollback cannot be completed, including operational backend errors. This matches the pinned YuanRong sandbox contract instead of documenting a narrow missing-checkpoint-only result. Align the public documentation and tests with the broader boolean contract while retaining an explicit error for backends that do not implement reload. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
Advance the YuanRong submodule to the published checkpoint branch revision that finalizes restore resource inheritance, directory artifacts, local-only affinity, and runtime contracts across its coordinated components. Replace the previous unavailable gitlink so a fresh recursive submodule checkout can resolve the exact source revision from the declared upstream. Signed-off-by: Tianyu Zhou <albert.zty@antgroup.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/run/akernel/rrt.sockCheckpoint creation remains an internal sandbox workload API. The SDK does not
expose checkpoint IDs, restore, list, or delete operations.
Testing
make sdk-checkmake deploy-script-check