From af0fc7cae7f42bb11d1514c753266e93662e6a3e Mon Sep 17 00:00:00 2001 From: yarrischen Date: Thu, 20 Aug 2026 20:20:49 +0800 Subject: [PATCH] feat(network): update sandbox egress policy in place, re-evaluating live flows Add PUT /sandboxes/{sandboxID}/network so a running sandbox's egress policy can be replaced without recreating it, and make the update reach traffic that already exists rather than only future connections. The route, the status codes (204/400/404/409) and the full-replacement semantics follow E2B, so an omitted field clears rather than preserves. One divergence is deliberate: E2B's nftables chain accepts ESTABLISHED,RELATED first, so tightening a policy there has no effect on open connections and a long-lived connection can keep using a revoked destination indefinitely. We re-evaluate instead. ## Datapath Each sandbox carries a policy generation (mvm_meta.policy_version), bumped only after both planes hold the new policy. Every session caches the generation it was admitted under, so the next packet on an established flow is re-judged exactly once per update: - same verdict: restamped and left alone, costing one policy lookup per flow; - no longer allowed, or the verdict changed: the session pair is deleted. TCP is answered with an RST, matching how every other unreachable TCP packet is handled here, so the guest fails fast instead of stalling on retransmits; UDP and ICMP have nothing to reset and are dropped. A verdict *change* retires the flow rather than migrating it. The SNAT and L7 paths disagree about both the reply tuple and which side terminates the TCP connection, so migration is not expressible; the client reconnects and is judged as any new flow. Deleting rather than flagging keeps the retirement self-enforcing, and is what lets this stay small. A later non-SYN packet finds no session and is reset, so a revoked flow cannot resume even if a subsequent update re-allows the destination, while a SYN legitimately opens a fresh connection under the current policy. Both directions go at once, so reply-path callers need no change either, and neither does the reaper. That property holds only once the legacy 80/443 connection drain in do_tcp_nat is removed, which this does. While that branch is present a deleted session is indistinguishable from a lost one, and the drain silently undoes the revocation. It had also lost its own justification independently: written (1b5d2c35) when L7 flows created no session and re-ran the policy lookup per packet, so a DNS-learned entry expiring broke live connections, it was made redundant by #1347, after which L7 flows cache their verdict in nat_session and an established flow never consults the policy maps again. Two of the three causes its comment cites do not hold either -- both session maps are pinned and no startup path flushes them, and they are BPF_MAP_TYPE_HASH rather than LRU, so nothing is evicted. The remaining case, a flow idle past the reaper timeout, contradicts the reaper it depends on: deleteSessions removes both directions, and the drain then revives the egress direction alone, without recreating a session, on the sole evidence that the proxy still holds a socket -- scoped by which flows happen to leave a host socket rather than by policy, which is why it only ever applied to 80/443. Consequently an L7 keepalive connection left idle past the reaper timeout is now reset rather than revived, and the application reconnects. policy_version fits in existing padding, so mvm_meta and nat_session keep their sizes and the pinned-map layout is unchanged. The kernel verifier accepts del_session() while the lookup pointer is still held, checked against mvmtap, nodenic and localgw. ## Control plane cubevs gains UpdateTAPDevicePolicy, a third apply mode beside applyNetPolicy (additive) and replaceNetPolicy (flush then refill). Neither fits a live sandbox: flushing blanks the policy for as long as the refill takes, and swapping the inner map defeats the HashOfMaps inner cache and pays a synchronize_rcu per update. So it diffs against the installed maps and writes only what changed. DNS-learned rows are left untouched, and deny_out's desired set always includes the always-denied private and link-local ranges. Ordering is part of the contract: CubeEgress, then CubeVS (which bumps the generation), then durable state. A failure before the bump leaves flows on their cached verdict instead of judging them against a half-applied map, and a failure before the state write replays the previous policy on restart. Nothing is rolled back; the diff is computed from the live maps, so replaying converges. For clone, snapshot and pause/resume the new policy is written to all three places that outlive the update: the network runtime state file (restart recovery), the Cubelet cubebox store (pause packaging) and Master's sandboxspec (what clone replays). The sandboxspec write happens synchronously before the response, so update then snapshot then clone is read-your-writes; a failure there is logged and does not fail the call, following the same best-effort convention the create path already applies to that store. ## DNS resolver allowance The create path folds the sandbox's resolver addresses into allow_out so domain rules can be resolved at all, but only when the policy names a domain. An update carries just the caller's targets, so it would silently revoke DNS and black-hole every domain rule it had just installed. The resolver list is now recorded on every create -- unconditionally, because a later update may introduce the first domain -- and folded back in under the same "policy still names a domain" condition, so dropping every domain also withdraws the implicit DNS access. Recording is separate from installing: an all-IP policy still gets no resolver access, since allow_out cannot be narrowed to port 53. The gate therefore asks whether a target is *installed* as a domain, not whether it looks like a DNS name. The latter accepts "10.0.0.1" -- digits are valid label characters and nothing requires a non-numeric top label -- and would fold the resolver into every policy written with bare IPv4 literals, granting one extra /32 on all ports that the caller never asked for. cubevs exports that install decision as IsAllowOutDomainTarget, mirroring splitAllowOutTargets, so the update gate cannot diverge from the create gate, which has always been IP-first. An update that neither installs nor clears L7 rules does not contact CubeEgress at all. That is decided from the old and new rule sets, not from whether the proxy happens to be configured, so an L3-only sandbox never depends on a component it does not use. ## Known limitation Addresses already learned for a domain outlive the rule that produced them until their DNS TTL expires, so revoking a domain promptly needs a short resolver TTL. Provenance tracking for learned entries is deferred. ## Testing Go unit tests cover the map diff (revocation, expanded L7 port sets, deny_out convergence, DNS port-set replacement), the generation bump and its survival across metadata rewrites, and the controller's ordering, resolver refold, L3-only skip and "a failed update keeps the old policy" guarantees. The resolver gate is exercised over bare, masked, subnet and L7-host target forms, because only the bare literals distinguish the install decision from a name-shape check. A new BPF case drives session_policy_revoked through the existing egress-policy harness across seven verdict/generation combinations. Ten end-to-end cases were added under a new network_dynamic_update capability. Two of them are what actually distinguish this feature: a revoked connection must be torn down, and a still-permitted one must be left alone -- the second is what stops an implementation that simply kills every session from passing. Their guest-side holder uses TCP keepalive rather than application bytes, because a peer that hangs up on unexpected data is indistinguishable from a policy reset, and it classifies RESET, ALIVE, EOF and DATA separately so only the unambiguous outcomes are asserted. Three more cover the compatibility requirement: a clone inherits the updated policy rather than the create-time one (which also covers read-your-writes, since clone snapshots immediately after the update returns), a clone of a narrowed policy is not more permissive, and the policy survives pause/resume. These have to be end-to-end -- clone and snapshot are not network-aware, they just replay a stored create request, so nothing under templatecenter changed here and review alone cannot tell whether the spec write-back landed. Verified on a live single-node deployment: all ten cases pass on repeated runs, and examples/network-policy/network_dynamic_update.py walks an IP allow list, a connection carried across a revoking update (observed as RESET), a domain allow list, and switching on L7 interception mid-run. Signed-off-by: yarrischen Co-authored-by: Cursor --- CubeAPI/src/cubemaster/mod.rs | 42 +++ CubeAPI/src/handlers/sandboxes.rs | 54 ++- CubeAPI/src/models/mod.rs | 15 + CubeAPI/src/openapi.rs | 5 +- CubeAPI/src/routes.rs | 4 + CubeAPI/src/services/sandboxes.rs | 40 ++- .../api/services/cubebox/v1/cubebox.pb.go | 143 ++++---- .../api/services/cubebox/v1/cubebox.proto | 4 + .../api/services/errorcode/v1/errorcode.pb.go | 8 +- .../api/services/errorcode/v1/errorcode.proto | 1 + .../pkg/service/httpservice/cube/cube.go | 1 + .../pkg/service/httpservice/cube/routes.go | 1 + .../pkg/service/httpservice/cube/update.go | 39 +++ .../pkg/service/sandbox/sandbox_update.go | 134 +++++++ CubeMaster/pkg/service/sandbox/types/types.go | 17 + CubeMaster/pkg/service/sandbox/util.go | 33 +- CubeNet/cubevs/cubevs.go | 9 +- CubeNet/cubevs/dnspolicy.go | 40 +++ CubeNet/cubevs/egress_policy_test.go | 189 +++++++++- CubeNet/cubevs/netpolicy.go | 313 ++++++++++++++--- CubeNet/cubevs/netpolicy_test.go | 322 +++++++++++++++++ CubeNet/cubevs/reaper.go | 25 +- CubeNet/cubevs/tap.go | 9 + CubeNet/src/cubevs.h | 17 +- CubeNet/src/egress_policy_test.bpf.c | 38 ++ CubeNet/src/mvmtap.bpf.c | 72 ++-- CubeNet/src/session.h | 71 ++++ Cubelet/api/services/cubebox/v1/cubebox.pb.go | 143 ++++---- Cubelet/api/services/cubebox/v1/cubebox.proto | 4 + .../api/services/errorcode/v1/errorcode.pb.go | 8 +- .../api/services/errorcode/v1/errorcode.proto | 1 + Cubelet/doc/cubelet-api.md | 2 + Cubelet/network/plugin_policy.go | 86 ++++- Cubelet/network/plugin_shim.go | 5 +- Cubelet/network/plugin_tap.go | 6 +- Cubelet/network/plugin_tap_create_test.go | 25 +- Cubelet/network/runtime/controller.go | 65 ++++ Cubelet/network/runtime/controller_test.go | 262 ++++++++++++++ Cubelet/network/runtime/cubeegress_adapter.go | 36 ++ Cubelet/network/runtime/cubevs_adapter.go | 21 ++ .../network/runtime/cubevs_adapter_test.go | 14 + Cubelet/network/runtime/network_runtime.go | 9 + Cubelet/network/runtime/policy_builder.go | 43 +++ Cubelet/network/runtime/state_store.go | 49 ++- Cubelet/network/runtime/types.go | 18 +- Cubelet/pkg/constants/const.go | 1 + Cubelet/services/cubebox/update.go | 51 +++ docs/guide/network-policy.md | 28 ++ docs/zh/guide/network-policy.md | 28 ++ examples/network-policy/README.md | 47 ++- .../network-policy/network_dynamic_update.py | 257 ++++++++++++++ examples/network-policy/requirements.txt | 3 + openapi.yml | 99 +++++- sdk/go/client.go | 30 +- sdk/go/models.go | 13 + sdk/go/policy.go | 37 ++ sdk/go/sandbox.go | 32 ++ sdk/node/src/index.ts | 1 + sdk/node/src/sandbox.ts | 91 +++-- sdk/python/cubesandbox/_policy.py | 35 ++ sdk/python/cubesandbox/sandbox.py | 68 ++-- tests/e2e/sdk_compat/README.md | 4 +- tests/e2e/sdk_compat/README_zh.md | 3 +- tests/e2e/sdk_compat/adapters/base.py | 9 + .../adapters/cubesandbox_adapter.py | 11 + .../sdk_compat/adapters/tracing_adapter.py | 24 ++ .../cases/network/test_policy_update.py | 330 ++++++++++++++++++ tests/e2e/sdk_compat/docs/test-coverage.md | 29 ++ tests/e2e/sdk_compat/docs/zh/test-coverage.md | 23 ++ .../e2e/sdk_compat/framework/capabilities.py | 4 + .../e2e/sdk_compat/framework/network_probe.py | 163 +++++++++ 71 files changed, 3502 insertions(+), 362 deletions(-) create mode 100644 examples/network-policy/network_dynamic_update.py create mode 100644 tests/e2e/sdk_compat/cases/network/test_policy_update.py diff --git a/CubeAPI/src/cubemaster/mod.rs b/CubeAPI/src/cubemaster/mod.rs index fc14dff8e..108a6f3c8 100644 --- a/CubeAPI/src/cubemaster/mod.rs +++ b/CubeAPI/src/cubemaster/mod.rs @@ -158,6 +158,22 @@ impl CubeMasterClient { parse_response(resp).await } + /// POST /cube/sandbox/network — replace a running sandbox's egress policy. + pub async fn update_sandbox_network( + &self, + req: &SandboxNetworkRequest, + ) -> Result { + let url = format!("{}/cube/sandbox/network", self.base_url); + let resp = self + .inner + .post(&url) + .json(req) + .send() + .await + .map_err(CubeMasterError::Http)?; + parse_response(resp).await + } + /// POST /cube/sandbox/refresh — extend TTL by a delta (seconds). /// ❌ New API required on CubeMaster. pub async fn refresh_sandbox( @@ -1430,6 +1446,32 @@ pub struct SandboxUpdateResponse { pub ret: RetCode, } +// ─── Update sandbox network policy ──────────────────────────────────────── +// ✅ Implemented: POST /cube/sandbox/network + +#[derive(Debug, Serialize)] +pub struct SandboxNetworkRequest { + #[serde(rename = "RequestID", alias = "requestID")] + pub request_id: String, + #[serde(rename = "sandboxID")] + pub sandbox_id: String, + #[serde(rename = "instanceType")] + pub instance_type: String, + /// Complete desired policy. Always sent, even when empty, because an empty + /// policy is a meaningful request: it clears every egress rule. + pub cube_network_config: CubeNetworkConfig, +} + +#[derive(Debug, Deserialize)] +#[allow(dead_code)] +pub struct SandboxNetworkResponse { + #[serde(rename = "RequestID", alias = "requestID")] + pub request_id: String, + #[serde(rename = "sandboxID", default)] + pub sandbox_id: String, + pub ret: RetCode, +} + // ─── Set sandbox timeout (absolute) ─────────────────────────────────────── // ✅ Implemented: POST /cube/sandbox/timeout diff --git a/CubeAPI/src/handlers/sandboxes.rs b/CubeAPI/src/handlers/sandboxes.rs index 8e1828ec7..2282adfd8 100644 --- a/CubeAPI/src/handlers/sandboxes.rs +++ b/CubeAPI/src/handlers/sandboxes.rs @@ -16,7 +16,7 @@ use crate::{ models::{ ApiError, ConnectSandbox, ListSandboxesQuery, ListSandboxesV2Query, NewSandbox, RefreshRequest, ResumedSandbox, Sandbox, SandboxDetail, SandboxLogsQuery, - SandboxLogsV2Query, SandboxLogsV2Response, SetTimeoutRequest, + SandboxLogsV2Query, SandboxLogsV2Response, SetTimeoutRequest, UpdateSandboxNetworkRequest, }, state::AppState, }; @@ -519,6 +519,58 @@ pub async fn set_sandbox_timeout( Ok(StatusCode::NO_CONTENT) } +// ─── PUT /sandboxes/:sandboxID/network ──────────────────────────────────────── + +#[utoipa::path( + put, + path = "/sandboxes/{sandboxID}/network", + params( + ("sandboxID" = String, Path, description = "Sandbox identifier") + ), + request_body = UpdateSandboxNetworkRequest, + responses( + (status = 204, description = "Network policy updated"), + (status = 400, description = "Invalid network policy", body = ApiError), + (status = 404, description = "Sandbox not found", body = ApiError), + (status = 409, description = "Sandbox is not running", body = ApiError), + (status = 500, description = "Unexpected backend error", body = ApiError) + ) +)] +pub async fn update_sandbox_network( + State(state): State, + Path(sandbox_id): Path, + Json(body): Json, +) -> AppResult { + state + .logger + .log( + LogEvent::new(LogLevel::Debug, "api.request") + .field("handler", "update_sandbox_network") + .field("sandbox_id", &sandbox_id), + ) + .await; + + state + .services + .sandboxes + .update_network( + &sandbox_id, + body.allow_internet_access, + body.network.as_ref(), + ) + .await?; + + tracing::info!(sandbox_id = %sandbox_id, "update_sandbox_network: success"); + state + .logger + .log( + LogEvent::new(LogLevel::Info, "sandbox.network.updated") + .field("sandbox_id", &sandbox_id), + ) + .await; + Ok(StatusCode::NO_CONTENT) +} + // ─── POST /sandboxes/:sandboxID/refreshes ───────────────────────────────────── #[utoipa::path( diff --git a/CubeAPI/src/models/mod.rs b/CubeAPI/src/models/mod.rs index 465f787c1..8d0766c38 100644 --- a/CubeAPI/src/models/mod.rs +++ b/CubeAPI/src/models/mod.rs @@ -521,6 +521,21 @@ fn default_log_limit() -> i32 { 1000 } +// ─── Sandbox — network policy ───────────────────────────────────────────── + +/// Request body for PUT /sandboxes/{id}/network. +/// +/// The body is the complete desired egress policy, not a patch: any field left +/// out clears what the sandbox currently has. This mirrors how `network` is +/// interpreted at create time, so the same object can be sent to either. +#[derive(Debug, Deserialize, ToSchema)] +pub struct UpdateSandboxNetworkRequest { + #[serde(rename = "allowInternetAccess", alias = "allow_internet_access")] + pub allow_internet_access: Option, + #[serde(default)] + pub network: Option, +} + // ─── Sandbox — timeout / refresh ────────────────────────────────────────── /// Request body for POST /sandboxes/{id}/timeout diff --git a/CubeAPI/src/openapi.rs b/CubeAPI/src/openapi.rs index 01bda5c99..a74d36a09 100644 --- a/CubeAPI/src/openapi.rs +++ b/CubeAPI/src/openapi.rs @@ -19,7 +19,8 @@ use crate::{ SetTimeoutRequest, SnapshotInfo, SnapshotListItem, TemplateAliasLookupResponse, TemplateBuildJob, TemplateBuildStatus, TemplateCompatAdoptResponseView, TemplateCompatMatrixView, TemplateCompatRowView, TemplateCompatSummaryView, TemplateDetail, - TemplateNodeCompatView, TemplateSummary, Volume, VolumeAndToken, + TemplateNodeCompatView, TemplateSummary, UpdateSandboxNetworkRequest, Volume, + VolumeAndToken, }, }; @@ -77,6 +78,7 @@ impl Modify for SecurityAddon { handlers::sandboxes::get_sandbox_logs, handlers::sandboxes::get_sandbox_logs_v2, handlers::sandboxes::set_sandbox_timeout, + handlers::sandboxes::update_sandbox_network, handlers::sandboxes::refresh_sandbox, handlers::snapshots::create_snapshot, handlers::snapshots::list_snapshots, @@ -111,6 +113,7 @@ impl Modify for SecurityAddon { ConnectSandbox, ResumedSandbox, SetTimeoutRequest, + UpdateSandboxNetworkRequest, RefreshRequest, SandboxLogEntry, SandboxLogs, diff --git a/CubeAPI/src/routes.rs b/CubeAPI/src/routes.rs index 8c92f64e7..a0ea88f60 100644 --- a/CubeAPI/src/routes.rs +++ b/CubeAPI/src/routes.rs @@ -98,6 +98,10 @@ fn build_sandbox_routes(state: &AppState, auth_configured: bool) -> Router, + network: Option<&SandboxNetworkConfig>, + ) -> AppResult<()> { + let cube_network_config = + build_cube_network_config(allow_internet_access, network)?.unwrap_or_default(); + + let req = SandboxNetworkRequest { + request_id: new_request_id(), + sandbox_id: sandbox_id.to_string(), + instance_type: self.instance_type.clone(), + cube_network_config, + }; + + let resp = self + .cubemaster + .update_sandbox_network(&req) + .await + .map_err(|e| sandbox_not_found_or_internal(e, sandbox_id))?; + + resp.ret + .into_result() + .map_err(|e| sandbox_not_found_or_internal(e, sandbox_id))?; + + Ok(()) + } + pub async fn refresh(&self, sandbox_id: &str, duration: i32) -> AppResult<()> { let req = SandboxRefreshRequest { request_id: new_request_id(), diff --git a/CubeMaster/api/services/cubebox/v1/cubebox.pb.go b/CubeMaster/api/services/cubebox/v1/cubebox.pb.go index 77413d739..aadbe9aaa 100644 --- a/CubeMaster/api/services/cubebox/v1/cubebox.pb.go +++ b/CubeMaster/api/services/cubebox/v1/cubebox.pb.go @@ -3976,9 +3976,13 @@ type UpdateCubeSandboxRequest struct { // new features that are opaque to the Kubernetes APIs (both user-facing // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. - Annotations map[string]string `protobuf:"bytes,3,rep,name=annotations,proto3" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + Annotations map[string]string `protobuf:"bytes,3,rep,name=annotations,proto3" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` + // Replacement egress policy for a running sandbox. Absent means "leave the + // network alone"; present means the complete desired state, so an omitted or + // empty field inside it clears whatever is currently installed. + CubeNetworkConfig *CubeNetworkConfig `protobuf:"bytes,4,opt,name=cube_network_config,json=cubeNetworkConfig,proto3,oneof" json:"cube_network_config,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *UpdateCubeSandboxRequest) Reset() { @@ -4032,6 +4036,13 @@ func (x *UpdateCubeSandboxRequest) GetAnnotations() map[string]string { return nil } +func (x *UpdateCubeSandboxRequest) GetCubeNetworkConfig() *CubeNetworkConfig { + if x != nil { + return x.CubeNetworkConfig + } + return nil +} + type UpdateCubeSandboxResponse struct { state protoimpl.MessageState `protogen:"open.v1"` // requestID reqID @@ -6892,14 +6903,16 @@ const file_api_services_cubebox_v1_cubebox_proto_rawDesc = "" + "\a_filterB\t\n" + "\a_option\"Y\n" + "\x17ListCubeSandboxResponse\x12>\n" + - "\x05items\x18\x01 \x03(\v2(.cubelet.services.cubebox.v1.CubeSandboxR\x05items\"\x80\x02\n" + + "\x05items\x18\x01 \x03(\v2(.cubelet.services.cubebox.v1.CubeSandboxR\x05items\"\xfd\x02\n" + "\x18UpdateCubeSandboxRequest\x12\x1c\n" + "\trequestID\x18\x01 \x01(\tR\trequestID\x12\x1c\n" + "\tsandboxID\x18\x02 \x01(\tR\tsandboxID\x12h\n" + - "\vannotations\x18\x03 \x03(\v2F.cubelet.services.cubebox.v1.UpdateCubeSandboxRequest.AnnotationsEntryR\vannotations\x1a>\n" + + "\vannotations\x18\x03 \x03(\v2F.cubelet.services.cubebox.v1.UpdateCubeSandboxRequest.AnnotationsEntryR\vannotations\x12c\n" + + "\x13cube_network_config\x18\x04 \x01(\v2..cubelet.services.cubebox.v1.CubeNetworkConfigH\x00R\x11cubeNetworkConfig\x88\x01\x01\x1a>\n" + "\x10AnnotationsEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + - "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x8b\x02\n" + + "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01B\x16\n" + + "\x14_cube_network_config\"\x8b\x02\n" + "\x19UpdateCubeSandboxResponse\x12\x1c\n" + "\trequestID\x18\x01 \x01(\tR\trequestID\x124\n" + "\x03ret\x18\x02 \x01(\v2\".cubelet.services.errorcode.v1.RetR\x03ret\x12^\n" + @@ -7380,64 +7393,65 @@ var file_api_services_cubebox_v1_cubebox_proto_depIdxs = []int32{ 59, // 70: cubelet.services.cubebox.v1.ListCubeSandboxRequest.option:type_name -> cubelet.services.cubebox.v1.ListCubeSandboxOption 50, // 71: cubelet.services.cubebox.v1.ListCubeSandboxResponse.items:type_name -> cubelet.services.cubebox.v1.CubeSandbox 99, // 72: cubelet.services.cubebox.v1.UpdateCubeSandboxRequest.annotations:type_name -> cubelet.services.cubebox.v1.UpdateCubeSandboxRequest.AnnotationsEntry - 104, // 73: cubelet.services.cubebox.v1.UpdateCubeSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 100, // 74: cubelet.services.cubebox.v1.UpdateCubeSandboxResponse.ext_info:type_name -> cubelet.services.cubebox.v1.UpdateCubeSandboxResponse.ExtInfoEntry - 104, // 75: cubelet.services.cubebox.v1.ExecCubeSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 40, // 76: cubelet.services.cubebox.v1.AppSnapshotRequest.create_request:type_name -> cubelet.services.cubebox.v1.RunCubeSandboxRequest - 104, // 77: cubelet.services.cubebox.v1.AppSnapshotResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 104, // 78: cubelet.services.cubebox.v1.CommitSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 104, // 79: cubelet.services.cubebox.v1.RollbackSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 68, // 80: cubelet.services.cubebox.v1.CleanupTemplateRequest.objects:type_name -> cubelet.services.cubebox.v1.CowObjectRef - 104, // 81: cubelet.services.cubebox.v1.CleanupTemplateResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 68, // 82: cubelet.services.cubebox.v1.ListSandboxSnapshotsRequest.objects:type_name -> cubelet.services.cubebox.v1.CowObjectRef - 104, // 83: cubelet.services.cubebox.v1.ListSandboxSnapshotsResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 72, // 84: cubelet.services.cubebox.v1.ListSandboxSnapshotsResponse.objects:type_name -> cubelet.services.cubebox.v1.CowObjectStatus - 104, // 85: cubelet.services.cubebox.v1.ListLocalSnapshotsResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 75, // 86: cubelet.services.cubebox.v1.ListLocalSnapshotsResponse.snapshots:type_name -> cubelet.services.cubebox.v1.LocalSnapshotInfo - 104, // 87: cubelet.services.cubebox.v1.GetLocalSnapshotResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 75, // 88: cubelet.services.cubebox.v1.GetLocalSnapshotResponse.snapshot:type_name -> cubelet.services.cubebox.v1.LocalSnapshotInfo - 104, // 89: cubelet.services.cubebox.v1.GetStorageMetricsResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 101, // 90: cubelet.services.cubebox.v1.GetStorageMetricsResponse.metrics:type_name -> cubelet.services.cubebox.v1.GetStorageMetricsResponse.MetricsEntry - 81, // 91: cubelet.services.cubebox.v1.SandboxStorageInfo.volumes:type_name -> cubelet.services.cubebox.v1.StorageVolumeInfo - 104, // 92: cubelet.services.cubebox.v1.InspectStorageVolumesResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 82, // 93: cubelet.services.cubebox.v1.InspectStorageVolumesResponse.sandboxes:type_name -> cubelet.services.cubebox.v1.SandboxStorageInfo - 104, // 94: cubelet.services.cubebox.v1.CleanupOrphanStorageFilesResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 86, // 95: cubelet.services.cubebox.v1.CleanupOrphanStorageFilesResponse.orphans:type_name -> cubelet.services.cubebox.v1.StorageOrphanEntry - 40, // 96: cubelet.services.cubebox.v1.CubeboxMgr.Create:input_type -> cubelet.services.cubebox.v1.RunCubeSandboxRequest - 48, // 97: cubelet.services.cubebox.v1.CubeboxMgr.Destroy:input_type -> cubelet.services.cubebox.v1.DestroyCubeSandboxRequest - 54, // 98: cubelet.services.cubebox.v1.CubeboxMgr.List:input_type -> cubelet.services.cubebox.v1.ListCubeSandboxRequest - 56, // 99: cubelet.services.cubebox.v1.CubeboxMgr.Update:input_type -> cubelet.services.cubebox.v1.UpdateCubeSandboxRequest - 60, // 100: cubelet.services.cubebox.v1.CubeboxMgr.Exec:input_type -> cubelet.services.cubebox.v1.ExecCubeSandboxRequest - 62, // 101: cubelet.services.cubebox.v1.CubeboxMgr.AppSnapshot:input_type -> cubelet.services.cubebox.v1.AppSnapshotRequest - 64, // 102: cubelet.services.cubebox.v1.CubeboxMgr.CommitSandbox:input_type -> cubelet.services.cubebox.v1.CommitSandboxRequest - 66, // 103: cubelet.services.cubebox.v1.CubeboxMgr.RollbackSandbox:input_type -> cubelet.services.cubebox.v1.RollbackSandboxRequest - 69, // 104: cubelet.services.cubebox.v1.CubeboxMgr.CleanupTemplate:input_type -> cubelet.services.cubebox.v1.CleanupTemplateRequest - 71, // 105: cubelet.services.cubebox.v1.CubeboxMgr.ListSandboxSnapshots:input_type -> cubelet.services.cubebox.v1.ListSandboxSnapshotsRequest - 74, // 106: cubelet.services.cubebox.v1.CubeboxMgr.ListLocalSnapshots:input_type -> cubelet.services.cubebox.v1.ListLocalSnapshotsRequest - 77, // 107: cubelet.services.cubebox.v1.CubeboxMgr.GetLocalSnapshot:input_type -> cubelet.services.cubebox.v1.GetLocalSnapshotRequest - 79, // 108: cubelet.services.cubebox.v1.CubeboxMgr.GetStorageMetrics:input_type -> cubelet.services.cubebox.v1.GetStorageMetricsRequest - 83, // 109: cubelet.services.cubebox.v1.CubeboxMgr.InspectStorageVolumes:input_type -> cubelet.services.cubebox.v1.InspectStorageVolumesRequest - 85, // 110: cubelet.services.cubebox.v1.CubeboxMgr.CleanupOrphanStorageFiles:input_type -> cubelet.services.cubebox.v1.CleanupOrphanStorageFilesRequest - 41, // 111: cubelet.services.cubebox.v1.CubeboxMgr.Create:output_type -> cubelet.services.cubebox.v1.RunCubeSandboxResponse - 49, // 112: cubelet.services.cubebox.v1.CubeboxMgr.Destroy:output_type -> cubelet.services.cubebox.v1.DestroyCubeSandboxResponse - 55, // 113: cubelet.services.cubebox.v1.CubeboxMgr.List:output_type -> cubelet.services.cubebox.v1.ListCubeSandboxResponse - 57, // 114: cubelet.services.cubebox.v1.CubeboxMgr.Update:output_type -> cubelet.services.cubebox.v1.UpdateCubeSandboxResponse - 61, // 115: cubelet.services.cubebox.v1.CubeboxMgr.Exec:output_type -> cubelet.services.cubebox.v1.ExecCubeSandboxResponse - 63, // 116: cubelet.services.cubebox.v1.CubeboxMgr.AppSnapshot:output_type -> cubelet.services.cubebox.v1.AppSnapshotResponse - 65, // 117: cubelet.services.cubebox.v1.CubeboxMgr.CommitSandbox:output_type -> cubelet.services.cubebox.v1.CommitSandboxResponse - 67, // 118: cubelet.services.cubebox.v1.CubeboxMgr.RollbackSandbox:output_type -> cubelet.services.cubebox.v1.RollbackSandboxResponse - 70, // 119: cubelet.services.cubebox.v1.CubeboxMgr.CleanupTemplate:output_type -> cubelet.services.cubebox.v1.CleanupTemplateResponse - 73, // 120: cubelet.services.cubebox.v1.CubeboxMgr.ListSandboxSnapshots:output_type -> cubelet.services.cubebox.v1.ListSandboxSnapshotsResponse - 76, // 121: cubelet.services.cubebox.v1.CubeboxMgr.ListLocalSnapshots:output_type -> cubelet.services.cubebox.v1.ListLocalSnapshotsResponse - 78, // 122: cubelet.services.cubebox.v1.CubeboxMgr.GetLocalSnapshot:output_type -> cubelet.services.cubebox.v1.GetLocalSnapshotResponse - 80, // 123: cubelet.services.cubebox.v1.CubeboxMgr.GetStorageMetrics:output_type -> cubelet.services.cubebox.v1.GetStorageMetricsResponse - 84, // 124: cubelet.services.cubebox.v1.CubeboxMgr.InspectStorageVolumes:output_type -> cubelet.services.cubebox.v1.InspectStorageVolumesResponse - 87, // 125: cubelet.services.cubebox.v1.CubeboxMgr.CleanupOrphanStorageFiles:output_type -> cubelet.services.cubebox.v1.CleanupOrphanStorageFilesResponse - 111, // [111:126] is the sub-list for method output_type - 96, // [96:111] is the sub-list for method input_type - 96, // [96:96] is the sub-list for extension type_name - 96, // [96:96] is the sub-list for extension extendee - 0, // [0:96] is the sub-list for field type_name + 43, // 73: cubelet.services.cubebox.v1.UpdateCubeSandboxRequest.cube_network_config:type_name -> cubelet.services.cubebox.v1.CubeNetworkConfig + 104, // 74: cubelet.services.cubebox.v1.UpdateCubeSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 100, // 75: cubelet.services.cubebox.v1.UpdateCubeSandboxResponse.ext_info:type_name -> cubelet.services.cubebox.v1.UpdateCubeSandboxResponse.ExtInfoEntry + 104, // 76: cubelet.services.cubebox.v1.ExecCubeSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 40, // 77: cubelet.services.cubebox.v1.AppSnapshotRequest.create_request:type_name -> cubelet.services.cubebox.v1.RunCubeSandboxRequest + 104, // 78: cubelet.services.cubebox.v1.AppSnapshotResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 104, // 79: cubelet.services.cubebox.v1.CommitSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 104, // 80: cubelet.services.cubebox.v1.RollbackSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 68, // 81: cubelet.services.cubebox.v1.CleanupTemplateRequest.objects:type_name -> cubelet.services.cubebox.v1.CowObjectRef + 104, // 82: cubelet.services.cubebox.v1.CleanupTemplateResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 68, // 83: cubelet.services.cubebox.v1.ListSandboxSnapshotsRequest.objects:type_name -> cubelet.services.cubebox.v1.CowObjectRef + 104, // 84: cubelet.services.cubebox.v1.ListSandboxSnapshotsResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 72, // 85: cubelet.services.cubebox.v1.ListSandboxSnapshotsResponse.objects:type_name -> cubelet.services.cubebox.v1.CowObjectStatus + 104, // 86: cubelet.services.cubebox.v1.ListLocalSnapshotsResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 75, // 87: cubelet.services.cubebox.v1.ListLocalSnapshotsResponse.snapshots:type_name -> cubelet.services.cubebox.v1.LocalSnapshotInfo + 104, // 88: cubelet.services.cubebox.v1.GetLocalSnapshotResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 75, // 89: cubelet.services.cubebox.v1.GetLocalSnapshotResponse.snapshot:type_name -> cubelet.services.cubebox.v1.LocalSnapshotInfo + 104, // 90: cubelet.services.cubebox.v1.GetStorageMetricsResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 101, // 91: cubelet.services.cubebox.v1.GetStorageMetricsResponse.metrics:type_name -> cubelet.services.cubebox.v1.GetStorageMetricsResponse.MetricsEntry + 81, // 92: cubelet.services.cubebox.v1.SandboxStorageInfo.volumes:type_name -> cubelet.services.cubebox.v1.StorageVolumeInfo + 104, // 93: cubelet.services.cubebox.v1.InspectStorageVolumesResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 82, // 94: cubelet.services.cubebox.v1.InspectStorageVolumesResponse.sandboxes:type_name -> cubelet.services.cubebox.v1.SandboxStorageInfo + 104, // 95: cubelet.services.cubebox.v1.CleanupOrphanStorageFilesResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 86, // 96: cubelet.services.cubebox.v1.CleanupOrphanStorageFilesResponse.orphans:type_name -> cubelet.services.cubebox.v1.StorageOrphanEntry + 40, // 97: cubelet.services.cubebox.v1.CubeboxMgr.Create:input_type -> cubelet.services.cubebox.v1.RunCubeSandboxRequest + 48, // 98: cubelet.services.cubebox.v1.CubeboxMgr.Destroy:input_type -> cubelet.services.cubebox.v1.DestroyCubeSandboxRequest + 54, // 99: cubelet.services.cubebox.v1.CubeboxMgr.List:input_type -> cubelet.services.cubebox.v1.ListCubeSandboxRequest + 56, // 100: cubelet.services.cubebox.v1.CubeboxMgr.Update:input_type -> cubelet.services.cubebox.v1.UpdateCubeSandboxRequest + 60, // 101: cubelet.services.cubebox.v1.CubeboxMgr.Exec:input_type -> cubelet.services.cubebox.v1.ExecCubeSandboxRequest + 62, // 102: cubelet.services.cubebox.v1.CubeboxMgr.AppSnapshot:input_type -> cubelet.services.cubebox.v1.AppSnapshotRequest + 64, // 103: cubelet.services.cubebox.v1.CubeboxMgr.CommitSandbox:input_type -> cubelet.services.cubebox.v1.CommitSandboxRequest + 66, // 104: cubelet.services.cubebox.v1.CubeboxMgr.RollbackSandbox:input_type -> cubelet.services.cubebox.v1.RollbackSandboxRequest + 69, // 105: cubelet.services.cubebox.v1.CubeboxMgr.CleanupTemplate:input_type -> cubelet.services.cubebox.v1.CleanupTemplateRequest + 71, // 106: cubelet.services.cubebox.v1.CubeboxMgr.ListSandboxSnapshots:input_type -> cubelet.services.cubebox.v1.ListSandboxSnapshotsRequest + 74, // 107: cubelet.services.cubebox.v1.CubeboxMgr.ListLocalSnapshots:input_type -> cubelet.services.cubebox.v1.ListLocalSnapshotsRequest + 77, // 108: cubelet.services.cubebox.v1.CubeboxMgr.GetLocalSnapshot:input_type -> cubelet.services.cubebox.v1.GetLocalSnapshotRequest + 79, // 109: cubelet.services.cubebox.v1.CubeboxMgr.GetStorageMetrics:input_type -> cubelet.services.cubebox.v1.GetStorageMetricsRequest + 83, // 110: cubelet.services.cubebox.v1.CubeboxMgr.InspectStorageVolumes:input_type -> cubelet.services.cubebox.v1.InspectStorageVolumesRequest + 85, // 111: cubelet.services.cubebox.v1.CubeboxMgr.CleanupOrphanStorageFiles:input_type -> cubelet.services.cubebox.v1.CleanupOrphanStorageFilesRequest + 41, // 112: cubelet.services.cubebox.v1.CubeboxMgr.Create:output_type -> cubelet.services.cubebox.v1.RunCubeSandboxResponse + 49, // 113: cubelet.services.cubebox.v1.CubeboxMgr.Destroy:output_type -> cubelet.services.cubebox.v1.DestroyCubeSandboxResponse + 55, // 114: cubelet.services.cubebox.v1.CubeboxMgr.List:output_type -> cubelet.services.cubebox.v1.ListCubeSandboxResponse + 57, // 115: cubelet.services.cubebox.v1.CubeboxMgr.Update:output_type -> cubelet.services.cubebox.v1.UpdateCubeSandboxResponse + 61, // 116: cubelet.services.cubebox.v1.CubeboxMgr.Exec:output_type -> cubelet.services.cubebox.v1.ExecCubeSandboxResponse + 63, // 117: cubelet.services.cubebox.v1.CubeboxMgr.AppSnapshot:output_type -> cubelet.services.cubebox.v1.AppSnapshotResponse + 65, // 118: cubelet.services.cubebox.v1.CubeboxMgr.CommitSandbox:output_type -> cubelet.services.cubebox.v1.CommitSandboxResponse + 67, // 119: cubelet.services.cubebox.v1.CubeboxMgr.RollbackSandbox:output_type -> cubelet.services.cubebox.v1.RollbackSandboxResponse + 70, // 120: cubelet.services.cubebox.v1.CubeboxMgr.CleanupTemplate:output_type -> cubelet.services.cubebox.v1.CleanupTemplateResponse + 73, // 121: cubelet.services.cubebox.v1.CubeboxMgr.ListSandboxSnapshots:output_type -> cubelet.services.cubebox.v1.ListSandboxSnapshotsResponse + 76, // 122: cubelet.services.cubebox.v1.CubeboxMgr.ListLocalSnapshots:output_type -> cubelet.services.cubebox.v1.ListLocalSnapshotsResponse + 78, // 123: cubelet.services.cubebox.v1.CubeboxMgr.GetLocalSnapshot:output_type -> cubelet.services.cubebox.v1.GetLocalSnapshotResponse + 80, // 124: cubelet.services.cubebox.v1.CubeboxMgr.GetStorageMetrics:output_type -> cubelet.services.cubebox.v1.GetStorageMetricsResponse + 84, // 125: cubelet.services.cubebox.v1.CubeboxMgr.InspectStorageVolumes:output_type -> cubelet.services.cubebox.v1.InspectStorageVolumesResponse + 87, // 126: cubelet.services.cubebox.v1.CubeboxMgr.CleanupOrphanStorageFiles:output_type -> cubelet.services.cubebox.v1.CleanupOrphanStorageFilesResponse + 112, // [112:127] is the sub-list for method output_type + 97, // [97:112] is the sub-list for method input_type + 97, // [97:97] is the sub-list for extension type_name + 97, // [97:97] is the sub-list for extension extendee + 0, // [0:97] is the sub-list for field type_name } func init() { file_api_services_cubebox_v1_cubebox_proto_init() } @@ -7455,6 +7469,7 @@ func file_api_services_cubebox_v1_cubebox_proto_init() { file_api_services_cubebox_v1_cubebox_proto_msgTypes[40].OneofWrappers = []any{} file_api_services_cubebox_v1_cubebox_proto_msgTypes[41].OneofWrappers = []any{} file_api_services_cubebox_v1_cubebox_proto_msgTypes[48].OneofWrappers = []any{} + file_api_services_cubebox_v1_cubebox_proto_msgTypes[50].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ diff --git a/CubeMaster/api/services/cubebox/v1/cubebox.proto b/CubeMaster/api/services/cubebox/v1/cubebox.proto index 13ba0279f..b6401b5ee 100644 --- a/CubeMaster/api/services/cubebox/v1/cubebox.proto +++ b/CubeMaster/api/services/cubebox/v1/cubebox.proto @@ -758,6 +758,10 @@ message UpdateCubeSandboxRequest { // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. map annotations = 3; + // Replacement egress policy for a running sandbox. Absent means "leave the + // network alone"; present means the complete desired state, so an omitted or + // empty field inside it clears whatever is currently installed. + optional CubeNetworkConfig cube_network_config = 4; } message UpdateCubeSandboxResponse { diff --git a/CubeMaster/api/services/errorcode/v1/errorcode.pb.go b/CubeMaster/api/services/errorcode/v1/errorcode.pb.go index 06baa2e99..379b76fec 100644 --- a/CubeMaster/api/services/errorcode/v1/errorcode.pb.go +++ b/CubeMaster/api/services/errorcode/v1/errorcode.pb.go @@ -75,6 +75,7 @@ const ( ErrorCode_DestroyImageFailed ErrorCode = 130569 ErrorCode_TaskPauseFailed ErrorCode = 130588 ErrorCode_TaskResumeFailed ErrorCode = 130589 + ErrorCode_UpdateNetworkFailed ErrorCode = 130590 ErrorCode_InitCommandPathError ErrorCode = 130445 ErrorCode_ContainerStateExitedByUser ErrorCode = 130451 ErrorCode_PullImageFailed ErrorCode = 130456 @@ -144,6 +145,7 @@ var ( 130569: "DestroyImageFailed", 130588: "TaskPauseFailed", 130589: "TaskResumeFailed", + 130590: "UpdateNetworkFailed", 130445: "InitCommandPathError", 130451: "ContainerStateExitedByUser", 130456: "PullImageFailed", @@ -208,6 +210,7 @@ var ( "DestroyImageFailed": 130569, "TaskPauseFailed": 130588, "TaskResumeFailed": 130589, + "UpdateNetworkFailed": 130590, "InitCommandPathError": 130445, "ContainerStateExitedByUser": 130451, "PullImageFailed": 130456, @@ -312,7 +315,7 @@ const file_api_services_errorcode_v1_errorcode_proto_rawDesc = "" + ")api/services/errorcode/v1/errorcode.proto\x12\x1dcubelet.services.errorcode.v1\"c\n" + "\x03Ret\x12C\n" + "\bret_code\x18\x01 \x01(\x0e2(.cubelet.services.errorcode.v1.ErrorCodeR\aretCode\x12\x17\n" + - "\aret_msg\x18\x02 \x01(\tR\x06retMsg*\xe4\f\n" + + "\aret_msg\x18\x02 \x01(\tR\x06retMsg*\xff\f\n" + "\tErrorCode\x12\x06\n" + "\x02OK\x10\x00\x12\x14\n" + "\aUnknown\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\x01\x12\f\n" + @@ -360,7 +363,8 @@ const file_api_services_errorcode_v1_errorcode_proto_rawDesc = "" + "\x12PreConditionFailed\x10\x88\xfc\a\x12\x18\n" + "\x12DestroyImageFailed\x10\x89\xfc\a\x12\x15\n" + "\x0fTaskPauseFailed\x10\x9c\xfc\a\x12\x16\n" + - "\x10TaskResumeFailed\x10\x9d\xfc\a\x12\x1a\n" + + "\x10TaskResumeFailed\x10\x9d\xfc\a\x12\x19\n" + + "\x13UpdateNetworkFailed\x10\x9e\xfc\a\x12\x1a\n" + "\x14InitCommandPathError\x10\x8d\xfb\a\x12 \n" + "\x1aContainerStateExitedByUser\x10\x93\xfb\a\x12\x15\n" + "\x0fPullImageFailed\x10\x98\xfb\a\x12\x16\n" + diff --git a/CubeMaster/api/services/errorcode/v1/errorcode.proto b/CubeMaster/api/services/errorcode/v1/errorcode.proto index 804f38946..220eeeeff 100644 --- a/CubeMaster/api/services/errorcode/v1/errorcode.proto +++ b/CubeMaster/api/services/errorcode/v1/errorcode.proto @@ -57,6 +57,7 @@ enum ErrorCode { DestroyImageFailed = 130569; TaskPauseFailed = 130588; TaskResumeFailed = 130589; + UpdateNetworkFailed = 130590; InitCommandPathError = 130445; ContainerStateExitedByUser = 130451; diff --git a/CubeMaster/pkg/service/httpservice/cube/cube.go b/CubeMaster/pkg/service/httpservice/cube/cube.go index 8a6c66fa4..e1365909a 100644 --- a/CubeMaster/pkg/service/httpservice/cube/cube.go +++ b/CubeMaster/pkg/service/httpservice/cube/cube.go @@ -21,6 +21,7 @@ const ( SandboxInfoAction = "/sandbox/info" SandboxExecAction = "/sandbox/exec" SandboxUpdateAction = "/sandbox/update" + SandboxNetworkAction = "/sandbox/network" SandboxTimeoutAction = "/sandbox/timeout" SandboxRefreshAction = "/sandbox/refresh" SandboxCommitAction = "/sandbox/commit" diff --git a/CubeMaster/pkg/service/httpservice/cube/routes.go b/CubeMaster/pkg/service/httpservice/cube/routes.go index 999096231..ea9aca12e 100644 --- a/CubeMaster/pkg/service/httpservice/cube/routes.go +++ b/CubeMaster/pkg/service/httpservice/cube/routes.go @@ -26,6 +26,7 @@ func RegisterCubeRoutes(g *gin.RouterGroup) { g.POST(SandboxRollbackAction, handleSandboxRollbackAction) g.POST(SandboxAction+"/:sandbox_id/rollback", handleSandboxRollbackAction) g.POST(SandboxUpdateAction, handleUpdateAction) + g.POST(SandboxNetworkAction, handleSandboxNetworkAction) g.POST(SandboxTimeoutAction, handleSandboxTimeoutAction) g.POST(SandboxRefreshAction, handleSandboxRefreshAction) g.POST(SandboxExecAction, handleExecAction) diff --git a/CubeMaster/pkg/service/httpservice/cube/update.go b/CubeMaster/pkg/service/httpservice/cube/update.go index cab9f159a..fe6b59f5b 100644 --- a/CubeMaster/pkg/service/httpservice/cube/update.go +++ b/CubeMaster/pkg/service/httpservice/cube/update.go @@ -8,6 +8,7 @@ import ( "net/http" "github.com/gin-gonic/gin" + "github.com/google/uuid" "github.com/tencentcloud/CubeSandbox/CubeMaster/api/services/cubebox/v1" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/base/log" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/base/utils" @@ -53,3 +54,41 @@ func handleUpdateAction(c *gin.Context) { rsp = sandbox.Update(CubeLog.WithRequestTrace(ctx, rt), req) common.WriteAPI(c, rsp) } + +// handleSandboxNetworkAction serves POST /cube/sandbox/network, which replaces +// the egress policy of a running sandbox. +func handleSandboxNetworkAction(c *gin.Context) { + rt := CubeLog.GetTraceInfo(c.Request.Context()) + + req := &types.UpdateNetworkRequest{} + if err := utils.DecodeHttpBody(c.Request.Body, req); err != nil { + rt.RetCode = int64(errorcode.ErrorCode_MasterParamsError) + common.WriteAPI(c, &types.UpdateNetworkRes{ + Ret: &types.Ret{ + RetCode: int(errorcode.ErrorCode_MasterParamsError), + RetMsg: err.Error(), + }, + }) + return + } + if req.RequestID == "" { + req.RequestID = uuid.New().String() + } + if req.InstanceType == "" { + req.InstanceType = cubebox.InstanceType_cubebox.String() + } + rt.RequestID = req.RequestID + rt.InstanceID = req.SandboxID + rt.InstanceType = req.InstanceType + + ctx := log.WithLogger(c.Request.Context(), log.G(c.Request.Context()).WithFields(map[string]interface{}{ + "RequestId": req.RequestID, + "InstanceId": req.SandboxID, + "InstanceType": req.InstanceType, + })) + res := sandbox.UpdateNetwork(CubeLog.WithRequestTrace(ctx, rt), req) + if res != nil && res.Ret != nil { + rt.RetCode = int64(res.Ret.RetCode) + } + common.WriteAPI(c, res) +} diff --git a/CubeMaster/pkg/service/sandbox/sandbox_update.go b/CubeMaster/pkg/service/sandbox/sandbox_update.go index f1c9498ce..4fd118102 100644 --- a/CubeMaster/pkg/service/sandbox/sandbox_update.go +++ b/CubeMaster/pkg/service/sandbox/sandbox_update.go @@ -8,12 +8,16 @@ import ( "context" "errors" + "github.com/tencentcloud/CubeSandbox/CubeMaster/api/services/cubebox/v1" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/base/config" + "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/base/constants" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/base/log" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/base/utils" + "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/cubelet" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/errorcode" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/localcache" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/sandboxlock" + "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/sandboxspec" "github.com/tencentcloud/CubeSandbox/CubeMaster/pkg/service/sandbox/types" ) @@ -112,3 +116,133 @@ func applyLifecycleLockError(rsp *types.Res, err error) { rsp.Ret.RetMsg = err.Error() } } + +// UpdateNetwork implements POST /cube/sandbox/network: it replaces a running +// sandbox's egress policy in place. +// +// Held under the same per-sandbox lifecycle lock as pause/resume/delete, so a +// policy update can neither interleave with a pause that is packaging the +// sandbox nor with a delete that is tearing it down. +// +// The node is updated before sandboxspec because sandboxspec is what a later +// clone or resume replays: recording a policy the node rejected would hand that +// policy to every descendant of this sandbox. A failed spec write is logged and +// not surfaced — the update itself did take effect — but it does mean a clone +// taken before the next successful write inherits the older policy. +func UpdateNetwork(ctx context.Context, req *types.UpdateNetworkRequest) (rsp *types.UpdateNetworkRes) { + rsp = &types.UpdateNetworkRes{ + RequestID: req.RequestID, + SandboxID: req.SandboxID, + Ret: &types.Ret{ + RetCode: int(errorcode.ErrorCode_Success), + RetMsg: errorcode.ErrorCode_Success.String(), + }, + } + if req.SandboxID == "" { + rsp.Ret.RetCode = int(errorcode.ErrorCode_MasterParamsError) + rsp.Ret.RetMsg = "should provide SandboxID" + return + } + if req.CubeNetworkConfig == nil { + rsp.Ret.RetCode = int(errorcode.ErrorCode_MasterParamsError) + rsp.Ret.RetMsg = "should provide cube_network_config" + return + } + if err := validateCubeNetworkConfig(req.CubeNetworkConfig); err != nil { + rsp.Ret.RetCode = int(errorcode.ErrorCode_MasterParamsError) + rsp.Ret.RetMsg = err.Error() + return + } + if r := normalizeSandboxIDInReq(ctx, &req.SandboxID); r != nil { + rsp.Ret = r + rsp.SandboxID = req.SandboxID + return + } + rsp.SandboxID = req.SandboxID + + lockErr := sandboxlock.WithLock(ctx, req.SandboxID, sandboxlock.Options{ + Value: "network", + TTL: sandboxlock.LifecycleTTL, + }, func(ctx context.Context) error { + // A client disconnect must not abandon the update between the node and + // sandboxspec, which would leave the two disagreeing until the next write. + ctx = context.WithoutCancel(ctx) + hostIP, ok := resolveSandboxHostIP(ctx, req.SandboxID) + if !ok { + rsp.Ret.RetCode = int(errorcode.ErrorCode_MasterParamsError) + rsp.Ret.RetMsg = "sandbox not found" + return nil + } + *rsp.Ret = *updateSandboxNetworkOnNode(ctx, req, hostIP) + return nil + }) + if lockErr != nil { + applyLifecycleLockError(&types.Res{Ret: rsp.Ret}, lockErr) + } + return +} + +// updateSandboxNetworkOnNode pushes the policy to Cubelet and, once the node has +// accepted it, records it as the sandbox's canonical spec. +func updateSandboxNetworkOnNode(ctx context.Context, req *types.UpdateNetworkRequest, hostIP string) *types.Ret { + cubeletReq := &cubebox.UpdateCubeSandboxRequest{ + RequestID: req.RequestID, + SandboxID: req.SandboxID, + Annotations: map[string]string{ + constants.CubeAnnotationsUpdateAction: "network", + constants.CubeAnnotationsInsType: req.InstanceType, + }, + CubeNetworkConfig: mapCubeNetworkConfig(req.CubeNetworkConfig), + } + cubeRsp, err := cubelet.Update(ctx, cubelet.GetCubeletAddr(hostIP), cubeletReq) + if err != nil || cubeRsp.GetRet() == nil { + msg := "cubelet update network response is nil" + if err != nil { + msg = err.Error() + } + return &types.Ret{RetCode: int(errorcode.ErrorCode_ReqCubeAPIFailed), RetMsg: msg} + } + result := &types.Ret{ + RetCode: int(cubeRsp.GetRet().GetRetCode()), + RetMsg: cubeRsp.GetRet().GetRetMsg(), + } + if result.RetCode != int(errorcode.ErrorCode_Success) { + return result + } + persistSandboxNetworkSpec(ctx, req, hostIP) + return result +} + +// persistSandboxNetworkSpec rewrites the stored create spec so clone, snapshot +// and resume replay the new policy instead of the one the sandbox was born +// with. Best-effort by design: the node already enforces the new policy, and +// sandboxspec is recovery-friendly. +func persistSandboxNetworkSpec(ctx context.Context, req *types.UpdateNetworkRequest, hostIP string) { + spec, err := sandboxspec.Get(ctx, req.SandboxID) + if err != nil || spec == nil { + log.G(ctx).Warnf("update network: load sandbox spec failed, clone will replay the old policy: sandbox=%s err=%v", + req.SandboxID, err) + return + } + spec.CubeNetworkConfig = req.CubeNetworkConfig.DeepCopy() + opts := sandboxspec.PutOptions{HostIP: hostIP} + if n, ok := localcache.GetNodesByIp(hostIP); ok { + opts.HostID = n.ID() + } + if err := sandboxspec.Put(ctx, req.SandboxID, spec, opts); err != nil { + log.G(ctx).Errorf("update network: persist sandbox spec failed, clone will replay the old policy: sandbox=%s err=%v", + req.SandboxID, err) + } +} + +// resolveSandboxHostIP finds the node currently hosting the sandbox, trying the +// same sources as the pause/resume path. +func resolveSandboxHostIP(ctx context.Context, sandboxID string) (string, bool) { + if v := localcache.GetSandboxCache(sandboxID); v != nil { + return v.HostIP, true + } + if proxyMap, ok := localcache.GetSandboxProxyMap(ctx, sandboxID); ok { + return proxyMap.HostIP, true + } + return resolvePauseHostIP(ctx, sandboxID) +} diff --git a/CubeMaster/pkg/service/sandbox/types/types.go b/CubeMaster/pkg/service/sandbox/types/types.go index 7658c379f..26db3caac 100644 --- a/CubeMaster/pkg/service/sandbox/types/types.go +++ b/CubeMaster/pkg/service/sandbox/types/types.go @@ -808,6 +808,23 @@ type UpdateRequest struct { Action string `json:"action"` } +// UpdateNetworkRequest is the wire shape for POST /cube/sandbox/network. The +// config is the complete desired egress policy, not a patch: an omitted or +// empty field clears whatever the sandbox currently has. +type UpdateNetworkRequest struct { + RequestID string `json:"requestID"` + SandboxID string `json:"sandboxID"` + InstanceType string `json:"instanceType"` + CubeNetworkConfig *CubeNetworkConfig `json:"cube_network_config"` +} + +// UpdateNetworkRes is the master-side response for /cube/sandbox/network. +type UpdateNetworkRes struct { + RequestID string `json:"requestID,omitempty"` + SandboxID string `json:"sandboxID,omitempty"` + Ret *Ret `json:"ret,omitempty"` +} + // SetTimeoutRequest is the wire shape for POST /cube/sandbox/timeout. // Mirrors CubeAPI's SandboxTimeoutRequest field-for-field. `timeout` is the // new idle TTL in seconds counted from "now": the master refreshes the diff --git a/CubeMaster/pkg/service/sandbox/util.go b/CubeMaster/pkg/service/sandbox/util.go index 6ac4dd44f..83716f210 100644 --- a/CubeMaster/pkg/service/sandbox/util.go +++ b/CubeMaster/pkg/service/sandbox/util.go @@ -87,22 +87,29 @@ func checkParam(req *types.CreateCubeSandboxReq) error { return ret.Err(errorcode.ErrorCode_MasterParamsError, "containers param is nil") } - if req.CubeNetworkConfig != nil { - if req.CubeNetworkConfig.MaskRequestHost != nil { - if err := validateMaskRequestHost(*req.CubeNetworkConfig.MaskRequestHost); err != nil { - return ret.Err(errorcode.ErrorCode_MasterParamsError, err.Error()) - } + return validateCubeNetworkConfig(req.CubeNetworkConfig) +} + +// validateCubeNetworkConfig runs the server-side egress policy checks shared by +// sandbox creation and in-place network updates, so the update path cannot +// accept a policy that creation would have rejected. +func validateCubeNetworkConfig(cfg *types.CubeNetworkConfig) error { + if cfg == nil { + return nil + } + if cfg.MaskRequestHost != nil { + if err := validateMaskRequestHost(*cfg.MaskRequestHost); err != nil { + return ret.Err(errorcode.ErrorCode_MasterParamsError, err.Error()) } - for i, rule := range req.CubeNetworkConfig.Rules { - if rule == nil { - continue - } - if err := validateEgressRuleMatch(rule.Match, i); err != nil { - return ret.Err(errorcode.ErrorCode_MasterParamsError, err.Error()) - } + } + for i, rule := range cfg.Rules { + if rule == nil { + continue + } + if err := validateEgressRuleMatch(rule.Match, i); err != nil { + return ret.Err(errorcode.ErrorCode_MasterParamsError, err.Error()) } } - return nil } diff --git a/CubeNet/cubevs/cubevs.go b/CubeNet/cubevs/cubevs.go index 8d87a242d..55ac3106a 100644 --- a/CubeNet/cubevs/cubevs.go +++ b/CubeNet/cubevs/cubevs.go @@ -54,12 +54,19 @@ type TAPDevice struct { // mvmMetadata is used to retrieve BPF map values. // The struct layout should be exactly the same as BPF side. +// mvmMetadata mirrors struct mvm_meta on the BPF side. PolicyVersion is the +// per-sandbox network-policy generation; the datapath compares it against the +// copy cached in each nat_session to decide whether an established flow needs +// re-evaluating. Version is a different thing entirely (TAP generation, part of +// session_key) and must not be reused for it. type mvmMetadata struct { Version uint32 IP uint32 UUID [64]byte DNSPolicyFlags uint8 - Reserved [55]uint8 + Reserved0 [3]uint8 + PolicyVersion uint32 + Reserved [48]uint8 } // TCDirection is used to specified attach point of a TC filter. diff --git a/CubeNet/cubevs/dnspolicy.go b/CubeNet/cubevs/dnspolicy.go index 836339690..6eb64e73e 100644 --- a/CubeNet/cubevs/dnspolicy.go +++ b/CubeNet/cubevs/dnspolicy.go @@ -196,6 +196,46 @@ func cleanupDNSAllow(ifindex uint32) error { return flushDNSAllowInnerMap(inner) } +// syncDNSAllowInner converges dns_allow_v2 for one TAP on the desired rules. +// +// Values are written as-is rather than through updateDNSAllowRule: that helper +// unions the installed flags and port tuples into the new value, which is right +// when several rules of one apply share a key, but on an update it would keep +// ports the caller just removed. buildNetPolicyPlan already merges same-key +// rules in userspace, so rules holds exactly the desired state. +func syncDNSAllowInner(ifindex uint32, rules []dnsAllowRule) error { + dnsAllow, err := loadPinnedMap(MapNameDNSAllowV2) + if err != nil { + return err + } + defer dnsAllow.Close() + + inner, err := acquireInnerMap(dnsAllow, ifindex, MapNameDNSAllowV2, newInnerDNSAllowMap) + if err != nil { + return err + } + + desired := make(map[dnsAllowKey]struct{}, len(rules)) + for _, rule := range rules { + desired[rule.key] = struct{}{} + } + + stale, err := staleKeys(inner, desired, func(*dnsAllowValue) bool { return true }) + if err != nil { + return err + } + if err := deleteKeys(inner, stale); err != nil { + return err + } + + for _, rule := range rules { + if err := inner.Update(&rule.key, &rule.value, ebpf.UpdateAny); err != nil { + return fmt.Errorf("dns allow update failed: %w, domain: %s", err, rule.domain) + } + } + return nil +} + // applyDNSAllow installs DNS allow rules parsed from MVMOptions. func applyDNSAllow(ifindex uint32, rules []dnsAllowRule, replace bool) error { if len(rules) == 0 && !replace { diff --git a/CubeNet/cubevs/egress_policy_test.go b/CubeNet/cubevs/egress_policy_test.go index 16eb34562..1758396e9 100644 --- a/CubeNet/cubevs/egress_policy_test.go +++ b/CubeNet/cubevs/egress_policy_test.go @@ -19,6 +19,7 @@ const ( type egressPolicyTestEnv struct { program *ebpf.Program + recheckProgram *ebpf.Program allowOut *ebpf.Map denyOut *ebpf.Map allowInnerSpec *ebpf.MapSpec @@ -60,12 +61,13 @@ func loadEgressPolicyTestEnv(t *testing.T) *egressPolicyTestEnv { env := &egressPolicyTestEnv{ program: coll.Programs["test_classify_egress_flow"], + recheckProgram: coll.Programs["test_session_policy_revoked"], allowOut: coll.Maps["allow_out_v3"], denyOut: coll.Maps["deny_out"], allowInnerSpec: allowInnerSpec, denyInnerSpec: denyInnerSpec, } - if env.program == nil || env.allowOut == nil || env.denyOut == nil { + if env.program == nil || env.recheckProgram == nil || env.allowOut == nil || env.denyOut == nil { t.Fatal("loaded egress policy program or maps missing") } return env @@ -307,3 +309,188 @@ func TestClassifyEgressFlowExpiredAllow(t *testing.T) { }) } } + +const sessionRecheckCaseLen = 28 + +// sessionRecheckCase mirrors struct session_recheck_case in +// egress_policy_test.bpf.c. +type sessionRecheckCase struct { + ifindex uint32 + daddr uint32 + sessPolicyVersion uint32 + metaPolicyVersion uint32 + dport uint16 + packetClass uint8 + l7Scheme uint8 +} + +type sessionRecheckResult struct { + revoked bool + sessPolicyVersion uint32 +} + +func runSessionRecheckCase(t *testing.T, prog *ebpf.Program, tc sessionRecheckCase) sessionRecheckResult { + t.Helper() + + data := make([]byte, sessionRecheckCaseLen) + binary.LittleEndian.PutUint32(data[0:4], tc.ifindex) + binary.LittleEndian.PutUint32(data[4:8], tc.daddr) + binary.LittleEndian.PutUint32(data[8:12], tc.sessPolicyVersion) + binary.LittleEndian.PutUint32(data[12:16], tc.metaPolicyVersion) + binary.LittleEndian.PutUint16(data[20:22], tc.dport) + data[22] = tc.packetClass + data[23] = tc.l7Scheme + + ret, out, err := prog.Test(data) + if err != nil { + if bpfTestUnavailable(err) { + t.Skipf("kernel BPF policy test-run unavailable: %v", err) + } + t.Fatalf("run session recheck test: %v", err) + } + if ret != 0 { + t.Fatalf("test_session_policy_revoked returned %d, want TC_ACT_OK", ret) + } + if len(out) < sessionRecheckCaseLen { + t.Fatalf("test output length=%d, want >=%d", len(out), sessionRecheckCaseLen) + } + return sessionRecheckResult{ + revoked: out[24] != 0, + sessPolicyVersion: binary.LittleEndian.Uint32(out[16:20]), + } +} + +// TestSessionPolicyRevoked exercises the datapath re-check an update relies on: +// an established flow is judged against the current policy the first time it is +// seen under a new generation, and the decision is then cached. +func TestSessionPolicyRevoked(t *testing.T) { + const ( + ifindex = uint32(700) + snatPort = uint16(0x5000) // 80, network byte order on little-endian + otherPort = uint16(0xBB01) // 443 + packetSNAT = uint8(0) + packetL7 = uint8(1) + schemeNone = uint8(0) + schemeHTTP = uint8(1) + schemeHTTPS = uint8(2) + ) + + env := loadEgressPolicyTestEnv(t) + allowInner, denyInner := env.attachInnerMaps(t, ifindex) + + allowed := mustParseCIDRForTest(t, "192.0.2.30") + l7Host := mustParseCIDRForTest(t, "192.0.2.31") + denied := mustParseCIDRForTest(t, "192.0.2.32") + + // classify_egress_flow defaults to SNAT, so "no longer allowed" has to be + // expressed as an explicit deny rather than the absence of an allow. + denyKey := lpmKey{Prefixlen: 32, IP: denied.IP} + denyVal := uint32(netPolicyValueStatic) + if err := denyInner.Update(&denyKey, &denyVal, ebpf.UpdateAny); err != nil { + t.Fatalf("seed deny: %v", err) + } + + // A plain allow for one host, and an HTTPS-only L7 rule for another. + plainKey := lpmKeyV3{Prefixlen: 32, IP: allowed.IP} + if err := allowInner.Update(&plainKey, &netPolicyValueV3{KeyPrefixlen: 32}, ebpf.UpdateAny); err != nil { + t.Fatalf("seed plain allow: %v", err) + } + l7Key := lpmKeyV3{Prefixlen: 48, IP: l7Host.IP, Port: otherPort} + if err := allowInner.Update(&l7Key, &netPolicyValueV3{ + Flags: netPolicyFlagL7Required, Scheme: L7SchemeHTTPS, KeyPrefixlen: 48, + }, ebpf.UpdateAny); err != nil { + t.Fatalf("seed L7 allow: %v", err) + } + + tests := []struct { + name string + tc sessionRecheckCase + wantRevoked bool + wantVersion uint32 + }{ + { + // Denied by the current policy, but the generation matches, so the + // flow keeps running on its cached verdict. + name: "same generation is not re-evaluated", + tc: sessionRecheckCase{ + ifindex: ifindex, daddr: denied.IP, dport: snatPort, + sessPolicyVersion: 5, metaPolicyVersion: 5, + }, + wantRevoked: false, + wantVersion: 5, + }, + { + name: "still allowed under the new generation is restamped", + tc: sessionRecheckCase{ + ifindex: ifindex, daddr: allowed.IP, dport: snatPort, + sessPolicyVersion: 5, metaPolicyVersion: 6, + }, + wantRevoked: false, + wantVersion: 6, + }, + { + name: "no longer allowed is revoked", + tc: sessionRecheckCase{ + ifindex: ifindex, daddr: denied.IP, dport: snatPort, + sessPolicyVersion: 5, metaPolicyVersion: 6, + }, + wantRevoked: true, + wantVersion: 5, + }, + { + // SNAT and L7 disagree on the reply tuple and on who terminates + // the connection, so a flow cannot migrate between them. + name: "verdict change SNAT to L7 is revoked", + tc: sessionRecheckCase{ + ifindex: ifindex, daddr: l7Host.IP, dport: otherPort, + packetClass: packetSNAT, l7Scheme: schemeNone, + sessPolicyVersion: 5, metaPolicyVersion: 6, + }, + wantRevoked: true, + wantVersion: 5, + }, + { + name: "verdict change L7 scheme is revoked", + tc: sessionRecheckCase{ + ifindex: ifindex, daddr: l7Host.IP, dport: otherPort, + packetClass: packetL7, l7Scheme: schemeHTTP, + sessPolicyVersion: 5, metaPolicyVersion: 6, + }, + wantRevoked: true, + wantVersion: 5, + }, + { + name: "unchanged L7 verdict is restamped", + tc: sessionRecheckCase{ + ifindex: ifindex, daddr: l7Host.IP, dport: otherPort, + packetClass: packetL7, l7Scheme: schemeHTTPS, + sessPolicyVersion: 5, metaPolicyVersion: 6, + }, + wantRevoked: false, + wantVersion: 6, + }, + { + // Sessions created before this field existed carry 0, so they are + // re-checked exactly once after an upgrade instead of being trusted. + name: "unset generation is treated as stale", + tc: sessionRecheckCase{ + ifindex: ifindex, daddr: allowed.IP, dport: snatPort, + sessPolicyVersion: 0, metaPolicyVersion: 1, + }, + wantRevoked: false, + wantVersion: 1, + }, + } + + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := runSessionRecheckCase(t, env.recheckProgram, tt.tc) + if got.revoked != tt.wantRevoked { + t.Errorf("revoked=%v, want %v", got.revoked, tt.wantRevoked) + } + if got.sessPolicyVersion != tt.wantVersion { + t.Errorf("session policy_version=%d, want %d", got.sessPolicyVersion, tt.wantVersion) + } + }) + } +} diff --git a/CubeNet/cubevs/netpolicy.go b/CubeNet/cubevs/netpolicy.go index 5acf9c185..8f208ff1e 100644 --- a/CubeNet/cubevs/netpolicy.go +++ b/CubeNet/cubevs/netpolicy.go @@ -1009,7 +1009,7 @@ func splitAllowOutTargets(targets []string) ([]string, []string, error) { if net.ParseIP(target) != nil || isDottedDecimalLikeTarget(target) { return nil, nil, fmt.Errorf("unsupported allow_out IP target: %s", target) //nolint:err113 } - if !isDNSAllowTarget(target) { + if !IsAllowOutDomainTarget(target) { return nil, nil, fmt.Errorf("invalid allow_out domain target: %s", target) //nolint:err113 } domains = append(domains, target) @@ -1044,6 +1044,25 @@ func isDottedDecimalLikeTarget(target string) bool { return true } +// IsAllowOutDomainTarget reports whether an allow_out target is installed as a +// dns_allow_v2 domain rule rather than an allow_out_v3 CIDR. +// +// IP and CIDR literals take precedence, exactly as splitAllowOutTargets decides +// it: isDNSAllowTarget() alone accepts an all-numeric name like "10.0.0.1" +// because digits are valid DNS label characters, so asking it directly reports a +// bare IPv4 literal as a domain. Callers outside this package need the install +// decision, not the name-shape check, so this is the one they get. +func IsAllowOutDomainTarget(target string) bool { + target = strings.TrimSpace(target) + if target == "" || isIPv4Target(target) || strings.Contains(target, "/") { + return false + } + if net.ParseIP(target) != nil || isDottedDecimalLikeTarget(target) { + return false + } + return isDNSAllowTarget(target) +} + func isDNSAllowTarget(target string) bool { domain := strings.ToLower(strings.TrimSuffix(target, ".")) if strings.HasPrefix(domain, "*.") { @@ -1106,62 +1125,81 @@ func populateAllowOutInnerMap(outerMap *ebpf.Map, ifindex uint32, entries []allo return populateAllowOutInner(inner, entries) } +// allowOutRow is one inner-map row a plan entry occupies. +type allowOutRow struct { + key lpmKeyV3 + val netPolicyValueV3 + // mergeLearnedFlags marks the exact (ip, port)/48 rows an L7 rule owns, + // which are the only keys a DNS-learned entry can also land on. Plain + // any-port rows overwrite outright: nothing else writes that key, so + // merging there would just resurrect stale flags. + mergeLearnedFlags bool +} + +// expandAllowOutEntry materialises the rows one plan entry occupies: an L7 +// entry becomes one exact (ip, port)/48 row per (port, scheme) tuple — an empty +// port set means the default {80/http, 443/https} — plus, when the host is also +// in plain allow_out (netPolicyFlagL3Allowed), the any-port row that keeps L3 +// access on every other port. A plain entry is a single any-port row. +// +// Shared by the populate and diff paths so the set of keys written can never +// drift from the set of keys considered current. +func expandAllowOutEntry(entry allowOutPolicyEntry) []allowOutRow { + plainRow := func(flags uint8) allowOutRow { + return allowOutRow{ + key: lpmKeyV3{Prefixlen: entry.key.Prefixlen, IP: entry.key.IP}, + val: netPolicyValueV3{Flags: flags, KeyPrefixlen: uint8(entry.key.Prefixlen)}, + } + } + if entry.flags&netPolicyFlagL7Required == 0 { + return []allowOutRow{plainRow(entry.flags)} + } + + ports := entry.ports + if len(ports) == 0 { + ports = expandDefaultPortSet() + } + rows := make([]allowOutRow, 0, len(ports)+1) + for _, p := range ports { + rows = append(rows, allowOutRow{ + key: lpmKeyV3{Prefixlen: 48, IP: entry.key.IP, Port: p.Port}, + val: netPolicyValueV3{Flags: entry.flags, Scheme: p.Scheme, KeyPrefixlen: 48}, + mergeLearnedFlags: true, + }) + } + if entry.flags&netPolicyFlagL3Allowed != 0 { + // The /48 rows win for the rule's ports (longest prefix); this one + // covers every other port via plain SNAT. Strip the marker bits so it + // reads as a plain allow. + rows = append(rows, plainRow(entry.flags&^(netPolicyFlagL7Required|netPolicyFlagL3Allowed))) + } + return rows +} + func populateAllowOutInner(inner *ebpf.Map, entries []allowOutPolicyEntry) error { for _, entry := range entries { - if entry.flags&netPolicyFlagL7Required != 0 { - ports := entry.ports - if len(ports) == 0 { - ports = expandDefaultPortSet() - } - for _, p := range ports { - key := lpmKeyV3{Prefixlen: 48, IP: entry.key.IP, Port: p.Port} - val := netPolicyValueV3{Flags: entry.flags, Scheme: p.Scheme, KeyPrefixlen: 48} + for _, row := range expandAllowOutEntry(entry) { + val := row.val + if row.mergeLearnedFlags { var oldVal netPolicyValueV3 - lerr := inner.Lookup(&key, &oldVal) - switch { - case lerr == nil: - // LPM lookup is longest-prefix: only merge with - // an entry written under the EXACT same key, - // never with a shorter covering entry (whose - // flags would otherwise leak into this /48). - // Flags only: the static (zero) expiry wins over - // a learned same-key entry, so the entry becomes - // permanent rather than ageing out at the old TTL. - if oldVal.KeyPrefixlen == uint8(key.Prefixlen) { + switch err := inner.Lookup(&row.key, &oldVal); { + case err == nil: + // LPM lookup is longest-prefix: only merge with an entry + // written under the EXACT same key, never with a shorter + // covering entry (whose flags would otherwise leak in). + // Flags only: the static (zero) expiry wins over a learned + // same-key entry, so the entry becomes permanent rather + // than ageing out at the old TTL. + if oldVal.KeyPrefixlen == uint8(row.key.Prefixlen) { val.Flags |= oldVal.Flags } - case !errors.Is(lerr, ebpf.ErrKeyNotExist): - return fmt.Errorf("inner map lookup failed: %w, cidr: %s", lerr, entry.source) - } - if uerr := inner.Update(&key, &val, ebpf.UpdateAny); uerr != nil { - return fmt.Errorf("inner map update failed: %w, cidr: %s", uerr, entry.source) + case !errors.Is(err, ebpf.ErrKeyNotExist): + return fmt.Errorf("inner map lookup failed: %w, cidr: %s", err, entry.source) } } - - // Coexistence: this host is also in plain allow_out, so write a - // plain /32 any-port entry alongside the /48 L7 entries. The /48 - // (longest-prefix) match wins for the rule's ports; the /32 covers - // all other ports via plain SNAT. Strip the L7/L3 marker bits so - // the entry reads as a plain allow. - if entry.flags&netPolicyFlagL3Allowed != 0 { - plainKey := lpmKeyV3{Prefixlen: entry.key.Prefixlen, IP: entry.key.IP, Port: 0} - plainVal := netPolicyValueV3{ - Flags: entry.flags &^ (netPolicyFlagL7Required | netPolicyFlagL3Allowed), - Scheme: L7SchemeNone, - KeyPrefixlen: uint8(plainKey.Prefixlen), - } - if uerr := inner.Update(&plainKey, &plainVal, ebpf.UpdateAny); uerr != nil { - return fmt.Errorf("inner map update failed: %w, cidr: %s", uerr, entry.source) - } + if err := inner.Update(&row.key, &val, ebpf.UpdateAny); err != nil { + return fmt.Errorf("inner map update failed: %w, cidr: %s", err, entry.source) } - continue - } - - // Plain allow: ip-only / subnet key with port = 0, scheme = NONE. - key := lpmKeyV3{Prefixlen: entry.key.Prefixlen, IP: entry.key.IP, Port: 0} - val := netPolicyValueV3{Flags: entry.flags, KeyPrefixlen: uint8(key.Prefixlen)} - if uerr := inner.Update(&key, &val, ebpf.UpdateAny); uerr != nil { - return fmt.Errorf("inner map update failed: %w, cidr: %s", uerr, entry.source) } } return nil @@ -1174,6 +1212,183 @@ func netPolicyValueV3Expired(value netPolicyValueV3, now uint64) bool { return value.ExpiresAtNS != 0 && value.ExpiresAtNS <= now } +// UpdateTAPDevicePolicy converges an already-registered TAP's egress policy on +// opts, then bumps the sandbox's policy generation so the datapath re-evaluates +// established flows. +// +// It is the third apply mode alongside applyNetPolicy (additive, create path) +// and replaceNetPolicy (flush + refill, recovery path). Neither fits a live +// sandbox: flushing would blank the policy for as long as the refill takes, and +// swapping the inner map would defeat the HashOfMaps inner cache and pay a +// synchronize_rcu() on every update. So this diffs against what is installed +// and issues only the required per-entry writes. +// +// Two ordering rules: +// +// - The generation bump is last. A failure before it leaves established flows +// on their cached verdict instead of judging them against a half-applied +// map — and because a revoked flow is deleted outright, a premature bump +// could retire flows the finished policy would have allowed. +// - Within each map, revocations land before additions. No intermediate state +// is then more permissive than both the old and the new policy. +// +// Nothing is rolled back on failure. The diff is computed from the live maps, +// so replaying the same request converges; and the caller's durable state is +// only written after this returns, so a restart re-applies the previous policy +// in full. +func UpdateTAPDevicePolicy(ifindex uint32, opts MVMOptions) error { + // Validate the whole desired state before touching anything: a rejected + // plan must leave both the L3 maps and the caller's L7 push untouched. + plan, err := buildNetPolicyPlan(opts) + if err != nil { + return err + } + + if err := syncAllowOutInner(ifindex, plan.allowOutEntries); err != nil { + return fmt.Errorf("sync %s failed: %w", MapNameAllowOutV3, err) + } + if err := syncDenyOutInner(ifindex, effectiveDenyOutEntriesForReplace(plan)); err != nil { + return fmt.Errorf("sync %s failed: %w", MapNameDenyOut, err) + } + if err := syncDNSAllowInner(ifindex, plan.dnsAllowRules); err != nil { + return fmt.Errorf("sync %s failed: %w", MapNameDNSAllowV2, err) + } + if err := setDNSPolicyFlags(ifindex, plan.dnsPolicyFlags); err != nil { + return err + } + return bumpPolicyVersion(ifindex) +} + +// syncAllowOutInner converges allow_out_v3 for one TAP on the desired entries. +// +// Only static rows are managed. DNS-learned rows (non-zero expiry) are left +// exactly as they are: their lifetime belongs to the TTL and the reaper, and a +// revoked domain rule stops producing new ones as soon as dns_allow_v2 is +// synced. Removing a domain therefore takes effect for already-resolved IPs +// only once they age out. +func syncAllowOutInner(ifindex uint32, entries []allowOutPolicyEntry) error { + outer, err := loadPinnedMap(MapNameAllowOutV3) + if err != nil { + return err + } + defer outer.Close() + + inner, err := acquireInnerMap(outer, ifindex, MapNameAllowOutV3, newInnerAllowOutMap) + if err != nil { + return err + } + + desired := make(map[lpmKeyV3]struct{}, len(entries)) + for _, entry := range entries { + for _, row := range expandAllowOutEntry(entry) { + desired[row.key] = struct{}{} + } + } + + stale, err := staleKeys(inner, desired, func(v *netPolicyValueV3) bool { + return v.ExpiresAtNS == 0 + }) + if err != nil { + return err + } + if err := deleteKeys(inner, stale); err != nil { + return err + } + return populateAllowOutInner(inner, entries) +} + +// syncDenyOutInner converges deny_out for one TAP. Every row is managed, so the +// caller must pass the effective set including the always-denied private and +// link-local ranges — otherwise an update would drop the invariant deny rules. +func syncDenyOutInner(ifindex uint32, entries []denyOutPolicyEntry) error { + outer, err := loadPinnedMap(MapNameDenyOut) + if err != nil { + return err + } + defer outer.Close() + + inner, err := acquireInnerMap(outer, ifindex, MapNameDenyOut, newInnerLPMMap) + if err != nil { + return err + } + + desired := make(map[lpmKey]struct{}, len(entries)) + for _, entry := range entries { + desired[entry.key] = struct{}{} + } + + stale, err := staleKeys(inner, desired, func(*uint32) bool { return true }) + if err != nil { + return err + } + if err := deleteKeys(inner, stale); err != nil { + return err + } + return populateDenyOutInner(inner, entries) +} + +// staleKeys returns the managed keys currently in inner that desired no longer +// covers. managed decides which rows this policy owns; rows it rejects are left +// alone. +// +// Keys are collected first and deleted afterwards: deleting while iterating a +// BPF hash map can make the cursor skip live entries. +func staleKeys[K comparable, V any](inner *ebpf.Map, desired map[K]struct{}, managed func(*V) bool) ([]K, error) { + var ( + key K + value V + stale []K + ) + iter := inner.Iterate() + for iter.Next(&key, &value) { + if !managed(&value) { + continue + } + if _, keep := desired[key]; !keep { + stale = append(stale, key) + } + } + if err := iter.Err(); err != nil { + return nil, fmt.Errorf("inner map iterate failed: %w", err) + } + return stale, nil +} + +func deleteKeys[K any](inner *ebpf.Map, keys []K) error { + for i := range keys { + if err := inner.Delete(&keys[i]); err != nil && !errors.Is(err, ebpf.ErrKeyNotExist) { + return fmt.Errorf("inner map delete failed: %w", err) + } + } + return nil +} + +// bumpPolicyVersion advances the sandbox's policy generation. Every established +// flow compares its cached copy against this value on the next packet, so this +// is what makes an update reach traffic that already exists. +func bumpPolicyVersion(ifindex uint32) error { + m, err := loadPinnedMap(MapNameIfindexToMVMMetadata) + if err != nil { + return err + } + defer m.Close() + + var meta mvmMetadata + if err := m.Lookup(&ifindex, &meta); err != nil { + return fmt.Errorf("map.Lookup failed: %w, name: %s", err, MapNameIfindexToMVMMetadata) + } + meta.PolicyVersion++ + if meta.PolicyVersion == 0 { + // 0 is reserved for "never set"; skip it so a wrap cannot make live + // sessions look freshly stamped. + meta.PolicyVersion = 1 + } + if err := m.Update(&ifindex, &meta, ebpf.UpdateAny); err != nil { + return fmt.Errorf("map.Update failed: %w, name: %s", err, MapNameIfindexToMVMMetadata) + } + return nil +} + // applyNetPolicy configures egress network policy for the given ifindex // based on MVMOptions. // diff --git a/CubeNet/cubevs/netpolicy_test.go b/CubeNet/cubevs/netpolicy_test.go index ffa615aeb..3388d22f5 100644 --- a/CubeNet/cubevs/netpolicy_test.go +++ b/CubeNet/cubevs/netpolicy_test.go @@ -2,6 +2,7 @@ package cubevs import ( "fmt" + "net" "reflect" "strings" "testing" @@ -1227,3 +1228,324 @@ func TestPopulateAllowOutPlainStaticOverwritesLearnedUnconditionally(t *testing. t.Fatalf("scheme=%d, want NONE", got.Scheme) } } + +// pinPolicyMaps mounts a scratch bpffs and pins the outer maps the update path +// loads by name, then creates this ifindex's inner maps. It leaves the TAP with +// the same map shape a freshly registered sandbox has. +func pinPolicyMaps(t *testing.T, ifindex uint32) { + t.Helper() + mountBpffs(t) + + allowOut := newAllowOutV3OuterMap(t) + if err := allowOut.Pin(pinPath(MapNameAllowOutV3)); err != nil { + t.Fatalf("pin %s: %v", MapNameAllowOutV3, err) + } + denyOut := newDenyOutOuterMap(t) + if err := denyOut.Pin(pinPath(MapNameDenyOut)); err != nil { + t.Fatalf("pin %s: %v", MapNameDenyOut, err) + } + dnsAllow := newDNSAllowOuterMap(t) + if err := dnsAllow.Pin(pinPath(MapNameDNSAllowV2)); err != nil { + t.Fatalf("pin %s: %v", MapNameDNSAllowV2, err) + } + if err := initNetPolicy(ifindex); err != nil { + t.Fatalf("initNetPolicy: %v", err) + } +} + +// newDenyOutOuterMap builds a deny_out outer map with the production shape. +func newDenyOutOuterMap(t *testing.T) *ebpf.Map { + t.Helper() + outer, err := ebpf.NewMap(&ebpf.MapSpec{ + Type: ebpf.HashOfMaps, + KeySize: uint32(unsafe.Sizeof(uint32(0))), + ValueSize: uint32(unsafe.Sizeof(uint32(0))), + MaxEntries: maxNetPolicyEntries, + InnerMap: &ebpf.MapSpec{ + Type: ebpf.LPMTrie, + KeySize: uint32(unsafe.Sizeof(lpmKey{})), + ValueSize: uint32(unsafe.Sizeof(uint32(0))), + MaxEntries: maxNetPolicyEntries, + Flags: unix.BPF_F_NO_PREALLOC, + }, + }) + if err != nil { + t.Fatalf("create deny_out outer map: %v", err) + } + t.Cleanup(func() { outer.Close() }) + return outer +} + +// pinTAPMetadataMaps pins the two maps UpsertTAPDeviceMetadata writes. +func pinTAPMetadataMaps(t *testing.T) { + t.Helper() + pin := func(name string, valueSize uint32) { + m, err := ebpf.NewMap(&ebpf.MapSpec{ + Type: ebpf.Hash, + KeySize: uint32(unsafe.Sizeof(uint32(0))), + ValueSize: valueSize, + MaxEntries: maxNetPolicyEntries, + }) + if err != nil { + t.Fatalf("create %s: %v", name, err) + } + t.Cleanup(func() { m.Close() }) + if err := m.Pin(pinPath(name)); err != nil { + t.Fatalf("pin %s: %v", name, err) + } + } + pin(MapNameIfindexToMVMMetadata, uint32(unsafe.Sizeof(mvmMetadata{}))) + pin(MapNameMVMIPToIfindex, uint32(unsafe.Sizeof(uint32(0)))) +} + +func mustInner(t *testing.T, mapName string, ifindex uint32) *ebpf.Map { + t.Helper() + outer, err := loadPinnedMap(mapName) + if err != nil { + t.Fatalf("load %s: %v", mapName, err) + } + defer outer.Close() + inner, err := lookupInnerMap(outer, ifindex, mapName) + if err != nil { + t.Fatalf("lookup %s inner for ifindex %d: %v", mapName, ifindex, err) + } + return inner +} + +// TestSyncAllowOutInnerRevokesStaticKeepsLearned is the core of the update +// contract on allow_out_v3: static rows the new policy dropped go away, static +// rows it still names stay, and DNS-learned rows are never touched. +func TestSyncAllowOutInnerRevokesStaticKeepsLearned(t *testing.T) { + ifindex := uint32(401) + pinPolicyMaps(t, ifindex) + inner := mustInner(t, MapNameAllowOutV3, ifindex) + + kept := mustParseCIDRForTest(t, "192.0.2.10").IP + revoked := mustParseCIDRForTest(t, "192.0.2.11").IP + learned := mustParseCIDRForTest(t, "192.0.2.12").IP + + seed := []allowOutPolicyEntry{ + {key: lpmKey{Prefixlen: 32, IP: kept}}, + {key: lpmKey{Prefixlen: 32, IP: revoked}}, + } + if err := populateAllowOutInner(inner, seed); err != nil { + t.Fatalf("seed static entries: %v", err) + } + learnedKey := lpmKeyV3{Prefixlen: 32, IP: learned} + if err := inner.Update(&learnedKey, &netPolicyValueV3{ + ExpiresAtNS: 1 << 40, KeyPrefixlen: 32, + }, ebpf.UpdateAny); err != nil { + t.Fatalf("seed learned entry: %v", err) + } + + desired := []allowOutPolicyEntry{{key: lpmKey{Prefixlen: 32, IP: kept}}} + if err := syncAllowOutInner(ifindex, desired); err != nil { + t.Fatalf("syncAllowOutInner: %v", err) + } + + assertV3Present(t, inner, lpmKeyV3{Prefixlen: 32, IP: kept}, true, "still-desired static entry") + assertV3Present(t, inner, lpmKeyV3{Prefixlen: 32, IP: revoked}, false, "revoked static entry") + assertV3Present(t, inner, learnedKey, true, "DNS-learned entry") +} + +// TestSyncAllowOutInnerRevokesExpandedL7Ports covers the expansion path: an L7 +// rule occupies one /48 per port, so narrowing its port set must delete exactly +// the rows for the ports that are gone. +func TestSyncAllowOutInnerRevokesExpandedL7Ports(t *testing.T) { + ifindex := uint32(402) + pinPolicyMaps(t, ifindex) + inner := mustInner(t, MapNameAllowOutV3, ifindex) + ip := mustParseCIDRForTest(t, "192.0.2.20").IP + + l7Entry := func(ports ...uint16) []allowOutPolicyEntry { + set := make([]l7PortEntry, 0, len(ports)) + for _, p := range ports { + set = append(set, l7PortEntry{Port: htonsPort(p), Scheme: L7SchemeHTTPS}) + } + return []allowOutPolicyEntry{{ + key: lpmKey{Prefixlen: 32, IP: ip}, + flags: netPolicyFlagL7Required, + ports: set, + }} + } + + if err := populateAllowOutInner(inner, l7Entry(443, 8443)); err != nil { + t.Fatalf("seed L7 entry: %v", err) + } + if err := syncAllowOutInner(ifindex, l7Entry(443)); err != nil { + t.Fatalf("syncAllowOutInner: %v", err) + } + + assertV3Present(t, inner, lpmKeyV3{Prefixlen: 48, IP: ip, Port: htonsPort(443)}, true, "kept L7 port row") + assertV3Present(t, inner, lpmKeyV3{Prefixlen: 48, IP: ip, Port: htonsPort(8443)}, false, "revoked L7 port row") +} + +// TestSyncDenyOutInnerConvergesOnDesired pins that deny_out is fully managed: +// every row the caller did not ask for is removed. Callers are responsible for +// including the always-denied ranges, which UpdateTAPDevicePolicy does via +// effectiveDenyOutEntriesForReplace. +func TestSyncDenyOutInnerConvergesOnDesired(t *testing.T) { + ifindex := uint32(403) + pinPolicyMaps(t, ifindex) + inner := mustInner(t, MapNameDenyOut, ifindex) + + seed, err := buildDenyOutPolicyEntries([]string{"198.51.100.0/24", "203.0.113.0/24"}) + if err != nil { + t.Fatalf("build seed: %v", err) + } + if err := populateDenyOutInner(inner, seed); err != nil { + t.Fatalf("seed deny_out: %v", err) + } + + desired, err := buildDenyOutPolicyEntries([]string{"203.0.113.0/24", "192.0.2.0/24"}) + if err != nil { + t.Fatalf("build desired: %v", err) + } + if err := syncDenyOutInner(ifindex, desired); err != nil { + t.Fatalf("syncDenyOutInner: %v", err) + } + + var val uint32 + for _, tc := range []struct { + cidr string + want bool + }{ + {"198.51.100.0/24", false}, + {"203.0.113.0/24", true}, + {"192.0.2.0/24", true}, + } { + key, perr := parseCIDR(tc.cidr) + if perr != nil { + t.Fatalf("parse %s: %v", tc.cidr, perr) + } + if got := inner.Lookup(&key, &val) == nil; got != tc.want { + t.Errorf("deny_out %s present=%v, want %v", tc.cidr, got, tc.want) + } + } +} + +// TestSyncDNSAllowInnerReplacesPortSet pins the one place the update path must +// NOT reuse the create path's merge: shrinking a domain's port set has to +// actually shrink it, or the removed ports keep being learned into allow_out. +func TestSyncDNSAllowInnerReplacesPortSet(t *testing.T) { + ifindex := uint32(405) + pinPolicyMaps(t, ifindex) + inner := mustInner(t, MapNameDNSAllowV2, ifindex) + + rule := func(ports ...uint16) dnsAllowRule { + key, value, err := makeDNSAllowRule("api.example.com", uint8(netPolicyFlagL7Required)) + if err != nil { + t.Fatalf("makeDNSAllowRule: %v", err) + } + set := make([]l7PortEntry, 0, len(ports)) + for _, p := range ports { + set = append(set, l7PortEntry{Port: htonsPort(p), Scheme: L7SchemeHTTPS}) + } + applyPortsToDNSAllowValue(&value, set) + return dnsAllowRule{domain: "api.example.com", key: key, value: value} + } + + wide := rule(443, 8443) + if err := populateDNSAllowInnerMap(inner, []dnsAllowRule{wide}); err != nil { + t.Fatalf("seed dns allow: %v", err) + } + if err := syncDNSAllowInner(ifindex, []dnsAllowRule{rule(443)}); err != nil { + t.Fatalf("syncDNSAllowInner: %v", err) + } + + var got dnsAllowValue + if err := inner.Lookup(&wide.key, &got); err != nil { + t.Fatalf("lookup dns allow: %v", err) + } + if got.PortCount != 1 || got.Ports[0].Port != htonsPort(443) { + t.Fatalf("port set not replaced: count=%d ports=%+v", got.PortCount, got.Ports[:got.PortCount]) + } +} + +// TestSyncDNSAllowInnerRevokesDomain covers removing a domain rule outright. +func TestSyncDNSAllowInnerRevokesDomain(t *testing.T) { + ifindex := uint32(406) + pinPolicyMaps(t, ifindex) + inner := mustInner(t, MapNameDNSAllowV2, ifindex) + + keptKey, keptVal, err := makeDNSAllowRule("kept.example.com", 0) + if err != nil { + t.Fatalf("makeDNSAllowRule: %v", err) + } + revokedKey, revokedVal, err := makeDNSAllowRule("gone.example.com", 0) + if err != nil { + t.Fatalf("makeDNSAllowRule: %v", err) + } + seed := []dnsAllowRule{ + {domain: "kept.example.com", key: keptKey, value: keptVal}, + {domain: "gone.example.com", key: revokedKey, value: revokedVal}, + } + if err := populateDNSAllowInnerMap(inner, seed); err != nil { + t.Fatalf("seed dns allow: %v", err) + } + if err := syncDNSAllowInner(ifindex, seed[:1]); err != nil { + t.Fatalf("syncDNSAllowInner: %v", err) + } + + var val dnsAllowValue + if err := inner.Lookup(&keptKey, &val); err != nil { + t.Errorf("still-desired domain was deleted: %v", err) + } + if err := inner.Lookup(&revokedKey, &val); err == nil { + t.Error("revoked domain survived the update") + } +} + +// TestBumpPolicyVersionAdvancesGeneration checks the one signal the datapath +// uses to notice an update at all, plus the invariant that a metadata rewrite +// must not reset it. +func TestBumpPolicyVersionAdvancesGeneration(t *testing.T) { + mountBpffs(t) + pinTAPMetadataMaps(t) + ifindex := uint32(404) + ip := net.ParseIP("10.0.0.5") + if err := UpsertTAPDeviceMetadata(ifindex, ip, "sandbox-404", 7); err != nil { + t.Fatalf("UpsertTAPDeviceMetadata: %v", err) + } + + if got := mustReadPolicyVersion(t, ifindex); got != 1 { + t.Fatalf("fresh TAP policy_version=%d, want 1 (0 is reserved for unset)", got) + } + if err := bumpPolicyVersion(ifindex); err != nil { + t.Fatalf("bumpPolicyVersion: %v", err) + } + if got := mustReadPolicyVersion(t, ifindex); got != 2 { + t.Fatalf("policy_version=%d, want 2", got) + } + + // Recovery bumps Version on every restart; that must not reset the policy + // generation, or every live session would look stale at once. + if err := UpsertTAPDeviceMetadata(ifindex, ip, "sandbox-404", 8); err != nil { + t.Fatalf("re-upsert metadata: %v", err) + } + if got := mustReadPolicyVersion(t, ifindex); got != 2 { + t.Fatalf("metadata rewrite reset policy_version to %d, want 2", got) + } +} + +func assertV3Present(t *testing.T, inner *ebpf.Map, key lpmKeyV3, want bool, what string) { + t.Helper() + var val netPolicyValueV3 + if got := inner.Lookup(&key, &val) == nil; got != want { + t.Errorf("%s present=%v, want %v", what, got, want) + } +} + +func mustReadPolicyVersion(t *testing.T, ifindex uint32) uint32 { + t.Helper() + m, err := loadPinnedMap(MapNameIfindexToMVMMetadata) + if err != nil { + t.Fatalf("load %s: %v", MapNameIfindexToMVMMetadata, err) + } + defer m.Close() + var meta mvmMetadata + if err := m.Lookup(&ifindex, &meta); err != nil { + t.Fatalf("lookup metadata for ifindex %d: %v", ifindex, err) + } + return meta.PolicyVersion +} diff --git a/CubeNet/cubevs/reaper.go b/CubeNet/cubevs/reaper.go index ee91c3aa3..c7d2e18c9 100644 --- a/CubeNet/cubevs/reaper.go +++ b/CubeNet/cubevs/reaper.go @@ -156,18 +156,19 @@ type sessionKey struct { } type natSession struct { - AccessTime uint64 - NodeIfindex uint32 - NodeIP uint32 - VMIfindex uint32 - VMIP uint32 - NodePort uint16 - VMPort uint16 - State uint8 - ActiveClose uint8 - PacketClass uint8 - L7Scheme uint8 - Reserved [32]uint8 + AccessTime uint64 + NodeIfindex uint32 + NodeIP uint32 + VMIfindex uint32 + VMIP uint32 + NodePort uint16 + VMPort uint16 + State uint8 + ActiveClose uint8 + PacketClass uint8 + L7Scheme uint8 + PolicyVersion uint32 + Reserved [28]uint8 } // timeout returns the timeout for the session in nanoseconds. diff --git a/CubeNet/cubevs/tap.go b/CubeNet/cubevs/tap.go index afca5d27b..2464a1c01 100644 --- a/CubeNet/cubevs/tap.go +++ b/CubeNet/cubevs/tap.go @@ -83,6 +83,9 @@ func UpsertTAPDeviceMetadata(ifindex uint32, ip net.IP, id string, version uint3 IP: mvmIP, UUID: stringToByteArray(id), Version: version, + // First generation for a brand-new TAP. 0 is reserved for "unset", so + // pre-upgrade sessions read as stale and get re-checked exactly once. + PolicyVersion: 1, } // ifindex <-> MVM metadata (IP, ID and tunnels) @@ -98,6 +101,12 @@ func UpsertTAPDeviceMetadata(ifindex uint32, ip net.IP, id string, version uint3 oldMVMIP = oldMVMID.IP mvmID.DNSPolicyFlags = oldMVMID.DNSPolicyFlags mvmID.Reserved = oldMVMID.Reserved + // Carry the policy generation across metadata rewrites (recovery bumps + // Version on every restart). Resetting it would make every live session + // look stale and force a re-check storm on a dense node. + if oldMVMID.PolicyVersion != 0 { + mvmID.PolicyVersion = oldMVMID.PolicyVersion + } } else if !errors.Is(err, ebpf.ErrKeyNotExist) { return fmt.Errorf("map.Lookup failed: %w, name: %s", err, MapNameIfindexToMVMMetadata) } diff --git a/CubeNet/src/cubevs.h b/CubeNet/src/cubevs.h index 60bbf7284..1b1799902 100644 --- a/CubeNet/src/cubevs.h +++ b/CubeNet/src/cubevs.h @@ -145,12 +145,24 @@ const volatile __u16 nodenic_macaddr_p2 = 0x16dd; const volatile __u32 nodegw_macaddr_p1 = 0x4732eefe; /* fe:ee:32:47:6b:93 */ const volatile __u16 nodegw_macaddr_p2 = 0x936b; +/* policy_version is the per-sandbox network-policy generation. It is bumped by + * userspace after a policy update has been fully applied, and every packet on + * an established flow compares it against the generation cached in nat_session + * to decide whether the flow must be re-evaluated. Starts at 1; 0 means "never + * set" (sessions and metadata written before this field existed), which reads + * as stale and therefore triggers exactly one re-evaluation after an upgrade. + * + * Do NOT reuse the `version` field above for this: it is part of session_key, + * so bumping it would orphan every live session for this TAP. + */ struct mvm_meta { __u32 version; __u32 ip; __u8 uuid[64]; __u8 dns_policy_flags; - __u8 reserved[55]; + __u8 reserved0[3]; /* aligns policy_version; reserved starts at an odd offset */ + __u32 policy_version; + __u8 reserved[48]; }; /* https://elixir.bootlin.com/linux/v5.4.217/source/include/uapi/linux/if_arp.h#L144 */ @@ -318,7 +330,8 @@ struct nat_session { __u8 active_close; __u8 packet_class; /* SNAT_PACKET or L7PROXY_PACKET */ __u8 l7_scheme; /* L7_SCHEME_*; NONE for non-L7 sessions */ - __u8 reserved[32]; + __u32 policy_version; /* mvm_meta.policy_version at create / last re-check */ + __u8 reserved[28]; }; struct ingress_session { diff --git a/CubeNet/src/egress_policy_test.bpf.c b/CubeNet/src/egress_policy_test.bpf.c index ca3f3d8b5..f97be6dae 100644 --- a/CubeNet/src/egress_policy_test.bpf.c +++ b/CubeNet/src/egress_policy_test.bpf.c @@ -30,4 +30,42 @@ int test_classify_egress_flow(struct __sk_buff *skb) return TC_ACT_OK; } +/* Mirrors sessionRecheckCase in egress_policy_test.go. */ +struct session_recheck_case { + __u32 ifindex; + __u32 daddr; + __u32 sess_policy_version; + __u32 meta_policy_version; + __u32 policy_version_out; + __u16 dport; + __u8 packet_class; + __u8 l7_scheme; + __u8 revoked; + __u8 reserved[3]; +}; + +SEC("tc") +int test_session_policy_revoked(struct __sk_buff *skb) +{ + struct session_recheck_case tc = {}; + struct nat_session sess = {}; + struct mvm_meta meta = {}; + + if (bpf_skb_load_bytes(skb, 0, &tc, sizeof(tc))) + return TC_ACT_SHOT; + + sess.packet_class = tc.packet_class; + sess.l7_scheme = tc.l7_scheme; + sess.policy_version = tc.sess_policy_version; + meta.policy_version = tc.meta_policy_version; + + tc.revoked = session_policy_revoked(&sess, &meta, tc.ifindex, tc.daddr, tc.dport); + tc.policy_version_out = sess.policy_version; + + if (bpf_skb_store_bytes(skb, 0, &tc, sizeof(tc), 0)) + return TC_ACT_SHOT; + + return TC_ACT_OK; +} + char __license[] SEC("license") = "Dual BSD/GPL"; diff --git a/CubeNet/src/mvmtap.bpf.c b/CubeNet/src/mvmtap.bpf.c index c265d6bc6..c444e9b02 100644 --- a/CubeNet/src/mvmtap.bpf.c +++ b/CubeNet/src/mvmtap.bpf.c @@ -465,6 +465,14 @@ static __always_inline __u32 do_icmp_nat(struct __sk_buff *skb, struct mvm_meta sess = bpf_map_lookup_elem(&egress_sessions, &key); if (sess) { + /* revoked by a policy update: retire the pair and drop, since + * there is nothing to reset on ICMP + */ + if (session_policy_revoked(sess, mvm_meta, skb->ingress_ifindex, + key.dst_ip, key.dst_port)) { + del_session(&key, sess); + return 0; + } update_icmp_session(IP_CT_DIR_ORIGINAL, sess, now); goto do_nat; } @@ -566,6 +574,14 @@ static __always_inline __u32 do_udp_nat_inline(struct __sk_buff *skb, sess = bpf_map_lookup_elem(&egress_sessions, &key); if (sess) { + /* revoked by a policy update: retire the pair and drop, since + * there is nothing to reset on UDP + */ + if (session_policy_revoked(sess, mvm_meta, skb->ingress_ifindex, + key.dst_ip, key.dst_port)) { + del_session(&key, sess); + return 0; + } update_udp_session(IP_CT_DIR_ORIGINAL, sess, now); goto do_nat; } @@ -737,7 +753,7 @@ static __always_inline __u64 do_tcp_nat(struct __sk_buff *skb, struct mvm_meta * goto do_create; } - goto do_update; + goto do_recheck; } do_create: /* Classify the flow with the unified egress policy. The verdict is @@ -783,47 +799,29 @@ static __always_inline __u64 do_tcp_nat(struct __sk_buff *skb, struct mvm_meta * /* lookup existing session */ sess = bpf_map_lookup_elem(&egress_sessions, &key); if (!sess) { - /* Legacy default-port (80/443) connection drain: the eBPF session - * entry was lost (agent restart, map eviction, or expiry) AND the - * allow_out_v3 /48 entry for this (ip, port) has aged out, so - * classify_egress_flow would return FLOW_REJECT. But the proxy - * still holds an established TPROXY socket for this 4-tuple — the - * connection was legitimately opened when the policy allowed it. - * Re-stamp the mark so iptables TPROXY steers the packet to the - * proxy, keeping the connection alive instead of resetting it. - * Custom-port connections are intentionally excluded: they should - * respect the current policy when their allow_out_v3 entry expires. - * No session is re-created, so each packet on the drained flow - * re-enters this path (per-packet socket lookup — acceptable for - * draining connections that will eventually close). + /* No session: the flow was never authorized, or it was retired + * (reaped, or revoked by a policy update). Either way there is no + * record that this connection is allowed, so answer like every + * other unreachable TCP packet here instead of trusting that a + * live proxy socket implies a past authorization. */ - if (l4->dest == bpf_htons(80) || l4->dest == bpf_htons(443)) { - struct bpf_sock *sk; - struct bpf_sock_tuple tuple = {}; - tuple.ipv4.saddr = key.src_ip; - tuple.ipv4.daddr = key.dst_ip; - tuple.ipv4.sport = l4->source; - tuple.ipv4.dport = l4->dest; - sk = bpf_skc_lookup_tcp(skb, &tuple, sizeof(tuple.ipv4), BPF_F_CURRENT_NETNS, 0); - if (sk) { - __u32 state = sk->state; - - bpf_sk_release(sk); - if (state == BPF_TCP_ESTABLISHED) { - if (l4->dest == bpf_htons(80)) { - skb->mark = (skb->mark & ~cube_l7_mark_mask) | cube_l7_mark_http; - } else { - skb->mark = (skb->mark & ~cube_l7_mark_mask) | cube_l7_mark_https; - } - return TCP_NAT_PACK(cubegw0_ifindex, TCP_L7PROXY_OK); - } - } - } return rst ? TCP_NAT_DROP : TCP_NAT_RESET; } } -do_update: +do_recheck: + /* A policy update revoked this flow: retire the session pair so the tuple is + * free for an immediate reconnect, and answer with an RST like every other + * unreachable TCP packet here, so the guest learns now instead of stalling + * until its retransmit timer gives up. Never RST an RST, or two peers that + * both consider the flow dead would trade resets forever. + */ + if (session_policy_revoked(sess, mvm_meta, skb->ingress_ifindex, + key.dst_ip, key.dst_port)) { + del_session(&key, sess); + return rst ? TCP_NAT_DROP : TCP_NAT_RESET; + } + /* update session */ update_session(IP_CT_DIR_ORIGINAL, sess, now, syn, ack, fin, rst); diff --git a/CubeNet/src/session.h b/CubeNet/src/session.h index 04c59f9e5..081a6e756 100644 --- a/CubeNet/src/session.h +++ b/CubeNet/src/session.h @@ -170,6 +170,69 @@ static __always_inline __u8 classify_egress_flow(__u32 ifindex, __u32 daddr, return FLOW_SNAT; } +/** + * session_verdict - the flow verdict this session was created under + * @sess: pointer to the NAT session + * + * Reconstructed from packet_class + l7_scheme so a re-check can compare the + * fresh verdict against the cached one. An L7 session with an unknown scheme + * is a corrupt value; report REJECT so it fails closed, matching + * classify_egress_flow(). + */ +static __always_inline __u8 session_verdict(const struct nat_session *sess) +{ + if (sess->packet_class != L7PROXY_PACKET) + return FLOW_SNAT; + if (sess->l7_scheme == L7_SCHEME_HTTP) + return FLOW_HTTP; + if (sess->l7_scheme == L7_SCHEME_HTTPS) + return FLOW_HTTPS; + return FLOW_REJECT; +} + +/** + * session_policy_revoked - re-evaluate an established flow after a policy update + * @sess: pointer to the NAT session + * @mvm_meta: sandbox metadata carrying the current policy generation + * @ifindex: TAP ifindex of the originating MVM + * @daddr: destination IP in network byte order + * @dport: destination port in network byte order (0 for ICMP) + * + * Returns true when this flow may no longer carry traffic. Callers retire the + * session pair with del_session() and reject the packet: TCP answers with an RST + * like every other unreachable packet here, so the guest fails fast instead of + * stalling on retransmits; UDP and ICMP have nothing to reset and simply drop. + * + * Deleting rather than flagging keeps the retirement self-enforcing. A later + * non-SYN packet on the same tuple finds no session and is reset, so a revoked + * flow cannot resume even if a subsequent update re-allows the destination, + * while a SYN legitimately opens a fresh connection under the current policy. + * + * A verdict *change* counts as revocation, not just FLOW_REJECT. Once a flow + * must switch between plain SNAT and L7 interception there is no way to migrate + * it -- the two paths disagree on both the reply tuple and who terminates the + * TCP connection -- so the flow is retired and the client reconnects. + * + * Called before update_session(): there is no point advancing the conntrack + * state of a flow that is about to be deleted. + */ +static __always_inline bool session_policy_revoked(struct nat_session *sess, + const struct mvm_meta *mvm_meta, + __u32 ifindex, __u32 daddr, + __u16 dport) +{ + __u8 verdict; + + if (sess->policy_version == mvm_meta->policy_version) + return false; + + verdict = classify_egress_flow(ifindex, daddr, dport); + if (verdict == FLOW_REJECT || verdict != session_verdict(sess)) + return true; + sess->policy_version = mvm_meta->policy_version; + return false; +} + /** * create_nat_session - create egress session with rollback on failure * @skb: packet skb, used to signal deny reason via skb->cb[] @@ -202,6 +265,7 @@ static __always_inline bool create_nat_session(struct __sk_buff *skb, { struct nat_session sess = {}; struct session_key ikey = {}; + struct mvm_meta *meta; long err; ikey.src_ip = ekey->dst_ip; @@ -227,6 +291,13 @@ static __always_inline bool create_nat_session(struct __sk_buff *skb, sess.state = initial_state; sess.packet_class = packet_class; sess.l7_scheme = l7_scheme; + /* Stamp the generation this verdict was taken under. A missing meta + * leaves 0, which reads as stale and costs one re-check on the next + * packet -- the safe direction. + */ + meta = bpf_map_lookup_elem(&ifindex_to_mvmmeta, &vm_ifindex); + if (meta) + sess.policy_version = meta->policy_version; err = bpf_map_update_elem(&egress_sessions, ekey, &sess, BPF_NOEXIST); if (err) { /* on failure, clean up the ingress slot we reserved earlier */ diff --git a/Cubelet/api/services/cubebox/v1/cubebox.pb.go b/Cubelet/api/services/cubebox/v1/cubebox.pb.go index 9c88b780b..353562f0f 100644 --- a/Cubelet/api/services/cubebox/v1/cubebox.pb.go +++ b/Cubelet/api/services/cubebox/v1/cubebox.pb.go @@ -3921,9 +3921,13 @@ type UpdateCubeSandboxRequest struct { // new features that are opaque to the Kubernetes APIs (both user-facing // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. - Annotations map[string]string `protobuf:"bytes,3,rep,name=annotations,proto3" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + Annotations map[string]string `protobuf:"bytes,3,rep,name=annotations,proto3" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` + // Replacement egress policy for a running sandbox. Absent means "leave the + // network alone"; present means the complete desired state, so an omitted or + // empty field inside it clears whatever is currently installed. + CubeNetworkConfig *CubeNetworkConfig `protobuf:"bytes,4,opt,name=cube_network_config,json=cubeNetworkConfig,proto3,oneof" json:"cube_network_config,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *UpdateCubeSandboxRequest) Reset() { @@ -3977,6 +3981,13 @@ func (x *UpdateCubeSandboxRequest) GetAnnotations() map[string]string { return nil } +func (x *UpdateCubeSandboxRequest) GetCubeNetworkConfig() *CubeNetworkConfig { + if x != nil { + return x.CubeNetworkConfig + } + return nil +} + type UpdateCubeSandboxResponse struct { state protoimpl.MessageState `protogen:"open.v1"` // requestID reqID @@ -6859,14 +6870,16 @@ const file_api_services_cubebox_v1_cubebox_proto_rawDesc = "" + "\a_filterB\t\n" + "\a_option\"Y\n" + "\x17ListCubeSandboxResponse\x12>\n" + - "\x05items\x18\x01 \x03(\v2(.cubelet.services.cubebox.v1.CubeSandboxR\x05items\"\x80\x02\n" + + "\x05items\x18\x01 \x03(\v2(.cubelet.services.cubebox.v1.CubeSandboxR\x05items\"\xfd\x02\n" + "\x18UpdateCubeSandboxRequest\x12\x1c\n" + "\trequestID\x18\x01 \x01(\tR\trequestID\x12\x1c\n" + "\tsandboxID\x18\x02 \x01(\tR\tsandboxID\x12h\n" + - "\vannotations\x18\x03 \x03(\v2F.cubelet.services.cubebox.v1.UpdateCubeSandboxRequest.AnnotationsEntryR\vannotations\x1a>\n" + + "\vannotations\x18\x03 \x03(\v2F.cubelet.services.cubebox.v1.UpdateCubeSandboxRequest.AnnotationsEntryR\vannotations\x12c\n" + + "\x13cube_network_config\x18\x04 \x01(\v2..cubelet.services.cubebox.v1.CubeNetworkConfigH\x00R\x11cubeNetworkConfig\x88\x01\x01\x1a>\n" + "\x10AnnotationsEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + - "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x8b\x02\n" + + "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01B\x16\n" + + "\x14_cube_network_config\"\x8b\x02\n" + "\x19UpdateCubeSandboxResponse\x12\x1c\n" + "\trequestID\x18\x01 \x01(\tR\trequestID\x124\n" + "\x03ret\x18\x02 \x01(\v2\".cubelet.services.errorcode.v1.RetR\x03ret\x12^\n" + @@ -7345,64 +7358,65 @@ var file_api_services_cubebox_v1_cubebox_proto_depIdxs = []int32{ 58, // 69: cubelet.services.cubebox.v1.ListCubeSandboxRequest.option:type_name -> cubelet.services.cubebox.v1.ListCubeSandboxOption 49, // 70: cubelet.services.cubebox.v1.ListCubeSandboxResponse.items:type_name -> cubelet.services.cubebox.v1.CubeSandbox 97, // 71: cubelet.services.cubebox.v1.UpdateCubeSandboxRequest.annotations:type_name -> cubelet.services.cubebox.v1.UpdateCubeSandboxRequest.AnnotationsEntry - 103, // 72: cubelet.services.cubebox.v1.UpdateCubeSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 98, // 73: cubelet.services.cubebox.v1.UpdateCubeSandboxResponse.ext_info:type_name -> cubelet.services.cubebox.v1.UpdateCubeSandboxResponse.ExtInfoEntry - 103, // 74: cubelet.services.cubebox.v1.ExecCubeSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 39, // 75: cubelet.services.cubebox.v1.AppSnapshotRequest.create_request:type_name -> cubelet.services.cubebox.v1.RunCubeSandboxRequest - 103, // 76: cubelet.services.cubebox.v1.AppSnapshotResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 103, // 77: cubelet.services.cubebox.v1.CommitSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 103, // 78: cubelet.services.cubebox.v1.RollbackSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 67, // 79: cubelet.services.cubebox.v1.CleanupTemplateRequest.objects:type_name -> cubelet.services.cubebox.v1.CowObjectRef - 103, // 80: cubelet.services.cubebox.v1.CleanupTemplateResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 67, // 81: cubelet.services.cubebox.v1.ListSandboxSnapshotsRequest.objects:type_name -> cubelet.services.cubebox.v1.CowObjectRef - 103, // 82: cubelet.services.cubebox.v1.ListSandboxSnapshotsResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 71, // 83: cubelet.services.cubebox.v1.ListSandboxSnapshotsResponse.objects:type_name -> cubelet.services.cubebox.v1.CowObjectStatus - 103, // 84: cubelet.services.cubebox.v1.ListLocalSnapshotsResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 74, // 85: cubelet.services.cubebox.v1.ListLocalSnapshotsResponse.snapshots:type_name -> cubelet.services.cubebox.v1.LocalSnapshotInfo - 103, // 86: cubelet.services.cubebox.v1.GetLocalSnapshotResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 74, // 87: cubelet.services.cubebox.v1.GetLocalSnapshotResponse.snapshot:type_name -> cubelet.services.cubebox.v1.LocalSnapshotInfo - 103, // 88: cubelet.services.cubebox.v1.GetStorageMetricsResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 99, // 89: cubelet.services.cubebox.v1.GetStorageMetricsResponse.metrics:type_name -> cubelet.services.cubebox.v1.GetStorageMetricsResponse.MetricsEntry - 80, // 90: cubelet.services.cubebox.v1.SandboxStorageInfo.volumes:type_name -> cubelet.services.cubebox.v1.StorageVolumeInfo - 103, // 91: cubelet.services.cubebox.v1.InspectStorageVolumesResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 81, // 92: cubelet.services.cubebox.v1.InspectStorageVolumesResponse.sandboxes:type_name -> cubelet.services.cubebox.v1.SandboxStorageInfo - 103, // 93: cubelet.services.cubebox.v1.CleanupOrphanStorageFilesResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret - 85, // 94: cubelet.services.cubebox.v1.CleanupOrphanStorageFilesResponse.orphans:type_name -> cubelet.services.cubebox.v1.StorageOrphanEntry - 39, // 95: cubelet.services.cubebox.v1.CubeboxMgr.Create:input_type -> cubelet.services.cubebox.v1.RunCubeSandboxRequest - 47, // 96: cubelet.services.cubebox.v1.CubeboxMgr.Destroy:input_type -> cubelet.services.cubebox.v1.DestroyCubeSandboxRequest - 53, // 97: cubelet.services.cubebox.v1.CubeboxMgr.List:input_type -> cubelet.services.cubebox.v1.ListCubeSandboxRequest - 55, // 98: cubelet.services.cubebox.v1.CubeboxMgr.Update:input_type -> cubelet.services.cubebox.v1.UpdateCubeSandboxRequest - 59, // 99: cubelet.services.cubebox.v1.CubeboxMgr.Exec:input_type -> cubelet.services.cubebox.v1.ExecCubeSandboxRequest - 61, // 100: cubelet.services.cubebox.v1.CubeboxMgr.AppSnapshot:input_type -> cubelet.services.cubebox.v1.AppSnapshotRequest - 63, // 101: cubelet.services.cubebox.v1.CubeboxMgr.CommitSandbox:input_type -> cubelet.services.cubebox.v1.CommitSandboxRequest - 65, // 102: cubelet.services.cubebox.v1.CubeboxMgr.RollbackSandbox:input_type -> cubelet.services.cubebox.v1.RollbackSandboxRequest - 68, // 103: cubelet.services.cubebox.v1.CubeboxMgr.CleanupTemplate:input_type -> cubelet.services.cubebox.v1.CleanupTemplateRequest - 70, // 104: cubelet.services.cubebox.v1.CubeboxMgr.ListSandboxSnapshots:input_type -> cubelet.services.cubebox.v1.ListSandboxSnapshotsRequest - 73, // 105: cubelet.services.cubebox.v1.CubeboxMgr.ListLocalSnapshots:input_type -> cubelet.services.cubebox.v1.ListLocalSnapshotsRequest - 76, // 106: cubelet.services.cubebox.v1.CubeboxMgr.GetLocalSnapshot:input_type -> cubelet.services.cubebox.v1.GetLocalSnapshotRequest - 78, // 107: cubelet.services.cubebox.v1.CubeboxMgr.GetStorageMetrics:input_type -> cubelet.services.cubebox.v1.GetStorageMetricsRequest - 82, // 108: cubelet.services.cubebox.v1.CubeboxMgr.InspectStorageVolumes:input_type -> cubelet.services.cubebox.v1.InspectStorageVolumesRequest - 84, // 109: cubelet.services.cubebox.v1.CubeboxMgr.CleanupOrphanStorageFiles:input_type -> cubelet.services.cubebox.v1.CleanupOrphanStorageFilesRequest - 40, // 110: cubelet.services.cubebox.v1.CubeboxMgr.Create:output_type -> cubelet.services.cubebox.v1.RunCubeSandboxResponse - 48, // 111: cubelet.services.cubebox.v1.CubeboxMgr.Destroy:output_type -> cubelet.services.cubebox.v1.DestroyCubeSandboxResponse - 54, // 112: cubelet.services.cubebox.v1.CubeboxMgr.List:output_type -> cubelet.services.cubebox.v1.ListCubeSandboxResponse - 56, // 113: cubelet.services.cubebox.v1.CubeboxMgr.Update:output_type -> cubelet.services.cubebox.v1.UpdateCubeSandboxResponse - 60, // 114: cubelet.services.cubebox.v1.CubeboxMgr.Exec:output_type -> cubelet.services.cubebox.v1.ExecCubeSandboxResponse - 62, // 115: cubelet.services.cubebox.v1.CubeboxMgr.AppSnapshot:output_type -> cubelet.services.cubebox.v1.AppSnapshotResponse - 64, // 116: cubelet.services.cubebox.v1.CubeboxMgr.CommitSandbox:output_type -> cubelet.services.cubebox.v1.CommitSandboxResponse - 66, // 117: cubelet.services.cubebox.v1.CubeboxMgr.RollbackSandbox:output_type -> cubelet.services.cubebox.v1.RollbackSandboxResponse - 69, // 118: cubelet.services.cubebox.v1.CubeboxMgr.CleanupTemplate:output_type -> cubelet.services.cubebox.v1.CleanupTemplateResponse - 72, // 119: cubelet.services.cubebox.v1.CubeboxMgr.ListSandboxSnapshots:output_type -> cubelet.services.cubebox.v1.ListSandboxSnapshotsResponse - 75, // 120: cubelet.services.cubebox.v1.CubeboxMgr.ListLocalSnapshots:output_type -> cubelet.services.cubebox.v1.ListLocalSnapshotsResponse - 77, // 121: cubelet.services.cubebox.v1.CubeboxMgr.GetLocalSnapshot:output_type -> cubelet.services.cubebox.v1.GetLocalSnapshotResponse - 79, // 122: cubelet.services.cubebox.v1.CubeboxMgr.GetStorageMetrics:output_type -> cubelet.services.cubebox.v1.GetStorageMetricsResponse - 83, // 123: cubelet.services.cubebox.v1.CubeboxMgr.InspectStorageVolumes:output_type -> cubelet.services.cubebox.v1.InspectStorageVolumesResponse - 86, // 124: cubelet.services.cubebox.v1.CubeboxMgr.CleanupOrphanStorageFiles:output_type -> cubelet.services.cubebox.v1.CleanupOrphanStorageFilesResponse - 110, // [110:125] is the sub-list for method output_type - 95, // [95:110] is the sub-list for method input_type - 95, // [95:95] is the sub-list for extension type_name - 95, // [95:95] is the sub-list for extension extendee - 0, // [0:95] is the sub-list for field type_name + 42, // 72: cubelet.services.cubebox.v1.UpdateCubeSandboxRequest.cube_network_config:type_name -> cubelet.services.cubebox.v1.CubeNetworkConfig + 103, // 73: cubelet.services.cubebox.v1.UpdateCubeSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 98, // 74: cubelet.services.cubebox.v1.UpdateCubeSandboxResponse.ext_info:type_name -> cubelet.services.cubebox.v1.UpdateCubeSandboxResponse.ExtInfoEntry + 103, // 75: cubelet.services.cubebox.v1.ExecCubeSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 39, // 76: cubelet.services.cubebox.v1.AppSnapshotRequest.create_request:type_name -> cubelet.services.cubebox.v1.RunCubeSandboxRequest + 103, // 77: cubelet.services.cubebox.v1.AppSnapshotResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 103, // 78: cubelet.services.cubebox.v1.CommitSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 103, // 79: cubelet.services.cubebox.v1.RollbackSandboxResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 67, // 80: cubelet.services.cubebox.v1.CleanupTemplateRequest.objects:type_name -> cubelet.services.cubebox.v1.CowObjectRef + 103, // 81: cubelet.services.cubebox.v1.CleanupTemplateResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 67, // 82: cubelet.services.cubebox.v1.ListSandboxSnapshotsRequest.objects:type_name -> cubelet.services.cubebox.v1.CowObjectRef + 103, // 83: cubelet.services.cubebox.v1.ListSandboxSnapshotsResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 71, // 84: cubelet.services.cubebox.v1.ListSandboxSnapshotsResponse.objects:type_name -> cubelet.services.cubebox.v1.CowObjectStatus + 103, // 85: cubelet.services.cubebox.v1.ListLocalSnapshotsResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 74, // 86: cubelet.services.cubebox.v1.ListLocalSnapshotsResponse.snapshots:type_name -> cubelet.services.cubebox.v1.LocalSnapshotInfo + 103, // 87: cubelet.services.cubebox.v1.GetLocalSnapshotResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 74, // 88: cubelet.services.cubebox.v1.GetLocalSnapshotResponse.snapshot:type_name -> cubelet.services.cubebox.v1.LocalSnapshotInfo + 103, // 89: cubelet.services.cubebox.v1.GetStorageMetricsResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 99, // 90: cubelet.services.cubebox.v1.GetStorageMetricsResponse.metrics:type_name -> cubelet.services.cubebox.v1.GetStorageMetricsResponse.MetricsEntry + 80, // 91: cubelet.services.cubebox.v1.SandboxStorageInfo.volumes:type_name -> cubelet.services.cubebox.v1.StorageVolumeInfo + 103, // 92: cubelet.services.cubebox.v1.InspectStorageVolumesResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 81, // 93: cubelet.services.cubebox.v1.InspectStorageVolumesResponse.sandboxes:type_name -> cubelet.services.cubebox.v1.SandboxStorageInfo + 103, // 94: cubelet.services.cubebox.v1.CleanupOrphanStorageFilesResponse.ret:type_name -> cubelet.services.errorcode.v1.Ret + 85, // 95: cubelet.services.cubebox.v1.CleanupOrphanStorageFilesResponse.orphans:type_name -> cubelet.services.cubebox.v1.StorageOrphanEntry + 39, // 96: cubelet.services.cubebox.v1.CubeboxMgr.Create:input_type -> cubelet.services.cubebox.v1.RunCubeSandboxRequest + 47, // 97: cubelet.services.cubebox.v1.CubeboxMgr.Destroy:input_type -> cubelet.services.cubebox.v1.DestroyCubeSandboxRequest + 53, // 98: cubelet.services.cubebox.v1.CubeboxMgr.List:input_type -> cubelet.services.cubebox.v1.ListCubeSandboxRequest + 55, // 99: cubelet.services.cubebox.v1.CubeboxMgr.Update:input_type -> cubelet.services.cubebox.v1.UpdateCubeSandboxRequest + 59, // 100: cubelet.services.cubebox.v1.CubeboxMgr.Exec:input_type -> cubelet.services.cubebox.v1.ExecCubeSandboxRequest + 61, // 101: cubelet.services.cubebox.v1.CubeboxMgr.AppSnapshot:input_type -> cubelet.services.cubebox.v1.AppSnapshotRequest + 63, // 102: cubelet.services.cubebox.v1.CubeboxMgr.CommitSandbox:input_type -> cubelet.services.cubebox.v1.CommitSandboxRequest + 65, // 103: cubelet.services.cubebox.v1.CubeboxMgr.RollbackSandbox:input_type -> cubelet.services.cubebox.v1.RollbackSandboxRequest + 68, // 104: cubelet.services.cubebox.v1.CubeboxMgr.CleanupTemplate:input_type -> cubelet.services.cubebox.v1.CleanupTemplateRequest + 70, // 105: cubelet.services.cubebox.v1.CubeboxMgr.ListSandboxSnapshots:input_type -> cubelet.services.cubebox.v1.ListSandboxSnapshotsRequest + 73, // 106: cubelet.services.cubebox.v1.CubeboxMgr.ListLocalSnapshots:input_type -> cubelet.services.cubebox.v1.ListLocalSnapshotsRequest + 76, // 107: cubelet.services.cubebox.v1.CubeboxMgr.GetLocalSnapshot:input_type -> cubelet.services.cubebox.v1.GetLocalSnapshotRequest + 78, // 108: cubelet.services.cubebox.v1.CubeboxMgr.GetStorageMetrics:input_type -> cubelet.services.cubebox.v1.GetStorageMetricsRequest + 82, // 109: cubelet.services.cubebox.v1.CubeboxMgr.InspectStorageVolumes:input_type -> cubelet.services.cubebox.v1.InspectStorageVolumesRequest + 84, // 110: cubelet.services.cubebox.v1.CubeboxMgr.CleanupOrphanStorageFiles:input_type -> cubelet.services.cubebox.v1.CleanupOrphanStorageFilesRequest + 40, // 111: cubelet.services.cubebox.v1.CubeboxMgr.Create:output_type -> cubelet.services.cubebox.v1.RunCubeSandboxResponse + 48, // 112: cubelet.services.cubebox.v1.CubeboxMgr.Destroy:output_type -> cubelet.services.cubebox.v1.DestroyCubeSandboxResponse + 54, // 113: cubelet.services.cubebox.v1.CubeboxMgr.List:output_type -> cubelet.services.cubebox.v1.ListCubeSandboxResponse + 56, // 114: cubelet.services.cubebox.v1.CubeboxMgr.Update:output_type -> cubelet.services.cubebox.v1.UpdateCubeSandboxResponse + 60, // 115: cubelet.services.cubebox.v1.CubeboxMgr.Exec:output_type -> cubelet.services.cubebox.v1.ExecCubeSandboxResponse + 62, // 116: cubelet.services.cubebox.v1.CubeboxMgr.AppSnapshot:output_type -> cubelet.services.cubebox.v1.AppSnapshotResponse + 64, // 117: cubelet.services.cubebox.v1.CubeboxMgr.CommitSandbox:output_type -> cubelet.services.cubebox.v1.CommitSandboxResponse + 66, // 118: cubelet.services.cubebox.v1.CubeboxMgr.RollbackSandbox:output_type -> cubelet.services.cubebox.v1.RollbackSandboxResponse + 69, // 119: cubelet.services.cubebox.v1.CubeboxMgr.CleanupTemplate:output_type -> cubelet.services.cubebox.v1.CleanupTemplateResponse + 72, // 120: cubelet.services.cubebox.v1.CubeboxMgr.ListSandboxSnapshots:output_type -> cubelet.services.cubebox.v1.ListSandboxSnapshotsResponse + 75, // 121: cubelet.services.cubebox.v1.CubeboxMgr.ListLocalSnapshots:output_type -> cubelet.services.cubebox.v1.ListLocalSnapshotsResponse + 77, // 122: cubelet.services.cubebox.v1.CubeboxMgr.GetLocalSnapshot:output_type -> cubelet.services.cubebox.v1.GetLocalSnapshotResponse + 79, // 123: cubelet.services.cubebox.v1.CubeboxMgr.GetStorageMetrics:output_type -> cubelet.services.cubebox.v1.GetStorageMetricsResponse + 83, // 124: cubelet.services.cubebox.v1.CubeboxMgr.InspectStorageVolumes:output_type -> cubelet.services.cubebox.v1.InspectStorageVolumesResponse + 86, // 125: cubelet.services.cubebox.v1.CubeboxMgr.CleanupOrphanStorageFiles:output_type -> cubelet.services.cubebox.v1.CleanupOrphanStorageFilesResponse + 111, // [111:126] is the sub-list for method output_type + 96, // [96:111] is the sub-list for method input_type + 96, // [96:96] is the sub-list for extension type_name + 96, // [96:96] is the sub-list for extension extendee + 0, // [0:96] is the sub-list for field type_name } func init() { file_api_services_cubebox_v1_cubebox_proto_init() } @@ -7420,6 +7434,7 @@ func file_api_services_cubebox_v1_cubebox_proto_init() { file_api_services_cubebox_v1_cubebox_proto_msgTypes[39].OneofWrappers = []any{} file_api_services_cubebox_v1_cubebox_proto_msgTypes[40].OneofWrappers = []any{} file_api_services_cubebox_v1_cubebox_proto_msgTypes[47].OneofWrappers = []any{} + file_api_services_cubebox_v1_cubebox_proto_msgTypes[49].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ diff --git a/Cubelet/api/services/cubebox/v1/cubebox.proto b/Cubelet/api/services/cubebox/v1/cubebox.proto index 8ef1c14a7..f7fdc1ea3 100644 --- a/Cubelet/api/services/cubebox/v1/cubebox.proto +++ b/Cubelet/api/services/cubebox/v1/cubebox.proto @@ -750,6 +750,10 @@ message UpdateCubeSandboxRequest { // and the CRI). Whenever possible, however, runtime authors SHOULD // consider proposing new typed fields for any new features instead. map annotations = 3; + // Replacement egress policy for a running sandbox. Absent means "leave the + // network alone"; present means the complete desired state, so an omitted or + // empty field inside it clears whatever is currently installed. + optional CubeNetworkConfig cube_network_config = 4; } message UpdateCubeSandboxResponse { diff --git a/Cubelet/api/services/errorcode/v1/errorcode.pb.go b/Cubelet/api/services/errorcode/v1/errorcode.pb.go index ea6b0df72..647e0d6af 100644 --- a/Cubelet/api/services/errorcode/v1/errorcode.pb.go +++ b/Cubelet/api/services/errorcode/v1/errorcode.pb.go @@ -75,6 +75,7 @@ const ( ErrorCode_DestroyImageFailed ErrorCode = 130569 ErrorCode_TaskPauseFailed ErrorCode = 130588 ErrorCode_TaskResumeFailed ErrorCode = 130589 + ErrorCode_UpdateNetworkFailed ErrorCode = 130590 ErrorCode_InitCommandPathError ErrorCode = 130445 ErrorCode_ContainerStateExitedByUser ErrorCode = 130451 ErrorCode_PullImageFailed ErrorCode = 130456 @@ -144,6 +145,7 @@ var ( 130569: "DestroyImageFailed", 130588: "TaskPauseFailed", 130589: "TaskResumeFailed", + 130590: "UpdateNetworkFailed", 130445: "InitCommandPathError", 130451: "ContainerStateExitedByUser", 130456: "PullImageFailed", @@ -208,6 +210,7 @@ var ( "DestroyImageFailed": 130569, "TaskPauseFailed": 130588, "TaskResumeFailed": 130589, + "UpdateNetworkFailed": 130590, "InitCommandPathError": 130445, "ContainerStateExitedByUser": 130451, "PullImageFailed": 130456, @@ -312,7 +315,7 @@ const file_api_services_errorcode_v1_errorcode_proto_rawDesc = "" + ")api/services/errorcode/v1/errorcode.proto\x12\x1dcubelet.services.errorcode.v1\"c\n" + "\x03Ret\x12C\n" + "\bret_code\x18\x01 \x01(\x0e2(.cubelet.services.errorcode.v1.ErrorCodeR\aretCode\x12\x17\n" + - "\aret_msg\x18\x02 \x01(\tR\x06retMsg*\xe4\f\n" + + "\aret_msg\x18\x02 \x01(\tR\x06retMsg*\xff\f\n" + "\tErrorCode\x12\x06\n" + "\x02OK\x10\x00\x12\x14\n" + "\aUnknown\x10\xff\xff\xff\xff\xff\xff\xff\xff\xff\x01\x12\f\n" + @@ -360,7 +363,8 @@ const file_api_services_errorcode_v1_errorcode_proto_rawDesc = "" + "\x12PreConditionFailed\x10\x88\xfc\a\x12\x18\n" + "\x12DestroyImageFailed\x10\x89\xfc\a\x12\x15\n" + "\x0fTaskPauseFailed\x10\x9c\xfc\a\x12\x16\n" + - "\x10TaskResumeFailed\x10\x9d\xfc\a\x12\x1a\n" + + "\x10TaskResumeFailed\x10\x9d\xfc\a\x12\x19\n" + + "\x13UpdateNetworkFailed\x10\x9e\xfc\a\x12\x1a\n" + "\x14InitCommandPathError\x10\x8d\xfb\a\x12 \n" + "\x1aContainerStateExitedByUser\x10\x93\xfb\a\x12\x15\n" + "\x0fPullImageFailed\x10\x98\xfb\a\x12\x16\n" + diff --git a/Cubelet/api/services/errorcode/v1/errorcode.proto b/Cubelet/api/services/errorcode/v1/errorcode.proto index 225f06ad7..feb5421c7 100644 --- a/Cubelet/api/services/errorcode/v1/errorcode.proto +++ b/Cubelet/api/services/errorcode/v1/errorcode.proto @@ -57,6 +57,7 @@ enum ErrorCode { DestroyImageFailed = 130569; TaskPauseFailed = 130588; TaskResumeFailed = 130589; + UpdateNetworkFailed = 130590; InitCommandPathError = 130445; ContainerStateExitedByUser = 130451; diff --git a/Cubelet/doc/cubelet-api.md b/Cubelet/doc/cubelet-api.md index 35c7cbd7a..41ff10427 100644 --- a/Cubelet/doc/cubelet-api.md +++ b/Cubelet/doc/cubelet-api.md @@ -1812,6 +1812,7 @@ TCPSocketAction describes an action based on opening a socket. | requestID | [string](#string) | | requestID reqID | | sandboxID | [string](#string) | | ID of the Sandbox. | | annotations | [UpdateCubeSandboxRequest.AnnotationsEntry](#cubelet-services-cubebox-v1-UpdateCubeSandboxRequest-AnnotationsEntry) | repeated | Annotations can also be useful for runtime authors to experiment with new features that are opaque to the Kubernetes APIs (both user-facing and the CRI). Whenever possible, however, runtime authors SHOULD consider proposing new typed fields for any new features instead. | +| cube_network_config | [CubeNetworkConfig](#cubelet-services-cubebox-v1-CubeNetworkConfig) | optional | Replacement egress policy for a running sandbox. Absent means "leave the network alone"; present means the complete desired state, so an omitted or empty field inside it clears whatever is currently installed. | @@ -2568,6 +2569,7 @@ CubeVMImageService is used by cube image converter service in vm. | DestroyImageFailed | 130569 | | | TaskPauseFailed | 130588 | | | TaskResumeFailed | 130589 | | +| UpdateNetworkFailed | 130590 | | | InitCommandPathError | 130445 | | | ContainerStateExitedByUser | 130451 | | | PullImageFailed | 130456 | | diff --git a/Cubelet/network/plugin_policy.go b/Cubelet/network/plugin_policy.go index 1dec27c7b..bf55eadcb 100644 --- a/Cubelet/network/plugin_policy.go +++ b/Cubelet/network/plugin_policy.go @@ -6,6 +6,7 @@ package network import ( "context" + "errors" "fmt" "net" "strings" @@ -13,6 +14,7 @@ import ( "github.com/tencentcloud/CubeSandbox/Cubelet/api/services/cubebox/v1" networkruntime "github.com/tencentcloud/CubeSandbox/Cubelet/network/runtime" "github.com/tencentcloud/CubeSandbox/Cubelet/pkg/constants" + "github.com/tencentcloud/CubeSandbox/Cubelet/pkg/container/netfile" "github.com/tencentcloud/CubeSandbox/Cubelet/pkg/log" ) @@ -171,7 +173,27 @@ func mergeDNSAllowOutCIDRs(ctx context.Context, cfg *networkruntime.CubeNetworkC if out == nil { out = &networkruntime.CubeNetworkConfig{} } - dnsAllowOutCIDRs := make([]string, 0, len(dnsServers)) + dnsAllowOutCIDRs := dnsServersToAllowOutCIDRs(ctx, dnsServers) + // CubeVS AllowOut entries are CIDR-only today and cannot express UDP/TCP port 53. + // These resolver CIDRs intentionally keep domain-based allow rules functional + // even when AllowInternetAccess=false; restricting them to DNS ports requires a + // network runtime/CubeVS policy-model extension. + out.AllowOut = appendUniqueString(out.AllowOut, dnsAllowOutCIDRs) + return out, dnsAllowOutCIDRs +} + +// dnsServersToAllowOutCIDRs converts resolved DNS server addresses into +// allow_out CIDRs, dropping the ones CubeVS cannot express. +// +// Kept separate from mergeDNSAllowOutCIDRs because the two questions are +// different: whether to *merge* these into the policy depends on the policy +// naming a domain, but the list itself is a property of the sandbox and is +// recorded unconditionally so a later policy update can fold it back in. +func dnsServersToAllowOutCIDRs(ctx context.Context, dnsServers []string) []string { + if ctx == nil { + ctx = context.Background() + } + cidrs := make([]string, 0, len(dnsServers)) for _, dnsServer := range dnsServers { cidr, ok := dnsServerToCIDR(dnsServer) if !ok { @@ -180,14 +202,9 @@ func mergeDNSAllowOutCIDRs(ctx context.Context, cfg *networkruntime.CubeNetworkC } continue } - dnsAllowOutCIDRs = append(dnsAllowOutCIDRs, cidr) + cidrs = append(cidrs, cidr) } - // CubeVS AllowOut entries are CIDR-only today and cannot express UDP/TCP port 53. - // These resolver CIDRs intentionally keep domain-based allow rules functional - // even when AllowInternetAccess=false; restricting them to DNS ports requires a - // network runtime/CubeVS policy-model extension. - out.AllowOut = appendUniqueString(out.AllowOut, dnsAllowOutCIDRs) - return out, dnsAllowOutCIDRs + return cidrs } // shouldAppendDNSAllowOut keeps the resolver exception narrow: pure IP/CIDR @@ -377,3 +394,56 @@ func appendUniqueString(base []string, extra []string) []string { } return out } + +// ErrSandboxNetworkNotActive reports that the sandbox has no active network, so +// its policy cannot be updated. Callers map it to a client-visible conflict +// rather than an internal failure. +var ErrSandboxNetworkNotActive = networkruntime.ErrNetworkNotActive + +// UpdateSandboxNetworkPolicy replaces a running sandbox's egress policy. +// +// cfg is the complete desired state as authored by the user; the runtime folds +// the sandbox's DNS resolver CIDRs back in, so callers pass exactly what the +// API received. Unlike Create there is no legacy-annotation fallback: the +// update API is new, so a caller that omits the config is a programming error +// rather than an old client. +func UpdateSandboxNetworkPolicy(ctx context.Context, sandboxID string, cfg *cubebox.CubeNetworkConfig) error { + if dnm == nil || dnm.tapPlugin == nil || dnm.tapPlugin.networkRuntime == nil { + return fmt.Errorf("network runtime is not initialized") + } + if sandboxID == "" { + return fmt.Errorf("sandbox id is empty") + } + return dnm.tapPlugin.networkRuntime.UpdateNetworkPolicy(ctx, &networkruntime.UpdateNetworkPolicyRequest{ + SandboxID: sandboxID, + CubeNetworkConfig: mapRunRequestCubeNetworkConfig(cfg), + DNSAllowOutCIDRs: hostDNSAllowOutCIDRs(ctx), + }) +} + +// hostDNSAllowOutCIDRs resolves the node's default DNS servers as allow-out +// CIDRs. The runtime only needs this for sandboxes created before it started +// recording its own resolver list; those it recorded win. Per-container DNS +// overrides are not recoverable here, so such a legacy sandbox falls back to +// the node defaults — still better than losing DNS outright. +func hostDNSAllowOutCIDRs(ctx context.Context) []string { + servers, err := netfile.ResolveEffectiveDNSServers(nil) + if err != nil { + log.G(ctx).Warnf("update network policy: resolve host dns servers failed: %v", err) + return nil + } + cidrs := make([]string, 0, len(servers)) + for _, server := range servers { + if cidr, ok := dnsServerToCIDR(server); ok { + cidrs = append(cidrs, cidr) + } + } + return cidrs +} + +// IsSandboxNetworkNotActive reports whether err means the sandbox has no active +// network, which is a caller mistake (wrong or stopped sandbox) rather than a +// node-side failure. +func IsSandboxNetworkNotActive(err error) bool { + return errors.Is(err, ErrSandboxNetworkNotActive) +} diff --git a/Cubelet/network/plugin_shim.go b/Cubelet/network/plugin_shim.go index 73833c68d..34377a874 100644 --- a/Cubelet/network/plugin_shim.go +++ b/Cubelet/network/plugin_shim.go @@ -16,7 +16,7 @@ import ( // buildEnsureNetworkRequestFromIntent converts the old workflow/shim intent into // the declarative runtime request. It keeps guest-side defaults such as eth0, // gateway ARP and loopback host-port bindings in one place. -func (l *local) buildEnsureNetworkRequestFromIntent(sandboxID, requestID string, exposedPorts []int64, shimReq *NetRequest, cubeNetworkConfig *networkruntime.CubeNetworkConfig) *networkruntime.EnsureNetworkRequest { +func (l *local) buildEnsureNetworkRequestFromIntent(sandboxID, requestID string, exposedPorts []int64, shimReq *NetRequest, cubeNetworkConfig *networkruntime.CubeNetworkConfig, dnsAllowOutCIDRs []string) *networkruntime.EnsureNetworkRequest { desired := &networkruntime.EnsureNetworkRequest{ SandboxID: sandboxID, IdempotencyKey: requestID, @@ -43,6 +43,9 @@ func (l *local) buildEnsureNetworkRequestFromIntent(sandboxID, requestID string, }, } desired.CubeNetworkConfig = cubeNetworkConfig + // Recorded, not re-derived: a later policy update carries only user-authored + // targets and must fold these same resolvers back in. + desired.DNSAllowOutCIDRs = dnsAllowOutCIDRs portReq := make(map[uint16]struct{}) for _, port := range exposedPorts { portReq[uint16(port)] = struct{}{} diff --git a/Cubelet/network/plugin_tap.go b/Cubelet/network/plugin_tap.go index 905fd5b4e..3c29c4a0c 100644 --- a/Cubelet/network/plugin_tap.go +++ b/Cubelet/network/plugin_tap.go @@ -190,7 +190,11 @@ func (l *local) Create(ctx context.Context, opts *workflow.CreateContext) (err e formatCubeNetworkAllowInternetAccess(cubeNetworkConfig), lenCubeNetworkList(cubeNetworkConfig, true), lenCubeNetworkList(cubeNetworkConfig, false), resolvedDNSServers, dnsAllowOutCIDRs, formatNetworkRuntimeCubeNetworkConfig(cubeNetworkConfigBeforeDNS), formatNetworkRuntimeCubeNetworkConfig(cubeNetworkConfig)) - ensureReq := l.buildEnsureNetworkRequestFromIntent(opts.SandboxID, request.GetRequestID(), request.ExposedPorts, req, cubeNetworkConfig) + // Record the sandbox's resolvers even when they were not merged into the + // policy: this create had no domain target, but a later update may add one, + // and by then the resolver list is not recoverable from anywhere else. + ensureReq := l.buildEnsureNetworkRequestFromIntent(opts.SandboxID, request.GetRequestID(), request.ExposedPorts, req, cubeNetworkConfig, + dnsServersToAllowOutCIDRs(ctx, resolvedDNSServers)) log.G(ctx).Infof("tap create ensure request: sandbox_id=%s interfaces=%d routes=%d arps=%d port_mappings=%d resolved_dns_servers=%v dns_allow_out_cidrs=%v cube_network_config=%s persist_metadata=%s", ensureReq.SandboxID, len(ensureReq.Interfaces), len(ensureReq.Routes), len(ensureReq.ARPNeighbors), len(ensureReq.PortMappings), resolvedDNSServers, dnsAllowOutCIDRs, formatNetworkRuntimeCubeNetworkConfig(ensureReq.CubeNetworkConfig), utils.InterfaceToString(ensureReq.PersistMetadata)) diff --git a/Cubelet/network/plugin_tap_create_test.go b/Cubelet/network/plugin_tap_create_test.go index a6571ab97..547bc2817 100644 --- a/Cubelet/network/plugin_tap_create_test.go +++ b/Cubelet/network/plugin_tap_create_test.go @@ -29,14 +29,18 @@ type fakeNetworkRuntime struct { lastEnsureRequest *networkruntime.EnsureNetworkRequest releaseCalled bool lastReleaseRequest *networkruntime.ReleaseNetworkRequest - listTaps []networkruntime.TapState - dumpPolicies map[string]map[string]any - healthErrs []error - healthCalls int - tapFiles []*os.File - getTapFileCalls int - lastTapSandboxID string - lastTapName string + + lastUpdatePolicyRequest *networkruntime.UpdateNetworkPolicyRequest + updatePolicyErr error + + listTaps []networkruntime.TapState + dumpPolicies map[string]map[string]any + healthErrs []error + healthCalls int + tapFiles []*os.File + getTapFileCalls int + lastTapSandboxID string + lastTapName string } func (c *fakeNetworkRuntime) EnsureNetwork(_ context.Context, req *networkruntime.EnsureNetworkRequest) (*networkruntime.EnsureNetworkResponse, error) { @@ -84,6 +88,11 @@ func (c *fakeNetworkRuntime) ReleaseNetwork(_ context.Context, req *networkrunti return &networkruntime.ReleaseNetworkResponse{Released: true, PersistMetadata: req.PersistMetadata}, nil } +func (c *fakeNetworkRuntime) UpdateNetworkPolicy(_ context.Context, req *networkruntime.UpdateNetworkPolicyRequest) error { + c.lastUpdatePolicyRequest = req + return c.updatePolicyErr +} + func (c *fakeNetworkRuntime) ListTaps(_ context.Context, _ *networkruntime.ListTapsRequest) (*networkruntime.ListTapsResponse, error) { stateCounts := map[string]int{} for _, tap := range c.listTaps { diff --git a/Cubelet/network/runtime/controller.go b/Cubelet/network/runtime/controller.go index 5670728f0..8e8e11993 100644 --- a/Cubelet/network/runtime/controller.go +++ b/Cubelet/network/runtime/controller.go @@ -511,6 +511,70 @@ func (s *NetworkController) EnsureNetwork(ctx context.Context, req *EnsureNetwor return state.ensureResponse(), nil } +// UpdateNetworkPolicy replaces the egress policy of a running sandbox. +// +// Step order is the contract, not an implementation detail: +// +// 1. CubeEgress before CubeVS. Whichever way the rule set moves the transient +// state errs towards more interception: a new rule is installed before +// anything is steered at it, and a removed rule is gone before the datapath +// stops steering, so leftover traffic meets the proxy's default deny. +// 2. CubeVS second, which also bumps the policy generation and so is the point +// where established flows start being re-evaluated. +// 3. Persist last. A crash before this replays the previous policy on restart +// — the safe direction, since the caller was never told it succeeded. +// +// There is no rollback. The CubeVS diff is computed from the live maps, so +// replaying the same request converges. +func (s *NetworkController) UpdateNetworkPolicy(ctx context.Context, req *UpdateNetworkPolicyRequest) error { + if req == nil || req.SandboxID == "" { + return fmt.Errorf("sandboxID is required") + } + unlock := func() {} + if s.locks != nil { + unlock = s.locks.Lock(req.SandboxID) + } + defer unlock() + + s.mu.Lock() + state, ok := s.states[req.SandboxID] + s.mu.Unlock() + if !ok { + return fmt.Errorf("%w: sandbox %q", ErrNetworkNotActive, req.SandboxID) + } + + CubeLog.WithContext(ctx).Infof( + "network runtime UpdateNetworkPolicy request: sandbox_id=%s sandbox_ip=%s cube_network_config=%s", + req.SandboxID, state.SandboxIP, formatCubeNetworkConfig(req.CubeNetworkConfig), + ) + + // A sandbox created before the runtime recorded its resolvers has them in + // its installed AllowOut but nowhere we can identify them, so fall back to + // the caller's list rather than silently revoking DNS. Persisting it below + // means each sandbox needs the fallback at most once. + resolverCIDRs := state.DNSAllowOutCIDRs + if len(resolverCIDRs) == 0 { + resolverCIDRs = req.DNSAllowOutCIDRs + } + cfg := withDNSResolverAllowOut(cloneCubeNetworkConfig(req.CubeNetworkConfig), resolverCIDRs) + if err := s.syncEgressPolicy(ctx, state, cfg); err != nil { + return fmt.Errorf("sync CubeEgress policy for sandbox %s: %w", req.SandboxID, err) + } + if err := s.updateCubeVSTapPolicy(state.TapIfIndex, req.SandboxID, cfg); err != nil { + return fmt.Errorf("update CubeVS policy for sandbox %s: %w", req.SandboxID, err) + } + + s.mu.Lock() + state.CubeNetworkConfig = cfg + state.DNSAllowOutCIDRs = resolverCIDRs + s.mu.Unlock() + + if err := s.store.RewriteSuccess(&state.persistedState); err != nil { + return fmt.Errorf("persist network policy for sandbox %s: %w", req.SandboxID, err) + } + return nil +} + // createState builds the network for a sandbox. It does NOT hold s.mu across // the heavy work: the global mutex is only taken briefly inside acquireTap / // releaseAcquiredTap / cleanupConflictingTap to mutate the in-memory pools and @@ -568,6 +632,7 @@ func (s *NetworkController) createState(ctx context.Context, req *EnsureNetworkR ARPNeighbors: slices.Clone(req.ARPNeighbors), PortMappings: actualMappings, CubeNetworkConfig: cloneCubeNetworkConfig(req.CubeNetworkConfig), + DNSAllowOutCIDRs: slices.Clone(req.DNSAllowOutCIDRs), PersistMetadata: s.persistMetadata(req.PersistMetadata, tap.Name, tap.IP.String()), }, tap: tap, diff --git a/Cubelet/network/runtime/controller_test.go b/Cubelet/network/runtime/controller_test.go index b83d9e0d7..b2680f80f 100644 --- a/Cubelet/network/runtime/controller_test.go +++ b/Cubelet/network/runtime/controller_test.go @@ -850,3 +850,265 @@ func TestLoadL7MarksConfig(t *testing.T) { } }) } + +// registerActiveSandbox puts a controller into the state UpdateNetworkPolicy +// requires: an in-memory managedState plus a committed success state file. +func registerActiveSandbox(t *testing.T, c *NetworkController, sandboxID string, cfg *CubeNetworkConfig, dnsCIDRs []string) *managedState { + t.Helper() + state := &managedState{persistedState: persistedState{ + SandboxID: sandboxID, + NetworkHandle: sandboxID, + TapName: "tap-" + sandboxID, + TapIfIndex: 77, + SandboxIP: "10.30.0.7", + CubeNetworkConfig: cfg, + DNSAllowOutCIDRs: dnsCIDRs, + }} + if err := c.store.WriteTmp(&state.persistedState); err != nil { + t.Fatalf("WriteTmp: %v", err) + } + if err := c.store.CommitCreating(sandboxID); err != nil { + t.Fatalf("CommitCreating: %v", err) + } + if err := c.store.CommitSuccess(sandboxID); err != nil { + t.Fatalf("CommitSuccess: %v", err) + } + c.states[sandboxID] = state + return state +} + +func TestUpdateNetworkPolicyAppliesAndPersists(t *testing.T) { + c := newCreateTestController(t, nil) + state := registerActiveSandbox(t, c, "sb-update", &CubeNetworkConfig{AllowOut: []string{"1.1.1.1/32"}}, nil) + + newCfg := &CubeNetworkConfig{AllowOut: []string{"2.2.2.2/32"}} + if err := c.UpdateNetworkPolicy(context.Background(), &UpdateNetworkPolicyRequest{ + SandboxID: "sb-update", + CubeNetworkConfig: newCfg, + }); err != nil { + t.Fatalf("UpdateNetworkPolicy: %v", err) + } + + adapter := c.cubevsAdapter.(*fakeCubeVSAdapter) + if len(adapter.updatedPolicies) != 1 { + t.Fatalf("CubeVS update calls=%d, want 1", len(adapter.updatedPolicies)) + } + if got := adapter.updatedPolicies[0].ifindex; got != 77 { + t.Errorf("updated ifindex=%d, want 77", got) + } + if allow := adapter.updatedPolicies[0].opts.AllowOut; allow == nil || len(*allow) != 1 || (*allow)[0] != "2.2.2.2/32" { + t.Errorf("CubeVS got allow_out %v, want [2.2.2.2/32]", allow) + } + + if len(state.CubeNetworkConfig.AllowOut) != 1 || state.CubeNetworkConfig.AllowOut[0] != "2.2.2.2/32" { + t.Errorf("in-memory state not updated: %v", state.CubeNetworkConfig.AllowOut) + } + persisted, err := c.store.Load("sb-update", StateFileSuccess) + if err != nil { + t.Fatalf("reload success state: %v", err) + } + if persisted.CubeNetworkConfig == nil || len(persisted.CubeNetworkConfig.AllowOut) != 1 || + persisted.CubeNetworkConfig.AllowOut[0] != "2.2.2.2/32" { + t.Errorf("state file not rewritten: %+v", persisted.CubeNetworkConfig) + } +} + +// TestUpdateNetworkPolicyRefoldsDNSResolvers guards the failure mode that would +// break every domain rule: an update carries only user targets, so the resolver +// CIDRs recorded at create must be folded back in. +func TestUpdateNetworkPolicyRefoldsDNSResolvers(t *testing.T) { + c := newCreateTestController(t, nil) + registerActiveSandbox(t, c, "sb-dns", &CubeNetworkConfig{}, []string{"169.254.0.53/32"}) + + if err := c.UpdateNetworkPolicy(context.Background(), &UpdateNetworkPolicyRequest{ + SandboxID: "sb-dns", + CubeNetworkConfig: &CubeNetworkConfig{AllowOut: []string{"api.example.com"}}, + }); err != nil { + t.Fatalf("UpdateNetworkPolicy: %v", err) + } + + allow := c.cubevsAdapter.(*fakeCubeVSAdapter).updatedPolicies[0].opts.AllowOut + if allow == nil { + t.Fatal("CubeVS received no allow_out") + } + var sawResolver bool + for _, target := range *allow { + if target == "169.254.0.53/32" { + sawResolver = true + } + } + if !sawResolver { + t.Errorf("resolver CIDR dropped by update: %v, DNS would break", *allow) + } +} + +// TestUpdateNetworkPolicyFallsBackToCallerResolvers covers the upgrade path: a +// sandbox created before the runtime recorded its resolvers has them installed +// but unidentifiable, so the caller's list is used instead of revoking DNS — +// and persisted, so the fallback is needed at most once per sandbox. +func TestUpdateNetworkPolicyFallsBackToCallerResolvers(t *testing.T) { + c := newCreateTestController(t, nil) + registerActiveSandbox(t, c, "sb-legacy", &CubeNetworkConfig{}, nil) + + if err := c.UpdateNetworkPolicy(context.Background(), &UpdateNetworkPolicyRequest{ + SandboxID: "sb-legacy", + CubeNetworkConfig: &CubeNetworkConfig{AllowOut: []string{"api.example.com"}}, + DNSAllowOutCIDRs: []string{"169.254.0.53/32"}, + }); err != nil { + t.Fatalf("UpdateNetworkPolicy: %v", err) + } + + allow := *c.cubevsAdapter.(*fakeCubeVSAdapter).updatedPolicies[0].opts.AllowOut + var sawResolver bool + for _, target := range allow { + if target == "169.254.0.53/32" { + sawResolver = true + } + } + if !sawResolver { + t.Errorf("caller-supplied resolver was ignored for a legacy sandbox: %v", allow) + } + + persisted, err := c.store.Load("sb-legacy", StateFileSuccess) + if err != nil { + t.Fatalf("reload success state: %v", err) + } + if len(persisted.DNSAllowOutCIDRs) != 1 || persisted.DNSAllowOutCIDRs[0] != "169.254.0.53/32" { + t.Errorf("resolver list not backfilled into state: %v", persisted.DNSAllowOutCIDRs) + } +} + +// TestUpdateNetworkPolicyDropsResolversWithoutDomains is the other half of the +// gate: once no rule needs DNS, the implicit resolver exception goes away too. +// TestUpdateNetworkPolicyDropsResolversWithoutDomains checks the other half of +// the resolver gate: an IP-only policy must not inherit DNS access. +// +// The bare-literal cases are the ones that matter. A DNS name-shape check +// accepts "2.2.2.2" because digits are valid label characters, so gating on it +// silently folded the resolver into every IP-only policy. Masked forms like +// "2.2.2.2/32" happen to fail that check on the slash, which is why they cannot +// stand in for this. +func TestUpdateNetworkPolicyDropsResolversWithoutDomains(t *testing.T) { + l7Port := 443 + for _, tc := range []struct { + name string + cfg *CubeNetworkConfig + }{ + {"bare IPv4", &CubeNetworkConfig{AllowOut: []string{"2.2.2.2"}}}, + {"masked IPv4", &CubeNetworkConfig{AllowOut: []string{"2.2.2.2/32"}}}, + {"subnet", &CubeNetworkConfig{AllowOut: []string{"203.0.113.0/24"}}}, + {"bare IPv4 L7 host", &CubeNetworkConfig{Rules: []*EgressRule{{ + Name: "ip-host", + Match: &EgressRuleMatch{ + Host: stringPtr("2.2.2.2"), Port: &l7Port, Scheme: stringPtr("https"), + }, + Action: &EgressRuleAction{Allow: true}, + }}}}, + } { + t.Run(tc.name, func(t *testing.T) { + c := newCreateTestController(t, nil) + registerActiveSandbox(t, c, "sb-nodns", &CubeNetworkConfig{}, []string{"169.254.0.53/32"}) + + if err := c.UpdateNetworkPolicy(context.Background(), &UpdateNetworkPolicyRequest{ + SandboxID: "sb-nodns", + CubeNetworkConfig: tc.cfg, + }); err != nil { + t.Fatalf("UpdateNetworkPolicy: %v", err) + } + + // A rules-only policy leaves AllowOut unset, which is itself the + // expected outcome here. + allow := c.cubevsAdapter.(*fakeCubeVSAdapter).updatedPolicies[0].opts.AllowOut + if allow == nil { + return + } + for _, target := range *allow { + if target == "169.254.0.53/32" { + t.Errorf("resolver CIDR kept for an IP-only policy: %v", *allow) + } + } + }) + } +} + +// TestUpdateNetworkPolicyEmptyRulesDeletesEgressPolicy pins the empty-rule-set +// fix: PutPolicy short-circuits on no rules, so clearing every L7 rule has to +// go through DeletePolicy or the old rules keep intercepting. +func TestUpdateNetworkPolicyEmptyRulesDeletesEgressPolicy(t *testing.T) { + c := newCreateTestController(t, nil) + registerActiveSandbox(t, c, "sb-clear", &CubeNetworkConfig{ + Rules: []*EgressRule{{Name: "r1"}}, + }, nil) + + if err := c.UpdateNetworkPolicy(context.Background(), &UpdateNetworkPolicyRequest{ + SandboxID: "sb-clear", + CubeNetworkConfig: &CubeNetworkConfig{}, + }); err != nil { + t.Fatalf("UpdateNetworkPolicy: %v", err) + } + + egress := c.cubeEgressAdapter.(*fakeCubeEgressAdapter) + if egress.deleteCalls != 1 { + t.Errorf("DeletePolicy calls=%d, want 1", egress.deleteCalls) + } + if egress.putCalls != 0 { + t.Errorf("PutPolicy calls=%d, want 0 for an empty rule set", egress.putCalls) + } +} + +// TestUpdateNetworkPolicyL7UntouchedSkipsCubeEgress pins that an L3-only update +// never talks to CubeEgress. Otherwise a sandbox that has no L7 rules — and so +// nothing installed on the proxy — would still fail its policy updates whenever +// CubeEgress happens to be down. +func TestUpdateNetworkPolicyL7UntouchedSkipsCubeEgress(t *testing.T) { + c := newCreateTestController(t, nil) + registerActiveSandbox(t, c, "sb-l3only", &CubeNetworkConfig{AllowOut: []string{"1.1.1.1/32"}}, nil) + egress := c.cubeEgressAdapter.(*fakeCubeEgressAdapter) + egress.deleteErr = errors.New("connection refused") + + if err := c.UpdateNetworkPolicy(context.Background(), &UpdateNetworkPolicyRequest{ + SandboxID: "sb-l3only", + CubeNetworkConfig: &CubeNetworkConfig{AllowOut: []string{"2.2.2.2/32"}}, + }); err != nil { + t.Fatalf("L3-only update failed because of CubeEgress: %v", err) + } + if egress.deleteCalls != 0 || egress.putCalls != 0 { + t.Errorf("CubeEgress was contacted for an L3-only update: delete=%d put=%d", + egress.deleteCalls, egress.putCalls) + } +} + +func TestUpdateNetworkPolicyUnknownSandbox(t *testing.T) { + c := newCreateTestController(t, nil) + err := c.UpdateNetworkPolicy(context.Background(), &UpdateNetworkPolicyRequest{ + SandboxID: "sb-missing", + CubeNetworkConfig: &CubeNetworkConfig{}, + }) + if !errors.Is(err, ErrNetworkNotActive) { + t.Fatalf("err=%v, want ErrNetworkNotActive so callers can return a conflict", err) + } +} + +// TestUpdateNetworkPolicyKeepsOldPolicyOnCubeVSFailure pins the no-rollback +// contract's safe half: a failed update must not advance the durable state, so +// a restart re-applies the policy the sandbox actually ran with. +func TestUpdateNetworkPolicyKeepsOldPolicyOnCubeVSFailure(t *testing.T) { + c := newCreateTestController(t, nil) + registerActiveSandbox(t, c, "sb-fail", &CubeNetworkConfig{AllowOut: []string{"1.1.1.1/32"}}, nil) + c.cubevsAdapter.(*fakeCubeVSAdapter).updateTAPPolicyErr = errors.New("map update failed") + + if err := c.UpdateNetworkPolicy(context.Background(), &UpdateNetworkPolicyRequest{ + SandboxID: "sb-fail", + CubeNetworkConfig: &CubeNetworkConfig{AllowOut: []string{"2.2.2.2/32"}}, + }); err == nil { + t.Fatal("UpdateNetworkPolicy succeeded despite a CubeVS failure") + } + + persisted, err := c.store.Load("sb-fail", StateFileSuccess) + if err != nil { + t.Fatalf("reload success state: %v", err) + } + if persisted.CubeNetworkConfig.AllowOut[0] != "1.1.1.1/32" { + t.Errorf("failed update was persisted: %v", persisted.CubeNetworkConfig.AllowOut) + } +} diff --git a/Cubelet/network/runtime/cubeegress_adapter.go b/Cubelet/network/runtime/cubeegress_adapter.go index 006ba88a5..852326d37 100644 --- a/Cubelet/network/runtime/cubeegress_adapter.go +++ b/Cubelet/network/runtime/cubeegress_adapter.go @@ -127,6 +127,42 @@ func (s *NetworkController) pushEgressForState(ctx context.Context, state *manag return s.putEgressPolicyForCreate(ctx, state, in) } +// syncEgressPolicy converges CubeEgress on cfg's rule set for a live sandbox. +// +// Whether CubeEgress is involved at all is decided by the rule sets alone: +// +// want != nil -> install the new rules +// want == nil, installed != nil -> clear the rules the sandbox still has +// want == nil, installed == nil -> L7 is not in play; do not touch CubeEgress +// +// The third row is why this exists. pushEgressForState can stop at "no rules, no +// call" because on create nothing is installed yet, but an update must also be +// able to clear. Deriving that from the rule sets keeps an L3-only sandbox — one +// that never had L7 rules and still doesn't — independent of CubeEgress instead +// of failing its policy updates whenever the proxy is unreachable. +// +// `installed` comes from the state's own config, which can under-report if an +// earlier update pushed rules and then failed before persisting. Those leftovers +// are inert (the datapath was never told to steer at them) and CubeEgress +// re-seeds from DumpEgressPolicies on its next reload. +func (s *NetworkController) syncEgressPolicy(ctx context.Context, state *managedState, cfg *CubeNetworkConfig) error { + want := toEgressInput(cfg) + installed := toEgressInput(state.CubeNetworkConfig) + if want == nil && installed == nil { + return nil + } + // L7 is in play, so CubeEgress has to be part of this deployment. An unset + // admin URL means it is not (dev mode), which the create and release paths + // also treat as a silent no-op. + if s.cubeEgressAdapter == nil || !s.cubeEgressAdapter.Configured() { + return nil + } + if want == nil { + return s.deleteEgressForState(ctx, state.SandboxID, state.SandboxIP) + } + return s.putEgressPolicyForCreate(ctx, state, want) +} + // putEgressPolicyForCreate keeps sandbox creation from failing on a single // transient CubeEgress admin blip. Three total attempts are deliberately small: // with the default 2s per-call timeout the worst case is bounded around six diff --git a/Cubelet/network/runtime/cubevs_adapter.go b/Cubelet/network/runtime/cubevs_adapter.go index 8bbee2d92..dc6ea3f04 100644 --- a/Cubelet/network/runtime/cubevs_adapter.go +++ b/Cubelet/network/runtime/cubevs_adapter.go @@ -21,6 +21,7 @@ type CubeVSAdapter interface { AddTAPDevice(ifindex uint32, ip net.IP, sandboxID string, version uint32, opts cubevs.MVMOptions) error UpsertTAPDevice(ifindex uint32, ip net.IP, sandboxID string, version uint32, opts cubevs.MVMOptions) error UpsertTAPDeviceMetadata(ifindex uint32, ip net.IP, sandboxID string, version uint32) error + UpdateTAPPolicy(ifindex uint32, opts cubevs.MVMOptions) error GetTAPDevice(ifindex uint32) (*cubevs.TAPDevice, error) CleanupTAPPolicy(ifindex uint32) error DeleteTAPDeviceMetadata(ifindex uint32, ip net.IP) error @@ -52,6 +53,10 @@ func (realCubeVSAdapter) UpsertTAPDeviceMetadata(ifindex uint32, ip net.IP, sand return cubevs.UpsertTAPDeviceMetadata(ifindex, ip, sandboxID, version) } +func (realCubeVSAdapter) UpdateTAPPolicy(ifindex uint32, opts cubevs.MVMOptions) error { + return cubevs.UpdateTAPDevicePolicy(ifindex, opts) +} + func (realCubeVSAdapter) GetTAPDevice(ifindex uint32) (*cubevs.TAPDevice, error) { return cubevs.GetTAPDevice(ifindex) } @@ -119,6 +124,22 @@ func (s *NetworkController) registerCubeVSTap(ifindex int, ip net.IP, sandboxID return err } +// updateCubeVSTapPolicy converges an active sandbox's CubeVS policy on cfg and +// bumps its policy generation so established flows are re-evaluated. Unlike +// register/replace it neither touches TAP metadata nor flushes the policy maps, +// because the sandbox is live and must not lose its policy mid-update. +func (s *NetworkController) updateCubeVSTapPolicy(ifindex int, sandboxID string, cfg *CubeNetworkConfig) error { + opts, err := cubeVSTapRegistration(cfg) + if err != nil { + return err + } + CubeLog.WithContext(context.Background()).Infof( + "network runtime update cubevs policy: sandbox_id=%s ifindex=%d cube_network_config=%s", + sandboxID, ifindex, formatCubeNetworkConfig(cfg), + ) + return s.cubevsAdapter.UpdateTAPPolicy(uint32(ifindex), opts) +} + // replaceCubeVSTap rewrites TAP metadata and the complete desired CubeVS policy. // Legacy recovery uses this to avoid carrying old allow/deny/DNS residue into // the recovered Active sandbox. diff --git a/Cubelet/network/runtime/cubevs_adapter_test.go b/Cubelet/network/runtime/cubevs_adapter_test.go index 650ccb4c8..6e6baf2a4 100644 --- a/Cubelet/network/runtime/cubevs_adapter_test.go +++ b/Cubelet/network/runtime/cubevs_adapter_test.go @@ -22,9 +22,18 @@ type fakeCubeVSAdapter struct { cleanupPolicyErr error deleteMetadataErr error deletePortMappingErr error + updatedPolicies []updatedPolicy + updateTAPPolicyErr error recorder *createOrderRecorder } +// updatedPolicy records one UpdateTAPPolicy call so tests can assert both that +// the policy update reached CubeVS and what it carried. +type updatedPolicy struct { + ifindex uint32 + opts cubevs.MVMOptions +} + func (f *fakeCubeVSAdapter) AddTAPDevice(_ uint32, _ net.IP, _ string, _ uint32, _ cubevs.MVMOptions) error { if f.recorder != nil { f.recorder.record("cubevs_tap") @@ -41,6 +50,11 @@ func (f *fakeCubeVSAdapter) UpsertTAPDeviceMetadata(_ uint32, _ net.IP, _ string return nil } +func (f *fakeCubeVSAdapter) UpdateTAPPolicy(ifindex uint32, opts cubevs.MVMOptions) error { + f.updatedPolicies = append(f.updatedPolicies, updatedPolicy{ifindex: ifindex, opts: opts}) + return f.updateTAPPolicyErr +} + func (f *fakeCubeVSAdapter) GetTAPDevice(ifindex uint32) (*cubevs.TAPDevice, error) { if _, deleted := f.deletedTAPDevices[ifindex]; deleted { return nil, ebpf.ErrKeyNotExist diff --git a/Cubelet/network/runtime/network_runtime.go b/Cubelet/network/runtime/network_runtime.go index 466756371..d0eb29efb 100644 --- a/Cubelet/network/runtime/network_runtime.go +++ b/Cubelet/network/runtime/network_runtime.go @@ -16,6 +16,11 @@ import ( // another retry may already have started after EnsureNetwork released its lock. var ErrEnsureNetworkCommitted = errors.New("network ensure success is already committed") +// ErrNetworkNotActive reports that a sandbox has no active network, so its +// policy cannot be updated. Distinct from a generic failure because callers +// surface it as a client-side conflict rather than a server error. +var ErrNetworkNotActive = errors.New("sandbox network is not active") + // NetworkRuntime is the in-process network runtime interface used by Cubelet. // Implementations must make EnsureNetwork and ReleaseNetwork idempotent for the // same sandbox so Cubelet can safely retry after process or RPC failures. @@ -26,6 +31,10 @@ type NetworkRuntime interface { // ownership has been handed to the runtime, not necessarily after every kernel // side effect has completed. ReleaseNetwork(ctx context.Context, req *ReleaseNetworkRequest) (*ReleaseNetworkResponse, error) + // UpdateNetworkPolicy replaces the egress policy of a running sandbox. It + // returns ErrNetworkNotActive when the sandbox has no active network, which + // callers map to a "not running" client error. + UpdateNetworkPolicy(ctx context.Context, req *UpdateNetworkPolicyRequest) error // ListTaps returns the TAP pool state machine snapshot used by diagnostics. ListTaps(ctx context.Context, req *ListTapsRequest) (*ListTapsResponse, error) // Health reports whether the runtime process can still serve requests. diff --git a/Cubelet/network/runtime/policy_builder.go b/Cubelet/network/runtime/policy_builder.go index 681c2afb2..e9f840e64 100644 --- a/Cubelet/network/runtime/policy_builder.go +++ b/Cubelet/network/runtime/policy_builder.go @@ -8,6 +8,7 @@ import ( "errors" "fmt" "net" + "slices" "strings" "github.com/tencentcloud/CubeSandbox/CubeNet/cubevs" @@ -123,6 +124,48 @@ func formatCubeNetworkConfig(in *CubeNetworkConfig) string { // allow_internet_access / allow_out / deny_out, and it also receives network // targets extracted from L7 rules as L7 allow targets. The complete L7 rules // are still pushed to CubeEgress separately. +// withDNSResolverAllowOut folds the sandbox's resolver CIDRs back into cfg. +// +// An update request carries only user-authored targets, but the create path +// appended the resolver /32s so domain rules could be resolved at all. Without +// this the first update of a domain-based policy would revoke DNS itself and +// black-hole every domain rule it just installed. +// +// Same condition as the create path: only a policy that still names a domain +// keeps the resolver exception, so an update that drops every domain also drops +// the implicit DNS access. +func withDNSResolverAllowOut(cfg *CubeNetworkConfig, resolverCIDRs []string) *CubeNetworkConfig { + if cfg == nil || len(resolverCIDRs) == 0 || !needsDNSResolution(cfg) { + return cfg + } + for _, cidr := range resolverCIDRs { + if !slices.Contains(cfg.AllowOut, cidr) { + cfg.AllowOut = append(cfg.AllowOut, cidr) + } + } + return cfg +} + +// needsDNSResolution reports whether any allow_out target or L7 rule host is a +// domain. It asks the predicate that mirrors where cubevs actually installs a +// target, so a bare IPv4 literal does not read as a domain -- a name-shape check +// accepts "10.0.0.1" because digits are valid DNS label characters, and folding +// the resolver in for an IP-only policy would grant access nobody asked for. +func needsDNSResolution(cfg *CubeNetworkConfig) bool { + if slices.ContainsFunc(cfg.AllowOut, cubevs.IsAllowOutDomainTarget) { + return true + } + targets, err := extractL7AllowOutTargetsFromRules(cfg.Rules) + if err != nil { + // Malformed rules are rejected later with a precise error. Assume DNS is + // needed so a bad request cannot silently strip resolver access. + return true + } + return slices.ContainsFunc(targets, func(t cubevs.L7Target) bool { + return cubevs.IsAllowOutDomainTarget(t.Host) + }) +} + func cubeVSTapRegistration(cfg *CubeNetworkConfig) (cubevs.MVMOptions, error) { if cfg == nil { allowInternetAccess := true diff --git a/Cubelet/network/runtime/state_store.go b/Cubelet/network/runtime/state_store.go index c1274ccae..2f2997e10 100644 --- a/Cubelet/network/runtime/state_store.go +++ b/Cubelet/network/runtime/state_store.go @@ -32,7 +32,13 @@ type persistedState struct { ARPNeighbors []ARPNeighbor `json:"arpNeighbors"` PortMappings []PortMapping `json:"portMappings"` CubeNetworkConfig *CubeNetworkConfig `json:"-"` - PersistMetadata map[string]string `json:"persistMetadata"` + // DNSAllowOutCIDRs are the resolver /32s already folded into + // CubeNetworkConfig.AllowOut so domain rules can be resolved at all. They + // are kept separately because a policy update replaces AllowOut wholesale + // and has to fold the same resolvers back in — the caller only knows the + // user-authored targets. + DNSAllowOutCIDRs []string `json:"dnsAllowOutCIDRs,omitempty"` + PersistMetadata map[string]string `json:"persistMetadata"` } // persistedStateOnDisk is the JSON compatibility layer. CubeNetworkConfig is @@ -50,6 +56,7 @@ type persistedStateOnDisk struct { PortMappings []PortMapping `json:"portMappings"` CubeNetworkConfig *CubeNetworkConfig `json:"cubeNetworkConfig,omitempty"` LegacyCubeVSContext *CubeNetworkConfig `json:"cubevsContext,omitempty"` + DNSAllowOutCIDRs []string `json:"dnsAllowOutCIDRs,omitempty"` PersistMetadata map[string]string `json:"persistMetadata"` } @@ -69,6 +76,7 @@ func (s *persistedState) MarshalJSON() ([]byte, error) { PortMappings: s.PortMappings, CubeNetworkConfig: s.CubeNetworkConfig, LegacyCubeVSContext: s.CubeNetworkConfig, + DNSAllowOutCIDRs: s.DNSAllowOutCIDRs, PersistMetadata: s.PersistMetadata, } return json.Marshal(&disk) @@ -95,6 +103,7 @@ func (s *persistedState) UnmarshalJSON(data []byte) error { } else { s.CubeNetworkConfig = disk.LegacyCubeVSContext } + s.DNSAllowOutCIDRs = disk.DNSAllowOutCIDRs s.PersistMetadata = disk.PersistMetadata return nil } @@ -245,6 +254,44 @@ func (s *stateStore) CommitSuccess(sandboxID string) error { return s.rename(sandboxID, StateFileCreating, StateFileSuccess) } +// RewriteSuccess replaces the committed state of an already-active sandbox. The +// policy update path uses it to persist a new CubeNetworkConfig without moving +// the sandbox out of the success stage. +// +// The new bytes land in a scratch file and are renamed over the success file, so +// a crash leaves either the old state or the new one, never a torn mix. The +// scratch suffix is deliberately not a state-file kind: parseStateFileName +// rejects it, so a leftover is invisible to Load/LoadAny/Scan rather than being +// mistaken for an interrupted create. +func (s *stateStore) RewriteSuccess(state *persistedState) error { + if state == nil { + return fmt.Errorf("state is nil") + } + if err := validateStateForStore(state); err != nil { + return err + } + p, err := s.path(state.SandboxID, StateFileSuccess) + if err != nil { + return err + } + if _, err := os.Stat(p); err != nil { + return fmt.Errorf("sandbox %s has no committed network state: %w", state.SandboxID, err) + } + data, err := json.MarshalIndent(state, "", " ") + if err != nil { + return err + } + scratch := p + ".new" + if err := s.writeStateFile(scratch, data, 0o600); err != nil { + return err + } + if err := os.Rename(scratch, p); err != nil { + _ = os.Remove(scratch) + return err + } + return s.maybeSyncDir(filepath.Dir(p)) +} + // MarkDeleting atomically transfers ownership from active runtime state to the // cleanup/recovery path. func (s *stateStore) MarkDeleting(sandboxID string) error { diff --git a/Cubelet/network/runtime/types.go b/Cubelet/network/runtime/types.go index 6ee398105..abde14937 100644 --- a/Cubelet/network/runtime/types.go +++ b/Cubelet/network/runtime/types.go @@ -14,7 +14,11 @@ type EnsureNetworkRequest struct { ARPNeighbors []ARPNeighbor `json:"arpNeighbors,omitempty"` PortMappings []PortMapping `json:"portMappings,omitempty"` CubeNetworkConfig *CubeNetworkConfig `json:"cubeNetworkConfig,omitempty"` - PersistMetadata map[string]string `json:"persistMetadata,omitempty"` + // DNSAllowOutCIDRs are the resolver /32s the caller already folded into + // CubeNetworkConfig.AllowOut. Recorded so a later policy update, which only + // carries user-authored targets, can fold the same resolvers back in. + DNSAllowOutCIDRs []string `json:"dnsAllowOutCIDRs,omitempty"` + PersistMetadata map[string]string `json:"persistMetadata,omitempty"` } // EnsureNetworkResponse is the concrete network shape assigned by the runtime. @@ -38,6 +42,18 @@ type ReleaseNetworkRequest struct { PersistMetadata map[string]string `json:"persistMetadata,omitempty"` } +// UpdateNetworkPolicyRequest replaces the egress policy of a running sandbox. +// CubeNetworkConfig is the complete desired state, not a patch: an omitted or +// empty field clears whatever is currently installed. +type UpdateNetworkPolicyRequest struct { + SandboxID string `json:"sandboxID,omitempty"` + CubeNetworkConfig *CubeNetworkConfig `json:"cubeNetworkConfig,omitempty"` + // DNSAllowOutCIDRs is a fallback resolver list, used only for sandboxes + // created before the runtime started recording its own. See + // NetworkController.UpdateNetworkPolicy. + DNSAllowOutCIDRs []string `json:"dnsAllowOutCIDRs,omitempty"` +} + // ReleaseNetworkResponse confirms the release handoff and returns the metadata // persisted at creation time when the network existed. type ReleaseNetworkResponse struct { diff --git a/Cubelet/pkg/constants/const.go b/Cubelet/pkg/constants/const.go index 1c8152688..d6ed3fe0c 100644 --- a/Cubelet/pkg/constants/const.go +++ b/Cubelet/pkg/constants/const.go @@ -338,6 +338,7 @@ const ( UpdateActionRemoveDevice = "removeDevice" UpdateActionPause = "pause" UpdateActionResume = "resume" + UpdateActionNetwork = "network" PreStopTypePause = "pause" PreStopTypeDestroy = "destroy" ) diff --git a/Cubelet/services/cubebox/update.go b/Cubelet/services/cubebox/update.go index 8f6f25edd..5530005c6 100644 --- a/Cubelet/services/cubebox/update.go +++ b/Cubelet/services/cubebox/update.go @@ -13,10 +13,12 @@ import ( containerd "github.com/containerd/containerd/v2/client" "github.com/containerd/containerd/v2/pkg/namespaces" "github.com/containerd/ttrpc" + "google.golang.org/protobuf/proto" "k8s.io/apimachinery/pkg/api/resource" "github.com/tencentcloud/CubeSandbox/Cubelet/api/services/cubebox/v1" "github.com/tencentcloud/CubeSandbox/Cubelet/api/services/errorcode/v1" + "github.com/tencentcloud/CubeSandbox/Cubelet/network" "github.com/tencentcloud/CubeSandbox/Cubelet/pkg/config" "github.com/tencentcloud/CubeSandbox/Cubelet/pkg/constants" "github.com/tencentcloud/CubeSandbox/Cubelet/pkg/log" @@ -108,6 +110,8 @@ func (s *service) Update(ctx context.Context, req *cubebox.UpdateCubeSandboxRequ return rsp, nil } return s.updateWithPauseCow(ctx, req, sb) + case constants.UpdateActionNetwork: + return s.updateNetworkPolicy(ctx, req, sb, rsp) case constants.UpdateActionResume: // Resume is Master Create(same sandboxID from pause snap), not Update(resume). rsp.Ret.RetMsg = "pause resume is owned by CubeMaster Create; Update(resume) is not supported" @@ -635,3 +639,50 @@ func reconcileStuckPausingSandbox(ctx context.Context, client *containerd.Client convergePauseStateFromShim(ctx, cb, st.Status, fmt.Sprintf("DeadGC stuck PAUSING for %s", stuckFor)) } + +// updateNetworkPolicy applies a new egress policy to a running sandbox. The +// caller owns the sandbox lifecycle lock. +// +// The datapath is converged first and the store second. Persisting first would +// risk claiming a policy that never reached the node; this way a failure leaves +// both the store and the node on the previous policy. +func (s *service) updateNetworkPolicy( + ctx context.Context, + req *cubebox.UpdateCubeSandboxRequest, + sb *cubeboxstore.CubeBox, + rsp *cubebox.UpdateCubeSandboxResponse, +) (*cubebox.UpdateCubeSandboxResponse, error) { + cfg := req.GetCubeNetworkConfig() + if cfg == nil { + rsp.Ret.RetMsg = "must provide cube_network_config for network update" + rsp.Ret.RetCode = errorcode.ErrorCode_InvalidParamFormat + return rsp, nil + } + if sb.GetStatus().IsPaused() { + rsp.Ret.RetMsg = "cannot update network policy of a paused sandbox" + rsp.Ret.RetCode = errorcode.ErrorCode_Conflict + return rsp, nil + } + + if err := network.UpdateSandboxNetworkPolicy(ctx, req.SandboxID, cfg); err != nil { + log.G(ctx).Errorf("update network policy failed sandbox=%s err=%v", req.SandboxID, err) + rsp.Ret.RetMsg = err.Error() + if network.IsSandboxNetworkNotActive(err) { + rsp.Ret.RetCode = errorcode.ErrorCode_Conflict + } else { + rsp.Ret.RetCode = errorcode.ErrorCode_UpdateNetworkFailed + } + return rsp, nil + } + + // Mirror the new policy into the sandbox object: it is what pause packages, + // so without this a pause/resume cycle would silently restore the policy the + // sandbox was created with. + sb.CubeNetworkConfig = proto.Clone(cfg).(*cubebox.CubeNetworkConfig) + if err := s.cubeboxMgr.cubeboxManger.SyncByID(ctx, sb.ID); err != nil { + // The datapath already carries the new policy, so this is not a failed + // update — only a durability gap that a later sync or resume may expose. + log.G(ctx).Errorf("persist updated network policy failed sandbox=%s err=%v", sb.ID, err) + } + return rsp, nil +} diff --git a/docs/guide/network-policy.md b/docs/guide/network-policy.md index c107a98f6..291d90152 100644 --- a/docs/guide/network-policy.md +++ b/docs/guide/network-policy.md @@ -256,6 +256,34 @@ Cubelet embedded network runtime finally programs different sources into differe If the same IP/CIDR appears in both plain `allow_out` and an L7 rule target, CubeVS preserves the `L7_REQUIRED` flag. Static `allow_out` entries do not expire; DNS-learned entries have `expires_at_ns` and expire according to DNS TTL. +## Updating the policy of a running sandbox + +`PUT /sandboxes/{sandboxID}/network` replaces the egress policy of a sandbox that is already running. The body is the same `network` object accepted at create time, and it is a **replacement, not a patch**: a field you leave out is cleared. + +```bash +curl -X PUT "$CUBE_API/sandboxes/$SANDBOX_ID/network" \ + -H 'Content-Type: application/json' \ + -d '{"allowInternetAccess": false, "network": {"allowOut": ["api.example.com"], "denyOut": ["0.0.0.0/0"]}}' +``` + +The SDKs expose it as `sandbox.update_network(...)` (Python), `sandbox.updateNetwork(...)` (Node) and `sandbox.UpdateNetwork(...)` (Go), taking the same argument shape as sandbox creation. + +### What happens to connections that are already open + +Unlike a plain map rewrite, an update also reaches traffic that already exists. Each sandbox carries a policy generation (`mvm_meta.policy_version`) that the update bumps once both CubeEgress and the CubeVS maps hold the new policy. Every session caches the generation it was admitted under, so the next packet on an established flow is re-evaluated exactly once per update: + +- **Still allowed with the same verdict** — the session is restamped with the new generation and continues untouched. This is the common case and costs one policy lookup per flow per update. +- **No longer allowed, or its verdict changed** (for example a host that moved between plain SNAT and L7 interception) — the session is retired immediately: both directions of its conntrack state are deleted, so replies stop being delivered and the 4-tuple is free for a reconnect. TCP is answered with an RST, matching how every other unreachable TCP packet is handled, so the guest fails fast instead of stalling on retransmits; UDP and ICMP have nothing to reset and are dropped. + +A verdict *change* retires the flow rather than migrating it, because the SNAT and L7 paths disagree about both the reply tuple and which side terminates the TCP connection. The reconnect that follows is evaluated against the new policy like any new flow. + +Re-evaluation is driven by traffic, not pushed: an idle established connection is only judged when the sandbox next sends on it. A connection that is open but silent therefore stays in the session table until it either sends again or times out normally. + +Two consequences worth planning for: + +- **Existing DNS-learned IPs outlive the domain rule that created them.** Removing a domain from `allow_out` stops new IPs from being learned immediately, but IPs already learned for it remain allowed until their DNS TTL expires. Revoking access to a domain promptly requires an `allow_internet_access=false` policy plus a short resolver TTL. +- **The update is not transactional across planes.** CubeEgress is updated before CubeVS, and the durable state is written last, so a failure leaves the sandbox on a state no more permissive than the old and new policies combined. Replaying the same request converges; a Cubelet restart re-applies the last policy that was successfully persisted. + ## How `from_cube` decides and forwards `from_cube` is the TC eBPF program attached to sandbox TAP ingress. Every packet sent by the sandbox enters this program first. You can think of it in these stages: diff --git a/docs/zh/guide/network-policy.md b/docs/zh/guide/network-policy.md index a0d6ed7db..1c27e500a 100644 --- a/docs/zh/guide/network-policy.md +++ b/docs/zh/guide/network-policy.md @@ -256,6 +256,34 @@ Cubelet 内置 network runtime 最终把不同来源写入不同 map： 如果同一个 IP/CIDR 既来自普通 `allow_out`，又来自 L7 规则，CubeVS 会保留 `L7_REQUIRED` 标记。静态 `allow_out` 条目不过期；DNS 学习出的条目带 `expires_at_ns`，会按 TTL 过期。 +## 更新运行中沙箱的策略 + +`PUT /sandboxes/{sandboxID}/network` 用于替换正在运行的沙箱的出站策略。请求体就是创建时那个 `network` 对象，且语义是**整体替换而不是增量打补丁**：没传的字段会被清空。 + +```bash +curl -X PUT "$CUBE_API/sandboxes/$SANDBOX_ID/network" \ + -H 'Content-Type: application/json' \ + -d '{"allowInternetAccess": false, "network": {"allowOut": ["api.example.com"], "denyOut": ["0.0.0.0/0"]}}' +``` + +各 SDK 分别暴露为 `sandbox.update_network(...)`（Python）、`sandbox.updateNetwork(...)`（Node）和 `sandbox.UpdateNetwork(...)`（Go），入参形状与创建沙箱时一致。 + +### 已经建立的连接会怎样 + +和单纯重写 map 不同，更新也会作用到已有流量。每个沙箱带一个策略代际（`mvm_meta.policy_version`），在 CubeEgress 和 CubeVS map 都写入新策略之后才递增。每条 session 会缓存自己被放行时的代际，因此每次更新后，存量流的下一个包会被重新判定一次，且只判定一次： + +- **仍然放行且判决不变** —— session 被重新盖上新代际后照常继续。这是常见情况，每条流每次更新只多付一次策略查表。 +- **不再放行，或判决发生变化**（例如某个 host 在普通 SNAT 和 L7 代理之间切换）—— session 立即作废：出入两个方向的连接跟踪记录都被删掉，因此回包不再投递，该 4 元组也立刻空出来可供重连。TCP 会收到 RST，与这里对其他所有不可达 TCP 报文的处理方式一致，让 guest 立即失败而不是卡在重传上；UDP 和 ICMP 没有可 reset 的东西，直接丢弃。 + +判决发生变化时选择作废而不是迁移这条流，是因为 SNAT 路径和 L7 路径对回包元组、以及由谁来终结这条 TCP 连接这两件事的理解并不一致。随后的重连会像任何新流一样按新策略判定。 + +重判是由流量驱动的，不是主动推送的：一条空闲的存量连接要等沙箱下一次在它上面发包时才会被判定。因此一条打开但一直不说话的连接会留在 session 表里，直到它再次发包或按正常超时被回收。 + +有两点需要提前规划： + +- **已经学到的 DNS IP 会比产生它的域名规则活得更久。** 从 `allow_out` 里删掉一个域名后，新的 IP 会立刻停止被学习，但此前已为它学到的 IP 会一直放行到 DNS TTL 过期。要想及时收回对某个域名的访问，需要配合 `allow_internet_access=false` 的策略和较短的解析 TTL。 +- **更新在多个平面之间不是事务性的。** 先更新 CubeEgress，再更新 CubeVS，最后才写持久化状态，因此中途失败时沙箱所处的状态不会比「新旧策略的并集」更宽松。重放同一个请求即可收敛；Cubelet 重启后会重新应用最后一次成功持久化的策略。 + ## `from_cube` 如何判断和转发 `from_cube` 是挂在沙箱 TAP ingress 上的 TC eBPF 程序。每个沙箱发出的包都会先进入这里。处理顺序可以理解为下面几个阶段。 diff --git a/examples/network-policy/README.md b/examples/network-policy/README.md index 5518b0ae9..0bada991c 100644 --- a/examples/network-policy/README.md +++ b/examples/network-policy/README.md @@ -1,7 +1,8 @@ # Network Policy -Control outbound network access for a Cube Sandbox at creation time. -Three modes are provided: fully air-gapped, CIDR allowlist, and CIDR denylist. +Control outbound network access for a Cube Sandbox. Three modes are set at +creation time — fully air-gapped, CIDR allowlist, CIDR denylist — and a fourth +example changes the policy of a sandbox that is already running. ## 1. Background @@ -66,6 +67,40 @@ Sandbox.create( ) ``` +### Mode 4 — Update a running sandbox (`network_dynamic_update.py`) + +Replace the egress policy of a sandbox that is already running, without losing +its filesystem or process state. Use this when an agent must be granted a +destination partway through a task, or must lose one as soon as a step finishes. + +```python +sandbox.update_network( + network={"allow_out": ["8.8.8.8/32"]}, + allow_internet_access=False, +) +``` + +It deliberately takes the same `network` / `allow_internet_access` pair as +`Sandbox.create`, so a policy can be moved between the two without rewriting it. + +Two things set this apart from the create-time modes: + +- **It is a replacement, not a patch.** A key you leave out is cleared, so + `sandbox.update_network(network={}, allow_internet_access=False)` revokes + everything. +- **It reaches connections that are already open.** A connection the new policy + no longer permits is reset, rather than surviving until the peer closes it. + This is what makes revocation take effect rather than merely apply to future + connections. + +This is a CubeSandbox extension, so the example uses the `cubesandbox` SDK +(`sandbox.updateNetwork(...)` in Node, `sandbox.UpdateNetwork(...)` in Go). + +The script walks four scenarios, in order: an IP allow list, a live connection +carried across a revoking update, a domain allow list, and switching on L7 +interception mid-run. Unlike E2B — where `network.rules` is create-only — L7 +rules can be changed on a running sandbox. + ## 3. Policy Summary | Mode | `allow_internet_access` | `network` key | Effect | @@ -73,6 +108,7 @@ Sandbox.create( | No internet | `False` | _(none)_ | All outbound traffic blocked | | Allowlist | `False` | `allow_out` | Only listed CIDRs reachable | | Denylist | `True` | `deny_out` | Listed CIDRs blocked, rest allowed | +| Dynamic update | either | any of the above | Replaces the policy of a running sandbox, resetting connections it no longer permits | ## 4. Prerequisites @@ -127,6 +163,9 @@ python network_allowlist.py # Denylist: block specific CIDRs, allow everything else python network_denylist.py + +# Grant, then revoke, a destination on a running sandbox +python network_dynamic_update.py ``` Expected output for `network_no_internet.py`: @@ -154,6 +193,7 @@ that is passed to Cubelet when the VM is created: | `allow_internet_access=False` | `AllowInternetAccess=false` | Drop all public-IP traffic | | `network.allow_out` | `AllowOut` (CIDR list) | Forward only matching destinations | | `network.deny_out` | `DenyOut` (CIDR list) | Drop matching destinations | +| `update_network(...)` | `UpdateNetworkPolicy` | Converge the live policy maps, then re-evaluate established connections | All enforcement happens in the tap network device of the KVM MicroVM, so policies are applied at the kernel level and cannot be bypassed from inside @@ -167,6 +207,8 @@ the sandbox. | Metadata endpoint still reachable | CIDR not in denylist | Add `169.254.0.0/16` to `deny_out` | | `Template not found` | Wrong template ID | Run `cubemastercli tpl list` | | `Connection refused` | CubeAPI not reachable | Check `E2B_API_URL` and port 3000 | +| `update_network` returns 409 | Sandbox is paused or already gone | Resume it first; a paused sandbox has no live policy to update | +| Update succeeded but a domain stopped resolving | Every domain was dropped from the policy, which also withdraws the implicit DNS-resolver allowance | Keep at least one domain target, or add the resolver IP to `allow_out` | ## 8. Directory Structure @@ -176,6 +218,7 @@ network-policy/ ├── network_no_internet.py # Mode 1: fully air-gapped sandbox ├── network_allowlist.py # Mode 2: outbound CIDR allowlist ├── network_denylist.py # Mode 3: outbound CIDR denylist +├── network_dynamic_update.py # Mode 4: change the policy of a running sandbox ├── env_utils.py # .env loader utility ├── requirements.txt # Python dependencies └── .env.example # Environment variable template diff --git a/examples/network-policy/network_dynamic_update.py b/examples/network-policy/network_dynamic_update.py new file mode 100644 index 000000000..08cae3ce7 --- /dev/null +++ b/examples/network-policy/network_dynamic_update.py @@ -0,0 +1,257 @@ +#!/usr/bin/env python3 +# Copyright (c) 2026 Tencent Inc. +# SPDX-License-Identifier: Apache-2.0 + +""" +network_dynamic_update.py — Change a running sandbox's egress policy in place. + +Use case: + An agent starts locked down and needs to be granted a destination partway + through a task, or has to lose one the moment a step finishes. Recreating + the sandbox would throw away its filesystem and process state, so the + policy has to change under a live VM. + +How it works: + sandbox.update_network() replaces the whole egress policy. It takes the same + `network` / `allow_internet_access` pair as Sandbox.create, so a policy can + be moved between the two without being rewritten, and like creation it is a + replacement rather than a patch — a key you leave out is cleared. + + What creation cannot do is re-evaluate connections that are already open. An + update does: a connection the new policy no longer permits is reset instead + of running until it closes on its own. That is what makes revocation take + effect rather than merely apply to future connections. + +This script walks four scenarios: + 1. IP allow list — grant one, confirm another stays blocked, revoke. + 2. A live connection carried across a revoking update, which gets reset. + 3. Domain allow list — grant one, then switch the policy to another. + 4. L7 rules — start intercepting a host mid-run, with no restart. + +Run: + cp .env.example .env # fill in values + pip install -r requirements.txt + python network_dynamic_update.py +""" + +import os +import sys + +from cubesandbox import Sandbox + +from env_utils import load_local_dotenv + +load_local_dotenv() + +TEMPLATE_ID = os.environ.get("CUBE_TEMPLATE_ID") +if not TEMPLATE_ID: + sys.exit("CUBE_TEMPLATE_ID is required") + +# Two reachable IPs, so "granted" and "still blocked" can be told apart in the +# same sandbox. Override if these are not routable for you. +GRANTED_IP = os.environ.get("GRANTED_IP", "8.8.8.8") +OTHER_IP = os.environ.get("OTHER_IP", "1.1.1.1") +PROBE_PORT = int(os.environ.get("PROBE_PORT", "53")) + +# Two stable HTTPS hosts for the domain scenario. Domain filtering works off the +# TLS SNI on 443, so the probe has to be a real request rather than a bare +# connect to a resolved address. +GRANTED_DOMAIN = os.environ.get("GRANTED_DOMAIN", "example.com") +OTHER_DOMAIN = os.environ.get("OTHER_DOMAIN", "example.org") + +# An echo host for the L7 scenario: it reflects request headers back, which is +# how an injected header becomes visible from inside the sandbox. +L7_HOST = os.environ.get("L7_HOST", "httpbun.com") +L7_HEADER = "X-Cube-Injected" + +HOLDER_PATH = "/tmp/hold_flow.py" + +# Holds one connection open and reports how it ends. It never writes to the +# socket: the peer speaks some protocol and arbitrary bytes would make it hang +# up, which is indistinguishable from the policy reset we want to observe. TCP +# keepalive gives us packets on the flow — the datapath only re-evaluates a flow +# when the guest sends — without touching the byte stream. +HOLDER_SCRIPT = """ +import socket, sys + +host, port, window = sys.argv[1], int(sys.argv[2]), float(sys.argv[3]) +s = socket.socket() +s.settimeout(5) +try: + s.connect((host, port)) +except Exception as exc: + print("CONNECT_FAILED:%s" % exc, flush=True) + sys.exit(0) + +s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) +for opt, value in (("TCP_KEEPIDLE", 1), ("TCP_KEEPINTVL", 1), ("TCP_KEEPCNT", 4)): + if hasattr(socket, opt): + s.setsockopt(socket.IPPROTO_TCP, getattr(socket, opt), value) +print("CONNECTED", flush=True) + +s.settimeout(window) +try: + print("EOF" if s.recv(1) == b"" else "DATA", flush=True) +except socket.timeout: + print("ALIVE", flush=True) +except ConnectionResetError: + print("RESET", flush=True) +except OSError as exc: + print("ERROR:%s" % type(exc).__name__, flush=True) +""" + + +def tcp_reachable(sandbox: Sandbox, ip: str) -> bool: + """Whether a fresh TCP connection to ip:PROBE_PORT succeeds.""" + result = sandbox.commands.run( + f"timeout 5 bash -c ' str: + """HTTP status from an HTTPS request; curl reports 000 when it never connected.""" + result = sandbox.commands.run( + f"curl -sS --max-time 8 -o /dev/null -w '%{{http_code}}\\n' https://{domain}/ || true", + timeout=30, + ) + status = result.stdout.strip().splitlines()[-1] if result.stdout.strip() else "000" + return "BLOCKED" if status == "000" else status + + +def l7_probe(sandbox: Sandbox, path: str) -> str: + """Status plus whether the injected header came back, for one HTTPS request. + + Uses curl -k on purpose. While a host is intercepted, CubeEgress terminates + the TLS session with its own certificate, so verification would fail unless + the guest image trusts the interception CA. Skipping verification keeps the + example about the policy rather than about CA distribution. + """ + result = sandbox.commands.run( + f"curl -sSk --max-time 10 -w '\\nSTATUS:%{{http_code}}\\n' https://{L7_HOST}{path} " + f"|| echo 'STATUS:000'", + timeout=30, + ) + out = result.stdout + status = "000" + for line in out.splitlines(): + if line.startswith("STATUS:"): + status = line.split(":", 1)[1].strip() + injected = L7_HEADER.lower() in out.lower() + if status == "000": + return "no connection" + return f"HTTP {status}" + (f", {L7_HEADER} injected" if injected else "") + + +def hold_across_update(sandbox: Sandbox, host: str, port: int, revoke) -> str: + """Open a connection, run revoke() while it is live, and report its fate. + + Returns RESET when the datapath tore the flow down, ALIVE when the update + left it running, or EOF/DATA/ERROR when the peer acted on its own — in which + case this run simply cannot tell the two apart. + """ + sandbox.files.write(HOLDER_PATH, HOLDER_SCRIPT) + sandbox.commands.run( + f"nohup python3 {HOLDER_PATH} {host} {port} 10 >/tmp/hold.out 2>&1 & echo started", + timeout=20, + ) + ready = sandbox.commands.run( + f"for i in $(seq 10); do grep -q . /tmp/hold.out && break; sleep 1; done; " + f"head -1 /tmp/hold.out", + timeout=30, + ) + if "CONNECTED" not in ready.stdout: + return f"NOT_ESTABLISHED({ready.stdout.strip()})" + + revoke() + + outcome = sandbox.commands.run( + "for i in $(seq 15); do test $(wc -l None: + # Start fully locked down: no allow list, no public egress. + with Sandbox.create( + template=TEMPLATE_ID, + allow_internet_access=False, + timeout=600, + ) as sandbox: + print(f"sandbox {sandbox.sandbox_id} created with no egress") + print(f" {GRANTED_IP} reachable: {tcp_reachable(sandbox, GRANTED_IP)} (expect False)") + + # --- 1. IP allow list ------------------------------------------------- + sandbox.update_network( + network={"allow_out": [GRANTED_IP]}, + allow_internet_access=False, + ) + print(f"\ngranted {GRANTED_IP}") + print(f" {GRANTED_IP} reachable: {tcp_reachable(sandbox, GRANTED_IP)} (expect True)") + print(f" {OTHER_IP} reachable: {tcp_reachable(sandbox, OTHER_IP)} (expect False)") + + # --- 2. A live connection carried across a revoking update ------------ + # Held on 443: the holder stays silent, and a TLS server waiting for a + # ClientHello tolerates that far longer than a DNS server tolerates an + # idle TCP/53 connection. The policy allows the whole IP, so the port + # makes no difference to what is being shown. + def revoke_everything(): + sandbox.update_network(network={}, allow_internet_access=False) + print("\nrevoked every destination while a connection was open") + + outcome = hold_across_update(sandbox, GRANTED_IP, 443, revoke_everything) + print(f" held connection ended as: {outcome} (expect RESET)") + print(f" {GRANTED_IP} reachable: {tcp_reachable(sandbox, GRANTED_IP)} (expect False)") + + # --- 3. Domain allow list --------------------------------------------- + # Domains need the default-deny that allow_internet_access=False gives, + # otherwise everything is reachable anyway and the allow list is moot. + # The resolver's own address is allowed automatically, so DNS keeps + # working without being named here. + sandbox.update_network( + network={"allow_out": [GRANTED_DOMAIN]}, + allow_internet_access=False, + ) + print(f"\ngranted domain {GRANTED_DOMAIN}") + print(f" https://{GRANTED_DOMAIN} -> {https_status(sandbox, GRANTED_DOMAIN)} (expect 200)") + print(f" https://{OTHER_DOMAIN} -> {https_status(sandbox, OTHER_DOMAIN)} (expect BLOCKED)") + + sandbox.update_network( + network={"allow_out": [OTHER_DOMAIN]}, + allow_internet_access=False, + ) + print(f"\nswitched the allow list to {OTHER_DOMAIN}") + print(f" https://{OTHER_DOMAIN} -> {https_status(sandbox, OTHER_DOMAIN)} (expect 200)") + + # --- 4. L7 rules ------------------------------------------------------ + # Start intercepting a host mid-run. A rule's host is allowed implicitly, + # and once any rule exists for it the host is L7 default-deny: only what + # a rule matches gets through, everything else is refused by the proxy. + sandbox.update_network( + network={ + "allow_out": [L7_HOST], + "rules": [ + { + "name": "inject_on_headers", + "match": {"scheme": "https", "sni": L7_HOST, "host": L7_HOST, + "path": "/headers"}, + "action": {"allow": True, + "inject": [{"header": L7_HEADER, "secret": "demo-token"}]}, + }, + ], + }, + allow_internet_access=False, + ) + print(f"\nstarted intercepting {L7_HOST} with an L7 rule on /headers") + print(f" GET /headers -> {l7_probe(sandbox, '/headers')} (expect HTTP 200, header injected)") + print(f" GET /get -> {l7_probe(sandbox, '/get')} (expect HTTP 403, no rule matches)") + + print("\ndynamic network update ok") + + +if __name__ == "__main__": + main() diff --git a/examples/network-policy/requirements.txt b/examples/network-policy/requirements.txt index e2f853e67..9e505f265 100644 --- a/examples/network-policy/requirements.txt +++ b/examples/network-policy/requirements.txt @@ -1,2 +1,5 @@ e2b-code-interpreter>=2.4.1 python-dotenv +# network_dynamic_update.py only: updating a running sandbox's policy is a +# CubeSandbox extension, so it uses the CubeSandbox SDK rather than the E2B one. +cubesandbox diff --git a/openapi.yml b/openapi.yml index 0d86c936f..7aa1bf6f4 100644 --- a/openapi.yml +++ b/openapi.yml @@ -239,6 +239,51 @@ paths: application/json: schema: $ref: '#/components/schemas/ApiError' + /sandboxes/{sandboxID}/network: + put: + tags: + - handlers::sandboxes + operationId: update_sandbox_network + parameters: + - name: sandboxID + in: path + description: Sandbox identifier + required: true + schema: + type: string + requestBody: + content: + application/json: + schema: + $ref: '#/components/schemas/UpdateSandboxNetworkRequest' + required: true + responses: + '204': + description: Network policy updated + '400': + description: Invalid network policy + content: + application/json: + schema: + $ref: '#/components/schemas/ApiError' + '404': + description: Sandbox not found + content: + application/json: + schema: + $ref: '#/components/schemas/ApiError' + '409': + description: Sandbox is not running + content: + application/json: + schema: + $ref: '#/components/schemas/ApiError' + '500': + description: Unexpected backend error + content: + application/json: + schema: + $ref: '#/components/schemas/ApiError' /sandboxes/{sandboxID}/pause: post: tags: @@ -800,7 +845,7 @@ paths: required: true responses: '200': - description: Alias set or cleared; returns the updated template detail + description: Alias updated content: application/json: schema: @@ -1418,6 +1463,11 @@ components: Multi-field semantics: AND across fields, OR within `method`. Comparisons on sni/host/scheme are case-insensitive. + + `port` + `scheme` together pin the (host, port) tuple CubeEgress intercepts. + Both nil keeps the legacy default {80/http, 443/https}. When `port` is set, + `scheme` MUST also be set — same-`(host, port)` rules across the policy + must agree on `scheme` (the server rejects the whole policy on mismatch). properties: host: type: @@ -1433,6 +1483,11 @@ components: type: - string - 'null' + port: + type: + - integer + - 'null' + format: int32 scheme: type: - string @@ -1636,18 +1691,6 @@ components: - integer - 'null' format: int32 - SetTemplateAliasRequest: - type: object - description: Body for PUT /templates/{id}/alias (set, reassign, or clear alias). - properties: - alias: - type: - - string - - 'null' - description: |- - New alias for the template. Validated against `^[a-z0-9][a-z0-9-]{0,63}$` - with `tpl-`/`snap-` prefixes rejected. Omitted, null, or empty string - clears the alias. ResumedSandbox: type: object description: Request body for POST /sandboxes/{id}/resume (deprecated). @@ -1942,6 +1985,19 @@ components: description: |- CubeSandbox extension: mount this volume read-only for this sandbox attachment. Defaults to false when omitted. + SetTemplateAliasRequest: + type: object + description: |- + Body for PUT /templates/:id/alias (set / modify / clear alias). + + `alias` is `None` / null / empty string ⇒ clear the current alias. + A non-empty value is validated by CubeMaster's `validateTemplateAlias` + (the single source of truth; CubeAPI does not re-validate). + properties: + alias: + type: + - string + - 'null' SetTimeoutRequest: type: object description: Request body for POST /sandboxes/{id}/timeout @@ -2254,6 +2310,23 @@ components: type: - string - 'null' + UpdateSandboxNetworkRequest: + type: object + description: |- + Request body for PUT /sandboxes/{id}/network. + + The body is the complete desired egress policy, not a patch: any field left + out clears what the sandbox currently has. This mirrors how `network` is + interpreted at create time, so the same object can be sent to either. + properties: + allowInternetAccess: + type: + - boolean + - 'null' + network: + oneOf: + - type: 'null' + - $ref: '#/components/schemas/SandboxNetworkConfig' Volume: type: object description: Volume descriptor returned in list responses (no token). diff --git a/sdk/go/client.go b/sdk/go/client.go index 9ed99bc9d..7e94e2bde 100644 --- a/sdk/go/client.go +++ b/sdk/go/client.go @@ -140,36 +140,10 @@ func (c *Client) createPayload(opts CreateOptions) (map[string]any, error) { payload["allowInternetAccess"] = false } - // Mirror the server-side contract: domain allowOut requires either - // allowInternetAccess=false or an explicit deny-all CIDR in denyOut. - if err := validateAllowOutDomainsRequireDenyAll(opts.Network.AllowOut, opts.Network.DenyOut, internetAccessDisabled); err != nil { + network, err := buildNetworkPayload(opts.Network, internetAccessDisabled) + if err != nil { return nil, err } - - network := map[string]any{} - if opts.Network.AllowPublicTraffic != nil { - network["allowPublicTraffic"] = *opts.Network.AllowPublicTraffic - } - if opts.Network.MaskRequestHost != nil { - network["maskRequestHost"] = *opts.Network.MaskRequestHost - } - if len(opts.Network.AllowOut) > 0 { - network["allowOut"] = opts.Network.AllowOut - } - if len(opts.Network.DenyOut) > 0 { - network["denyOut"] = opts.Network.DenyOut - } - if len(opts.Network.Rules) > 0 { - rules := make([]Rule, len(opts.Network.Rules)) - for i, rule := range opts.Network.Rules { - if err := rule.Match.validate(); err != nil { - return nil, fmt.Errorf("network.rules[%d] %q: %w", i, rule.Name, err) - } - rule.Match = rule.Match.normalized() - rules[i] = rule - } - network["rules"] = rules - } if len(network) > 0 { payload["network"] = network } diff --git a/sdk/go/models.go b/sdk/go/models.go index 8a0ec81ac..cd9406a44 100644 --- a/sdk/go/models.go +++ b/sdk/go/models.go @@ -73,6 +73,19 @@ type CreateOptions struct { Extra map[string]any } +// UpdateNetworkOptions is the desired egress policy for Sandbox.UpdateNetwork. +// It mirrors the network half of CreateOptions so a policy can be moved between +// the two without being rewritten. +// +// The whole struct is the desired state, not a patch: a zero-valued field is +// cleared rather than left as it was. +type UpdateNetworkOptions struct { + Network NetworkOptions + // AllowInternetAccess gates traffic outside the allow list; nil keeps the + // server default (allow). + AllowInternetAccess *bool +} + // DurationPtr returns a pointer to d. It is a convenience for optional // duration fields such as CreateOptions.Timeout and Sandbox.Resume, where nil // means "not provided; let the server decide". diff --git a/sdk/go/policy.go b/sdk/go/policy.go index 721770b92..dc227e8ba 100644 --- a/sdk/go/policy.go +++ b/sdk/go/policy.go @@ -108,6 +108,43 @@ const ( // allowing specific domains is meaningful only when all other egress is denied, // either by allowInternetAccess=false (defaultDenyAll) or by listing // 0.0.0.0/0 in denyOut. Returns nil when allowOut carries no domain target. +// buildNetworkPayload translates NetworkOptions into the API's camelCase +// network object, validating it the same way the server will. Shared by sandbox +// creation and Sandbox.UpdateNetwork so both accept identical input. +func buildNetworkPayload(opts NetworkOptions, internetAccessDisabled bool) (map[string]any, error) { + // Mirror the server-side contract: domain allowOut requires either + // allowInternetAccess=false or an explicit deny-all CIDR in denyOut. + if err := validateAllowOutDomainsRequireDenyAll(opts.AllowOut, opts.DenyOut, internetAccessDisabled); err != nil { + return nil, err + } + + network := map[string]any{} + if opts.AllowPublicTraffic != nil { + network["allowPublicTraffic"] = *opts.AllowPublicTraffic + } + if opts.MaskRequestHost != nil { + network["maskRequestHost"] = *opts.MaskRequestHost + } + if len(opts.AllowOut) > 0 { + network["allowOut"] = opts.AllowOut + } + if len(opts.DenyOut) > 0 { + network["denyOut"] = opts.DenyOut + } + if len(opts.Rules) > 0 { + rules := make([]Rule, len(opts.Rules)) + for i, rule := range opts.Rules { + if err := rule.Match.validate(); err != nil { + return nil, fmt.Errorf("network.rules[%d] %q: %w", i, rule.Name, err) + } + rule.Match = rule.Match.normalized() + rules[i] = rule + } + network["rules"] = rules + } + return network, nil +} + func validateAllowOutDomainsRequireDenyAll(allowOut, denyOut []string, defaultDenyAll bool) error { hasDomain := false for _, target := range allowOut { diff --git a/sdk/go/sandbox.go b/sdk/go/sandbox.go index 472e7e6ee..88b71fa31 100644 --- a/sdk/go/sandbox.go +++ b/sdk/go/sandbox.go @@ -142,6 +142,38 @@ func (s *Sandbox) SetTimeout(ctx context.Context, timeout time.Duration) error { return s.client.doJSON(ctx, http.MethodPost, path, payload, nil, http.StatusNoContent) } +// UpdateNetwork replaces the sandbox's egress policy. +// +// opts is the complete desired policy, not a patch: a field left at its zero +// value clears whatever the sandbox currently has. It carries the same fields +// as the network half of CreateOptions. +// +// The new policy applies to established connections as well as new ones — a +// connection it no longer permits is reset rather than left running. +// +// Errors wrap ErrSandboxNotFound (404) or an *APIError for other HTTP errors, +// including 409 when the sandbox is not running. +func (s *Sandbox) UpdateNetwork(ctx context.Context, opts UpdateNetworkOptions) error { + if err := s.ensureClient(); err != nil { + return err + } + internetAccessDisabled := opts.AllowInternetAccess != nil && !*opts.AllowInternetAccess + netPayload, err := buildNetworkPayload(opts.Network, internetAccessDisabled) + if err != nil { + return err + } + + payload := map[string]any{} + if internetAccessDisabled { + payload["allowInternetAccess"] = false + } + if len(netPayload) > 0 { + payload["network"] = netPayload + } + path := "/sandboxes/" + url.PathEscape(s.SandboxID) + "/network" + return s.client.doJSON(ctx, http.MethodPut, path, payload, nil, http.StatusNoContent) +} + func (s *Sandbox) Kill(ctx context.Context) error { if err := s.ensureClient(); err != nil { return err diff --git a/sdk/node/src/index.ts b/sdk/node/src/index.ts index d3e4f9ffb..0a8819731 100644 --- a/sdk/node/src/index.ts +++ b/sdk/node/src/index.ts @@ -11,6 +11,7 @@ export { type PauseOptions, type ListSnapshotsOptions, type NetworkOptions, + type UpdateNetworkOptions, type LifecycleOptions, } from "./sandbox.js"; diff --git a/sdk/node/src/sandbox.ts b/sdk/node/src/sandbox.ts index 9facc5c5b..60259ce2d 100644 --- a/sdk/node/src/sandbox.ts +++ b/sdk/node/src/sandbox.ts @@ -44,6 +44,19 @@ export interface LifecycleOptions { } /** Options for {@link Sandbox.create}. */ +/** + * Desired egress policy for {@link Sandbox.updateNetwork}. Mirrors the network + * half of {@link CreateOptions} so a policy can be moved between the two + * without being rewritten. + * + * The whole object is the desired state, not a patch: an omitted field is + * cleared rather than left as it was. + */ +export interface UpdateNetworkOptions { + network?: NetworkOptions; + allowInternetAccess?: boolean; +} + export interface CreateOptions { template?: string; /** Alias for {@link CreateOptions.template}, matching the Issue #760 / E2B shape. */ @@ -190,6 +203,34 @@ async function checkControlResponse(resp: { throw new ApiError(msg, code); } +/** + * Translate NetworkOptions into the API's camelCase network object, running the + * same client-side validation the server applies. Shared by sandbox creation + * and `Sandbox.updateNetwork` so both accept identical input. + */ +function buildNetworkPayload( + net: NetworkOptions, + internetAccessDisabled: boolean, +): Record { + validateAllowOutDomainsRequireDenyAll(net.allowOut, net.denyOut, internetAccessDisabled); + const wire: Record = {}; + if (net.allowOut !== undefined) wire.allowOut = net.allowOut; + if (net.denyOut !== undefined) wire.denyOut = net.denyOut; + if (net.allowPublicTraffic !== undefined) { + wire.allowPublicTraffic = net.allowPublicTraffic; + } + if (net.maskRequestHost !== undefined) { + wire.maskRequestHost = net.maskRequestHost; + } + if (net.rules) { + const normalized = normalizeRulesArg(net.rules); + if (normalized.length > 0) { + wire.rules = normalized.map(serializeRule); + } + } + return wire; +} + function sleep(ms: number): Promise { return new Promise((resolve) => setTimeout(resolve, ms)); } @@ -377,27 +418,7 @@ export class Sandbox { payload.allow_internet_access = false; } if (options.network) { - const net = options.network; - validateAllowOutDomainsRequireDenyAll( - net.allowOut, - net.denyOut, - options.allowInternetAccess === false, - ); - const wire: Record = {}; - if (net.allowOut !== undefined) wire.allowOut = net.allowOut; - if (net.denyOut !== undefined) wire.denyOut = net.denyOut; - if (net.allowPublicTraffic !== undefined) { - wire.allowPublicTraffic = net.allowPublicTraffic; - } - if (net.maskRequestHost !== undefined) { - wire.maskRequestHost = net.maskRequestHost; - } - if (net.rules) { - const normalized = normalizeRulesArg(net.rules); - if (normalized.length > 0) { - wire.rules = normalized.map(serializeRule); - } - } + const wire = buildNetworkPayload(options.network, options.allowInternetAccess === false); if (Object.keys(wire).length > 0) { payload.network = wire; } @@ -619,6 +640,34 @@ export class Sandbox { await checkControlResponse(resp); } + /** + * PUT /sandboxes/:id/network — replace the sandbox's egress policy. + * + * `options` is the complete desired policy, not a patch: an omitted field + * clears whatever the sandbox currently has. The new policy also applies to + * established connections, which are reset if it no longer permits them. + */ + async updateNetwork(options: UpdateNetworkOptions = {}): Promise { + const { network, allowInternetAccess } = options; + const payload: Record = {}; + if (allowInternetAccess === false) { + payload.allowInternetAccess = false; + } + if (network) { + payload.network = buildNetworkPayload(network, allowInternetAccess === false); + } + const resp = await controlFetch( + this.config, + `${this.config.apiUrl}/sandboxes/${this.sandboxId}/network`, + { + method: "PUT", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify(payload), + }, + ); + await checkControlResponse(resp); + } + /** DELETE /sandboxes/:id — destroy a sandbox. */ async kill(): Promise { const resp = await controlFetch(this.config, `${this.config.apiUrl}/sandboxes/${this.sandboxId}`, { diff --git a/sdk/python/cubesandbox/_policy.py b/sdk/python/cubesandbox/_policy.py index cb15d7e2d..90144215f 100644 --- a/sdk/python/cubesandbox/_policy.py +++ b/sdk/python/cubesandbox/_policy.py @@ -426,6 +426,41 @@ def _validate_allow_out_domains_require_deny_all( raise ApiError(ALLOW_OUT_DOMAIN_REQUIRES_DENY_ALL, 400) +def _build_network_payload( + network: Dict[str, Any], + *, + allow_internet_access: bool = True, +) -> Dict[str, Any]: + """Translate the SDK's snake_case ``network`` argument into the API body. + + Shared by sandbox creation and :meth:`Sandbox.update_network` so both accept + exactly the same argument shape and run the same client-side validation. + """ + _validate_allow_out_domains_require_deny_all( + network.get("allow_out"), + network.get("deny_out"), + default_deny_all=not allow_internet_access, + ) + net: Dict[str, Any] = {} + if "allow_out" in network: + net["allowOut"] = network["allow_out"] + if "deny_out" in network: + net["denyOut"] = network["deny_out"] + if "allow_public_traffic" in network: + net["allowPublicTraffic"] = network["allow_public_traffic"] + if "mask_request_host" in network: + net["maskRequestHost"] = network["mask_request_host"] + if network.get("rules"): + # ``rules`` accepts either CubeEgress's list-of-Rule shape or E2B's + # per-host transform mapping (``{host: [{transform: {...}}]}``). + # ``_normalize_rules_arg`` collapses both into a list of rule dicts + # that ``_serialize_rule`` understands. + normalized_rules = _normalize_rules_arg(network["rules"]) + if normalized_rules: + net["rules"] = [_serialize_rule(r) for r in normalized_rules] + return net + + def _is_domain_allow_out_target(target: object) -> bool: import ipaddress diff --git a/sdk/python/cubesandbox/sandbox.py b/sdk/python/cubesandbox/sandbox.py index 7bb6fc4f5..3500a4c35 100644 --- a/sdk/python/cubesandbox/sandbox.py +++ b/sdk/python/cubesandbox/sandbox.py @@ -16,9 +16,7 @@ from ._models import Execution, ExecutionError, OutputMessage, Result, SandboxInfo, SnapshotInfo from ._policy import ( Rule, - _normalize_rules_arg, - _serialize_rule, - _validate_allow_out_domains_require_deny_all, + _build_network_payload, ) from ._pty import Pty from ._stream import _parse_line @@ -287,28 +285,7 @@ def create( if not allow_internet_access: payload["allow_internet_access"] = False if network: - _validate_allow_out_domains_require_deny_all( - network.get("allow_out"), - network.get("deny_out"), - default_deny_all=not allow_internet_access, - ) - net: dict = {} - if "allow_out" in network: - net["allowOut"] = network["allow_out"] - if "deny_out" in network: - net["denyOut"] = network["deny_out"] - if "allow_public_traffic" in network: - net["allowPublicTraffic"] = network["allow_public_traffic"] - if "mask_request_host" in network: - net["maskRequestHost"] = network["mask_request_host"] - if "rules" in network and network["rules"]: - # ``rules`` accepts either CubeEgress's list-of-Rule shape or - # E2B's per-host transform mapping (``{host: [{transform: {...}}]}``). - # ``_normalize_rules_arg`` collapses both into a list of rule - # dicts that ``_serialize_rule`` understands. - normalized_rules = _normalize_rules_arg(network["rules"]) - if normalized_rules: - net["rules"] = [_serialize_rule(r) for r in normalized_rules] + net = _build_network_payload(network, allow_internet_access=allow_internet_access) if net: payload["network"] = net # Lifecycle: opt-in. Wire shape mirrors e2b @@ -548,6 +525,47 @@ def set_timeout(self, timeout: int) -> None: ) _check_response(resp) + def update_network( + self, + *, + network: Dict[str, Any] | None = None, + allow_internet_access: bool = True, + ) -> None: + """PUT /sandboxes/:sandboxID/network - Replace the egress policy. + + Takes the same ``network`` / ``allow_internet_access`` pair as + :meth:`create`, and as keyword arguments for the same reason, so a policy + can be moved between the two without being rewritten. + + Takes effect on established connections too, not just new ones: a + connection the new policy no longer allows is reset rather than left + running until it closes. + + Args: + network: The complete desired policy, using the same keys as + :meth:`create`. This is a replacement, not a patch — omitting a + key clears it, and passing ``None`` clears the whole policy. + allow_internet_access: Whether traffic outside the allow list may + still reach the internet. Same meaning as in :meth:`create`. + + Raises: + SandboxNotFoundError: If the sandbox does not exist (HTTP 404). + ApiError: If the policy is invalid (HTTP 400), the sandbox is not + running (HTTP 409), or on unexpected backend error. + """ + body: Dict[str, Any] = {} + if not allow_internet_access: + body["allowInternetAccess"] = False + if network: + body["network"] = _build_network_payload( + network, allow_internet_access=allow_internet_access + ) + resp = self._session.put( + f"{self._config.api_url}/sandboxes/{self.sandbox_id}/network", + json=body, + ) + _check_response(resp) + def kill(self) -> None: """DELETE /sandboxes/:sandboxID - Destroy a sandbox. diff --git a/tests/e2e/sdk_compat/README.md b/tests/e2e/sdk_compat/README.md index 996a7b077..155778b80 100644 --- a/tests/e2e/sdk_compat/README.md +++ b/tests/e2e/sdk_compat/README.md @@ -384,7 +384,9 @@ Current capability domains: - `cases/commands/`: stdout, stderr, exit code, env, special characters, multiline output, missing command. - `cases/filesystem/`: read/write, overwrite, multiline content, file API and shell interoperability. - `cases/run_code/`: expression text, stdout, kernel state, Python error reporting. -- `cases/network/`: create-time network policy for allow/deny and public egress access. +- `cases/network/`: create-time network policy for allow/deny and public egress access, + plus in-place policy updates on a running sandbox including re-evaluation of + already-established connections (`test_policy_update.py`, CubeSandbox only). - `cases/concurrency/`: simultaneous multi-sandbox isolation. - `cases/host-mount/`: host-directory mount extension — happy path plus create-time validation, runtime bind-mount failures, and cross-sandbox sharing boundary cases. diff --git a/tests/e2e/sdk_compat/README_zh.md b/tests/e2e/sdk_compat/README_zh.md index 24e27265c..e879befe0 100644 --- a/tests/e2e/sdk_compat/README_zh.md +++ b/tests/e2e/sdk_compat/README_zh.md @@ -371,7 +371,8 @@ tests/e2e/sdk_compat/ 输出和缺失命令； - `cases/filesystem/`：读写、覆盖、多行内容、文件 API 与 shell 互操作； - `cases/run_code/`：表达式结果、stdout、kernel 状态和 Python 错误； -- `cases/network/`：创建时的 allow/deny 和公网出站策略； +- `cases/network/`：创建时的 allow/deny 和公网出站策略，以及运行中沙箱的策略原地 + 更新（含存量连接重判，见 `test_policy_update.py`，仅 CubeSandbox）； - `cases/concurrency/`：同时运行多个 sandbox 时的数据隔离； - `cases/host-mount/`：宿主目录挂载扩展——happy path，以及创建时校验、 运行期 bind-mount 失败和跨 sandbox 共享等边界用例。 diff --git a/tests/e2e/sdk_compat/adapters/base.py b/tests/e2e/sdk_compat/adapters/base.py index a741fe5d8..f29d8355c 100644 --- a/tests/e2e/sdk_compat/adapters/base.py +++ b/tests/e2e/sdk_compat/adapters/base.py @@ -112,6 +112,15 @@ def get_host(self, port: int) -> str: def traffic_access_token(self) -> str | None: raise UnsupportedCapability(self.backend, "network_public_access") + def update_network( + self, + *, + network: dict | None = None, + allow_internet_access: bool = True, + ) -> None: + """Replace the egress policy of the running sandbox.""" + raise UnsupportedCapability(self.backend, "network_dynamic_update") + @abstractmethod def kill(self) -> None: raise NotImplementedError diff --git a/tests/e2e/sdk_compat/adapters/cubesandbox_adapter.py b/tests/e2e/sdk_compat/adapters/cubesandbox_adapter.py index f76b5e3af..e993549ee 100644 --- a/tests/e2e/sdk_compat/adapters/cubesandbox_adapter.py +++ b/tests/e2e/sdk_compat/adapters/cubesandbox_adapter.py @@ -250,6 +250,17 @@ def traffic_access_token(self) -> str | None: return str(raw[key]) return None + def update_network( + self, + *, + network: dict | None = None, + allow_internet_access: bool = True, + ) -> None: + self._sandbox.update_network( + network=network, + allow_internet_access=allow_internet_access, + ) + def kill(self) -> None: self._sandbox.kill() diff --git a/tests/e2e/sdk_compat/adapters/tracing_adapter.py b/tests/e2e/sdk_compat/adapters/tracing_adapter.py index 2eece0c3f..7312914a7 100644 --- a/tests/e2e/sdk_compat/adapters/tracing_adapter.py +++ b/tests/e2e/sdk_compat/adapters/tracing_adapter.py @@ -290,6 +290,30 @@ def traffic_access_token(self) -> str | None: output=lambda token: {"token_present": bool(token)}, ) + def update_network( + self, + *, + network: dict | None = None, + allow_internet_access: bool = True, + ) -> None: + return self._trace.capture( + "update_network", + { + "backend": self.backend, + "sandbox_id": self.sandbox_id, + # The policy itself, not just its keys: when a case asserts that + # a connection was or was not torn down, the trace has to show + # which policy caused it. + "network": network, + "allow_internet_access": allow_internet_access, + }, + lambda: self._wrapped.update_network( + network=network, + allow_internet_access=allow_internet_access, + ), + output=lambda _: {"updated": True}, + ) + def kill(self) -> None: return self._trace.capture( "kill", diff --git a/tests/e2e/sdk_compat/cases/network/test_policy_update.py b/tests/e2e/sdk_compat/cases/network/test_policy_update.py new file mode 100644 index 000000000..6326f2610 --- /dev/null +++ b/tests/e2e/sdk_compat/cases/network/test_policy_update.py @@ -0,0 +1,330 @@ +# Copyright (c) 2026 Tencent Inc. +# SPDX-License-Identifier: Apache-2.0 + +"""In-place egress policy updates on a running sandbox. + +What separates these cases from cases/network/test_policy.py: there the policy is +fixed at create time, so only the initial verdict is under test. Here the policy +changes under a live VM, which adds two things worth proving — new connections +follow the new policy, and connections that already exist are re-evaluated rather +than grandfathered. +""" + +from __future__ import annotations + +import pytest + +from framework.assertions import assert_command_ok +from framework.capabilities import ( + NETWORK_ALLOW_DENY, + NETWORK_DYNAMIC_UPDATE, + PAUSE_RESUME, + ROLLBACK_CLONE, +) +from framework.lifecycle import ( + wait_until_data_plane_ready, + wait_until_paused, + wait_until_running, +) +from framework.network_probe import ( + ALTERNATE_TCP_TARGET_IP, + TCP_TARGET_IP, + assert_tcp_blocked, + assert_tcp_reachable, + require_conclusive_flow_outcome, + start_established_flow, + tcp_probe_command, + wait_established_flow_outcome, +) + +pytestmark = [ + pytest.mark.e2e, + pytest.mark.sdk_compat, + pytest.mark.network, + pytest.mark.p1, + pytest.mark.requires_internet, + pytest.mark.requires_capability(NETWORK_DYNAMIC_UPDATE), +] + + +def probe(sdk_sandbox, sdk_e2e_config, target: str): + return sdk_sandbox.run_command( + tcp_probe_command(target, timeout=sdk_e2e_config.network_probe_timeout), + timeout=sdk_e2e_config.command_timeout, + ) + + +@pytest.mark.requires_capability(NETWORK_ALLOW_DENY) +@pytest.mark.sandbox_create_options(allow_internet_access=False) +def test_update_grants_new_destination(sdk_sandbox, sdk_e2e_config): + """A destination blocked at create time becomes reachable after an update.""" + assert_tcp_blocked(probe(sdk_sandbox, sdk_e2e_config, TCP_TARGET_IP), TCP_TARGET_IP) + + sdk_sandbox.update_network( + network={"allow_out": [TCP_TARGET_IP]}, + allow_internet_access=False, + ) + + assert_tcp_reachable(probe(sdk_sandbox, sdk_e2e_config, TCP_TARGET_IP), TCP_TARGET_IP) + + +@pytest.mark.requires_capability(NETWORK_ALLOW_DENY) +@pytest.mark.sandbox_create_options( + allow_internet_access=False, + network={"allow_out": [TCP_TARGET_IP]}, +) +def test_update_revokes_destination_for_new_connections(sdk_sandbox, sdk_e2e_config): + """Dropping a target from the allow list blocks subsequent connections.""" + assert_tcp_reachable(probe(sdk_sandbox, sdk_e2e_config, TCP_TARGET_IP), TCP_TARGET_IP) + + # Empty policy: the update replaces rather than patches, so this revokes all. + sdk_sandbox.update_network(network={}, allow_internet_access=False) + + assert_tcp_blocked(probe(sdk_sandbox, sdk_e2e_config, TCP_TARGET_IP), TCP_TARGET_IP) + + +@pytest.mark.requires_capability(NETWORK_ALLOW_DENY) +@pytest.mark.sandbox_create_options( + allow_internet_access=False, + network={"allow_out": [TCP_TARGET_IP]}, +) +def test_update_swaps_allowed_destination(sdk_sandbox, sdk_e2e_config): + """Replacing the allow list moves access rather than accumulating it.""" + sdk_sandbox.update_network( + network={"allow_out": [ALTERNATE_TCP_TARGET_IP]}, + allow_internet_access=False, + ) + + assert_tcp_reachable( + probe(sdk_sandbox, sdk_e2e_config, ALTERNATE_TCP_TARGET_IP), + ALTERNATE_TCP_TARGET_IP, + ) + assert_tcp_blocked(probe(sdk_sandbox, sdk_e2e_config, TCP_TARGET_IP), TCP_TARGET_IP) + + +@pytest.mark.requires_capability(NETWORK_ALLOW_DENY) +@pytest.mark.sandbox_create_options( + allow_internet_access=False, + network={"allow_out": [TCP_TARGET_IP]}, +) +def test_update_tears_down_established_connection(sdk_sandbox, sdk_e2e_config): + """The differentiating case: a revoked connection dies instead of surviving. + + Without datapath re-evaluation an established flow keeps its create-time + verdict, so revocation would only apply to future connections and an already + open channel would stay usable indefinitely. + """ + if not start_established_flow( + sdk_sandbox, + TCP_TARGET_IP, + command_timeout=sdk_e2e_config.command_timeout, + ): + pytest.skip(f"could not establish a connection to {TCP_TARGET_IP} to hold open") + + sdk_sandbox.update_network(network={}, allow_internet_access=False) + + outcome = require_conclusive_flow_outcome( + wait_established_flow_outcome( + sdk_sandbox, + command_timeout=sdk_e2e_config.command_timeout, + ), + TCP_TARGET_IP, + ) + assert outcome == "RESET", ( + f"established connection to {TCP_TARGET_IP} survived after its policy was " + f"revoked; holder reported {outcome!r}" + ) + + +@pytest.mark.requires_capability(NETWORK_ALLOW_DENY) +@pytest.mark.sandbox_create_options( + allow_internet_access=False, + network={"allow_out": [TCP_TARGET_IP]}, +) +def test_update_preserves_still_allowed_established_connection( + sdk_sandbox, + sdk_e2e_config, +): + """The other half: an update must not disturb flows it still permits. + + Re-evaluating every flow on every update is only safe if unchanged verdicts + are left alone. Without this case a datapath that simply killed all sessions + on any update would pass the revocation test above. + """ + if not start_established_flow( + sdk_sandbox, + TCP_TARGET_IP, + command_timeout=sdk_e2e_config.command_timeout, + ): + pytest.skip(f"could not establish a connection to {TCP_TARGET_IP} to hold open") + + # Still allows TCP_TARGET_IP; only widens the policy with a second target. + sdk_sandbox.update_network( + network={"allow_out": [TCP_TARGET_IP, ALTERNATE_TCP_TARGET_IP]}, + allow_internet_access=False, + ) + + outcome = require_conclusive_flow_outcome( + wait_established_flow_outcome( + sdk_sandbox, + command_timeout=sdk_e2e_config.command_timeout, + ), + TCP_TARGET_IP, + ) + assert outcome == "ALIVE", ( + "an update that still permits this destination tore down its established " + f"connection; holder reported {outcome!r}" + ) + + +@pytest.mark.requires_capability(NETWORK_ALLOW_DENY) +@pytest.mark.sandbox_create_options( + allow_internet_access=False, + network={"allow_out": ["dns.google"]}, +) +def test_update_keeps_dns_working_for_domain_policy(sdk_sandbox, sdk_e2e_config): + """Updating a domain policy must not withdraw resolver access along with it. + + The resolver CIDRs are injected by the control plane, not authored by the + caller, so an update carrying only user targets could silently drop them and + black-hole every domain rule it just installed. + """ + sdk_sandbox.update_network( + network={"allow_out": ["dns.google", "one.one.one.one"]}, + allow_internet_access=False, + ) + + result = sdk_sandbox.run_command( + "getent hosts dns.google >/dev/null && echo RESOLVED || echo UNRESOLVED", + timeout=sdk_e2e_config.command_timeout, + ) + assert_command_ok(result) + assert "RESOLVED" in result.stdout, ( + "DNS resolution broke after updating a domain-based policy; the resolver " + f"allowance was likely dropped. stdout={result.stdout!r}" + ) + + +@pytest.mark.requires_capability(NETWORK_ALLOW_DENY) +@pytest.mark.requires_capability(ROLLBACK_CLONE) +@pytest.mark.sandbox_create_options( + allow_internet_access=False, + network={"allow_out": [TCP_TARGET_IP]}, +) +def test_clone_after_update_inherits_the_new_policy(sdk_sandbox, sdk_e2e_config): + """A clone must carry the policy the sandbox has now, not the one it was born with. + + Clone is a snapshot plus a create from that snapshot, and the create replays + the spec Master has stored. So this also covers read-your-writes: the update + returns only after the spec is written, and the snapshot here is taken + immediately afterwards. + """ + sdk_sandbox.update_network( + network={"allow_out": [ALTERNATE_TCP_TARGET_IP]}, + allow_internet_access=False, + ) + + clones = sdk_sandbox.clone(1) + try: + clone = clones[0] + assert_tcp_reachable( + probe(clone, sdk_e2e_config, ALTERNATE_TCP_TARGET_IP), + ALTERNATE_TCP_TARGET_IP, + ) + assert_tcp_blocked(probe(clone, sdk_e2e_config, TCP_TARGET_IP), TCP_TARGET_IP) + finally: + for clone in clones: + clone.kill() + + +@pytest.mark.requires_capability(NETWORK_ALLOW_DENY) +@pytest.mark.requires_capability(ROLLBACK_CLONE) +@pytest.mark.sandbox_create_options( + allow_internet_access=False, + network={"allow_out": [TCP_TARGET_IP, ALTERNATE_TCP_TARGET_IP]}, +) +def test_clone_after_tightening_update_is_not_more_permissive(sdk_sandbox, sdk_e2e_config): + """The dangerous direction: a stale spec would hand the clone wider access. + + If Master's spec lagged behind the node, a clone of a sandbox whose policy was + just narrowed would come up with the older, broader allow list — a silent + privilege escalation for every descendant of that sandbox. + """ + sdk_sandbox.update_network( + network={"allow_out": [TCP_TARGET_IP]}, + allow_internet_access=False, + ) + + clones = sdk_sandbox.clone(1) + try: + clone = clones[0] + assert_tcp_blocked( + probe(clone, sdk_e2e_config, ALTERNATE_TCP_TARGET_IP), + ALTERNATE_TCP_TARGET_IP, + ) + assert_tcp_reachable(probe(clone, sdk_e2e_config, TCP_TARGET_IP), TCP_TARGET_IP) + finally: + for clone in clones: + clone.kill() + + +@pytest.mark.requires_capability(PAUSE_RESUME) +@pytest.mark.requires_capability(NETWORK_ALLOW_DENY) +@pytest.mark.sandbox_create_options( + allow_internet_access=False, + network={"allow_out": [TCP_TARGET_IP]}, +) +def test_update_survives_pause_resume(sdk_sandbox, sdk_e2e_config): + """Resume must restore the updated policy, not the create-time one. + + Pause packages the sandbox from Cubelet's own store rather than from the + network runtime's state file, so an update has to reach that store too or the + policy silently reverts on resume. + """ + sdk_sandbox.update_network( + network={"allow_out": [ALTERNATE_TCP_TARGET_IP]}, + allow_internet_access=False, + ) + + sdk_sandbox.pause(timeout=sdk_e2e_config.default_timeout) + wait_until_paused(sdk_sandbox, timeout=sdk_e2e_config.default_timeout) + resumed = sdk_sandbox.resume_or_connect(timeout=sdk_e2e_config.default_timeout) + try: + wait_until_running(resumed, timeout=sdk_e2e_config.default_timeout) + wait_until_data_plane_ready( + resumed, + timeout=sdk_e2e_config.default_timeout, + command_timeout=sdk_e2e_config.command_timeout, + ) + assert_tcp_reachable( + probe(resumed, sdk_e2e_config, ALTERNATE_TCP_TARGET_IP), + ALTERNATE_TCP_TARGET_IP, + ) + assert_tcp_blocked(probe(resumed, sdk_e2e_config, TCP_TARGET_IP), TCP_TARGET_IP) + finally: + resumed.close() + + +@pytest.mark.requires_capability(NETWORK_ALLOW_DENY) +@pytest.mark.sandbox_create_options(allow_internet_access=False) +def test_repeated_updates_converge(sdk_sandbox, sdk_e2e_config): + """Replaying updates is safe: the control plane diffs against live state. + + Guards the incremental map programming — a diff that leaked or double-freed + entries would drift after a few rounds instead of landing on the same policy. + """ + for _ in range(3): + sdk_sandbox.update_network( + network={"allow_out": [TCP_TARGET_IP]}, + allow_internet_access=False, + ) + sdk_sandbox.update_network( + network={"allow_out": [ALTERNATE_TCP_TARGET_IP]}, + allow_internet_access=False, + ) + + assert_tcp_reachable( + probe(sdk_sandbox, sdk_e2e_config, ALTERNATE_TCP_TARGET_IP), + ALTERNATE_TCP_TARGET_IP, + ) + assert_tcp_blocked(probe(sdk_sandbox, sdk_e2e_config, TCP_TARGET_IP), TCP_TARGET_IP) diff --git a/tests/e2e/sdk_compat/docs/test-coverage.md b/tests/e2e/sdk_compat/docs/test-coverage.md index 81ee207bb..fdf4dadb7 100644 --- a/tests/e2e/sdk_compat/docs/test-coverage.md +++ b/tests/e2e/sdk_compat/docs/test-coverage.md @@ -116,6 +116,35 @@ These scenarios require Code Interpreter support and validate normalized `e2b-traffic-access-token` and `cube-traffic-access-token` both work with the correct token. +`cases/network/test_policy_update.py` covers in-place policy replacement on a +running sandbox (`network_dynamic_update`, CubeSandbox only): + +- granting a destination that was blocked at create time; +- revoking every destination, since the update replaces rather than patches; +- swapping the allow list so access moves instead of accumulating; +- **tearing down an established connection whose policy was revoked** — the case + that distinguishes this feature from create-time policy, because without + datapath re-evaluation an open channel would keep its create-time verdict; +- **leaving an established connection alone when the update still permits it** — + the necessary counterpart, since a datapath that killed every session on any + update would otherwise pass the case above; +- keeping DNS working after a domain-policy update, guarding the control-plane + resolver allowance that the caller never authors and could silently lose; +- **a clone inheriting the updated policy rather than the create-time one**, which + also covers read-your-writes: clone snapshots immediately after the update + returns, so it fails if the spec write lagged the response; +- **a clone of a narrowed policy not being more permissive** — the direction where + a stale spec would silently widen access for every descendant; +- **the updated policy surviving pause/resume**, since pause packages the sandbox + from Cubelet's own store rather than from the network runtime state file; +- converging under repeated updates, guarding the incremental map diff against + leaked or double-freed entries. + +The established-connection cases use the guest-side holder in +`framework/network_probe.py`, which keeps poking the peer on purpose: a revoked +flow is re-evaluated when the guest next sends, so nothing would be observed on +an idle socket. + `cases/lifecycle/test_pause_resume_network.py` covers the same create-time policies after an SDK pause + connect resume: diff --git a/tests/e2e/sdk_compat/docs/zh/test-coverage.md b/tests/e2e/sdk_compat/docs/zh/test-coverage.md index d3934c67d..bfad8d4ff 100644 --- a/tests/e2e/sdk_compat/docs/zh/test-coverage.md +++ b/tests/e2e/sdk_compat/docs/zh/test-coverage.md @@ -105,6 +105,29 @@ pytest --run-e2e -m "lifecycle and slow" - 限制公网 URL 访问时，缺失/错误 token 返回 403，`e2b-traffic-access-token` 与 `cube-traffic-access-token` 携带正确 token 时均可访问。 +`cases/network/test_policy_update.py` 覆盖运行中沙箱的策略原地替换 +（`network_dynamic_update`，仅 CubeSandbox）： + +- 放通创建时被阻断的目标； +- 传空策略即撤销全部——更新是整体替换而不是增量打补丁； +- 替换 allow list 时访问权限是「转移」而不是「叠加」； +- **被撤销的存量连接会被拆掉**——这一条是本功能与创建时策略的分界线：没有数据面 + 重判的话，已经打开的通道会一直沿用创建时的判决； +- **更新后仍放行的存量连接不受影响**——上一条的必要配套，否则一个「任何更新都 + 杀光所有 session」的实现也能通过上一条； +- 域名策略更新后 DNS 仍可用，守住那批由控制面注入、调用方从不书写、因而可能被 + 静默丢掉的 resolver 放行条目； +- **clone 继承的是更新后的策略而不是创建时的策略**，这条同时覆盖 read-your-writes： + clone 在 update 返回后立刻做快照，spec 写入慢于响应就会失败； +- **收紧型 update 之后的 clone 不会更宽松** —— 这是 spec 落后时会静默放大权限的方向； +- **更新后的策略能扛过 pause/resume**，因为 pause 打包读的是 Cubelet 自己的 store， + 不是 network runtime 的 state file； +- 反复更新可收敛，守住增量 map diff 不漏删、不重复删。 + +存量连接相关用例使用 `framework/network_probe.py` 里的 guest 侧 holder，它会 +刻意持续向对端发包：被撤销的流是在 guest 下一次发包时才被重判的，光挂着一个 +空闲 socket 观察不到任何现象。 + `cases/lifecycle/test_pause_resume_network.py` 覆盖 SDK pause + connect resume 后同一批创建时策略仍生效： diff --git a/tests/e2e/sdk_compat/framework/capabilities.py b/tests/e2e/sdk_compat/framework/capabilities.py index db960261a..196b2ce64 100644 --- a/tests/e2e/sdk_compat/framework/capabilities.py +++ b/tests/e2e/sdk_compat/framework/capabilities.py @@ -13,6 +13,9 @@ SET_TIMEOUT = "set_timeout" ROLLBACK_CLONE = "rollback_clone" NETWORK_ALLOW_DENY = "network_allow_deny" +# In-place egress policy replacement on a running sandbox, including +# re-evaluation of already-established connections. +NETWORK_DYNAMIC_UPDATE = "network_dynamic_update" NETWORK_PUBLIC_ACCESS = "network_public_access" NETWORK_MASK_REQUEST_HOST = "network_mask_request_host" NETWORK_L7_CUSTOM_PORT = "network_l7_custom_port" @@ -59,6 +62,7 @@ NETWORK_ALWAYS_DENIED, NETWORK_L7_EGRESS, NETWORK_TEMPLATE_MERGE, + NETWORK_DYNAMIC_UPDATE, PLATFORM_LIFECYCLE, HOST_MOUNT, VOLUME_PLUGIN, diff --git a/tests/e2e/sdk_compat/framework/network_probe.py b/tests/e2e/sdk_compat/framework/network_probe.py index 0e961855d..a67c876bb 100644 --- a/tests/e2e/sdk_compat/framework/network_probe.py +++ b/tests/e2e/sdk_compat/framework/network_probe.py @@ -86,6 +86,169 @@ def _is_timeout_block(output: str) -> bool: ) +# Guest-side holder for testing what a policy update does to a connection that +# is already established. +# +# Two constraints shape this, and they pull against each other: +# +# - The datapath only re-evaluates a flow when the guest sends on it, so a +# completely idle socket would observe nothing at all. +# - Sending application bytes is not an option. The peer speaks some protocol +# (a DNS server on TCP/53, say) and arbitrary bytes make it hang up — which +# arrives as a closed connection and is indistinguishable, to a naive probe, +# from the policy reset we are trying to detect. +# +# TCP keepalive satisfies both: the probes are bare ACKs that keep packets +# flowing on the 4-tuple without ever touching the byte stream. +# +# Writes READY_PATH once connected, then RESULT_PATH with one of: +# RESET - ECONNRESET, i.e. the datapath tore the flow down. The only outcome +# that unambiguously comes from policy enforcement. +# ALIVE - nothing happened for the whole observation window. +# EOF - the peer closed on its own (idle timeout, protocol policy). Says +# nothing about our policy, so callers treat it as inconclusive. +# DATA - the peer sent something unexpected; also inconclusive. +_ESTABLISHED_FLOW_HOLDER = """ +import socket, sys + +host, port, ready_path, result_path, window = ( + sys.argv[1], int(sys.argv[2]), sys.argv[3], sys.argv[4], float(sys.argv[5]) +) + + +def record(value): + with open(result_path, "w") as fh: + fh.write(value) + + +s = socket.socket() +s.settimeout(5) +try: + s.connect((host, port)) +except Exception as exc: + record("CONNECT_FAILED:%s" % exc) + sys.exit(0) + +# Probe every second after one second of idleness, so a revoked flow meets the +# datapath within ~1s of the update instead of waiting for application traffic. +s.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1) +for opt, value in (("TCP_KEEPIDLE", 1), ("TCP_KEEPINTVL", 1), ("TCP_KEEPCNT", 4)): + if hasattr(socket, opt): + s.setsockopt(socket.IPPROTO_TCP, getattr(socket, opt), value) + +with open(ready_path, "w") as fh: + fh.write("CONNECTED") + +s.settimeout(window) +try: + record("EOF" if s.recv(1) == b"" else "DATA") +except socket.timeout: + record("ALIVE") +except ConnectionResetError: + record("RESET") +except OSError as exc: + record("ERROR:%s" % type(exc).__name__) +""" + +ESTABLISHED_HOLDER_PATH = "/tmp/cube_e2e_hold_flow.py" +ESTABLISHED_READY_PATH = "/tmp/cube_e2e_hold_ready" +ESTABLISHED_RESULT_PATH = "/tmp/cube_e2e_hold_result" + + +# A revoked flow is reset within about a second of the update (keepalive probes +# run at 1s), so the window only needs a little slack. Every extra second adds +# exposure to the peer's own idle timeout, which arrives as an inconclusive EOF. +ESTABLISHED_OBSERVE_SECONDS = int( + os.environ.get("SDK_E2E_ESTABLISHED_WINDOW_SECONDS", "5") +) + +# Held connections use 443 rather than the reachability probes' port. The holder +# never writes to the socket, and a TLS server waiting for a ClientHello tolerates +# that for tens of seconds, whereas a DNS server closes an idle TCP/53 connection +# almost immediately. The policies under test allow the whole IP, so the port +# makes no difference to what is being asserted. +ESTABLISHED_TARGET_PORT = int( + os.environ.get("SDK_E2E_ESTABLISHED_TARGET_PORT", "443") +) + + +def start_established_flow( + sdk_sandbox, + host: str = TCP_TARGET_IP, + port: int = ESTABLISHED_TARGET_PORT, + *, + window_seconds: int = ESTABLISHED_OBSERVE_SECONDS, + command_timeout: int = 30, +) -> bool: + """Open a TCP connection inside the guest and leave it running in the background. + + window_seconds bounds how long the holder watches the connection. Keep it + short: a revoked flow is reset within about a second of the update, while a + long window only adds time for the peer to hit its own idle timeout and + muddy the result with an EOF. + + Returns whether the connection was established, so callers can skip rather + than fail when the target is simply not reachable in this environment. + """ + sdk_sandbox.write_file(ESTABLISHED_HOLDER_PATH, _ESTABLISHED_FLOW_HOLDER) + sdk_sandbox.run_command( + f"rm -f {ESTABLISHED_READY_PATH} {ESTABLISHED_RESULT_PATH}", + timeout=command_timeout, + ) + sdk_sandbox.run_command( + f"nohup python3 {ESTABLISHED_HOLDER_PATH} {host} {port} " + f"{ESTABLISHED_READY_PATH} {ESTABLISHED_RESULT_PATH} {window_seconds} " + f">/dev/null 2>&1 & echo started", + timeout=command_timeout, + ) + for _ in range(10): + probe = sdk_sandbox.run_command( + f"test -f {ESTABLISHED_READY_PATH} && echo READY || echo WAIT", + timeout=command_timeout, + ) + if "READY" in probe.stdout: + return True + time.sleep(1) + return False + + +def require_conclusive_flow_outcome(outcome: str, target: str) -> str: + """Skip unless the holder's verdict actually reflects policy enforcement. + + Only RESET and ALIVE do. EOF/DATA mean the peer acted on its own and a + PENDING/ERROR verdict means the probe never reached a conclusion — asserting + on any of those would report an environment quirk as a product bug. + """ + import pytest + + if outcome not in ("RESET", "ALIVE"): + pytest.skip( + f"held connection to {target} gave an inconclusive verdict {outcome!r}; " + "the peer closed it or the probe never settled, so this run cannot " + "tell a policy reset from a peer-side close" + ) + return outcome + + +def wait_established_flow_outcome( + sdk_sandbox, + *, + attempts: int = 25, + command_timeout: int = 30, +) -> str: + """Poll for the holder's verdict; returns RESET, ALIVE, EOF, DATA, ERROR:*, or PENDING.""" + for _ in range(attempts): + probe = sdk_sandbox.run_command( + f"cat {ESTABLISHED_RESULT_PATH} 2>/dev/null || echo PENDING", + timeout=command_timeout, + ) + outcome = probe.stdout.strip() + if outcome and outcome != "PENDING": + return outcome + time.sleep(1) + return "PENDING" + + def public_url(sdk_sandbox) -> str: host = sdk_sandbox.get_host(PUBLIC_ACCESS_PORT).rstrip("/") path = PUBLIC_ACCESS_PATH if PUBLIC_ACCESS_PATH.startswith("/") else f"/{PUBLIC_ACCESS_PATH}"