feat(cubenet): support arbitrary custom L7 port + scheme in egress rules

[zhouxianping]

Summary

L7 egress rules can now pin the TCP port CubeEgress intercepts on, via
Match.port + Match.scheme. This extends the existing host/path/SNI
matching so a rule applies to a specific (host, port, scheme) tuple instead
of always the default {80/http, 443/https} set.

Changes by layer

• CubeNet dataplane (mvmtap/session/tcp): port+scheme → skb->mark
  on egress SYN; spec tests updated.
• CubeEgress: Lua policy + iptables mark validation for custom
  (host, port, scheme).
• Cubelet: embedded network runtime + policy builder handle port/scheme.
• CubeAPI / CubeMaster: validate egress rule port/scheme at API and master.
• SDKs (go / node / python): port/scheme validation in egress rule match.
• deploy/one-click: bind L7 skb-mark config into the install flow.
• docs (security-proxy.md EN + ZH): document custom L7 ports, the four
  port/scheme quadrants, and constraints.
• examples: L7 custom-port echo demos.

SDK compatibility e2e (this session)

• cases/network/test_l7_custom_port.py: 5 cases — custom HTTP port injection,
  scheme-only default-set interception, HTTPS custom-port interception,
  port-without-scheme rejection, subnet-host rejection.
• framework/capabilities.py: NETWORK_L7_CUSTOM_PORT.
• CubeNet/cubevs/Makefile: test: gen so stale BPF objects are regenerated
  before make test (prevents verifier failures from outdated .o).

Validation

• go test ./... under CubeNet/cubevs passes (after make gen).
• SDK compatibility e2e: test_l7_custom_port.py 5 passed against a template
  that trusts the L7 interception CA.

[fslongjin]

Review of PR #1347 — arbitrary custom L7 port + scheme in egress rules

Overall this is a well-engineered change: the unified classify_egress_flow() verdict, the /48 port-in-key LPM trie, the key_prefixlen guard for exact-key merge, the fail-closed unknown-scheme handling, and the static asserts mirroring BPF structs are all high quality. The findings below are what I'd want addressed before merge, in severity order.

Correctness

1. [High] Migration rollback ordering can permanently lose allow_out policy
CubeNet/cubevs/migration.go:98-118 (with Init rollback at miscs.go:299-305)

migratePersistentPolicyMaps() deletes the legacy allow_out/allow_out_v2 pin (line 109) immediately after migrateAllowOutMap succeeds, before migrateDNSAllowMap (line 114). If the DNS migration then fails, Init's deferred rollback removes the freshly-populated allow_out_v3/dns_allow_v2 pins; on the next restart allowOutMigrationSource() returns "" (legacy pin is gone) and the allow_out policy is never re-migrated. The "leave the legacy pin in place so the migration can be retried" comment only covers an allow-map failure, not a later DNS failure. Suggestion: migrate both maps first and only unlink the legacy pins after both succeed. The existing rollback tests (TestMigrateDNSAllowMapFailureKeepsLegacyPin, TestMigrateAllowOutMapFailureKeepsLegacyPin) only exercise single-map failure and miss this combined case.

2. [High] Port-less deny rules are silently narrowed to {80/http, 443/https}; a custom-port allow can bypass a broader deny
CubeEgress/lua/access_phase.lua:144-152 via port_scheme.matches

Pre-PR, a deny rule with a host + scheme matched all intercepted traffic to that host. Post-PR, port_scheme.matches(nil, nil, dst_port, scheme) expands to only {80/http, 443/https}, so a deny rule with no port matches only the default set. Combined with a custom-port allow rule (e.g. deny host="*.example.com" + allow host="api.example.com", port=8443, scheme=https), api.example.com:8443 traffic is allowed because the deny rule never matches port 8443 and decide() is first-match-wins. For deny rules the narrowing is fail-open (for allow rules it's fail-closed). Consider making deny apply port-agnostically within the host, or at minimum documenting the narrowing prominently and auditing existing deny rules on upgrade.

3. [Medium] Non-atomic iptables chain install can leave a fail-open interception state
CubeEgress/scripts/cube-proxy-iptables-init.sh:124-144

The chain is built by iptables -F then two sequential -A TPROXY rules with no rollback and no post-condition check. If the second (HTTPS) rule fails after the first succeeds, the script aborts before install_routing(), so mark-based ip rule routing is never installed and HTTPS-marked packets fall through to the default route — a silent bypass of all L7 interception/deny for the HTTPS scheme. Consider building into a scratch chain + atomic swap (or verifying both rules are present before proceeding).

4. [Medium] Cubelet cloneEgressRules shallow-copies the new Match.Port pointer
Cubelet/network/runtime/policy_builder.go:47-51

match := *r.Match shares Port *int (and the pre-existing Host/Path/SNI/Scheme pointers) with the caller's request, despite the function contract claiming a deep copy. Method is correctly deep-copied. CubeMaster fixed this with cloneIntPtr (types.go:227); Cubelet should mirror it.

Test coverage / CI

5. [High] The new CubeNet/cubevs unit tests (~2000 lines, 8 files) are not executed by CI or run.sh
.github/workflows/unit-test-check.yml:185-189 maps CubeNet/* → cubelet → cubelet-pkg-test (go test -short ./pkg/... under Cubelet). CubeNet/cubevs is a separate Go module, so custom_port_test.go, dns_learn_test.go, egress_policy_test.go, l7_mark_test.go, migration_test.go, tcp_state_test.go, netpolicy_test.go, dump_test.go are never compiled/run. Add a go test ./... for the cubevs module to the workflow (and tests/unittest/run.sh).

6. [High] The FLOW_HTTP verdict — the primary new plaintext path — is never asserted
CubeNet/cubevs/egress_policy_test.go:12-17

The verdict constants define reject=0, SNAT=1, HTTPS=3 but no HTTP=2, and the seed cases only cover HTTPS. Add flowVerdictHTTP = 2 and a case seeding a /48 with Scheme: L7SchemeHTTP.

7. [High] iptables TPROXY rule generation is stubbed out in tests
CubeEgress/tests/cube-proxy-iptables-init_test.sh:98 overrides install_chain() with echo, so the actual two mark→TPROXY rules (--on-port 8080/8443, mark/mask match) and the gateway-from-CIDR arithmetic are never asserted. A regression in --on-port or the mark match would pass CI.

8. [Medium] Migration success path and dump port rendering are untested

• CubeNet/cubevs/migration_test.go has full bpffs success tests for DNS migration but only rollback tests for allow_out_v2→allow_out_v3; the 16-byte L7+expires → /48 expansion upgrade path is unproven.
• CubeNet/cubevs/dump_test.go never exercises l7PortEntriesToDump / dumpPolicyInnerMap NBO→host port rendering; a byte-order regression would show 0x20EB instead of 8443.

9. [Medium] HTTPS custom-port e2e asserts nothing that distinguishes interception from passthrough
tests/e2e/sdk_compat/cases/network/test_l7_custom_port.py asserts only code=200 + non-empty body against a public endpoint. Plain SNAT would satisfy the same invariant, and the case is gated behind SDK_E2E_L7_HTTPS_CAP_TRUSTED=1 (silently skipped by default). Consider an endpoint that echoes a request-observable signal, or document that the leg only proves reachability.

Docs / conventions

10. [Medium] Non-English character in a Go comment
CubeNet/cubevs/dump.go:184 — Scheme string json:"scheme" // "http" | "https" | ""(未知) violates the project's non-English-character policy. Change to "" (unknown).

11. [Medium] "Enforced by the SDK client" overstates SDK coverage
docs/guide/security-proxy.md:148 (and ZH :135): the SDK clients only validate the port range + scheme + port-requires-scheme; the subnet-CIDR rejection, the (host, port) scheme-conflict rejection, and the 8-tuple cap are server-side only. Reword to "validated server-side; the SDK client also pre-validates the port/scheme pair".

Minor

12. [Low] Scheme-only rules silently narrow datapath interception to a single port
Cubelet/network/runtime/policy_builder.go:218 expands scheme-only https to port 443 only. Pre-PR the host stayed intercepted on the full {80,443} set and the scheme was a CubeEgress filter; with a CubeEgress default-deny sandbox, plaintext HTTP to that host now bypasses the proxy entirely. Docs were updated to the new semantics, but the Python SDK Match.__post_init__ docstring (sdk/python/cubesandbox/_policy.py:87-92) and the Go SDK test comment still describe the old behavior — callers will be misled.

13. [Low] Dead validation helpers with a contradictory counting model
CubeNet/cubevs/netpolicy.go:759-805 — validateNetPolicyEntryCounts/countUniqueLPMEntries/countUniqueDNSAllowEntries are only referenced from tests and count one entry per LPM key, contradicting expandedAllowOutEntryCount (per-port). If re-wired into production they'd reintroduce the E2BIG overflow the PR's own comment describes. Recommend deleting them.

14. [Low] from_envoy reply path now does 2 dependent hash lookups on every TCP reply packet
CubeNet/src/localgw.bpf.c:19-46 — update_l7_session_from_cubedev runs on every L7-proxy reply packet (pull headers + lookup_session + bpf_ktime_get_ns), where the previous from_envoy did zero map lookups. Gate on control-flag packets (syn||fin||rst) to remove the steady-state data/ACK cost; verify against the tcp-state tests before landing.

15. [Low] Cross-layer validation parity: CubeAPI scheme check doesn't trim whitespace
CubeAPI/src/services/sandboxes.rs:1107 uses eq_ignore_ascii_case without trim(), so " https " is rejected by CubeAPI but accepted by CubeMaster and all SDKs. Also: persistentPolicyGenerationExists (miscs.go:274-291) hard-fails with no self-healing when exactly one of allow_out_v3/dns_allow_v2 is pinned.

---

Thanks for the thorough spec-test harness and the e2e coverage — these are above average for the repo. The correctness items (#1, #2) and the CI gap (#5) are the ones I'd prioritize.

[cubesandboxbot]

Minor / plausible edge case: an expired DNS-learned /48 L7 entry shadows a live /32 static allow.

classify_egress_flow performs a single LPM lookup with key.prefixlen = 48. LPM returns the longest-prefix match regardless of the value's expiry, so when a DNS-learned /48 for (ip, port) has expired but has not yet been reaped, this lookup still returns that /48 (it out-ranks the /32). The expiry check then fails, and the function falls through to the deny_out / default-allow phase without ever consulting the /32.

Failure scenario: default-deny (deny_all 0.0.0.0/0), a static /32 plain allow for IP X, and a DNS-learned L7 /48 for (X, 80). While the /48 is expired-but-unreaped, a flow to (X, 80) is rejected — the static allow is effectively revoked — even though the rule author explicitly allowed X. Under default-allow this degrades harmlessly to FLOW_SNAT, which is why it's narrow; but the /32 fallback is unreachable by construction, not by design.

Suggested fix: when the matched value is present but expired, re-lookup with key.prefixlen = 32 (the plain /32/subnet fallback) before evaluating deny_out, or treat "matched but expired" as a miss so the LPM fallback to /32 applies.

[cubesandboxbot]

Minor / plausible gap: custom-port L7 connections are RST'd on session loss even while the policy entry is still live.

This lost-session drain path only covers dport 80/443. For an established custom-port L7 flow whose egress_sessions entry is lost (e.g. CubeNet restarts and re-creates the session map while the pinned allow_out_v3 map survives), the socket lookup succeeds (the TPROXY socket is still ESTABLISHED) but the dport guard excludes it, and the packet falls through to return rst ? TCP_NAT_DROP : TCP_NAT_RESET.

The result is asymmetric: the same restart preserves established 80/443 connections via mark re-stamp + redirect, but resets an established custom-port connection — even though classify_egress_flow would still return FLOW_HTTP/FLOW_HTTPS for this flow, and a fresh SYN on the same tuple would be admitted and re-proxied.

The comment motivates the exclusion with "they should respect the current policy when their allow_out_v3 entry expires" — that covers the expired-policy case, but not the live-policy, lost-session case, where the RST contradicts the still-valid allow.

Suggested fix: in the !sess branch, for non-80/443 ports, consult classify_egress_flow first; if it still returns FLOW_HTTP/FLOW_HTTPS, re-create the L7 session (or at minimum re-stamp the mark and redirect), and fall back to reset only when the policy no longer allows the flow.

[cubesandboxbot]

F1 (Medium · availability): half-pinned map generation permanently bricks startup.

When only one of the two new-generation pins exists, this returns an error, and Init returns immediately — before the deferred cleanup that removes orphaned new-generation pins has been registered (that defer is only installed inside the if !generationExists branch below). If a previous boot died between pinning allow_out_v3 and dns_allow_v2 (crash, OOM-kill, power loss mid-migration), the orphan pin survives forever and every restart fails this same consistency check; the node cannot start until an operator manually unlinks bpffs.

Suggestion: register the cleanup defer unconditionally (before the generation check), or treat a mismatch as an incomplete generation — remove the orphan pin(s) and proceed with a fresh migration (generationExists = false) instead of returning an error.

[cubesandboxbot]

F2 (Medium · data loss): legacy allow_out source is removed before DNS migration completes.

The legacy allow_out pin is deleted here, immediately after migrateAllowOutMap, but before migrateDNSAllowMap runs below. If the DNS migration then fails, Init's deferred cleanup removes the freshly-migrated allow_out_v3 pin, and on the next restart allowOutMigrationSource() returns "" (the legacy pin is already gone) — so the allow migration is silently skipped and all allow_out policy is permanently lost while DNS rules are retried. The comment on the migrateAllowOutMap failure path ("Leave the legacy pin in place so the migration can be retried") shows the intent; the current ordering defeats it.

Suggestion: move removePinnedMap(allowSrc) to the end of the function so the legacy source is removed only after the entire generation (allow + DNS) has been migrated successfully.

[cubesandboxbot]

F3 (Medium · security/consistency): UDP/ICMP to an L7-pinned (host, port) bypasses the proxy.

Only FLOW_REJECT is handled; FLOW_HTTP/FLOW_HTTPS falls through to plain SNAT. But classify_egress_flow returns an L7 verdict for any flow whose (daddr, dport) matches an L7-required /48 entry, regardless of L4 protocol — while the proxy only listens on TCP 8080/8443. So a UDP flow to a pinned port (QUIC/HTTP-3 on 443, or any custom UDP L7 port) is SNAT'd straight to the internet with no proxy inspection, silently bypassing L7 audit/deny rules for that (host, port). The same pattern exists in do_icmp_nat (~line 473).

Suggestion: for UDP/ICMP, treat FLOW_HTTP/FLOW_HTTPS as an unsupported transport for L7 interception and drop (or explicitly log/audit the bypass) rather than silently passing; at minimum document that L7 pinning is TCP-only and consider rejecting UDP L7 ports at validation time.

[cubesandboxbot]

Potential regression — plain L3 allow for IP/CIDR hosts is silently dropped when an L7 rule is added for the same host.

When a host appears in both plain allow_out (as an IP//32) and as an L7 rule host, this merge ORs the L7 flag onto the entry and replaces its port set. Downstream, populateAllowOutInnerMap writes only the /48 (ip, port) entries for any NET_POLICY_FLAG_L7_REQUIRED entry and skips the plain /32 write entirely (if entry.flags&netPolicyFlagL7Required != 0 { ...continue }). So the explicit plain allow for that IP (any port) is replaced by L7-only /48 entries for exactly the rule's ports.

This is asymmetric with the domain path: mergeDNSAllowRules (below) marks the merged domain entry with NET_POLICY_FLAG_L3_ALLOWED so the datapath learns the plain /32 alongside the L7 /48 entries — its own comment says "otherwise the L7 rule would silently narrow the same-domain plain allow_out to only the rule's ports." The domain behavior is pinned by TestBuildNetPolicyPlanL3AllowedCoexistence; there is no equivalent test for the IP/CIDR path.

Failure scenario: allowOut = ["10.0.0.5"] + L7 rule {host: "10.0.0.5", port: 8443, scheme: https}. After merge, the /32 10.0.0.5 entry is gone. A flow to 10.0.0.5:22 misses /48 (10.0.0.5, 22) and hits the deny check; with any deny rule covering the host (e.g. a default-deny backstop) it is now rejected, whereas the base code allowed it via the explicit /32. Even without a deny rule, the explicit allow is silently removed — contradicting the documented "adding an L7 rule does not remove plain L3 access" coexistence (docs/guide/security-proxy.md, which is tested for domains only).

Suggestion: mirror the domain path — preserve the plain /32 (or covering subnet) entry when merging, and add a coexistence test for the IP/CIDR case analogous to TestBuildNetPolicyPlanL3AllowedCoexistence.

[cubesandboxbot]

P2 — this budget counter undercounts multi-port L7 entries, and the refactored budget tests now rely on it.

countUniqueLPMEntries dedups by lpmKey{Prefixlen, IP} — there is no port dimension. A host with N pinned L7 ports occupies N /48 entries in the allow_out_v3 inner LPM trie but counts as 1 here. The production path is correct: validateNetPolicyPlan uses expandedAllowOutEntryCount (which expands to len(ports), default {80,443}, plus +1 for the L3_ALLOWED /32), and that helper's docstring explicitly warns the per-host count "undercounts multi-port L7 rules."

The problem is that the budget tests were refactored off the production builder (buildNetPolicyPlan) onto this standalone helper, so the 8193 > 8192 boundary assertions and the new TestValidateNetPolicyEntryCountsDeduplicatesByMapKey no longer exercise the counting that actually governs map population. A plan that passes the deduped count but would overflow the inner trie mid-write (E2BIG → half-populated map) can false-pass.

Either make the dedup key port-aware (include the port, e.g. reuse lpmKeyV3), or drop the helper and keep the budget tests on buildNetPolicyPlan. The generic validateNetPolicyEntryCounts name/signature also reads as production-ready, which is a footgun for future callers.

[cubesandboxbot]

P2/P3 — terminal L7 sessions block immediate 4-tuple reuse with RST until the userspace reaper removes them.

The rationale in the comment (identity reverse tuple ⇒ delayed packets from the old connection could mutate a fresh session) is sound, but the window is real: the reaper ticks every 5 s, and the terminal-session timeouts are 10 s for CLOSE/active-close and up to 2 min for TIME_WAIT. A client that closes and immediately reconnects on the same tuple is RST'd on its SYN for up to ~10 s in the common close case.

This only affects L7-intercepted custom ports (default 80/443 drain and fresh SNAT sessions are unaffected), but it is a plausible source of transient connection failures for fast-reconnect clients. Consider removing terminal L7 sessions more aggressively (or via a shorter reaper pass), or at least documenting the "avoid immediate client-port reuse" caveat in security-proxy.md.

[cubesandboxbot]

P3 — dead code added in this PR. ingressSession is never referenced anywhere; the //nolint:unused silences exactly the lint that would catch it. Either wire it into an event/log path (it mirrors egressSession) or drop it.

[cubesandboxbot]

P3 (nit) — mixed-language comment. // "http" | "https" | ""(未知) — the rest of the file comments in English; this should be "" (unknown) or similar.

[cubesandboxbot]

Review: feat(cubenet): support arbitrary custom L7 port + scheme in egress rules

PR: #1347 · Author: zhouxianping · Base: master · Head: feat/custom_l7_port
Scope: ~8,604 additions / 575 deletions across ~85 files (CubeNet BPF/userspace, CubeEgress Lua + iptables, Cubelet runtime, CubeAPI/CubeMaster, SDKs go/node/python, docs, examples).

AI-generated review. This review was produced by an automated reviewer; it has not been approved by a human. Findings are ranked by severity; inline comments are posted on the diff. Please verify each item before acting on it.

---

Overview

The PR extends L7 egress rules so a rule can pin the TCP port CubeEgress intercepts via Match.port + Match.scheme, replacing the hardcoded {80/http, 443/https} set with an arbitrary (host, port, scheme) tuple. This is a deep, cross-cutting change:

• Dataplane (CubeNet/src): classify_egress_flow returns a unified verdict (FLOW_SNAT / FLOW_HTTP / FLOW_HTTPS / FLOW_REJECT); allow/deny maps migrate v2→v3 with the port moved into the key (lpm_key_v3 /48 exact (ip,port), /32 any-port, /<32 subnet); L7 sessions carry packet_class/l7_scheme and stamp skb->mark so CubeEgress's TPROXY can steer by fwmark instead of dport.
• CubeEgress: new port_scheme.lua implements the four port/scheme quadrants with fail-closed semantics; cube-proxy-iptables-init.sh now validates the shared /etc/cubeegress/l7-marks.conf and swaps chains atomically via a scratch chain.
• Orchestration: Cubelet extracts (host, port, scheme) targets and normalizes scheme-only rules to default ports; CubeAPI/CubeMaster/SDKs validate port/scheme consistently.
• Migration: allow_out and dns_allow migrate one-way with rollback; legacy pins are removed only after both migrations succeed.

The change is unusually well-engineered for its size: validation is consistent across every layer, the port/scheme semantics are fail-closed rather than silently permissive, verifier constraints are handled carefully (#pragma unroll with guard-in-body, __u8 states), and there is a genuine test suite at every layer (BPF spec tests, Lua, iptables shell, Go migration, SDK e2e). The production budget counter expandedAllowOutEntryCount is correct and its docstring is honest about why the naive count is wrong.

The findings below are mostly test-quality and edge-case concerns rather than merge-blocking defects. Finding 1 should be fixed before the budget tests are relied on; Finding 2 deserves a documented mitigation or a faster reaper for terminal L7 sessions.

---

Findings

1. P2 — validateNetPolicyEntryCounts/countUniqueLPMEntries undercount multi-port L7 entries, and the refactored budget tests no longer exercise the production path

CubeNet/cubevs/netpolicy.go:842

countUniqueLPMEntries dedups by lpmKey{Prefixlen, IP} — there is no port dimension. A host with 8 pinned L7 ports is counted once, even though it occupies 8 /48 entries in the allow_out_v3 inner LPM trie. The production path is correct: validateNetPolicyPlan uses expandedAllowOutEntryCount (expands to len(ports), default {80,443}, +1 for the L3_ALLOWED /32 coexistence entry), and that helper's docstring explicitly warns that a per-host count "undercounts multi-port L7 rules."

The problem is that the budget tests were refactored off the production builder onto the standalone helper:

• TestValidateNetPolicyEntryCount* now calls validateNetPolicyEntryCounts(allowOutCIDRs, l7AllowOutCIDRs, nil, nil, nil) with raw CIDR slices instead of buildNetPolicyPlan(MVMOptions{...}), so the 8193 > 8192 boundary assertions no longer exercise the counting that actually governs map population.
• The new TestValidateNetPolicyEntryCountsDeduplicatesByMapKey asserts the undercounting behavior (no error for one host shared across plain-allow + L7), which is exactly the case where production would count 2–9 entries.

Consequence: a policy that passes the deduped budget check but overflows the inner trie mid-write (E2BIG → half-populated map) can false-pass. The helper's generic name and production-looking signature are a footgun for future callers. Either make countUniqueLPMEntries port-aware (include the port in the dedup key, e.g. reuse lpmKeyV3), or delete the helper and keep the budget tests on buildNetPolicyPlan.

2. P2/P3 — L7 session tuple-reuse after CLOSE/TIME_WAIT returns RST until the reaper removes the old session

CubeNet/src/mvmtap.bpf.c:733

In do_tcp_nat, when an existing L7 session is terminal (TCP_CONNTRACK_CLOSE/TIME_WAIT) and a new SYN arrives on the same 4-tuple, the datapath returns TCP_NAT_RESET and leaves the old pair in place until the userspace reaper deletes it. The comment explains the rationale (identity reverse tuple ⇒ delayed packets from the old connection could mutate a fresh session), which is sound. But the window is real: the reaper ticks every 5 s, and the terminal-session timeouts are 10 s (CLOSE / active-close) up to 2 min (TIME_WAIT) — so a client that closes and immediately reconnects on the same tuple can be RST'd for up to ~10 s in the common close case. This only affects L7-intercepted custom ports (the default 80/443 drain path and fresh SNAT sessions are unaffected), but it is a plausible source of transient connection failures for fast-reconnect clients. Consider deleting terminal L7 sessions more aggressively, or documenting the behavior in security-proxy.md alongside the "avoid reusing client ports immediately" caveat.

3. P3 — dead ingressSession helper added in reaper.go

CubeNet/cubevs/reaper.go:227

ingressSession(key, value) string is added but never referenced anywhere; it is silenced with //nolint:unused. Dead code — either wire it into an event/log path or drop it (the //nolint suppresses exactly the lint that would catch it).

4. P3 — stray Chinese comment in dump.go

CubeNet/cubevs/dump.go:187

Scheme string `json:"scheme"` // "http" | "https" | ""(未知)

Mixed-language comment in a Go source file; the rest of the codebase comments in English. Cosmetic but worth a one-line fix.

---

Behavior & upgrade notes (no change required, but worth knowing)

• DNS-learned L7 rules are now fail-closed on non-rule ports. In v2, dns_learn_response_ip wrote L7-required domains as /32 any-port allow entries, so any port to a DNS-learned L7 host was allowed. In v3, only /48 per-rule-port entries are written (default {80,443} when the rule has no explicit ports). After upgrade, a non-rule port (e.g. 22, or 8443 when only 443 is pinned) to an L7-only domain that used to pass now falls through to deny_out and is rejected. The migration copies L7 entries as default-port /48 entries, so it does not preserve any-port semantics unless the host is also present in plain allow_out (which adds the L3_ALLOWED /32). This is documented, and the fail-closed direction is defensible, but it is a silent behavior change on upgrade for existing DNS-learned L7 domains.
• from_cube node-IP corner case. The nodenic_ip branch calls classify_egress_flow(ifindex, daddr, 0) with dport = 0, which can never match a /48 L7 entry. Traffic from a sandbox to an L7-only domain's node IP (the node's own address, e.g. a hostNetworked service) that v2 allowed via the /32 L7 entry is now denied under deny-all. Narrow corner case, but the dport=0 sentinel silently changes semantics for these flows.
• L7 deny rules only cover intercepted tuples. A deny rule with a pinned (port, scheme) matches only traffic CubeEgress intercepts on that tuple; other ports to the same host are governed by deny_out (L3/L4). Internally consistent with the fail-closed design, but operators should not assume an L7 deny rule blocks the host on all ports.
• Shared mark config is now a cross-component contract. /etc/cubeegress/l7-marks.conf is parsed by both the iptables init script and Cubelet's loadL7MarksConfig; both tolerate hex, an optional export prefix, and trailing comments. Nice defensive parsing on both sides — just keep the two parsers in sync, since a divergence would silently desync mark-based steering.

---

Overall

High-quality, coherent, and unusually well-tested PR. The port/scheme model is consistent from API validation down to the BPF key layout and Lua matching, and the fail-closed defaults are the right call for a security proxy. None of the findings block merge on their own; I'd ask for a fix (or an explicit doc/name change) on Finding 1 and a documented mitigation on Finding 2 before considering this fully shippable.

[lkml-likexu]

@FakeLearne Please confirm whether the following test cases are already present in tests/e2e/sdk_compat/ (pls delete for yes). If not, please move them there.

tests/test_l7_custom_port_e2e.py::test_l7_custom_http_inject_and_https_deny SKIPPED (use --run-e2e or set CUBE_E2E=1)
tests/test_l7_custom_port_e2e.py::test_l7_custom_https_allow_reaches_real_upstream SKIPPED (use --run-e2e or set CUBE_E2E=1)
tests/test_l7_custom_port_validation_e2e.py::test_l7_custom_scheme_conflict_rejected SKIPPED (use --run-e2e or set CUBE_E2E=1)
tests/test_l7_custom_port_validation_e2e.py::test_l7_custom_port_budget_exceeded_rejected SKIPPED (use --run-e2e or set CUBE_E2E=1)