perf(hypervisor): speed up incremental snapshots with PM_FILE (pagemap bit 61)

[fslongjin]

Summary

Incremental snapshots only need to save the pages the Guest has written. On newer host kernels, this PR replaces "query /proc/kpageflags per present page" with "a single sequential pagemap read," deciding via pagemap bit 61 (PM_FILE).

• New kernels use bit 61: one sequential read, and CAP_SYS_ADMIN is no longer required.
• Old kernels keep the existing kpageflags path; behavior unchanged.

The snapshot file format and the baseline-file convention are unchanged. Existing snapshots need no migration.

Performance:

• Scan alone (no disk writes): 64 MiB / 256 MiB / 1 GiB mappings are about 35× / 35× / 29× faster.
• Production path (PVM, 2 GiB sandbox, first create_snapshot()), same size points as the scan: 64 MiB wash (155 → 154 ms); 256 MiB 209 → 178 ms (−31 ms); 1 GiB 384 → 305 ms (−79 ms). Almost no Guest writes is also a wash. The end-to-end speedup is far smaller than the scan's, because writing the dirty pages to the snapshot file dominates total time.
• The new path writes the same amount of data as the old one (within 1%), so it neither under-saves (which would corrupt the snapshot) nor degrades into a full memory dump.

Background

Cube's fast restore backs Guest RAM by a read-only baseline file. When the Guest only reads a page, that page comes straight from the baseline; once the Guest writes, the kernel copies it into a process-private page (copy-on-write, CoW). An incremental snapshot only needs to save these private pages back; untouched pages are reused from the baseline.

So "pick the pages to save" is equivalent to "pick the pages the Guest has written."

flowchart LR
  base["read-only baseline file"] --> fault{"how does the Guest touch this page?"}
  fault -->|"never accessed"| skip1["no PTE: don't save"]
  fault -->|"read only"| filePage["still a baseline page: don't save"]
  fault -->|"writes"| cow["copied to a private page: must save"]
  cow --> swap{"swapped out?"}
  swap -->|yes| saveSwap["swapped page: still must save"]
  swap -->|no| saveRam["in memory and not a file page: save"]

Loading

Guest page state	Save to incremental?
Never accessed (no PTE)	No
Only read, still a baseline page	No (already in the baseline)
Guest wrote, copied to a private page	Yes
Private page swapped out	Yes

Why the old path is slow

The old implementation reads pagemap once, then for every present page queries /proc/kpageflags once to ask "is this an anonymous page?"

When Guest RAM is large but few pages were actually written, this becomes hundreds of thousands of 8-byte random reads, just to confirm "this page need not be saved." Even with few dirty pages, the incremental snapshot can stall here. Also, /proc/kpageflags is usually only openable by root; without CAP_SYS_ADMIN the whole step fails.

flowchart TD
  start["pick pages to save"] --> readPm["one sequential pagemap read"]
  readPm --> loop["for each present page"]
  loop --> seek["query /proc/kpageflags"]
  seek --> kpf{"anonymous page?"}
  kpf -->|yes| save["write to incremental"]
  kpf -->|no| skip["skip"]

Loading

The new path

The key observation: the old path actually answers two questions—"is this page in memory?" and "is this a private page the Guest wrote?". The first is already answered by pagemap (bit 63, present); the second detours to /proc/kpageflags for KPF_ANON, so every present page issues another syscall.

But in the same 64-bit pagemap entry, bit 61 (PM_FILE) already marks whether this is a file page. A page the Guest wrote is CoW'd into a private anonymous page, which is not a file page, so PM_FILE = 0. In other words, the answer the old path went to kpageflags for is already in the pagemap read it already did:

must_save = swapped, or (present and not a file page)
          = swapped || (present && !PM_FILE)

On kernels that correctly set PM_FILE on file huge pages, this predicate selects exactly the same set of pages as the old present && KPF_ANON—neither fewer nor more—just trading "one kpageflags query per page" for "a bit already in the one sequential read." Since we no longer need to pull a PFN out of pagemap to query another file, CAP_SYS_ADMIN is no longer required either.

flowchart TD
  start["pick pages to save"] --> gate{"host kernel supports PM_FILE?"}
  gate -->|yes| bit61["one sequential pagemap read<br/>decide by bit 61"]
  gate -->|no| kpf["pagemap + per-page kpageflags<br/>still needs CAP_SYS_ADMIN"]
  bit61 --> out["list of pages to save"]
  kpf --> out

Loading

Choosing by kernel version

Correct PM_FILE marking on file huge pages was fixed only later (upstream 3f9f022). So we can't simply check "version ≥ 6.6.44": 6.7–6.10 are numerically newer, but mainline only merged the fix around 6.11.

This PR's enablement:

Host kernel	Incremental snapshot path
6.6.44+ (6.6.x only), 6.11+, 7+	bit 61
5.x, 6.1, 6.6.43, 6.7–6.10, parse failure	kpageflags (unchanged)

The first scan emits one log line for verification: pagemap_anon: kernel=... path=bit61|kpageflags.

Performance

Scan alone: ~30×

Measures only the "pick pages to save" step, no disk writes. Release build, Linux 7.0, read all pages sequentially then write 10%.

Mapping	Old path	New path	Speedup
64 MiB	4.17 ms	0.12 ms	35.7×
256 MiB	16.66 ms	0.48 ms	34.9×
1 GiB	67.51 ms	2.34 ms	28.8×

Production path: same sizes as the scan (64 / 256 / 1 GiB)

PVM host kernel Linux 6.6.69-opencloudos9.cubesandbox.pvm.host-gb85200d80fa2 , 2 GiB sandbox, same template. A fresh sandbox each round, measuring only the first create_snapshot(). 1 warm-up + 5 measured rounds per size; the table shows p50. Guest dirtying uses dd into /dev/shm (same sizes as the scan table). Logs confirm the new path uses bit 61; at each size the new path writes the same amount of data as the old one (within 1%), so it neither under-saves (which would corrupt the snapshot) nor degrades into a full memory dump.

Guest writes	Old path	New path	Time change
Almost none	141 ms	136 ms	wash
64 MiB	155 ms	154 ms	wash
256 MiB	209 ms	178 ms	−31 ms
1 GiB	384 ms	305 ms	−79 ms

The end-to-end speedup is far below the scan's 30×, as expected: total create_snapshot() time is dominated by writing dirty pages to the snapshot file; scanning is just one part of it. At 64 MiB the scan tax is still buried in noise; at 256 MiB it shows (~30 ms); at ~1 GiB the old path's per-page query cost across hundreds of thousands of present pages surfaces (~80 ms), matching the order of magnitude of "1 GiB scan 67 ms → 2 ms" above.

Correctness

Under-saving would silently corrupt a snapshot, which is this PR's main concern.

• A page the Guest wrote is always CoW'd into a private page; the new path saves it, the old path saves it too—there is no path that under-saves.
• On kernels that don't mark PM_FILE correctly, a file huge page can be misclassified as anonymous, causing over-saving (the snapshot is still valid, just larger). This PR gates those kernels out by version and keeps them on kpageflags.

Compatibility

• The snapshot format is unchanged; files written by the old and new paths are mutually compatible.
• Old kernels behave exactly as before.
• The first scan emits a path= log line so you can confirm at runtime which path was taken.

[cubesandboxbot]

Review: perf(hypervisor): speed up incremental snapshots with PM_FILE (pagemap bit 61)

AI-generated review — not a human approval.

Overall this is a well-structured, well-tested PR. The central idea is sound: for the fast-restore scenario the Guest RAM is a MAP_PRIVATE mapping of a read-only baseline file, so "must save" is exactly swapped || (present && !PM_FILE), and reading that bit from the single sequential pagemap read is a strict improvement over one /proc/kpageflags seek+read per present page. The kernel-version gate is a sensible way to keep kernels with broken file-huge-page PM_FILE on the old path, and the added tests (decoder, MAP_PRIVATE fixture, cross-check against kpageflags, privileged benchmark) are genuinely useful.

I verified the code against the base tree: signatures (get_anon_pages, filter_memory_ranges_by_pagemap_anon) are unchanged for callers in memory_manager.rs and soft_dirty.rs, dependencies (once_cell, libc, log) are already present in the vmm crate, the mod benchmark/mod kernel_release wiring compiles, and the soft-dirty test now correctly skips when /proc/kpageflags cannot be opened. The behavior change for the intended MAP_PRIVATE fast-restore path is correct.

The main substantive concern is one correctness edge in the equivalence claim (below).

---

Finding 1 (medium): the bit-61 predicate is not exactly equivalent to KPF_ANON — it can under-save when a CoW page has page_mapcount > 1 (KSM)

PM_FILE is set by the kernel when page_mapcount(page) != 1 || page_is_file_cache(page) (fs/proc/task_mmu.c). For the normal fast-restore case — single process, private CoW pages with mapcount 1 — present && !PM_FILE ≡ present && KPF_ANON, so the equivalence claim holds. But the two diverge for anonymous pages that have more than one PTE reference:

• A KSM-merged anonymous page has mapcount > 1 → PM_FILE = 1, while KPF_ANON = 1. The old kpageflags path saved it; the new bit-61 path skips it → under-save → silent snapshot corruption, which is exactly the failure mode the PR says it is most concerned about.
• Cube explicitly supports mergeable=on (guest RAM gets madvise(MADV_MERGEABLE) in allocate_address_space/create_userspace_mapping, documented in docs/memory.md), and fast-restored RAM is precisely a MAP_PRIVATE baseline mapping whose CoW'd pages are mergeable. On a host running ksmd, a Guest-written page identical to another VM's page can be merged; at snapshot time it reads as present && PM_FILE and is dropped from the incremental.

Pagemap alone cannot distinguish a KSM page from a file page (both have PM_FILE = 1), so the kernel-version gate cannot protect against this — KSM state is a runtime property, not a version property. Recommendation: gate the bit-61 path on !config.mergeable (or MADV_UNMERGEABLE the RAM before the scan), or at minimum document that bit-61 incremental snapshots require KSM-mergeable guest RAM to be disabled. (An inline comment is posted at pagemap_entry_is_cow_anon.)

Finding 2 (low): the version gate is a conservative heuristic — worth documenting its limits

KernelRelease::parse takes only the leading major.minor.patch, so 6.6.44-rc1-style strings are treated as final 6.6.44, and any vendor kernel that reports ≥ 6.6.44 / ≥ 6.11 without carrying the PM_FILE backport is gated to bit 61. Both directions are safe — a wrongly-enabled kernel only over-saves file huge pages (snapshot stays valid, just larger), and a wrongly-disabled kernel falls back to kpageflags (slower, unchanged behavior) — but the "within 1%" data-equivalence claim only holds when the gate is right. A short note in the module doc that the gate means "≥ version, assuming the vendor ships the upstream stable backport" would help future readers.

Finding 3 (low, test robustness): the fixture tests compare two live scans of the same mapping

test_scan_kpageflags_anon_cap_or_matches_bit61 and the benchmark run the bit-61 and kpageflags scans back-to-back on the same live mapping; a page evicted/swapped between the two scans could in principle flake the "no under-save" assertion under memory pressure. The 8-page fixture makes this unlikely, and the benchmark is #[ignore]-d, so this is a minor note rather than a blocker. Also, the unit test's under count only checks the bit61 && !kpf direction; the reverse (bit-61 under-saving) is covered indirectly by assert_expected_bitmap, which is fine.

---

Positive notes

• Clean separation of the two scan paths (scan_pagemap_cow_anon / scan_kpageflags_anon) with a lazy, once-only path decision and a single info! line for runtime verification.
• swapped_pages in PagemapAnonStats is now actually populated (it was always 0 before).
• No CAP_SYS_ADMIN requirement on the new path is a real improvement for unprivileged snapshotting.
• The kpageflags path preserves the old PFN-zero → NoCapSysAdmin behavior, and the soft-dirty test now degrades gracefully when /proc/kpageflags is unreadable.

Questions

1. Are incremental snapshots ever produced for VMs configured with mergeable=on? If so, please address Finding 1.
2. Could Guest RAM ever be mapped at two host addresses in the VMM process (which would also set mapcount > 1 and trip Finding 1)? A quick audit would settle this.

[fslongjin]

On the review findings:

1. tmpfs fixture / full-dump on a tmpfs baseline — false positive. PM_FILE is !PageAnon in fs/proc/task_mmu.c, not !PageSwapBacked. Read-only tmpfs file pages have bit 61 set (verified on Linux 7.0 /dev/shm).

2. Shared-anon under-save — false positive. Shared-anon/shmem pages have bit 61 set; KPF_ANON is also clear (PageAnon in fs/proc/page.c). Old and new paths both skip them. Incremental snapshots are file-backed MAP_PRIVATE anyway.

3. Version-gate heuristic / seccomp on /proc/sys/kernel/osrelease — the gate already falls back to kpageflags on parse failure (over-save or slower, never under-save). The seccomp part is a false positive: the VMM filter allows open/openat/read, so osrelease is readable (that is how we got path=bit61 on PVM). A blocked uname(2) would SIGSYS, not silently keep the old path; the primary path does not call uname(2).

4. #[ignore] referencing a script not in this PR — agreed. Will drop the script reference and point at --ignored plus CUBE_PAGEMAP_BENCH_MIB / CUBE_PAGEMAP_BENCH_ITERS instead. Not adding the script to this PR.

[cubesandboxbot]

The kernel sets PM_FILE (bit 61) when page_mapcount(page) != 1 || page_is_file_cache(page) (fs/proc/task_mmu.c). For the fast-restore case — a private CoW page with mapcount 1 — present && !PM_FILE is equivalent to the old present && KPF_ANON. But they diverge when an anonymous page has more than one PTE reference:

• A KSM-merged anonymous page has mapcount > 1 → PM_FILE = 1 while KPF_ANON = 1. The old kpageflags path saved it; this predicate skips it → under-save → silent snapshot corruption.
• Cube supports mergeable=on (MADV_MERGEABLE on guest RAM via create_userspace_mapping, docs/memory.md), and fast-restored RAM is a MAP_PRIVATE baseline mapping whose CoW pages are mergeable — so on a host running ksmd, a Guest-written page can be merged and then dropped here.

Pagemap alone can't distinguish a KSM page from a file page (both have PM_FILE = 1), so the version gate can't protect against this. Consider also gating the bit-61 path on !config.mergeable (or breaking KSM before the scan), or documenting that bit-61 snapshots require non-mergeable guest RAM.

[cubesandboxbot]

KernelRelease::parse takes only the leading major.minor.patch, so a string like 6.6.44-rc1 is treated as final 6.6.44, and any vendor kernel that reports ≥ 6.6.44 / ≥ 6.11 without carrying the PM_FILE backport is also gated to bit 61. Both directions are safe — a wrongly-enabled kernel only over-saves file huge pages (valid but larger snapshot), a wrongly-disabled one falls back to kpageflags — but the PR's "writes the same amount of data (within 1%)" claim assumes the gate is accurate. A one-line note that the gate means "≥ this version, assuming the vendor ships the upstream stable backport" would help future readers.

[lisongqian]

Can we add some integration cases to cover the two methods of obtaining anon pages? @fslongjin