[WIP] Support co-locate training and inference (#81)#92

Draft

zhubohao911 wants to merge 61 commits into

lightseekorg:mainfrom

zhubohao911:feature/colocate-training-inference

Commits on May 7, 2026

WIP: Support co-locate training and inference (#81)
zhubohao911
committed

Commits on May 11, 2026

init doc
zhubohao911
committed

Commits on May 12, 2026

Merge branch 'lightseekorg:main' into feature/colocate-training-inference
zhubohao911
committed

Commits on May 13, 2026

feat(colocate): phases 0-2 of MPS-strategy colocate training
zhubohao911
committed
Phase 3: NCCL P2P data plane (dummy tensors)
zhubohao911
committed
Phase 4: NCCL hidden-state connector + multi-tensor data plane
zhubohao911
committed
Phases 5 + 6: controller trim, init-order fence, MPS hygiene
zhubohao911
committed
Phase 7: numeric parity & convergence test skeletons
zhubohao911
committed
Phase 8: colocate usage docs + Qwen3-8B example
zhubohao911
committed
Phase 4: ship the colocate sglang patch + wire Modal image to apply it
zhubohao911
committed
Modal: layer colocate.patch on top of overlay + assert patch surface in probe
zhubohao911
committed
colocate: implement Phase-5 sync training loop + driver-side union spec
zhubohao911
committed
colocate: bring up MPS pre-Ray and propagate pipe env to controller
zhubohao911
committed
colocate: dump MPS daemon log on CUDA error 805
zhubohao911
committed
tests/colocate/one_step: dump nvidia-mps daemon log on failure
zhubohao911
committed
colocate: detect 'MPS not supported' and fall back to fractional GPU sharing
zhubohao911
committed
colocate: probe MPS via real CUDA client subprocess (cuInit/cuDeviceGetCount)
zhubohao911
committed
colocate: switch union world to lazy NCCL init to tolerate slow engine startup
zhubohao911
committed
tests/colocate/one_step: stream subprocess output to log file (so we can read it on timeout)
zhubohao911
committed
tests/colocate/one_step: bump timeout to 30min for cold HF cache
zhubohao911
committed
docs/colocate: log the Modal MPS / eager-NCCL discoveries from Phase-4 debugging
zhubohao911
committed
tests/colocate: skip Phase-4+ tests when MPS server can't start
zhubohao911
committed
tests/colocate/test_placement: handle None handle from MPS-fallback path
zhubohao911
committed
docs/colocate: final Phase 1-7 verification matrix on Modal sandbox
zhubohao911
committed
colocate: add cheap-host MPS smoke (1×GPU, Qwen3-0.6B) + fix mps-helper unit tests
zhubohao911
committed
docs/colocate: cheap-host test plan + --full runner mode for agent handoff
zhubohao911
committed
scripts/colocate: harden runner with real MPS pre-flight + auto-report
zhubohao911
committed
mooncake/store: lazy-import so colocate doesn't need libibverbs/libnuma
zhubohao911
committed
docs/colocate: add bilingual GPU/CUDA knowledge supplement
zhubohao911
committed
utils/logging: configure 'torchspec' namespace so submodule INFO surfaces
zhubohao911
committed

Commits on May 14, 2026

Commits on May 15, 2026

colocate: dp_attention rank offset must be the engine's own union rank

authored and
Xing Han
committed
colocate: scope set_model_state_dict broadcast to the trainer mesh

authored and
Xing Han
committed
docs/colocate: Vast session #4 — 4xH100 --full suite green (runs #1-#7)

authored and
Xing Han
committed
docs/colocate: expand session #4 — debug methodology + next steps

authored and
Xing Han
committed
colocate: scope dcp.save / dcp.load to the trainer-only group

authored and
Xing Han
committed
docs/colocate: session #5 — verification re-run + single_rank audit + dcp fix

authored and
Xing Han
committed