Skip to content

feat(optimizer): complete dynamic AdamW8bit state routing - #543

Merged
xsuler merged 4 commits into
mainfrom
feat/adamw8bit-paper-complete
Sep 2, 2026
Merged

feat(optimizer): complete dynamic AdamW8bit state routing#543
xsuler merged 4 commits into
mainfrom
feat/adamw8bit-paper-complete

Conversation

@xsuler

@xsuler xsuler commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • implement paper-compatible signed/unsigned dynamic 8-bit codebooks in CUDA and MLX
  • keep token-embedding Adam moments in FP32 using explicit role/identity routing, with tied-parameter deduplication and explicit precision overrides
  • support mixed-state checkpointing, DP sharding, CPU/disk offload, clearing, and state-memory metrics without changing pretrained model forward behavior
  • preserve AdamW4bit behavior while sharing the optimizer base

Validation

  • .venv-test/bin/python -m pytest -q tests/test_adamw_8bit_blockwise_cpu.py tests/test_adamw_4bit_cpu.py — 24 passed
  • targeted MLX AdamW8bit tests — 5 passed
  • targeted Ruff checks/formatting, py_compile, and git diff --check

CUDA compilation/runtime and the issue three-seed training/throughput experiments require the project GPU environment and were not run locally.

Per maintainer direction, this intentionally does not support optimizer checkpoints produced by the previous linear AdamW8bit implementation.

Addresses #541

@xsuler
xsuler deployed to push_check September 2, 2026 06:52 — with GitHub Actions Active
@xsuler
xsuler deployed to push_check September 2, 2026 06:58 — with GitHub Actions Active
@xsuler
xsuler deployed to push_check September 2, 2026 07:07 — with GitHub Actions Active
@xsuler
xsuler merged commit 12103ca into main Sep 2, 2026
6 checks passed
@xsuler
xsuler deleted the feat/adamw8bit-paper-complete branch September 3, 2026 03:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant