Skip to content

feat(models): add Phi-4 Multimodal text-only adapter - #521

Open
zitai-wang wants to merge 5 commits into
inclusionAI:mainfrom
zitai-wang:feat/phi4-text-adapter
Open

feat(models): add Phi-4 Multimodal text-only adapter#521
zitai-wang wants to merge 5 commits into
inclusionAI:mainfrom
zitai-wang:feat/phi4-text-adapter

Conversation

@zitai-wang

Copy link
Copy Markdown

Summary

  • Add a text-only adapter for Microsoft Phi-4-Multimodal-Instruct using AReno native layers and runtime.
  • Implement Phi partial LongRoPE, true tied input/output embeddings, text generation, and text-only save/reload.
  • Load the official base-language checkpoint with explicit vision, speech, and LoRA classification.
  • Add TP-aware packed-section mapping for fused QKV and gate/up projections, validated with TP1, TP2, and TP4.

Validation

  • Phi-4 focused CPU tests: 39 passed.
  • Full repository suite with CUDA hidden: 643 passed, 15 skipped, 1 failed.
    • The sole failure is tests/test_mlx_training_cpu.py::test_mlx_decode_progress_matches_cuda_log_shape.
    • This MLX logging failure reproduces unchanged on clean upstream/main at 4cc3d388d17b60b534c57d25f69fdcad393433dc; it is not introduced by this PR.
  • Available CUDA tests on an NVIDIA GeForce RTX 3090: 12 passed.
  • Ruff lint/format, all manual pre-commit hooks, compileall, and git diff --check: passed.
  • Official BF16 checkpoint audit: 194 base-language keys consumed, 0 unknown keys, and 0 missing required base-language keys.
  • HF/AReno parity: exact 16-token greedy and KV-cache token parity; last-token logits cosine similarity 0.999983340331.
  • Tensor parallel parity: exact greedy token parity for TP1, TP2, and TP4.
  • Text-only save/reload: bitwise-equal logits and true weight tying preserved.

Known limitations

  • This PR supports only the text path. Image and audio support will be submitted in follow-up PRs.
  • Cached generation crossing the Phi LongRoPE 4096-token regime boundary currently fails fast by design.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant