Skip to content

feat(models): add Phi-4 multimodal vision support - #523

Open
zitai-wang wants to merge 7 commits into
inclusionAI:mainfrom
zitai-wang:feat/phi4-vision-adapter
Open

feat(models): add Phi-4 multimodal vision support#523
zitai-wang wants to merge 7 commits into
inclusionAI:mainfrom
zitai-wang:feat/phi4-vision-adapter

Conversation

@zitai-wang

Copy link
Copy Markdown

Summary

Depends on #521.

  • add the native Phi-4-Multimodal SigLIP vision encoder and HD projector
  • merge projected image features using the official Phi image placeholder layout
  • support Vision LoRA across the language backbone
  • preserve per-sequence Vision LoRA state across chunked prefill and KV-cache decode
  • support multi-image, variable-resolution, and mixed text/image batches
  • load and save the vision encoder, projector, separators, and Vision LoRA weights
  • support TP-aware Vision LoRA checkpoint sharding for TP1, TP2, and TP4

Validation

  • Phi vision/checkpoint tests: 30 passed
  • repository CPU suite: 662 passed, 3 skipped; 1 MLX baseline failure reproduced on clean upstream/main
  • CUDA tests: 12 passed
  • projected vision features and merged embeddings match the HF reference exactly
  • 8-token KV-cache greedy generation matches the HF reference
  • TP1/2/4 produce identical greedy tokens; gathered last-token logits remain aligned
  • save/reload consumes all 904 supported language and vision tensors
  • logits are bitwise identical before and after reload (max_abs=0)
  • Ruff, formatting, pre-commit, compileall, and git diff --check pass

Validation used Microsoft Phi-4-Multimodal-Instruct in BF16 on RTX 3090 GPUs. The local snapshot corresponds to ModelScope commit 7641bf905e6965ee54166808d275266371e28343 and Hugging Face commit 93f923e1a7727d1c4f446756212d9d3e8fcc5d81.

The MLX failure is test_mlx_decode_progress_matches_cuda_log_shape: the log is emitted to stderr but is not captured by caplog. The same failure reproduces at clean upstream/main commit 4cc3d38.

Known limitations

  • this PR supports text and image paths only; audio support is deferred to a follow-up PR
  • cached generation crossing the Phi LongRoPE 4096-token boundary continues to fail fast
  • small BF16 hidden-state differences versus HF can accumulate because AReno uses its native attention/runtime implementation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant