convert --quantize routes text-only checkpoints of vision families to the vision backend

[mzau]

Summary

mlxk convert … --quantize dispatches on model_type alone. A checkpoint whose model_type is on
the verified vision list but which carries no vision tower is still sent to the vision backend,
where it fails with an error that does not point at the cause.

Text-only derivatives of multimodal families are the affected class — e.g. TranslateGemma, which is
built on the Gemma-3 architecture and keeps model_type: gemma3 while shipping no vision config and
no vision weights.

Repro

mlxk clone mlx-community/translategemma-27b-it-8bit translategemma-27b-it-8bit
mlxk convert translategemma-27b-it-8bit out-4bit --quantize 4

[INFO] Loading
convert: Error: Quantization failed: VisionConfig.__init__() missing 6 required positional
arguments: 'model_type', 'num_hidden_layers', 'hidden_size', 'intermediate_size',
'num_attention_heads', and 'patch_size'

Reproduces identically on the bf16 variant, so it is driven by the config, not by the weights.

Cause

classify_convert_target() (mlxk2/core/capabilities.py) returns "vision" for any model_type
in VISION_QUANTIZE_TYPES (gemma3, gemma4, idefics2, idefics3, llava, mimo, mistral3,
mllama, pixtral, qwen2_vl) without checking whether this particular checkpoint has a vision
tower.

The checkpoint in the repro declares architectures: ["Gemma3ForConditionalGeneration"],
model_type: gemma3 and the full image-token wiring (image_token_index, boi/eoi_token_index,
mm_tokens_per_image), but has vision_config: null and 0 of 1678 tensors under
vision_tower.* / multi_modal_projector.*. The vision backend then constructs VisionConfig
from nothing and raises.

For contrast, checkpoints that genuinely are multimodal classify correctly:
gemma-4-31b-bf16 carries 355 vision tensors, pixtral-12b-bf16 222.

The function's docstring covers the neighbouring case — "routed to the vision backend even when it
also carries vision_config" — but not a model_type on the list whose vision_config is absent
or null.

Suggested predicate

Deciding on model_type and the actual presence of a vision tower separates the three cases:

model_type whitelisted	vision tower in the safetensors index	dispatch
yes	present	vision backend (unchanged)
yes	absent, and vision_config absent/null	text backend
yes	present, but vision_config absent/null	reject with a message naming the inconsistency

The third row is the stale-conversion case and should not be silently routed to the text backend:
that backend drops vision weights on save. Reading model.safetensors.index.json is enough to tell
the rows apart; no weights need to be loaded.

Relation to upstream

Blaizzy/mlx-vlm#1776 reports the same precondition (vision_config: null
on a multimodal model_type) from a different crash site — utils.py skip_vision lookup, and only
for already-quantized checkpoints. The crash reported here happens earlier, during config
construction, and also on unquantized input.

That issue is not a blocker for this one: routing such a checkpoint to the text backend avoids
mlx-vlm entirely, and mlx_lm/models/gemma3.py handles this shape natively — its ModelArgs has no
vision_config field at all, and its sanitize() expects the language_model. prefix that these
checkpoints use.

Environment

mlx-knife 2.0.7 · macOS/Apple Silicon · Python 3.10