Repository navigation
Description
Summary
The pre-quantized w4a16 asset for intern3_5_vl_2b (InternVL3.5-2B) appears to ship at least one tensor that was left in FP32. All 9 graphs compile successfully, but part 1 (layers 0–8) and the vision encoder fail at the device compose stage on the Dragonwing RB3 Gen 2 Vision Kit (QCS6490). Parts 2–4 compose, link, and profile normally on the same device with the same toolchain, so the failure is localized to the part 1 / vision asset, not the export path or device support.
Environment
- qai-hub-models 0.66.0
- Device: Dragonwing RB3 Gen 2 Vision Kit (QCS6490)
- Precision: w4a16 (
DEFAULT_W4A16, asset described asfp16mix), context length 512, prompt chunk 128 - Path: manual QAI Hub API chain (compile each split part to QNN DLC with input shapes pinned, then link / profile per graph). The same chain produced complete, published measurements for Qwen3-0.6B and Llama 3.2 1B on this device.
What happens
All 9 compiles succeed. Linking part 1 fails server-side with:
Tensor '/model/model/layers.2/mlp/down_proj/Conv_bias' has a floating-point type which is not supported by the targeted device
Profiling the part 1 graphs and the vision encoder graph fails at compose time with:
QnnModel_composeGraphsFromDlc: MODEL_GRAPH_ERROR
What we ruled out
- Compile options: part 1 was recompiled with
-O1instead of the default optimization level (compile jobjp1m6kvnp→ target modelmq3e65zdl, profile jobjp1m6rd8p) — identical compose failure. - Link granularity: whole-model link and per-part links were both tried. Part 3 links successfully (context binary
mqkkv24rw); parts 1 and 2 fail (part 2's link failure is expected — it depends on part 1 — but part 1 fails on its own). - Device/toolchain generally: parts 2–4 profile fine on the real device (e.g. prompt graphs 68,980 µs / 69,011 µs / 36,122 µs per 128 tokens; decode 20,224 µs / 20,300 µs / 22,291 µs per token).
Job IDs (AI Hub)
- Compiles (all SUCCESS):
j5qdvwreg,jgz0nwr45,jgnwkx9vg,jp1m63qkp,j560wvr75,jgdm208lg,jp2zejemp,jpyymn87p, visionjp42384vg - Links: whole-model
jglql788pFAILED; part 1jg9qd8qlpFAILED; part 2jglql7q8pFAILED; part 3 SUCCESS →mqkkv24rw - Failing profiles: part 1
jg9qd87wp,jgkk8dwwg,jpvn7evzg,jg9qdkrwp,jp1m67x8p,j560wwqn5,jpenyzovp,jp1m6jdnp; visionjgj8qyee5,jpyym2w7p,j5ml8lw9g - Successful profiles:
jp42392vg,jp1m66z8p,jgj8qql15(prompt);jp423ymqg,jp3r6z6xp,jp0x6ov0p(decode)
Suspected cause
The link error names an FP32 tensor inside the w4a16 asset: a Conv_bias in the MLP of layer 2 that appears not to have been quantized when the fp16mix asset was produced. Since the same tensor class in parts 2–4 causes no problem, this looks like an asset-generation gap specific to part 1 (and, judging by the compose failures, the vision encoder as well), rather than a user-side configuration issue.
Full measurement record, including the parts that do run: https://github.com/anguoyang/qcs-bench and https://huggingface.co/anguoyang/internvl3.5-2b-qcs6490
Happy to provide the split ONNX files or run a patched asset if that helps verify a fix.
Activity
shreyajn commentedon Nov 16, 2026
We don't support this model on RB Gen 2. We bumped up a layer to be in floating point to improve the output quality of the model. Since RB Gen2 only supports integer math, we never had it as a supported device for this model.