Skip to content

intern3_5_vl_2b w4a16 asset: part 1 + vision encoder fail device compose on RB3 Gen 2 (FP32 Conv_bias tensor in fp16mix asset) #369

Closed

Description

Summary

The pre-quantized w4a16 asset for intern3_5_vl_2b (InternVL3.5-2B) appears to ship at least one tensor that was left in FP32. All 9 graphs compile successfully, but part 1 (layers 0–8) and the vision encoder fail at the device compose stage on the Dragonwing RB3 Gen 2 Vision Kit (QCS6490). Parts 2–4 compose, link, and profile normally on the same device with the same toolchain, so the failure is localized to the part 1 / vision asset, not the export path or device support.

Environment

  • qai-hub-models 0.66.0
  • Device: Dragonwing RB3 Gen 2 Vision Kit (QCS6490)
  • Precision: w4a16 (DEFAULT_W4A16, asset described as fp16mix), context length 512, prompt chunk 128
  • Path: manual QAI Hub API chain (compile each split part to QNN DLC with input shapes pinned, then link / profile per graph). The same chain produced complete, published measurements for Qwen3-0.6B and Llama 3.2 1B on this device.

What happens

All 9 compiles succeed. Linking part 1 fails server-side with:

Tensor '/model/model/layers.2/mlp/down_proj/Conv_bias' has a floating-point type which is not supported by the targeted device

Profiling the part 1 graphs and the vision encoder graph fails at compose time with:

QnnModel_composeGraphsFromDlc: MODEL_GRAPH_ERROR

What we ruled out

  • Compile options: part 1 was recompiled with -O1 instead of the default optimization level (compile job jp1m6kvnp → target model mq3e65zdl, profile job jp1m6rd8p) — identical compose failure.
  • Link granularity: whole-model link and per-part links were both tried. Part 3 links successfully (context binary mqkkv24rw); parts 1 and 2 fail (part 2's link failure is expected — it depends on part 1 — but part 1 fails on its own).
  • Device/toolchain generally: parts 2–4 profile fine on the real device (e.g. prompt graphs 68,980 µs / 69,011 µs / 36,122 µs per 128 tokens; decode 20,224 µs / 20,300 µs / 22,291 µs per token).

Job IDs (AI Hub)

  • Compiles (all SUCCESS): j5qdvwreg, jgz0nwr45, jgnwkx9vg, jp1m63qkp, j560wvr75, jgdm208lg, jp2zejemp, jpyymn87p, vision jp42384vg
  • Links: whole-model jglql788p FAILED; part 1 jg9qd8qlp FAILED; part 2 jglql7q8p FAILED; part 3 SUCCESS → mqkkv24rw
  • Failing profiles: part 1 jg9qd87wp, jgkk8dwwg, jpvn7evzg, jg9qdkrwp, jp1m67x8p, j560wwqn5, jpenyzovp, jp1m6jdnp; vision jgj8qyee5, jpyym2w7p, j5ml8lw9g
  • Successful profiles: jp42392vg, jp1m66z8p, jgj8qql15 (prompt); jp423ymqg, jp3r6z6xp, jp0x6ov0p (decode)

Suspected cause

The link error names an FP32 tensor inside the w4a16 asset: a Conv_bias in the MLP of layer 2 that appears not to have been quantized when the fp16mix asset was produced. Since the same tensor class in parts 2–4 causes no problem, this looks like an asset-generation gap specific to part 1 (and, judging by the compose failures, the vision encoder as well), rather than a user-side configuration issue.

Full measurement record, including the parts that do run: https://github.com/anguoyang/qcs-bench and https://huggingface.co/anguoyang/internvl3.5-2b-qcs6490

Happy to provide the split ONNX files or run a patched asset if that helps verify a fix.

Activity

  1. shreyajn commented on Nov 16, 2026

    @shreyajn
    Contributor

    We don't support this model on RB Gen 2. We bumped up a layer to be in floating point to improve the output quality of the model. Since RB Gen2 only supports integer math, we never had it as a supported device for this model.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions