Add XNNPACK, MobileNetV2, MobileBERT, Llama, ResNet18 to RISC-V testing matrix

[luhenry]

Summary

This is the continuation of #19399 and #19521, to deliver on Phase 2 of #18991

Test plan

This code is exclusively test code. Everything works out of the box, and CI will validate.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19617

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• Run pull request jobs on OSDC runners in shadow mode

❌ 1 New Failure, 38 Unrelated Failures, 11 Unclassified Failures

As of commit 4b5e42e with merge base 7355d7b ():

NEW FAILURE - The following job has failed:

• pull / unittest-arm-backend-with-no-deps (test_pytest_ops_tosa) / linux-job (gh)
  RuntimeError: Command docker exec -t f4357ea903270f9c9ecceb31c0fa001a5dbf4b22c910e4f5c001234d249eca9d /exec failed with exit code 1

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Build Windows Wheels / pytorch/executorch / build-wheel-py3_10-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• Build Windows Wheels / pytorch/executorch / upload / upload-wheel-py3_10-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found for name: pytorch_executorch__3.10_cpu_x64
• RISC-V / test-riscv (add, true, false) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• RISC-V / test-riscv (llama2, true, false) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 09d2c76fc67126e24e93855517295ae0f8bbbae8b81e2560225c1e28afc4606b /exec failed with exit code 1
• RISC-V / test-riscv (llama2, true, true) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]The operation was canceled.
• RISC-V / test-riscv (mobilebert, true, false) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• RISC-V / test-riscv (mobilebert, true, true) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  ##[error]The operation was canceled.
• RISC-V / test-riscv (mv2, true, false) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  RuntimeError: Command docker exec -t 5af2926c2a680a408bc7d362487dcd5cc99f0ba125c6b7f6f8b1bc744784f4aa /exec failed with exit code 1
• RISC-V / test-riscv (mv2, true, true) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• RISC-V / test-riscv (resnet18, true, false) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
• RISC-V / test-riscv (resnet18, true, true) / run / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• pull / test-qnn-delegate-linux / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• pull / test-qnn-python-imports-linux / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, models) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• trunk / test-qnn-model (fp32, conv_former) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• trunk / test-qnn-model (fp32, ic3) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• trunk / test-qnn-model (fp32, mv3) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• trunk / test-qnn-model (fp32, vit) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• trunk / test-qnn-model (fp32, w2l) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• trunk / test-qnn-optimum-model (fp32, albert) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• trunk / test-qnn-optimum-model (fp32, bert) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• trunk / test-qnn-optimum-model (fp32, distilbert) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• trunk / test-qnn-optimum-model (fp32, mobilevit_v1) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• trunk / test-qnn-optimum-model (fp32, pvt) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• trunk / test-qnn-optimum-model (fp32, roberta) / linux-job (gh) (similar failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh) (trunk failure)
• pull / test-qnn-models-linux (dl3) / linux-job (gh) (trunk failure)
• pull / test-qnn-models-linux (mv2) / linux-job (gh) (trunk failure)
• pull / test-qnn-models-linux (mv3) / linux-job (gh) (trunk failure)
• pull / test-qnn-passes-linux / linux-job (gh) (trunk failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, operators) / linux-job (gh) (trunk failure)
• pull / test-sqnr-static-llm-qnn-linux (smollm2_135m) / linux-job (gh) (trunk failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• pull / test-static-llama-qnn-linux (stories_110m) / linux-job (gh) (trunk failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• pull / test-static-llama-qnn-linux (stories_260k_bc) / linux-job (gh) (trunk failure)
  /pytorch/executorch/examples/qualcomm/oss_scripts/llama/qnn_multimodal_runner.cpp:185:7: error: type 'float' cannot be narrowed to 'bool' in initializer list [-Wc++11-narrowing]
• Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh) (trunk failure)
• Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh) (trunk failure)
• trunk / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh) (trunk failure)
• trunk / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh) (trunk failure)
• trunk / test-qnn-model (fp32, dl3) / linux-job (gh) (trunk failure)
• trunk / test-qnn-model (fp32, ic4) / linux-job (gh) (trunk failure)
• trunk / test-qnn-model (fp32, mb) / linux-job (gh) (trunk failure)
• trunk / test-qnn-model (fp32, mv2) / linux-job (gh) (trunk failure)
• trunk / test-qnn-optimum-model (fp32, cvt) / linux-job (gh) (trunk failure)
• trunk / test-qnn-optimum-model (fp32, dit) / linux-job (gh) (trunk failure)
• trunk / test-qnn-optimum-model (fp32, efficientnet) / linux-job (gh) (trunk failure)
• trunk / test-qnn-optimum-model (fp32, focalnet) / linux-job (gh) (trunk failure)
• trunk / test-qnn-optimum-model (fp32, mobilevit_v2) / linux-job (gh) (trunk failure)
• trunk / test-qnn-optimum-model (fp32, swin) / linux-job (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[luhenry]

cc @rascani @mergennachin, and @GregoryComer @digantdesai for XNNPACK

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[rascani]

The boolean conditionals need an "or empty string."

${{ inputs.xnnpack && '--xnnpack' || '' }} ${{ inputs.quantize && '--quantize' || '' }}

Alternatively, the script could take --xnnpack=true/false --quantize=true/false

[rascani]

We typically also set fail-fast: false

[rascani]

gcc-version: ${{ matrix.xnnpack && 14 || 11 }}
docker-image: ${{ matrix.xnnpack && 'ci-image:executorch-ubuntu-24.04-gcc14' || 'ci-image:executorch-ubuntu-22.04-gcc11' }}