Benchmark HF optimum-executorch

[guangy10]

Benchmark LLMs from optimum-executorch. With all the work recently happening in optimum-executorch, we are able to boost the out-of-the-box performance. Putting these models on benchmark infra to gather perf numbers and understand the remaining perf gaps between the in-house generated model via export_llama.

We are able to do apple-to-apple comparison for CPU backend by introducing quant, custom SPDA, custom KV cache to native Hugging Face models in optimum-executorch: hf_xnnpack_custom_spda_kv_cache_8da4w represents the recipe used by optimum-et, et_xnnpack_custom_spda_kv_cache_8da4w is the counterpart for etLLM.

Here are the benchmark jobs in our infra:

• Android: https://github.com/pytorch/executorch/actions/runs/15597347625
• Apple: https://github.com/pytorch/executorch/actions/runs/15597340954

Note there may be failures when running optimum-et models on-device due to lack of support HF tokenizers in the benchmark apps. I have to remove packing tokenizer.json from the .zip in order to unblock collecting raw latency on forward() call.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11450

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures, 1 Pending

As of commit b25c0d2 with merge base cbd3874 ():

NEW FAILURES - The following jobs have failed:

• apple-perf (private devices) / apple / upload-benchmark-results (gh)
  Process completed with exit code 1.
• pull / unittest / linux / linux-job (gh)
  .ci/scripts/tests/test_gather_benchmark_configs.py::TestGatehrBenchmarkConfigs::test_generate_compatible_configs_quantized_llama_model
• pull / unittest-editable / linux / linux-job (gh)
  .ci/scripts/tests/test_gather_benchmark_configs.py::TestGatehrBenchmarkConfigs::test_generate_compatible_configs_quantized_llama_model
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command docker exec -t 51674ab09dbf95746970c5d898d0ac67d7880ab0255cf21de173da8c17fd3a1c /exec failed with exit code 1
• trunk / unittest-release / linux / linux-job (gh)
  .ci/scripts/tests/test_gather_benchmark_configs.py::TestGatehrBenchmarkConfigs::test_generate_compatible_configs_quantized_llama_model

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[guangy10]

@huydhn In the apple's workflow, though I have specified the python version to be "3.11", it still install the python "3.13". Then when trying to pip install executorch, it ended with no package found. That's because we only publish with python 3.10, 3.11, and 3.12. https://github.com/pytorch/executorch/actions/runs/15500604843/job/43647388676#step:9:13372

Okay, it turns out that I need to run install with ${CONDA_RUN}

[guangy10]

@guangy10 for the reported 5x higher numbers compared to etLLM, could this be comparing the results for the HF pte running non-LLM benchmark tests vs the etLLM pte. running LLM benchmark test? Just a guess.

Now I can TPS reported from the Android app. I manually pull the results and it turns seems the TPS is almost same between the etLLM generated Qwen3 and the optimum-et generated Qwen3:

[guangy10]

For some reason doesn't see the results from the hf recipe got uploaded here. Upon manual checking the results, same perf on iOS devices as well.