Benchmark HF optimum-executorch

[guangy10]

Benchmark LLMs from optimum-executorch. With all the work recently happening in optimum-executorch, we are able to boost the out-of-the-box performance. Putting these models on benchmark infra to gather perf numbers and understand the remaining perf gaps between the in-house generated model via export_llama.

We are able to do apple-to-apple comparison for CPU backend by introducing quant, custom SPDA, custom KV cache to native Hugging Face models in optimum-executorch: hf_xnnpack_custom_spda_kv_cache_8da4w represents the recipe used by optimum-et, et_xnnpack_custom_spda_kv_cache_8da4w is the counterpart for etLLM.

Here are the benchmark jobs in our infra:

• Android: https://github.com/pytorch/executorch/actions/runs/15597347625
• Apple: https://github.com/pytorch/executorch/actions/runs/15597340954

Fixed in benchmark apps (Android and iOS) to support using HF's tokenizers

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/11450

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures

As of commit b25c0d2 with merge base cbd3874 ():

NEW FAILURES - The following jobs have failed:

• apple-perf / benchmark-on-device (google/gemma-3-1b-it, hf_xnnpack_custom_spda_kv_cache_8da4w, apple_iphone_15... / mobile-job (ios) (gh)
  Final attempt failed. Child_process exited with error code 1
• pull / unittest / linux / linux-job (gh)
  .ci/scripts/tests/test_gather_benchmark_configs.py::TestGatehrBenchmarkConfigs::test_generate_compatible_configs_quantized_llama_model
• pull / unittest-editable / linux / linux-job (gh)
  .ci/scripts/tests/test_gather_benchmark_configs.py::TestGatehrBenchmarkConfigs::test_generate_compatible_configs_quantized_llama_model
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command docker exec -t 51674ab09dbf95746970c5d898d0ac67d7880ab0255cf21de173da8c17fd3a1c /exec failed with exit code 1
• trunk / unittest-release / linux / linux-job (gh)
  .ci/scripts/tests/test_gather_benchmark_configs.py::TestGatehrBenchmarkConfigs::test_generate_compatible_configs_quantized_llama_model

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[guangy10]

@huydhn In the apple's workflow, though I have specified the python version to be "3.11", it still install the python "3.13". Then when trying to pip install executorch, it ended with no package found. That's because we only publish with python 3.10, 3.11, and 3.12. https://github.com/pytorch/executorch/actions/runs/15500604843/job/43647388676#step:9:13372

Okay, it turns out that I need to run install with ${CONDA_RUN}

[guangy10]

@guangy10 for the reported 5x higher numbers compared to etLLM, could this be comparing the results for the HF pte running non-LLM benchmark tests vs the etLLM pte. running LLM benchmark test? Just a guess.

Now I can TPS reported from the Android app. I manually pull the results and it turns seems the TPS is almost same between the etLLM generated Qwen3 and the optimum-et generated Qwen3:

[guangy10]

For some reason doesn't see the results from the hf recipe got uploaded here. Upon manual checking the results, same perf on iOS devices as well.

[guangy10]

One nit that will follow up in a separate PR is that we should allow bypass overriding installed et and deps when installing optimum-executorch. This is because in optimum-executorch the stack is running in dev mode with pinned nightly instead of source for stability, however, in ET, we would want to benchmark continuously to catch issues on trunk, so it should be using ET from source instead. I will need to address this discrepancy in both optimum-executorch and executorch