🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/16506

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 6 New Failures

As of commit 7b64dff with merge base 7492d0d ():

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Hi @cccclai,
This PR is to:

1. Enable SQNR evaluation for Static LLM models. Please refer to test_qnn_delegate.py for the sqnr scores for each model.
2. Added SQNR eval in mainline CI. It only takes 30 min to lower and eval sqnr for smollm2_135m on x86 emulator, so we can put this under pull.yml, making the CI more robust.
3. Refactored the inference flow for llama.py, making it easier for users to support new eval methods in the future.
   Please have a look.
   Thanks.

@YIWENX14 can you take a look at the PR and see if it meets the need?

@winskuo-quic Could you clarify whether the comparison is being made using nn.Module rather than the exported model? IIUC nn.Module is the floating point model before quantization —is that the case?

@winskuo-quic Could you clarify whether the comparison is being made using nn.Module rather than the exported model? IIUC nn.Module is the floating point model before quantization —is that the case?

Hi @YIWENX14,
Your understanding is correct. We are using nn.Module FP32 as the golden to compare against QNN.

@winskuo-quic Could you clarify whether the comparison is being made using nn.Module rather than the exported model? IIUC nn.Module is the floating point model before quantization —is that the case?

Hi @YIWENX14, Your understanding is correct. We are using nn.Module FP32 as the golden to compare against QNN.

Thank you for clarifying. As a follow-up, is it possible to obtain the exported program (quantized) results? Essentially, the process involves two steps: fp32 → quantized → delegated. It would be helpful if we could get the SNR (quantized, delegated) in addition to the end-to-end SNR, so we can better understand the delegation gap.

@winskuo-quic Could you clarify whether the comparison is being made using nn.Module rather than the exported model? IIUC nn.Module is the floating point model before quantization —is that the case?

Hi @YIWENX14, Your understanding is correct. We are using nn.Module FP32 as the golden to compare against QNN.

Thank you for clarifying. As a follow-up, is it possible to obtain the exported program (quantized) results? Essentially, the process involves two steps: fp32 → quantized → delegated. It would be helpful if we could get the SNR (quantized, delegated) in addition to the end-to-end SNR, so we can better understand the delegation gap.

Sure, that is possible. However, please notice that the one of the reasons I did not use quantized model is that user can quickly compare nn.Module's result with a pre-generated delegated model's result quickly, since it takes only a couple of seconds to retrieve a nn.Module graph. On the other hand, if a user wants to compare quantized cpu result with pre-generated delegated model's result, they will need to perform export -> prepare_pt2e -> calibration -> convert_pt2e every time before we start comparing, which could take some time.
If this seems fine to you, I can have the feature supported.
If you have some recommendations on the official flow to save quantized model (so we don't have to quant every time when comparing sqnr), it will also be appreciated.

Thanks for the explanation. Yeah there's a way to save and load the quantized exported program (https://docs.pytorch.org/docs/stable/export.html#serialization). We can do:

torch.export.save(exported_program, 'exported_program.pt2')
saved_exported_program = torch.export.load('exported_program.pt2')
res = saved_exported_program.module()(*example_inputs)

This can be done as a follow up, which can help to better understand the gap between eager and delegation. Thank you so much for the support!

Thanks for the explanation. Yeah there's a way to save and load the quantized exported program (https://docs.pytorch.org/docs/stable/export.html#serialization). We can do:

torch.export.save(exported_program, 'exported_program.pt2')
saved_exported_program = torch.export.load('exported_program.pt2')
res = saved_exported_program.module()(*example_inputs)

This can be done as a follow up, which can help to better understand the gap between eager and delegation. Thank you so much for the support!

Thanks for the suggestion and the help. I have pushed a new commit that should support:

• FP32 nn.Module VS QNN
• FP32 nn.Module VS CPU QDQ
• CPU QDQ VS QNN

Please have a look. Thanks

Thank you for adding the detailed comparison on quantized model. The changes LGTM!