Skip to content

Qualcomm: cap inf replacement value to fix 16a16w accuracy regression #20471

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
psiddh wants to merge 4 commits into
base: main
Choose a base branch
from
+7 −2
Open

Qualcomm: cap inf replacement value to fix 16a16w accuracy regression #20471

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
2333727
Qualcomm: cap inf replacement value to fix 16a16w accuracy regression
psiddh Jun 24, 2026
45a7414
Merge branch 'main' into fix-qnn-16a16w-inf-replacement
psiddh Jun 24, 2026
87c4feb
Merge branch 'main' into fix-qnn-16a16w-inf-replacement
psiddh Jun 25, 2026
8875528
Bump test-llama-runner-qnn-linux to linux.8xlarge.memory
psiddh Jun 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/pull.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -856,7 +856,7 @@ jobs:
mode: [qnn]
fail-fast: false
with:
runner: linux.2xlarge
runner: linux.8xlarge.memory
docker-image: ci-image:executorch-ubuntu-22.04-qnn-sdk
submodules: 'recursive'
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
Expand Down
7 changes: 6 additions & 1 deletion backends/qualcomm/quantizer/quantizer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -416,7 +416,12 @@ def _get_quant_range(self, node):
if quant_info.output_qspec.quant_min is None
else quant_info.output_qspec.quant_min
)
return quant_range
# Cap the inf stand-in so it does not dominate the tensor's
# dynamic range. For >8-bit activations the full range (e.g.
# 65535 for uint16) would blow up the attention-mask quant scale
# and wreck accuracy; 255 keeps a reasonable scale for
# Llama-style attention masks.
return min(quant_range, 255)

@winskuo-quic winskuo-quic Jun 24, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @psiddh,
Thanks for identifying the issue.
Could you please let me do a quick test on the CI testing model and validate the accuracy issue?
Ideally, we don't want hard coded numbers, and that is why we replaced them with quant range in the PR: #19660

@psiddh psiddh Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, please go ahead and validate on CI, happy to wait for your results.

On the hardcoded 255: agree we should avoid magic numbers. perhaps a cleaner version would be to derive the cap from the int8 range rather than hardcoding: ..

return min(quant_range, torch.iinfo(torch.int8).max - torch.iinfo(torch.int8).min)

@psiddh psiddh Jun 24, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@winskuo-quic Any updates ? We would like to land this asap as this is stalling viable/strict branch from moving forward.

@psiddh psiddh Jun 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@winskuo-quic The CI runner for test-llama-runner-qnn-linux (qnn_16a16w) is OOM-killing the export step, exit code 137 (SIGKILL), The job is never getting far enough , the Python export process gets killed before it finishes , it looks like.

@winskuo-quic winskuo-quic Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @psiddh,
Thanks for sharing all the test results.
I am also encountering the same issue.
I noticed it is killed during qnn_preprocess. I believe there's a chance OOM happened inside QNN SDK, which is weird as this is a small model.
I am still working on finding a solution to resolve the issue.
If you happened to find a work around, please feel free to merge first to unblock the CI error.
Thanks again for all the help.

@psiddh psiddh Jun 25, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is the quick work-around : #20511 , This will give us time to investigate OOM issue

@winskuo-quic winskuo-quic Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the workaround. I will try to find re-enable 16a16w test.


def _get_candidates_with_infinity_args(self, graph_module: GraphModule):
binary_op_sources = [
Expand Down
Loading