Skip to content

Commit 1b7cbdd

Browse files
SS-JIApytorchbot
authored andcommitted
[ET-VK][ops] Fuse a trailing relu into q8ta_pixel_shuffle
## Ulterior Motive Quantized models that upsample with `PixelShuffle` followed by `ReLU` lower to `dequantize -> pixel_shuffle -> relu -> quantize`, four fp32 dispatches per upsample. This commit fuses that chain into the existing int8 `q8ta_pixel_shuffle` kernel. ## Rationale **What**: A new overload `et_vk.q8ta_pixel_shuffle.with_activation(..., str activation)`, which the quantized pixel shuffle pattern emits when an activation sits between `pixel_shuffle` and `quantize`. `relu` is the only supported activation. **Why**: The pattern only matched `dequantize -> pixel_shuffle -> quantize`. With a `relu` in between, nothing fused and the upsample ran in fp32. ## Details - Pattern (`quantized_pixel_shuffle.py`) - **Before**: `dequantize -> [clone] -> pixel_shuffle -> [clone] -> quantize` fuses into `q8ta_pixel_shuffle.default`. - **After**: an optional activation (`relu`, with optional clones after it) is also matched, and the match emits `q8ta_pixel_shuffle.with_activation` with the activation name. Supported activations are listed in one table, `_ACTIVATIONS`. - Runtime: both overloads register the same dispatch and shader, so no shader variant is added. - For a positive output scale, `quantize(relu(x)) == max(quantize(x), zp_out)`. The kernel therefore applies relu by raising the lower clamp bound of the requantized value from -128 to `zp_out`, passed as a new push constant `q_min`. - This is bit-exact with the unfused chain, because `q8ta_quantize` uses the same `round(x * inv_scale) + zp` arithmetic. - With an activation, the pure byte-shuffle (`passthrough`) path is disabled. - `add_q8ta_pixel_shuffle_node` takes an `ActivationType`, reusing the enum and parser from the q8ta conv2d ops. - Tests - `etvk_ops_bench`: the reference and throughput functions are registered for the new overload. `correctness.json` adds one case with 8 argument sets: 4 shapes (including odd sizes and batch 2), each with requantizing and with matching quantization params, across the 3 x 3 input/output layout sweep. `performance.json` adds one case with 3 upsample shapes. - `test_vulkan_passes.py`: `test_fuse_quantized_pixel_shuffle_relu` checks the fused op, its arguments, that no quant/dequant/relu nodes remain, and that the fused graph matches the eager module exactly. Authored with Claude Code. Differential Revision: [D133330257](https://our.internmc.facebook.com/intern/diff/D133330257/) **NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D133330257/)! ghstack-source-id: 478906449 Pull-Request: #25316
1 parent abfcd96 commit 1b7cbdd

11 files changed

Lines changed: 304 additions & 27 deletions

File tree

‎backends/vulkan/custom_ops_lib.py‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1110,6 +1110,36 @@ def q8ta_pixel_shuffle_impl(
11101110
lib.impl(name, q8ta_pixel_shuffle_impl, "CompositeExplicitAutograd")
11111111
q8ta_pixel_shuffle_op = getattr(getattr(torch.ops, namespace), name)
11121112

1113+
1114+
def q8ta_pixel_shuffle_with_activation_impl(
1115+
input: torch.Tensor,
1116+
input_scale: float,
1117+
input_zero_point: int,
1118+
output_inv_scale: float,
1119+
output_zero_point: int,
1120+
upscale_factor: int,
1121+
activation: str,
1122+
):
1123+
output_scale = 1.0 / output_inv_scale
1124+
dequant = torch.ops.quantized_decomposed.dequantize_per_tensor(
1125+
input, input_scale, input_zero_point, -128, 127, input.dtype
1126+
)
1127+
out = torch.nn.functional.pixel_shuffle(dequant, upscale_factor)
1128+
if activation == "relu":
1129+
out = torch.nn.functional.relu(out)
1130+
else:
1131+
assert activation == "none", f"Unsupported activation: {activation}"
1132+
return torch.ops.quantized_decomposed.quantize_per_tensor(
1133+
out, output_scale, output_zero_point, -128, 127, torch.int8
1134+
)
1135+
1136+
1137+
name = "q8ta_pixel_shuffle.with_activation"
1138+
lib.define(
1139+
f"{name}(Tensor input, float input_scale, int input_zero_point, float output_inv_scale, int output_zero_point, int upscale_factor, str activation) -> Tensor"
1140+
)
1141+
lib.impl(name, q8ta_pixel_shuffle_with_activation_impl, "CompositeExplicitAutograd")
1142+
11131143
########################
11141144
## embedding_q4gsw ##
11151145
########################

‎backends/vulkan/op_registry.py‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -670,7 +670,12 @@ def register_q8ta_relu():
670670
# =============================================================================
671671

672672

673-
@update_features(exir_ops.edge.et_vk.q8ta_pixel_shuffle.default)
673+
@update_features(
674+
[
675+
exir_ops.edge.et_vk.q8ta_pixel_shuffle.default,
676+
exir_ops.edge.et_vk.q8ta_pixel_shuffle.with_activation,
677+
]
678+
)
674679
def register_q8ta_pixel_shuffle():
675680
# The fused kernel is restricted to the channels-packed family
676681
# (PACKED_INT8_4W4C, PACKED_INT8_4C1W, PACKED_INT8_CONV2D), all of which

‎backends/vulkan/patterns/quantized_pixel_shuffle.py‎

Lines changed: 30 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
# This source code is licensed under the BSD-style license found in the
55
# LICENSE file in the root directory of this source tree.
66

7-
from typing import List, Optional, Set
7+
from typing import Dict, List, Optional, Set
88

99
import executorch.backends.vulkan.utils as utils
1010

@@ -30,6 +30,12 @@
3030
exir_ops.edge.dim_order_ops._clone_dim_order.default,
3131
}
3232

33+
# Activations that can follow the pixel_shuffle, mapped to the activation name
34+
# passed to q8ta_pixel_shuffle.with_activation.
35+
_ACTIVATIONS: Dict[object, str] = {
36+
exir_ops.edge.aten.relu.default: "relu",
37+
}
38+
3339

3440
def _is_noop_passthrough(node: torch.fx.Node) -> bool:
3541
return node.op == "call_function" and node.target in _NOOP_PASSTHROUGH_TARGETS
@@ -61,6 +67,8 @@ class QuantizedPixelShuffleMatch(PatternMatch):
6167
[optional] clone / _clone_dim_order
6268
aten.pixel_shuffle.default (upscale_factor = r)
6369
[optional] clone / _clone_dim_order
70+
[optional] activation (aten.relu.default)
71+
[optional] clone / _clone_dim_order
6472
q8ta_quantize_per_tensor (fp32 -> int8)
6573
6674
The anchor is the dequantize node since it is a unique entry point.
@@ -88,8 +96,14 @@ def __init__(self, dequantize_node: torch.fx.Node) -> None:
8896
if pixel_shuffle_node.target != exir_ops.edge.aten.pixel_shuffle.default:
8997
return
9098

91-
# Walk forward to the quantize node (skipping any clones).
99+
# Walk forward to the quantize node (skipping any clones), through an
100+
# optional activation.
92101
quantize_node = _skip_passthrough_user(pixel_shuffle_node, self.all_nodes)
102+
self.activation_node: Optional[torch.fx.Node] = None
103+
if quantize_node is not None and quantize_node.target in _ACTIVATIONS:
104+
self.activation_node = quantize_node
105+
self.all_nodes.append(self.activation_node)
106+
quantize_node = _skip_passthrough_user(self.activation_node, self.all_nodes)
93107
if quantize_node is None or not utils.is_quant_node(quantize_node):
94108
return
95109

@@ -139,8 +153,6 @@ def make_quantized_pixel_shuffle_custom_op(
139153
graph_module: torch.fx.GraphModule,
140154
match: QuantizedPixelShuffleMatch,
141155
) -> None:
142-
op_target = exir_ops.edge.et_vk.q8ta_pixel_shuffle.default
143-
144156
# The fused op takes the *inverse* of the output scale to match the
145157
# runtime kernel's expectation.
146158
output_scale = match.output_scales_node
@@ -162,19 +174,21 @@ def make_quantized_pixel_shuffle_custom_op(
162174
"quantized_pixel_shuffle pattern only supports scalar output scales"
163175
)
164176

177+
args = (
178+
match.input_int8_node,
179+
match.input_scales_node,
180+
match.input_zeros_node,
181+
inv_output_scale,
182+
match.output_zeros_node,
183+
match.upscale_factor,
184+
)
185+
op_target = exir_ops.edge.et_vk.q8ta_pixel_shuffle.default
186+
if match.activation_node is not None:
187+
op_target = exir_ops.edge.et_vk.q8ta_pixel_shuffle.with_activation
188+
args = (*args, _ACTIVATIONS[match.activation_node.target])
189+
165190
with graph_module.graph.inserting_before(match.output_node):
166-
new_node = graph_module.graph.create_node(
167-
"call_function",
168-
op_target,
169-
args=(
170-
match.input_int8_node,
171-
match.input_scales_node,
172-
match.input_zeros_node,
173-
inv_output_scale,
174-
match.output_zeros_node,
175-
match.upscale_factor,
176-
),
177-
)
191+
new_node = graph_module.graph.create_node("call_function", op_target, args)
178192

179193
new_node.meta["val"] = match.output_node.meta["val"]
180194
match.quantize_output_node.replace_all_uses_with(new_node)

‎backends/vulkan/runtime/graph/ops/glsl/q8ta_pixel_shuffle.glsl‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,14 +35,16 @@ layout(push_constant) uniform restrict Block {
3535
float inv_scale_out;
3636
int zp_in;
3737
int zp_out;
38+
// Lower clamp bound of requantized values; raised to zp_out to apply relu.
39+
int q_min;
3840
};
3941

4042
layout(local_size_x_id = 0, local_size_y_id = 1, local_size_z_id = 2) in;
4143

4244
${layout_declare_spec_const(C, "int", "inp_layout", "CONTIG_LAYOUT_INT")}
4345
${layout_declare_spec_const(C, "int", "outp_layout", "CONTIG_LAYOUT_INT")}
44-
// Set by the host when input and output quantization params match, making the
45-
// shader a pure byte shuffle.
46+
// Set by the host when input and output quantization params match and no
47+
// activation is applied, making the shader a pure byte shuffle.
4648
${layout_declare_spec_const(C, "int", "passthrough", "0")}
4749

4850
/*
@@ -66,7 +68,7 @@ int requantize_word(const int packed) {
6668
const int q_in = (packed << (24 - 8 * i)) >> 24;
6769
const float dq = float(q_in - zp_in) * scale_in;
6870
const int q_out =
69-
clamp(int(round(dq * inv_scale_out)) + zp_out, -128, 127);
71+
clamp(int(round(dq * inv_scale_out)) + zp_out, q_min, 127);
7072
result |= (q_out & 0xFF) << (8 * i);
7173
}
7274
return result;

‎backends/vulkan/runtime/graph/ops/impl/Q8taPixelShuffle.cpp‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,16 @@
66
* LICENSE file in the root directory of this source tree.
77
*/
88

9+
#include <executorch/backends/vulkan/runtime/graph/ops/impl/Q8taPixelShuffle.h>
10+
911
#include <executorch/backends/vulkan/runtime/graph/ops/OperatorRegistry.h>
1012

1113
#include <executorch/backends/vulkan/runtime/graph/ops/impl/Common.h>
1214
#include <executorch/backends/vulkan/runtime/graph/ops/utils/ShaderNameUtils.h>
1315

1416
#include <executorch/backends/vulkan/runtime/graph/Logging.h>
1517

18+
#include <algorithm>
1619
#include <cmath>
1720
#include <cstdint>
1821

@@ -100,6 +103,7 @@ void add_q8ta_pixel_shuffle_node(
100103
const ValueRef output_inv_scale,
101104
const ValueRef output_zp,
102105
const ValueRef upscale_factor,
106+
const ActivationType activation,
103107
const ValueRef packed_int8_output) {
104108
// dtype must be the int8x4 packed type
105109
VK_CHECK_COND(graph.dtype_of(packed_int8_input) == vkapi::kInt8x4);
@@ -145,14 +149,21 @@ void add_q8ta_pixel_shuffle_node(
145149
float inv_scale_out = graph.extract_scalar<float>(output_inv_scale);
146150
int32_t zp_in = graph.extract_scalar<int32_t>(input_zp);
147151
int32_t zp_out = graph.extract_scalar<int32_t>(output_zp);
152+
// quantize(relu(x)) == max(quantize(x), zp_out), so relu only raises the
153+
// lower clamp bound of the requantized value.
154+
int32_t q_min = -128;
155+
if (activation == ActivationType::kRelu) {
156+
q_min = std::clamp(zp_out, -128, 127);
157+
}
148158

149159
// Detect the pure-byte-shuffle case: same scale & same zero-point. In that
150160
// case the shader can skip the requantize math entirely.
151161
// Use a small relative tolerance on the scales.
152162
const float scale_diff = std::abs(scale_in - scale_out_actual);
153163
const float scale_thresh = 1e-7f * std::max(std::abs(scale_in), 1e-7f);
154164
int32_t passthrough = 0;
155-
if (scale_diff <= scale_thresh && zp_in == zp_out) {
165+
if (scale_diff <= scale_thresh && zp_in == zp_out &&
166+
activation == ActivationType::kNone) {
156167
passthrough = 1;
157168
}
158169

@@ -161,6 +172,7 @@ void add_q8ta_pixel_shuffle_node(
161172
PushConstantDataInfo(&inv_scale_out, sizeof(inv_scale_out)),
162173
PushConstantDataInfo(&zp_in, sizeof(zp_in)),
163174
PushConstantDataInfo(&zp_out, sizeof(zp_out)),
175+
PushConstantDataInfo(&q_min, sizeof(q_min)),
164176
};
165177

166178
// UBOs
@@ -209,6 +221,12 @@ void q8ta_pixel_shuffle(
209221
const ValueRef output_inv_scale = args.at(idx++);
210222
const ValueRef output_zp = args.at(idx++);
211223
const ValueRef upscale_factor = args.at(idx++);
224+
// q8ta_pixel_shuffle.with_activation passes an activation before the output.
225+
ActivationType activation = ActivationType::kNone;
226+
if (args.size() == 8) {
227+
activation =
228+
activation_type_from_string(graph.extract_string(args.at(idx++)));
229+
}
212230
const ValueRef packed_int8_output = args.at(idx);
213231

214232
add_q8ta_pixel_shuffle_node(
@@ -219,11 +237,13 @@ void q8ta_pixel_shuffle(
219237
output_inv_scale,
220238
output_zp,
221239
upscale_factor,
240+
activation,
222241
packed_int8_output);
223242
}
224243

225244
REGISTER_OPERATORS {
226245
VK_REGISTER_OP(et_vk.q8ta_pixel_shuffle.default, q8ta_pixel_shuffle);
246+
VK_REGISTER_OP(et_vk.q8ta_pixel_shuffle.with_activation, q8ta_pixel_shuffle);
227247
}
228248

229249
} // namespace vkcompute

‎backends/vulkan/runtime/graph/ops/impl/Q8taPixelShuffle.h‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,17 @@
1010

1111
#include <executorch/backends/vulkan/runtime/graph/ComputeGraph.h>
1212

13+
#include <executorch/backends/vulkan/runtime/graph/ops/impl/convolution/conv2d/q8ta/Q8taConv2dCommon.h>
14+
1315
namespace vkcompute {
1416

1517
//
1618
// Fused PixelShuffle operating on int8x4 packed tensors.
1719
//
1820
// Replaces the decomposed chain:
19-
// q8ta_dequantize -> view -> permute -> view -> q8ta_quantize
20-
// with a single byte-shuffle (and optional requantize when scales differ).
21+
// q8ta_dequantize -> view -> permute -> view -> [relu] -> q8ta_quantize
22+
// with a single byte-shuffle (and optional requantize when scales differ or an
23+
// activation is applied).
2124
//
2225

2326
void add_q8ta_pixel_shuffle_node(
@@ -28,6 +31,7 @@ void add_q8ta_pixel_shuffle_node(
2831
const ValueRef output_inv_scale,
2932
const ValueRef output_zp,
3033
const ValueRef upscale_factor,
34+
const ActivationType activation,
3135
const ValueRef packed_int8_output);
3236

3337
} // namespace vkcompute

‎backends/vulkan/test/ops/benchmark_v2/operators/pixel_shuffle/q8ta/Q8taPixelShuffleReference.cpp‎

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@
77
#include <executorch/backends/vulkan/test/ops/benchmark_v2/framework/ReferenceRegistry.h>
88
#include <executorch/backends/vulkan/test/ops/benchmark_v2/operators/quantize/q8ta/Q8taReferenceCommon.h>
99

10+
#include <algorithm>
11+
1012
namespace vkcompute::shader_profiling::reference {
1113
static Tensor q8ta_pixel_shuffle_reference(
1214
const Json& spec,
@@ -18,6 +20,9 @@ static Tensor q8ta_pixel_shuffle_reference(
1820
inverse_out_scale = q8ta::scale(spec, 3);
1921
const int64_t iz = q8ta::integer(spec, 2), oz = q8ta::integer(spec, 4),
2022
r = q8ta::integer(spec, 5);
23+
// q8ta_pixel_shuffle.with_activation passes an activation before the output.
24+
const bool relu = spec.at("args").size() == 8 &&
25+
spec.at("args").at(6).at("string") == "relu";
2126
const float actual_out_scale = 1.0f / inverse_out_scale;
2227
const bool passthrough = std::abs(is - actual_out_scale) <=
2328
1e-7f * std::max(std::abs(is), 1e-7f) &&
@@ -34,16 +39,22 @@ static Tensor q8ta_pixel_shuffle_reference(
3439
input.sizes[3] +
3540
x / r;
3641
const int64_t value = int64_t(input.values[source]);
37-
out.values[i] = passthrough
38-
? double(value)
39-
: double(q8ta::requantize(
40-
q8ta::dequantize(value, is, iz), inverse_out_scale, oz));
42+
int64_t q = passthrough
43+
? value
44+
: q8ta::requantize(
45+
q8ta::dequantize(value, is, iz), inverse_out_scale, oz);
46+
if (relu) {
47+
q = std::max(q, oz);
48+
}
49+
out.values[i] = double(q);
4150
});
4251
return out;
4352
}
4453
REGISTER_REFERENCE_FUNCTIONS {
4554
REGISTER_REFERENCE_FN(q8ta_pixel_shuffle, q8ta_pixel_shuffle_reference);
4655
REGISTER_OPERATOR_REFERENCE(
4756
et_vk.q8ta_pixel_shuffle.default, q8ta_pixel_shuffle);
57+
REGISTER_OPERATOR_REFERENCE(
58+
et_vk.q8ta_pixel_shuffle.with_activation, q8ta_pixel_shuffle);
4859
}
4960
} // namespace vkcompute::shader_profiling::reference

‎backends/vulkan/test/ops/benchmark_v2/operators/pixel_shuffle/q8ta/Q8taPixelShuffleThroughput.cpp‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,10 @@ REGISTER_THROUGHPUT_FUNCTIONS {
2525
et_vk.q8ta_pixel_shuffle.default, throughput::tensor_traffic);
2626
REGISTER_OPERATOR_COMPUTE_FLOPS(
2727
et_vk.q8ta_pixel_shuffle.default, pixel_shuffle_flops);
28+
REGISTER_OPERATOR_MEMORY_TRAFFIC(
29+
et_vk.q8ta_pixel_shuffle.with_activation, throughput::tensor_traffic);
30+
REGISTER_OPERATOR_COMPUTE_FLOPS(
31+
et_vk.q8ta_pixel_shuffle.with_activation, pixel_shuffle_flops);
2832
}
2933
} // namespace
3034
} // namespace vkcompute::shader_profiling

0 commit comments

Comments
 (0)