[ET-VK][tools] Report RMS error and dispatches in etvk_ops_bench validation by SS-JIA · Pull Request #24855 · pytorch/executorch

SS-JIA · 2026-11-10T20:26:59Z

Stack from ghstack (oldest at bottom):

Ulterior Motive

Correctness gating for automated ET-VK operator optimization loops. Each candidate build is validated against its parent; the gate must show that the candidate dispatched the intended shaders, that its error did not grow, and that the gate itself can fail.

Rationale

What: etvk_ops_bench validation reports per-output mean absolute error, RMS error and reference RMS; validation-only operator reports list the dispatched execute_nodes; the --fault_rel flag injects a known relative error into GPU outputs.

Why: Pass/fail tolerances hide precision regressions that stay under atol/rtol, and max-abs error is dominated by single elements. Validation-only runs did not report which shaders ran, so a candidate silently falling back to the parent's shader still passed. Nothing showed that a gate could fail.

Details

Error statistics
- Before: each output reports mismatch_count, max_abs_error and max_rel_error.
- After: each output also reports mean_abs_error, rms_error and reference_rms over its finite elements, or null when none is finite, like max_rel_error. Comparing rms_error between two builds on the same inputs measures error growth.
Dispatched shaders
- Before: only timed operator reports include execute_nodes.
- After: validation-only operator reports include the same execute_nodes (the operator's own nodes, without staging copies).
Fault hook
- Before: none.
- After: --fault_rel=<r> scales every GPU output by 1 + r after readback, before comparison, and the validation report records fault_rel. A value that is not a finite number exits with code 2. An element fails only when r > rtol + atol / |expected|, so r = 0.01 fails the FP32 defaults but not FP16 tolerances; the README suggests 0.1 or more for FP16, or gating on rms_error.

Differential Revision: D130986953

[ghstack-poisoned]

pytorch-bot · 2026-11-10T20:27:04Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24855

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

[Mitigated] PR CI jobs created 12:15–12:55 PM PT on 11/11 stuck in Queued (GitHub Actions incident): push again to restart

❌ You can merge normally! (3 Unrelated Failures), 1 Unclassified Failure

As of commit 794b545 with merge base 115655c ():

UNCLASSIFIED FAILURE - DrCI could not classify the following job because the workflow did not run on the merge base. The failure may be pre-existing on trunk or introduced by this PR:

Test Vulkan Backend (specialized runners) / test-vulkan-nvidia / linux-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest / linux / linux-job (gh) (trunk failure)
[ FAILED ] OpLeTensorOutTest.BroadcastBoolInputs
pull / unittest / macos / macos-job (gh) (trunk failure)
[ FAILED ] OpLeTensorOutTest.BroadcastBoolInputs
pull / unittest / windows / windows-job (gh) (trunk failure)
[ FAILED ] OpLeTensorOutTest.BroadcastBoolInputs

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-11-10T20:28:36Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24859 * #24858 * #24857 * __->__ #24856 * #24855 ## Ulterior Motive Optimizing the FP16 1x1 convolutions of the DSR model on mobile GPUs. ## Rationale **What**: A `mul_add(a, b, c)` helper in `common.glslh` computes `a * b + c`. In shaders that define `HAS_FLOAT16_ACCUMULATION`, it uses `*` and `+` for `f16vec4` and `fma()` for `vec4`; in other shaders it is `fma()`. The GEMM tile helper and `conv2d_tiled` call it instead of `fma()`, and the conv2d shaders that accumulate in f16 vectors define the marker: `conv2d_pw_tiled`, `conv2d_tiled` and the buffer input `conv2d_gemm`. **Why**: Adreno 740 and 750 run `fma()` on f16 vectors at about half the speed of a multiply and an add. With this commit, Half pointwise convolutions take 0.37 to 0.49 of the time on a Galaxy S23 (Adreno 740) and a Galaxy S24 (Adreno 750). Replacing `fma()` in every shader regardless of type made Float shaders up to 1.35 times as slow on the same phones, so the form depends on the type. ## Details - **Before**: Multiply-accumulates call `fma()` for both dtypes. - **After**: In the Half variants of `conv2d_pw_tiled`, `conv2d_tiled` and the buffer input `conv2d_gemm`, multiply-accumulates on f16 vectors compute `c + a * b`. Every other shader still calls `fma()`, including the other users of the GEMM tile helper (linear, matmul, Q4 and Q8 linear, fused SDPA). `f16vec4` is a type only when `GL_EXT_shader_explicit_arithmetic_types_float16` is enabled, so a shader opts in by defining `HAS_FLOAT16_ACCUMULATION` before it includes `common.glslh`; without it, `mul_add` is a macro for `fma()`. Two alternatives do not work: - Enabling the extension in `common.glslh` for every shader changes implicit conversions in shaders that do not use f16 arithmetic (`max_pool2d`'s `mix()` call becomes ambiguous). - The extension's own macro cannot be the guard: glslang defines `GL_EXT_shader_explicit_arithmetic_types_float16` whether or not the extension is enabled. Other shaders that accumulate in f16 vectors can opt in once their accuracy and performance are measured. `c + a * b` rounds the product before the add, which `fma()` does not. On Adreno 740 and 750, the f16 error statistics of some conv2d correctness configurations change; all still pass. Outputs on Adreno 840, Mali-G715 and Xclipse 940 are bit-identical to the parent's. Only 16 SPIR-V modules change: the Half `conv2d_pw_tiled` (4), `conv2d_tiled` (11) and `conv2d_gemm_buffer_half` modules. Differential Revision: D130986954 Pull Request resolved: #24856

Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24859 * #24858 * __->__ #24857 * #24856 * #24855 ## Ulterior Motive Optimizing the FP16 1x1 convolutions of the DSR model on mobile GPUs. ## Rationale **What**: `f16vec4` texels are stored to rgba16f images without the shader generator's FP32 rounding, and the conv2d epilogue (residual, scale, clamp, ReLU, leaky ReLU) computes in the shader's texel type. Half pointwise (`conv2d_pw_tiled`), direct (`conv2d_tiled`) and buffer-input im2col (`conv2d_gemm`) convolutions now pass their f16 accumulators to the store as they are. **Why**: A texel computed in f16 is exact in f16, so the rounding never changes it, but it costs about 40 instructions per stored texel on Adreno 740. The epilogue took and returned `vec4`, so every Half store paid for the rounding, with or without an epilogue. With this commit three DSR 1x1 layers take 0.84 to 0.93 of the time on a Pixel 9 Pro XL (Mali-G715) and 0.87 to 0.91 on a Galaxy S23 (Adreno 740), and a DSR 3x3 direct convolution 0.92 and 0.96. A Galaxy S26 Ultra (Adreno 840) is within noise. ## Details - Store rounding - **Before**: Every store to an rgba16f image passes its texel through `round_to_f16(vec4)`, which rounds to the nearest f16 with FP32 bit manipulation because some devices round toward zero. An `f16vec4` texel converts to `vec4` and is rounded too, which never changes it. - **After**: The generator also declares `round_to_f16(f16vec4)`, which only widens. `f16vec4` is a type only where `GL_EXT_shader_explicit_arithmetic_types_float16` is enabled, so the overload is declared under `#ifdef HAS_FLOAT16_ACCUMULATION`, which the conv2d shaders that accumulate in f16 define before their output image declaration. Other shaders keep rounding their texels. - Epilogue - **Before**: `apply_epilogue()` computes in `vec4`; callers cast f16 accumulators to `vec4`. - **After**: It computes in `VEC4_T`, including the residual alpha multiply and the leaky ReLU slope, and the pointwise, direct and im2col shaders pass their accumulators without a cast. Where `VEC4_T` is `vec4` (Float variants, depthwise and transposed convolutions, texture-input im2col), the code compiles to the same Adreno 740 ISA. - Epilogue parameters - **Before**: The alpha, slope and clamp bounds reach the shaders as given. - **After**: For the shaders that compute the epilogue in f16, the host rounds them to the nearest f16, ties to even, so their conversion to `f16vec4` is exact even where the GPU rounds conversions toward zero. `half_to_float` and `float_to_half` move out of `StagingBuffer.cpp`'s anonymous namespace for this. Outputs without an epilogue are bit-identical. With an epilogue, Half variants that compute in f16 round to f16 after the residual add, the scale and the leaky ReLU, where they rounded once at the store before. Clamps are bit-identical: rounding is monotonic, so clamping an f16 value to the rounded bounds gives the rounded clamp of it. `FLT_MAX` bounds round to infinity. 16 SPIR-V modules change: - `conv2d_pw_tiled_half`, `conv2d_pw_tiled_bufw_half`, `conv2d_pw_tiled_4x8_half`, `conv2d_pw_tiled_4x8_bufw_half` - the 11 Half `conv2d_tiled` modules (`conv2d_1x1x4_half`, `conv2d_2x2x8_border_half`, ...) - `conv2d_gemm_buffer_half` 32 more modules (the Float conv2d modules, and the Half depthwise, transposed and texture-input im2col modules) differ in the form of the epilogue's conversions and compile to identical Adreno 740 ISA. Differential Revision: D130986955 Pull Request resolved: #24857

Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * #24859 * __->__ #24858 * #24857 * #24856 * #24855 ## Ulterior Motive Optimizing the FP16 1x1 convolutions of the DSR model on mobile GPUs. ## Rationale **What**: On Adreno, Half wide tile `conv2d_pw_tiled` convolutions keep their weights in a texture and fold their accumulation: the accumulators start from the bias, and the four products of each output texel are summed in a temporary before they are added. A specialization constant selects the folded accumulation in the wide tile texture weight Half variant, which also reads through `mediump` samplers. No shader variant is added. **Why**: Over the nine 1x1 layers of the DSR model, these convolutions take 0.829 of the parent's time on a Galaxy S26 Ultra (Adreno 840), 0.949 on a Galaxy S24 (Adreno 750) and 0.949 on a Galaxy S23 (Adreno 740). In the Adreno offline compiler, the temporary's multiply-adds fold into repeat (`rpt3`) instructions, which loop-carried accumulators do not allow: the Adreno 740 shader issues 45 multiply-add instructions instead of 108. The error against the CPU reference also shrinks, to 0.50 to 0.78 of the parent's, since each K step rounds the running sum once instead of four times. The route stays on Adreno: Mali-G715 and Xclipse 940 take 1.80 and 1.28 times as long with it. ## Details - **Before**: Half wide tile pointwise convolutions dispatch `conv2d_pw_tiled_4x8_bufw_half` where 16-bit storage buffers are available, and `conv2d_pw_tiled_4x8_half` with texture weights elsewhere: accumulators that start at zero and take one multiply-add per product, and a bias pass after the K loop. - **After**: On Adreno, they prepack their weights into a 2D texture (the buffer fallback stays for weights past the texture extent limit) and dispatch `conv2d_pw_tiled_4x8_half` with the `folded_accumulation` specialization constant set. Other GPUs, Float and the narrow tile dispatch the same shaders as before, whose SPIR-V is the parent's apart from the order of their type declarations. In the shader: - `accumulate_folded` sums each output texel's four products in a temporary, which the compiler places in consecutive registers so that the multiply-adds fold into `(rpt3)` instructions. The accumulators keep their lanes reversed (`wzyx`) so that the accumulating add folds too, and are reversed back after the K loop. - With `folded_accumulation`, the accumulators start from the bias and the bias pass is skipped. - The wide tile texture weight Half variant (`FOLDABLE` in the template) declares `folded_accumulation` as a bool specialization constant and reads the input, weights and bias through `mediump` samplers, which is exact for Half data. The folded accumulation needs them: with `highp` samplers it took 1.064 to 1.211 of the time of the `mediump` version on the three Adreno phones. Buffer weight variants keep `highp`, which was faster for them: with `mediump` inputs, narrow tile shapes took 1.09 to 1.20 of the time on the Galaxy S26 Ultra. - The other variants fix `folded_accumulation` at false with a plain constant, so its branches fold away in the SPIR-V. With a specialization constant left at false their Adreno instruction scheduling changed, and narrow tile shapes took up to 1.063 of the time on the Galaxy S23. Their SPIR-V is the parent's, apart from the order of their type declarations. `use_folded_accumulation` in `Conv2dPW.cpp` selects the route (Half, Adreno, wide tile) and the texture weight prepacking, and sets the specialization constant. Both are fixed when the node is built, while the tile is picked again on resize; after a resize to the narrow tile, the narrow texture weight variant ignores the constant. Differential Revision: D130986958 Pull Request resolved: #24858

…e convs on Mali Stack from [ghstack](https://github.com/ezyang/ghstack/tree/0.15.0) (oldest at bottom): * __->__ #24859 * #24858 * #24857 * #24856 * #24855 ## Ulterior Motive Optimizing the FP16 1x1 convolutions of the DSR model on Mali (Pixel 9 Pro XL). ## Rationale **What**: On Mali, Half wide tile `conv2d_pw_tiled` convolutions with at least as many 4-channel output slices as input slices, 8, 16, 32 or 64 channel tiles and at least 8192 pixel tiles use work groups of up to 896 threads: y spans the channel tiles and x divides the row of pixel tiles. Other convolutions keep the 512 thread picks. On every GPU, wide tile work groups also keep x large enough for the work group count on x to stay within the device limit. **Why**: The six DSR 1x1 layers that qualify take 0.771 to 1.005 of the time with these work groups on a Pixel 9 Pro XL (Mali-G715), and the nine 1x1 layers of the model take 0.928. Applied to every convolution with at least as many output as input slices, the same work groups took up to 1.11 of the time on other channel tile counts and 0.92 to 1.06 on outputs below 8192 pixel tiles, so those shapes keep their current work groups. The rule stays on Mali: on an earlier version of this stack, Adreno 740, 750 and 840 took 1.030, 1.038 and 1.045 of the time over the nine layers with it (G7 up to 1.33), and Xclipse 940 up to 1.99 per layer. ## Details - **Before**: Mali Half wide tile dispatches use 512 thread work groups: y is the channel tile count rounded up to a power of two (at most 32), and x takes the rest. - **After**: with N4 = ceil(C_out / 4), K4 = ceil(C_in / 4) and P = ceil(N4 / 2) (the channel tiles), a convolution with N4 >= K4, P equal to 8, 16, 32 or 64 and at least 8192 pixel tiles (ceil(W / 2) * ceil(H / 2)) uses the local size {x, P, 1}, where x is the largest divisor of ceil(W / 2) with x * P <= 896, or floor(896 / P) if that divisor is below half of it. The 896 threads are halved while they exceed the device's work group invocation or x size limit, as the 512 thread pick already is. No shader changes; Adreno, Xclipse, Float and the narrow tile are unchanged. - Work group count on x - **Before**: Wide tile dispatches put every pixel tile on x, and nothing bounds the resulting work group count. With 64 thread work groups (GPUs other than Mali and Adreno, x = 4 above 8 channel tiles), a 1024x1024 output already needs 65536 work groups on x, one more than the usual `maxComputeWorkGroupCount[0]`, and the dispatch throws. - **After**: x is at least ceil(pixel tiles / `maxComputeWorkGroupCount[0]`). The row-aligned pick falls back to floor(896 / P) when its divisor is smaller, and the other picks raise min_x to the smallest power of two at or above it, up to the thread count. Below the limit, every pick is unchanged. Differential Revision: D130986963 Pull Request resolved: #24859

…dation ## Ulterior Motive Correctness gating for automated ET-VK operator optimization loops. Each candidate build is validated against its parent; the gate must show that the candidate dispatched the intended shaders, that its error did not grow, and that the gate itself can fail. ## Rationale **What**: `etvk_ops_bench` validation reports per-output mean absolute error, RMS error and reference RMS; validation-only operator reports list the dispatched `execute_nodes`; the `--fault_rel` flag injects a known relative error into GPU outputs. **Why**: Pass/fail tolerances hide precision regressions that stay under `atol`/`rtol`, and max-abs error is dominated by single elements. Validation-only runs did not report which shaders ran, so a candidate silently falling back to the parent's shader still passed. Nothing showed that a gate could fail. ## Details - Error statistics - **Before**: each output reports `mismatch_count`, `max_abs_error` and `max_rel_error`. - **After**: each output also reports `mean_abs_error`, `rms_error` and `reference_rms` over its finite elements, or null when none is finite, like `max_rel_error`. Comparing `rms_error` between two builds on the same inputs measures error growth. - Dispatched shaders - **Before**: only timed operator reports include `execute_nodes`. - **After**: validation-only operator reports include the same `execute_nodes` (the operator's own nodes, without staging copies). - Fault hook - **Before**: none. - **After**: `--fault_rel=<r>` scales every GPU output by `1 + r` after readback, before comparison, and the validation report records `fault_rel`. A value that is not a finite number exits with code 2. An element fails only when `r > rtol + atol / |expected|`, so `r` = 0.01 fails the FP32 defaults but not FP16 tolerances; the README suggests 0.1 or more for FP16, or gating on `rms_error`. Differential Revision: [D130986953](https://our.internmc.facebook.com/intern/diff/D130986953/) ghstack-source-id: 469962061 Pull-Request: #24855

Update

794b545

[ghstack-poisoned]

meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 10, 2026

SS-JIA temporarily deployed to cadence November 10, 2026 20:27 — with GitHub Actions Inactive

meta-codesync Bot added the meta-exported label Nov 10, 2026

SS-JIA temporarily deployed to cadence November 10, 2026 20:39 — with GitHub Actions Inactive

SS-JIA temporarily deployed to cadence November 10, 2026 20:42 — with GitHub Actions Inactive

SS-JIA temporarily deployed to cadence November 10, 2026 20:44 — with GitHub Actions Inactive

metascroy approved these changes Nov 11, 2026

View reviewed changes

meta-codesync Bot merged commit 1d225ec into gh/SS-JIA/706/base Nov 12, 2026
212 of 226 checks passed

meta-codesync Bot deleted the gh/SS-JIA/706/head branch November 12, 2026 00:07

meta-codesync Bot deployed to cherry-pick-bot November 12, 2026 00:07 Active

pytorchbot mentioned this pull request Nov 12, 2026

[ET-VK][tools] Report RMS error and dispatches in etvk_ops_bench validation #24938

Merged

Navigation Menu

Repository navigation

[ET-VK][tools] Report RMS error and dispatches in etvk_ops_bench validation - #24855

[ET-VK][tools] Report RMS error and dispatches in etvk_ops_bench validation#24855
meta-codesync[bot] merged 1 commit into
gh/SS-JIA/706/basefrom
gh/SS-JIA/706/head

SS-JIA commented Nov 10, 2026 •

edited

Loading

Uh oh!

pytorch-bot Bot commented Nov 10, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 10, 2026

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

SS-JIA commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Ulterior Motive

Rationale

Details

Uh oh!

pytorch-bot Bot commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24855

❗ 1 Active SEVs

❌ You can merge normally! (3 Unrelated Failures), 1 Unclassified Failure

Uh oh!

github-actions Bot commented Nov 10, 2026

This PR needs a release notes: label

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SS-JIA commented Nov 10, 2026 •

edited

Loading

pytorch-bot Bot commented Nov 10, 2026 •

edited

Loading

This PR needs a `release notes:` label