[akashveramd]

Summary

• Update ROCm CI from 10.0 to 10.1.

• This PR switch CI to pytorch 2.15 #5033 upgrades python 3.10 -> 3.11 but not build workflow .github/workflows/build_wheels_linux_x86.yml

• Note: CUDA and rocm10.0 still have older pins such as
  2.15.0.dev20260923 for cp310; that exact build is not on rocm10.1
  (rocm10.1 nightlies are 2.16.0.dev… and start at cp311) https://github.com/pytorch/ao/actions/runs/41744610851/job/126294477898#step:14:71.

  

  The Python bump aligns the wheel matrix with what rocm10.1 actually publishes.

The PR is currently blocked by the PRs, #5036 and #4941.

Made with Cursor

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/ao/5032

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 73b8d1a with merge base 9e9dca9 ():

NEW FAILURE - The following job has failed:

• Run Regression Tests on ROCm / test-nightly (ROCM Nightly, linux.rocm.gpu.ecosystem.mi350.1, --pre torchvision torch --index-url... / linux-job (gh)
  RuntimeError: Command docker exec -t 2f39111513cc41208a239cc0c7865ba657beecfef8a58a6ec3cf799c61d7eb06 /exec failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[vkuzo]

@claude review

[claude]

Claude finished @vkuzo's task in 56s —— View job

---

PR Review: [ROCm][CI] Update ROCm version to 10.1

• Read diff
• Check CI failure
• Post review

Diff: The changes look good. In both regression_test_rocm.yml and regression_test_pt2e_gpu.yml (both test and test-rocm jobs), all three ROCm version strings move from 10.0 to 10.1 together: torch-spec index URL, gpu-arch-version, and docker-image. No other workflow pins rocm10.0, so nothing is left behind.

Python 3.10 → 3.11 bump (commit f09b01f): This is a reasonable change, but the PR description doesn't mention it. Please add one line explaining why, e.g. whether the rocm10.1 nightly wheels are no longer published for cp310. That way someone reading the history later knows the 3.11 pin is required, not optional.

CI failure: Run Regression Tests on ROCm / test-nightly

Results: 4 failed, 1867 passed. Setup worked: the docker image, wheel index and Python 3.11 env all installed fine. The 4 failures are Triton compile errors, and all have the same cause:

triton.compiler.errors.CompilationError: too many indices for tensor of rank 0

Test	Kernel location
test/prototype/blockwise_fp8_training/test_blockwise_kernels.py::test_triton_quantize_fp8_weight_quant_transposed_rhs[128,256]	torchao/prototype/blockwise_fp8_training/kernels.py:915-916 (scale_k[:, None], scale_m[None, :])
test/prototype/moe_training/test_kernels.py::test_fp8_rowwise_3d_transpose_rhs_atomic[True,False]	torchao/prototype/moe_training/kernels/float8_rowwise.py:196 and :256 (expert_idx[:, None])

In these kernels, scale_k/scale_m (= pid_n/pid_m) and expert_idx are scalars from tl.program_id, and the code indexes them with [:, None]. The nightly installed triton-rocm==3.9.0+gitb7ddcf86, and this Triton version rejects that indexing. Older versions accepted it. So the failures most likely come from the Triton that ships with the new nightly wheel, not from anything ROCm 10.1 specific in this PR. The same kernels will probably break on CUDA once the CUDA nightlies pick up this Triton.

Suggested fix (separate PR or on top of this one): remove the [:, None] / [None, :] on the scalar terms, since broadcasting a scalar against the block offsets already gives the right shape. For example:

scale_offs = scale_k * s_stride_dim_0 + scale_m * s_stride_dim_1

If the store or mask must be 2D, broadcast against a real block, e.g. expert_idx * stride_scales_dim0 + k_offs[None, :] * stride_scales_dim1.

Fix this →

Verdict

The workflow change itself is fine. Before merging, decide how to handle the 4 kernel failures: fix the kernels, or land this with the failures tracked in a follow-up issue. The failures are real bugs exposed by the newer Triton, not CI flakiness.

[vkuzo]

could you look into the CI failures, are they real?

[vkuzo]

just as a heads up, i'm going to do the python 3.10 -> 3.11 upgrade in this PR: #5033

[akashveramd]

could you look into the CI failures, are they real?

@vkuzo: @jagadish-amd is currently looking into the CI failure.

[jagadish-amd]

The failures are fixed in this PR #5036
cc @vkuzo