Skip to content

fix(autograd): support 1-D and arbitrary-rank inputs for k-bit linear backward (#2110, #2108) - #2146

Open
shotsan wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
shotsan:fix/kbit-linear-bias-gradient-1d
Open

shotsan wants to merge 1 commit into
bitsandbytes-foundation:mainfrom
shotsan:fix/kbit-linear-bias-gradient-1d

Conversation

@shotsan

@shotsan shotsan commented Nov 17, 2026

Copy link
Copy Markdown

Summary

Resolves #2110 and #2108.

  1. Fix bias gradient reduction for 1-D and >2-D inputs (k-bit linear bias gradient is invalid for 1-D inputs (and wrong rank for >2-D) #2110):
    In MatMulLt, MatMul8bitFp, and MatMul4Bit, grad_bias was calculated with grad_output.sum(0, dtype=ctx.dtype_bias). For 1-D inputs (shape (out_features,)), sum(0) yielded a 0-D scalar [], which PyTorch autograd rejected with RuntimeError: Function MatMul4BitBackward returned an invalid gradient at index 2 - got [] but expected shape compatible with [out_features]. For >2-D inputs, sum(0) left unreduced leading dimensions.
    Reducing across all leading batch dimensions via grad_output.reshape(-1, grad_output.shape[-1]).sum(0, dtype=ctx.dtype_bias) consistently returns a 1-D tensor of shape (out_features,) across all input ranks (1-D, 2-D, 3-D, 4-D).

  2. Fix weight gradient (grad_B) for non-2D inputs in MatMul8bitFp (Linear8bitLt CPU/XPU backward crashes for non-2D inputs in MatMul8bitFp #2108):
    In MatMul8bitFp.backward (CPU/XPU), grad_B was computed with torch.matmul(A.t(), grad_output).t(). A.t() failed with RuntimeError: t() expects a tensor with <= 2 dimensions for 3-D/4-D inputs, and rank-mismatched on 1-D inputs. Flattening A and grad_output to 2-D matrices (A.reshape(-1, A.shape[-1]) and grad_output.reshape(-1, grad_output.shape[-1])) makes grad_B work across arbitrary activation ranks.

  3. Generalize MatMulLt reshape:
    Updated the len(grad_output.shape) == 3 check to len(grad_output.shape) != 2 to ensure non-2D activations (1-D, 3-D, 4-D) are consistently flattened to 2-D for int8 quantization.

Testing

Added regression test suites across 1-D, 2-D, 3-D, and 4-D shapes:

  • tests/test_linear4bit.py::test_linear4bit_bias_gradient_input_ranks (4 passed)
  • tests/test_linear8bitlt.py::test_linear8bitlt_bias_gradient_input_ranks (4 passed)
  • tests/test_linear8bitlt.py::test_linear8bitlt_fp16_weights_backward_non_2d_ranks (3 passed)
  • Full test_linear8bitlt.py test suite passed (58 passed, 4 skipped).

… backward (bitsandbytes-foundation#2110, bitsandbytes-foundation#2108)

- Compute k-bit linear bias gradient over all leading dimensions via
  `grad_output.reshape(-1, grad_output.shape[-1]).sum(0)` instead of `sum(0)`
  alone. For 1-D inputs, `sum(0)` yielded a 0-D scalar which was rejected by
  autograd; for >2-D inputs, it returned extra leading dimensions (bitsandbytes-foundation#2110).
- Reshape `A` and `grad_output` in `MatMul8bitFp` (CPU/XPU) to 2-D when
  computing `grad_B`, avoiding `RuntimeError: t() expects a tensor with <= 2
  dimensions` on 1-D, 3-D, and 4-D activations (bitsandbytes-foundation#2108).
- Reshape non-2D `grad_output` in `MatMulLt` to 2-D when `grad_output.ndim != 2`.
- Add regression unit tests across 1-D, 2-D, 3-D, and 4-D input shapes in
  `tests/test_linear4bit.py` and `tests/test_linear8bitlt.py`.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

k-bit linear bias gradient is invalid for 1-D inputs (and wrong rank for >2-D)

1 participant