Skip to content

Ensure floats are returned losslessly by the C ABI on 32-bit x86 - #161950

Merged
rust-bors[bot] merged 1 commit into
rust-lang:mainfrom
beetrees:x86-ret-snan-c
Nov 16, 2026
Merged

rust-bors[bot] merged 1 commit into
rust-lang:mainfrom
beetrees:x86-ret-snan-c

Conversation

@beetrees

@beetrees beetrees commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor

View all comments

The x86 C ABI returns f32 and f64 using the x87 extended precision floating point format (henceforth x86_fp80) on the x87 floating point stack. Currently, LLVM uses regular fld/fstp instructions to convert f32/f64 to and from x86_fp80 when returning from a function with the C ABI, however these instructions perform floating point format conversions that quieten signalling NaNs, which breaks Rust's (and LLVM's) guarantees. This is a long-standing LLVM bug (llvm/llvm-project#66803) but is difficult to fix on the LLVM side as it ties into larger problems with the way LLVM handles x87 registers holding f32/f64, combined with the general lack of developer interest in 32-bit x86 without SSE (although the ABI bug specifically also affects targets with SSE2 enabled). This problem has been tracked on the Rust side in #115567, with #123351 ensuring that Rustic ABIs avoid the x87 stack altogether and #113053 adding notes to the platform support page detailing how the targets are non-complaint (including the tier 1 targets i686-unknown-linux-gnu and i686-pc-windows-msvc).

This PR fixes #115567 by manually converting NaNs to and from x86_fp80 when a function returns f32/f64 (this was briefly discussed in #t-compiler > `x87_f80` is weird @ 💬).

r? @tgross35
cc @RalfJung

try-job: i686-*

@rustbot rustbot added A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Aug 28, 2026
@rustbot

rustbot commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

tgross35 is currently at their maximum review capacity.
They may take a while to respond.

@rust-log-analyzer

This comment has been minimized.

@folkertdev

Copy link
Copy Markdown
Contributor

@bors delegate try

@rust-bors

rust-bors Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

✌️ @beetrees, you can now perform try builds on this pull request!

You can now post @bors try to start a try build.

@beetrees

Copy link
Copy Markdown
Contributor Author

@bors try

@rust-bors

This comment has been minimized.

rust-bors Bot pushed a commit that referenced this pull request Aug 28, 2026
Ensure floats are returned losslessly by the C ABI on 32-bit x86

try-job: i686-*
@rust-log-analyzer

This comment has been minimized.

@rust-bors

rust-bors Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

☀️ Try build successful (CI)
Build commit: a9e9507 (a9e95070267a88d3be44d2d95464f64d39bb751d)
Base parent: 17fd5b8 (17fd5b8a37b6667b6cc137f3cc35f09759768a3b)

@RalfJung RalfJung left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't review this, but I love it. :) I see that there's a separate path for nnan, so if LLVM can infer nnan all the new code should optimize away. That might be worth a test as well if we don't already have one?

View changes since this review

Comment thread compiler/rustc_codegen_llvm/src/abi.rs Resolved
@beetrees
beetrees force-pushed the x86-ret-snan-c branch 3 times, most recently from 68fdb80 to bb9b569 Compare August 29, 2026 14:05
Comment thread compiler/rustc_codegen_llvm/src/abi.rs Resolved
Comment thread compiler/rustc_codegen_llvm/src/abi.rs Outdated
let fraction = self.zext(fraction, self.type_ix(80));

let is_nan_res = self.or(exp_and_sign, fraction);
let is_nan_res = self.bitcast(is_nan_res, self.type_x86_fp80());

@folkertdev folkertdev Aug 29, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

slightly confusing that this works, but apparently it does? (because when stored x86_fp80 would use 12 or 16 bytes).

View changes since the review

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The LLVM x86_fp80 type is exactly 80 bits - padding is added when needed by Clang AFAICT:

https://github.com/llvm/llvm-project/blob/05da143c63d333420b99eab198e1a89c97480929/llvm/lib/IR/Type.cpp#L208

Comment thread compiler/rustc_codegen_llvm/src/abi.rs Resolved
@@ -44,6 +72,12 @@ where
// float aggregates directly in a floating-point register.
if fn_abi.ret.layout.is_single_fp_element(cx) {

@folkertdev folkertdev Aug 29, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, is this actually what we want for is_single_fp_element? Should it not also return true on f16 and f128 (or have a clearer name)?

pub fn is_single_fp_element<C>(self, cx: &C) -> bool
where
Ty: TyAbiInterface<'a, C>,
C: HasDataLayout,
{
match self.backend_repr {
BackendRepr::Scalar(scalar) => {
matches!(scalar.primitive(), Primitive::Float(Float::F32 | Float::F64))
}
BackendRepr::Memory { .. } => {
if self.fields.count() == 1 && self.fields.offset(0).bytes() == 0 {
self.field(cx, 0).is_single_fp_element(cx)
} else {
false
}
}
_ => false,
}
}

View changes since the review

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking into it, it definitely isn't what we want, and at least on s390x #[repr(C)] struct Foo(f16) is passed incorrectly.

The rust version moves ldgr %f0, %r0 from a general into a float register. With the C compilers the value is already there.

I'll try to clean that up.

@beetrees beetrees Aug 29, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've left it as is in this PR as I didn't want to put to many changes in one PR, but is_single_fp_element is definitely not correct here. Both Clang and GCC also match f16 on x86, and they also require the struct to have no padding (which isn't checked here at the moment). is_single_fp_element also doesn't handle #[repr(transparent)] structs containing 1ZST fields. is_single_fp_element is also used in the s390x ABI, where GCC and Clang compilers currently implement slightly different rules: padding is fine as long as the overall size if either 2, 4 bytes or 8 bytes. This appears to be GCC and Clang compilers violating the s390x ABI specification, which says "A structure equivalent to one of the above. A structure is equivalent to a type 𝑇 if and only if it has exactly one member, which is either of type 𝑇 itself or a structure equivalent to type 𝑇.", which implies that padding should actually be ignored.

(EDIT: That's strange, GitHub didn't show me your second comment until after I'd posted mine.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah is_single_fp_element is completely wrong.
Strange that tests/ui/abi/compatibility.rs didn't catch it, it's meant exactly for things like this. Even when I add an f32 test, which will check whether a repr(transparent) wrapper around f32 has the same ABI as f32, nothing fails: #161991

@beetrees
beetrees force-pushed the x86-ret-snan-c branch 2 times, most recently from 26bbbba to 552a4cb Compare August 29, 2026 16:34
@rust-log-analyzer

This comment has been minimized.

@rust-log-analyzer

This comment has been minimized.

Comment on lines +25 to +40
// CHECK: ucomiss [[XMM]], [[XMM]]
// CHECK: jp [[NAN_LABEL:.*]]
// CHECK: movss dword ptr [esp + [[#OFFSET:]]], [[XMM]]
// CHECK: fld dword ptr [esp + [[#OFFSET]]]
// CHECK: ret
// CHECK: [[NAN_LABEL]]:
// CHECK: movd [[BITS:.*]], [[XMM]]
// CHECK: mov dword ptr [esp + [[#OFFSET:]]], 0
// CHECK: mov e[[SIGN_AND_EXP:.*]], [[BITS]]
// CHECK: shl [[BITS]], 8
// CHECK: shr e[[SIGN_AND_EXP]], 16
// CHECK: mov dword ptr [esp + [[#OFFSET+4]]], [[BITS]]
// CHECK: or e[[SIGN_AND_EXP]], 32767
// CHECK: mov word ptr [esp + [[#OFFSET+8]]], [[SIGN_AND_EXP]]
// CHECK: fld tbyte ptr [esp + [[#OFFSET]]]
// CHECK: ret

@RalfJung RalfJung Aug 30, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to extract the NaN branch into a function to not blow up the assembly so much?

View changes since the review

@beetrees beetrees Aug 30, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would definitely be possible. The downsides would be that if the function called in the NaN branch was compiled separately (e.g. compiler-builtins) LLVM wouldn't be able to constant fold NaNs, and it would require leaf functions to do more stack setup, negating some of the benefit (a quick check based on the IR for extern "C" fn f(x: f64) -> f64 { x } shows that calling a function in the NaN branch would save a net 4 instructions in that case). Given that extern "C" functions that return f32/f64 are rare in Rust programs (as opposed to C where extern "C" is the default ABI) I don't think it's worth the extra complexity (part of my aim with this PR was to minimise the extra compiler complexity that goes towards supporting 32-bit x86 as much as possible). For comparison, the VaList setup prelude of a extern "C" fn f(x: ...) { f2(x) } function on x86_64-unknown-linux-gnu takes over 20 instructions.

@rust-bors

rust-bors Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

💔 Test for bf9ccb1 failed: CI. Failed jobs:

@beetrees

Copy link
Copy Markdown
Contributor Author

@bors try jobs=-i686-

@rust-bors

This comment has been minimized.

rust-bors Bot pushed a commit that referenced this pull request Nov 14, 2026
Ensure floats are returned losslessly by the C ABI on 32-bit x86


try-job: *-i686-*
@rust-bors

rust-bors Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

☀️ Try build successful (CI)
Build commit: 1ed708f (1ed708f8477614a7577072ecf978780a7653167a)
Base parent: 5f4effd (5f4effd579500175af844241eb179e756190f1e7)

@beetrees

Copy link
Copy Markdown
Contributor Author

@rustbot review

@rustbot rustbot added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Nov 14, 2026
@folkertdev

Copy link
Copy Markdown
Contributor

@bors r+

@rust-bors

rust-bors Bot commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

📌 Commit 44927e5 has been approved by folkertdev

It is now in the queue for this repository.

🌲 The tree is currently closed for pull requests below priority 100. This pull request will be tested once the tree is reopened.

Reason for tree closure: consistent failures on main

@rust-bors rust-bors Bot added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Nov 14, 2026
jhpratt added a commit to jhpratt/rust that referenced this pull request Nov 14, 2026
Ensure floats are returned losslessly by the C ABI on 32-bit x86

The x86 C ABI returns `f32` and `f64` using the x87 extended precision floating point format (henceforth `x86_fp80`) on the x87 floating point stack. Currently, LLVM uses regular `fld`/`fstp` instructions to convert `f32`/`f64` to and from `x86_fp80` when returning from a function with the C ABI, however these instructions perform floating point format conversions that quieten signalling NaNs, which breaks Rust's (and LLVM's) guarantees. This is a long-standing LLVM bug (llvm/llvm-project#66803) but is difficult to fix on the LLVM side as it ties into larger problems with the way LLVM handles x87 registers holding `f32`/`f64`, combined with the general lack of developer interest in 32-bit x86 without SSE (although the ABI bug specifically also affects targets with SSE2 enabled). This problem has been tracked on the Rust side in rust-lang#115567, with rust-lang#123351 ensuring that Rustic ABIs avoid the x87 stack altogether and rust-lang#113053 adding notes to the platform support page detailing how the targets are non-complaint (including the tier 1 targets `i686-unknown-linux-gnu` and `i686-pc-windows-msvc`).

This PR fixes rust-lang#115567 by manually converting NaNs to and from `x86_fp80` when a function returns `f32`/`f64` (this was briefly discussed in [#t-compiler > &rust-lang#96;x87_f80&rust-lang#96; is weird @ 💬](https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/.60x87_f80.60.20is.20weird/near/619651653)).

r? @tgross35
cc @RalfJung

try-job: i686-*
jhpratt added a commit to jhpratt/rust that referenced this pull request Nov 14, 2026
Ensure floats are returned losslessly by the C ABI on 32-bit x86

The x86 C ABI returns `f32` and `f64` using the x87 extended precision floating point format (henceforth `x86_fp80`) on the x87 floating point stack. Currently, LLVM uses regular `fld`/`fstp` instructions to convert `f32`/`f64` to and from `x86_fp80` when returning from a function with the C ABI, however these instructions perform floating point format conversions that quieten signalling NaNs, which breaks Rust's (and LLVM's) guarantees. This is a long-standing LLVM bug (llvm/llvm-project#66803) but is difficult to fix on the LLVM side as it ties into larger problems with the way LLVM handles x87 registers holding `f32`/`f64`, combined with the general lack of developer interest in 32-bit x86 without SSE (although the ABI bug specifically also affects targets with SSE2 enabled). This problem has been tracked on the Rust side in rust-lang#115567, with rust-lang#123351 ensuring that Rustic ABIs avoid the x87 stack altogether and rust-lang#113053 adding notes to the platform support page detailing how the targets are non-complaint (including the tier 1 targets `i686-unknown-linux-gnu` and `i686-pc-windows-msvc`).

This PR fixes rust-lang#115567 by manually converting NaNs to and from `x86_fp80` when a function returns `f32`/`f64` (this was briefly discussed in [#t-compiler > &rust-lang#96;x87_f80&rust-lang#96; is weird @ 💬](https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/.60x87_f80.60.20is.20weird/near/619651653)).

r? @tgross35
cc @RalfJung

try-job: i686-*
rust-bors Bot pushed a commit that referenced this pull request Nov 16, 2026
…uwer

Rollup of 23 pull requests

Successful merges:

 - #165928 (Replace most uses of `type_array` with `type_byte_array` in `rustc_codegen_llvm`)
 - #160618 (require a consistent codegen backend when -Zllvm-target-feature is present)
 - #161950 (Ensure floats are returned losslessly by the C ABI on 32-bit x86)
 - #162267 (Fix an edge within `std::fs::create_dir_all` for certain platforms that views trailing cur dir components as a separate missing directory components)
 - #164043 (alloc: Unstably add `Vec::repeat` and `Vec::repeat_in`)
 - #164808 (asm const ptr: move name generation to asm-specific logic)
 - #164823 (Use `DerefMut` and `IndexMut` before `&raw mut` autoborrow)
 - #165568 (rustdoc: Suggest corrections for reference label typos)
 - #165641 (Treat all proc macro dependencies as host dependencies in bootstrap)
 - #165924 (rustc_codegen_gcc subtree update 2026-11-13)
 - #165927 (Remove `const_struct` from `rustc_codegen_ssa`)
 - #163696 (Stabilize `int_from_ascii`)
 - #163882 (docs(vec): clarify into_boxed_slice allocation guarantee)
 - #164362 (Emit note for `non_exhaustive` within defining crate for `unreachable_patterns`)
 - #164465 (Under `guard_patterns` allow parenthesized guarded patterns in match arms)
 - #164997 (Fix as str suggestion in subpattern)
 - #165825 (make `Complex<x87_f80>` ABI-compatible with C)
 - #165868 (Small changes to dyn compatibility)
 - #165879 (Add `#[rustc_test_entrypoint_marker]` back !)
 - #165902 (Add allocator() to vec_deque::IntoIter and vec_deque::Drain)
 - #165903 (Fix borrowck ICE when a borrow outlives a local in a const block)
 - #165914 (Use `trim_prefix` in bootstrap)
 - #165919 (tests: accept better behavior on underaligned accesses)

Failed merges:

 - #165893 (Prune un/under used methods of `ast::Attribute`)
rust-bors Bot pushed a commit that referenced this pull request Nov 16, 2026
…uwer

Rollup of 23 pull requests

Successful merges:

 - #165928 (Replace most uses of `type_array` with `type_byte_array` in `rustc_codegen_llvm`)
 - #160618 (require a consistent codegen backend when -Zllvm-target-feature is present)
 - #161950 (Ensure floats are returned losslessly by the C ABI on 32-bit x86)
 - #162267 (Fix an edge within `std::fs::create_dir_all` for certain platforms that views trailing cur dir components as a separate missing directory components)
 - #164043 (alloc: Unstably add `Vec::repeat` and `Vec::repeat_in`)
 - #164808 (asm const ptr: move name generation to asm-specific logic)
 - #164823 (Use `DerefMut` and `IndexMut` before `&raw mut` autoborrow)
 - #165568 (rustdoc: Suggest corrections for reference label typos)
 - #165641 (Treat all proc macro dependencies as host dependencies in bootstrap)
 - #165924 (rustc_codegen_gcc subtree update 2026-11-13)
 - #165927 (Remove `const_struct` from `rustc_codegen_ssa`)
 - #163696 (Stabilize `int_from_ascii`)
 - #163882 (docs(vec): clarify into_boxed_slice allocation guarantee)
 - #164362 (Emit note for `non_exhaustive` within defining crate for `unreachable_patterns`)
 - #164465 (Under `guard_patterns` allow parenthesized guarded patterns in match arms)
 - #164997 (Fix as str suggestion in subpattern)
 - #165825 (make `Complex<x87_f80>` ABI-compatible with C)
 - #165868 (Small changes to dyn compatibility)
 - #165879 (Add `#[rustc_test_entrypoint_marker]` back !)
 - #165902 (Add allocator() to vec_deque::IntoIter and vec_deque::Drain)
 - #165903 (Fix borrowck ICE when a borrow outlives a local in a const block)
 - #165914 (Use `trim_prefix` in bootstrap)
 - #165919 (tests: accept better behavior on underaligned accesses)

Failed merges:

 - #165893 (Prune un/under used methods of `ast::Attribute`)
@rust-bors
rust-bors Bot merged commit 6f920e9 into rust-lang:main Nov 16, 2026
15 checks passed
@rustbot rustbot added this to the 1.102.0 milestone Nov 16, 2026
rust-bors Bot pushed a commit that referenced this pull request Nov 16, 2026
Rollup merge of #161950 - beetrees:x86-ret-snan-c, r=folkertdev

Ensure floats are returned losslessly by the C ABI on 32-bit x86

The x86 C ABI returns `f32` and `f64` using the x87 extended precision floating point format (henceforth `x86_fp80`) on the x87 floating point stack. Currently, LLVM uses regular `fld`/`fstp` instructions to convert `f32`/`f64` to and from `x86_fp80` when returning from a function with the C ABI, however these instructions perform floating point format conversions that quieten signalling NaNs, which breaks Rust's (and LLVM's) guarantees. This is a long-standing LLVM bug (llvm/llvm-project#66803) but is difficult to fix on the LLVM side as it ties into larger problems with the way LLVM handles x87 registers holding `f32`/`f64`, combined with the general lack of developer interest in 32-bit x86 without SSE (although the ABI bug specifically also affects targets with SSE2 enabled). This problem has been tracked on the Rust side in #115567, with #123351 ensuring that Rustic ABIs avoid the x87 stack altogether and #113053 adding notes to the platform support page detailing how the targets are non-complaint (including the tier 1 targets `i686-unknown-linux-gnu` and `i686-pc-windows-msvc`).

This PR fixes #115567 by manually converting NaNs to and from `x86_fp80` when a function returns `f32`/`f64` (this was briefly discussed in [#t-compiler > &#96;x87_f80&#96; is weird @ 💬](https://rust-lang.zulipchat.com/#narrow/channel/131828-t-compiler/topic/.60x87_f80.60.20is.20weird/near/619651653)).

r? @tgross35
cc @RalfJung

try-job: i686-*
@beetrees
beetrees deleted the x86-ret-snan-c branch November 16, 2026 06:39
@RalfJung

Copy link
Copy Markdown
Member

This is awesome, thanks a ton. :)

On our non-SSE targets, this now means that the extern "C" ABI works better than the "Rust" ABI, right? Since we are only applying this mitigation to "C"?

@beetrees

Copy link
Copy Markdown
Contributor Author

The "Rust" ABI mitigation is also applied on non-SSE x86, so this PR means that extern "C" has the same level of mitigations as extern "Rust".

When SSE is disabled, the mitigations will have little practical affect: while a few edge cases might theoretically benefit, in practice LLVM is extremely likely to load the f32/f64 into x87 registers either before or after the return (e.g. moving it around and performing operations on it), nullifying any benefits the mitigations might of had. The mitigations are enabled on non-SSE targets as they will benefit functions that are #[target_feature(enable = "sse")]/#[target_feature(enable = "sse2")] and for ease of implementation.

@RalfJung

RalfJung commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

The "Rust" ABI mitigation is also applied on non-SSE x86

I am not sure what you mean... the usual "Rust" ABI mitigation is that we use SSE registers to return floats, which obviously cannot be applied on non-SSE x86?

@beetrees

Copy link
Copy Markdown
Contributor Author

@RalfJung

Copy link
Copy Markdown
Member

Oh right I entirely forgot about that. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-LLVM Area: Code generation parts specific to LLVM. Both correctness bugs and optimization-related issues. S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tracking issue: 32bit x86 targets lose float NaN payload in return values

7 participants