Skip to content

Nightly regressed igvm crate and now emits SIGILL at opt-level higher than 1 #136361

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
durin42 opened this issue Jan 31, 2025 · 9 comments · Fixed by #136450
Closed

Nightly regressed igvm crate and now emits SIGILL at opt-level higher than 1 #136361

durin42 opened this issue Jan 31, 2025 · 9 comments · Fixed by #136450
Assignees
Labels
A-mir-opt Area: MIR optimizations A-mir-opt-GVN Area: MIR opt Global Value Numbering (GVN) C-bug Category: This is a bug. I-miscompile Issue: Correct Rust code lowers to incorrect machine code I-unsound Issue: A soundness hole (worst kind of bug), see: https://en.wikipedia.org/wiki/Soundness P-critical Critical priority regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. S-has-bisection Status: a bisection has been found for this issue S-has-mcve Status: A Minimal Complete and Verifiable Example has been found for this issue T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Comments

@durin42
Copy link
Contributor

durin42 commented Jan 31, 2025

Code

I tried this code:

durin42/igvm@1880200
cargo +nightly test --release

I expected to see this happen: Tests all pass.

Instead, this happened: Tests die with a SIGILL.

Version it worked on

This works on stable, and also up until b6b8361 (merge of #133324)

Version with regression

b6b8361 breaks, aka the merge of #133324. All nightlies since then show the same behavior.

Backtrace

I don't get one, I assume because it's a crash.

bisect-rustc output:

searched nightlies: from nightly-2025-01-07 to nightly-2025-01-13
regressed nightly: nightly-2025-01-10
searched commit range: a580b5c...8247594
regressed commit: b6b8361

bisected with cargo-bisect-rustc v0.6.8

Host triple: x86_64-unknown-linux-gnu
Reproduce with:

cargo bisect-rustc --start 2025-01-07 --end 2025-01-13 -- test --release

It's plausible to me that one of the crates in here is guilty of some unsafe crimes, but I'm not sure how to prove who's at fault at this point.

@rustbot modify labels: +regression-from-stable-to-nightly -regression-untriaged

@durin42 durin42 added C-bug Category: This is a bug. regression-untriaged Untriaged performance or correctness regression. labels Jan 31, 2025
@rustbot rustbot added I-prioritize Issue: Indicates that prioritization has been requested for this issue. needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. and removed regression-untriaged Untriaged performance or correctness regression. labels Jan 31, 2025
@durin42
Copy link
Contributor Author

durin42 commented Jan 31, 2025

Update: a colleague has further reduced this to just zerocopy as a dep, and will share the sample here shortly

@AdamCDunlap
Copy link

This is a much simpler reproduction on rust playground. Note nightly and release mode must be selected for the SIGILL to occur.

https://play.rust-lang.org/?version=nightly&mode=release&edition=2021&gist=e522e9c608dea108a0a278b333523660

@lqd
Copy link
Member

lqd commented Jan 31, 2025

cc @scottmcm on that bisection

@jieyouxu jieyouxu added E-needs-mcve Call for participation: This issue has a repro, but needs a Minimal Complete and Verifiable Example requires-nightly This issue requires a nightly compiler in some way. A-mir-opt Area: MIR optimizations S-has-bisection Status: a bisection has been found for this issue A-mir-opt-GVN Area: MIR opt Global Value Numbering (GVN) T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. and removed requires-nightly This issue requires a nightly compiler in some way. needs-triage This issue may need triage. Remove it if it has been sufficiently triaged. labels Jan 31, 2025
@cyrgani
Copy link
Contributor

cyrgani commented Jan 31, 2025

Smaller:

use std::slice;

fn main() {
    let vp_ctx: &Box<[u8; 2]> = &Box::new([0, 0]);
    let slf: *const [u8; 2] = &raw const **vp_ctx;

    let bytes = unsafe { slice::from_raw_parts(slf.cast::<u8>(), 2) };

    for _ in bytes {}
}

@jieyouxu jieyouxu added S-has-mcve Status: A Minimal Complete and Verifiable Example has been found for this issue and removed E-needs-mcve Call for participation: This issue has a repro, but needs a Minimal Complete and Verifiable Example labels Jan 31, 2025
@cuviper
Copy link
Member

cuviper commented Feb 1, 2025

Cute -- @cyrgani's example compiles to almost nothing:

0000000000014b40 <_ZN11rust_1363614main17hc26238fcb10c7fa6E>:
   14b40:       48 8b 05 81 3f 04 00    mov    0x43f81(%rip),%rax        # 58ac8 <_DYNAMIC+0x220>
   14b47:       0f b6 00                movzbl (%rax),%eax
   14b4a:       0f 0b                   ud2

And of course that ud2 is the source of SIGILL, presumably from LLVM's trap-unreachable.

FWIW, cranelift panics on this in release mode:

cranelift panic and mir dump
$ RUSTFLAGS=-Zcodegen-backend=cranelift cargo +nightly run --release
   Compiling rust-136361 v0.1.0 (/tmp/rust-136361)
warning: target feature `x87` must be enabled to ensure that the ABI of the current target can be implemented correctly
  |
  = note: this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
  = note: for more information, see issue #116344 <https://github.com/rust-lang/rust/issues/116344>


thread 'rustc' panicked at compiler/rustc_codegen_cranelift/src/value_and_place.rs:589:9:
assertion `left == right` failed
  left: Size(8 bytes)
 right: Size(16 bytes)
stack backtrace:
   0:     0x7f0cc442b760 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::hdd797bb3d9453305
   1:     0x7f0cc4c138e6 - core::fmt::write::h8903f82ae713d84a
   2:     0x7f0cc5b54011 - std::io::Write::write_fmt::h4c071e9d390cbd83
   3:     0x7f0cc442b5c2 - std::sys::backtrace::BacktraceLock::print::hfb2f5541238e1583
   4:     0x7f0cc442da42 - std::panicking::default_hook::{{closure}}::h8fe9847ada2da135
   5:     0x7f0cc442d8ca - std::panicking::default_hook::hcfe19f0ffc592091
   6:     0x7f0cc3587059 - std[41a6e8152209c0a6]::panicking::update_hook::<alloc[9f80a3905cc8b55b]::boxed::Box<rustc_driver_impl[3ce1408bca960349]::install_ice_hook::{closure#1}>>::{closure#0}
   7:     0x7f0cc442e583 - std::panicking::rust_panic_with_hook::hdb647d40fc260137
   8:     0x7f0cc442e27a - std::panicking::begin_panic_handler::{{closure}}::h7323aa792739fc4e
   9:     0x7f0cc442bc49 - std::sys::backtrace::__rust_end_short_backtrace::h304d83d1d823b3b1
  10:     0x7f0cc442df3d - rust_begin_unwind
  11:     0x7f0cc1052a00 - core::panicking::panic_fmt::h9a9c479b3cce79d6
  12:     0x7f0cc2fa74c6 - core::panicking::assert_failed_inner::hf444ff32ba58f969
  13:     0x7f0cc3b328a4 - core[1b69f308fde5021f]::panicking::assert_failed::<rustc_abi[2bf90cc0e24882f2]::Size, rustc_abi[2bf90cc0e24882f2]::Size>
  14:     0x7f0cb5893d6e - <rustc_codegen_cranelift[cced2f27739003b0]::value_and_place::CPlace>::write_cvalue_maybe_transmute
  15:     0x7f0cb5842a64 - rustc_codegen_cranelift[cced2f27739003b0]::base::codegen_fn_body
  16:     0x7f0cb5838483 - rustc_codegen_cranelift[cced2f27739003b0]::base::codegen_fn
  17:     0x7f0cb585d831 - rustc_codegen_cranelift[cced2f27739003b0]::driver::aot::codegen_cgu_content
  18:     0x7f0cb585fe48 - rustc_codegen_cranelift[cced2f27739003b0]::driver::aot::module_codegen
  19:     0x7f0cb586b541 - rustc_codegen_cranelift[cced2f27739003b0]::driver::aot::run_aot::{closure#3}::{closure#0}
  20:     0x7f0cb5807af9 - <core[1b69f308fde5021f]::iter::adapters::filter_map::FilterMap<alloc[9f80a3905cc8b55b]::vec::into_iter::IntoIter<(usize, &rustc_middle[80127f98e0fedb5b]::mir::mono::CodegenUnit)>, rustc_data_structures[8ba96ecce1330312]::sync::parallel::par_map<(usize, &rustc_middle[80127f98e0fedb5b]::mir::mono::CodegenUnit), alloc[9f80a3905cc8b55b]::vec::Vec<(usize, &rustc_middle[80127f98e0fedb5b]::mir::mono::CodegenUnit)>, rustc_codegen_cranelift[cced2f27739003b0]::driver::aot::OngoingModuleCodegen, alloc[9f80a3905cc8b55b]::vec::Vec<rustc_codegen_cranelift[cced2f27739003b0]::driver::aot::OngoingModuleCodegen>, rustc_codegen_cranelift[cced2f27739003b0]::driver::aot::run_aot::{closure#3}::{closure#0}>::{closure#0}::{closure#1}> as core[1b69f308fde5021f]::iter::traits::iterator::Iterator>::next
  21:     0x7f0cb5866d81 - rustc_codegen_cranelift[cced2f27739003b0]::driver::aot::run_aot
  22:     0x7f0cb58969d9 - <rustc_codegen_cranelift[cced2f27739003b0]::CraneliftCodegenBackend as rustc_codegen_ssa[57384e5daf4b3948]::traits::backend::CodegenBackend>::codegen_crate
  23:     0x7f0cc5c380f4 - <rustc_interface[444c50c8c58a8997]::queries::Linker>::codegen_and_build_linker
  24:     0x7f0cc5c2e91d - rustc_interface[444c50c8c58a8997]::passes::create_and_enter_global_ctxt::<core[1b69f308fde5021f]::option::Option<rustc_interface[444c50c8c58a8997]::queries::Linker>, rustc_driver_impl[3ce1408bca960349]::run_compiler::{closure#0}::{closure#2}>::{closure#2}::{closure#0}
  25:     0x7f0cc5b245d8 - rustc_interface[444c50c8c58a8997]::interface::run_compiler::<(), rustc_driver_impl[3ce1408bca960349]::run_compiler::{closure#0}>::{closure#1}
  26:     0x7f0cc5a8c5f5 - std[41a6e8152209c0a6]::sys::backtrace::__rust_begin_short_backtrace::<rustc_interface[444c50c8c58a8997]::util::run_in_thread_with_globals<rustc_interface[444c50c8c58a8997]::util::run_in_thread_pool_with_globals<rustc_interface[444c50c8c58a8997]::interface::run_compiler<(), rustc_driver_impl[3ce1408bca960349]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}, ()>
  27:     0x7f0cc5a8c2d9 - <<std[41a6e8152209c0a6]::thread::Builder>::spawn_unchecked_<rustc_interface[444c50c8c58a8997]::util::run_in_thread_with_globals<rustc_interface[444c50c8c58a8997]::util::run_in_thread_pool_with_globals<rustc_interface[444c50c8c58a8997]::interface::run_compiler<(), rustc_driver_impl[3ce1408bca960349]::run_compiler::{closure#0}>::{closure#1}, ()>::{closure#0}, ()>::{closure#0}::{closure#0}, ()>::{closure#1} as core[1b69f308fde5021f]::ops::function::FnOnce<()>>::call_once::{shim:vtable#0}
  28:     0x7f0cc5a8ba6f - std::sys::pal::unix::thread::Thread::new::thread_start::h57c67f308f577aff
  29:     0x7f0cbfc7e168 - start_thread
  30:     0x7f0cbfd0214c - __clone3
  31:                0x0 - <unknown>

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: please make sure that you have updated to the latest nightly

note: please attach the file at `/tmp/rust-136361/rustc-ice-2025-02-01T00_09_10-27048.txt` to your bug report

note: compiler flags: --crate-type bin -C opt-level=3 -C embed-bitcode=no -C strip=debuginfo -Z codegen-backend=cranelift

note: some of the compiler flags provided by cargo are hidden

query stack during panic:
end of query stack
stmt _16 = copy _12 as *mut u8 (Transmute)
fn main() -> () {
    let mut _0: ();
    let _1: &std::boxed::Box<[u8; 2]>;
    let _2: std::boxed::Box<[u8; 2]>;
    let mut _3: [u8; 2];
    let mut _4: *const u8;
    let mut _5: std::slice::Iter<'_, u8>;
    let mut _7: std::option::Option<&u8>;
    let mut _8: &mut std::slice::Iter<'_, u8>;
    let mut _9: isize;
    let mut _10: std::boxed::Box<[u8; 2]>;
    let mut _11: *const [u8; 2];
    scope 1 {
        debug vp_ctx => _1;
        scope 2 {
            debug slf => _11;
            scope 3 {
                debug bytes => _12;
                let mut _6: std::slice::Iter<'_, u8>;
                scope 4 {
                    debug iter => _6;
                }
                scope 13 (inlined core::slice::iter::<impl std::iter::IntoIterator for &[u8]>::into_iter) {
                    scope 14 (inlined core::slice::<impl [u8]>::iter) {
                        scope 15 (inlined std::slice::Iter::<'_, u8>::new) {
                            let mut _15: *mut u8;
                            let mut _16: *mut u8;
                            scope 16 {
                                let _13: std::ptr::NonNull<u8>;
                                scope 17 {
                                    let _14: *const u8;
                                    scope 18 {
                                    }
                                    scope 23 (inlined std::ptr::without_provenance::<u8>) {
                                        scope 24 (inlined std::ptr::without_provenance_mut::<u8>) {
                                        }
                                    }
                                    scope 25 (inlined std::ptr::NonNull::<u8>::as_ptr) {
                                    }
                                    scope 26 (inlined std::ptr::mut_ptr::<impl *mut u8>::add) {
                                    }
                                }
                                scope 19 (inlined <std::ptr::NonNull<[u8]> as std::convert::From<&[u8]>>::from) {
                                    scope 20 (inlined std::ptr::NonNull::<[u8]>::from_ref) {
                                    }
                                }
                                scope 21 (inlined std::ptr::NonNull::<[u8]>::cast::<u8>) {
                                    scope 22 (inlined std::ptr::NonNull::<[u8]>::as_ptr) {
                                    }
                                }
                            }
                        }
                    }
                }
            }
            scope 5 (inlined std::ptr::const_ptr::<impl *const [u8; 2]>::cast::<u8>) {
            }
            scope 6 (inlined std::slice::from_raw_parts::<'_, u8>) {
                let _12: *const [u8];
                scope 7 (inlined core::ub_checks::check_language_ub) {
                    scope 8 (inlined core::ub_checks::check_language_ub::runtime) {
                    }
                }
                scope 9 (inlined std::mem::size_of::<u8>) {
                }
                scope 10 (inlined std::mem::align_of::<u8>) {
                }
                scope 11 (inlined std::ptr::slice_from_raw_parts::<u8>) {
                    scope 12 (inlined std::ptr::from_raw_parts::<[u8], u8>) {
                    }
                }
            }
        }
    }

    bb0: {
        StorageLive(_2);
        StorageLive(_3);
        _3 = [const 0_u8, const 0_u8];
        _2 = std::boxed::Box::<[u8; 2]>::new(move _3) -> [return: bb1, unwind continue];
    }

    bb1: {
        StorageDead(_3);
        _1 = &_2;
        _10 = copy _2;
        _11 = copy ((_10.0: std::ptr::Unique<[u8; 2]>).0: std::ptr::NonNull<[u8; 2]>) as *const [u8; 2] (Transmute);
        _4 = copy _11 as *const u8 (Transmute);
        _12 = *const [u8] from (copy _4, const 2_usize);
        StorageLive(_5);
        StorageLive(_13);
        _13 = std::ptr::NonNull::<u8> { pointer: copy _4 };
        StorageLive(_15);
        StorageLive(_16);
        _16 = copy _12 as *mut u8 (Transmute);
        _15 = Offset(copy _16, const 2_usize);
        StorageDead(_16);
        _14 = move _15 as *const u8 (PtrToPtr);
        StorageDead(_15);
        _5 = std::slice::Iter::<'_, u8> { ptr: copy _13, end_or_len: copy _14, _marker: const std::marker::PhantomData::<&u8> };
        StorageDead(_13);
        StorageLive(_6);
        _6 = move _5;
        goto -> bb2;
    }

    bb2: {
        StorageLive(_7);
        _8 = &mut _6;
        _7 = <std::slice::Iter<'_, u8> as std::iter::Iterator>::next(move _8) -> [return: bb3, unwind: bb8];
    }

    bb3: {
        _9 = discriminant(_7);
        switchInt(move _9) -> [0: bb6, 1: bb5, otherwise: bb4];
    }

    bb4: {
        unreachable;
    }

    bb5: {
        StorageDead(_7);
        goto -> bb2;
    }

    bb6: {
        StorageDead(_7);
        StorageDead(_6);
        StorageDead(_5);
        drop(_2) -> [return: bb7, unwind continue];
    }

    bb7: {
        StorageDead(_2);
        return;
    }

    bb8 (cleanup): {
        drop(_2) -> [return: bb9, unwind terminate(cleanup)];
    }

    bb9 (cleanup): {
        resume;
    }
}

warning: `rust-136361` (bin "rust-136361") generated 1 warning
error: could not compile `rust-136361` (bin "rust-136361"); 1 warning emitted

Caused by:
  process didn't exit successfully: `/home/jistone/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/rustc --crate-name rust_136361 --edition=2021 src/main.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type bin --emit=dep-info,link -C opt-level=3 -C embed-bitcode=no --check-cfg 'cfg(docsrs,test)' --check-cfg 'cfg(feature, values())' -C metadata=90c88872ae7e4ae0 -C extra-filename=-c76ad6e9ced65976 --out-dir /tmp/rust-136361/target/release/deps -C strip=debuginfo -L dependency=/tmp/rust-136361/target/release/deps -Zcodegen-backend=cranelift` (exit status: 101)

@saethlin saethlin added the I-unsound Issue: A soundness hole (worst kind of bug), see: https://en.wikipedia.org/wiki/Soundness label Feb 1, 2025
@saethlin
Copy link
Member

saethlin commented Feb 1, 2025

Is this a duplicate of #135997?

@theemathas

This comment has been minimized.

@theemathas
Copy link
Contributor

Reproducer without unsafe:

fn main() {
    let vp_ctx: &Box<u8> = &Box::new(0);
    let slf: *const u8 = &raw const **vp_ctx;
    let bytes = std::ptr::slice_from_raw_parts(slf, 1);
    let _x = foo(bytes);
}

fn foo(bytes: *const [u8]) -> *mut u8 {
    bytes as *const u8 as *mut u8 as *const u8 as *mut u8
}

@theemathas
Copy link
Contributor

The issue reproduces even with a ZST.

fn main() {
    let vp_ctx: &Box<()>= &Box::new(());
    let slf: *const () = &raw const **vp_ctx;
    let bytes = std::ptr::slice_from_raw_parts(slf, 1);
    let _x = foo(bytes);
}

fn foo(bytes: *const [()]) -> *mut () {
    bytes as *const () as *mut () as *const () as *mut ()
}

@Noratrieb Noratrieb added P-critical Critical priority I-miscompile Issue: Correct Rust code lowers to incorrect machine code and removed I-prioritize Issue: Indicates that prioritization has been requested for this issue. labels Feb 1, 2025
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 2, 2025
Don't reset cast kind without also updating the operand in `simplify_cast` in GVN

Consider this heavily elided segment of the pre-GVN example code that was committed as a test:

```
          let _4: *const ();
          let _5: *const [()];
          let mut _6: *const ();
          let _7: *mut ();
          let mut _8: *const [()];
          let mut _9: std::boxed::Box<()>;
          let mut _10: *const ();
          /* ... */
          _10 = copy ((_9.0: std::ptr::Unique<()>).0: std::ptr::NonNull<()>) as *const () (Transmute);
          _4 = copy _10;
          _6 = copy _4;
          _5 = *const [()] from (copy _6, copy _11);
          _8 = copy _5;
          _7 = copy _8 as *mut () (PtrToPtr);
```

A malformed optimization was changing `_7` to:

```
          _7 = copy _5 as *mut () (Transmute);
```

(where `_8` was just replaced with `_5` bc of simple copy propagation, that part is not important... the CastKind changing to Transmute is the important part here).

In rust-lang#133324, two new functionalities were implemented:
* Peeking through unsized -> sized PtrToPtr casts whose operand is `AggregateKind::RawPtr`, to turn it into PtrToPtr casts of the base of the aggregate. In this case, this allows us to see that the value of `_7` is just a ptr-to-ptr cast of `_6`.
* Folding a PtrToPtr cast of an operand which is a Transmute cast into just a single Transmute, which (theoretically) allows us to treat `_7` as a transmute into `*mut ()` of the base of the cast of `_10`, which is the place projection of `((_9.0: std::ptr::Unique<()>).0: std::ptr::NonNull<()>)`.

However, when applying those two subsequent optimizations, we must *not* update the CastKind of the final cast *unless* we also update the operand of the cast, since the operand may no longer make sense with the updated CastKind.

In this case, this is problematic because the type of `_8` is `*const [()]`, but that operand in assignment statement of `_7` does *not* get turned into something like `((_9.0: std::ptr::Unique<()>).0: std::ptr::NonNull<()>)` -- **in other words, `try_to_operand` fails** -- because GVN only turns value nodes into locals or consts, not projections of locals. So we fail to update the operand, but we still update the CastKind to Transmute, which means we now are transmuting types of different sizes (a wide pointer and a thin pointer).

r? `@scottmcm` or `@cjgillot`

Fixes rust-lang#136361
@compiler-errors compiler-errors self-assigned this Feb 3, 2025
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 6, 2025
…thlin

Don't reset cast kind without also updating the operand in `simplify_cast` in GVN

Consider this heavily elided segment of the pre-GVN example code that was committed as a test:

```rust
          let _4: *const ();
          let _5: *const [()];
          let mut _6: *const ();
          let _7: *mut ();
          let mut _8: *const [()];
          let mut _9: std::boxed::Box<()>;
          let mut _10: *const ();
          /* ... */
          // Deref a box
          _10 = copy ((_9.0: std::ptr::Unique<()>).0: std::ptr::NonNull<()>) as *const () (Transmute);
          _4 = copy _10;
          _6 = copy _4;
          // Inlined body of `slice::from_raw_parts`, to turn a unit pointer into a slice-of-unit pointer
          _5 = *const [()] from (copy _6, copy _11);
          _8 = copy _5;
          // Cast the raw slice-of-unit pointer back to a unit pointer
          _7 = copy _8 as *mut () (PtrToPtr);
```

A malformed optimization was changing `_7` (which casted the slice-of-unit ptr to a unit ptr) to:

```
          _7 = copy _5 as *mut () (Transmute);
```

...where `_8` was just replaced with `_5` bc of simple copy propagation, that part is not important... the CastKind changing to Transmute is the important part here.

In rust-lang#133324, two new functionalities were implemented:
* Peeking through unsized -> sized PtrToPtr casts whose operand is `AggregateKind::RawPtr`, to turn it into PtrToPtr casts of the base of the aggregate. In this case, this allows us to see that the value of `_7` is just a ptr-to-ptr cast of `_6`.
* Folding a PtrToPtr cast of an operand which is a Transmute cast into just a single Transmute, which (theoretically) allows us to treat `_7` as a transmute into `*mut ()` of the base of the cast of `_10`, which is the place projection of `((_9.0: std::ptr::Unique<()>).0: std::ptr::NonNull<()>)`.

However, when applying those two subsequent optimizations, we must *not* update the CastKind of the final cast *unless* we also update the operand of the cast, since the operand may no longer make sense with the updated CastKind.

In this case, this is problematic because the type of `_8` is `*const [()]`, but that operand in assignment statement of `_7` does *not* get turned into something like `((_9.0: std::ptr::Unique<()>).0: std::ptr::NonNull<()>)` -- **in other words, `try_to_operand` fails** -- because GVN only turns value nodes into locals or consts, not projections of locals. So we fail to update the operand, but we still update the CastKind to Transmute, which means we now are transmuting types of different sizes (a wide pointer and a thin pointer).

r? `@scottmcm` or `@cjgillot`

Fixes rust-lang#136361
Fixes rust-lang#135997
bors added a commit to rust-lang-ci/rust that referenced this issue Feb 7, 2025
…thlin

Don't reset cast kind without also updating the operand in `simplify_cast` in GVN

Consider this heavily elided segment of the pre-GVN example code that was committed as a test:

```rust
          let _4: *const ();
          let _5: *const [()];
          let mut _6: *const ();
          let _7: *mut ();
          let mut _8: *const [()];
          let mut _9: std::boxed::Box<()>;
          let mut _10: *const ();
          /* ... */
          // Deref a box
          _10 = copy ((_9.0: std::ptr::Unique<()>).0: std::ptr::NonNull<()>) as *const () (Transmute);
          _4 = copy _10;
          _6 = copy _4;
          // Inlined body of `slice::from_raw_parts`, to turn a unit pointer into a slice-of-unit pointer
          _5 = *const [()] from (copy _6, copy _11);
          _8 = copy _5;
          // Cast the raw slice-of-unit pointer back to a unit pointer
          _7 = copy _8 as *mut () (PtrToPtr);
```

A malformed optimization was changing `_7` (which casted the slice-of-unit ptr to a unit ptr) to:

```
          _7 = copy _5 as *mut () (Transmute);
```

...where `_8` was just replaced with `_5` bc of simple copy propagation, that part is not important... the CastKind changing to Transmute is the important part here.

In rust-lang#133324, two new functionalities were implemented:
* Peeking through unsized -> sized PtrToPtr casts whose operand is `AggregateKind::RawPtr`, to turn it into PtrToPtr casts of the base of the aggregate. In this case, this allows us to see that the value of `_7` is just a ptr-to-ptr cast of `_6`.
* Folding a PtrToPtr cast of an operand which is a Transmute cast into just a single Transmute, which (theoretically) allows us to treat `_7` as a transmute into `*mut ()` of the base of the cast of `_10`, which is the place projection of `((_9.0: std::ptr::Unique<()>).0: std::ptr::NonNull<()>)`.

However, when applying those two subsequent optimizations, we must *not* update the CastKind of the final cast *unless* we also update the operand of the cast, since the operand may no longer make sense with the updated CastKind.

In this case, this is problematic because the type of `_8` is `*const [()]`, but that operand in assignment statement of `_7` does *not* get turned into something like `((_9.0: std::ptr::Unique<()>).0: std::ptr::NonNull<()>)` -- **in other words, `try_to_operand` fails** -- because GVN only turns value nodes into locals or consts, not projections of locals. So we fail to update the operand, but we still update the CastKind to Transmute, which means we now are transmuting types of different sizes (a wide pointer and a thin pointer).

r? `@scottmcm` or `@cjgillot`

Fixes rust-lang#136361
Fixes rust-lang#135997
@bors bors closed this as completed in 550e035 Feb 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-mir-opt Area: MIR optimizations A-mir-opt-GVN Area: MIR opt Global Value Numbering (GVN) C-bug Category: This is a bug. I-miscompile Issue: Correct Rust code lowers to incorrect machine code I-unsound Issue: A soundness hole (worst kind of bug), see: https://en.wikipedia.org/wiki/Soundness P-critical Critical priority regression-from-stable-to-nightly Performance or correctness regression from stable to nightly. S-has-bisection Status: a bisection has been found for this issue S-has-mcve Status: A Minimal Complete and Verifiable Example has been found for this issue T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.
Projects
None yet
Development

Successfully merging a pull request may close this issue.