`rustc` proxy may not invoke the correct `rustc` `override` when used with external `cargo`.

[cr1901]

Problem
This too may be a cargo problem, but since it involves the rustc proxy, I'll open here first.

Consider a crate with the following src/lib.rs and Cargo.toml:

#![no_std]

use bit_reverse;

#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
        assert_eq!(2 + 2, 4);
    }
}

[package]
name = "rust-check"
version = "0.1.0"
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
bit_reverse = "*"

My default rustup toolchain is nightly, and I have an override to use stable in this directory. rustup is on the path first, so the proxies will be picked up first. In addition, I have an external cargo that I built for testing that I'm using instead of the proxy that rustup provides. When invoking this particular cargo binary, which is not a proxy, I get the following error:

william@xubuntu-dtrain:~/Projects/toolchains/rust-check$ ../cargo/target/debug/cargo build
    Updating crates.io index
   Compiling bit_reverse v0.1.8
   Compiling rust-check v0.1.0 (/home/william/Projects/toolchains/rust-check)
error[E0514]: found crate `bit_reverse` compiled by an incompatible version of rustc
 --> src/lib.rs:3:5
  |
3 | use bit_reverse;
  |     ^^^^^^^^^^^
  |
  = help: please recompile that crate using this compiler (rustc 1.51.0 (2fd73fabe 2021-03-23))
  = note: the following crate versions were found:
          crate `bit_reverse` compiled by rustc 1.53.0-nightly (9d9c2c92b 2021-04-19): /home/william/Projects/toolchains/rust-check/target/debug/deps/libbit_reverse-4ef4b144e0bd8cec.rmeta

error: aborting due to previous error

error: could not compile `rust-check`

To learn more, run the command again with --verbose.

Steps

1. Compile your own cargo from source, e.g. git clone https://github.com/rust-lang/cargo; cd cargo; cargo build.
2. Create a crate such as above with any external dependency that's used.
3. In that crate directory, set up a rustup override to anything besides your default toolchain.
4. Attempt to compile the crate using your shiny new cargo, and not the cargo proxy provided by rustup.

Possible Solution(s)
Since cargo invokes rustc, and the rustc proxy is first on my path, the external cargo is invoking the rustc proxy. However, the override is not taking effect except for the final crate. I can confirm that all artifacts are built with the default rustc in my case (nightly), until the final crate is compiled. At this point, rustc invoked from external cargo starts honoring the override, and crate compilation fails due to version mismatch.

Is this not supported? It's fine if using an external cargo unsupported, but I do wonder if:

1. A better error message/warning could notify the user that this is not supported.
2. How does one debug cargo in the presence of rustup on the path?
3. Additionally, I thought cargo could auto-detect whether there's a version-mismatch that warrants recompiling dependencies? Does this setup break auto-detect?

Notes

System Information:

william@xubuntu-dtrain:~/Projects/toolchains/rust-check$ uname -a
Linux xubuntu-dtrain 5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

william@xubuntu-dtrain:~/Projects/toolchains/rust-check$ rustup -V
rustup 1.24.1 (a01bd6b0d 2021-04-27)
info: This is the version for the rustup toolchain manager, not the rustc compiler.
info: The currently active `rustc` version is `rustc 1.51.0 (2fd73fabe 2021-03-23)`

william@xubuntu-dtrain:~/Projects/toolchains/rust-check$ rustup show
Default host: x86_64-unknown-linux-gnu
rustup home:  /home/william/.rustup

installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu
nightly-x86_64-unknown-linux-gnu
msp430-fix
shared-rust

active toolchain
----------------

stable-x86_64-unknown-linux-gnu (directory override for '/home/william/Projects/toolchains/rust-check')
rustc 1.51.0 (2fd73fabe 2021-03-23)

william@xubuntu-dtrain:~/Projects/toolchains/rust-check$ rustc +nightly -V
rustc 1.53.0-nightly (9d9c2c92b 2021-04-19)

[kinnison]

Rustup is not meant to be used in this fashion certainly. I think you would probably do best to discuss with the Cargo team how they develop and debug in this kind of situation. I'm not sure how overrides, partial toolchains, and so on would interact in this kind of situation.

[kinnison]

If it turns out that we're doing something wrong within Rustup itself once you've had your discussions with Cargo, we can revisit it on this issue.

[ehuss]

This may be very difficult to solve. Cargo would somehow need to discover which toolchain should be used, which can be difficult to discover correctly. That would need to be cached, and the existing caching is already a bit tricky and buggy.

There is documentation at https://doc.crates.io/contrib/process/working-on-cargo.html#running-cargo for how to work around this issue.

[cr1901]

@ehuss

There is documentation at https://doc.crates.io/contrib/process/working-on-cargo.html#running-cargo for how to work around this issue.

Ahhh, this is very useful to know. Anyways, if it's very difficult to solve, then it may as well remain unsupported. As a possible solution: how difficult would it be for rustup proxies to notify a non-proxy cargo that its running a proxy rustc (doesn't matter which toolchain, just that the proxy was run)? Then non-proxy cargo could print out a nice big warning that "this is unsupported, expect things to break" for the next person who tries what I did?

[kinnison]

When Rustup invokes something via a proxy we set the RUSTUP_TOOLCHAIN environment variable I believe. You might be able to use that to detect it in Cargo, but honestly it's not a guaranteed API so I wouldn't want to tie Cargo to it. In reality it's probably better to use the mechanisms the Cargo team recommend.

[cr1901]

In reality it's probably better to use the mechanisms the Cargo team recommend.

Right, but there will be other people, like me, who still try what I did first before reading the docs. My question is more like "can't there be an error/big warning that points back to the Cargo docs?", knowing that people will experiment without reading the docs? And if so, is there a way for cargo to know that you're doing something unsupported from the rustup side of things.

If RUSTUP_TOOLCHAIN is not a stable API, and there's no way to reliably detect the proxy from cargo, then I guess "read the docs when cargo invokes the wrong rustc" is the correct answer. I supposed "cargo stopped working when it previously worked" if a big enough warning sign. Anyways I'll migrate both my issues over if infra hasn't done so already when I get the chance, and come back to this issue if necessary.

[cr1901]

Should this issue be xferred to cargo as well? I can't seem to tag @rust-lang/infra, like the previous issue did.

[kinnison]

@rust-lang/infra could one of you move this to cargo ?