`linux-musl`-to-`gnu` cross-compilation not working since Rustup v1.25.1

[giordano]

Background

BinaryBuilder is a toolbox written in Julia which provides a sandbox environment to cross-compile packages for a variety of platforms and compilers, including Rust. The host platform in the environment is always x86_64-linux-musl, from which you can target all the other platforms using the cross-compilers provided by BinaryBuilder. The Rust toolchain is obtained with Rustup, you can find the full script at https://github.com/JuliaPackaging/Yggdrasil/blob/b5fef352621f293f0e0b1091cbe86135a3408277/0_RootFS/Rust/build_tarballs.jl.

Description of the problem with the linker in cross-compilation

Everything worked fine with the toolchains obtained with Rustup v1.24.3, but after upgrading to Rustup v1.25.1 we started having problems with what seems to be a wrong selection of the linker. For example I have working toolchains for Rust v1.61.0 and v1.65.0 obtained with Rustup v1.24.3, but broken toolchains for the same versions of Rust when obtained with Rustup v1.25.1. This seems a good indication that something changed in Rustup (or something related to it) but the problem is independent from the specific version of Rust used.

More specifically on the issue we're encountering, when trying to cross-compile hyperfine for x86_64-linux-gnu I get

sandbox:${WORKSPACE}/srcdir/hyperfine-1.15.0 # echo ${CARGO_BUILD_TARGET}                                                                                                                                         
x86_64-unknown-linux-gnu                                                                                                                                                                                          
sandbox:${WORKSPACE}/srcdir/hyperfine-1.15.0 # cat ${CARGO_HOME}/config.toml                                                                                                                                      
# Configuration file for `cargo`                                                                                                                                                                                  
[target.x86_64-unknown-linux-gnu]                                                                                                                                                                                 
linker = "x86_64-linux-gnu-cc"                                                                                                                                                                                    
[target.x86_64-unknown-linux-musl]                                                                                                                                                                                
linker = "x86_64-linux-musl-cc"                                                                                                                                                                                   
sandbox:${WORKSPACE}/srcdir/hyperfine-1.15.0 # cargo build --release --target x86_64-unknown-linux-gnu                                                                                                            
   Compiling libc v0.2.132                                                                                                                                                                                        
   Compiling proc-macro2 v1.0.43                                                                                                                                                                                  
   Compiling quote v1.0.21                                                                                                                                                                                        
   Compiling num-traits v0.2.15                                                                                                                                                                                   
   Compiling syn v1.0.99                                                                                                                                                                                          
   Compiling indexmap v1.9.1                                                                                                                                                                                      
   Compiling num-integer v0.1.45                                                                                                                                                                                  
   Compiling serde_derive v1.0.144                                                                                                                                                                                
   Compiling serde v1.0.144                                                                                                                                                                                       
   Compiling num-bigint v0.2.6                                                                                                                                                                                    
   Compiling num-complex v0.2.4                                                                                                                                                                                   
   Compiling num-rational v0.2.4                                                                                                                                                                                  
   Compiling num-iter v0.1.43                                                                                                                                                                                     
   Compiling rand_chacha v0.1.1                                                                                                                                                                                   
   Compiling rand_pcg v0.1.2                                                                                                                                                                                      
   Compiling rand v0.6.5                                                                                                                                                                                          
   Compiling memoffset v0.6.5                                                                                                                                                                                     
   Compiling serde_json v1.0.85                                                                                                                                                                                   
   Compiling anyhow v1.0.64
   Compiling regex v1.6.0
error: linking with `x86_64-linux-musl-cc` failed: exit status: 1
  |
  = note: "x86_64-linux-musl-cc" "-m64" "/tmp/rustcedIU7R/symbols.o" "/workspace/srcdir/hyperfine-1.15.0/target/release/build/num-traits-e4b2e0af064b2249/build_script_build-e4b2e0af064b2249.build_script_build.1
513a843-cgu.0.rcgu.o" "/workspace/srcdir/hyperfine-1.15.0/target/release/build/num-traits-e4b2e0af064b2249/build_script_build-e4b2e0af064b2249.build_script_build.1513a843-cgu.1.rcgu.o" "/workspace/srcdir/hyperf
ine-1.15.0/target/release/build/num-traits-e4b2e0af064b2249/build_script_build-e4b2e0af064b2249.build_script_build.1513a843-cgu.2.rcgu.o" "/workspace/srcdir/hyperfine-1.15.0/target/release/build/num-traits-e4b2
e0af064b2249/build_script_build-e4b2e0af064b2249.build_script_build.1513a843-cgu.3.rcgu.o" "/workspace/srcdir/hyperfine-1.15.0/target/release/build/num-traits-e4b2e0af064b2249/build_script_build-e4b2e0af064b224
9.build_script_build.1513a843-cgu.4.rcgu.o" "/workspace/srcdir/hyperfine-1.15.0/target/release/build/num-traits-e4b2e0af064b2249/build_script_build-e4b2e0af064b2249.build_script_build.1513a843-cgu.5.rcgu.o" "/w
orkspace/srcdir/hyperfine-1.15.0/target/release/build/num-traits-e4b2e0af064b2249/build_script_build-e4b2e0af064b2249.3jtcfnczj1416vo.rcgu.o" "-Wl,--as-needed" "-L" "/workspace/srcdir/hyperfine-1.15.0/target/re
lease/deps" "-L" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/workspace/srcdir/hyperfine-1.15.0/target/release/deps/libautocfg-f
df990b8c499d903.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-05737cf45bd30456.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-
unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-9f873b61fdec9b03.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/li
bobject-7f13930fcac1846f.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-098633b847612f3b.rlib" "/opt/x86_64-linux-musl/toolchains/1.
65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-f14b73d282b0245e.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-g
nu/lib/libgimli-2c5b4433ebc1d822.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-59591a7b405fe395.rlib" "/opt/x86_64-linux-mu
sl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-384947c6d5f697ff.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_6
4-unknown-linux-gnu/lib/libhashbrown-b08a86c6880b47a8.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-58adeee671f9ba8e.rlib" "/o
pt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-f156b880fc73e7f0.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/
rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-4458c5022988e1ab.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-02
e61e5ec4aa9e8b.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-a0d9b33b5161957b.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_6
4-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-04cec55a79224c36.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc
-3fb6d8496dc7d6a6.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-7d46c016841a97d4.rlib" "/opt/x86_64-linux-musl/to
olchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-a1f7b8b60464cc57.rlib" "/opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-l
inux-gnu/lib/libcompiler_builtins-272ca28f0b8538d5.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/opt/x86_64-linux-musl/toolchains
/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/workspace/srcdir/hyperfine-1.15.0/target/release/build/num-traits-e4b2e0af064b2249/build_script_build-e4b2e0af064b2249" "-Wl,--
gc-sections" "-pie" "-Wl,-zrelro,-znow" "-nodefaultlibs"
  = note: /opt/x86_64-linux-musl/bin/../lib/gcc/x86_64-linux-musl/8.1.0/../../../../x86_64-linux-musl/bin/ld: /opt/x86_64-linux-musl/toolchains/1.65.0-x86_64-unknown-linux-musl/lib/rustlib/x86_64-unknown-linux-
gnu/lib/libstd-05737cf45bd30456.rlib(std-05737cf45bd30456.std.428b0c25-cgu.0.rcgu.o): in function `std::sys::unix::os::glibc_version':
          /rustc/897e37553bba8b42751c67658967889d11ecd120/library/std/src/sys/unix/os.rs:663: undefined reference to `gnu_get_libc_version'
          collect2: error: ld returned 1 exit status 
           
  = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link

Despite the fact that the Cargo config file indicates that the linker for x86_64-unknown-linux-gnu is x86_64-linux-gnu-cc and the target platform is specified with both the environment variable CARGO_BUILD_TARGET (which is exported) and the --target option to cargo build, Cargo uses x86_64-linux-musl-cc to link object files compiled for x86_64-unknown-linux-gnu, which of course leads to errors like

undefined reference to `gnu_get_libc_version'

Unfortunately this isn't very easy to reproduce because there are no released versions of BinaryBuilder which provide the broken Rust toolchain, I discovered the issues by doing local tests before cutting a new release. If someone is willing to install Julia I can try provide lengthy instructions to try and reproduce this locally, but perhaps someone has a clue of what could lead to a wrong selection of the linker in cross-compilation, or what changed between Rustup v1.24.3 and v1.25.1.

In case someone wants to compare a working a non-working toolchain, here they are (remember that the host is x86_64-linux-musl):

• working Rust v1.65.0 (based on Rustup v1.24.3)
• non-working Rust v1.65.0 (based on Rustup v1.25.1)

It's also totally possible that the script to obtain the toolchain has some problems, we are far from being experienced Rust developers, so any help with improving that would be appreciated, but I'd like to reiterate that things worked fine until Rustup v1.24.3, problems emerged with v1.25.1.

[rami3l]

@giordano I see that JuliaPackaging/BinaryBuilderBase.jl#307 has been closed. Is this issue still relevant?

[giordano]

I see that JuliaPackaging/BinaryBuilderBase.jl#307 has been closed. Is this issue still relevant?

We updated the Rust toolchain, but we're still using Rustup 1.24 for doing that, because of the problems outlined above: https://github.com/JuliaPackaging/Yggdrasil/blob/7f3f8a1bd526f964108c9e881f3e9d088fd155ac/0_RootFS/Rust/build_tarballs.jl#L18-L20

[rami3l]

I see that JuliaPackaging/BinaryBuilderBase.jl#307 has been closed. Is this issue still relevant?

We updated the Rust toolchain, but we're still using Rustup 1.24 for doing that, because of the problems outlined above: https://github.com/JuliaPackaging/Yggdrasil/blob/7f3f8a1bd526f964108c9e881f3e9d088fd155ac/0_RootFS/Rust/build_tarballs.jl#L18-L20

@giordano Thanks for your prompt response! Since I wasn't in the Rustup team during that time, I guess I'd like to first make sure that the issue is still there. Would you mind trying again with the latest stable (v1.27.1)?

Also, GCC 8.1 (from 2018) being used in the linkage is very old even back in 2022... Does an upgrade to GCC changes anything? Or, does cargo zigbuild --target x86_64-unknown-linux-musl fix it for you?

Finally, as hyperfine is a well-known crate, I guess we can ping its maintainer(s) to see if they have encountered anything similar.

[giordano]

I can't easily check right now because I'll be quite busy with work over the next 2-3 weeks, but I wanted to comment on a bit:

Also, GCC 8.1 (from 2018) being used in the linkage is very old even back in 2022...

We generally purposefully use "old" GCC/binutils versions to to maximise compatibility: we produce generic binaries to be used by as many users as possible, using super new toolchains versions has a compatibility cost. Certainly when I'll get the time to get back to this issue I can try using newer toolchains for the sake of checking that, but that's beside the point: the main issue is that rustc is targeting the wrong platform during cross-compilation. And I seem to remember I didn't have that issue only with hyperfine, it was a generic issue with the Rust toolchain coming out of Rustup 1.25, hence why I opened this issue in Rustup, and not in Rustc nor hyperfine.

Thanks for taking the time to look into this.

[djc]

@giordano I definitely want to make sure we can get you back onto a newer rustup. I guess the hard part here is that this feels like it's at the intersection of rustup itself and the compiler toolchain? It seems very surprising to me that changes in rustup can cause the kind of linking failures. Would it be feasible to bisect rustup changes between the working and first failing release?