fix output from runners to IOBuffer #201

KristofferC · 2021-12-30T14:37:02Z

src/Runner.jl

test/runners.jl

giordano · 2021-12-30T15:03:11Z

Uhm, this is a bit unexpected:

julia> out = IOBuffer(); err = IOBuffer(); run_interactive(preferred_runner()(mktempdir(); platform=Platform("x86_64", "linux"; libc="musl")), `/bin/bash -c "echo hello; echo world >&2"`, stdout=out, stderr=err); @show String(take!(out)); @show String(take!(err));
world
String(take!(out)) = "hello\n"
String(take!(err)) = ""

But it's also unrelated to your changes, I see the same before #197.

giordano · 2021-12-30T15:04:46Z

Tests in BinaryBuilder with the Docker runner are failing: https://dev.azure.com/JuliaPackaging/BinaryBuilder.jl/_build/results?buildId=15585&view=logs&j=54759826-d195-5ca2-1571-21806663cd36&t=9eb882d6-5a34-5e37-1506-d0b4bf16214f

Test Summary:              | Pass  Total
Auditor - cxxabi selection |   29     29
  0.000292 seconds (636 allocations: 54.172 KiB)
  0.000128 seconds (367 allocations: 32.141 KiB)
  0.000262 seconds (1.18 k allocations: 103.438 KiB)
llvm-objdump: error: 'libfoo.0.dylib': truncated or malformed object (bad section index: 3 for symbol at index 2)
Auditor - .la removal: Error During Test at /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/auditing.jl:289
  Test threw exception
  Expression: begin
    build_output_meta = autobuild(build_path, "libfoo", v"1.0.0", [DirectorySource(build_tests_dir)], libfoo_autotools_script * "\ntouch \${prefix}/lib/libbar.la", [platform], libfoo_products, Dependency[])
end
  failed process: Process(`docker run --rm --privileged -w //workspace/ -v /data/staticfloat/bb_azp/rootfs/agent_3/depot/artifacts/597f3ec099059ced3acae4427576fedbaa3fd21c:/opt/x86_64-apple-darwin14/PlatformSupport-2021.8.10:ro -v /data/staticfloat/bb_azp/rootfs/agent_3/depot/artifacts/303e65f6896f295459c81aa311b5eb993f702d89:/opt/x86_64-apple-darwin14/GCCBootstrap-4.8.5:ro -v /data/staticfloat/bb_azp/rootfs/agent_3/depot/artifacts/aac62b1de8767c6b83ea3e15753a06afdad87e43:/opt/x86_64-linux-musl/LLVMBootstrap-12.0.0:ro -v /data/staticfloat/bb_azp/rootfs/agent_3/depot/artifacts/047fec931a6c54eb4d5df41acf4d35c704dac1c1:/opt/x86_64-linux-musl/PlatformSupport-2021.8.10:ro -v /data/staticfloat/bb_azp/rootfs/agent_3/depot/artifacts/e5229ab63cfb1c1deb9c8f461889787e648eaeba:/opt/x86_64-linux-musl/GCCBootstrap-4.8.5:ro -v /data/staticfloat/bb_azp/rootfs/agent_3/tmp/jl_eykGoW/build/x86_64-apple-darwin/SGAto66t/x86_64-apple-darwin14-libgfortran3-cxx03/destdir/lib:/workspace -v /data/staticfloat/bb_azp/rootfs/agent_3/tmp/jl_Frj0ST:/opt/bin -v /data/staticfloat/bb_azp/rootfs/agent_3/tmp/jl_COJkxT:/opt/toolchains -v /data/staticfloat/bb_azp/rootfs/agent_3/storage/ccache:/root/.ccache -e FC_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -e PATH=/opt/bin/x86_64-apple-darwin14-libgfortran3-cxx03:/opt/x86_64-apple-darwin14/bin:/opt/bin/x86_64-linux-musl-cxx11:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/x86_64-linux-musl-cxx11/destdir/bin:/workspace/destdir/bin -e CXX=c++ -e HOSTFC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -e HOSTSTRIP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -e nproc=4 -e CC=cc -e OBJCOPY_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -e FC_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -e HISTFILE=/meta/.bash_history -e LD_LIBRARY_PATH=/usr/lib/csl-musl-x86_64:/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/x86_64-apple-darwin14/x86_64-apple-darwin14/lib:/opt/x86_64-apple-darwin14/lib:/workspace/destdir/lib64:/workspace/destdir/lib -e AS_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -e HOSTOBJCOPY=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -e BUILD_AR=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -e target=x86_64-apple-darwin14 -e HOSTRANLIB=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -e HOSTLIPO=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -e VERBOSE=false -e BUILD_STRIP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -e MESON_HOST_TOOLCHAIN=/opt/toolchains/x86_64-linux-musl-cxx11/host_x86_64-linux-musl.meson -e BUILD_RANLIB=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -e BUILD_CXX=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -e STRIP_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -e HOSTAR=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -e USER=ubuntu -e CMAKE_HOST_TOOLCHAIN=/opt/toolchains/x86_64-linux-musl-cxx11/host_x86_64-linux-musl.cmake -e LLVM_TARGET=x86_64-apple-darwin14 -e LD=/opt/bin/x86_64-apple-darwin14-libgfortran3-cxx03/ld -e HOSTOBJDUMP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -e proc_family=intel -e includedir=/workspace/destdir/include -e BUILD_LD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -e V=false -e BUILD_AS=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -e host_includedir=/workspace/x86_64-linux-musl-cxx11/destdir/include -e SHELL=/bin/bash -e LD_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -e BUILD_READELF=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -e rust_host=x86_64-unknown-linux-musl -e HOSTCXX=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -e STRIP_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -e LD_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -e BUILD_OBJDUMP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -e bindir=/workspace/destdir/bin -e exeext= -e PKG_CONFIG_SYSROOT_DIR=/workspace/destdir -e SOURCE_DATE_EPOCH=0 -e TERM=screen -e NM_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -e MESON_TARGET_TOOLCHAIN=/opt/toolchains/x86_64-apple-darwin14-libgfortran3-cxx03/target_x86_64-apple-darwin14.meson -e OBJCOPY_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -e CXX_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -e HOSTCC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -e RANLIB_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -e BUILD_LIPO=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -e rust_target=x86_64-apple-darwin -e CCACHE_COMPILERCHECK=content -e BUILD_OBJCOPY=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -e HOSTLD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -e HOSTNM=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -e BUILD_CC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -e AR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -e USE_CCACHE=true -e HOSTREADELF=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -e NM_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -e CXX_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -e bb_full_target=x86_64-apple-darwin14-libgfortran3-cxx03 -e 'HIDDEN_PS1=\[\]sandbox\[\]:\[\]${PWD//$WORKSPACE/$\{WORKSPACE\}}\[\] \$ ' -e LIPO_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -e prefix=/workspace/destdir -e nbits=64 -e DSYMUTIL_FOR_BUILD=dsymutil -e BUILD_DSYMUTIL=dsymutil -e MACHTYPE=x86_64-linux-musl -e LIPO_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -e FC=gfortran -e HOSTDSYMUTIL=dsymutil -e BUILD_FC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -e CC_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -e RANLIB_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -e host_bindir=/workspace/x86_64-linux-musl-cxx11/destdir/bin -e LLVM_HOST_TARGET=x86_64-linux-musl -e READELF_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -e READELF_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -e MACOSX_DEPLOYMENT_TARGET=10.8 -e libdir=/workspace/destdir/lib -e host_libdir=/workspace/x86_64-linux-musl-cxx11/destdir/lib -e CC_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -e host_prefix=/workspace/x86_64-linux-musl-cxx11/destdir -e OBJDUMP_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -e AS_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -e DSYMUTIL_BUILD=dsymutil -e BUILD_NM=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -e dlext=dylib -e CMAKE_TARGET_TOOLCHAIN=/opt/toolchains/x86_64-apple-darwin14-libgfortran3-cxx03/target_x86_64-apple-darwin14.cmake -e OBJDUMP_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -e SRC_NAME= -e PKG_CONFIG_PATH=/workspace/destdir/lib/pkgconfig:/workspace/destdir/lib64/pkgconfig:/workspace/destdir/share/pkgconfig -e ZERO_AR_DATE=1 -e bb_target=x86_64-apple-darwin14 -e WORKSPACE=/workspace -e HOSTAS=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -e AR_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -i -i julia_binarybuilder_rootfs:v2021.7.13-545434 /bin/bash -c 'llvm-objdump -d libfoo.0.dylib'`, ProcessExited(1)) [1]
  
  Stacktrace:
    [1] pipeline_error
      @ ./process.jl:525 [inlined]
    [2] run(::Base.CmdRedirect; wait::Bool)
      @ Base ./process.jl:440
    [3] run
      @ ./process.jl:438 [inlined]
    [4] run_interactive(dr::DockerRunner, cmd::Cmd; stdin::Nothing, stdout::IOBuffer, stderr::Base.DevNull, verbose::Bool)
      @ BinaryBuilderBase /data/staticfloat/bb_azp/rootfs/agent_3/depot/packages/BinaryBuilderBase/xdpR0/src/DockerRunner.jl:215
    [5] instruction_mnemonics(path::String, platform::Platform)
      @ BinaryBuilder.Auditor /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/src/auditor/instruction_set.jl:44
    [6] analyze_instruction_set(oh::MachOHandle{IOStream}, platform::Platform; verbose::Bool)
      @ BinaryBuilder.Auditor /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/src/auditor/instruction_set.jl:148
    [7] check_isa(oh::MachOHandle{IOStream}, platform::Platform, prefix::Prefix; verbose::Bool, silent::Bool)
      @ BinaryBuilder.Auditor /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/src/Auditor.jl:293
    [8] (::BinaryBuilder.Auditor.var"#94#100"{Platform, Bool, Bool, Bool, Prefix, String, Vector{String}, String})(oh::MachOHandle{IOStream})
      @ BinaryBuilder.Auditor /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/src/Auditor.jl:91
    [9] readmeta(f::BinaryBuilder.Auditor.var"#94#100"{Platform, Bool, Bool, Bool, Prefix, String, Vector{String}, String}, file::String)
      @ ObjectFile /data/staticfloat/bb_azp/rootfs/agent_3/depot/packages/ObjectFile/lbue6/src/Abstract/ObjectHandle.jl:158
   [10] audit(prefix::Prefix, src_name::String; io::Base.TTY, platform::Platform, verbose::Bool, silent::Bool, autofix::Bool, has_csl::Bool, require_license::Bool)
      @ BinaryBuilder.Auditor /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/src/Auditor.jl:84
   [11] autobuild(dir::AbstractString, src_name::AbstractString, src_version::VersionNumber, sources::Vector{var"#s952"} where var"#s952"<:AbstractSource, script::AbstractString, platforms::Vector{T} where T, products::Vector{var"#s951"} where var"#s951"<:Product, dependencies::Vector{var"#s950"} where var"#s950"<:AbstractDependency; verbose::Bool, debug::Bool, skip_audit::Bool, ignore_audit_errors::Bool, autofix::Bool, code_dir::Union{Nothing, String}, require_license::Bool, kwargs::Any)
      @ BinaryBuilder /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/src/AutoBuild.jl:804
   [12] autobuild(dir::AbstractString, src_name::AbstractString, src_version::VersionNumber, sources::Vector{var"#s942"} where var"#s942"<:AbstractSource, script::AbstractString, platforms::Vector{T} where T, products::Vector{var"#s941"} where var"#s941"<:Product, dependencies::Vector{var"#s232"} where var"#s232"<:AbstractDependency)
      @ BinaryBuilder /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/src/AutoBuild.jl:676
   [13] (::var"#77#79"{String, Platform})()
      @ Main ./none:0
   [14] with_logstate(f::Function, logstate::Any)
      @ Base.CoreLogging ./logging.jl:491
   [15] with_logger
      @ ./logging.jl:603 [inlined]
   [16] #collect_test_logs#47
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/logging.jl:51 [inlined]
   [17] collect_test_logs
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/logging.jl:50 [inlined]
   [18] #match_logs#48
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/logging.jl:186 [inlined]
   [19] (::var"#76#78"{Platform})(build_path::String)
      @ Main /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/auditing.jl:289
   [20] mktempdir(fn::var"#76#78"{Platform}, parent::String; prefix::String)
      @ Base.Filesystem ./file.jl:729
   [21] mktempdir(fn::Function, parent::String) (repeats 2 times)
      @ Base.Filesystem ./file.jl:727
   [22] macro expansion
      @ /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/auditing.jl:287 [inlined]
   [23] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [24] top-level scope
      @ /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/auditing.jl:285
Auditor - .la removal: Error During Test at /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/auditing.jl:308
  Test threw exception
  Expression: haskey(build_output_meta, platform)
  MethodError: no method matching haskey(::Nothing, ::Platform)
  Closest candidates are:
    haskey(::DataStructures.DefaultOrderedDict, ::Any...) at /data/staticfloat/bb_azp/rootfs/agent_3/depot/packages/DataStructures/vSp4s/src/delegate.jl:21
    haskey(::DataStructures.MultiDict, ::Any...) at /data/staticfloat/bb_azp/rootfs/agent_3/depot/packages/DataStructures/vSp4s/src/delegate.jl:21
    haskey(::DataStructures.SwissDict, ::Any) at /data/staticfloat/bb_azp/rootfs/agent_3/depot/packages/DataStructures/vSp4s/src/swiss_dict.jl:534
    ...
  Stacktrace:
   [1] (::var"#76#78"{Platform})(build_path::String)
     @ Main /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/auditing.jl:308
   [2] mktempdir(fn::var"#76#78"{Platform}, parent::String; prefix::String)
     @ Base.Filesystem ./file.jl:729
   [3] mktempdir(fn::Function, parent::String) (repeats 2 times)
     @ Base.Filesystem ./file.jl:727
   [4] macro expansion
     @ /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/auditing.jl:287 [inlined]
   [5] macro expansion
     @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
   [6] top-level scope
     @ /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/auditing.jl:285
Auditor - .la removal: Error During Test at /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/auditing.jl:284
  Got exception outside of a @test
  MethodError: no method matching getindex(::Nothing, ::Platform)
  Stacktrace:
    [1] (::var"#76#78"{Platform})(build_path::String)
      @ Main /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/auditing.jl:309
    [2] mktempdir(fn::var"#76#78"{Platform}, parent::String; prefix::String)
      @ Base.Filesystem ./file.jl:729
    [3] mktempdir(fn::Function, parent::String) (repeats 2 times)
      @ Base.Filesystem ./file.jl:727
    [4] macro expansion
      @ /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/auditing.jl:287 [inlined]
    [5] macro expansion
      @ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Test/src/Test.jl:1151 [inlined]
    [6] top-level scope
      @ /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/auditing.jl:285
    [7] include(fname::String)
      @ Base.MainInclude ./client.jl:444
    [8] top-level scope
      @ /data/staticfloat/bb_azp/rootfs/agent_3/_work/1/s/test/runtests.jl:71
    [9] include(fname::String)
      @ Base.MainInclude ./client.jl:444
   [10] top-level scope
      @ none:6
   [11] eval
      @ ./boot.jl:360 [inlined]
   [12] exec_options(opts::Base.JLOptions)
      @ Base ./client.jl:261
   [13] _start()
      @ Base ./client.jl:485
Test Summary:         | Pass  Error  Total
Auditor - .la removal |    5      3      8

KristofferC · 2021-12-30T15:06:46Z

With this PR? In that case, I guess just do the revert and I'll go through this properly again?

I don't really see how llvm-objdump: error: 'libfoo.0.dylib': truncated or malformed object (bad section index: 3 for symbol at index 2) was caused by this though.

giordano · 2021-12-30T15:08:22Z

Yes, I'm running the tests in JuliaPackaging/BinaryBuilder.jl#1032

giordano · 2021-12-30T15:10:53Z

I don't really see how llvm-objdump: error: 'libfoo.0.dylib': truncated or malformed object (bad section index: 3 for symbol at index 2) was caused by this though.

That's unrelated, I've seen it before: https://dev.azure.com/JuliaPackaging/BinaryBuilder.jl/_build/results?buildId=15571&view=logs&j=54759826-d195-5ca2-1571-21806663cd36&t=9eb882d6-5a34-5e37-1506-d0b4bf16214f&l=813

KristofferC · 2021-12-30T15:16:49Z

Aha, okay. Well, debugging that will probably take a while so #199 it is I guess.

giordano · 2021-12-30T15:22:08Z

Note that isn't only the Docker runner, also the other jobs eventually failed, they just took longer.

KristofferC · 2021-12-30T15:28:05Z

Maybe we should set up reverse dependency testing on BinaryBuilder on this repo? What's the reason this is a separate package anyway? The only thing depending on it is BinaryBuilder and they seem to be quite closely coupled.

giordano · 2021-12-30T15:58:17Z

Yeah, the separation is moot and complicated. In principle also BinaryProvider could depend on this package, there's a lot of code duplication between BinaryProvider and BinaryBuilder (with the result people end up opening PRs to the wrong package: JuliaPackaging/BinaryBuilder.jl#793 (comment)), but BinaryProvider is practically unmaintained today, so we have zero motivation for actually doing the work to integrate BinaryBuilderBase in BinaryProvider.

BinaryBuilderBase has only the main building blocks needed by BinaryBuilder: the runners (to be eventually replaced by Sandbox.jl), the dependencies, the sources, the products, etc... BinaryBuilder is the "integration" of BinaryBuilderBase: it does the actual building of the recipes, the audit passes, and the wizard (BinaryBuilderBase knows nothing about them).

Good part of the motivation for the separation was an infrastructure problem: CI of BinaryBuilder takes a lot of time and storage, it now runs on dedicated hardware, but we can have only few (three at the moment) runners, so opening multiple PRs at the same time is always problematic. Splitting the "base" components into their own package let us develop much quicker, with faster CI (ironic I'm saying this in a PR whose CI jobs just timed out, but I believe it's a network issue on GitHub side) which can run on GitHub-hosted runners. As a matter of fact, I never ran BinaryBuilder tests locally, it just takes too long, while I'm able to test BinaryBuilderBase. CI for this package used to be way faster initially, less than 10 minutes, but it keeps growing. Of course everything works as long as we have enough unit tests here, instead of relying on integration tests in BinaryBuilder, but that's not always the case.

KristofferC · 2021-12-30T16:10:40Z

Of course everything works as long as we have enough unit tests here, instead of relying on integration tests in BinaryBuilder, but that's not always the case.

Yeah, but that feels kind of hard. Just thinking about what we can do to not end up in a situation like the current one where BinaryBuilder is broken.

giordano · 2021-12-30T21:50:43Z

Alright, I have a small reproducer.

On master (with #197 reverted)

julia> run_interactive(preferred_runner()(mktempdir(); platform=Platform("x86_64", "linux")), `/bin/bash -c 'false'`)
[ Info: Running privileged container via `sudo`, may ask for your password:
[sudo] password for mose: 
ERROR: failed process: Process(setenv(`sudo -E FC_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -E PATH=/opt/bin/x86_64-linux-gnu-libgfortran3-cxx03:/opt/x86_64-linux-gnu/bin:/opt/bin/x86_64-linux-musl-cxx11:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/x86_64-linux-musl-cxx11/destdir/bin:/workspace/destdir/bin -E CXX=c++ -E HOSTFC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -E HOSTSTRIP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -E nproc=3 -E CC=cc -E OBJCOPY_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -E FC_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -E HISTFILE=/meta/.bash_history -E LD_LIBRARY_PATH=/usr/lib/csl-musl-x86_64:/usr/lib/csl-glibc-x86_64:/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib64:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib:/workspace/destdir/lib64:/workspace/destdir/lib -E AS_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -E HOSTOBJCOPY=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -E BUILD_AR=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -E target=x86_64-linux-gnu -E HOSTRANLIB=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -E HOSTLIPO=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -E VERBOSE=false -E BUILD_STRIP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -E MESON_HOST_TOOLCHAIN=/opt/toolchains/x86_64-linux-musl-cxx11/host_x86_64-linux-musl.meson -E BUILD_RANLIB=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -E BUILD_CXX=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -E STRIP_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -E HOSTAR=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -E USER=mose -E CMAKE_HOST_TOOLCHAIN=/opt/toolchains/x86_64-linux-musl-cxx11/host_x86_64-linux-musl.cmake -E LLVM_TARGET=x86_64-linux-gnu -E HOSTOBJDUMP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -E proc_family=intel -E includedir=/workspace/destdir/include -E BUILD_LD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -E V=false -E BUILD_AS=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -E host_includedir=/workspace/x86_64-linux-musl-cxx11/destdir/include -E SHELL=/bin/bash -E LD_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -E BUILD_READELF=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -E rust_host=x86_64-unknown-linux-musl -E HOSTCXX=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -E STRIP_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -E LD_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -E BUILD_OBJDUMP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -E bindir=/workspace/destdir/bin -E exeext= -E PKG_CONFIG_SYSROOT_DIR=/workspace/destdir -E SOURCE_DATE_EPOCH=0 -E TERM=screen -E NM_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -E MESON_TARGET_TOOLCHAIN=/opt/toolchains/x86_64-linux-gnu-libgfortran3-cxx03/target_x86_64-linux-gnu.meson -E OBJCOPY_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -E CXX_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -E HOSTCC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -E RANLIB_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -E BUILD_LIPO=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -E rust_target=x86_64-unknown-linux-gnu -E CCACHE_COMPILERCHECK=content -E BUILD_OBJCOPY=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -E HOSTLD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -E HOSTNM=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -E BUILD_CC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -E AR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -E USE_CCACHE=true -E HOSTREADELF=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -E NM_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -E CXX_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -E bb_full_target=x86_64-linux-gnu-libgfortran3-cxx03 -E 'HIDDEN_PS1=\[\]sandbox\[\]:\[\]${PWD//$WORKSPACE/$\{WORKSPACE\}}\[\] \$ ' -E LIPO_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -E prefix=/workspace/destdir -E nbits=64 -E DSYMUTIL_FOR_BUILD=dsymutil -E BUILD_DSYMUTIL=dsymutil -E 'GNU_LIBC_VERSION=glibc 2.12.2' -E MACHTYPE=x86_64-linux-musl -E LIPO_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -E FC=gfortran -E HOSTDSYMUTIL=dsymutil -E BUILD_FC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -E CC_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -E RANLIB_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -E host_bindir=/workspace/x86_64-linux-musl-cxx11/destdir/bin -E LLVM_HOST_TARGET=x86_64-linux-musl -E READELF_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -E READELF_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -E libdir=/workspace/destdir/lib -E host_libdir=/workspace/x86_64-linux-musl-cxx11/destdir/lib -E CC_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -E host_prefix=/workspace/x86_64-linux-musl-cxx11/destdir -E OBJDUMP_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -E AS_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -E DSYMUTIL_BUILD=dsymutil -E BUILD_NM=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -E dlext=so -E CMAKE_TARGET_TOOLCHAIN=/opt/toolchains/x86_64-linux-gnu-libgfortran3-cxx03/target_x86_64-linux-gnu.cmake -E OBJDUMP_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -E SRC_NAME= -E PKG_CONFIG_PATH=/workspace/destdir/lib/pkgconfig:/workspace/destdir/lib64/pkgconfig:/workspace/destdir/share/pkgconfig -E ZERO_AR_DATE=1 -E bb_target=x86_64-linux-gnu -E WORKSPACE=/workspace -E HOSTAS=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -E AR_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar /home/mose/.julia/artifacts/56df63e8265adb2316d6599632113da025545434/sandbox --rootfs /home/mose/.julia/artifacts/56df63e8265adb2316d6599632113da025545434 --workspace /tmp/jl_YAdI8o:/workspace --workspace /tmp/jl_cqAyX3:/opt/bin --workspace /tmp/jl_lyXa9r:/opt/toolchains --workspace /home/mose/.julia/dev/BinaryBuilderBase/deps/ccache:/root/.ccache --workspace /tmp/jl_7NjQBr:/etc/resolv.conf --map /home/mose/.julia/artifacts/93f4959433015551e5a77a47768c01a06a7ba4a2:/opt/x86_64-linux-gnu/PlatformSupport-2021.8.10 --map /home/mose/.julia/artifacts/305c8b171016c46dfbb8ca234a543a99b6e3d045:/opt/x86_64-linux-gnu/GCCBootstrap-4.8.5 --map /home/mose/.julia/artifacts/aac62b1de8767c6b83ea3e15753a06afdad87e43:/opt/x86_64-linux-musl/LLVMBootstrap-12.0.0 --map /home/mose/.julia/artifacts/047fec931a6c54eb4d5df41acf4d35c704dac1c1:/opt/x86_64-linux-musl/PlatformSupport-2021.8.10 --map /home/mose/.julia/artifacts/e5229ab63cfb1c1deb9c8f461889787e648eaeba:/opt/x86_64-linux-musl/GCCBootstrap-4.8.5 -- /bin/bash -c false`,["FC_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran", "PATH=/opt/bin/x86_64-linux-gnu-libgfortran3-cxx03:/opt/x86_64-linux-gnu/bin:/opt/bin/x86_64-linux-musl-cxx11:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/x86_64-linux-musl-cxx11/destdir/bin:/workspace/destdir/bin", "CXX=c++", "HOSTFC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran", "HOSTSTRIP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip", "nproc=3", "CC=cc", "OBJCOPY_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy", "FC_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran", "HISTFILE=/meta/.bash_history"  …  "dlext=so", "CMAKE_TARGET_TOOLCHAIN=/opt/toolchains/x86_64-linux-gnu-libgfortran3-cxx03/target_x86_64-linux-gnu.cmake", "OBJDUMP_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump", "SRC_NAME=", "PKG_CONFIG_PATH=/workspace/destdir/lib/pkgconfig:/workspace/destdir/lib64/pkgconfig:/workspace/destdir/share/pkgconfig", "ZERO_AR_DATE=1", "bb_target=x86_64-linux-gnu", "WORKSPACE=/workspace", "HOSTAS=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as", "AR_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar"]), ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:440
 [3] run
   @ ./process.jl:438 [inlined]
 [4] run_interactive(ur::UserNSRunner, user_cmd::Cmd; stdin::Nothing, stdout::Nothing, stderr::Nothing, verbose::Bool)
   @ BinaryBuilderBase ~/.julia/dev/BinaryBuilderBase/src/UserNSRunner.jl:220
 [5] run_interactive(ur::UserNSRunner, user_cmd::Cmd)
   @ BinaryBuilderBase ~/.julia/dev/BinaryBuilderBase/src/UserNSRunner.jl:187
 [6] top-level scope
   @ REPL[2]:1

julia> run_interactive(preferred_runner()(mktempdir(); platform=Platform("x86_64", "linux")), `/bin/bash -c 'false'`; stdout=IOBuffer())
false

So passing stdout=IOBuffer() makes a failing process not throw an error, contrary to what happens when you stdout=nothing (the default).

Instead on this PR (with #197 included):

julia> run_interactive(preferred_runner()(mktempdir(); platform=Platform("x86_64", "linux")), `/bin/bash -c 'false'`)
ERROR: failed process: Process(setenv(`sudo -E FC_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -E PATH=/opt/bin/x86_64-linux-gnu-libgfortran3-cxx03:/opt/x86_64-linux-gnu/bin:/opt/bin/x86_64-linux-musl-cxx11:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/x86_64-linux-musl-cxx11/destdir/bin:/workspace/destdir/bin -E CXX=c++ -E HOSTFC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -E HOSTSTRIP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -E nproc=3 -E CC=cc -E OBJCOPY_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -E FC_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -E HISTFILE=/meta/.bash_history -E LD_LIBRARY_PATH=/usr/lib/csl-musl-x86_64:/usr/lib/csl-glibc-x86_64:/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib64:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib:/workspace/destdir/lib64:/workspace/destdir/lib -E AS_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -E HOSTOBJCOPY=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -E BUILD_AR=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -E target=x86_64-linux-gnu -E HOSTRANLIB=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -E HOSTLIPO=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -E VERBOSE=false -E BUILD_STRIP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -E MESON_HOST_TOOLCHAIN=/opt/toolchains/x86_64-linux-musl-cxx11/host_x86_64-linux-musl.meson -E BUILD_RANLIB=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -E BUILD_CXX=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -E STRIP_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -E HOSTAR=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -E USER=mose -E CMAKE_HOST_TOOLCHAIN=/opt/toolchains/x86_64-linux-musl-cxx11/host_x86_64-linux-musl.cmake -E LLVM_TARGET=x86_64-linux-gnu -E HOSTOBJDUMP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -E proc_family=intel -E includedir=/workspace/destdir/include -E BUILD_LD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -E V=false -E BUILD_AS=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -E host_includedir=/workspace/x86_64-linux-musl-cxx11/destdir/include -E SHELL=/bin/bash -E LD_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -E BUILD_READELF=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -E rust_host=x86_64-unknown-linux-musl -E HOSTCXX=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -E STRIP_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -E LD_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -E BUILD_OBJDUMP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -E bindir=/workspace/destdir/bin -E exeext= -E PKG_CONFIG_SYSROOT_DIR=/workspace/destdir -E SOURCE_DATE_EPOCH=0 -E TERM=screen -E NM_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -E MESON_TARGET_TOOLCHAIN=/opt/toolchains/x86_64-linux-gnu-libgfortran3-cxx03/target_x86_64-linux-gnu.meson -E OBJCOPY_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -E CXX_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -E HOSTCC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -E RANLIB_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -E BUILD_LIPO=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -E rust_target=x86_64-unknown-linux-gnu -E CCACHE_COMPILERCHECK=content -E BUILD_OBJCOPY=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -E HOSTLD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -E HOSTNM=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -E BUILD_CC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -E AR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -E USE_CCACHE=true -E HOSTREADELF=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -E NM_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -E CXX_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -E bb_full_target=x86_64-linux-gnu-libgfortran3-cxx03 -E 'HIDDEN_PS1=\[\]sandbox\[\]:\[\]${PWD//$WORKSPACE/$\{WORKSPACE\}}\[\] \$ ' -E LIPO_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -E prefix=/workspace/destdir -E nbits=64 -E DSYMUTIL_FOR_BUILD=dsymutil -E BUILD_DSYMUTIL=dsymutil -E 'GNU_LIBC_VERSION=glibc 2.12.2' -E MACHTYPE=x86_64-linux-musl -E LIPO_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -E FC=gfortran -E HOSTDSYMUTIL=dsymutil -E BUILD_FC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -E CC_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -E RANLIB_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -E host_bindir=/workspace/x86_64-linux-musl-cxx11/destdir/bin -E LLVM_HOST_TARGET=x86_64-linux-musl -E READELF_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -E READELF_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -E libdir=/workspace/destdir/lib -E host_libdir=/workspace/x86_64-linux-musl-cxx11/destdir/lib -E CC_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -E host_prefix=/workspace/x86_64-linux-musl-cxx11/destdir -E OBJDUMP_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -E AS_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -E DSYMUTIL_BUILD=dsymutil -E BUILD_NM=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -E dlext=so -E CMAKE_TARGET_TOOLCHAIN=/opt/toolchains/x86_64-linux-gnu-libgfortran3-cxx03/target_x86_64-linux-gnu.cmake -E OBJDUMP_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -E SRC_NAME= -E PKG_CONFIG_PATH=/workspace/destdir/lib/pkgconfig:/workspace/destdir/lib64/pkgconfig:/workspace/destdir/share/pkgconfig -E ZERO_AR_DATE=1 -E bb_target=x86_64-linux-gnu -E WORKSPACE=/workspace -E HOSTAS=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -E AR_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar /home/mose/.julia/artifacts/56df63e8265adb2316d6599632113da025545434/sandbox --rootfs /home/mose/.julia/artifacts/56df63e8265adb2316d6599632113da025545434 --workspace /tmp/jl_AVhuGr:/workspace --workspace /tmp/jl_T30f0z:/opt/bin --workspace /tmp/jl_HRFU9x:/opt/toolchains --workspace /home/mose/.julia/dev/BinaryBuilderBase/deps/ccache:/root/.ccache --workspace /tmp/jl_VMjJfw:/etc/resolv.conf --map /home/mose/.julia/artifacts/93f4959433015551e5a77a47768c01a06a7ba4a2:/opt/x86_64-linux-gnu/PlatformSupport-2021.8.10 --map /home/mose/.julia/artifacts/305c8b171016c46dfbb8ca234a543a99b6e3d045:/opt/x86_64-linux-gnu/GCCBootstrap-4.8.5 --map /home/mose/.julia/artifacts/aac62b1de8767c6b83ea3e15753a06afdad87e43:/opt/x86_64-linux-musl/LLVMBootstrap-12.0.0 --map /home/mose/.julia/artifacts/047fec931a6c54eb4d5df41acf4d35c704dac1c1:/opt/x86_64-linux-musl/PlatformSupport-2021.8.10 --map /home/mose/.julia/artifacts/e5229ab63cfb1c1deb9c8f461889787e648eaeba:/opt/x86_64-linux-musl/GCCBootstrap-4.8.5 -- /bin/bash -c false`,["FC_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran", "PATH=/opt/bin/x86_64-linux-gnu-libgfortran3-cxx03:/opt/x86_64-linux-gnu/bin:/opt/bin/x86_64-linux-musl-cxx11:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/x86_64-linux-musl-cxx11/destdir/bin:/workspace/destdir/bin", "CXX=c++", "HOSTFC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran", "HOSTSTRIP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip", "nproc=3", "CC=cc", "OBJCOPY_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy", "FC_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran", "HISTFILE=/meta/.bash_history"  …  "dlext=so", "CMAKE_TARGET_TOOLCHAIN=/opt/toolchains/x86_64-linux-gnu-libgfortran3-cxx03/target_x86_64-linux-gnu.cmake", "OBJDUMP_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump", "SRC_NAME=", "PKG_CONFIG_PATH=/workspace/destdir/lib/pkgconfig:/workspace/destdir/lib64/pkgconfig:/workspace/destdir/share/pkgconfig", "ZERO_AR_DATE=1", "bb_target=x86_64-linux-gnu", "WORKSPACE=/workspace", "HOSTAS=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as", "AR_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar"]), ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:525 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:440
 [3] run
   @ ./process.jl:438 [inlined]
 [4] run_interactive(ur::UserNSRunner, user_cmd::Cmd; stdin::Nothing, stdout::Nothing, stderr::Nothing, verbose::Bool)
   @ BinaryBuilderBase ~/.julia/dev/BinaryBuilderBase/src/UserNSRunner.jl:206
 [5] run_interactive(ur::UserNSRunner, user_cmd::Cmd)
   @ BinaryBuilderBase ~/.julia/dev/BinaryBuilderBase/src/UserNSRunner.jl:186
 [6] top-level scope
   @ REPL[4]:1

julia> run_interactive(preferred_runner()(mktempdir(); platform=Platform("x86_64", "linux")), `/bin/bash -c 'false'`; stdout=IOBuffer())
ERROR: failed process: Process(setenv(`sudo -E FC_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -E PATH=/opt/bin/x86_64-linux-gnu-libgfortran3-cxx03:/opt/x86_64-linux-gnu/bin:/opt/bin/x86_64-linux-musl-cxx11:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/x86_64-linux-musl-cxx11/destdir/bin:/workspace/destdir/bin -E CXX=c++ -E HOSTFC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -E HOSTSTRIP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -E nproc=3 -E CC=cc -E OBJCOPY_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -E FC_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -E HISTFILE=/meta/.bash_history -E LD_LIBRARY_PATH=/usr/lib/csl-musl-x86_64:/usr/lib/csl-glibc-x86_64:/usr/local/lib64:/usr/local/lib:/usr/lib64:/usr/lib:/lib64:/lib:/workspace/x86_64-linux-musl-cxx11/destdir/lib:/opt/x86_64-linux-musl/x86_64-linux-musl/lib64:/opt/x86_64-linux-musl/x86_64-linux-musl/lib:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib64:/opt/x86_64-linux-gnu/x86_64-linux-gnu/lib:/workspace/destdir/lib64:/workspace/destdir/lib -E AS_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -E HOSTOBJCOPY=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -E BUILD_AR=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -E target=x86_64-linux-gnu -E HOSTRANLIB=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -E HOSTLIPO=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -E VERBOSE=false -E BUILD_STRIP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -E MESON_HOST_TOOLCHAIN=/opt/toolchains/x86_64-linux-musl-cxx11/host_x86_64-linux-musl.meson -E BUILD_RANLIB=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -E BUILD_CXX=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -E STRIP_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -E HOSTAR=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -E USER=mose -E CMAKE_HOST_TOOLCHAIN=/opt/toolchains/x86_64-linux-musl-cxx11/host_x86_64-linux-musl.cmake -E LLVM_TARGET=x86_64-linux-gnu -E HOSTOBJDUMP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -E proc_family=intel -E includedir=/workspace/destdir/include -E BUILD_LD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -E V=false -E BUILD_AS=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -E host_includedir=/workspace/x86_64-linux-musl-cxx11/destdir/include -E SHELL=/bin/bash -E LD_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -E BUILD_READELF=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -E rust_host=x86_64-unknown-linux-musl -E HOSTCXX=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -E STRIP_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip -E LD_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -E BUILD_OBJDUMP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -E bindir=/workspace/destdir/bin -E exeext= -E PKG_CONFIG_SYSROOT_DIR=/workspace/destdir -E SOURCE_DATE_EPOCH=0 -E TERM=screen -E NM_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -E MESON_TARGET_TOOLCHAIN=/opt/toolchains/x86_64-linux-gnu-libgfortran3-cxx03/target_x86_64-linux-gnu.meson -E OBJCOPY_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -E CXX_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -E HOSTCC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -E RANLIB_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -E BUILD_LIPO=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -E rust_target=x86_64-unknown-linux-gnu -E CCACHE_COMPILERCHECK=content -E BUILD_OBJCOPY=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy -E HOSTLD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ld -E HOSTNM=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -E BUILD_CC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -E AR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar -E USE_CCACHE=true -E HOSTREADELF=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -E NM_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -E CXX_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-g++ -E bb_full_target=x86_64-linux-gnu-libgfortran3-cxx03 -E 'HIDDEN_PS1=\[\]sandbox\[\]:\[\]${PWD//$WORKSPACE/$\{WORKSPACE\}}\[\] \$ ' -E LIPO_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -E prefix=/workspace/destdir -E nbits=64 -E DSYMUTIL_FOR_BUILD=dsymutil -E BUILD_DSYMUTIL=dsymutil -E 'GNU_LIBC_VERSION=glibc 2.12.2' -E MACHTYPE=x86_64-linux-musl -E LIPO_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-lipo -E FC=gfortran -E HOSTDSYMUTIL=dsymutil -E BUILD_FC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran -E CC_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -E RANLIB_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ranlib -E host_bindir=/workspace/x86_64-linux-musl-cxx11/destdir/bin -E LLVM_HOST_TARGET=x86_64-linux-musl -E READELF_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -E READELF_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-readelf -E libdir=/workspace/destdir/lib -E host_libdir=/workspace/x86_64-linux-musl-cxx11/destdir/lib -E CC_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gcc -E host_prefix=/workspace/x86_64-linux-musl-cxx11/destdir -E OBJDUMP_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -E AS_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -E DSYMUTIL_BUILD=dsymutil -E BUILD_NM=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-nm -E dlext=so -E CMAKE_TARGET_TOOLCHAIN=/opt/toolchains/x86_64-linux-gnu-libgfortran3-cxx03/target_x86_64-linux-gnu.cmake -E OBJDUMP_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump -E SRC_NAME= -E PKG_CONFIG_PATH=/workspace/destdir/lib/pkgconfig:/workspace/destdir/lib64/pkgconfig:/workspace/destdir/share/pkgconfig -E ZERO_AR_DATE=1 -E bb_target=x86_64-linux-gnu -E WORKSPACE=/workspace -E HOSTAS=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as -E AR_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar /home/mose/.julia/artifacts/56df63e8265adb2316d6599632113da025545434/sandbox --rootfs /home/mose/.julia/artifacts/56df63e8265adb2316d6599632113da025545434 --workspace /tmp/jl_DhrIKO:/workspace --workspace /tmp/jl_Q2KzVD:/opt/bin --workspace /tmp/jl_hfB5Ir:/opt/toolchains --workspace /home/mose/.julia/dev/BinaryBuilderBase/deps/ccache:/root/.ccache --workspace /tmp/jl_omnq5t:/etc/resolv.conf --map /home/mose/.julia/artifacts/93f4959433015551e5a77a47768c01a06a7ba4a2:/opt/x86_64-linux-gnu/PlatformSupport-2021.8.10 --map /home/mose/.julia/artifacts/305c8b171016c46dfbb8ca234a543a99b6e3d045:/opt/x86_64-linux-gnu/GCCBootstrap-4.8.5 --map /home/mose/.julia/artifacts/aac62b1de8767c6b83ea3e15753a06afdad87e43:/opt/x86_64-linux-musl/LLVMBootstrap-12.0.0 --map /home/mose/.julia/artifacts/047fec931a6c54eb4d5df41acf4d35c704dac1c1:/opt/x86_64-linux-musl/PlatformSupport-2021.8.10 --map /home/mose/.julia/artifacts/e5229ab63cfb1c1deb9c8f461889787e648eaeba:/opt/x86_64-linux-musl/GCCBootstrap-4.8.5 -- /bin/bash -c false`,["FC_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran", "PATH=/opt/bin/x86_64-linux-gnu-libgfortran3-cxx03:/opt/x86_64-linux-gnu/bin:/opt/bin/x86_64-linux-musl-cxx11:/opt/x86_64-linux-musl/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/workspace/x86_64-linux-musl-cxx11/destdir/bin:/workspace/destdir/bin", "CXX=c++", "HOSTFC=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran", "HOSTSTRIP=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-strip", "nproc=3", "CC=cc", "OBJCOPY_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objcopy", "FC_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-gfortran", "HISTFILE=/meta/.bash_history"  …  "dlext=so", "CMAKE_TARGET_TOOLCHAIN=/opt/toolchains/x86_64-linux-gnu-libgfortran3-cxx03/target_x86_64-linux-gnu.cmake", "OBJDUMP_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-objdump", "SRC_NAME=", "PKG_CONFIG_PATH=/workspace/destdir/lib/pkgconfig:/workspace/destdir/lib64/pkgconfig:/workspace/destdir/share/pkgconfig", "ZERO_AR_DATE=1", "bb_target=x86_64-linux-gnu", "WORKSPACE=/workspace", "HOSTAS=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-as", "AR_FOR_BUILD=/opt/bin/x86_64-linux-musl-cxx11/x86_64-linux-musl-ar"]), ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:525 [inlined]
 [2] run(::Base.CmdRedirect; wait::Bool)
   @ Base ./process.jl:440
 [3] run
   @ ./process.jl:438 [inlined]
 [4] run_interactive(ur::UserNSRunner, user_cmd::Cmd; stdin::Nothing, stdout::IOBuffer, stderr::Nothing, verbose::Bool)
   @ BinaryBuilderBase ~/.julia/dev/BinaryBuilderBase/src/UserNSRunner.jl:206
 [5] top-level scope
   @ REPL[5]:1

the behaviour is the same, whatever is the type of stdout: this always throws an error.

I believe you simply uncovered a bug which went unnoticed for ages, and which we even relied on. Sigh.

giordano · 2021-12-30T22:07:17Z

This is probably because the stdout isa IOBuffer branch

BinaryBuilderBase.jl/src/UserNSRunner.jl

Lines 207 to 218 in 769d8e2

    
           if stdout isa IOBuffer 
        
               if !(stdin isa IOBuffer) 
        
                   stdin = devnull 
        
               end 
        
               process = open(cmd, "r", stdin) 
        
               @async begin 
        
                   while !eof(process) 
        
                       write(stdout, read(process)) 
        
                   end 
        
               end 
        
               wait(process) 
        
               return success(process)

uses success(process), which doesn't throw, while success(run(cmd)) does

giordano · 2021-12-30T22:26:36Z

~~And ignorestatus doesn't work with Base.CmdRedirect. Double sigh.~~ Nevermind, that shouldn't be a problem.

giordano · 2021-12-30T22:46:28Z

This should do it: JuliaPackaging/BinaryBuilder.jl#1143

Co-authored-by: Mosè Giordano <[email protected]>

… an `IOBuffer`" (#199)" This reverts commit 769d8e2.

KristofferC commented Dec 30, 2021

View reviewed changes

src/Runner.jl Show resolved Hide resolved

giordano reviewed Dec 30, 2021

View reviewed changes

src/Runner.jl Show resolved Hide resolved

test/runners.jl Outdated Show resolved Hide resolved

KristofferC and others added 3 commits December 30, 2021 23:49

fix output from runners to IOBuffer

f6258f5

Update test/runners.jl

e93a4fe

Co-authored-by: Mosè Giordano <[email protected]>

Revert "Revert "simplify run_interactive, pipeline can now accept…

fac1994

… an `IOBuffer`" (#199)" This reverts commit 769d8e2.

giordano force-pushed the kc/fix_any_redirect_type branch from f05997a to affc698 Compare December 30, 2021 23:01

Add more tests for run_interactive

a6094a2

giordano force-pushed the kc/fix_any_redirect_type branch from affc698 to a6094a2 Compare December 30, 2021 23:03

giordano merged commit e1345f7 into master Dec 31, 2021

giordano deleted the kc/fix_any_redirect_type branch December 31, 2021 00:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix output from runners to IOBuffer #201

fix output from runners to IOBuffer #201

KristofferC commented Dec 30, 2021 •

edited

Loading

giordano commented Dec 30, 2021

giordano commented Dec 30, 2021 •

edited

Loading

KristofferC commented Dec 30, 2021 •

edited

Loading

giordano commented Dec 30, 2021

giordano commented Dec 30, 2021

KristofferC commented Dec 30, 2021

giordano commented Dec 30, 2021

KristofferC commented Dec 30, 2021

giordano commented Dec 30, 2021

KristofferC commented Dec 30, 2021

giordano commented Dec 30, 2021 •

edited

Loading

giordano commented Dec 30, 2021

giordano commented Dec 30, 2021 •

edited

Loading

giordano commented Dec 30, 2021 •

edited

Loading

fix output from runners to IOBuffer #201

fix output from runners to IOBuffer #201

Conversation

KristofferC commented Dec 30, 2021 • edited Loading

giordano commented Dec 30, 2021

giordano commented Dec 30, 2021 • edited Loading

KristofferC commented Dec 30, 2021 • edited Loading

giordano commented Dec 30, 2021

giordano commented Dec 30, 2021

KristofferC commented Dec 30, 2021

giordano commented Dec 30, 2021

KristofferC commented Dec 30, 2021

giordano commented Dec 30, 2021

KristofferC commented Dec 30, 2021

giordano commented Dec 30, 2021 • edited Loading

giordano commented Dec 30, 2021

giordano commented Dec 30, 2021 • edited Loading

giordano commented Dec 30, 2021 • edited Loading

KristofferC commented Dec 30, 2021 •

edited

Loading

giordano commented Dec 30, 2021 •

edited

Loading

KristofferC commented Dec 30, 2021 •

edited

Loading

giordano commented Dec 30, 2021 •

edited

Loading

giordano commented Dec 30, 2021 •

edited

Loading

giordano commented Dec 30, 2021 •

edited

Loading