Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test failures caused by Downloads.jl upgrade #56116

Closed
antonio-rojas opened this issue Oct 11, 2024 · 14 comments
Closed

Test failures caused by Downloads.jl upgrade #56116

antonio-rojas opened this issue Oct 11, 2024 · 14 comments

Comments

@antonio-rojas
Copy link
Contributor

I'm getting some test failures with 1.11.0, both using system curl 8.10.1 or bundled one

Error in testset Artifacts:
Error During Test at /build/julia/src/julia-1.11.0/test/testdefs.jl:24
  Got exception outside of a @test
  LoadError: failed process: Process(setenv(`/build/julia/src/julia-1.11.0/usr/bin/julia -C native -J/build/julia/src/julia-1.11.0/usr/lib/julia/sys.so --depwarn=error --check-bounds=yes -g1 --startup-file=no --color=no /build/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/Artifacts/test/refresh_artifacts.jl /tmp/jl_mKjpny`,["PYTHONHASHSEED=0", "SUDO_HOME=/root", "COMMAND_MODE=legacy", "PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl", "PWD=/build/julia/src/julia-1.11.0/test", "TEXTDOMAIN=pacman-scripts", "FCFLAGS=  -ffile-prefix-map=/build/julia/src=/usr/src/debug/julia", "SUDO_GID=0", "MAKEFLAGS=-j96", "SHELL=/bin/bash", "CFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -g -ffile-prefix-map=/build/julia/src=/usr/src/debug/julia", "CXXFLAGS=-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=3 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection         -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -Wp,-D_GLIBCXX_ASSERTIONS -g -ffile-prefix-map=/build/julia/src=/usr/src/debug/julia", "OPENBLAS_NUM_THREADS=1", "FFLAGS=  -ffile-prefix-map=/build/julia/src=/usr/src/debug/julia", "DEBUGINFOD_URLS=https://debuginfod.archlinux.org ", "MAIL=/var/mail/builduser", "LANG=C.UTF-8", "LDFLAGS=-Wl,-O1 -Wl,--sort-common -Wl,--as-needed -Wl,-z,relro -Wl,-z,now          -Wl,-z,pack-relative-relocs", "SUDO_USER=root", "LOGNAME=builduser", "SHLVL=1", "TEXTDOMAINDIR=/usr/share/locale", "OLDPWD=/build/julia/src", "SUDO_UID=0", "JULIA_LOAD_PATH=@:@v#.#:@stdlib", "_=../julia", "SUDO_COMMAND=/bin/bash -c bash -c cd\\ /startdir;\\ makepkg\\ \"\$@\" -bash --syncdeps --noconfirm --log --holdver --skipinteg --install", "DEBUG_RUSTFLAGS=-C debuginfo=2 --remap-path-prefix=/build/julia/src=/usr/src/debug/julia", "BUILDTOOLVER=1:1.2.1-1-any", "SOURCE_DATE_EPOCH=1728666311", "OPENBLAS_DEFAULT_NUM_THREADS=1", "USER=builduser", "CHOST=x86_64-pc-linux-gnu", "HOME=/build", "RUSTFLAGS=-Cforce-frame-pointers=yes -C debuginfo=2 --remap-path-prefix=/build/julia/src=/usr/src/debug/julia", "TERM=dumb", "JULIA_DEPOT_PATH=/build/.julia:/build/julia/src/julia-1.11.0/usr/local/share/julia:/build/julia/src/julia-1.11.0/usr/share/julia", "OPENBLAS_MAIN_FREE=1", "DEBUG_FFLAGS= -ffile-prefix-map=/build/julia/src=/usr/src/debug/julia", "BUILDTOOL=devtools"]), ProcessSignaled(6)) [0]
  
  Stacktrace:
    [1] pipeline_error
      @ ./process.jl:598 [inlined]
    [2] run(::Cmd; wait::Bool)
      @ Base ./process.jl:513
    [3] run(::Cmd)
      @ Base ./process.jl:510
    [4] top-level scope
      @ ~/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/Artifacts/test/runtests.jl:10
    [5] include
      @ ./Base.jl:557 [inlined]
    [6] macro expansion
      @ ~/julia/src/julia-1.11.0/test/testdefs.jl:33 [inlined]
    [7] macro expansion
      @ ~/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:1700 [inlined]
    [8] macro expansion
      @ ~/julia/src/julia-1.11.0/test/testdefs.jl:26 [inlined]
    [9] macro expansion
      @ ./timing.jl:578 [inlined]
   [10] runtests(name::String, path::String, isolate::Bool; seed::UInt128)
      @ Main ~/julia/src/julia-1.11.0/test/testdefs.jl:24
   [11] runtests
      @ ~/julia/src/julia-1.11.0/test/testdefs.jl:5 [inlined]
   [12] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::@Kwargs{seed::UInt128})
      @ Base ./essentials.jl:1056
   [13] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed ~/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:287
   [14] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed ~/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:70
   [15] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ~/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:287
  in expression starting at /build/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/Artifacts/test/runtests.jl:10
Error in testset LazyArtifacts:
Error During Test at /build/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/LazyArtifacts/test/runtests.jl:18
  Test threw exception
  Expression: success(run(setenv(`$(Base.julia_cmd()) --depwarn=no --startup-file=no -e '
    using Artifacts, Pkg
    using Test
    mktempdir() do tempdir
        Artifacts.with_artifacts_directory(tempdir) do
            redirect_stderr(devnull) do
                socrates_dir = @test_logs(
                        (:warn, "using Pkg instead of using LazyArtifacts is deprecated"),
                        artifact"socrates")
                @test isdir(socrates_dir)
            end
        end
    end'`, dir = #= /build/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/LazyArtifacts/test/runtests.jl:31 =# @__DIR__())))
  failed process: Process(setenv(`/build/julia/src/julia-1.11.0/usr/bin/julia -C native -J/build/julia/src/julia-1.11.0/usr/lib/julia/sys.so --depwarn=error --check-bounds=yes -g1 --startup-file=no --depwarn=no --startup-file=no -e '
      using Artifacts, Pkg
      using Test
      mktempdir() do tempdir
          Artifacts.with_artifacts_directory(tempdir) do
              redirect_stderr(devnull) do
                  socrates_dir = @test_logs(
                          (:warn, "using Pkg instead of using LazyArtifacts is deprecated"),
                          artifact"socrates")
                  @test isdir(socrates_dir)
              end
          end
      end'`; dir="/build/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/LazyArtifacts/test"), ProcessSignaled(6)) [0]
  
  Stacktrace:
   [1] pipeline_error
     @ ./process.jl:598 [inlined]
   [2] run(::Cmd; wait::Bool)
     @ Base ./process.jl:513
   [3] run(::Cmd)
     @ Base ./process.jl:510
   [4] macro expansion
     @ ~/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/LazyArtifacts/test/runtests.jl:18 [inlined]
   [5] macro expansion
     @ ~/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:676 [inlined]
   [6] top-level scope
     @ ~/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/LazyArtifacts/test/runtests.jl:515
Error in testset download:
Error During Test at /build/julia/src/julia-1.11.0/test/testdefs.jl:24
  Got exception outside of a @test
  LoadError: download test failed, cmd : `/build/julia/src/julia-1.11.0/usr/bin/julia -C native -J/build/julia/src/julia-1.11.0/usr/lib/julia/sys.so --depwarn=error --check-bounds=yes -g1 --startup-file=no --depwarn=no --startup-file=no download_exec.jl`
  Stacktrace:
    [1] error(s::String)
      @ Base ./error.jl:35
    [2] top-level scope
      @ ~/julia/src/julia-1.11.0/test/download.jl:5
    [3] include
      @ ./Base.jl:557 [inlined]
    [4] macro expansion
      @ ~/julia/src/julia-1.11.0/test/testdefs.jl:33 [inlined]
    [5] macro expansion
      @ ~/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/Test/src/Test.jl:1700 [inlined]
    [6] macro expansion
      @ ~/julia/src/julia-1.11.0/test/testdefs.jl:26 [inlined]
    [7] macro expansion
      @ ./timing.jl:578 [inlined]
    [8] runtests(name::String, path::String, isolate::Bool; seed::UInt128)
      @ Main ~/julia/src/julia-1.11.0/test/testdefs.jl:24
    [9] runtests
      @ ~/julia/src/julia-1.11.0/test/testdefs.jl:5 [inlined]
   [10] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::@Kwargs{seed::UInt128})
      @ Base ./essentials.jl:1056
   [11] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed ~/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:287
   [12] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed ~/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:70
   [13] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ~/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/Distributed/src/process_messages.jl:287
  in expression starting at /build/julia/src/julia-1.11.0/test/download.jl:4
Error in testset loading:
Test Failed at /build/julia/src/julia-1.11.0/test/loading.jl:1384
  Expression: success(addenv(`$(Base.julia_cmd()) --project=$project_path --startup-file=no -e 'import Pkg; Pkg.develop("Foo51989"); Pkg.precompile(); exit(0)'`, "JULIA_DEPOT_PATH" => depot))

ERROR: LoadError: Test run finished with errors
in expression starting at /build/julia/src/julia-1.11.0/test/runtests.jl:100

Traced it back to the Downloads upgrade 598a4de, in particular to the commit JuliaLang/Downloads.jl@1061ecc. Reverting that commit fixes the tests.

@ararslan
Copy link
Member

cc @IanButterworth

@IanButterworth
Copy link
Member

Seems like the actual error from the sub process is getting swallowed. Can you run the command separately and see what it says.

@antonio-rojas
Copy link
Contributor Author

Tried that but I wasn't able to get any useful output.

@IanButterworth
Copy link
Member

Given this is a custom setup, I think debugging it is going to require seeing all that.

@antonio-rojas
Copy link
Contributor Author

Looks like this is actually related to curl 8.10. Even with USE_SYSTEM_CURL=0, julia is somehow loading the system libcurl (possibly via another system dependency). Downgrading to curl 8.9 in the build chroot makes the tests pass (always with USE_SYSTEM_CURL=0)

@IanButterworth
Copy link
Member

Share versioninfo to see what env vars are set?

@antonio-rojas
Copy link
Contributor Author

julia> versioninfo(verbose=true)
Julia Version 1.11.0
Commit 501a4f25c2 (2024-10-07 11:40 UTC)
Build Info:

    Note: This is an unofficial build, please report bugs to the project
    responsible for this build and not to the Julia project unless you can
    reproduce the issue using official builds available at https://julialang.org/downloads

Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  uname: Linux 6.11.2-arch1-1 #1 SMP PREEMPT_DYNAMIC Fri, 04 Oct 2024 21:51:11 +0000 x86_64 unknown
  CPU: AMD Ryzen 7 5700U with Radeon Graphics: 
                 speed         user         nice          sys         idle          irq
       #1-16  1929 MHz    1472536 s       2616 s     496437 s   79182456 s          1 s
  Memory: 30.735687255859375 GB (8967.19140625 MB free)
  Uptime: 509327.04 sec
  Load Avg:  0.78  0.77  0.81
  WORD_SIZE: 64
  LLVM: libLLVM-16.0.6 (ORCJIT, znver2)
Threads: 1 default, 0 interactive, 1 GC (on 16 virtual cores)
Environment:
  XDG_SESSION_PATH = /org/freedesktop/DisplayManager/Session1
  HOME = /home/antonio
  PROFILEHOME = 
  XDG_SEAT_PATH = /org/freedesktop/DisplayManager/Seat0
  TERM = xterm-256color
  PATH = /usr/local/sbin:/usr/local/bin:/usr/bin:/var/lib/flatpak/exports/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/antonio/.local/bin:/home/antonio/.local/bin

Build flags:

    prefix=/usr
    bindir=/usr/bin
    sysconfdir=/etc
    libexecdir=/usr/lib
    USE_BINARYBUILDER=0
    USE_SYSTEM_CSL=1
    USE_SYSTEM_LLVM=1
    USE_SYSTEM_LLD=1
    USE_SYSTEM_LIBUNWIND=1
    USE_SYSTEM_PCRE=1
    USE_SYSTEM_BLAS=1
    USE_SYSTEM_LAPACK=1
    USE_SYSTEM_LIBBLASTRAMPOLINE=1
    USE_SYSTEM_GMP=1
    USE_SYSTEM_MPFR=1
    USE_SYSTEM_LIBSUITESPARSE=1
    USE_SYSTEM_LIBWHICH=1
    USE_SYSTEM_DSFMT=0
    USE_SYSTEM_LIBUV=0
    USE_SYSTEM_UTF8PROC=1
    USE_SYSTEM_LIBGIT2=1
    USE_SYSTEM_LIBSSH2=1
    USE_SYSTEM_MBEDTLS=1
    USE_SYSTEM_CURL=0
    USE_SYSTEM_PATCHELF=1
    USE_SYSTEM_ZLIB=1
    USE_SYSTEM_P7ZIP=1
    USE_SYSTEM_OPENLIBM=1
    USE_BLAS64=1
    LIBBLAS=-lblas64
    LIBBLASNAME=libblas64
    LIBLAPACK=-llapack64
    LIBLAPACKNAME=liblapack64
    MARCH=x86-64
    VERBOSE=1

@IanButterworth
Copy link
Member

IanButterworth commented Oct 14, 2024

Ok, and you tried running this but didn't get anything helpful out?

% cd /build/julia/src/julia-1.11.0/usr/share/julia/stdlib/v1.11/LazyArtifacts/test
% julia --depwarn=no --startup-file=no -e '
      using Artifacts, Pkg
      using Test
      mktempdir() do tempdir
          Artifacts.with_artifacts_directory(tempdir) do
              redirect_stderr(devnull) do
                  socrates_dir = @test_logs(
                          (:warn, "using Pkg instead of using LazyArtifacts is deprecated"),
                          artifact"socrates")
                  @test isdir(socrates_dir)
              end
          end
      end'

@antonio-rojas
Copy link
Contributor Author

Correct, no output at all actually. Returns error code 0.

@IanButterworth
Copy link
Member

So it succeeds, which it doesn't in tests. There must be some setup that's different in the tests.

@giordano
Copy link
Contributor

Looks like this is actually related to curl 8.10.

Did you see JuliaLang/Downloads.jl#260 and JuliaLang/Downloads.jl#261 already?

@antonio-rojas
Copy link
Contributor Author

Looks like this is actually related to curl 8.10.

Did you see JuliaLang/Downloads.jl#260 and JuliaLang/Downloads.jl#261 already?

I can reproduce the test failures at JuliaLang/Downloads.jl#261 (with curl 8.10 installed but julia compiled with USE_SYSTEM_CURL=0). However, they still fail before JuliaLang/Downloads.jl@1061ecc so it doesn't seem to be the same issue.

@antonio-rojas
Copy link
Contributor Author

So it succeeds, which it doesn't in tests. There must be some setup that's different in the tests.

Could it be parallelization? Is there a way to force running the julia test suite with a single worker to confirm?

@antonio-rojas
Copy link
Contributor Author

"patch 1" from JuliaLang/Downloads.jl#260 (comment) fixes the issue with curl 1.11. Closing as duplicate

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants