Skip to content

How to add JLLs as test-only dependencies? #4236

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jonas-schulze opened this issue May 15, 2025 · 9 comments
Closed

How to add JLLs as test-only dependencies? #4236

jonas-schulze opened this issue May 15, 2025 · 9 comments

Comments

@jonas-schulze
Copy link
Contributor

jonas-schulze commented May 15, 2025 •
edited
Loading

Consider a Project.toml like

name = "MWE"
uuid = "57eeb600-edbd-463a-acb9-87ceabc7a853"
authors = ["Jonas Schulze"]
version = "0.1.0"

[compat]
pocl_jll = "7"

[extras]
OpenCL = "08131aa3-fb12-5dee-8b74-c09406e224a2"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
pocl_jll = "627d6b7a-bbe6-5189-83e7-98cc0a5aeadd"

[targets]
test = ["OpenCL", "Test", "pocl_jll"]

and a test/runtests.jl like

using OpenCL, pocl_jll
using Test

data = CLArray(1:3)
@test sum(data) == 6

Note that for the example above it doesn't really matter what the MWE package does. Trying to run the tests leads to the failure

ERROR: LoadError: ArgumentError: Package TOML not found in current path.

that is caused by some .pkg/select_artifacts.jl. I could not find who or why that file was being executed, but it seems to stem from here:

https://github.com/JuliaPackaging/BinaryBuilder.jl/blob/aa0df0e2bcb983185bb989adce7e97698b705fb5/src/AutoBuild.jl#L1540

Duplicating the [extras] under [weakdeps] did not help.

julia> versioninfo()
Julia Version 1.10.9
Commit 5595d20a287 (2025-03-10 12:51 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: 4 × Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-15.0.7 (ORCJIT, skylake)
Threads: 1 default, 0 interactive, 1 GC (on 4 virtual cores)
Environment:
  JULIA_PROJECT = @.

(MWE) pkg> test
     Testing MWE
      Status `/tmp/jl_sCPcaB/Project.toml`
ERROR: LoadError: ArgumentError: Package TOML not found in current path.
- Run `import Pkg; Pkg.add("TOML")` to install the TOML package.
Stacktrace:
 [1] macro expansion
   @ ./loading.jl:1842 [inlined]
 [2] macro expansion
   @ ./lock.jl:267 [inlined]
 [3] __require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1823
 [4] #invoke_in_world#3
   @ ./essentials.jl:926 [inlined]
 [5] invoke_in_world
   @ ./essentials.jl:923 [inlined]
 [6] require(into::Module, mod::Symbol)
   @ Base ./loading.jl:1816
 [7] include(fname::String)
   @ Base.MainInclude ./client.jl:494
 [8] top-level scope
   @ none:5
in expression starting at /home/jonas/.julia/packages/pocl_jll/RNyNp/.pkg/select_artifacts.jl:3
ERROR: failed process: Process(`/opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/bin/julia -C native -J/opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/lib/julia/sys.so -g1 -O0 --color=no --history-file=no --startup-file=no --project=/tmp/jl_sCPcaB/Project.toml --eval 'append!(empty!(Base.DEPOT_PATH), ["/home/jonas/.julia", "/opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/local/share/julia", "/opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia"])
append!(empty!(Base.DL_LOAD_PATH), String[])

cd("/home/jonas/.julia/packages/pocl_jll/RNyNp/.pkg")
include("/home/jonas/.julia/packages/pocl_jll/RNyNp/.pkg/select_artifacts.jl")
' -t1 --startup-file=no x86_64-linux-gnu-libgfortran5-cxx11-libstdcxx30-julia_version+1.10.9`, ProcessExited(1)) [1]

Stacktrace:
  [1] pipeline_error
    @ ./process.jl:565 [inlined]
  [2] read(cmd::Cmd)
    @ Base ./process.jl:449
  [3] collect_artifacts(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:734
  [4] collect_artifacts
    @ /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:720 [inlined]
  [5] check_artifacts_downloaded(pkg_root::String; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:778
  [6] check_artifacts_downloaded
    @ /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:777 [inlined]
  [7] is_package_downloaded(manifest_file::String, pkg::Pkg.Types.PackageSpec; platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2211
  [8] is_package_downloaded
    @ /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2206 [inlined]
  [9] print_status(env::Pkg.Types.EnvCache, old_env::Nothing, registries::Vector{Pkg.Registry.RegistryInstance}, header::Symbol, uuids::Vector{Base.UUID}, names::Vector{String}; manifest::Bool, diff::Bool, ignore_indent::Bool, outdated::Bool, extensions::Bool, io::Base.TTY, mode::Pkg.Types.PackageMode, hidden_upgrades_info::Bool, show_usagetips::Bool)
    @ Pkg.Operations /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2332
 [10] print_status
    @ /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2264 [inlined]
 [11] status(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}; header::Nothing, mode::Pkg.Types.PackageMode, git_diff::Bool, env_diff::Nothing, ignore_indent::Bool, io::Base.TTY, outdated::Bool, extensions::Bool, hidden_upgrades_info::Bool, show_usagetips::Bool)
    @ Pkg.Operations /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2495
 [12] status
    @ /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:2466 [inlined]
 [13] (::Pkg.Operations.var"#130#134"{Bool, Cmd, Cmd, Nothing, Pkg.Types.Context, Vector{Tuple{String, Base.Process}}, String, Pkg.Types.PackageSpec})()
    @ Pkg.Operations /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1974
 [14] withenv(::Pkg.Operations.var"#130#134"{Bool, Cmd, Cmd, Nothing, Pkg.Types.Context, Vector{Tuple{String, Base.Process}}, String, Pkg.Types.PackageSpec}, ::Pair{String, String}, ::Vararg{Pair{String}})
    @ Base ./env.jl:257
 [15] (::Pkg.Operations.var"#117#122"{String, Bool, Bool, Bool, Pkg.Operations.var"#130#134"{Bool, Cmd, Cmd, Nothing, Pkg.Types.Context, Vector{Tuple{String, Base.Process}}, String, Pkg.Types.PackageSpec}, Pkg.Types.PackageSpec})()
    @ Pkg.Operations /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1840
 [16] with_temp_env(fn::Pkg.Operations.var"#117#122"{String, Bool, Bool, Bool, Pkg.Operations.var"#130#134"{Bool, Cmd, Cmd, Nothing, Pkg.Types.Context, Vector{Tuple{String, Base.Process}}, String, Pkg.Types.PackageSpec}, Pkg.Types.PackageSpec}, temp_env::String)
    @ Pkg.Operations /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1721
 [17] (::Pkg.Operations.var"#115#120"{Dict{String, Any}, Bool, Bool, Bool, Pkg.Operations.var"#130#134"{Bool, Cmd, Cmd, Nothing, Pkg.Types.Context, Vector{Tuple{String, Base.Process}}, String, Pkg.Types.PackageSpec}, Pkg.Types.Context, Pkg.Types.PackageSpec, String, Pkg.Types.Project, String})(tmp::String)
    @ Pkg.Operations /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1810
 [18] mktempdir(fn::Pkg.Operations.var"#115#120"{Dict{String, Any}, Bool, Bool, Bool, Pkg.Operations.var"#130#134"{Bool, Cmd, Cmd, Nothing, Pkg.Types.Context, Vector{Tuple{String, Base.Process}}, String, Pkg.Types.PackageSpec}, Pkg.Types.Context, Pkg.Types.PackageSpec, String, Pkg.Types.Project, String}, parent::String; prefix::String)
    @ Base.Filesystem ./file.jl:766
 [19] mktempdir(fn::Function, parent::String)
    @ Base.Filesystem ./file.jl:762
 [20] mktempdir
    @ ./file.jl:762 [inlined]
 [21] sandbox(fn::Function, ctx::Pkg.Types.Context, target::Pkg.Types.PackageSpec, target_path::String, sandbox_path::String, sandbox_project_override::Pkg.Types.Project; preferences::Dict{String, Any}, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
    @ Pkg.Operations /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1768
 [22] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
    @ Pkg.Operations /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1971
 [23] test
    @ /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/Operations.jl:1915 [inlined]
 [24] test(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{io::Base.TTY})
    @ Pkg.API /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/API.jl:444
 [25] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::@Kwargs{})
    @ Pkg.API /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/API.jl:159
 [26] test(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/API.jl:148
 [27] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/REPLMode/REPLMode.jl:412
 [28] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/REPLMode/REPLMode.jl:390
 [29] do_cmd
    @ /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/REPLMode/REPLMode.jl:380 [inlined]
 [30] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/REPLMode/REPLMode.jl:557
 [31] #invokelatest#2
    @ ./essentials.jl:892 [inlined]
 [32] invokelatest
    @ ./essentials.jl:889 [inlined]
 [33] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/LineEdit.jl:2656
 [34] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:1312
 [35] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL /opt/dot-julia/juliaup/julia-1.10.9+0.x64.linux.gnu/share/julia/stdlib/v1.10/REPL/src/REPL.jl:386

How do I fix this (properly)? Who should declare TOML as a dependency?

This is a re-post of JuliaPackaging/BinaryBuilder.jl#1384 after that one had been closed. I could not find a duplicate issue here, and I verified that the behavior is the same for Julia 1.11.5.
@IanButterworth
Copy link
Member

Whoever owns the .pkg/select_artifacts.jl should declare TOML as a dependency.
I think they might be able to declare it as a build dependency, but not sure.

[extras]
TOML = "xxx"

[targets]
build = ["TOML"]

@IanButterworth
Copy link
Member

Seems like BB should be adding these as dependencies? (again, hopefully build-only)
https://github.com/JuliaPackaging/BinaryBuilder.jl/blob/aa0df0e2bcb983185bb989adce7e97698b705fb5/src/AutoBuild.jl#L1544

cc. @giordano

@giordano
Copy link
Contributor

It's already a dep: https://github.com/JuliaPackaging/BinaryBuilder.jl/blob/aa0df0e2bcb983185bb989adce7e97698b705fb5/src/AutoBuild.jl#L1845. Why it should be also in extras?

@IanButterworth
Copy link
Member

Ok, then this appears to be a Pkg bug where this is assuming all stdlibs are available, which they aren't in Pkg.test during its install time.

and I don't think even adding TOML as a test dependency would fix this..

Pkg.jl/src/Operations.jl

Line 827 in 12c3eaf

function collect_artifacts(pkg_root::String; platform::AbstractPlatform=HostPlatform())

@vchuravy
Copy link
Member

Just to be sure, which build of pocl_jll was this?

There recently was an issue where we removed a platform augmentation from pocl_jll, yet the .pkg files remained behind and caused issues like this. That should now be fixed.

@IanButterworth
Copy link
Member 

[[deps.pocl_jll]]
deps = ["Artifacts", "Clang_unified_jll", "Hwloc_jll", "JLLWrappers", "LLD_unified_jll", "Libdl", "OpenCL_Headers_jll", "OpenCL_jll", "SPIRV_LLVM_Translator_jll", "SPIRV_Tools_jll"]
git-tree-sha1 = "49f2e110ce71ac083cb2f2a2ae5b8a904e786bcc"
uuid = "627d6b7a-bbe6-5189-83e7-98cc0a5aeadd"
version = "7.0.0+3"

@IanButterworth
Copy link
Member

IanButterworth commented May 15, 2025 •
edited
Loading

TOML is missing from the deps list, yeah https://github.com/JuliaBinaryWrappers/pocl_jll.jl/tree/7f58a189c9588cfa64ff9a239f7a771be350b6cd

@IanButterworth IanButterworth closed this as not planned Won't fix, can't repro, duplicate, stale May 15, 2025
@jonas-schulze
Copy link
Contributor Author

Thanks for investigating, everyone!

Just to be sure, which build of pocl_jll was this?

There recently was an issue where we removed a platform augmentation from pocl_jll, yet the .pkg files remained behind and caused issues like this. That should now be fixed.

@vchuravy, with pocl_jll version pocl-v6.0.1+2 it does indeed work, but with any of the v7 versions it does not. Shall I open an issue on some other project, or is a fix for v7 already on the way?

@jonas-schulze
Copy link
Contributor Author

Issue seems resolved starting pocl-v7.0.0+5, which does no longer contain .pkg/select_artifacts.jl. 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@vchuravy @giordano @IanButterworth @jonas-schulze