Normalize version range spacing in compress function

[eschnett]

Fix an issue where version ranges with different spacing (e.g., "4.3.0-4" vs. "4.3.0 - 4")
are treated as different keys in the compressed dictionary, causing duplicate entries
instead of merging them properly.

This change strips all spaces from version range strings before using them as dictionary
keys, ensuring consistent treatment regardless of formatting variations.

This is a temporary work-around that will become unnecessary when "all this is rewirtten to use VersionNumbers", as suggested in the beginning of this file.

[GunnarFarneback]

An attempt to solve this was tried in #102 and reverted in #106 due to incorrect output (which wasn't caught in this package's tests).

(I'm fine with this lower tech approach if it's effective and passes downstream tests.)

[fingolfin]

This issue is causing more and more JLL updates to get stuck. Can we get this merged & deployed?

[eschnett]

This issue only appeared some time last week. Another option would be to undo the respective change. (HDF5_jll.jl 2.0.0+1 was handled correctly, its successor HDF5_jll.jl 2.0.0+2 is now having problems.)

[giordano]

Can we get a regression test? My understanding is that this issue is Julia-version specific, which also explains why it popped up only recently in BinaryBuilder (we switched to Julia v1.12 only last week)

[giordano]

Bump

[GunnarFarneback]

I missed that Dilum had rebased on master after #109 was merged. Since the downstream tests pass I'm confident enough to merge this. However, it doesn't seem to be entirely effective.

With

filename = expanduser("~/.julia/registries/General/A/AMDGPU/Compat.toml")
data = load(filename)
compress(filename, data)

there are still entries like

   "1.2.2-1.2.7" => Dict("GPUArrays" => "11.2.1-11")
     "1.2.8-1.2" => Dict("Random123" => "1.6.0 - 1", "AcceleratedKernels" => "0.3.1 - 0.3", "ROCmDeviceLibs_jll" => "5.6.1 - 5", "LLD_jll" => "15 - 19", "CEnum" => "0.4 - 0.5", "julia" => "1.10.0 - 1", "GPUArrays" => "11.2.1 - 11", "GPUToolbox" => "0.1 - 0.2", "KernelAbstractions" => "0.9.2 - 0.9", "RandomNumbers" => "1.5.0 - 1"…)
     "1.3-1.3.3" => Dict("RandomNumbers" => "1.5.0-1", "GPUArrays" => "11.2.1-11", "KernelAbstractions" => "0.9.2-0.9", "julia" => "1.10.0-1", "CEnum" => "0.4-0.5", "GPUToolbox" => "0.1-0.2", "LLD_jll" => "15-19", "LLVM_jll" => "15-19", "Random123" => "1.6.0-1")
   "1.3.4-2.1.2" => Dict("GPUArrays" => "11.2.1 - 11")

where GPUArrays is split too much because some entries have spaces and some don't.

I will add a commit to attempt to solve this and we'll see if that passes the downstream tests.

[giordano]

I still think more tests here would be useful, instead of just relying on downstream ones 🙂

[GunnarFarneback]

Absolutely, I wouldn't hesitate accepting a PR that adds tests.

[GunnarFarneback]

But until some volunteer gets to that, the downstream tests are a lot better than nothing. Now I need to investigate whether the LocalRegistry regression tests found something significant or something harmless. Won't happen until tomorrow though. Ah, RegistryCI also flagged something. Then it's probably a real problem.

[giordano]

I thought you had a test right there: #107 (comment) 🙂

[GunnarFarneback]

Not really, that involves far too much visual inspection and may or may not be on point.

Frankly I still don't understand why this is causing errors in JLLs where it otherwise just has made the registry files less compact than they could be.

[giordano]

As far as I understand, RegistryTools doesn't realise "1.6 - 1" is the same as "1.6-1" and it creates duplicate overlapping compat bounds which fail registry checks

[GunnarFarneback]

Sure, but why do overlapping bounds only happen for JLLs? Registrator has had this issue in production for quite a while now, and the only visible effect has been a loss of conciseness.

[giordano]

Because we switched to Julia v1.12 last week.

[GunnarFarneback]

That explains the timing but not why only JLLs. The extra spacing was introduced in JuliaLang/Pkg.jl#3580 and has been present since Julia 1.11. Registrator has been running with this since July, which has resulted in a lot of split ranges but never in an overlapping range. I have never seen a bug report of this for LocalRegistry, and I would expect at least some users to be on 1.11 or 1.12.

Are the JLLs using RegistryTools differently somehow?
How can we test whether this PR will actually resolve the JLL problems?
Are there any downstream tests we can run?
Can we extract one example where it goes badly (i.e. overlapping ranges) into a test?

[giordano]

All usage of RegistryTools is in this file: https://github.com/JuliaPackaging/BinaryBuilder.jl/blob/ee05c44e940128fca1fa9980c6cc00c78a21e88e/src/AutoBuild.jl. We don't have any test about registration. Last time I attempted to understand RegistryTools code I wanted to scream in the void: #94

[GunnarFarneback]

Presumably https://github.com/JuliaPackaging/BinaryBuilder.jl/blob/ee05c44e940128fca1fa9980c6cc00c78a21e88e/src/AutoBuild.jl#L634-L643 is what matters. That's a function I don't call from LocalRegistry but Registrator should be using it.

With knowledge of the argument values and using a registry fork on local disk, it may be possible to extract a stand-alone reproducer.

[GunnarFarneback]

I know what is wrong with my latest change and will fix it tonight.

In the absence of useful tests, would it be possible to run this PR in production to see if it empirically solves the problem?

[eschnett]

The special thing that happened here might be that we tried to register a new version that differs only in its build number (HDF5_jll 2.0.0+3). This is not usually done for non-jll packages. In this case, there must be no changes at all to the package's dependencies because the registry does not look at the build number when resolving dependencies.

I assume what happened was approximately the following:

• the old dependencies (for HDF5_jll 2.0.0) were kept, without spaces
• the new dependencies (also for HDF5_jll 2.0.0 because the build number does not matter) were added, with spaces
• these two dependency sets were the same, but were not unified
• this leads to overlapping version ranges

In other words, we first registered dependencies for HDF5_jll 2.0.0+2, and later for HDF5_jll 2.0.0+3. Internally, both were recorded as dependencies for HDF5_jll 2.0.0 (without build number), and these duplicate sets were not unified.

[giordano]

That sounds plausible, I think all cases where we've seen the issue were in non +0 build versions, most recently I've seen it in JuliaRegistries/General#143840.

[GunnarFarneback]

Thanks. That's a good hint and maybe something I can verify with LocalRegistry.

[GunnarFarneback]

Indeed I can. With Julia 1.12 and RegistryTools 2.4.1 in the manifest, this script

using LocalRegistry: register
mktempdir() do tmpdir
    cd(tmpdir) do
        run(`git clone https://github.com/JuliaRegistries/General.git`)
        run(`git -C General checkout ad9f281452c6754555f5485fb9f7c21ec40cd362`)
        run(`git clone https://github.com/JuliaBinaryWrappers/GANAK_jll.jl.git`)
        run(`git -C GANAK_jll.jl checkout GANAK-v2.5.2+1`)
        register("./GANAK_jll.jl", registry = "./General", commit = false)
        run(`git -C General diff`)
    end
end

generates the original diff in JuliaRegistries/General#143840 and with this PR instead, it generates the diff after the corrections.

That's good enough verification for me.