Convert remaining JLL stdlibs to LazyLibraries #58444

IanButterworth · 2025-05-17T00:38:26Z

Builds on #58405 (which I rebased, so the merge base here is master for now)

TODO:

Convert the new Zstd_jll from [deps] enable zstd support #58344
~~Convert LLD_jll?~~ No, it's an executable not a lib
~~Convert p7zip_jll?~~ No, it's an executable not a lib
Blocked by --trim failure trimming: no way to ccall into a dynamically-computed library name #57707
Fix linux RR test failure - release Fix _ossl_modules_path() returning wrong path to ossl-modules JuliaWeb/OpenSSL.jl#40

topolarity · 2025-05-17T01:57:20Z

Unfortunately the --trim failure is expected until we resolve #57707:

Core.MethodError(f=Base.Libc.Libdl.var"#dlsym"(), args=("/cache/julia-buildkite-plugin/depots/53de1ea5-6dd3-4008-b976-161c9f34b0ad/artifacts/4db1e58d71ac6bbb35ecc832033264c630d5d3b3/lib/libzstd.so", :ZSTD_versionString), world=0x00000000000097b7)

(LinearAlgebra is already broken for --trim on master for the same reason)

We do have a plan to fix it, but I don't know when it's going to happen since @Keno would like to revamp our ccall semantics (#57931) around the same time

IanButterworth · 2025-05-17T03:41:12Z

Ah ok.
In its current state, along with JuliaSparse/SparseArrays.jl#626 this PR gives

macOS M2 compared to results in #58436

julia> @time using Plots
  0.850111 seconds (1.29 M allocations: 74.658 MiB, 6.23% gc time, 2.34% compilation time)

Linux AMD Ryzen 9 5950X

julia> @time using Plots
  0.757390 seconds (1.38 M allocations: 77.337 MiB, 4.01% gc time, 3.59% compilation time)

And basic startup appears to be faster (if that makes sense?)
master vs. PR

% hyperfine './julia --start=no -e "1+1"' '/Users/ian/.julia/juliaup/julia-nightly/bin/julia --start=no -e "1+1"'
Benchmark 1: ./julia --start=no -e "1+1"
  Time (mean ± σ):      89.3 ms ±   3.6 ms    [User: 58.6 ms, System: 29.0 ms]
  Range (min … max):    85.4 ms …  98.2 ms    30 runs

Benchmark 2: /Users/ian/.julia/juliaup/julia-nightly/bin/julia --start=no -e "1+1"
  Time (mean ± σ):      99.2 ms ±   2.1 ms    [User: 67.8 ms, System: 29.7 ms]
  Range (min … max):    97.0 ms … 107.3 ms    27 runs

Summary
  ./julia --start=no -e "1+1" ran
    1.11 ± 0.05 times faster than /Users/ian/.julia/juliaup/julia-nightly/bin/julia --start=no -e "1+1"

IanButterworth · 2025-05-17T03:42:41Z

stdlib/GMP_jll/test/runtests.jl

@@ -3,6 +3,6 @@
 using Test, Libdl, GMP_jll

 @testset "GMP_jll" begin
-    vn = VersionNumber(unsafe_string(unsafe_load(cglobal((:__gmp_version, libgmp), Ptr{Cchar}))))
+    vn = VersionNumber(unsafe_string(unsafe_load(cglobal(dlsym(libgmp, :__gmp_version), Ptr{Cchar}))))


Should/could cglobal be taught how to handle a LazyLibrary?

It could, the most annoying thing is just that cglobal() is an intrinsic, not a function, so to add an overload for that involves writing C code, not just Julia code.

stdlib/GMP_jll/src/GMP_jll.jl

stdlib/libLLVM_jll/src/libLLVM_jll.jl

stdlib/OpenLibm_jll/src/OpenLibm_jll.jl

stdlib/PCRE2_jll/src/PCRE2_jll.jl

stdlib/OpenLibm_jll/src/OpenLibm_jll.jl

staticfloat · 2025-05-17T04:50:51Z

stdlib/SuiteSparse_jll/src/SuiteSparse_jll.jl

+const libamd = LazyLibrary(_libamd_path, dependencies=_libamd_dependencies)
+const libbtf = LazyLibrary(_libbtf_path, dependencies=_libbtf_dependencies)
+const libcamd = LazyLibrary(_libcamd_path, dependencies=_libcamd_dependencies)
+const libccolamd = LazyLibrary(_libccolamd_path, dependencies=_libccolamd_dependencies)
+const libcholmod = LazyLibrary(_libcholmod_path, dependencies=_libcholmod_dependencies)
+const libcolamd = LazyLibrary(_libcolamd_path, dependencies=_libcolamd_dependencies)
+const libklu = LazyLibrary(_libklu_path, dependencies=_libklu_dependencies)
+const libldl = LazyLibrary(_libldl_path, dependencies=_libldl_dependencies)
+const librbio = LazyLibrary(_librbio_path, dependencies=_librbio_dependencies)
+const libspqr = LazyLibrary(_libspqr_path, dependencies=_libspqr_dependencies)
+const libsuitesparseconfig = LazyLibrary(_libsuitesparseconfig_path, dependencies=_libsuitesparseconfig_dependencies)
+const libumfpack = LazyLibrary(_libumfpack_path, dependencies=_libumfpack_dependencies)


This is going to be tedious, but you need to express the interdependencies between each of these libraries. E.g. libamd depends on libsuitesparseconfig, and libcholmod depends on libamd, libcamd, libccolamd, etc...

I used a couple of bots to produce and check the list. Just pushed. Does it look reasonable?

topolarity · 2025-05-17T15:10:21Z

Can / should we add a test that checks the dependency list against the ldd (or DT_NEEDED entries) of the .so's?

vtjnash · 2025-05-17T15:43:33Z

BinaryBuilder should already audit that, except for a few know broken libraries

topolarity · 2025-05-17T16:43:55Z

BinaryBuilder should already audit that, except for a few know broken libraries

Right, but it would be good to make sure that we haven't fallen out-of-sync with BinaryBuilder

staticfloat · 2025-05-19T15:07:02Z

it would be good to make sure that we haven't fallen out-of-sync with BinaryBuilder

I agree, I added a test that can be run on macOS or Linux. I (or someone else feeling brave) can make one for Windows as well. It already catches plenty of minor issues:

┌ Warning: Dependency mismatch
│   jll = "SuiteSparse_jll"
│   library = "libbtf"
│   missing_deps = ""
│   extraneous_deps = "libsuitesparseconfig, libamd, libcolamd"
└ @ Main ~/src/julia/test/stdlib_dependencies.jl:169

┌ Warning: Dependency mismatch
│   jll = "SuiteSparse_jll"
│   library = "libldl"
│   missing_deps = ""
│   extraneous_deps = "libsuitesparseconfig"
└ @ Main ~/src/julia/test/stdlib_dependencies.jl:169

┌ Warning: Dependency mismatch
│   jll = "SuiteSparse_jll"
│   library = "libspqr"
│   missing_deps = "libstdc++, libgcc_s"
│   extraneous_deps = "libamd, libcolamd"
└ @ Main ~/src/julia/test/stdlib_dependencies.jl:169

┌ Warning: Dependency mismatch
│   jll = "SuiteSparse_jll"
│   library = "libumfpack"
│   missing_deps = ""
│   extraneous_deps = "libcolamd"
└ @ Main ~/src/julia/test/stdlib_dependencies.jl:169

┌ Warning: Dependency mismatch
│   jll = "MPFR_jll"
│   library = "libmpfr"
│   missing_deps = ""
│   extraneous_deps = "libgmpxx"
└ @ Main ~/src/julia/test/stdlib_dependencies.jl:169
Test Summary:               | Pass  Fail  Total  Time
Stdlib JLL dependency check |   29     5     34  3.9s
RNG of the outermost testset: Random.Xoshiro(0x1329e9815a60ab0e, 0x784c9414423a89d0, 0x1a3aea28fda405f6, 0x8c1dfb6d4dda4f7b, 0xd4081888363c92a9)
ERROR: LoadError: Some tests did not pass: 29 passed, 5 failed, 0 errored, 0 broken.
in expression starting at /home/sabae/src/julia/test/stdlib_dependencies.jl:132

Note that there are some slight differences between macOS and Linux, so you'll have to add some conditionals for those things.

IanButterworth · 2025-05-19T15:37:56Z

Nice

vtjnash · 2025-05-19T21:05:29Z

Blocked by --trim failure

That should not actually be blocking. The required code dependencies should be vendored into the test directory anyways, rather than relying on changeable state elsewhere, since otherwise we can lose test coverage simply if these package files change, which should not be true of test code.

stdlib/LibSSH2_jll/src/LibSSH2_jll.jl

stdlib/SuiteSparse_jll/src/SuiteSparse_jll.jl

staticfloat · 2025-05-20T04:52:24Z

The required code dependencies should be vendored into the test directory anyways, rather than relying on changeable state elsewhere, since otherwise we can lose test coverage simply if these package files change, which should not be true of test code.

I don't quite follow what you mean, can you be more explicit?

This converts more JLL stdlibs to use lazily-loaded libraries, and as a useful side-effect, causes them to be loaded by absolute path, isolating them all from `LD_LIBRARY_PATH`-like effects. fixups lazify dSFMT_jll lazify GMP_jll lazify libLLVM_jll lazify MPFR_jll lazify OpenLibm_jll lazify PCRE2_jll lazify SuiteSparse_jll add small precompiles Add `is_available()` for `LibUV_jll` Add initial `stdlib_dependencies` test This allows for auditing the expressed library dependencies in our LazyLibrary JLL definitions, to ensure that we don't get out of sync with the actual binaries. fix typo add stdlib_dependencies.jl to choosetests test fixes etc. test fixes etc. more test fixes morefix missing eager_modes dep fixes

topolarity · 2025-05-20T23:12:12Z

stdlib/LibCURL_jll/src/LibCURL_jll.jl

 libcurl_path::String = ""

+if Sys.isfreebsd()
+    _libcurl_dependencies = LazyLibrary[]


All of these zero-dependency lists for FreeBSD are surprising - why is it so common for that platform?

IanButterworth requested a review from staticfloat May 17, 2025 00:38

IanButterworth requested a review from giordano as a code owner May 17, 2025 00:38

IanButterworth mentioned this pull request May 17, 2025

Convert more stdlibs to LazyLibraries #58405

Open

IanButterworth force-pushed the ib/more_lazy branch from 6bb0540 to f8388c5 Compare May 17, 2025 03:09

IanButterworth commented May 17, 2025

View reviewed changes

IanButterworth added the latency Latency label May 17, 2025

staticfloat requested changes May 17, 2025

View reviewed changes

IanButterworth force-pushed the ib/more_lazy branch 4 times, most recently from 7fb4a84 to 46fe088 Compare May 17, 2025 12:19

This comment was marked as resolved.

Sign in to view

IanButterworth force-pushed the ib/more_lazy branch from 46fe088 to 75229fe Compare May 17, 2025 18:18

nsajko added the JLLs label May 18, 2025

IanButterworth mentioned this pull request May 18, 2025

precompile the eager_mode noop JuliaPackaging/JLLWrappers.jl#75

Merged

IanButterworth mentioned this pull request May 19, 2025

Package load time regressions in 1.12 #57436

Open

staticfloat approved these changes May 20, 2025

View reviewed changes

stdlib/LibSSH2_jll/src/LibSSH2_jll.jl Show resolved Hide resolved

stdlib/SuiteSparse_jll/src/SuiteSparse_jll.jl Show resolved Hide resolved

IanButterworth force-pushed the ib/more_lazy branch 2 times, most recently from 20f4d4f to a415b58 Compare May 20, 2025 17:12

IanButterworth force-pushed the ib/more_lazy branch from a415b58 to 488c8e9 Compare May 20, 2025 21:38

topolarity reviewed May 20, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert remaining JLL stdlibs to LazyLibraries #58444

Convert remaining JLL stdlibs to LazyLibraries #58444

IanButterworth commented May 17, 2025 •

edited

Loading

topolarity commented May 17, 2025

IanButterworth commented May 17, 2025 •

edited

Loading

IanButterworth May 17, 2025

staticfloat May 17, 2025

staticfloat May 17, 2025

IanButterworth May 17, 2025

topolarity commented May 17, 2025

This comment was marked as resolved.

vtjnash commented May 17, 2025

topolarity commented May 17, 2025

staticfloat commented May 19, 2025

IanButterworth commented May 19, 2025

vtjnash commented May 19, 2025

staticfloat commented May 20, 2025

topolarity May 20, 2025

Convert remaining JLL stdlibs to LazyLibraries #58444

Are you sure you want to change the base?

Convert remaining JLL stdlibs to LazyLibraries #58444

Conversation

IanButterworth commented May 17, 2025 • edited Loading

topolarity commented May 17, 2025

IanButterworth commented May 17, 2025 • edited Loading

IanButterworth May 17, 2025

Choose a reason for hiding this comment

staticfloat May 17, 2025

Choose a reason for hiding this comment

staticfloat May 17, 2025

Choose a reason for hiding this comment

IanButterworth May 17, 2025

Choose a reason for hiding this comment

topolarity commented May 17, 2025

This comment was marked as resolved.

vtjnash commented May 17, 2025

topolarity commented May 17, 2025

staticfloat commented May 19, 2025

IanButterworth commented May 19, 2025

vtjnash commented May 19, 2025

staticfloat commented May 20, 2025

topolarity May 20, 2025

Choose a reason for hiding this comment

IanButterworth commented May 17, 2025 •

edited

Loading

IanButterworth commented May 17, 2025 •

edited

Loading