Lazy artifacts cause precompilation-time warning

[maleadt]

As reported in JuliaGPU/CUDA.jl#2415: If a package (e.g. a JLL) has a lazy artifact that's used during precompilation, e.g. from __init__, that causes a precompilation-time warning: waiting for IO to finish::

julia> using CUDA
 │ Package CUDA not found, but a package named CUDA is available from a registry. 
 │ Install package?
 │   (docs) pkg> add CUDA 
 └ (y/n/o) [y]: y
   Resolving package versions...
   Installed CUDA_Runtime_Discovery ─ v0.3.3
   Installed SentinelArrays ───────── v1.4.3
   Installed LLVM ─────────────────── v7.2.1
   Installed UnsafeAtomicsLLVM ────── v0.1.4
   Installed CUDA_Driver_jll ──────── v0.9.0+0
   Installed GPUArrays ────────────── v10.2.0
   Installed KernelAbstractions ───── v0.9.20
   Installed CUDA_Runtime_jll ─────── v0.14.0+1
   Installed PrettyTables ─────────── v2.3.2
   Installed GPUCompiler ──────────── v0.26.5
   Installed CUDA ─────────────────── v5.4.2
    Updating `~/.julia/dev/Ferrite/docs/Project.toml`
  [052768ef] + CUDA v5.4.2
    Updating `~/.julia/dev/Ferrite/docs/Manifest.toml`
  Downloaded artifact: CUDA_Driver

[pid 611155] waiting for IO to finish:
 Handle type        uv_handle_t->data
 timer              0x1631cf0->0x7f50886d75b0
This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.

This is not a great user experience. It can be reproduced with plain JLLWrappers-generated code, as demonstrated in JuliaGPU/CUDA.jl#2415 (comment). I'm not sure whether this is a "bug"/feature of Pkg, or whether the JLLWrapper-generated code could be improved (e.g., shouldn't the download be postponed until run time, instead of precompilation time?).

[topolarity]

I agree - I'm not sure why we'd want lazy JLL artifacts to move the artifact download from instantiate to precompile. It seems like in either case we end up downloading the artifact as a result of Pkg's usual automatic operations if it's anywhere in your Manifest.

What I really want is that if the package is never using'd it never downloads its artifacts.

[topolarity]

The progress bar is not even visible, so the precompile process can easily appear hung

[vchuravy]

This is a bug/issue with the phase ordering in precompilation.

Precompilation of the current package is special, e.g. we don't run __init__, but when you load said package to precompile another package, we run __init__ because we have to create the "normal" state a user would see.

So if a packages __init__ function uses a lazy artifact then we must download it during precompilation.
Now an alternative might be for JLLWrappers to instead throw a PrecompilationError and say "hey, your precompilation script is doing fishy stuff".

What I really want is that if the package is never using'd it never downloads its artifacts.

Pkg, is only aware of the dependency graph, so we don't know under which conditions using happens.

[giordano]

This was fixed in 1.10 by JuliaLang/Pkg.jl#3950, and in 1.11 by JuliaLang/Downloads.jl#257

[giordano]

What I really want is that if the package is never using'd it never downloads its artifacts.

Problem is that precompilation workloads nowadays cause loading packages before runtime.

[topolarity]

So what's the purpose of a JLL with lazy artifacts nowadays?

[giordano]

MKL_jll was made lazy in JuliaPackaging/Yggdrasil#260 to address JuliaMath/FFTW.jl#133, and back then it did the job, because precompilation was a much lighter process and didn't typically cause packages to be actually loaded. With caching of native code in Julia v1.9 it's now much more likely that in a way or another a JLL is loaded thus triggering a lazy artifact to be downloaded.

If anything, lazy artifacts are useful in cases where there are multiple alternative variants of builds, with lazy artifacts we effectively download only the default build, if they were all non-lazy we'd download loads of them (think of Enzyme_jlll which has about 300 different builds)

[topolarity]

That issue suggests that we've regressed on the intended behavior:

With latest release of MKL_jll, v2019.0.117+2 (don't dare to install v2019.0.117+1, make sure to update the registry) the tarball of MKL_jll is not downloaded upon just installing the package, but only when doing using MKL_jll.

In the context of FFTW.jl, this means that the tarball is never downloaded unless you had set JULIA_FFTW_PROVIDER to MKL. This should effectively fix the issue, so I'm going to close the ticket.

This isn't true any more - installing the package triggers parallel pre-compilation which triggers MKL_jll artifact downloads

[topolarity]

I don't think the download is being triggered by __init__ either.

It seems to happen when pre-compiling the JLL itself:

(@v1.10) pkg> precompile
Precompiling project...
297 dependencies successfully precompiled in 743 seconds. 190 already precompiled.
2 dependencies had output during precompilation:
┌ MKL_jll
│ Downloading artifact: MKL
└
┌ IntelOpenMP_jll
│ Downloading artifact: IntelOpenMP
└

edit: Looks like that probably regressed with #53?

[giordano]

edit: Looks like that probably regressed with #53?

Uh, that looks possible. With JLLWrappers v1.4.1:

$ JULIA_DEPOT_PATH=$(mktemp -d) julia +1.10 -e 'using Pkg; Pkg.add(; name="JLLWrappers", version=v"1.4.1"); Pkg.add("IntelOpenMP_jll")'
  Installing known registries into `/tmp/tmp.ThLHUkl8qt`
    Updating registry at `/tmp/tmp.ThLHUkl8qt/registries/General.toml`
   Resolving package versions...
   Installed Preferences ─ v1.4.3
   Installed JLLWrappers ─ v1.4.1
    Updating `/tmp/tmp.ThLHUkl8qt/environments/v1.10/Project.toml`
⌃ [692b3bcd] + JLLWrappers v1.4.1
    Updating `/tmp/tmp.ThLHUkl8qt/environments/v1.10/Manifest.toml`
⌃ [692b3bcd] + JLLWrappers v1.4.1
  [21216c6a] + Preferences v1.4.3
  [ade2ca70] + Dates
  [de0858da] + Printf
  [fa267f1f] + TOML v1.0.3
  [4ec0a83e] + Unicode
        Info Packages marked with ⌃ have new versions available and may be upgradable.
Precompiling project...
  2 dependencies successfully precompiled in 2 seconds
   Resolving package versions...
   Installed IntelOpenMP_jll ─ v2024.2.1+0
    Updating `/tmp/tmp.ThLHUkl8qt/environments/v1.10/Project.toml`
  [1d5cc7b8] + IntelOpenMP_jll v2024.2.1+0
    Updating `/tmp/tmp.ThLHUkl8qt/environments/v1.10/Manifest.toml`
  [1d5cc7b8] + IntelOpenMP_jll v2024.2.1+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.10.0
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [a4e569a6] + Tar v1.10.0
 [cf7118a7] + UUIDs
  [deac9b47] + LibCURL_jll v8.4.0+0
  [e37daf67] + LibGit2_jll v1.6.4+0
  [29816b5a] + LibSSH2_jll v1.11.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.2+1
  [14a3606d] + MozillaCACerts_jll v2023.1.10
  [83775a58] + Zlib_jll v1.2.13+1
  [8e850ede] + nghttp2_jll v1.52.0+1
  [3f19e933] + p7zip_jll v17.4.0+2
Precompiling project...
  3 dependencies successfully precompiled in 1 seconds. 2 already precompiled.

With latest version (v1.5.0):

$ JULIA_DEPOT_PATH=$(mktemp -d) julia +1.10 -e 'using Pkg; Pkg.add("IntelOpenMP_jll")'
  Installing known registries into `/tmp/tmp.xH9SI1wIde`
    Updating registry at `/tmp/tmp.xH9SI1wIde/registries/General.toml`
   Resolving package versions...
   Installed Preferences ───── v1.4.3
   Installed JLLWrappers ───── v1.5.0
   Installed IntelOpenMP_jll ─ v2024.2.1+0
    Updating `/tmp/tmp.xH9SI1wIde/environments/v1.10/Project.toml`
  [1d5cc7b8] + IntelOpenMP_jll v2024.2.1+0
    Updating `/tmp/tmp.xH9SI1wIde/environments/v1.10/Manifest.toml`
  [692b3bcd] + JLLWrappers v1.5.0
  [21216c6a] + Preferences v1.4.3
  [1d5cc7b8] + IntelOpenMP_jll v2024.2.1+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [4af54fe1] + LazyArtifacts
  [b27032c2] + LibCURL v0.6.4
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg v1.10.0
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.7.0
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.10.0
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll v8.4.0+0
  [e37daf67] + LibGit2_jll v1.6.4+0
  [29816b5a] + LibSSH2_jll v1.11.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.2+1
  [14a3606d] + MozillaCACerts_jll v2023.1.10
  [83775a58] + Zlib_jll v1.2.13+1
  [8e850ede] + nghttp2_jll v1.52.0+1
  [3f19e933] + p7zip_jll v17.4.0+2
Precompiling project...
  5 dependencies successfully precompiled in 34 seconds
  1 dependency had output during precompilation:
┌ IntelOpenMP_jll
│   Downloading artifact: IntelOpenMP
│
│  [pid 2947122] waiting for IO to finish:
│   Handle type        uv_handle_t->data
│   timer              0x23cc2c0->0x7fc56b8efb50
│  This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to b
e closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
│
│  [pid 2947122] waiting for IO to finish:
│   Handle type        uv_handle_t->data
│   timer              0x23cc2c0->0x7fc56b8efb50
│  This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to b
e closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
└

Maybe we can turn the explicit call to find_artifact_dir() into a less side-effect'ful Base.precompile.