Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

soften circular dep warning during precompilation #55543

Conversation

IanButterworth
Copy link
Member

@IanButterworth IanButterworth commented Aug 20, 2024

Master

┌ Warning: Circular dependency detected. Precompilation will be skipped for:
│ Base.PkgId(Base.UUID("7f7a1694-90dd-40f0-9382-eb1efda571ba"), "Optimization")
│ Base.PkgId(Base.UUID("bca83a33-5cc9-4baa-983d-23429ab6bcbb"), "OptimizationBase")
│ Base.PkgId(Base.UUID("4297ee4d-0239-47d8-ba5d-195ecdf594fe"), "SymbolicAnalysis")
│ Base.PkgId(Base.UUID("04dc0061-c1f3-5338-88cf-aa3b7fdaee92"), "OptimizationForwardDiffExt")
│ Base.PkgId(Base.UUID("4a213a23-c09c-5cde-9712-b631ad2c72df"), "SymbolicsForwardDiffExt")
│ Base.PkgId(Base.UUID("d479e226-fb54-5ebe-a75e-a7af7f39127f"), "SymbolicsPreallocationToolsExt")
└ @ Base.Precompilation precompilation.jl:552

This PR

┌ Info: Circular dependency detected. Loading may still succeed, but parallel precompilation will be skipped for:
│   Optimization [7f7a1694-90dd-40f0-9382-eb1efda571ba]
│   OptimizationBase [bca83a33-5cc9-4baa-983d-23429ab6bcbb]
│   SymbolicAnalysis [4297ee4d-0239-47d8-ba5d-195ecdf594fe]
│   OptimizationForwardDiffExt [04dc0061-c1f3-5338-88cf-aa3b7fdaee92]
│   SymbolicsForwardDiffExt [4a213a23-c09c-5cde-9712-b631ad2c72df]
└   SymbolicsPreallocationToolsExt [d479e226-fb54-5ebe-a75e-a7af7f39127f]

The full output, showing that in this case loading kind of worked but the experience was quite alarming...

(@v1.12) pkg> add Optimization
...
    Updating `~/.julia/environments/v1.12/Project.toml`
  [7f7a1694] + Optimization v3.27.0
    Updating `~/.julia/environments/v1.12/Manifest.toml`
...
┌ Warning: Circular dependency detected. Loading may still succeed, but parallel precompilation will be skipped for:
│   Optimization [7f7a1694-90dd-40f0-9382-eb1efda571ba]
│   OptimizationBase [bca83a33-5cc9-4baa-983d-23429ab6bcbb]
│   SymbolicAnalysis [4297ee4d-0239-47d8-ba5d-195ecdf594fe]
│   OptimizationForwardDiffExt [04dc0061-c1f3-5338-88cf-aa3b7fdaee92]
│   SymbolicsForwardDiffExt [4a213a23-c09c-5cde-9712-b631ad2c72df]
│   SymbolicsPreallocationToolsExt [d479e226-fb54-5ebe-a75e-a7af7f39127f]
└ @ Base.Precompilation precompilation.jl:560
Precompiling all packages...
  174 dependencies successfully precompiled in 56 seconds. 37 already precompiled. 6 skipped due to circular dependency.

julia> using Optimization
┌ Warning: Circular dependency detected. Loading may still succeed, but parallel precompilation will be skipped for:
│   Optimization [7f7a1694-90dd-40f0-9382-eb1efda571ba]
│   OptimizationBase [bca83a33-5cc9-4baa-983d-23429ab6bcbb]
│   SymbolicAnalysis [4297ee4d-0239-47d8-ba5d-195ecdf594fe]
│   OptimizationForwardDiffExt [04dc0061-c1f3-5338-88cf-aa3b7fdaee92]
│   SymbolicsForwardDiffExt [4a213a23-c09c-5cde-9712-b631ad2c72df]
│   SymbolicsPreallocationToolsExt [d479e226-fb54-5ebe-a75e-a7af7f39127f]
└ @ Base.Precompilation precompilation.jl:560
[ Info: Precompiling Optimization [7f7a1694-90dd-40f0-9382-eb1efda571ba] 
┌ Warning: Module SymbolicsPreallocationToolsExt with build ID ffffffff-ffff-ffff-0009-02c06562b1df is missing from the cache.
│ This may mean SymbolicsPreallocationToolsExt [d479e226-fb54-5ebe-a75e-a7af7f39127f] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:2439
┌ Error: Error during loading of extension SymbolicsPreallocationToolsExt of Symbolics, use `Base.retry_load_extensions()` to retry.
│   exception =
│    1-element ExceptionStack:
│    Error when precompiling module, potentially caused by a __precompile__(false) declaration in the module.
│    Stacktrace:
│      [1] _require(pkg::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:2443
│      [2] __require_prelocked(uuidkey::Base.PkgId, env::Nothing)
│        @ Base ./loading.jl:2291
│      [3] #invoke_in_world#3
│        @ ./essentials.jl:1082 [inlined]
│      [4] invoke_in_world
│        @ ./essentials.jl:1079 [inlined]
│      [5] _require_prelocked
│        @ ./loading.jl:2278 [inlined]
│      [6] _require_prelocked
│        @ ./loading.jl:2277 [inlined]
│      [7] run_extension_callbacks(extid::Base.ExtensionId)
│        @ Base ./loading.jl:1479
│      [8] run_extension_callbacks(pkgid::Base.PkgId)
│        @ Base ./loading.jl:1514
│      [9] run_package_callbacks(modkey::Base.PkgId)
│        @ Base ./loading.jl:1335
│     [10] __require_prelocked(uuidkey::Base.PkgId, env::String)
│        @ Base ./loading.jl:0
│     [11] #invoke_in_world#3
│        @ ./essentials.jl:1082 [inlined]
│     [12] invoke_in_world
│        @ ./essentials.jl:1079 [inlined]
│     [13] _require_prelocked
│        @ ./loading.jl:2278 [inlined]
│     [14] macro expansion
│        @ ./loading.jl:2217 [inlined]
│     [15] macro expansion
│        @ ./lock.jl:273 [inlined]
│     [16] __require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:2174
│     [17] #invoke_in_world#3
│        @ ./essentials.jl:1082 [inlined]
│     [18] invoke_in_world
│        @ ./essentials.jl:1079 [inlined]
│     [19] require(into::Module, mod::Symbol)
│        @ Base ./loading.jl:2167
│     [20] include
│        @ ./Base.jl:582 [inlined]
│     [21] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt128}}, source::String)
│        @ Base ./loading.jl:2796
│     [22] top-level scope
│        @ stdin:4
│     [23] eval
│        @ ./boot.jl:439 [inlined]
│     [24] include_string(mapexpr::typeof(identity), mod::Module, code::String, filename::String)
│        @ Base ./loading.jl:2618
│     [25] include_string
│        @ ./loading.jl:2628 [inlined]
│     [26] exec_options(opts::Base.JLOptions)
│        @ Base ./client.jl:322
│     [27] _start()
│        @ Base ./client.jl:553
└ @ Base loading.jl:1485

@vtjnash
Copy link
Member

vtjnash commented Aug 20, 2024

The error occurs because it fails to declare __precompile__(false). The circularity is not disallowed, but it is forbidden in code that declared it can be precompiled

@nsajko nsajko added compiler:precompilation Precompilation of modules error messages Better, more actionable error messages labels Aug 21, 2024
@jishnub
Copy link
Contributor

jishnub commented Aug 21, 2024

This needs to be documented better

@IanButterworth
Copy link
Member Author

@vtjnash also there could be circularity in the dependency tree but conditional loading might mean the circularity isn't complete.

@vtjnash
Copy link
Member

vtjnash commented Aug 21, 2024

That could be true, but as the message there said "SymbolicsPreallocationToolsExt does not support precompile, but packages that depend on it fail to declare __precompile__(false)"

@IanButterworth
Copy link
Member Author

Sorry, yes. I missed that you were talking about that part.

@topolarity
Copy link
Member

topolarity commented Aug 21, 2024

It's very unusual from my perspective that:

  1. I get this warning repeatedly upon both using X and ] precompile
  2. Despite the statements in this PR that this circularity is not allowed, loading seems to work anyway...

Does (1) go away after __precompile__(false) is added?

(2) presumably means that some extensions do not load, but otherwise loading succeeds? The experience is pretty bad right now though due to #55557

@topolarity
Copy link
Member

Also it's worth mentioning that the most common case is that you hit the error regarding the missing __precompile__(...) and then spend a very long time silently doing pre-compilation

As a user, it's hard to know that the error wasn't fatal and that the using X hasn't hung...

@IanButterworth
Copy link
Member Author

IIUC #55589 should remove the need for this warning

@IanButterworth IanButterworth deleted the ib/precompile_circular_warn_soften branch September 5, 2024 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler:precompilation Precompilation of modules error messages Better, more actionable error messages
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants