create_app never finishes on Julia v1.11.1 when JULIA_NUM_THREADS is set

[juanromerohb]

The same app I was successfully compiling in Julia v.1.10.6 under 15 minutes no longer compiles after updating Julia to v1.11.1.

It stucks +20 minutes on

⠸ [03m:30s] PackageCompiler: compiling nonincremental system image

The error that it shows me after Ctrl^C is

InterruptException:
Stacktrace:
  [1] try_yieldto(undo::typeof(Base.ensure_rescheduled))       
    @ Base .\task.jl:958
  [2] wait()
    @ Base .\task.jl:1022
  [3] wait(c::Base.GenericCondition{Base.Threads.SpinLock}; first::Bool)
    @ Base .\condition.jl:130
  [4] wait
    @ .\condition.jl:125 [inlined]
  [5] wait(x::Base.Process, syncd::Bool)
    @ Base .\process.jl:694
  [6] wait
    @ .\process.jl:687 [inlined]
  [7] success
    @ .\process.jl:556 [inlined]
  [8] run(::Cmd; wait::Bool)
    @ Base .\process.jl:513
  [9] run
    @ .\process.jl:510 [inlined]
 [10] #20
    @ C:\Users\juanh\.julia\packages\PackageCompiler\dFEAA\ext\TerminalSpinners.jl:157 [inlined]
 [11] spin(f::PackageCompiler.var"#20#22"{Cmd}, s::PackageCompiler.TerminalSpinners.Spinner{Base.TTY})
    @ PackageCompiler.TerminalSpinners C:\Users\juanh\.julia\packages\PackageCompiler\dFEAA\ext\TerminalSpinners.jl:164       
 [12] macro expansion
    @ C:\Users\juanh\.julia\packages\PackageCompiler\dFEAA\ext\TerminalSpinners.jl:157 [inlined]
 [13] create_sysimg_object_file(object_file::String, packages::Vector{String}, packages_sysimg::Set{Base.PkgId}; project::String, base_sysimage::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, extra_precompiles::String, incremental::Bool, import_into_main::Bool)      
    @ PackageCompiler C:\Users\juanh\.julia\packages\PackageCompiler\dFEAA\src\PackageCompiler.jl:134
 [14] create_sysimg_object_file
    @ C:\Users\juanh\.julia\packages\PackageCompiler\dFEAA\src\PackageCompiler.jl:315 [inlined]
 [15] create_sysimage(packages::Vector{String}; sysimage_path::String, project::String, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, cpu_target::String, script::Nothing, sysimage_build_args::Cmd, include_transitive_dependencies::Bool, base_sysimage::Nothing, julia_init_c_file::Nothing, julia_init_h_file::Nothing, version::Nothing, soname::Nothing, compat_level::String, extra_precompiles::String, import_into_main::Bool)
    @ PackageCompiler C:\Users\juanh\.julia\packages\PackageCompiler\dFEAA\src\PackageCompiler.jl:652
 [16] create_sysimage
    @ C:\Users\juanh\.julia\packages\PackageCompiler\dFEAA\src\PackageCompiler.jl:540 [inlined]
 [17] create_app(package_dir::String, app_dir::String; executables::Nothing, precompile_execution_file::Vector{String}, precompile_statements_file::Vector{String}, incremental::Bool, filter_stdlibs::Bool, force::Bool, c_driver_program::String, cpu_target::String, include_lazy_artifacts::Bool, sysimage_build_args::Cmd, include_transitive_dependencies::Bool, include_preferences::Bool, script::Nothing)
    @ PackageCompiler C:\Users\juanh\.julia\packages\PackageCompiler\dFEAA\src\PackageCompiler.jl:899
 [18] create_app(package_dir::String, app_dir::String)
    @ PackageCompiler C:\Users\juanh\.julia\packages\PackageCompiler\dFEAA\src\PackageCompiler.jl:842
 [19] top-level scope
    @ REPL[4]:1
 [20] eval
    @ .\boot.jl:430 [inlined]
 [21] eval
    @ .\Base.jl:130 [inlined]
 [22] repleval(m::Module, code::Expr, ::String)
    @ VSCodeServer c:\Users\juanh\.vscode\extensions\julialang.language-julia-1.127.2\scripts\packages\VSCodeServer\src\repl.jl:229
 [23] #112
    @ c:\Users\juanh\.vscode\extensions\julialang.language-julia-1.127.2\scripts\packages\VSCodeServer\src\repl.jl:192 [inlined]
 [24] with_logstate(f::VSCodeServer.var"#112#114"{Module, Expr, REPL.LineEditREPL, REPL.LineEdit.Prompt}, logstate::Base.CoreLogging.LogState)
    @ Base.CoreLogging .\logging\logging.jl:522
 [25] with_logger
    @ .\logging\logging.jl:632 [inlined]
 [26] (::VSCodeServer.var"#111#113"{Module, Expr, REPL.LineEditREPL, REPL.LineEdit.Prompt})()
    @ VSCodeServer c:\Users\juanh\.vscode\extensions\julialang.language-julia-1.127.2\scripts\packages\VSCodeServer\src\repl.jl:193
 [27] #invokelatest#2
    @ .\essentials.jl:1055 [inlined]
 [28] invokelatest(::Any)
    @ Base .\essentials.jl:1052
 [29] (::VSCodeServer.var"#64#65")()
    @ VSCodeServer c:\Users\juanh\.vscode\extensions\julialang.language-julia-1.127.2\scripts\packages\VSCodeServer\src\eval.jl:34

The difference I noted was that before PackageCompiler: compiling fresh sysimage (incremental=false), it throws me thousands of warnings like these

Warning: .drectve `-exclude-symbols:"julia_IRInterpretationState_54964.reloc_slot" ' unrecognized
Warning: .drectve `-exclude-symbols:"julia_YY.tarjanNOT.YY.379_64959.reloc_slot" ' unrecognized
Warning: .drectve `-exclude-symbols:jl_fvar_count_5 ' unrecognized
Warning: .drectve `-exclude-symbols:jl_fvar_ptrs_5 ' unrecognized
Warning: .drectve `-exclude-symbols:jl_clone_slots_5 ' unrecognized
Warning: .drectve `-exclude-symbols:jl_clone_idxs_5 ' unrecognized
Warning: corrupt .drectve at end of def file
Warning: .drectve `-exclude-symbols:jl_shard_tables ' unrecognized
Warning: .drectve `-exclude-symbols:jl_pgcstack_func_slot ' unrecognized
Warning: .drectve `-exclude-symbols:jl_pgcstack_key_slot ' unrecognized
Warning: .drectve `-exclude-symbols:jl_tls_offset ' unrecognized
Warning: .drectve `-exclude-symbols:jl_ptls_table ' unrecognized
Warning: .drectve `-exclude-symbols:jl_small_typeof ' unrecognized

[DilumAluthge]

1. Which operating system are you using?
2. Do you have an MWE?

[juanromerohb]

1. Which operating system are you using?
2. Do you have an MWE?

I'm working on Windows 11 using Visual Studio Code with the Julia REPL extension..

A MWE I just tried starts by generating an empty Julia application:

using Pkg
Pkg.generate("MyApp")

Next, I modified the MyApp/src/MyApp.jl file to create a basic "Hello World" program:

module MyApp

function julia_main()::Cint
    println("Hello, World!")
end

end

Then, I run

using PackageCompiler
create_app("MyApp", "MyAppCompiled")

When I tried to compile the application using PackageCompiler:

PackageCompiler: bundled libraries:
  ├── Base:
  │    ├── libLLVM-16jl.dll - 81.921 MiB
  │    ├── libatomic-1.dll - 262.406 KiB
  │    ├── libdSFMT.dll - 114.227 KiB
  │    ├── libgcc_s_seh-1.dll - 767.898 KiB
  │    ├── libgfortran-5.dll - 11.162 MiB
  │    ├── libgmp-10.dll - 1.054 MiB
  │    ├── libgmp.dll - 1.054 MiB
  │    ├── libgmpxx-4.dll - 293.766 KiB
  │    ├── libgmpxx.dll - 293.766 KiB
  │    ├── libgomp-1.dll - 1.645 MiB
  │    ├── libjulia-codegen.dll - 103.560 MiB
  │    ├── libjulia-internal.dll - 12.976 MiB
  │    ├── libmpfr-6.dll - 2.526 MiB
  │    ├── libmpfr.dll - 2.527 MiB
  │    ├── libopenlibm.dll - 384.742 KiB
  │    ├── libpcre2-16-0.dll - 712.625 KiB
  │    ├── libpcre2-16.dll - 712.625 KiB
  │    ├── libpcre2-32-0.dll - 684.148 KiB
  │    ├── libpcre2-32.dll - 683.281 KiB
  │    ├── libpcre2-8-0.dll - 773.922 KiB
  │    ├── libpcre2-8.dll - 774.789 KiB
  │    ├── libpcre2-posix-3.dll - 127.914 KiB
  │    ├── libquadmath-0.dll - 1.137 MiB
  │    ├── libssp-0.dll - 144.766 KiB
  │    ├── libstdc++-6.dll - 25.186 MiB
  │    ├── libuv-2.dll - 984.586 KiB
  │    ├── libwinpthread-1.dll - 330.039 KiB
  │    ├── libz.dll - 233.195 KiB
  │    ├── libjulia.dll - 226.492 KiB
  ├── Stdlibs:
  │   ├── OpenBLAS_jll
  │   │   ├── libopenblas64_.dll - 37.669 MiB
  │   ├── libblastrampoline_jll
  │   │   ├── libblastrampoline-5.dll - 2.250 MiB
  Total library file size: 292.973 MiB
⢰ [01m:04s] PackageCompiler: creating compiler .ji image (incre✔ [04m:09s] PackageCompiler: creating compiler .ji image (incremental=false)
⣠ [01m:35s] PackageCompiler: compiling fresh sysimage (incremental=false)
[pid 17396] waiting for IO to finish:
 Handle type        uv_handle_t->data
⠋ [02m:52s] PackageCompiler: compiling fresh sysimage (incremental=false)

## [thousands of warning lines]

Warning: .drectve `-exclude-symbols:"julia_copytoNOT._61547.reloc_slot" ' unrecognized 
Warning: .drectve `-exclude-symbols:"julia_domsort_ssaNOT._68697.reloc_slot" ' unrecognized
Warning: .drectve `-exclude-symbols:"julia_heapifyNOT._68559.reloc_slot" ' unrecognized
Warning: .drectve `-exclude-symbols:"julia__unsetindexNOT._61390.reloc_slot" ' unrecognized
Warning: .drectve `-exclude-symbols:"julia_heapifyNOT._68550.reloc_slot" ' unrecognized
Warning: .drectve `-exclude-symbols:"julia_valid_typeof_tparam_65525.reloc_slot" ' unrecognized
⣠ [05m:08s] PackageCompiler: compiling fresh sysimage (incremental=false) unrecognized

## [thousands of warning lines]

Warning: .drectve `-exclude-symbols:jl_clone_idxs_5 ' unrecognized
Warning: corrupt .drectve at end of def file
Warning: .drectve `-exclude-symbols:jl_shard_tables ' unrecognized
Warning: .drectve `-exclude-symbols:jl_pgcstack_func_slot ' unrecognized
Warning: .drectve `-exclude-symbols:jl_pgcstack_key_slot ' unrecognized
Warning: .drectve `-exclude-symbols:jl_tls_offset ' unrecognized
Warning: .drectve `-exclude-symbols:jl_ptls_table ' unrecognized
Warning: .drectve `-exclude-symbols:jl_small_typeof ' unrecognized
✔ [05m:08s] PackageCompiler: compiling fresh sysimage (incremental=false)
Precompiling project...
  1 dependency successfully precompiled in 4 seconds
⠸ [01m:39s] PackageCompiler: compiling nonincremental system image
⢰ [20m:17s] PackageCompiler: compiling nonincremental system image

[DilumAluthge]

Any chance you could do a git bisect on Julia between 1.10 and 1.11, to identify the first broken commit?

[juanromerohb]

I don't know how to do that but I tried it in 1.11.0 and I obtained the same bug

[markusgumbel]

I have a very similar or probably the same problem on Ubuntu and can confirm the issue.

Julia Version 1.11.1
Commit 8f5b7ca12ad (2024-10-16 10:53 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 8 × Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
WORD_SIZE: 64
LLVM: libLLVM-16.0.6 (ORCJIT, skylake)
Threads: 6 default, 0 interactive, 3 GC (on 8 virtual cores)
Environment:
JULIA_HOME = /home/markus/.juliaup
JULIA_NUM_THREADS = 6

[matthewgcooper]

I am having the same problem using the same MWE. I ran multiple tests and found out that 1.11.1 only gets stuck if the env variable JULIA_NUM_THREADS is set. The tests I performed are listed below:

• Julia 1.10.5 run with either 1 thread, "-t auto", or with the env var set to "auto". These three cases work as expected and take about 3 minutes to make an app.
• Julia 1.11.1 run with 1 thread or "-t auto". These two cases also work as expected with similar times.
• Julia 1.11.1 run with env var set to "auto". This case breaks with the compilation never ending (see below - I waited for 20mins).

✔ [01m:50s] PackageCompiler: creating compiler .ji image (incremental=false)
⠋ [01m:39s] PackageCompiler: compiling fresh sysimage (incremental=false)
Precompiling project...
1 dependency successfully precompiled in 2 seconds
✖ [20m:29s] PackageCompiler: compiling nonincremental system image

The specs for the failed case are as follows:

Julia Version 1.10.5
Commit 6f3fdf7b362 (2024-08-27 14:19 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
Ubuntu 22.04.4 LTS
uname: Linux 6.8.0-47-generic #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Oct 2 16:16:55 UTC 2 x86_64 x86_64
CPU: 13th Gen Intel(R) Core(TM) i7-13800H:
speed user nice sys idle irq
#1-20 1001 MHz 435481 s 2002 s 86515 s 6243735 s 0 s
Memory: 31.034202575683594 GB (19788.57421875 MB free)
Uptime: 81331.34 sec
Load Avg: 3.25 3.15 3.03
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, goldmont)
Threads: 20 default, 0 interactive, 10 GC (on 20 virtual cores)
Environment:
JULIA_NUM_THREADS = auto
TERMINATOR_DBUS_PATH = /net/tenshu/Terminator2
WINDOWPATH = 2

[DilumAluthge]

@juanromerohb Can you check if JULIA_NUM_THREADS is defined in your environment when you are running create_app()?

[juanromerohb]

@juanromerohb Can you check if JULIA_NUM_THREADS is defined in your environment when you are running create_app()?

I obtain the same results as @matthewgcooper