include julia.h with shared libraries

[rekabrnalla]

A bug was found in julia where @ccallable functions allocate memory for each argument triggering garbage collection. For a high-throughput loop this is a no go. It was observed that @cfunction function pointers do not allocate per each argument.

JuliaLang/julia#51894

Unfortunately for the shared-library use case with the PackageCompiler.jl using @cfunction interface requires a julia install because the "julia.h" include file is not available to shared libraries created with the PackageCompiler.jl.

Can julia.h be made available to shared libraries made with PackageCompiler.jl so that additional embedded functionality can also be accessed via embedded C julia commands? This would save an installation of Julia being required to link in with the shared library.

[sloede]

You can, by populating the julia_init_h_file option,

PackageCompiler.jl/src/PackageCompiler.jl

Lines 959 to 961 in 466aec3

	- `julia_init_h_file::::Union{String, Vector{String}}`: A file or list of files to include
	in the library bundle, with declarations for the functions defined in the file(s) provided
	via `julia_init_c_file` (default: `PackageCompiler.default_julia_init_header()`).

with appropriate values. For example, to include the default init file header plus the Julia header file, you could try something like

julia_init_h_file = [PackageCompiler.default_julia_init_header(),
                     joinpath(Sys.BINDIR, Base.INCLUDEDIR, "julia", "julia.h")]

[rekabrnalla]

Thanks. I will try that. I couldn't find it in the documentation anywhere.
Will it automatically pull in all of the julia.h include dependencies?

[sloede]

I couldn't find it in the documentation anywhere.

It is considered an "advanced" option and thus only documented in the References section. I'd be open to adding a section on custom header/init files in the main docs though.

Will it automatically pull in all of the julia.h include dependencies?

No, you're right. You'd have to do this manually. I also see currently no sane way to automate this, since you'd have to transitively also include those headers. Except maybe to include the entire joinpath(Sys.BINDIR, Base.INCLUDEDIR, "julia") folder in the library bundle.

[rekabrnalla]

So it seems like this is still a valid issue? To somehow make this more manageable, or specify which functions to rehost in the shared library somehow.

[sloede]

Yes, I still consider this to be a valid issue. However, it is unclear to me if such a functionality should be included in PackageCompiler.jl to be automated, or whether it is something that could be documented but would be left up to the user to do manually.

Maybe @KristofferC has on opinion here?