Wrap LLD into python

[ArachnidAbby]

Based on: Add dependency on LLD, wrap it into python

todo:

• Bring the old pr up-to-date with the 800 commits made to llvmlite since the pr opened
• Fix any broken code from the original pr
• add .binding.lld module with functions for each platform lld supports
• complete remaining items on the original PR
• add documentation for any newly added functions

items remaining in the original PR

• Make all Travis tests pass, test LLD on 64bit linux
• Write a test for 32 bits linux? Currently only 64 bits are tested. (not required)
• Get it to compile on macOS
• Get it to compile on Windows
• Get the PyPy test working (not required)
• Implement LLD for macOS and test it
• Implement LLD for Windows and test it
  (all non-required items are going to be checked off)

I will update you as I complete items.

[gmarkall]

Many thanks for opening this PR!

[ArachnidAbby]

@esc damn, can you rerun the tests? One of them had an http error while doing a conda install.

[esc]

@esc damn, can you rerun the tests? One of them had an http error while doing a conda install.

done

[ArachnidAbby]

sick, thank you!

[PrimozGodec]

Thank you for preparing those changes. It is a great addition. What is the state of this PR? Do you think it can be merged soon?

[esc]

@PrimozGodec the state is: pending on review and scheduled for inclusion in the 0.43 release of llvmlite. If you want, feel free to test this and add a review in case you discover any snags or comment with: "works for me" if no snags are to be found.

[ArachnidAbby]

If you decide to try it out, remember to run conda install -c conda-forge libstdcxx-ng=12 if you are on linux. I couldn't get it working with llvm14 without doing this.

[ArachnidAbby]

btw, changed my GH name so my old @ isn't formatted anymore.

[esc]

Unfortunately we couldn't find a reviewer for this, so it has been punted to the 0.44 release.

[ArachnidAbby]

damn, that's a bummer :(

Just tell me if there is anything I can do to make it easier when you do find a reviewer. Good luck with the next release!

[dlee992]

[Not a reviewer] Thanks for this PR! In Numba side, I often have an issue: when using typed.dict or typed.list, currently numba compiles a C-version of them as the underlying implementation. Then when users call a method of these types in jit-ed code, the actual call chain is: user jitted func -> numba type wrapper -> c implementation by ctypes.

So the issue is when calling into the c impl, runtime has to create a call frame for c func, but sometimes the c func is so small, then the creation of call frame becomes the performance bottleneck. A more concrete example can be found: numba/numba#9520 (comment).

I wonder if this PR can help solve the issue fundamentally? ex: compiling and linking the C code as a LLVM IR module, then enable further optimizations, e.g., inlining, vectorization. I am not sure if what I want is another feature, not this PR.

[ArachnidAbby]

This PR wont work with the conda-forge version of the llvm-dev. It only works with the one on the numba channel because the one on the numba channel includes lld. Using another lld package on conda (for the llvm16 tests only) caused another problem it seems. I can try to work to fix this, but it might be easier to just put out a llvm-dev release for llvm16 on the conda channel.

I'll work towards getting to work with a separately installed lld package for the llvm16 tests. Could take me a minute because my linux install is acting really goofy since updating it. At some point I'll reinstall linux but I'll just deal with the annoyances for now.