Linking julia to Graphcore's LLVM

[giordano]

Graphcore recently open-sourced their LLVM backend. This hasn't been upstreamed yet, but we can try to link julia to a libllvm built supporting the Colossus target. In this ticket I'll collect the steps I'm following to try and build Julia with Graphcore's LLVM.

Note: graphcore's LLVM fork currently is built on top of LLVM 15, which means we have to use it with current (as of 2023-06-12) Julia's master branch, what will become Julia v1.10 in the next few months. Following instructions are updated as of today, if you attempt to replicate this in the future, you may have to do something else.

---

Clone Julia's repository

git clone https://github.com/JuliaLang/julia
cd julia

Inside this directory create a file called Make.user with the following content

# -*- makefile -*-

USE_BINARYBUILDER_LLVM = 0
DEPS_GIT = llvm
LLVM_GIT_URL = https://github.com/graphcore/llvm-project-fork
LLVM_BRANCH = standalone_rel_15_x
LLVM_SHA1 = $(LLVM_BRANCH)
LLVM_TARGETS = host;NVPTX;AMDGPU;WebAssembly;BPF;AVR;Colossus
# Apply Julia's patches to LLVM
LLVM_APPLY_JULIA_PATCHES = 1
# Clang is currently needed by Graphcore's LLVM, otherwise we don't really need it
BUILD_LLVM_CLANG = 1
# For faster re-compilation in case something fails, it requires `ccache` in `PATH`
USECCACHE = 1

Then run the build with

make -j24

replacing 24 with an appropriate number of parallel jobs you want to use on your machine.

[giordano]

For the record, I managed to link Julia to Graphcore's LLVM. I need to put better documentation for how to build it in Julia, but the plumbing work is done. Note: maleadt/LLVM.jl#354 is required (but that will be in LLVM.jl v6, you probably have to make the change locally for the time being).

[giordano]

Other PRs related to this effort:

• Adapt to upstream changes. JuliaGPU/GPUCompiler.jl#473 (requires GPUCompiler v0.21)
• [build] Allow automatically applying Julia's patches to LLVM source JuliaLang/julia#50208 (this would make it easier to apply Julia patches to Graphcore's fork, the setup I tried worked out-of-the-box without any conflicts)
• [deps] Deal with USECCACHE=1 in common CMake options JuliaLang/julia#50146 (not strictly necessary, but this allows us to use ccache also for re-building LLVM, if needed)

Also, I had to make some other changes in this package, collected in branch https://github.com/giordano/IPUToolkit.jl/tree/mg/colossus.

One weird thing is that if annotating the for loop in the pi example with @simd most of the kernel gets optimised out, leaving the compiled kernel with

define void @_Z11_2___Pi_229() local_unnamed_addr #0 !dbg !36 {
top:
  %0 = call i64 @get_vec_ptr_Pi(i32 0), !dbg !40
  %1 = call i32 @get_vec_size_Pi(i32 0), !dbg !40
  %2 = call i64 @get_vec_ptr_Pi(i32 1), !dbg !40
  %3 = call i32 @get_vec_size_Pi(i32 1), !dbg !40
  %4 = call i64 @get_vec_ptr_Pi(i32 2), !dbg !40
  %5 = call i32 @get_vec_size_Pi(i32 2), !dbg !40
  %6 = call i32 @llvm.colossus.get.scount.l(), !dbg !41
  unreachable, !dbg !50
}

I looked into this a little bit with @gbaraldi, we added some @show @__LINE__, mod debugging lines to optimize! and found that the kernel is mostly optimised out very early on, already after https://github.com/JuliaGPU/GPUCompiler.jl/blob/e1f9a7d916164017c7e091edecc652a3da1ce53f/src/optim.jl#L181-L185. It looks like the call to the 3-vector-argument function Pi disappears completely, somehow.

Not sure I'll have much time to further investigate this, but I'm leaving the breadcrumbs for future reference.

[giordano]

Weirdly enough, trying to ]build this package with the Graphcore's fork of LLVM fails with

/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found
/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found
/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found
/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found
/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found
/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found
/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:11: fatal error: 'gnu/stubs-32.h' file not found
CLCursor (CLTranslationUnit) /cm/shared/apps/graphcore/sdk/3.0.0/include/poplar/VectorLayout.hpp
CLCursor (CLTranslationUnit) /cm/shared/apps/graphcore/sdk/3.0.0/include/poplar/DeviceManager.hpp
ERROR: LoadError: AssertionError: ptr_ref[] != C_NULL
Stacktrace:
  [1] TokenList(tu::Ptr{CXTranslationUnitImpl}, sr::CXSourceRange)
    @ Clang ~/.julia/packages/Clang/skgwS/src/token.jl:13 [inlined]
  [2] tokenize(c::CLParmDecl)
    @ Clang ~/.julia/packages/Clang/skgwS/src/token.jl:105
  [3] optionals(args::Vector{CLParmDecl})
    @ Main ~/repo/julia-ipu/deps/build.jl:149
  [4] constructor_handler(ctx::DefaultBindgenContext, method::CLConstructor)
    @ Main ~/repo/julia-ipu/deps/build.jl:262
  [5] iterate_children(ctx::DefaultBindgenContext, childvec::Vector{CLCursor})
    @ Main ~/repo/julia-ipu/deps/build.jl:544
  [6] iterate_children(ctx::DefaultBindgenContext, childvec::Vector{CLCursor}) (repeats 2 times)
    @ Main ~/repo/julia-ipu/deps/build.jl:570
  [7] gen_bindings(headers::Vector{String}, blacklist::Vector{String})
    @ Main ~/repo/julia-ipu/deps/build.jl:601
  [8] build_bindings(; path::String, generate_bindings::Bool, compile::Bool)
    @ Main ~/repo/julia-ipu/deps/build.jl:609
  [9] top-level scope
    @ ~/repo/julia-ipu/deps/build.jl:643
 [10] include(fname::String)
    @ Base.MainInclude ./client.jl:478
 [11] top-level scope
    @ REPL[2]:1
in expression starting at /home/mose/repo/julia-ipu/deps/build.jl:643

I don't know if that's relevant, but the fatal error to find system header files is unexpected, the output should usually be free of warnings/errors. Nothing of this happens with official build of Julia (tried both JuliaLang/julia@c58e508 and JuliaLang/julia@5939e2d), I fear this is a weird interaction with libclang_jll, will have to try to use Graphcore's libclang.

[vchuravy]

Yeah... This is not at all a well established build scenario

[giordano]

Ok, I can confirm using Graphcore's libclang with

julia> set_preferences!(
           "LocalPreferences.toml",
           "Clang_jll",
           "libclang_path" => "/home/mose/repo/julia/usr/lib/libclang.so",
           "libclang_cpp_path" => "/home/mose/repo/julia/usr/lib/libclang-cpp.so",
           "clang_path" => "/home/mose/repo/julia/usr/bin/clang",
       )

does the trick, building the wrappers is now successful 🥳

[giordano]

For my own record, for debugging the problem above with the kernel being completely wiped out and replaced by uneachable, Tim suggested using LLVM.clopts("--print-after-all", "--filter-print-funcs=kernel"), with the caveat that kernel is the name of the function, and I'll need to fight name mangling a little bit.