Linux support

[IkerGalardi]

Issue to point out some issues I noticed while trying to compile the engine. Will use this in order to track the progress and I'll go adding more tasks when I encounter more problems.

• Get the engine compiling
  • Prepare build scripts
  • Fix compiler warnings and errors
• Link the engine
  • Prepare docker image
    • dxcompiler
    • assimp
    • fmod
    • freeimage
    • freetype
    • bullet
    • SDL2
    • compressonator
    • FSR2
    • LinearMath
    • spirv-cross
    • Open Image Denoiser
  • Fix linking errors and warnings
• Setup github workflows so that linux builds never break.
• Implement audio for linux port

[PanosK92]

This list is a good idea.

The best way to take care of the backslashes would probably be a script with some regular expression.

As for FMOD and Windows.h, I'll remove these, hopefully today.

[IkerGalardi]

I've been thinking on moving the dependencies inside the project, but this could be a pain when compiling for the first time (and cloning the project). What about using system libraries on linux instead of providing precompiled binaries?

[PanosK92]

That's fine, my only worry is the amount of work it might involve.

[IkerGalardi]

No worries, I love challenges 😋. Now that there is at least a small roadmap I can pick up any task on my free time and work on it (or anyone can more easily hop in and help if they want).

[PanosK92]

Had a look at OpenAL Soft, SDL2 and SDL2 mixer, they are more low-level and require more work, so that's something that will probably happen over the weekend, I hope.

[IkerGalardi]

No worries! I can disable the sound subsystem for the linux build in the mean time to see if everything works. Once it builds and launches I can more easily help.

[PanosK92]

802b792 makes FileSystem::OpenDirectoryWindow() cross-platform and removes the windows header on windows as well.

[IkerGalardi]

I'm back! Have more spare time now and would like to start working on this again.

I've been looking at FMOD and it seems to be available as a shared library (DLL), but don't know how well it will work on all linux distributions as I had certain issues redistributing binaries in the past. The ideal solution would be to ditch FMOD and only use SDL or other open source libraries as that could mean that we could compile it for every distribution. Is there any specific feature that is necessary from FMOD that another open source library doesnt have?

[fucksophie]

FMOD supports Linux? https://www.fmod.com/docs/2.00/api/platforms-linux.html

This seems like the only serious issue for the engine to have linux support? I'd love if it did.

[pattakosn]

Hi, I have spend some time trying to compile spartan on linux, here is what i think needs to be done:

• modify premake configuration file for linux
• add linux build script
• fix gcc and/or clang compiler errors (preferably warnings as well)
• handle dependency libraries for linux (include in third parties zip or add in distro specific installation scripts):

• some are easily found on distro's repositories (eg assimp, bullet, freeimage, SDL2, freetype, pugixml-devel)
• some are available for linux but might require manual installation (eg compressonator, dxc shader compiler, fsr2)
• some are not available on linux (disable or replace) (fmod, windres)

[IkerGalardi]

I agree with the steps and will modify the issue to reflect them.

Regarding the dependency stuff, this is a tricky problem with different solutions:

• Including the source code of the dependencies (with FMOD simply include the library and headers). Then build it each time its cloned. This would be the "best" solution so the project depends as little as possible on the linux distribution. Distribution of this engine could be done by bundling all the dependencies and setting the LD_LIBRARY_PATH variable when running the engine/game.
• Building the engine inside a docker image like a modified alpine with all the dependencies and generating a completelly static elf executable. Alpine would be used as it includes a static library for musl libc.
• The third would be to use a package manager like conan or vcpkg to handle all the library dependencies, and distribute like in the first option.

The easiest solution would be to use the docker stuff, but it could be overkill for a research engine. The next easiest solution could be a package manager (conan uses precompiled libraries so no building of dependencies is required).

[PanosK92]

Just a reminder that if you would like to discuss (and coordinate) in a more real time way, the discord server is a good place to do that.

[IkerGalardi]

I've been investigating about the fully static binary using alpine and there is a catch! FMOD only works on glibc based systems, where normally don't support static linking, AFAIK.

I've created a table with dependencies and different linux distros to use as a base:

Library	Available Debian	Available Alpine
dxcompiler	Buildable?	Buildable?
assimp	libassimp-dev	assimp-dev
fmod	Downloadable	No
FreeImageLib	libfreeimage-dev	freeimage-dev
freetype	libfreetype-dev	freetype-dev
Bullet	libbullet-dev	bullet-dev
SDL2	libsdl2-dev	sdl2-dev
Compressonator	Buildable?	Not easy

The Buildable? cases have a question mark I have not tested if they build or not in the given distro. In compressonators case, the build documentation indicates that on linux the init_ubuntu.sh script should be used, but as Ubuntu is based on debian, there should not be any problem when building.

All this could be used in order to create the docker image in which the engine would be built. Distribution could be done by using tools like deptree (shameless spam) to bring all the required dependencies.

If anyone is has any better solution or want's to talk about anything related I'll soon enter the discord so ping me and we could talk more directly.

How should I proceed? Should I wait to see if there are any responses or should I start implementing it?

[PanosK92]

• FMOD: Let's ifdef out FMOD-related code for now. I can handle the audio programming once we find a compatible library. I'll also look into Wwise's (FMOD alternative) glibc dependencies.
• dxcompiler: Building DirectXShaderCompiler should be straightforward, I remember I did it in the past. Ping me if you need any help.
• Dependency Management: The idea of using deptree sounds pretty cool.

Start implementing it, the other Linux contributor is busy with life at the moment :-)

[IkerGalardi]

Cool! I'll start working on getting the engine fully linking and executing next week.

@PanosK92 Have you tried to compile the engine with clang on windows? If that worked I should not encounter any problem with undefined behavior related stuff during execution on linux...

[PanosK92]

Long time ago. Once I finish a couple of commits I am working on now, I'll switch to clang and check the situation 😉

[IkerGalardi]

Is the engine statically linked just because it's easier or is there any specific reason? Linking everything dynamically should make it easier in linux as static linking is a bit more complicated.

What do you think on linking dynamically on linux?

[PanosK92]

It's statically linked so we carry as few DLLs around as possible. If dynamic linking on Linux is easier then let's do that.

[IkerGalardi]

I've recently seen that the Open Image Denoiser dependency has been added so I updated the initial comment of the issue. Is there any other dependency that I have missed?

[PanosK92]

This is the only one.

[heavyrain266]

Hello, let me shade some light on how building and linking engines/software works (or actually doesn't) on Linux. Overly fragmented environment causes numerous issues across popular distributions. Each of them is a very own ecosystem designed around the central package manager, from which developers and end users are pulling binaries and shared objects of selected software and its dependencies. Such design won't work well for paid games of course.

Shared Objects (*.so) files (same as .dlls on Windows) are useful for hot reloading during development. For typical distribution that is targetting either testers or production, you may want to statically link everything (along with libc) in order to avoid so called "dynamic library hell". For example an engine or the game dynamically linked on Ubuntu in 99% cases won't work on Fedora or Arch Linux without weird symlinks to treat older versions of glibc or some other deps as the new ones.

Unfortunatelly, the most commonly used GNU Libc (glibc) cannot be statically linked due to licensing issues (copyleft, GPL). However, the Musl Libc exists as a MIT licensed alternative built to allow full static linking of Linux binaries.

Musl is much smaller in size and was built upon kernel's syscall APIs. From all the known issues, only the very slow implementation of malloc could be a problem, however it can be easly mitigated with Microsoft's mimalloc that is a arena-based alias for default malloc that fixes slow allocation issues as briefly explained by Tweag in their blog post about statically compiled software that is written in Rust and depends on musl instead of glibc,, but mimalloc itself is a cross-platform C/C++ library.

[IkerGalardi]

Commercial games usually target the steam runtime, which is AFAIK a Debian based runtime (basically games run using libraries provided by steam). That would be ideal in order to publish on steam, but as this engine is more focused on being a research project I would discard that idea.

Instead, I proposed to just installing the engine(+game) in the /opt/ directory with all its runtime libraries. A startup script would simply add spartan internal libraries directory to LD_LIBRARY_PATH and start the engine. Its a bit messy, but I think its a good aproximation to kickstart the linux port.

I also thought on using musl and statically linking everything but there is a major roadblock: FMOD. FMOD requires glibc in order to run in linux, so this would mean that the engine would link to both standard libraries and that would be a mess to say the least. If the engine would only depend on open source libraries it would be more achievable, but that is not the case sadly.

And I can imagine that getting rid of FMOD is a pain in the ass so I would not count on that in the near future.

[heavyrain266]

Hm, at this point audio(engine) is very subjective. If game developer don't want to use FMOD, they should have easy way to use e.g. miniaudio or even raw PipeWire if they want. FMOD on the other hand is not providing any value to gamers, only to developers but then both of them will suffer from poor audio at the end because of very low quality drivers...

[IkerGalardi]

Apparently FSR2 is not officially supported by AMD on linux, but I have found a fork that makes it build in linux.

For building on ubuntu you first need to install the Vulkan SDK as this will bring some binaries like glslValidator that are not available on the official ubuntu repositories. This can be done with the next commands:

wget -qO- https://packages.lunarg.com/lunarg-signing-key-pub.asc | sudo tee /etc/apt/trusted.gpg.d/lunarg.asc
sudo wget -qO /etc/apt/sources.list.d/lunarg-vulkan-jammy.list https://packages.lunarg.com/vulkan/lunarg-vulkan-jammy.list
sudo apt update
sudo apt install vulkan-sdk

After that, clone the fork and go to the folder src/ffx-fsr2-api in order to start building the actual static library with the next commands:

mkdir build && cd build/
cmake -DFFX_FSR2_API_DX12=OFF -DFFX_FSR2_API_VK=ON ..
make

Just leaving this here in order to add the library to the build environment when I finish fixing a minor issue in the #155 pull request. Depending on a random fork is not the best idea but it is a workaround that could help bootstrap the linux port, so I will add it anyways.

[IkerGalardi]

@PanosK92 Have you thought of any way to disable certain aspects of the engine at runtime? I've been looking into the new dependencies and linux support for FSR3 and Brixelizer seems to be bad, so it would be a nice idea to disable those technologies for the linux build.

Have you thought on doing some build configuration stuff or would you prefer if I simply #ifdef the implementations out on the linux build for now?

[PanosK92]

Let's #ifdef out anything that is an obstacle to compiling 😉