Hardware acceleration does not work without LD_LIBRARY_PATH hacks #454

Pleune · 2024-04-16T15:28:38Z

Bug description

At this time, the WSL2/WSLg D3D12 hardware acceleration does not work out of the box due to library problems, even though the mesa.drivers package does have D3D12 support enabled by default.

I do not understand exactly what is going on. but there are a few ways to set LD_LIBRARY_PATH as a hack to fix it:

LD_LIBRARY_PATH=/run/opengl-driver/lib works with MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia but not Intel (My laptop has two GPUs)
LD_LIBRARY_PATH=ub:/run/opengl-driver/lib works with both Intel and Nvidia from above, where the ub folder has some basic library files copied from the WSL2 Ubuntu system - see below for contents

In both examples above, /run/opengl-driver/lib can be swapped with /usr/lib/wsl/lib. I'm not sure what the difference is here, other than that /usr/lib/wsl/lib has many more files.

I believe this could be two problems:

the mesa.drivers files are not able to dlopen the WSLg files, since it has no way to know where they are.
The Intel version of the D3D12 sub-driver-??? seems to need some additional libraries to link to that are not currently available.

To Reproduce

With the following config:


  programs.nix-ld = {
    enable = true;
    # I am using this nix-ld-rs too:
    #   package = nix-ld-rs.packages."${pkgs.system}".nix-ld-rs;
  };
  wsl = {
    enable = true;
    defaultUser = "${username}";
    nativeSystemd = true;
    useWindowsDriver = true;
    startMenuLaunchers = true;
  };

  hardware.opengl = {
    enable = true;
    driSupport = true;

    extraPackages = with pkgs; [
      mesa.drivers
      libvdpau-va-gl
      vaapiVdpau
    ];
  };

Then run any of the following commands:

glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel LD_LIBRARY_PATH=ub:/run/opengl-driver/lib glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia LD_LIBRARY_PATH=/run/opengl-driver/lib glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel LD_LIBRARY_PATH=/usr/lib/wsl/lib glxinfo -B
MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia LD_LIBRARY_PATH=ub:/usr/lib/wsl/lib glxinfo -B

The ub folder I am using is: ub.tar.gz

Logs

Example of hardware acceleration working:

> LD_LIBRARY_PATH=ub:/usr/lib/wsl/lib glxinfo -B
name of display: :0
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
WARNING: dzn is not a conformant Vulkan implementation, testing use only.
WARNING: Some incorrect rendering might occur because the selected Vulkan device (Microsoft Direct3D12 (NVIDIA RTX A500 Laptop GPU)) doesn't support base Zink requirements: feats.features.logicOp have_EXT_custom_border_color have_EXT_line_rasterization
glx: failed to create drisw screen
failed to load driver: zink
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Microsoft Corporation (0xffffffff)
    Device: D3D12 (Intel(R) Iris(R) Xe Graphics) (0xffffffff)
    Version: 24.0.3
    Accelerated: yes
    Video memory: 8146MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.1
    Max compat profile version: 4.1
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.0

< snip >

What happens without the LD_LIBRARY_PATH hack:

> glxinfo -B
name of display: :0
MESA: error: ZINK: failed to choose pdev
glx: failed to create drisw screen
failed to load driver: zink
display: :0  screen: 0
direct rendering: Yes
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Mesa (0xffffffff)
    Device: llvmpipe (LLVM 17.0.6, 256 bits) (0xffffffff)
    Version: 24.0.3
    Accelerated: no
    Video memory: 7770MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.5
    Max compat profile version: 4.5
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2

< snip >

Where the ub folder has the following files:

> ls ub
libbsd.so.0
libedit.so.2
libmd.so.0
libstdc++.so.6
libtinfo.so.6
libz.so.1

WSL version

WSL version: 2.2.2.0
Kernel version: 5.15.150.1-2
WSLg version: 1.0.61
MSRDC version: 1.2.5105
Direct3D version: 1.611.1-81528511
DXCore version: 10.0.25131.1002-220531-1700.rs-onecore-base2-hyp
Windows version: 10.0.22631.3296

The text was updated successfully, but these errors were encountered:

SuperSandro2000 · 2024-04-16T16:09:24Z

/usr/lib/wsl/lib is symlinked into /run/opengl-driver/lib

Can you run the same commands with strace to see if it searches the right directories?

Pleune · 2024-04-16T16:50:36Z

Here is the output of MESA_D3D12_DEFAULT_ADAPTER_NAME=Nvidia strace -o strace.log glxinfo -B, where llvmpipe is used instead of the GPU

strace.log

I can see many references to /run/opengl-driver/lib here, so it must be searching that path. Maybe the initial d3d12 library is being loaded, but then when the code inside that runs, dlopen fails?

SuperSandro2000 · 2024-04-16T17:38:30Z

It is not searching for d3d12 in the right location.

openat(AT_FDCWD, "/etc/drirc", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/mitch/.drirc", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/vcxkk3l25hlix62aivl2prfhpczrvymp-expat-2.6.2/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/zph9xw0drmq3rl2ik5slg0n2frw9lw5m-zlib-1.3.1/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/hqvkggqxqslgy2ahp1zswgh7im2kl5im-libxcb-1.16/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/qmy3imvwp6l129avxifnczw4p3fkdz0j-libX11-1.8.7/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/zmpw448943piz8vdav28j8dqfy54xwjc-libxshmfence-1.3.2/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ss6gh67xv5jw6jh0l7dwmyx9823wvb60-zstd-1.5.5/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/blw10rx1cayp2n2pkmyihpipifzgj2xq-wayland-1.22.0/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/1birlqjpkhsadz9zxa1in8z1apz7l1x4-systemd-minimal-libs-255.4/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ixi9ca45786dq510fk6h7crmi9259qcd-libdrm-2.4.120/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/c2yb135iv4maadia5f760b3xhbh6jh61-gcc-13.2.0-lib/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/5xylmnrx48vqlcfv1daj63kql91gflx3-mesa-24.0.3-drivers/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/ddwyrxif62r8n6xclvskjyy6szdhvj60-glibc-2.39-5/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v3/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v3/", 0x7ffdb4a032e0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v2/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/glibc-hwcaps/x86-64-v2/", 0x7ffdb4a032e0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/libd3d12.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/nix/store/rxganm4ibf31qngal3j3psp20mak37yy-xgcc-13.2.0-libgcc/lib/", {st_mode=S_IFDIR|0555, st_size=4096, ...}, 0) = 0
futex(0x7fe8aa4b4e1c, FUTEX_WAKE_PRIVATE, 2147483647) = 0
openat(AT_FDCWD, "/nix/store/bhfs2c58cvgfmr5lh2j3p5032sy0bg11-mesa-24.0.3/share/drirc.d", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
fstat(4, {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0
getdents64(4, 0x186db00 /* 4 entries */, 32768) = 144
getdents64(4, 0x186db00 /* 0 entries */, 32768) = 0
close(4)                                = 0
openat(AT_FDCWD, "/nix/store/bhfs2c58cvgfmr5lh2j3p5032sy0bg11-mesa-24.0.3/share/drirc.d/00-mesa-defaults.conf", O_RDONLY) = 4

Pleune · 2024-04-16T17:57:38Z

hmm, libd3d12.so does exist in my nix store in from the wsl-lib and opengl-drivers packages:

> find /nix -name libd3d12.so
/nix/store/4mpn0189n0wrbi6xscricgsrdkx26bmx-wsl-lib/lib/libd3d12.so
/nix/store/gbfz16cazfd57f45654x916g06cwifxf-opengl-drivers/lib/libd3d12.so
/nix/store/1h8c6cx18mwn0i0kcz25mzbbf87mf8jp-opengl-drivers/lib/libd3d12.so
/nix/store/lk3pnpksmdw28m0nlj069vvg63l7ljjk-opengl-drivers/lib/libd3d12.so
/nix/store/0lxfb1j4yi3r0kaggq0hg0ad6q0i57ix-opengl-drivers/lib/libd3d12.so
/nix/store/57nb4gx6pmn46mvc8hlr26jmgm5j6i18-opengl-drivers/lib/libd3d12.so
/nix/store/6xq076h9n17fhb1w3f1pv1flspx1386m-opengl-drivers/lib/libd3d12.so
/nix/store/izbp5v8nvzs18gipcmd31iyywaqx10da-opengl-drivers/lib/libd3d12.so
/nix/store/v4wwzpgiazb98xjb0v4z0qpn4g2jwv84-opengl-drivers/lib/libd3d12.so
/nix/store/xzxjmcny8xdqv7b2sm0ws77zz65x9ylr-opengl-drivers/lib/libd3d12.so
/nix/store/10qydkiksz0zb9xn6fvlwr2s7swanzn1-opengl-drivers/lib/libd3d12.so
/nix/store/qq421zmhp9h5702mqlhirpxbf2jp5xb8-opengl-drivers/lib/libd3d12.so
/nix/store/hsd21kw4lpaxpivdyjrgz8c5vqm5iaq7-opengl-drivers/lib/libd3d12.so

I'm a good bit out of my depth here, but is there a way to add these packages as runtime dependencies for the mesa package?

Setting any of these directories as the LD_LIBRARY_PATH does allow at least the Nvidia d3d12 adapter to work

Pleune · 2024-04-16T18:43:07Z

Thank you, you have given my enough information to discover the hardware.opengl.setLdLibraryPath = true; option, which makes the Nvidia driver work. Perhaps it would be best to set this as default.

For MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel, I need to also get libedit and libtinfo into /run/opengl-drivers somehow. Perhaps this is a NixOS bug and not a NixOS on WSL bug.

Pleune · 2024-04-16T19:34:49Z

Alright, for a minimal configuration.nix only snippet, I have the following:

  hardware.opengl = {
    enable = true;
    driSupport = true;
    setLdLibraryPath = true;

    extraPackages = with pkgs; [
      mesa.drivers
      libvdpau-va-gl
      (libedit.overrideAttrs (attrs: {postInstall = (attrs.postInstall or "") + ''ln -s $out/lib/libedit.so $out/lib/libedit.so.2'';}))
    ];
  };

The libedit package only proides libedit.so.5 and libedit.so.0. d3d12 is looking for .2, so for the time being, this will work around that. It seems to pull in all of the other required library files as well.

sjhaleprogrammer · 2024-06-01T23:42:50Z

Traceback (most recent call last):
  File "/home/samuel/git/stable-diffusion-webui/launch.py", line 48, in <module>
    main()
  File "/home/samuel/git/stable-diffusion-webui/launch.py", line 39, in main
    prepare_environment()
  File "/home/samuel/git/stable-diffusion-webui/modules/launch_utils.py", line 386, in prepare_environment
    raise RuntimeError(
RuntimeError: Torch is not able to use GPU; add --skip-torch-cuda-test to COMMANDLINE_ARGS variable to disable this check
(venv)

how do i get my nvidia gpu to work

Parsifa1 · 2024-08-02T19:35:29Z

Thank you, you have given my enough information to discover the hardware.opengl.setLdLibraryPath = true; option, which makes the Nvidia driver work. Perhaps it would be best to set this as default.

For MESA_D3D12_DEFAULT_ADAPTER_NAME=Intel, I need to also get libedit and libtinfo into /run/opengl-drivers somehow. Perhaps this is a NixOS bug and not a NixOS on WSL bug.

It seems that the option hardware.opengl.setLdLibraryPath = true; no longer exists

Pleune · 2024-08-12T15:03:48Z

@Parsifa1 Replace with

environment.sessionVariables.LD_LIBRARY_PATH = [ "/run/opengl-driver/lib" ];

@SuperSandro2000, what do you think about adding this to the repo by default, particularly since this easier option is removed now? It is still needed on my system for my GPU to function, and I also don't think it would cause anyone else any issues. I would be happy to submit a PR with more documentation and explanation if you think that would be proper.

Perhaps we could even add options to enable the other paths I have had to added for Cuda and whatnot to work. Below is the full environment section I have been using now:

environment.sessionVariables = {
  CUDA_PATH = "${pkgs.cudatoolkit}";
  EXTRA_LDFLAGS = "-L/lib -L${pkgs.linuxPackages.nvidia_x11}/lib";
  EXTRA_CCFLAGS = "-I/usr/include";
  LD_LIBRARY_PATH = [
    "/usr/lib/wsl/lib"
    "${pkgs.linuxPackages.nvidia_x11}/lib"
    "${pkgs.ncurses5}/lib"
    "/run/opengl-driver/lib"
  ];
  MESA_D3D12_DEFAULT_ADAPTER_NAME = "Nvidia";
};

SuperSandro2000 · 2024-08-12T16:45:10Z

LD_LIBRARY_PATH is a big foot gun and can easily cause weird and hard to debug loading errors. I would love to really avoid it. If it is necessary we can add an option for it but I wouldn't turn it on by default.

The extra variables seem to be very user specific. I don't think they are a good fit.

polybluez · 2024-08-14T18:08:02Z

Maybe something like nixGL would be useful

Pleune · 2024-08-15T14:32:32Z

Maybe something like nixGL would be useful

I have actually tried this and it does not really change anything.

As to Sandro's comments, Everything other than "/run/opengl-driver/lib" is for the windows/wsl nvidia drivers to work, and to support cuda properly. It would be better to include the two pkgs additions in another way, like symlinking them into the same opengl lib folder so they don't end up effecting other parts of the system, but I was not able to get that to work.

polybluez · 2024-08-15T15:54:37Z

Maybe something like nixGL would be useful

I have actually tried this and it does not really change anything.

No, I mean having an utility like nixGL for running an app with WSL DirectX drivers or even adding support for it in nixGL

anticrisis · 2024-08-22T10:30:24Z

@SuperSandro2000 you said:

It is not searching for d3d12 in the right location.

I'm having the same ENOENT errors in strace when I launch emacs. I just installed NixOS on WSL in the past week or so, just trying to set everything up.

You also said:

/usr/lib/wsl/lib is symlinked into /run/opengl-driver/lib

but on my system, /usr/lib/wsl/lib is not symlinked. It contains all the cuda and d3d12 library files. And I apparently have no /run/opengl-driver/lib directory.

Pleune · 2024-08-22T10:53:54Z

@SuperSandro2000 you said:

It is not searching for d3d12 in the right location.

I'm having the same ENOENT errors in strace when I launch emacs. I just installed NixOS on WSL in the past week or so, just trying to set everything up.

You also said:

/usr/lib/wsl/lib is symlinked into /run/opengl-driver/lib

but on my system, /usr/lib/wsl/lib is not symlinked. It contains all the cuda and d3d12 library files. And I apparently have no /run/opengl-driver/lib directory.

You may have to reboot your wsl system to get the run folder back. For some reason mine is still unreliable, particularly after a laptop sleep.

But I have been looking more into exactly what files are needed to be linked into to keep WSLg happy, without causing any ld hack issues. To settle onto a "perfect solution" I'm considering trying changing the symlink from wsl/lib into a copy and elfpatch instead. I've had some success doing this manually now, but have not had the time to properly add it to the repo config and try it out.

anticrisis · 2024-08-22T11:14:40Z

You may have to reboot your wsl system to get the run folder back. For some reason mine is still unreliable, particularly after a laptop sleep.

wsl --shutdown wasn't doing it, but a reboot of the machine, did bring back the /run/opengl-driver/lib/ directory.

jacereda · 2024-08-22T11:37:11Z

Are you setting this?

NixOS-WSL/modules/wsl-distro.nix

Line 77 in 0bec2bf

extraPackages = mkIf cfg.useWindowsDriver [

anticrisis · 2024-08-22T22:26:40Z

Are you setting this?

NixOS-WSL/modules/wsl-distro.nix

Line 77 in 0bec2bf

extraPackages = mkIf cfg.useWindowsDriver [

No! Seems quite important. I only read the documentation, I should have read the code. I will give that a go.

cmspam · 2024-10-22T19:42:40Z

Trying to get this working on AMD graphics. I have dual AMD/Nvidia. Nvidia is fine with the fixes here.
I've looked at strace but can't figure out what else to check to see what could be missing for AMD. Any advice to check?

cmspam · 2024-10-23T05:05:22Z

To fix it for radeon, I had to add the following to my configuration.nix

environment.sessionVariables = {
LD_LIBRARY_PATH = [
"${pkgs.openssl.out}/lib"
];
};

For some reason it requires libssl.so.3 to have radeon work. I wonder if this couldn't be done in a better way though.

Pleune added the bug Something isn't working label Apr 16, 2024

SuperSandro2000 mentioned this issue Oct 26, 2024

Unable to use GPU inside NVIDIA docker container #578

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hardware acceleration does not work without LD_LIBRARY_PATH hacks #454

Hardware acceleration does not work without LD_LIBRARY_PATH hacks #454

Pleune commented Apr 16, 2024 •

edited

Loading

SuperSandro2000 commented Apr 16, 2024

Pleune commented Apr 16, 2024

SuperSandro2000 commented Apr 16, 2024

Pleune commented Apr 16, 2024 •

edited

Loading

Pleune commented Apr 16, 2024 •

edited

Loading

Pleune commented Apr 16, 2024 •

edited

Loading

sjhaleprogrammer commented Jun 1, 2024 •

edited by SuperSandro2000

Loading

Parsifa1 commented Aug 2, 2024

Pleune commented Aug 12, 2024 •

edited

Loading

SuperSandro2000 commented Aug 12, 2024 •

edited

Loading

polybluez commented Aug 14, 2024

Pleune commented Aug 15, 2024

polybluez commented Aug 15, 2024 •

edited

Loading

anticrisis commented Aug 22, 2024 •

edited

Loading

Pleune commented Aug 22, 2024 •

edited

Loading

anticrisis commented Aug 22, 2024

jacereda commented Aug 22, 2024

anticrisis commented Aug 22, 2024

cmspam commented Oct 22, 2024

cmspam commented Oct 23, 2024 •

edited

Loading

Hardware acceleration does not work without LD_LIBRARY_PATH hacks #454

Hardware acceleration does not work without LD_LIBRARY_PATH hacks #454

Comments

Pleune commented Apr 16, 2024 • edited Loading

Bug description

To Reproduce

Logs

WSL version

SuperSandro2000 commented Apr 16, 2024

Pleune commented Apr 16, 2024

SuperSandro2000 commented Apr 16, 2024

Pleune commented Apr 16, 2024 • edited Loading

Pleune commented Apr 16, 2024 • edited Loading

Pleune commented Apr 16, 2024 • edited Loading

sjhaleprogrammer commented Jun 1, 2024 • edited by SuperSandro2000 Loading

Parsifa1 commented Aug 2, 2024

Pleune commented Aug 12, 2024 • edited Loading

SuperSandro2000 commented Aug 12, 2024 • edited Loading

polybluez commented Aug 14, 2024

Pleune commented Aug 15, 2024

polybluez commented Aug 15, 2024 • edited Loading

anticrisis commented Aug 22, 2024 • edited Loading

Pleune commented Aug 22, 2024 • edited Loading

anticrisis commented Aug 22, 2024

jacereda commented Aug 22, 2024

anticrisis commented Aug 22, 2024

cmspam commented Oct 22, 2024

cmspam commented Oct 23, 2024 • edited Loading

Pleune commented Apr 16, 2024 •

edited

Loading

Pleune commented Apr 16, 2024 •

edited

Loading

Pleune commented Apr 16, 2024 •

edited

Loading

Pleune commented Apr 16, 2024 •

edited

Loading

sjhaleprogrammer commented Jun 1, 2024 •

edited by SuperSandro2000

Loading

Pleune commented Aug 12, 2024 •

edited

Loading

SuperSandro2000 commented Aug 12, 2024 •

edited

Loading

polybluez commented Aug 15, 2024 •

edited

Loading

anticrisis commented Aug 22, 2024 •

edited

Loading

Pleune commented Aug 22, 2024 •

edited

Loading

cmspam commented Oct 23, 2024 •

edited

Loading