any updates on visualization in remote server?

[jhchong0405]

Hi I am having this issue while setting up SAPIEN simulator in VNC server on a remote server.

File "/root/miniconda3/envs/dexart2/lib/python3.8/site-packages/sapien/example/hello_world.py", line 38, in main
viewer = Viewer(renderer) # Create a viewer (window)
File "/root/miniconda3/envs/dexart2/lib/python3.8/site-packages/sapien/utils/viewer.py", line 223, in init
self.window = self.renderer.create_window(
RuntimeError: vk::PhysicalDevice::getSurfaceFormatsKHR: ErrorUnknown

I read through the issues and am aware that you mentioned in Aug 2021 that you don't recommend doing SAPIEN visualization remotely, I would like to know if there is any updates to this issue. Is there any way to set up SAPIEN simulator for visualization in VNC server for remote server now (Jan 2024)?

"The first thing you should try is setting the environment variable VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json (make sure the file exists). If other issues happen with present surfaces, you can then disable display. visualization is very hard to set up on a remote server and there are specific VNC parameters required. I recommend not doing visualization remotely."

Originally posted by @fbxiang in #34 (comment)

[fbxiang]

Hi, I have extensively tested remote visualization recently and here are the findings.
First, Vulkan does not officially support any kind of X-forwarding, so I cannot guarantee that remote visualization will work. (It can depend on many factors, including OS version, driver version, GPU vendor, and luck.) However, if you do not need on-screen rendering (i.e. open a window), Vulkan is very reliable.

If you are lucky, the following steps will make VNC to work.

Offscreen rendering on a server

To use SAPIEN on a GPU server without display, the only system dependencies required are libegl1 and libxext6. If using NVIDIA docker environment, enable graphics, utility, and compute by setting the environment variable in the
Dockerfile.

ENV NVIDIA_DRIVER_CAPABILITIES graphics,utility,compute

Virtual desktop on a server

To use SAPIEN on a GPU server with virtual display, additionally install xvfb, x11vnc, and any window manager such as fluxbox or xfce. Add display capabilities for NVIDIA docker.

ENV NVIDIA_DRIVER_CAPABILITIES graphics,utility,compute,display

Assuming fluxbox, start a VNC server by

x11vnc -create -env FD_PROG=/usr/bin/fluxbox  -env X11VNC_FINDDISPLAY_ALWAYS_FAILS=1 -env X11VNC_CREATE_GEOM=${99:-1920x1080x16} -gone 'pkill Xvfb' -nopw
# Note: you should only allow local access and tunnel through ssh

Now you can connect to the server at port 5900. SAPIEN should be fully functional, test with

python -m sapien.exapmle.hello_world

[jhchong0405]

Thank you for your helpful reply! If you have any updates on remote visualization, please let me know!

[lucywang720]

Hi, if i just want it to generate a video instead of matplotlib rendering, is this issue suitable for my problem?

[wzds2015]

Hi, I have extensively tested remote visualization recently and here are the findings. First, Vulkan does not officially support any kind of X-forwarding, so I cannot guarantee that remote visualization will work. (It can depend on many factors, including OS version, driver version, GPU vendor, and luck.) However, if you do not need on-screen rendering (i.e. open a window), Vulkan is very reliable.

If you are lucky, the following steps will make VNC to work.

Offscreen rendering on a server

To use SAPIEN on a GPU server without display, the only system dependencies required are libegl1 and libxext6. If using NVIDIA docker environment, enable graphics, utility, and compute by setting the environment variable in the Dockerfile.

ENV NVIDIA_DRIVER_CAPABILITIES graphics,utility,compute

Virtual desktop on a server

To use SAPIEN on a GPU server with virtual display, additionally install xvfb, x11vnc, and any window manager such as fluxbox or xfce. Add display capabilities for NVIDIA docker.

ENV NVIDIA_DRIVER_CAPABILITIES graphics,utility,compute,display

Assuming fluxbox, start a VNC server by

x11vnc -create -env FD_PROG=/usr/bin/fluxbox  -env X11VNC_FINDDISPLAY_ALWAYS_FAILS=1 -env X11VNC_CREATE_GEOM=${99:-1920x1080x16} -gone 'pkill Xvfb' -nopw
# Note: you should only allow local access and tunnel through ssh

Now you can connect to the server at port 5900. SAPIEN should be fully functional, test with

python -m sapien.exapmle.hello_world

Hi fbxiang,
I tried your instruction to open a virtual desktop with GUI. But I see the same error in the thread. Is there any alternative way I can try? I only have remote server. I worry the headless mode won't be sufficient for simulation (I am new to this field, correct me if wrong).

[fbxiang]

Remote X is not officially supported, and getting VNC to work requires some luck. However, it can also indicate an incorrect or incomplete graphics driver installation. If you use Ubuntu, use some kind of RTX GPU (commercial gaming GPU), and have installed NVIDIA driver from apt, and have proper Docker setups, VNC should work.
If remote desktop does not work. You can at least test python -m sapien.example.offscreen to see if an image is generated. If it works fine, you have all features except that you cannot open an interactive window to play with the scene yourself. If you do not plan to model environments yourself but only to use existing benchmarks, you are probably still okay.
If everything above totally fails, you will need to debug your GPU driver. For example, you can try to get Vulkan official test programs such as vulkaninfo and vkcube to run first. If they cannot run, you can probably seek help in the Vulkan community.