[Build] [Alpine linux] Microsoft.ML.OnnxRuntime.OnnxRuntimeException : [ErrorCode:RuntimeException] Exception caught: No error information

[bartosz-jarmuz]

Describe the issue

One of the teams in my company started using Microsoft.ML.OnnxRuntime nuget v 1.16 for some internal processings.
They produce a nuget which our .net core service consumes.

We have a CI pipeline in gitlab where we use alpine linux containers in the runner of Build and Test stage.
Unfortunately, the Test stage (dotnet test) fails with the following error:

Error Message:
System.TypeInitializationException : The type initializer for 'Microsoft.ML.OnnxRuntime.NativeMethods' threw an exception.
----> System.DllNotFoundException : Unable to load shared library 'onnxruntime' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library libonnxruntime: No such file or directory
Stack Trace:
at Microsoft.ML.OnnxRuntime.SessionOptions..ctor()
at Microsoft.ML.OnnxRuntime.InferenceSession..ctor(String modelPath)

These tests running in VS locally on a Windows machine work OK, its only the gitlab pipeline that fails.

I have then added this package to the docker container https://pkgs.alpinelinux.org/package/edge/testing/x86_64/onnxruntime, as well as required dependencies, so the Dockerfile has these instructions:

FROM docker-public.dev.docker.env.works/alpine:3.18.4 AS Downloads
########## some non-relevant stuff here
FROM docker-public.dev.docker.env.works/dotnet/sdk:6.0.414-alpine3.18
######### more non-relevant stuff here

######### dependencies of onnxruntime
RUN apk add libssl1.1
RUN apk add libstdc++
RUN apk add musl
RUN apk add libgomp
RUN apk add abseil-cpp-log-internal-check-op --repository https://dl-cdn.alpinelinux.org/alpine/edge/main/
RUN apk add abseil-cpp-log-internal-message --repository https://dl-cdn.alpinelinux.org/alpine/edge/main/
RUN apk add abseil-cpp-raw-hash-set --repository https://dl-cdn.alpinelinux.org/alpine/edge/main/
RUN apk add libprotobuf-lite --repository https://dl-cdn.alpinelinux.org/alpine/edge/main/
RUN apk add icu --repository https://dl-cdn.alpinelinux.org/alpine/edge/main/
RUN apk add icu-data-full --repository https://dl-cdn.alpinelinux.org/alpine/edge/main/
RUN apk add icu-doc --repository https://dl-cdn.alpinelinux.org/alpine/edge/main/
RUN apk add icu-dev --repository https://dl-cdn.alpinelinux.org/alpine/edge/main/
RUN apk add icu-static --repository https://dl-cdn.alpinelinux.org/alpine/edge/main/
RUN apk add re2=2023.09.01-r0 --repository https://dl-cdn.alpinelinux.org/alpine/edge/community/

RUN apk update
&& apk add onnxruntime --repository https://dl-cdn.alpinelinux.org/alpine/edge/testing/

ENV LD_DEBUG=all

Unfortunately, it doesn't change a thing.
I have added a post build target to copy the shared obj lib to the bin folder, but also no luck

Urgency

High, cannot roll out the component which uses Onnxruntime

Target platform

alpine linux

Build script

Described in the description

Error / output

Error Message:
System.TypeInitializationException : The type initializer for 'Microsoft.ML.OnnxRuntime.NativeMethods' threw an exception.
----> System.DllNotFoundException : Unable to load shared library 'onnxruntime' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Error loading shared library libonnxruntime: No such file or directory
Stack Trace:
at Microsoft.ML.OnnxRuntime.SessionOptions..ctor()
at Microsoft.ML.OnnxRuntime.InferenceSession..ctor(String modelPath)

Visual Studio Version

2022

GCC / Compiler Version

No response

[bartosz-jarmuz]

I have ran ldd command which pointed me to a missing dependency:

$ ldd /builds/Path/To/TestProject/bin/Release/net6/libonnxruntime.so
/lib/ld-musl-x86_64.so.1 (0x7d6b46c14000)
libdl.so.2 => /lib/ld-musl-x86_64.so.1 (0x7d6b46c14000)
librt.so.1 => /lib/ld-musl-x86_64.so.1 (0x7d6b46c14000)
libpthread.so.0 => /lib/ld-musl-x86_64.so.1 (0x7d6b46c14000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x7d6b4594b000)
libm.so.6 => /lib/ld-musl-x86_64.so.1 (0x7d6b46c14000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7d6b4592d000)
libc.so.6 => /lib/ld-musl-x86_64.so.1 (0x7d6b46c14000)
Error loading shared library ld-linux-x86-64.so.2: No such file or directory (needed by /builds/Path/To/TestProject/bin/Release/net6/libonnxruntime.so)
Error relocating /builds/Path/To/TestProject/bin/Release/net6/libonnxruntime.so: __snprintf_chk: symbol not found
Error relocating /builds/Path/To/TestProject/bin/Release/net6/libonnxruntime.so: __vsnprintf_chk: symbol not found
Error relocating /builds/Path/To/TestProject/bin/Release/net6/libonnxruntime.so: __memmove_chk: symbol not found
Error relocating /builds/Path/To/TestProject/bin/Release/net6/libonnxruntime.so: __memset_chk: symbol not found
Error relocating /builds/Path/To/TestProject/bin/Release/net6/libonnxruntime.so: __fprintf_chk: symbol not found
Error relocating /builds/Path/To/TestProject/bin/Release/net6/libonnxruntime.so: __memcpy_chk: symbol not found

After some googling, I have added gcompat package to the docker image and the error during the test execution is now different:

Error Message:
Microsoft.ML.OnnxRuntime.OnnxRuntimeException : [ErrorCode:RuntimeException] Exception caught: No error information
Stack Trace:
at Microsoft.ML.OnnxRuntime.OrtEnv.CreateDefaultEnv(OrtLoggingLevel logLevel, Byte[] logIdUtf8)
at Microsoft.ML.OnnxRuntime.OrtEnv.CreateInstance()
at System.Lazy`1.ViaFactory(LazyThreadSafetyMode mode)

[snnn]

The Linux distro is not supported by the prebuilt binaries. You may either build the binary by yourself on that Linux distro, or consider using Mariner. I highly recommend Mariner.

Sorry I will close this issue as it is not in our roadmap.

[bartosz-jarmuz]

ok, thanks - I understand it's getting closed as not planned, but what I don't understand is 'The linux distro is not supported by the prebuilt binaries'.

What are the prebuilt binaries, and then what is the point of this package
https://pkgs.alpinelinux.org/package/edge/testing/x86_64/onnxruntime
mentioned in issue #2909?

[snnn]

No. Packages at alpinelinux.org are not published/maintained by us. I was saying the packages at nuget.org/pypi.org and https://github.com/microsoft/onnxruntime/releases .