Custom Op Library does not work for CUDA

[Positronx]

Describe the issue

I built onnxruntime v1.18 (cuda build) for Windows from source and among the generated dll files there is custom_op_library.dll. I wanted to test if the operator CustomOpOne for which there exists a CUDA defintion works when I use the CUDA execution provider. I made sure to call the RegisterCustomOpsLibrary to add my dll to the session_options. The error I obtain is that the return code is different than 0 (which is no the case when I use the CPU execution provider). I try to catch an exception but I failed.

To reproduce

This is the link to the onnx model I used https://mega.nz/file/kNsyWDKa#aEr8vsuh5olCpWn021w0eTsk5o1F3X0uhTs8QZXl4q8
The following is the code to reproduce the error

#include <onnxruntime_cxx_api.h>
#include <onnxruntime_lite_custom_op.h>
#include <iostream>

int main() {
    try {
        Ort::SessionOptions session_options;

        const wchar_t* custom_library_path = L"custom_op_library.dll";

        Ort::CustomOpConfigs cutom_op_configs;

        OrtCUDAProviderOptions cuda_options;
        cuda_options.device_id = 0;
        session_options.AppendExecutionProvider_CUDA(cuda_options);

        session_options.RegisterCustomOpsLibrary(custom_library_path, cutom_op_configs);

        Ort::Env env(ORT_LOGGING_LEVEL_WARNING, "test");

        const wchar_t* model_path = L"custom_op_one_test.onnx";

        Ort::Session session(env, model_path, session_options);

        std::vector<Ort::Value> input_tensors;

        std::vector<float> input_data_1(3 * 5);
        std::vector<float> input_data_2(3 * 5);

        std::vector<int64_t> input_shape = { 3, 5 };

        float idx = 0.5f;
        for (auto& v : input_data_1) {
            v = idx++;
        }
        idx = 0.f;
        for (auto& v : input_data_2)  {
            v = idx++;
        }

        Ort::MemoryInfo memory_info = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);
        Ort::Value input_1_tensor = Ort::Value::CreateTensor<float>(memory_info, input_data_1.data(), input_data_1.size(),
            input_shape.data(), input_shape.size());
        Ort::Value input_2_tensor = Ort::Value::CreateTensor<float>(memory_info, input_data_2.data(), input_data_2.size(),
            input_shape.data(), input_shape.size());

        input_tensors.push_back(std::move(input_1_tensor));
        input_tensors.push_back(std::move(input_2_tensor));

        std::vector<const char*> input_names = { "input_1", "input_2" };
        std::vector<const char*> output_names = { "output" };

        auto output_tensors = session.Run(Ort::RunOptions{ nullptr }, input_names.data(), input_tensors.data(), 2, output_names.data(), 1);

        float* int_array = output_tensors.front().GetTensorMutableData<float>();
        std::cout << "Inference results :" << std::endl << '\t';
        for (size_t i = 0; i < (3 * 5); ++i) {
            std::cout << int_array[i] << " ";
            if ((i + 1) % 5 == 0) std::cout << std::endl << '\t';
        }

        return 0;
    }
    catch (const std::exception& e) {
        std::cout << e.what() << std::endl;
        return -1;
    }
}

Urgency

This issue is stopping from developping custom operators that can be executed using CUDA execution provider

Platform

Windows

OS Version

10

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

1.18.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

12.3

[skottmckay]

Where does it fail?

Try using the fully qualified path to the custom ops library to rule out any issue with it finding the dll.

[Positronx]

Hi @skottmckay,

I tried the absolute path but the issue persists. I'm certain it has no problem finding the dll.
The code does what it's required i.e. the addition of the inputs. However, when I run the following command
echo %ERRORLEVEL% I obtain a negative value. I don't understand why the error level is different from zero.

When I comment the line

session_options.AppendExecutionProvider_CUDA(cuda_options);

the value of the ERRORLEVEL after the execution of the code above is 0, whereas it's different from zero when CUDA is added as a provider.

[skottmckay]

You need to have an environment for it to work. This should be created first and be the last thing destroyed. Most likely the code to add the CUDA EP is attempting to log a message, which requires a logger, which comes from the environment.

https://onnxruntime.ai/docs/api/c/struct_ort_1_1_env.html#details

Ort::Env env;

You can set the overall log level when creating the environment if needed to debug further issues.

[Positronx]

Hi @skottmckay,

You are correct. Creating the environnment as the first thing and destroying it as the last solved the issue. Thanks for your help, I should have paid attention to where I set up the ORT environment.