Segmentation fault with CUDA execution provider

[prattcmp]

Describe the issue

The ONNX runtime produces a segmentation fault when used with the CUDA execution provider. GDB inspect returns:

Successfully registered `CUDAExecutionProvider`
Successfully registered `CPUExecutionProvider`
new_cpu; allocator=Arena memory_type=Default
new_cpu; allocator=Arena memory_type=Default
new_cpu; allocator=Arena memory_type=Default
new_cpu; allocator=Arena memory_type=Default
new_cpu; allocator=Arena memory_type=Default
new_cpu; allocator=Arena memory_type=Default
Segmentation fault (core dumped)

#0  0x00007fc64625c520 in onnxruntime::common::Status onnxruntime::contrib::cuda::QkvToContext<__half>(cudaDeviceProp const&, cublasContext*&, onnxruntime::Stream*, onnxruntime::contrib::AttentionParameters&, onnxruntime::contrib::cuda::AttentionData<__half>&) () from libonnxruntime_providers_cuda.so
#1  0x00007fc6461acc7b in onnxruntime::contrib::cuda::Attention<onnxruntime::MLFloat16>::ComputeInternal(onnxruntime::OpKernelContext*) const () from libonnxruntime_providers_cuda.so
#2  0x00007fc645c35b86 in onnxruntime::cuda::CudaKernel::Compute(onnxruntime::OpKernelContext*) const () from libonnxruntime_providers_cuda.so
#3  0x00007fc671c4dd0d in ?? () from /libonnxruntime.so
#4  0x00007fc671c42ea9 in ?? () from /libonnxruntime.so
#5  0x00007fc671c51a4d in ?? () from /libonnxruntime.so

To reproduce

1. Install ONNX runtime onto AWS EKS container.
2. Load BAAI/bge-base-en-v1.5 model.
3. Run inference using inference session.

Urgency

This is blocking us from deploying to GPU in production.

Platform

Linux

OS Version

Ubuntu 22.04

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.16.3

ONNX Runtime API

C++

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

CUDA 11.6

[xadupre]

Did you try on CPU to see whether it is related the kernel implementation in CUDA or maybe an error in the model?

[prattcmp]

Did you try on CPU to see whether it is related the kernel implementation in CUDA or maybe an error in the model?

Yes, it works fine on CPU.

[prattcmp]

This appears to be an issue with the K8s Device Plugin.

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.