2023-09-22 12:39:57.324946075 [E:onnxruntime:, sequential_executor.cc:514 ExecuteKernel] Non-zero status code returned while running Slice node. Name:'Slice_37' Status Message: CUDA error cudaErrorNoKernelImageForDevice:no kernel image is available for execution on the device

[mahesh11T]

Describe the issue

i have no error locally in both docker way and conda env, but while running in docker image aws sagemaker ml p2 instance, i get this error.

To reproduce

docker
python:3.8.16 base image for docker
debian buster os
latest onnruntime-gpu
cuda 11.7 cudnn 8.5
ml.p2.xlarge instance
in aws sagemaker

Urgency

urgent

Platform

Linux

OS Version

debian buster 11

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

11.6

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

11.7

[mahesh11T]

compute capability was low on ml.p2 , ml.p3 instance solved it.