Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: CUDA: no CUDA-capable device is detected

[mojtabaAmir]

Description

I'm running nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04 container and this is nvidia-smi output in the container:

Tue Jul 11 07:30:55 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.125.06 Driver Version: 525.125.06 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:01:00.0 Off | N/A |
| 33% 31C P8 16W / 250W | 11MiB / 11264MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
After installing mxnet-cu101mkl==1.6.0.post0 using pip on python 3.7, I got this error for this sample code:

import mxnet as mx
mx.nd.array([1, 2, 3], ctx=mx.gpu(0))

Error Message

[07:35:23] src/base.cc:49: GPU context requested, but no GPUs found.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/mxnet/ndarray/utils.py", line 146, in array
    return _array(source_array, ctx=ctx, dtype=dtype)
  File "/usr/local/lib/python3.7/site-packages/mxnet/ndarray/ndarray.py", line 3284, in array
    arr = empty(source_array.shape, ctx, dtype)
  File "/usr/local/lib/python3.7/site-packages/mxnet/ndarray/ndarray.py", line 4735, in empty
    return NDArray(handle=_new_alloc_handle(shape, ctx, False, dtype))
  File "/usr/local/lib/python3.7/site-packages/mxnet/ndarray/ndarray.py", line 175, in _new_alloc_handle
    ctypes.byref(hdl)))
  File "/usr/local/lib/python3.7/site-packages/mxnet/base.py", line 255, in check_call
    raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [07:35:23] src/storage/storage.cc:100: Check failed: e == cudaSuccess || e == cudaErrorCudartUnloading: CUDA: no CUDA-capable device is detected
Stack trace:
  [bt] (0) /usr/local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x6d554b) [0x7f2c613c454b]
  [bt] (1) /usr/local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x41a14c4) [0x7f2c64e904c4]
  [bt] (2) /usr/local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x41a5042) [0x7f2c64e94042]
  [bt] (3) /usr/local/lib/python3.7/site-packages/mxnet/libmxnet.so(+0x41a6942) [0x7f2c64e95942]
  [bt] (4) /usr/local/lib/python3.7/site-packages/mxnet/libmxnet.so(mxnet::NDArray::NDArray(mxnet::TShape const&, mxnet::Context, bool, int)+0x4c6) [0x7f2c613f9d56]
  [bt] (5) /usr/local/lib/python3.7/site-packages/mxnet/libmxnet.so(void CreateNDArray<unsigned int, unsigned int>(unsigned int const*, unsigned int, int, int, int, int, void**)+0x2f3) [0x7f2c644f3373]
  [bt] (6) /usr/local/lib/python3.7/site-packages/mxnet/libmxnet.so(MXNDArrayCreateEx+0x4a) [0x7f2c644f382a]
  [bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f2cb8046dae]
  [bt] (8) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x22f) [0x7f2cb804671f]

To Reproduce

import mxnet as mx
mx.nd.array([1, 2, 3], ctx=mx.gpu(0))

Environment

----------Python Info----------
Version : 3.7.3
Compiler : GCC 7.5.0
Build : ('default', 'Jul 10 2023 09:10:16')
Arch : ('64bit', '')
------------Pip Info-----------
Version : 23.1.2
Directory : /usr/local/lib/python3.7/site-packages/pip
----------MXNet Info-----------
Version : 1.6.0
Directory : /usr/local/lib/python3.7/site-packages/mxnet
Num GPUs : 0
Commit Hash : 6eec9da
----------System Info----------
Platform : Linux-5.15.0-76-generic-x86_64-with-debian-buster-sid
system : Linux
node : a823acf4def6
release : 5.15.0-76-generic
version : #83~20.04.1-Ubuntu SMP Wed Jun 21 20:23:31 UTC 2023
----------Hardware Info----------
machine : x86_64
processor : x86_64
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 94
Model name: Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Stepping: 3
CPU MHz: 3400.000
CPU max MHz: 4000.0000
CPU min MHz: 800.0000
BogoMIPS: 6799.81
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 8192K
NUMA node0 CPU(s): 0-7
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities

[github-actions]

Welcome to Apache MXNet (incubating)! We are on a mission to democratize AI, and we are glad that you are contributing to it by opening this issue.
Please make sure to include all the relevant context, and one of the @apache/mxnet-committers will be here shortly.
If you are interested in contributing to our project, let us know! Also, be sure to check out our guide on contributing to MXNet and our development guides wiki.

[barzan-hayati]

...

Same error for mxnet in Docker contanier