Why does GPU memory consumption increase after USE_CUDNN is selected? RTX3090

[netxue]

Ubuntu 20.04
Graphics card :Nvidia RTX3090 24G Driver Version: 460.39
CUDA: 11.1.1 install by deb

nvcc: NVIDIA (R) Cuda compiler driver

Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Oct_12_20:09:46_PDT_2020
Cuda compilation tools, release 11.1, V11.1.105
Build cuda_11.1.TC455_06.29190527_0

CUDNN:8.0.5 install by deb

OpenPose: 1.7.0
Caffe version: Default from OpenPose
CMake version:3.16.3
OpenCV version: pre-compiled apt-get install libopencv-dev 4.5.1

Select USE_CUDNN in cmake-gui, GPU memory consumed:
Body only: 9G
Body + face + hands: 19G
Maximum Accuracy cannot be used: /build/examples/openpose/openpose.bin --net_resolution "1312x736" --scale_number 4 --scale_gap 0.25
display:
F0130 21:28:00.355384 15412 cudnn_conv_layer.cpp:53] Check failed: status == CUDNN_STATUS_SUCCESS (1 vs. 0) CUDNN_STATUS_NOT_INITIALIZED

When I cancel USE_CUDNN in cmake-gui, GPU memory consumption:
Body only: 5G
Body + face + hands: 11G
Maximum Accuracy cannot be used: /build/examples/openpose/openpose.bin --net_resolution "1312x736" --scale_number 4 --scale_gap 0.25
display:
F0130 21:09:31.572585 7285 syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0) out of memory

Why does GPU memory consumption increase after USE_CUDNN is selected?
With a total of 24G of GPU memory, the maximum accuracy still cannot be achieved. I don’t know where I went wrong?

[netxue]

I tried cuda 10.1 and cudnn 7.5. It can run with USE_CUDNN selected on NVIDIA1080 11G graphics card. It may be that the GPU memory is small, and Maximum Accuracy" ./build/examples/openpose/openpose.bin --net_resolution "1312x736" --scale_number 4 --scale_gap 0.25" cannot run. But cuda10.1 and cudnn7.5 cannot run openpose on RTX3090.
I tried to install cuda11.2 and cudnn8.1.0 on the RTX3090 24G graphics card, cmake-gui USE_CUDNN selected,but it couldn't run. The error was CUDNN_STATUS_NOT_INITIALIZED. So, I canceled USE_CUDNN in cmake-gui and it can run.You can see that the GPU memory usage is about 11G, with --face --hand.

but Maximum Accuracy "./build/examples/openpose/openpose.bin --net_resolution "1312x736" --scale_number 4 --scale_gap 0.25" cannot run

Does anyone know what's going on?

[gineshidalgo99]

v1.7.0 was modified to allow cuDNN 8, which was a pain. I am not an expert, so I am sure there must be a better way to run the cuDNN convolutions using less memory, but I am not an expert on it. I am very open to suggestions about the cudnn_conv implementation to minimize memory:
https://github.com/CMU-Perceptual-Computing-Lab/caffe/blob/master/src/caffe/layers/cudnn_conv_layer.cpp

Please, continue this discursion in #1864, to centralize messages and hopefully focus efforts to fix the issue. Thanks!