Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extreme (as in, this is extremely frustrating) dependency conflict: CUDA 12.4, torch 2.5.1, onnxruntime-gpu==1.20.1 #495

Open
3 tasks done
mbac opened this issue Nov 25, 2024 · 2 comments
Open
3 tasks done

Extreme (as in, this is extremely frustrating) dependency conflict: CUDA 12.4, torch 2.5.1, onnxruntime-gpu==1.20.1 #495

mbac opened this issue Nov 25, 2024 · 2 comments
Labels
bug Something isn't working ⛔ dependencies conflict

Comments

@mbac
Copy link

mbac commented Nov 25, 2024

First, confirm

  • I have read the instruction carefully
  • I have searched the existing issues
  • I have updated the extension to the latest version

What happened?

Hi,

AFAIK the main issue with CUDA 12.x is older versions of onnxruntime-[gpu], so when I got one of those error messages, I went around searching and found that the latest stable version available for my system (Ubuntu Linux on AMD CPU, Nvidia A6000 GPU) is 1.20.1. According to Microsoft, it should work with any 12.x CUDA version that they know of since onnxruntime-gpu 1.19.0, and I'm on 1.20.1… and yet…

!!! Exception during processing !!! [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Conv node. Name:'Conv_0' Status Message: CUDNN failure 1002: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH ; GPU=0 ; hostname=4e3610f4-2466-4a18-bc2b-dd509ec0ebcb ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cudnn_common.cc ; line=31 ; expr=cudnnCreateTensorDescriptor(&tensor_);

… and so on, and so forth…

My status is:

- torch 2.5.1 and updated dependencies
- CUDA 12.4
- cuDNN libraries 9.2.1
- onnxruntime-gpu 1.20.1

DWPose nodes (can't recall the package name) for Comfy , which are usually super-picky about the onnxruntime-gpu version, report GPU acceleration is detected just fine.

Can anyone help please?

Steps to reproduce the problem

  • Start with functioning system featuring torch 2.5.1 CUDA 12.4 cuDNN 9.2.1
  • install comfyui-reactor-node package and run its install.py
  • Installation completes smoothly, with onnxruntime-gpu installed first, currently version 1.20.1 (installer sees correct versions of torch & CUDA).
  • Run any workflow with Fast ReActor Node
  • boom:
[ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Conv node.
Name:'Conv_0' 
Status Message:
CUDNN failure 1002: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH ; GPU=0 ;
hostname=4e3610f4-2466-4a18-bc2b-dd509ec0ebcb ;
file=/onnxruntime_src/onnxruntime/core/providers/cuda/cudnn_common.cc ; line=31 ;
expr=cudnnCreateTensorDescriptor(&tensor_);

[my line breaks for readability here]

Appreciate any help you guys can give!

Sysinfo

Distributor ID: Ubuntu
Description:    Ubuntu 22.04.5 LTS
Release:        22.04
Codename:       jammy

4x AMD EPYC 7443 vCPUs
Nvidia Ada A6000 GPU (48GB VRAM)
System RAM 32GB

Cloud instance hosted on TensorDock

Relevant console log

# ComfyUI Error Report
## Error Details
- **Node ID:** 203
- **Node Type:** ReActorFaceSwap
- **Exception Type:** onnxruntime.capi.onnxruntime_pybind11_state.Fail
- **Exception Message:** [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Conv node. Name:'Conv_0' Status Message: CUDNN failure 1002: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH ; GPU=0 ; hostname=4e3610f4-2466-4a18-bc2b-dd509ec0ebcb ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cudnn_common.cc ; line=31 ; expr=cudnnCreateTensorDescriptor(&tensor_); 
## Stack Trace

  File "/workspace/ComfyUI/execution.py", line 323, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)

  File "/workspace/ComfyUI/execution.py", line 198, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)

  File "/workspace/ComfyUI/execution.py", line 169, in _map_node_over_list
    process_inputs(input_dict, i)

  File "/workspace/ComfyUI/execution.py", line 158, in process_inputs
    results.append(getattr(obj, func)(**inputs))

  File "/workspace/ComfyUI/custom_nodes/comfyui-reactor-node/nodes.py", line 353, in execute
    script.process(

  File "/workspace/ComfyUI/custom_nodes/comfyui-reactor-node/scripts/reactor_faceswap.py", line 101, in process
    result = swap_face(

  File "/workspace/ComfyUI/custom_nodes/comfyui-reactor-node/scripts/reactor_swapper.py", line 275, in swap_face
    source_faces = analyze_faces(source_img)

  File "/workspace/ComfyUI/custom_nodes/comfyui-reactor-node/scripts/reactor_swapper.py", line 181, in analyze_faces
    faces = face_analyser.get(img_data)

  File "/home/user/.local/lib/python3.10/site-packages/insightface/app/face_analysis.py", line 59, in get
    bboxes, kpss = self.det_model.detect(img,

  File "/home/user/.local/lib/python3.10/site-packages/insightface/model_zoo/retinaface.py", line 224, in detect
    scores_list, bboxes_list, kpss_list = self.forward(det_img, self.det_thresh)

  File "/home/user/.local/lib/python3.10/site-packages/insightface/model_zoo/retinaface.py", line 152, in forward
    net_outs = self.session.run(self.output_names, {self.input_name : blob})

  File "/home/user/.local/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 266, in run
    return self._sess.run(output_names, input_feed, run_options)

System Information

  • ComfyUI Version: v0.3.4-1-g839ed33
  • Arguments: main.py
  • OS: posix
  • Python Version: 3.10.12 (main, Nov 6 2024, 20:22:13) [GCC 11.4.0]
  • Embedded Python: false
  • PyTorch Version: 2.5.1+cu124

Devices

  • Name: cuda:0 NVIDIA RTX 6000 Ada Generation : cudaMallocAsync
    • Type: cuda
    • VRAM Total: 47803596800
    • VRAM Free: 29130528706
    • Torch VRAM Total: 17246978048
    • Torch VRAM Free: 2922434

Logs

2024-11-25T22:10:39.787464 - [START] Security scan2024-11-25T22:10:39.787491 - 
2024-11-25T22:10:41.819883 - [DONE] Security scan2024-11-25T22:10:41.819916 - 
2024-11-25T22:10:41.894635 - ## ComfyUI-Manager: installing dependencies done.2024-11-25T22:10:41.894735 - 
2024-11-25T22:10:41.894804 - ** ComfyUI startup time:2024-11-25T22:10:41.894873 -  2024-11-25T22:10:41.894932 - 2024-11-25 22:10:41.8947602024-11-25T22:10:41.894984 - 
2024-11-25T22:10:41.895045 - ** Platform:2024-11-25T22:10:41.895155 -  2024-11-25T22:10:41.895236 - Linux2024-11-25T22:10:41.895316 - 
2024-11-25T22:10:41.895371 - ** Python version:2024-11-25T22:10:41.895423 -  2024-11-25T22:10:41.895473 - 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0]2024-11-25T22:10:41.895521 - 
2024-11-25T22:10:41.895738 - ** Python executable:2024-11-25T22:10:41.895825 -  2024-11-25T22:10:41.895887 - /usr/bin/python32024-11-25T22:10:41.895949 - 
2024-11-25T22:10:41.896014 - ** ComfyUI Path:2024-11-25T22:10:41.896080 -  2024-11-25T22:10:41.896136 - /workspace/ComfyUI2024-11-25T22:10:41.896189 - 
2024-11-25T22:10:41.896294 - ** Log path:2024-11-25T22:10:41.896373 -  2024-11-25T22:10:41.896446 - /workspace/ComfyUI/comfyui.log2024-11-25T22:10:41.896523 - 
2024-11-25T22:10:41.911985 - 
Prestartup times for custom nodes:2024-11-25T22:10:41.912043 - 
2024-11-25T22:10:41.912099 -    0.0 seconds:2024-11-25T22:10:41.912146 -  2024-11-25T22:10:41.912186 - /workspace/ComfyUI/custom_nodes/rgthree-comfy2024-11-25T22:10:41.912231 - 
2024-11-25T22:10:41.912270 -    0.0 seconds:2024-11-25T22:10:41.912304 -  2024-11-25T22:10:41.912338 - /workspace/ComfyUI/custom_nodes/ComfyUI-Easy-Use2024-11-25T22:10:41.912370 - 
2024-11-25T22:10:41.912416 -    2.1 seconds:2024-11-25T22:10:41.912455 -  2024-11-25T22:10:41.912487 - /workspace/ComfyUI/custom_nodes/ComfyUI-Manager2024-11-25T22:10:41.912526 - 
2024-11-25T22:10:41.912566 - 
2024-11-25T22:10:43.305875 - Total VRAM 45589 MB, total RAM 32093 MB
2024-11-25T22:10:43.306123 - pytorch version: 2.5.1+cu124
2024-11-25T22:10:44.658436 - xformers version: 0.0.28.post3
2024-11-25T22:10:44.658634 - Set vram state to: NORMAL_VRAM
2024-11-25T22:10:44.658761 - Device: cuda:0 NVIDIA RTX 6000 Ada Generation : cudaMallocAsync
2024-11-25T22:10:44.743864 - Using xformers cross attention
2024-11-25T22:10:45.383437 - [Prompt Server] web root: /workspace/ComfyUI/web
2024-11-25T22:10:45.386600 - Adding extra search path checkpoints /workspace/models/checkpoints
2024-11-25T22:10:45.386701 - Adding extra search path clip /workspace/models/clip
2024-11-25T22:10:45.386772 - Adding extra search path clip_vision /workspace/models/clip_vision
2024-11-25T22:10:45.386853 - Adding extra search path configs /workspace/models/configs
2024-11-25T22:10:45.386930 - Adding extra search path controlnet /workspace/models/controlnet
2024-11-25T22:10:45.386997 - Adding extra search path diffusion_models /workspace/models/diffusion_models
2024-11-25T22:10:45.387081 - Adding extra search path diffusion_models /workspace/models/unet
2024-11-25T22:10:45.387155 - Adding extra search path embeddings /workspace/models/embeddings
2024-11-25T22:10:45.387232 - Adding extra search path loras /workspace/models/loras
2024-11-25T22:10:45.387307 - Adding extra search path upscale_models /workspace/models/upscale_models
2024-11-25T22:10:45.387377 - Adding extra search path vae /workspace/models/vae
2024-11-25T22:10:45.387441 - Adding extra search path ultralytics /workspace/models/ultralytics
2024-11-25T22:10:45.387515 - Adding extra search path style_models /workspace/models/style_models
2024-11-25T22:10:45.387620 - Adding extra search path LLM /workspace/models/LLM
2024-11-25T22:10:45.387718 - Adding extra search path sams /workspace/models/sams
2024-11-25T22:10:45.387788 - Adding extra search path pulid /workspace/models/pulid
2024-11-25T22:10:45.711048 - ### Loading: ComfyUI-Impact-Pack (V7.11.5)2024-11-25T22:10:45.711112 - 
2024-11-25T22:10:46.176420 - ### Loading: ComfyUI-Impact-Pack (Subpack: V0.8)2024-11-25T22:10:46.176486 - 
2024-11-25T22:10:46.194872 - [Impact Pack] Wildcards loading done.2024-11-25T22:10:46.195014 - 
2024-11-25T22:10:46.204660 - 
2024-11-25T22:10:46.204731 - �[92m[rgthree-comfy] Loaded 42 epic nodes. 🎉�[00m2024-11-25T22:10:46.204764 - 
2024-11-25T22:10:46.204794 - 
2024-11-25T22:10:46.206193 - Initializing ControlAltAI Nodes2024-11-25T22:10:46.206242 - 
2024-11-25T22:10:46.452135 - ------------------------------------------2024-11-25T22:10:46.452205 - 
2024-11-25T22:10:46.452239 - �[34mComfyroll Studio v1.76 : �[92m 175 Nodes Loaded�[0m2024-11-25T22:10:46.452269 - 
2024-11-25T22:10:46.452298 - ------------------------------------------2024-11-25T22:10:46.452340 - 
2024-11-25T22:10:46.452373 - ** For changes, please see patch notes at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/blob/main/Patch_Notes.md2024-11-25T22:10:46.452401 - 
2024-11-25T22:10:46.452431 - ** For help, please see the wiki at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/wiki2024-11-25T22:10:46.452460 - 
2024-11-25T22:10:46.452488 - ------------------------------------------2024-11-25T22:10:46.452516 - 
2024-11-25T22:10:46.453679 - �[94mtheUpsiders Logic Nodes: �[92mLoaded�[0m2024-11-25T22:10:46.453729 - 
2024-11-25T22:10:47.384683 - �[34mWAS Node Suite: �[0mOpenCV Python FFMPEG support is enabled�[0m2024-11-25T22:10:47.384838 - 
2024-11-25T22:10:47.384989 - �[34mWAS Node Suite �[93mWarning: �[0m`ffmpeg_bin_path` is not set in `/workspace/ComfyUI/custom_nodes/was-node-suite-comfyui/was_suite_config.json` config file. Will attempt to use system ffmpeg binaries if available.�[0m2024-11-25T22:10:47.385092 - 
2024-11-25T22:10:47.978965 - �[34mWAS Node Suite: �[0mFinished.�[0m �[32mLoaded�[0m �[0m218�[0m �[32mnodes successfully.�[0m2024-11-25T22:10:47.979078 - 
2024-11-25T22:10:47.979176 - 
	�[3m�[93m"Art is not what you see, but what you make others see."�[0m�[3m - Edgar Degas�[0m
2024-11-25T22:10:47.979257 - 
2024-11-25T22:10:48.272213 - [Crystools �[0;32mINFO�[0m] Crystools version: 1.21.0
2024-11-25T22:10:48.286571 - [Crystools �[0;32mINFO�[0m] CPU: AMD EPYC 7443 24-Core Processor - Arch: x86_64 - OS: Linux 5.15.0-125-generic
2024-11-25T22:10:48.286749 - [Crystools �[0;32mINFO�[0m] Pynvml (Nvidia) initialized.
2024-11-25T22:10:48.286956 - [Crystools �[0;32mINFO�[0m] GPU/s:
2024-11-25T22:10:48.287112 - [Crystools �[0;32mINFO�[0m] 0) NVIDIA RTX 6000 Ada Generation
2024-11-25T22:10:48.287233 - [Crystools �[0;32mINFO�[0m] NVIDIA Driver: 550.127.05
2024-11-25T22:10:48.696495 - �[34m[ComfyUI-Easy-Use] server: �[0mv1.2.5 �[92mLoaded�[0m2024-11-25T22:10:48.696564 - 
2024-11-25T22:10:48.696603 - �[34m[ComfyUI-Easy-Use] web root: �[0m/workspace/ComfyUI/custom_nodes/ComfyUI-Easy-Use/web_version/v2 �[92mLoaded�[0m2024-11-25T22:10:48.696634 - 
2024-11-25T22:10:48.797535 - �[36;20m[comfyui_controlnet_aux] | INFO -> Using ckpts path: /workspace/ComfyUI/custom_nodes/comfyui_controlnet_aux/ckpts�[0m
2024-11-25T22:10:48.797734 - �[36;20m[comfyui_controlnet_aux] | INFO -> Using symlinks: False�[0m
2024-11-25T22:10:48.797905 - �[36;20m[comfyui_controlnet_aux] | INFO -> Using ort providers: ['CUDAExecutionProvider', 'DirectMLExecutionProvider', 'OpenVINOExecutionProvider', 'ROCMExecutionProvider', 'CPUExecutionProvider', 'CoreMLExecutionProvider']�[0m
2024-11-25T22:10:48.813891 - DWPose: Onnxruntime with acceleration providers detected2024-11-25T22:10:48.813945 - 
2024-11-25T22:10:48.824703 - Total VRAM 45589 MB, total RAM 32093 MB
2024-11-25T22:10:48.824815 - pytorch version: 2.5.1+cu124
2024-11-25T22:10:48.824907 - xformers version: 0.0.28.post3
2024-11-25T22:10:48.825042 - Set vram state to: NORMAL_VRAM
2024-11-25T22:10:48.825146 - Device: cuda:0 NVIDIA RTX 6000 Ada Generation : cudaMallocAsync
2024-11-25T22:10:50.074452 - Nvidia APEX normalization not installed, using PyTorch LayerNorm2024-11-25T22:10:50.074519 - 
2024-11-25T22:10:50.174492 - Adding2024-11-25T22:10:50.174558 -  2024-11-25T22:10:50.174589 - /workspace/ComfyUI/custom_nodes2024-11-25T22:10:50.174620 -  2024-11-25T22:10:50.174647 - to sys.path2024-11-25T22:10:50.174678 - 
2024-11-25T22:10:50.289785 - 
�[36mEfficiency Nodes:�[0m Attempting to add Control Net options to the 'HiRes-Fix Script' Node (comfyui_controlnet_aux add-on)...�[92mSuccess!�[0m2024-11-25T22:10:50.289848 - 
2024-11-25T22:10:50.290695 - Loaded Efficiency nodes from2024-11-25T22:10:50.290745 -  2024-11-25T22:10:50.290776 - /workspace/ComfyUI/custom_nodes/efficiency-nodes-comfyui2024-11-25T22:10:50.290805 - 
2024-11-25T22:10:50.292773 - Loaded ControlNetPreprocessors nodes from2024-11-25T22:10:50.292822 -  2024-11-25T22:10:50.292853 - /workspace/ComfyUI/custom_nodes/comfyui_controlnet_aux2024-11-25T22:10:50.292881 - 
2024-11-25T22:10:50.293139 - Could not find AdvancedControlNet nodes2024-11-25T22:10:50.293185 - 
2024-11-25T22:10:50.293572 - Could not find AnimateDiff nodes2024-11-25T22:10:50.293619 - 
2024-11-25T22:10:50.293879 - Could not find IPAdapter nodes2024-11-25T22:10:50.293923 - 
2024-11-25T22:10:50.297240 - Loaded VideoHelperSuite from2024-11-25T22:10:50.297291 -  2024-11-25T22:10:50.297322 - /workspace/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite2024-11-25T22:10:50.297351 - 
2024-11-25T22:10:50.297979 - ### Loading: ComfyUI-Impact-Pack (V7.11.5)2024-11-25T22:10:50.298026 - 
2024-11-25T22:10:50.298889 - Loaded ImpactPack nodes from2024-11-25T22:10:50.298937 -  2024-11-25T22:10:50.298989 - /workspace/ComfyUI/custom_nodes/ComfyUI-Impact-Pack2024-11-25T22:10:50.299144 - 
2024-11-25T22:10:50.313613 - [Impact Pack] Wildcards loading done.2024-11-25T22:10:50.313782 - 
2024-11-25T22:10:50.317931 - ### Loading: ComfyUI-Manager (V2.51.9)2024-11-25T22:10:50.318057 - 
2024-11-25T22:10:50.384412 - ### ComfyUI Revision: 2855 [839ed336] | Released on '2024-11-22'2024-11-25T22:10:50.384500 - 
2024-11-25T22:10:50.455861 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json2024-11-25T22:10:50.456013 - 
2024-11-25T22:10:50.527091 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json2024-11-25T22:10:50.527305 - 
2024-11-25T22:10:50.601893 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json2024-11-25T22:10:50.601982 - 
2024-11-25T22:10:50.639077 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json2024-11-25T22:10:50.639847 - 
2024-11-25T22:10:50.693657 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json2024-11-25T22:10:50.693713 - 
2024-11-25T22:10:51.270254 - XTNodes: Successfully imported /workspace/ComfyUI/custom_nodes/ComfyUI-EasyCivitai-XTNodes/civitaiNodes/civitai_url_nodes.py2024-11-25T22:10:51.270313 - 
2024-11-25T22:10:51.270585 - XTNodes: Successfully imported /workspace/ComfyUI/custom_nodes/ComfyUI-EasyCivitai-XTNodes/civitaiNodes/local_loader_nodes.py2024-11-25T22:10:51.270632 - 
2024-11-25T22:10:51.341494 - XTNodes: Successfully imported /workspace/ComfyUI/custom_nodes/ComfyUI-EasyCivitai-XTNodes/civitaiNodes/prompt_concatenate.py2024-11-25T22:10:51.341549 - 
2024-11-25T22:10:51.352263 - �[0;33m[ReActor]�[0m - �[38;5;173mSTATUS�[0m - �[0;32mRunning v0.5.2-a1 in ComfyUI�[0m2024-11-25T22:10:51.352313 - 
2024-11-25T22:10:51.478527 - Torch version: 2.5.1+cu1242024-11-25T22:10:51.478579 - 
2024-11-25T22:10:51.494848 - Plush - Running on python installation: /usr/bin/python3, ver: 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0]2024-11-25T22:10:51.494898 - 
2024-11-25T22:10:51.494931 - Plush - Current Openai Version: 2024-11-25T22:10:51.494972 -  2024-11-25T22:10:51.495010 - 1.55.02024-11-25T22:10:51.495038 - 
2024-11-25T22:10:51.506819 - Plush - Version:2024-11-25T22:10:51.506865 -  2024-11-25T22:10:51.506903 - 1.21.192024-11-25T22:10:51.506932 - 
2024-11-25T22:10:51.630168 - 
Import times for custom nodes:
2024-11-25T22:10:51.630301 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Crystools-save
2024-11-25T22:10:51.630385 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/websocket_image_save.py
2024-11-25T22:10:51.630460 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/image-resize-comfyui
2024-11-25T22:10:51.630524 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/efficiency-nodes-comfyui
2024-11-25T22:10:51.630586 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-mxToolkit
2024-11-25T22:10:51.630657 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Detail-Daemon
2024-11-25T22:10:51.630717 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Inpaint-CropAndStitch
2024-11-25T22:10:51.630787 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Logic
2024-11-25T22:10:51.630855 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/comfy-image-saver
2024-11-25T22:10:51.630928 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/sd-dynamic-thresholding
2024-11-25T22:10:51.630989 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Image-Saver
2024-11-25T22:10:51.631050 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/lora-info
2024-11-25T22:10:51.631118 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/civitai_comfy_nodes
2024-11-25T22:10:51.631186 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/florence_dw
2024-11-25T22:10:51.631258 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/cg-use-everywhere
2024-11-25T22:10:51.631326 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/comfy-easy-grids
2024-11-25T22:10:51.631393 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_nodes
2024-11-25T22:10:51.631454 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ControlAltAI-Nodes
2024-11-25T22:10:51.631515 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/wlsh_nodes
2024-11-25T22:10:51.631633 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/comfyui-inpaint-nodes
2024-11-25T22:10:51.631753 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Florence-2
2024-11-25T22:10:51.631826 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Custom-Scripts
2024-11-25T22:10:51.631902 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Florence2
2024-11-25T22:10:51.631962 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-LogicUtils
2024-11-25T22:10:51.632030 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/Derfuu_ComfyUI_ModdedNodes
2024-11-25T22:10:51.632104 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/rgthree-comfy
2024-11-25T22:10:51.632166 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-KJNodes
2024-11-25T22:10:51.632234 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_ExtraModels
2024-11-25T22:10:51.632295 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_Comfyroll_CustomNodes
2024-11-25T22:10:51.632365 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_essentials
2024-11-25T22:10:51.632432 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/facerestore_cf
2024-11-25T22:10:51.632499 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-GGUF
2024-11-25T22:10:51.632560 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI_UltimateSDUpscale
2024-11-25T22:10:51.632629 -    0.0 seconds: /workspace/ComfyUI/custom_nodes/comfyui_controlnet_aux
2024-11-25T22:10:51.632689 -    0.1 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite
2024-11-25T22:10:51.632757 -    0.1 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager
2024-11-25T22:10:51.632830 -    0.1 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Easy-Use
2024-11-25T22:10:51.632899 -    0.1 seconds: /workspace/ComfyUI/custom_nodes/Plush-for-ComfyUI
2024-11-25T22:10:51.632973 -    0.1 seconds: /workspace/ComfyUI/custom_nodes/comfyui-art-venture
2024-11-25T22:10:51.633034 -    0.2 seconds: /workspace/ComfyUI/custom_nodes/comfyui-reactor-node
2024-11-25T22:10:51.633094 -    0.2 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Tara-LLM-Integration
2024-11-25T22:10:51.633161 -    0.2 seconds: /workspace/ComfyUI/custom_nodes/comfyui-tensorops
2024-11-25T22:10:51.633228 -    0.3 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-eesahesNodes
2024-11-25T22:10:51.633287 -    0.3 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Crystools
2024-11-25T22:10:51.633359 -    0.5 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-Impact-Pack
2024-11-25T22:10:51.633420 -    0.5 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-PuLID-Flux-Enhanced
2024-11-25T22:10:51.633492 -    0.6 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-SUPIR
2024-11-25T22:10:51.633552 -    0.9 seconds: /workspace/ComfyUI/custom_nodes/ComfyUI-EasyCivitai-XTNodes
2024-11-25T22:10:51.633624 -    1.5 seconds: /workspace/ComfyUI/custom_nodes/was-node-suite-comfyui
2024-11-25T22:10:51.633684 - 
2024-11-25T22:10:51.650206 - Starting server

2024-11-25T22:10:51.650528 - To see the GUI go to: http://127.0.0.1:8188
2024-11-25T22:10:59.872193 - FETCH DATA from: /workspace/ComfyUI/custom_nodes/ComfyUI-Manager/extension-node-map.json2024-11-25T22:10:59.872473 - 2024-11-25T22:10:59.882015 -  [DONE]2024-11-25T22:10:59.882126 - 
2024-11-25T22:11:02.039417 - HTTP Request: GET https://api.openai.com/v1/models "HTTP/1.1 200 OK"
2024-11-25T22:11:02.383980 - HTTP Request: GET https://api.groq.com/openai/v1/models "HTTP/1.1 200 OK"
2024-11-25T23:05:52.318912 - got prompt
2024-11-25T23:05:52.361862 - Using xformers attention in VAE
2024-11-25T23:05:52.364258 - Using xformers attention in VAE
2024-11-25T23:05:52.738934 - Requested to load FluxClipModel_
2024-11-25T23:05:52.739142 - Loading 1 new model
2024-11-25T23:05:52.754209 - loaded completely 0.0 4903.231597900391 True
2024-11-25T23:05:53.960277 - model weight dtype torch.float8_e4m3fn, manual cast: torch.bfloat16
2024-11-25T23:05:53.961061 - model_type FLUX
2024-11-25T23:05:58.995988 - �[33m[rgthree-comfy][Power Lora Loader]�[00m Lora "flux\flux_realism_lora.safetensors" not found, skipping.�[00m2024-11-25T23:05:58.996061 - 
2024-11-25T23:05:58.996248 - �[33m[rgthree-comfy][Power Lora Loader]�[00m Lora "flux\aidmaImageUpgrader-FLUX-V0.1.safetensors" not found, skipping.�[00m2024-11-25T23:05:58.996291 - 
2024-11-25T23:05:59.623950 - Requested to load Flux
2024-11-25T23:05:59.624321 - Loading 1 new model
2024-11-25T23:06:02.015175 - loaded completely 0.0 11350.069839477539 True
2024-11-25T23:06:16.243498 - 
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:14<00:00,  2.08it/s]2024-11-25T23:06:16.244232 - 
100%|████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:14<00:00,  2.11it/s]2024-11-25T23:06:16.244306 - 
2024-11-25T23:06:16.392214 - Requested to load AutoencodingEngine
2024-11-25T23:06:16.392327 - Loading 1 new model
2024-11-25T23:06:16.426233 - loaded completely 0.0 159.87335777282715 True
2024-11-25T23:06:17.182107 - [ReActor] 23:06:17 - �[38;5;173mSTATUS�[0m - Working: source face index [0], target face index [0]
2024-11-25T23:06:17.187283 - [ReActor] 23:06:17 - �[38;5;173mSTATUS�[0m - Analyzing Source Image...
2024-11-25T23:06:19.222319 - !!! Exception during processing !!! [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Conv node. Name:'Conv_0' Status Message: CUDNN failure 1002: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH ; GPU=0 ; hostname=4e3610f4-2466-4a18-bc2b-dd509ec0ebcb ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cudnn_common.cc ; line=31 ; expr=cudnnCreateTensorDescriptor(&tensor_); 
2024-11-25T23:06:19.225183 - Traceback (most recent call last):
  File "/workspace/ComfyUI/execution.py", line 323, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
  File "/workspace/ComfyUI/execution.py", line 198, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)
  File "/workspace/ComfyUI/execution.py", line 169, in _map_node_over_list
    process_inputs(input_dict, i)
  File "/workspace/ComfyUI/execution.py", line 158, in process_inputs
    results.append(getattr(obj, func)(**inputs))
  File "/workspace/ComfyUI/custom_nodes/comfyui-reactor-node/nodes.py", line 353, in execute
    script.process(
  File "/workspace/ComfyUI/custom_nodes/comfyui-reactor-node/scripts/reactor_faceswap.py", line 101, in process
    result = swap_face(
  File "/workspace/ComfyUI/custom_nodes/comfyui-reactor-node/scripts/reactor_swapper.py", line 275, in swap_face
    source_faces = analyze_faces(source_img)
  File "/workspace/ComfyUI/custom_nodes/comfyui-reactor-node/scripts/reactor_swapper.py", line 181, in analyze_faces
    faces = face_analyser.get(img_data)
  File "/home/user/.local/lib/python3.10/site-packages/insightface/app/face_analysis.py", line 59, in get
    bboxes, kpss = self.det_model.detect(img,
  File "/home/user/.local/lib/python3.10/site-packages/insightface/model_zoo/retinaface.py", line 224, in detect
    scores_list, bboxes_list, kpss_list = self.forward(det_img, self.det_thresh)
  File "/home/user/.local/lib/python3.10/site-packages/insightface/model_zoo/retinaface.py", line 152, in forward
    net_outs = self.session.run(self.output_names, {self.input_name : blob})
  File "/home/user/.local/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 266, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Conv node. Name:'Conv_0' Status Message: CUDNN failure 1002: CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH ; GPU=0 ; hostname=4e3610f4-2466-4a18-bc2b-dd509ec0ebcb ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cudnn_common.cc ; line=31 ; expr=cudnnCreateTensorDescriptor(&tensor_); 

2024-11-25T23:06:19.225763 - Prompt executed in 26.90 seconds

Attached Workflow

Please make sure that workflow does not contain any sensitive information such as API keys or passwords.

Workflow too large. Please manually upload the workflow from local file system.

Additional Context

(Please add any additional context or steps to reproduce the error here)



### Additional information

_No response_
@mbac mbac added bug Something isn't working new labels Nov 25, 2024
@Gourieff
Copy link
Owner

Try to downgrade Cuda to 12.1
I use torch 2.5.1+cu121 alongside ort-gpu 1.20.1 and everything works pretty fine with no errors

@mbac
Copy link
Author

mbac commented Dec 2, 2024

Thanks for the info, but downgrading is really a problematic option, both for the risk of even more conflicts with other libraries and the fact that the cloud provider I'm invested only has 12.4 images, which means I'd have to run docker on top of the installation.

After running several updates, I got to this point (notice slightly different error message)… Can you make anything else out of it?

# ComfyUI Error Report
## Error Details
- **Node ID:** 3585
- **Node Type:** ReActorFaceSwap
- **Exception Type:** onnxruntime.capi.onnxruntime_pybind11_state.Fail
- **Exception Message:** [ONNXRuntimeError] : 1 : FAIL : Non-zero status code returned while running Conv node. Name:'Conv_0' Status Message: CUDNN_FE failure 7: GRAPH_EXECUTION_FAILED ; GPU=0 ; hostname=eedcce05-9d6c-4f8f-8246-9d7b38a3f200 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/nn/conv.cc ; line=483 ; expr=s_.cudnn_fe_graph->execute(cudnn_handle, s_.variant_pack, ws.get()); 
## Stack Trace

  File "/workspace/ComfyUI/execution.py", line 323, in execute
    output_data, output_ui, has_subgraph = get_output_data(obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)

  File "/workspace/ComfyUI/execution.py", line 198, in get_output_data
    return_values = _map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb)

  File "/workspace/ComfyUI/execution.py", line 169, in _map_node_over_list
    process_inputs(input_dict, i)

  File "/workspace/ComfyUI/execution.py", line 158, in process_inputs
    results.append(getattr(obj, func)(**inputs))

  File "/workspace/ComfyUI/custom_nodes/comfyui-reactor-node/nodes.py", line 353, in execute
    script.process(

  File "/workspace/ComfyUI/custom_nodes/comfyui-reactor-node/scripts/reactor_faceswap.py", line 101, in process
    result = swap_face(

  File "/workspace/ComfyUI/custom_nodes/comfyui-reactor-node/scripts/reactor_swapper.py", line 275, in swap_face
    source_faces = analyze_faces(source_img)

  File "/workspace/ComfyUI/custom_nodes/comfyui-reactor-node/scripts/reactor_swapper.py", line 181, in analyze_faces
    faces = face_analyser.get(img_data)

  File "/workspace/ComfyUI/vcomfy/lib/python3.10/site-packages/insightface/app/face_analysis.py", line 59, in get
    bboxes, kpss = self.det_model.detect(img,

  File "/workspace/ComfyUI/vcomfy/lib/python3.10/site-packages/insightface/model_zoo/retinaface.py", line 224, in detect
    scores_list, bboxes_list, kpss_list = self.forward(det_img, self.det_thresh)

  File "/workspace/ComfyUI/vcomfy/lib/python3.10/site-packages/insightface/model_zoo/retinaface.py", line 152, in forward
    net_outs = self.session.run(self.output_names, {self.input_name : blob})

  File "/workspace/ComfyUI/vcomfy/lib/python3.10/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 266, in run
    return self._sess.run(output_names, input_feed, run_options)

## System Information
- **ComfyUI Version:** v0.3.5-11-g20a560eb
- **Arguments:** /workspace/ComfyUI/main.py --output-directory /workspace/output/comfyui
- **OS:** posix
- **Python Version:** 3.10.12 (main, Nov  6 2024, 20:22:13) [GCC 11.4.0]
- **Embedded Python:** false
- **PyTorch Version:** 2.5.1+cu124
## Devices

- **Name:** cuda:0 NVIDIA GeForce RTX 4090 : cudaMallocAsync
  - **Type:** cuda
  - **VRAM Total:** 25386352640
  - **VRAM Free:** 4413828794
  - **Torch VRAM Total:** 19662897152
  - **Torch VRAM Free:** 73051834

Thanks!

@LukeG89 LukeG89 mentioned this issue Dec 18, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working ⛔ dependencies conflict
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mbac @Gourieff