torch.compile on windows is almost there

[WingeD123]

action1: I installed triton from https://github.com/jakaline-dev/Triton_win/releases on windows 10.
result1: comfyui fail to start because of no cuda_patch.
action2: install CUDA.
result2: comfyui start successful.
action3: tried to do torchcomile with cuda backend for flux.
result3: failed with error.
action4: restart comfyui with --disable-cuda-malloc and do action3 again.
result4: success with image generated, but cant see any speed increase.
action5: try generate another image to check speed.
result5: failed with error like: static pointer changed.
action6: restart comfyui and try torchcomile with inductor backend.
result6: got error with c compile fail (exit with code 1/2).
action7: install msvc build tool/mingw tool.
result7: same error as result6.
action8: try to modify code refer to who succeed in jakaline-dev/Triton_win#2.
result8: same error as above.

though I didn't succeed in make torch.compile work on my windows, I believe it's only an environment issue when using inductor backend since there are one/ones succeed like jakaline-dev/Triton_win#2.
for cuda backend, I have no idea what's wrong and whether it's possible to fix.

[woct0rdho]

It's already there if you can follow the instructions in my fork:
https://github.com/woct0rdho/triton

I just need to figure out how to automatically find some dependencies on other people's computers, then I'll build wheels and collect more feedback

[woct0rdho]

@comfyanonymous I've published the wheels for Triton, but they do not work with the embeded Python in ComfyUI, because Triton needs the include and libs folders in the usual installation of Python. Users have confirmed that just copy-pasting these two folders makes Triton work.

Would you include these two folders in the next release?

[elen07zz]

How can install the wheels ?
I have copied the mentioned folders but I don't know what to do next with the .whl file

[ltdrdata]

pip install <whl_file_name>

if you are using portable version.

python_embeded\python.exe -m pip install <whl_file_name>

[woct0rdho]

Also, maybe we need a bundled vcredist. The DLLs used by Triton are msvcp140.dll and vcruntime140.dll, and for completeness we can also include concrt140.dll, vcamp140.dll, vccorlib140.dll, vcomp140.dll

vcruntime140.dll is already bundled in the embeded Python, but a version mismatch with the msvcp140.dll installed in C:\Windows\System32\ can cause DLL import error

See https://www.reddit.com/r/StableDiffusion/comments/1g45n6n/triton_3_wheels_published_for_windows_and_working/ls7i5a0/

[elen07zz]

I'm getting tons of errors after trying to execute your code :https://www.reddit.com/r/StableDiffusion/comments/1g45n6n/comment/ls82cyk/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

[elen07zz]

I have cuda 12.4 in the environment

[woct0rdho]

Do you have an environment variable CUDA_PATH that is set to the path of CUDA 11? Set it to CUDA 12 instead

[woct0rdho]

Also, when using the embeded Python or a venv, I suggest to set the environment variable TRITON_CACHE_DIR to a folder specifically for that Python version. If there is a version mismatch between the Python that created the cache and the currently running Python, then Triton will fail

See triton-lang/triton#4809

[elen07zz]

After changing that now i'm getting this error instead:

Traceback (most recent call last):
  File "F:\ComfyUI_windows_portable\test.py", line 29, in <module>
    b_compiled = add(a, a)
                 ^^^^^^^^^
  File "F:\ComfyUI_windows_portable\test.py", line 21, in add
    add_kernel[grid](x, y, output, n_elements, BLOCK_SIZE=1024)
  File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\runtime\jit.py", line 345, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\runtime\jit.py", line 662, in run
    kernel = self.compile(
             ^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\compiler\compiler.py", line 244, in compile
    key = f"{triton_key()}-{src.hash()}-{backend.hash()}-{options.hash()}-{str(sorted(env_vars.items()))}"
                                         ^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\backends\nvidia\compiler.py", line 333, in hash
    version = get_ptxas_version()
              ^^^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\backends\nvidia\compiler.py", line 38, in get_ptxas_version
    version = subprocess.check_output([_path_to_binary("ptxas")[0], "--version"]).decode("utf-8")
                                       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\triton\backends\nvidia\compiler.py", line 33, in _path_to_binary
    raise RuntimeError(f"Cannot find {binary}")
RuntimeError: Cannot find ptxas.exe

[woct0rdho]

Maybe you did not install 'CUDA Development' when installing CUDA 12.4. Read the instructions to set the path for CUDA here:
https://github.com/woct0rdho/triton-windows#install-from-wheel

If you open a new PowerShell, type ptxas --version, and it shows your CUDA version like Cuda compilation tools, release 12.4 ..., then you're doing right

If there is no ptxas.exe in your C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4\bin\, then you need to reinstall CUDA 12.4, and choose both 'CUDA Development' and 'CUDA Runtime'

[elen07zz]

After reinstalling I think its fixed

Is this the correct result right?

[woct0rdho]

Yes, now you can try more things in ComfyUI

[woct0rdho]

Update: torch 2.5.0 just announced compiling to CPU on Windows, with prototype-level support

[frankyifei]

I found install vs-build-tools with PortableBuildTools very easy and saved me a lot of diskspace, It only required 1.2Gb disk space for MSVC and Windows Kit, all you need to do is to add those path into system environment variables.

[jepjoo]

I was surprised to see that torch.compile works with GGUF when using Mochi (with Kijai's node) since it doesn't work with Flux. Or am I just missing something and maybe there is a way to use torch.compile + GGUF with Flux/SD3.5 somehow after all?