Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Build] Cuda 12.5 #20765

Closed
johnnynunez opened this issue May 22, 2024 · 4 comments
Closed

[Build] Cuda 12.5 #20765

johnnynunez opened this issue May 22, 2024 · 4 comments
Assignees
Labels
build build issues; typically submitted using template ep:CUDA issues related to the CUDA execution provider quantization issues related to quantization

Comments

@johnnynunez
Copy link

johnnynunez commented May 22, 2024

Describe the issue

[ 80%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/home/johnny/Projects/onnxruntime/onnxruntime/contrib_ops/cuda/quantization/dequantize_blockwise_bnb4.cu.o
/home/johnny/Projects/onnxruntime/onnxruntime/contrib_ops/cuda/moe/ft_moe/moe_kernel.cu(68): error: identifier "FLT_MAX" is undefined
      float threadData(-FLT_MAX);
                        ^

1 error detected in the compilation of "/home/johnny/Projects/onnxruntime/onnxruntime/contrib_ops/cuda/moe/ft_moe/moe_kernel.cu".
gmake[2]: *** [CMakeFiles/onnxruntime_providers_cuda.dir/build.make:6984: CMakeFiles/onnxruntime_providers_cuda.dir/home/johnny/Projects/onnxruntime/onnxruntime/contrib_ops/cuda/moe/ft_moe/moe_kernel.cu.o] Error 2
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:2131: CMakeFiles/onnxruntime_providers_cuda.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2
Traceback (most recent call last):
  File "/home/johnny/Projects/onnxruntime/tools/ci_build/build.py", line 2950, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/johnny/Projects/onnxruntime/tools/ci_build/build.py", line 2842, in main
    build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
  File "/home/johnny/Projects/onnxruntime/tools/ci_build/build.py", line 1731, in build_targets
    run_subprocess(cmd_args, env=env)
  File "/home/johnny/Projects/onnxruntime/tools/ci_build/build.py", line 861, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/johnny/Projects/onnxruntime/tools/python/util/run.py", line 49, in run
    completed_process = subprocess.run(
                        ^^^^^^^^^^^^^^^
  File "/home/johnny/miniconda3/envs/py311/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/johnny/miniconda3/envs/py311/bin/cmake', '--build', '/home/johnny/Projects/onnxruntime/build/Linux/Release', '--config', 'Release', '--', '-j12']' returned non-zero exit status 2.

Urgency

No response

Target platform

arm64

Build script

git clone --recursive https://github.com/microsoft/onnxruntime
cd onnxruntime 

export PATH="/usr/local/cuda/bin:${PATH}"
export CUDACXX="/usr/local/cuda/bin/nvcc"

pip3 install -U packaging
./build.sh --config Release --update --parallel --build --build_wheel --build_shared_lib --skip_tests \
--use_tensorrt --cuda_home /usr/local/cuda --cudnn_home /usr/lib/aarch64-linux-gnu \
--tensorrt_home /usr/lib/aarch64-linux-gnu --cmake_extra_defines CMAKE_CXX_FLAGS="-Wno-unused-variable -I/usr/local/cuda/include" --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES="87"

Error / output

[ 80%] Building CUDA object CMakeFiles/onnxruntime_providers_cuda.dir/home/johnny/Projects/onnxruntime/onnxruntime/contrib_ops/cuda/quantization/dequantize_blockwise_bnb4.cu.o
/home/johnny/Projects/onnxruntime/onnxruntime/contrib_ops/cuda/moe/ft_moe/moe_kernel.cu(68): error: identifier "FLT_MAX" is undefined
      float threadData(-FLT_MAX);
                        ^

1 error detected in the compilation of "/home/johnny/Projects/onnxruntime/onnxruntime/contrib_ops/cuda/moe/ft_moe/moe_kernel.cu".
gmake[2]: *** [CMakeFiles/onnxruntime_providers_cuda.dir/build.make:6984: CMakeFiles/onnxruntime_providers_cuda.dir/home/johnny/Projects/onnxruntime/onnxruntime/contrib_ops/cuda/moe/ft_moe/moe_kernel.cu.o] Error 2
gmake[2]: *** Waiting for unfinished jobs....
gmake[1]: *** [CMakeFiles/Makefile2:2131: CMakeFiles/onnxruntime_providers_cuda.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2
Traceback (most recent call last):
  File "/home/johnny/Projects/onnxruntime/tools/ci_build/build.py", line 2950, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/johnny/Projects/onnxruntime/tools/ci_build/build.py", line 2842, in main
    build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
  File "/home/johnny/Projects/onnxruntime/tools/ci_build/build.py", line 1731, in build_targets
    run_subprocess(cmd_args, env=env)
  File "/home/johnny/Projects/onnxruntime/tools/ci_build/build.py", line 861, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/johnny/Projects/onnxruntime/tools/python/util/run.py", line 49, in run
    completed_process = subprocess.run(
                        ^^^^^^^^^^^^^^^
  File "/home/johnny/miniconda3/envs/py311/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/johnny/miniconda3/envs/py311/bin/cmake', '--build', '/home/johnny/Projects/onnxruntime/build/Linux/Release', '--config', 'Release', '--', '-j12']' returned non-zero exit status 2.

Visual Studio Version

No response

GCC / Compiler Version

3.29.3

@johnnynunez johnnynunez added the build build issues; typically submitted using template label May 22, 2024
@github-actions github-actions bot added ep:CUDA issues related to the CUDA execution provider quantization issues related to quantization labels May 22, 2024
@johnnynunez
Copy link
Author

johnnynunez commented May 22, 2024

maybe needs #include <cfloat> ??

@tianleiwu
Copy link
Contributor

maybe needs #include <cfloat> ??

I think so.

@johnnynunez
Copy link
Author

maybe needs #include <cfloat> ??

I think so.

it is working

tianleiwu pushed a commit that referenced this issue May 28, 2024
yf711 pushed a commit that referenced this issue Jun 21, 2024
@tianleiwu
Copy link
Contributor

The issue is fixed in 1.18.1 and main branches.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build build issues; typically submitted using template ep:CUDA issues related to the CUDA execution provider quantization issues related to quantization
Projects
None yet
Development

No branches or pull requests

3 participants