Build failed due to gcc

[yulong-chen551]

${PYTHON_BIN_PATH} configure.py --backend CUDA \
                                --os LINUX \
                                --host_compiler GCC \
                                --cuda_compiler NVCC \
                                --cuda_compute_capabilities ${TF_CUDA_COMPUTE_CAPABILITIES} \
                                --python_bin_path ${PYTHON_BIN_PATH} \
                                --local_cuda_path ${CUDA_INSTALL_PATH} \
                                --local_cudnn_path ${CUDNN_INSTALL_PATH} \
                                --gcc_path ${GCC_HOST_COMPILER_PATH}

bazel build //xla/tools:run_hlo_module 

After configured, I tried to build the run_hlo_module. But I got an error like this:

gcc: error: unrecognized command-line option '--cuda-path=external/cuda_nvcc'

The entire log is here:

INFO:root:Writing bazelrc to /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/xla_configure.bazelrc...
INFO: Reading 'startup' options from /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'build' from /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/xla_configure.bazelrc:
  'build' options: --action_env GCC_HOST_COMPILER_PATH=/data/common/gcc-10.2.0/bin/gcc --config cuda --config nonccl --action_env LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:/usr/local/cudnn-8.6/lib:/data/common/gcc-10.2.0//lib64:/data/common/gcc-10.2.0/lib64:/home/hadoop/software/java/lib/server/:/usr/local/lib64/:/home/mpi/install/lib:/home/hadoop/software/java/jre/lib/amd64/server:/data/common/gcc-10.2.0/lib64:/home/hadoop/software/java/lib/server/:/usr/local/lib64/:/home/mpi/install/lib:/home/hadoop/software/java/jre/lib/amd64/server:/data/common/gcc-10.2.0/lib64:/home/hadoop/software/java/lib/server/:/usr/local/lib64/:/home/mpi/install/lib:/home/hadoop/software/java/jre/lib/amd64/server:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 --action_env PYTHON_BIN_PATH=/root/anaconda3/envs/openxla/bin/python3.9 --python_path /root/anaconda3/envs/openxla/bin/python3.9 --copt -Wno-sign-compare --build_tag_filters -no_oss --test_tag_filters -no_oss
INFO: Found applicable config definition build:short_logs in file /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:cuda in file /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda --repo_env=HERMETIC_CUDA_VERSION=12.3.2 --repo_env=HERMETIC_CUDNN_VERSION=9.3.0 --@local_config_cuda//cuda:include_cuda_libs=true
INFO: Found applicable config definition build:cuda in file /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/xla_configure.bazelrc: --repo_env HERMETIC_CUDA_COMPUTE_CAPABILITIES=8.0,8.6,9.0 --repo_env LOCAL_CUDA_PATH=/usr/local/cuda-11.8/ --repo_env LOCAL_CUDNN_PATH=/usr/local/cudnn-8.6/,/usr/include/,/usr/lib64/
INFO: Found applicable config definition build:nonccl in file /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/.bazelrc: --define=no_nccl_support=true
INFO: Found applicable config definition build:linux in file /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
Loading: 
DEBUG: /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'llvm-raw' because it already exists.
DEBUG: /root/.cache/bazel/_bazel_root/5806b38faff86167fbbc6a1cf9564e98/external/tsl/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'nvtx_archive' because it already exists.
DEBUG: /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/third_party/repo.bzl:132:14: 
Warning: skipping import of repository 'jsoncpp_git' because it already exists.
Loading: 
Loading: 0 packages loaded
Analyzing: target //xla/tools:run_hlo_module (0 packages loaded, 0 targets configured)
INFO: Analyzed target //xla/tools:run_hlo_module (27 packages loaded, 8023 targets configured).
 checking cached actions
INFO: Found 1 target...
[0 / 6] [Prepa] BazelWorkspaceStatusAction stable-status.txt
ERROR: /root/.cache/bazel/_bazel_root/5806b38faff86167fbbc6a1cf9564e98/external/llvm-project/mlir/BUILD.bazel:9788:11: Compiling mlir/lib/ExecutionEngine/SparseTensor/Storage.cpp failed: (Exit 1): gcc failed: error executing command (from target @llvm-project//mlir:SparseTensorRuntime) 
  (cd /root/.cache/bazel/_bazel_root/5806b38faff86167fbbc6a1cf9564e98/execroot/xla && \
  exec env - \
    GCC_HOST_COMPILER_PATH=/data/common/gcc-10.2.0/bin/gcc \
    LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:/usr/local/cudnn-8.6/lib:/data/common/gcc-10.2.0//lib64:/data/common/gcc-10.2.0/lib64:/home/hadoop/software/java/lib/server/:/usr/local/lib64/:/home/mpi/install/lib:/home/hadoop/software/java/jre/lib/amd64/server:/data/common/gcc-10.2.0/lib64:/home/hadoop/software/java/lib/server/:/usr/local/lib64/:/home/mpi/install/lib:/home/hadoop/software/java/jre/lib/amd64/server:/data/common/gcc-10.2.0/lib64:/home/hadoop/software/java/lib/server/:/usr/local/lib64/:/home/mpi/install/lib:/home/hadoop/software/java/jre/lib/amd64/server:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 \
    PATH=/root/.cache/bazelisk/downloads/sha256/a40ac69263440761199fcb8da47ad4e3f328cbe79ffbf4ecc14e5ba252857307/bin:.:/root/anaconda3/envs/openxla/bin:/root/anaconda3/condabin:/root/.local/bin:/home/mpi/install/bin:/home/hadoop/software/java/bin:/home/hadoop/software/hadoop/bin:/root/anaconda3/bin:/media/disk1/framework-sim/chenyulong05/soft/nsight-compute:/data/common/gcc-10.2.0/bin:/root/.local/bin:/home/mpi/install/bin:/home/hadoop/software/java/bin:/home/hadoop/software/hadoop/bin:/root/anaconda3/bin:/root/.local/bin:/home/mpi/install/bin:/home/hadoop/software/java/bin:/home/hadoop/software/hadoop/bin:/root/anaconda3/bin:/root/.local/bin:/home/hadoop/software/java/bin:/home/hadoop/software/hadoop/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/bin:/media/disk1/framework-sim/chenyulong05/soft/cmake-3.30.1-linux-x86_64/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/root/anaconda3/envs/openxla/bin/python3.9 \
    TF2_BEHAVIOR=1 \
  /data/common/gcc-10.2.0/bin/gcc -MD -MF bazel-out/k8-opt/bin/external/llvm-project/mlir/_objs/SparseTensorRuntime/Storage.d '-frandom-seed=bazel-out/k8-opt/bin/external/llvm-project/mlir/_objs/SparseTensorRuntime/Storage.o' '-DBAZEL_CURRENT_REPOSITORY="llvm-project"' -iquote external/llvm-project -iquote bazel-out/k8-opt/bin/external/llvm-project -isystem external/llvm-project/mlir/include -isystem bazel-out/k8-opt/bin/external/llvm-project/mlir/include -fmerge-all-constants -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIE -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wno-invalid-partial-specialization -fno-omit-frame-pointer -no-canonical-prefixes -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections '--cuda-path=external/cuda_nvcc' -Wno-all -Wno-extra -Wno-deprecated -Wno-deprecated-declarations -Wno-ignored-attributes -Wno-array-bounds -Wunused-result '-Werror=unused-result' -Wswitch '-Werror=switch' '-Wno-error=unused-but-set-variable' -DAUTOLOAD_DYNAMIC_KERNELS -Wno-sign-compare '-std=c++17' -c external/llvm-project/mlir/lib/ExecutionEngine/SparseTensor/Storage.cpp -o bazel-out/k8-opt/bin/external/llvm-project/mlir/_objs/SparseTensorRuntime/Storage.o)
# Configuration: 660f29ed51dcc38c3b5336d29fc0fbe0348e3f0d5b0fb0ecb17aaeca8c1f8100
# Execution platform: @local_execution_config_platform//:platform
gcc: error: unrecognized command-line option '--cuda-path=external/cuda_nvcc'
ERROR: /media/disk1/framework-sim/chenyulong05/repo_except_kc/openxla/xla/xla/service/cpu/BUILD:524:11: Compiling xla/service/cpu/runtime_fp16.cc failed: (Exit 1): gcc failed: error executing command (from target //xla/service/cpu:runtime_fp16) 
  (cd /root/.cache/bazel/_bazel_root/5806b38faff86167fbbc6a1cf9564e98/execroot/xla && \
  exec env - \
    GCC_HOST_COMPILER_PATH=/data/common/gcc-10.2.0/bin/gcc \
    LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib64:/usr/local/cudnn-8.6/lib:/data/common/gcc-10.2.0//lib64:/data/common/gcc-10.2.0/lib64:/home/hadoop/software/java/lib/server/:/usr/local/lib64/:/home/mpi/install/lib:/home/hadoop/software/java/jre/lib/amd64/server:/data/common/gcc-10.2.0/lib64:/home/hadoop/software/java/lib/server/:/usr/local/lib64/:/home/mpi/install/lib:/home/hadoop/software/java/jre/lib/amd64/server:/data/common/gcc-10.2.0/lib64:/home/hadoop/software/java/lib/server/:/usr/local/lib64/:/home/mpi/install/lib:/home/hadoop/software/java/jre/lib/amd64/server:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 \
    PATH=/root/.cache/bazelisk/downloads/sha256/a40ac69263440761199fcb8da47ad4e3f328cbe79ffbf4ecc14e5ba252857307/bin:.:/root/anaconda3/envs/openxla/bin:/root/anaconda3/condabin:/root/.local/bin:/home/mpi/install/bin:/home/hadoop/software/java/bin:/home/hadoop/software/hadoop/bin:/root/anaconda3/bin:/media/disk1/framework-sim/chenyulong05/soft/nsight-compute:/data/common/gcc-10.2.0/bin:/root/.local/bin:/home/mpi/install/bin:/home/hadoop/software/java/bin:/home/hadoop/software/hadoop/bin:/root/anaconda3/bin:/root/.local/bin:/home/mpi/install/bin:/home/hadoop/software/java/bin:/home/hadoop/software/hadoop/bin:/root/anaconda3/bin:/root/.local/bin:/home/hadoop/software/java/bin:/home/hadoop/software/hadoop/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/root/bin:/media/disk1/framework-sim/chenyulong05/soft/cmake-3.30.1-linux-x86_64/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/root/anaconda3/envs/openxla/bin/python3.9 \
    TF2_BEHAVIOR=1 \
  /data/common/gcc-10.2.0/bin/gcc -MD -MF bazel-out/k8-opt/bin/xla/service/cpu/_objs/runtime_fp16/runtime_fp16.d '-frandom-seed=bazel-out/k8-opt/bin/xla/service/cpu/_objs/runtime_fp16/runtime_fp16.o' '-DBAZEL_CURRENT_REPOSITORY=""' -iquote . -iquote bazel-out/k8-opt/bin -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -fmerge-all-constants -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIE -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wno-invalid-partial-specialization -fno-omit-frame-pointer -no-canonical-prefixes -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections '--cuda-path=external/cuda_nvcc' -Wno-all -Wno-extra -Wno-deprecated -Wno-deprecated-declarations -Wno-ignored-attributes -Wno-array-bounds -Wunused-result '-Werror=unused-result' -Wswitch '-Werror=switch' '-Wno-error=unused-but-set-variable' -DAUTOLOAD_DYNAMIC_KERNELS -Wno-sign-compare '-std=c++17' -DEIGEN_AVOID_STL_ARRAY -c xla/service/cpu/runtime_fp16.cc -o bazel-out/k8-opt/bin/xla/service/cpu/_objs/runtime_fp16/runtime_fp16.o)
# Configuration: 660f29ed51dcc38c3b5336d29fc0fbe0348e3f0d5b0fb0ecb17aaeca8c1f8100
# Execution platform: @local_execution_config_platform//:platform
gcc: error: unrecognized command-line option '--cuda-path=external/cuda_nvcc'
Target //xla/tools:run_hlo_module failed to build
INFO: Elapsed time: 2.440s, Critical Path: 0.13s
INFO: 198 processes: 198 internal.
FAILED: Build did NOT complete successfully

Is this a common error? And I just wonder how to track an error like this. Thank you very much for your answer.

[ddunl]

Ah this is maybe a change introduced with the move to hermetic cuda. I should add a warning to configure.py that says that we only really support clang, and we just hope gcc works but it's never tested. Happy to accept any patch to configure.py that fixes things for gcc though.

[yulong-chen551]

Alright, thank you.

[ybaturina]

FYI: I've added experimental GCC support in CUDA repository rules in #16865.