build failed in conda environment

[s34296216]

Issue type

Build/Install

Have you reproduced the bug with TensorFlow Nightly?

No

Source

source

TensorFlow version

2.14.0.600

Custom code

Yes

OS platform and distribution

Linux Ubuntu 22.04

Mobile device

No response

Python version

3.10

Bazel version

bazel 6.5.0- (@non-git)

GCC/compiler version

gcc (conda-forge gcc 12.1.0-17) 12.1.0

CUDA/cuDNN version

No response

GPU model and memory

6700xt (gfx1031)

Current behavior?

Install clang 17 already

Standalone code to reproduce the issue

./build_rocm_python3

Relevant log output

You have bazel 6.5.0- (@non-git) installed.
Found possible Python library paths:
  /usr/lib/python3/dist-packages
  /usr/local/lib/python3.10/dist-packages
Please input the desired Python library path to use.  Default is [/usr/lib/python3/dist-packages]
Do you wish to build TensorFlow with CUDA support? [y/N]: No CUDA support will be enabled for TensorFlow.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]: 

Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
	--config=mkl         	# Build with MKL support.
	--config=mkl_aarch64 	# Build with oneDNN and Compute Library for the Arm Architecture (ACL).
	--config=monolithic  	# Config for mostly static monolithic build.
	--config=numa        	# Build with NUMA support.
	--config=dynamic_kernels	# (Experimental) Build kernels into separate shared objects.
	--config=v1          	# Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
	--config=nogcp       	# Disable GCP support.
	--config=nonccl      	# Disable NVIDIA NCCL support.
Configuration finished
WARNING: The following configs were expanded more than once: [rocm, rocm_base, no_tfrt, release_cpu_linux_base]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Reading 'startup' options from /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=183
INFO: Reading rc options for 'build' from /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/lib/python3/dist-packages --python_path=/usr/bin/python3 --config=rocm --action_env ROCM_PATH=/opt/rocm-6.1.2
INFO: Found applicable config definition build:short_logs in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:rocm in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --config=rocm_base --config=release_cpu_linux_base --action_env=CLANG_COMPILER_PATH=/usr/lib/llvm-17/bin/clang --action_env=TF_ROCM_CLANG=1 --linkopt=-fuse-ld=lld --linkopt=-Wl,--undefined-version --copt=-Wno-gnu-offsetof-extensions --copt=-Wno-unused-result
INFO: Found applicable config definition build:rocm_base in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --crosstool_top=@local_config_rocm//crosstool:toolchain --define=using_rocm_hipcc=true --define=tensorflow_mkldnn_contraction_kernel=0 --repo_env TF_NEED_ROCM=1 --config=no_tfrt
INFO: Found applicable config definition build:no_tfrt in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/ir,tensorflow/compiler/mlir/tfrt/ir/mlrt,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ifrt,tensorflow/compiler/mlir/tfrt/tests/mlrt,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/compiler/mlir/tfrt/transforms/mlrt,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/runtime_fallback/test,tensorflow/core/runtime_fallback/test/gpu,tensorflow/core/runtime_fallback/test/saved_model,tensorflow/core/runtime_fallback/test/testdata,tensorflow/core/tfrt/stubs,tensorflow/core/tfrt/tfrt_session,tensorflow/core/tfrt/mlrt,tensorflow/core/tfrt/mlrt/attribute,tensorflow/core/tfrt/mlrt/kernel,tensorflow/core/tfrt/mlrt/bytecode,tensorflow/core/tfrt/mlrt/interpreter,tensorflow/compiler/mlir/tfrt/translate/mlrt,tensorflow/compiler/mlir/tfrt/translate/mlrt/testdata,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils,tensorflow/core/tfrt/utils/debug,tensorflow/core/tfrt/saved_model/python,tensorflow/core/tfrt/graph_executor/python,tensorflow/core/tfrt/saved_model/utils
INFO: Found applicable config definition build:release_cpu_linux_base in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --repo_env=CC=/usr/lib/llvm-17/bin/clang --repo_env=BAZEL_COMPILER=/usr/lib/llvm-17/bin/clang --action_env=CLANG_COMPILER_PATH=/usr/lib/llvm-17/bin/clang --linkopt=-fuse-ld=lld
INFO: Found applicable config definition build:opt in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.tf_configure.bazelrc: --copt=-Wno-sign-compare --host_copt=-Wno-sign-compare
INFO: Found applicable config definition build:rocm in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --config=rocm_base --config=release_cpu_linux_base --action_env=CLANG_COMPILER_PATH=/usr/lib/llvm-17/bin/clang --action_env=TF_ROCM_CLANG=1 --linkopt=-fuse-ld=lld --linkopt=-Wl,--undefined-version --copt=-Wno-gnu-offsetof-extensions --copt=-Wno-unused-result
INFO: Found applicable config definition build:rocm_base in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --crosstool_top=@local_config_rocm//crosstool:toolchain --define=using_rocm_hipcc=true --define=tensorflow_mkldnn_contraction_kernel=0 --repo_env TF_NEED_ROCM=1 --config=no_tfrt
INFO: Found applicable config definition build:no_tfrt in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --deleted_packages=tensorflow/compiler/mlir/tfrt,tensorflow/compiler/mlir/tfrt/benchmarks,tensorflow/compiler/mlir/tfrt/ir,tensorflow/compiler/mlir/tfrt/ir/mlrt,tensorflow/compiler/mlir/tfrt/jit/python_binding,tensorflow/compiler/mlir/tfrt/jit/transforms,tensorflow/compiler/mlir/tfrt/python_tests,tensorflow/compiler/mlir/tfrt/tests,tensorflow/compiler/mlir/tfrt/tests/ifrt,tensorflow/compiler/mlir/tfrt/tests/mlrt,tensorflow/compiler/mlir/tfrt/tests/ir,tensorflow/compiler/mlir/tfrt/tests/analysis,tensorflow/compiler/mlir/tfrt/tests/jit,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_tfrt,tensorflow/compiler/mlir/tfrt/tests/lhlo_to_jitrt,tensorflow/compiler/mlir/tfrt/tests/tf_to_corert,tensorflow/compiler/mlir/tfrt/tests/tf_to_tfrt_data,tensorflow/compiler/mlir/tfrt/tests/saved_model,tensorflow/compiler/mlir/tfrt/transforms/lhlo_gpu_to_tfrt_gpu,tensorflow/compiler/mlir/tfrt/transforms/mlrt,tensorflow/core/runtime_fallback,tensorflow/core/runtime_fallback/conversion,tensorflow/core/runtime_fallback/kernel,tensorflow/core/runtime_fallback/opdefs,tensorflow/core/runtime_fallback/runtime,tensorflow/core/runtime_fallback/util,tensorflow/core/runtime_fallback/test,tensorflow/core/runtime_fallback/test/gpu,tensorflow/core/runtime_fallback/test/saved_model,tensorflow/core/runtime_fallback/test/testdata,tensorflow/core/tfrt/stubs,tensorflow/core/tfrt/tfrt_session,tensorflow/core/tfrt/mlrt,tensorflow/core/tfrt/mlrt/attribute,tensorflow/core/tfrt/mlrt/kernel,tensorflow/core/tfrt/mlrt/bytecode,tensorflow/core/tfrt/mlrt/interpreter,tensorflow/compiler/mlir/tfrt/translate/mlrt,tensorflow/compiler/mlir/tfrt/translate/mlrt/testdata,tensorflow/core/tfrt/gpu,tensorflow/core/tfrt/run_handler_thread_pool,tensorflow/core/tfrt/runtime,tensorflow/core/tfrt/saved_model,tensorflow/core/tfrt/graph_executor,tensorflow/core/tfrt/saved_model/tests,tensorflow/core/tfrt/tpu,tensorflow/core/tfrt/utils,tensorflow/core/tfrt/utils/debug,tensorflow/core/tfrt/saved_model/python,tensorflow/core/tfrt/graph_executor/python,tensorflow/core/tfrt/saved_model/utils
INFO: Found applicable config definition build:release_cpu_linux_base in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --repo_env=CC=/usr/lib/llvm-17/bin/clang --repo_env=BAZEL_COMPILER=/usr/lib/llvm-17/bin/clang --action_env=CLANG_COMPILER_PATH=/usr/lib/llvm-17/bin/clang --linkopt=-fuse-ld=lld
INFO: Found applicable config definition build:linux in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --linkopt=-Wl,--undefined-version --host_linkopt=-Wl,--undefined-version --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/misty_nep/Downloads/tensorflow/build_tf/tensorflow-upstream/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
WARNING: The following configs were expanded more than once: [rocm, rocm_base, no_tfrt, release_cpu_linux_base]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Analyzed target //tensorflow/tools/pip_package:wheel (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/misty_nep/miniconda3/envs/tensorflow/share/bazel/f67e28085f301ecc1d6f9c0d326c4335/external/com_google_absl/absl/types/BUILD.bazel:165:11: Compiling absl/types/bad_variant_access.cc failed: (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command (from target @com_google_absl//absl/types:bad_variant_access) 
  (cd /home/misty_nep/miniconda3/envs/tensorflow/share/bazel/f67e28085f301ecc1d6f9c0d326c4335/execroot/org_tensorflow && \
  exec env - \
    CLANG_COMPILER_PATH=/usr/lib/llvm-17/bin/clang \
    PATH=/opt/rocm-6.1.2/hcc/bin:/opt/rocm-6.1.2/hip/bin:/home/misty_nep/miniconda3/envs/tensorflow/bin:/home/misty_nep/miniconda3/condabin:/home/misty_nep/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/usr/bin/python3 \
    PYTHON_LIB_PATH=/usr/lib/python3/dist-packages \
    ROCM_PATH=/opt/rocm-6.1.2 \
    TF2_BEHAVIOR=1 \
    TF_ROCM_CLANG=1 \
    project_name=tf_nightly_rocm \
  external/local_config_rocm/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections -fdata-sections '-std=c++14' -MD -MF bazel-out/k8-opt/bin/external/com_google_absl/absl/types/_objs/bad_variant_access/bad_variant_access.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/com_google_absl/absl/types/_objs/bad_variant_access/bad_variant_access.pic.o' -fPIC '-DBAZEL_CURRENT_REPOSITORY="com_google_absl"' -iquote external/com_google_absl -iquote bazel-out/k8-opt/bin/external/com_google_absl -Wno-all -Wno-extra -Wno-deprecated -Wno-deprecated-declarations -Wno-ignored-attributes -Wno-array-bounds -Wunused-result '-Werror=unused-result' -Wswitch '-Werror=switch' '-Wno-error=unused-but-set-variable' -DAUTOLOAD_DYNAMIC_KERNELS -Wno-gnu-offsetof-extensions -Wno-unused-result -Wno-sign-compare -Wno-gnu-offsetof-extensions -Wno-unused-result '-std=c++17' -Wall -Wextra -Wcast-qual -Wconversion-null -Wformat-security -Wmissing-declarations -Woverlength-strings -Wpointer-arith -Wundef -Wunused-local-typedefs -Wunused-result -Wvarargs -Wvla -Wwrite-strings -DNOMINMAX -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' '-DTENSORFLOW_USE_ROCM=1' -D__HIP_PLATFORM_AMD__ -DEIGEN_USE_HIP -no-canonical-prefixes -fno-canonical-system-headers -c external/com_google_absl/absl/types/bad_variant_access.cc -o bazel-out/k8-opt/bin/external/com_google_absl/absl/types/_objs/bad_variant_access/bad_variant_access.pic.o)
# Configuration: 0a8b5b74ae6364e22e02a2851874ddfff500b5d52386a595dfc5315ab5a91efd
# Execution platform: @local_execution_config_platform//:platform
In file included from external/com_google_absl/absl/types/bad_variant_access.cc:15:
external/com_google_absl/absl/types/bad_variant_access.h:24:10: fatal error: 'stdexcept' file not found
   24 | #include <stdexcept>
      |          ^~~~~~~~~~~
1 error generated.
Target //tensorflow/tools/pip_package:wheel failed to build
INFO: Elapsed time: 0.364s, Critical Path: 0.09s
INFO: 22 processes: 20 internal, 2 local.
FAILED: Build did NOT complete successfully

[berndporr]

ERROR: /home/bernd/.cache/bazel/_bazel_bernd/2716ba2e215c041a26405d6ef9e43b69/external/com_google_absl/absl/types/BUILD.bazel:165:11: Compiling absl/types/bad_variant_access.cc failed: (Exit 1): clang failed: error executing command (from target @com_google_absl//absl/types:bad_variant_access) /usr/lib/llvm-17/bin/clang -MD -MF bazel-out/k8-opt/bin/external/com_google_absl/absl/types/_objs/bad_variant_access/bad_variant_access.pic.d ... (remaining 78 arguments skipped)
clang: warning: argument unused during compilation: '--cuda-path=/usr/local/cuda-12.5' [-Wunused-command-line-argument]
In file included from external/com_google_absl/absl/types/bad_variant_access.cc:15:
external/com_google_absl/absl/types/bad_variant_access.h:24:10: fatal error: 'stdexcept' file not found

I've got the same issue. Admittedly I'm using cuda-12.5 but I doubt that random basic STL classes are missing. Also had it with 'vector' not found.

BTW this is not a conda environment but a plain vanilla Ubuntu environment. llvm-17 installed via apt.

[berndporr]

Now 100% according to the recommendations with cuda-12.3 but now bombs by not finding stdc++!

ERROR: /home/bernd/.cache/bazel/_bazel_bernd/2716ba2e215c041a26405d6ef9e43b69/external/com_google_protobuf/BUILD.bazel:827:10: Linking external/com_google_protobuf/python/google/protobuf/internal/_api_implementation.so failed: (Exit 1): clang failed: error executing command (from target @com_google_protobuf//:python/google/protobuf/internal/_api_implementation.so) /usr/lib/llvm-17/bin/clang @bazel-out/k8-opt/bin/external/com_google_protobuf/python/google/protobuf/internal/_api_implementation.so-2.params
/usr/bin/ld: cannot find -lstdc++: No such file or directory
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //tensorflow/tools/pip_package:wheel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 133.160s, Critical Path: 0.67s
INFO: 136 processes: 76 internal, 60 local.
FAILED: Build did NOT complete successfully

[harkgill-amd]

Hi @s34296216 and @berndporr, I wasn't able to reproduce the 'stdexcept' file not found errors on both a baremetal installation and within a Conda environment. The build was done with the develop-upstream branch alongside the following variables:

Python Version: 3.10
ROCm Version: 6.2.4
Bazel Version: 6.5.0
GCC Version: conda-forge gcc 14.2.0-1

Could you please try this and let me know if you encounter any further issues? The C++ Standard Library headers are provided by the build-essential package so make sure to have it installed as well.

sudo apt-get install build-essential

I also noticed you're running on a 6700XT which is not officially supported for use with ROCm. Please be sure to set the environment variable HSA_OVERRIDE_GFX_VERSION=10.3.0 which will mimic a supported architecture.