Build failure with ROCm 5.4.3 - due to missing header inclusion dependencies

[bondhugula]

Issue Type

Bug

Have you reproduced the bug with TF nightly?

Yes

Source

source

Tensorflow Version

git 49d0ea4

Custom Code

No

OS Platform and Distribution

Linux Ubuntu 22.04

Mobile device

No response

Python version

3.10

Bazel version

6.1.0

GCC/Compiler version

gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0

CUDA/cuDNN version

No response

GPU model and memory

Radeon 7900 XTX

Current Behaviour?

Build failed with errors below.

Standalone code to reproduce the issue

The build fails with the errors pasted above:


# ./configure with ROCM enabled and everything else default.
$ bazel build tensorflow/tools/pip_package:build_pip_package

ROCM 5.4.3 is installed at /opt/rocm-5.4.3/ and symlinked from /opt/rocm. (Installed via apt on Ubuntu 22.04 from official repos https://repo.radeon.com/rocm/apt/5.4.3 jammy/main amd64 Packages)

### Relevant log output

```shell
ERROR: /home/uday/tensorflow-upstream-rocm/tensorflow/compiler/xla/stream_executor/rocm/BUILD:477:11: Compiling tensorflow/compiler/xla/stream_executor/rocm/rocm_helpers.cu.cc failed: undeclared inclusion(s) in rule '//tensorflow/compiler/xla/stream_executor/rocm:rocm_helpers':
this rule is missing dependency declarations for the following files included by 'tensorflow/compiler/xla/stream_executor/rocm/rocm_helpers.cu.cc':
  '/opt/rocm-5.4.3/include/hip/hip_version.h'
  '/opt/rocm-5.4.3/include/hip/hip_runtime.h'
  '/opt/rocm-5.4.3/include/hip/hip_common.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_hip_runtime.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_hip_common.h'
  '/opt/rocm-5.4.3/include/hip/hip_runtime_api.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/host_defines.h'
  '/opt/rocm-5.4.3/include/hip/driver_types.h'
  '/opt/rocm-5.4.3/include/hip/texture_types.h'
  '/opt/rocm-5.4.3/include/hip/channel_descriptor.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_channel_descriptor.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_hip_vector_types.h'
  '/opt/rocm-5.4.3/include/hip/surface_types.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_hip_runtime_pt_api.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/hip_ldg.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_hip_atomic.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_device_functions.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/math_fwd.h'
  '/opt/rocm-5.4.3/include/hip/hip_vector_types.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/device_library_decls.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/llvm_intrinsics.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_warp_functions.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_hip_unsafe_atomics.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_surface_functions.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/ockl_image.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/texture_fetch_functions.h'
  '/opt/rocm-5.4.3/include/hip/hip_texture_types.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/texture_indirect_functions.fh'
  '/opt/rocm-5.4.3/include/hip/amd_detail/amd_math_functions.h'
  '/opt/rocm-5.4.3/include/hip/amd_detail/hip_fp16_math_fwd.h'
  '/opt/rocm-5.4.3/include/hip/library_types.h'

I can reproduce the same issue with the latest Tensorflow upstream git version. (514005d Jun 3, 2023).

[Aegis90]

Hi there I encountered the exact same problem building TF2.12on Rocm5.6. I was able to solve the problem by following the instructions on https://github.com/ROCmSoftwarePlatform/tensorflow-upstream/blob/develop-upstream/rocm_docs/tensorflow-build-from-source.md
'''
Follow steps at Basic Installation to install ROCm stack. NOTE: ROCm install instructions recommend a purge and reinstall of ROCm rather than upgrading from previous release. For details of the ROCm instructions, please refer to the ROCm QuickStart Installation Guide.

To build with ROCm3.10, set the following environment variables, and add those environment variables at the end of ~/.bashrc

export ROCM_PATH=/opt/rocm-4.0.0
export HCC_HOME=$ROCM_PATH/hcc
export HIP_PATH=$ROCM_PATH/hip
export PATH=$HCC_HOME/bin:$HIP_PATH/bin:$PATH
export ROCM_TOOLKIT_PATH=$ROCM_PATH
''''
Obviously you need to replace the rocm version in the file, which in your case would be /opt/rocm-5.4.3

[briansp2020]

Are there still people who are waiting for 7900XTX support? Though the performance is still a bit poor, TensorFlow-upstream now runs when built on the latest ROCm release. I was looking into the status of ROCm support for 7900XTX and found a few issues opened by different people and wanted to link all to the issue I opened in MIOpen repo. Though there has not been any confirmation from the developer, I think the performance issues are due to insufficient optimization of MIOpen.
ROCm/MIOpen#2342

[ppanchad-amd]

@bondhugula Has your issue been resolved? If so, please close the ticket. Thanks!

[bondhugula]

It's been over a year since this issue was filed. All the associated packages/software have now moved forward significantly. I haven't tried out with recent versions. But if others have reported that the build works fine, please go ahead and close this.

[taylding-amd]

Hi, this issue will be closed now as I could not reproduce it with the newest release. Please feel free to reopen it or ask follow-up questions should further assistance be required. Thanks!