[BUG] subprocess.CalledProcessError

[jagadish-amd]

Due to the PR #6498
we are seeing issues on ROCm.
1.
[rank7]: raise child_exception_type(errno_num, err_msg, err_filename)
[rank7]: FileNotFoundError: [Errno 2] No such file or directory: "/opt/rocm/bin/rocminfo | grep -Eo -m1 'Wavefront Size:[[:space:]]+[0-9]+' | grep -Eo '[0-9]+'"
[rank0]: Traceback (most recent call last):

Building extension module transformer_inference...
Using envvar MAX_JOBS (32) as the number of workers...
Using /root/.cache/torch_extensions/py39_cpu as PyTorch extensions root...
Using /root/.cache/torch_extensions/py39_cpu as PyTorch extensions root...
Using /root/.cache/torch_extensions/py39_cpu as PyTorch extensions root...
ninja: error: build.ninja:8: unexpected '='

[rank4]: Traceback (most recent call last):
[rank4]: File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 2107, in _run_ninja_build
[rank4]: subprocess.run(
[rank4]: File "/opt/conda/envs/py_3.9/lib/python3.9/subprocess.py", line 528, in run
[rank4]: raise CalledProcessError(retcode, process.args,
[rank4]: subprocess.CalledProcessError: Command '['ninja', '-v', '-j', '32']' returned non-zero exit status 1.

Simple code to repro the issue.
import deepspeed
deepspeed.ops.op_builder.InferenceBuilder().load()

@loadams @jithunnair-amd @pruthvistony @jeffdaily

[jagadish-amd]

ping @tjruwase

[jagadish-amd]

shlex.split() is not working as expected for the strings. (rocm_wavefront_size_cmd, rocm_gpu_arch_cmd)
The below patch resolves the errors. (bringing back the old code partially)

diff --git a/op_builder/builder.py b/op_builder/builder.py
index e935a179..aa564f74 100644
--- a/op_builder/builder.py
+++ b/op_builder/builder.py
@@ -254,7 +254,7 @@ class OpBuilder(ABC):
         rocm_gpu_arch_cmd = str(rocm_info) + " | grep -o -m 1 'gfx.*'"
         try:
             safe_cmd = shlex.split(rocm_gpu_arch_cmd)
-            result = subprocess.check_output(safe_cmd)
+            result = subprocess.check_output(rocm_gpu_arch_cmd, shell=True)
             rocm_gpu_arch = result.decode('utf-8').strip()
         except subprocess.CalledProcessError:
             rocm_gpu_arch = ""
@@ -273,7 +273,7 @@ class OpBuilder(ABC):
             rocm_info) + " | grep -Eo -m1 'Wavefront Size:[[:space:]]+[0-9]+' | grep -Eo '[0-9]+'"
         try:
             safe_cmd = shlex.split(rocm_wavefront_size_cmd)
-            result = subprocess.check_output(rocm_wavefront_size_cmd)
+            result = subprocess.check_output(rocm_wavefront_size_cmd, shell=True)
             rocm_wavefront_size = result.decode('utf-8').strip()
         except subprocess.CalledProcessError:
             rocm_wavefront_size = "32"

Note that, safe_cmd is not used in get_rocm_wavefront_size(). (in the checked-in code)

[tjruwase]

@jagadish-amd, thanks! This fix looks good since there is no user-input involved in the cmds.

[tjruwase]

@jagadish-amd, can you please share a PR with this fix?