[Build] onnxruntime-dnnl fails to build a wheel with --build_wheel flag on Ubuntu 22.04

[matthiasmindee]

Describe the issue

When trying to build onnxruntime-dnnl to use the oneDNN optimizations, and producing a wheel at the same time, the python script responsible for building the wheel seems to fail, even though onnxruntime-dnnl got built by cmake without problems.
The command run is ./build.sh --use_dnnl --build_wheel --parallel, and works without the --build_wheel flag.

Urgency

This is an issue of medium urgency, I can use onnxruntime without the oneDNN optimizations, but it's far from ideal.

Target platform

Ubuntu 22.04

Build script

The command is solely ./build.sh --use_dnnl --build_wheel --parallel and works when I don't try to build the wheel

Error / output

I'm including the last few lines that happen before the crash to show that it built correctly before.

[100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/home/matthias/Projects/onnxruntime/onnxruntime/test/providers/internal_testing/internal_testing_partitioning_tests.cc.o
[100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/home/matthias/Projects/onnxruntime/onnxruntime/test/providers/internal_testing/internal_testing_tests.cc.o
[100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/home/matthias/Projects/onnxruntime/onnxruntime/test/quantization/quantization_test.cc.o
[100%] Building CXX object CMakeFiles/onnxruntime_test_all.dir/home/matthias/Projects/onnxruntime/onnxruntime/test/unittest_main/test_main.cc.o
[100%] Linking CXX executable onnxruntime_test_all
[100%] Built target onnxruntime_test_all
gmake: *** [Makefile:166: all] Error 2
Traceback (most recent call last):
  File "/home/matthias/Projects/onnxruntime/tools/ci_build/build.py", line 2699, in <module>
    sys.exit(main())
  File "/home/matthias/Projects/onnxruntime/tools/ci_build/build.py", line 2592, in main
    build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
  File "/home/matthias/Projects/onnxruntime/tools/ci_build/build.py", line 1496, in build_targets
    run_subprocess(cmd_args, env=env)
  File "/home/matthias/Projects/onnxruntime/tools/ci_build/build.py", line 801, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
  File "/home/matthias/Projects/onnxruntime/tools/python/util/run.py", line 49, in run
    completed_process = subprocess.run(
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/usr/local/bin/cmake', '--build', '/home/matthias/Projects/onnxruntime/build/Linux/Debug', '--config', 'Debug', '--', '-j16']' returned non-zero exit status 2.

another shot at directly running the command from my terminal instead of running it via a python subprocess gave this output

/home/matthias/Projects/onnxruntime/onnxruntime/python/onnxruntime_pybind_sparse_tensor.cc:11:10: fatal error: numpy/arrayobject.h: No such file or directory
   11 | #include <numpy/arrayobject.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/onnxruntime_pybind11_state.dir/build.make:174: CMakeFiles/onnxruntime_pybind11_state.dir/home/matthias/Projects/onnxruntime/onnxruntime/python/onnxruntime_pybind_sparse_tensor.cc.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
/home/matthias/Projects/onnxruntime/onnxruntime/python/onnxruntime_pybind_ortvalue.cc:11:10: fatal error: numpy/arrayobject.h: No such file or directory
   11 | #include <numpy/arrayobject.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/onnxruntime_pybind11_state.dir/build.make:132: CMakeFiles/onnxruntime_pybind11_state.dir/home/matthias/Projects/onnxruntime/onnxruntime/python/onnxruntime_pybind_ortvalue.cc.o] Error 1
/home/matthias/Projects/onnxruntime/onnxruntime/python/onnxruntime_pybind_mlvalue.cc:11:10: fatal error: numpy/arrayobject.h: No such file or directory
   11 | #include <numpy/arrayobject.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/onnxruntime_pybind11_state.dir/build.make:104: CMakeFiles/onnxruntime_pybind11_state.dir/home/matthias/Projects/onnxruntime/onnxruntime/python/onnxruntime_pybind_mlvalue.cc.o] Error 1
/home/matthias/Projects/onnxruntime/onnxruntime/python/onnxruntime_pybind_iobinding.cc:11:10: fatal error: numpy/arrayobject.h: No such file or directory
   11 | #include <numpy/arrayobject.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
/home/matthias/Projects/onnxruntime/onnxruntime/python/onnxruntime_pybind_state.cc:10:10: fatal error: numpy/arrayobject.h: No such file or directory
   10 | #include <numpy/arrayobject.h>
      |          ^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/onnxruntime_pybind11_state.dir/build.make:188: CMakeFiles/onnxruntime_pybind11_state.dir/home/matthias/Projects/onnxruntime/onnxruntime/python/onnxruntime_pybind_state.cc.o] Error 1
gmake[2]: *** [CMakeFiles/onnxruntime_pybind11_state.dir/build.make:90: CMakeFiles/onnxruntime_pybind11_state.dir/home/matthias/Projects/onnxruntime/onnxruntime/python/onnxruntime_pybind_iobinding.cc.o] Error 1
[100%] Built target onnxruntime_test_all
gmake[1]: *** [CMakeFiles/Makefile2:2288: CMakeFiles/onnxruntime_pybind11_state.dir/all] Error 2
gmake: *** [Makefile:166: all] Error 2

Visual Studio Version

No response

GCC / Compiler Version

gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0

[bashirmindee]

I have exactly the same issue. Any updates ?

[matthiasmindee]

so the root of the issue was that requirements should be installed before attempting to build this wheel, closing this issue now.