[Python] Error when executing the command to build pyarrow

[llama90]

Describe the bug, including details regarding any error messages, version, and platform.

I encounter an error when executing the command(python setup.py build_ext --inplace) to build pyarrow.

Based on examining the git commits, it appears to be related as follows.

I use M1 mackbook.

• /Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/lib.cpp:75745:101: error: no member named 'MakeTensor' in 'arrow::extension::FixedShapeTensorType'
  • GH-37484: [Python] Add a FixedSizeTensorScalar class #37533
• /Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/lib.cpp:118724:67: error: no member named 'ExportChunkedArray' in namespace 'arrow'
• /Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/lib.cpp:118897:98: error: no member named 'ImportChunkedArray' in namespace 'arrow'
  • GH-39984: [Python] Add ChunkedArray import/export to/from C  #39985

error message

$ python setup.py build_ext --inplace 
/Users/lama/workspace/arrow-latest/python/setup.py:34: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
/Users/lama/mambaforge/envs/pyarrow-dev/lib/python3.10/site-packages/setuptools/__init__.py:80: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!

        ********************************************************************************
        Requirements should be satisfied by a PEP 517 installer.
        If you are using pip, you can try `pip install --use-pep517`.
        ********************************************************************************

!!
  dist.fetch_build_eggs(dist.setup_requires)
running build_ext
-- Running cmake for PyArrow
cmake -DCMAKE_INSTALL_PREFIX=/Users/lama/workspace/arrow-latest/python/pyarrow -DPYTHON_EXECUTABLE=/Users/lama/mambaforge/envs/pyarrow-dev/bin/python -DPython3_EXECUTABLE=/Users/lama/mambaforge/envs/pyarrow-dev/bin/python -DPYARROW_CXXFLAGS= -DPYARROW_BUILD_CUDA=off -DPYARROW_BUILD_SUBSTRAIT=off -DPYARROW_BUILD_FLIGHT=off -DPYARROW_BUILD_GANDIVA=off -DPYARROW_BUILD_ACERO=off -DPYARROW_BUILD_DATASET=off -DPYARROW_BUILD_ORC=off -DPYARROW_BUILD_PARQUET=off -DPYARROW_BUILD_PARQUET_ENCRYPTION=off -DPYARROW_BUILD_GCS=off -DPYARROW_BUILD_S3=off -DPYARROW_BUILD_HDFS=off -DPYARROW_BUNDLE_ARROW_CPP=off -DPYARROW_BUNDLE_CYTHON_CPP=off -DPYARROW_GENERATE_COVERAGE=off -DCMAKE_BUILD_TYPE=release /Users/lama/workspace/arrow-latest/python
CMake Warning at /Users/lama/mambaforge/envs/pyarrow-dev/share/cmake-3.27/Modules/Platform/Darwin-Initialize.cmake:306 (message):
  Ignoring CMAKE_OSX_SYSROOT value:

   /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

  because the directory does not exist.
Call Stack (most recent call first):
  /Users/lama/mambaforge/envs/pyarrow-dev/share/cmake-3.27/Modules/CMakeSystemSpecificInitialize.cmake:34 (include)
  CMakeLists.txt:22 (project)


-- Using ccache: /Users/lama/anaconda3/envs/pyarrow-dev/bin/ccache
-- System processor: arm64
-- Arrow build warning level: PRODUCTION
-- Build Type: RELEASE
-- CMAKE_C_FLAGS:  -Wall -Wno-unknown-warning-option -Wno-pass-failed -march=armv8-a -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /Users/lama/anaconda3/envs/pyarrow-dev/include -Qunused-arguments -fcolor-diagnostics  -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -Wno-parentheses-equality -Wno-constant-logical-operand -Wno-missing-declarations -Wno-sometimes-uninitialized -Wno-return-type-c-linkage
-- CMAKE_CXX_FLAGS:  -fno-aligned-new  -Wall -Wno-unknown-warning-option -Wno-pass-failed -march=armv8-a -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem /Users/lama/anaconda3/envs/pyarrow-dev/include -Qunused-arguments -fcolor-diagnostics  -fno-omit-frame-pointer -Wno-unused-variable -Wno-maybe-uninitialized -Wno-parentheses-equality -Wno-constant-logical-operand -Wno-missing-declarations -Wno-sometimes-uninitialized -Wno-return-type-c-linkage
-- Generator: Unix Makefiles
-- Build output directory: /Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/release
CMake Warning at /Users/lama/anaconda3/envs/pyarrow-dev/lib/cmake/Arrow/Findlz4Alt.cmake:29 (find_package):
  By not providing "Findlz4.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "lz4", but
  CMake did not find one.

  Could not find a package configuration file provided by "lz4" with any of
  the following names:

    lz4Config.cmake
    lz4-config.cmake

  Add the installation prefix of "lz4" to CMAKE_PREFIX_PATH or set "lz4_DIR"
  to a directory containing one of the above files.  If "lz4" provides a
  separate development package or SDK, be sure it has been installed.
Call Stack (most recent call first):
  /Users/lama/mambaforge/envs/pyarrow-dev/share/cmake-3.27/Modules/CMakeFindDependencyMacro.cmake:76 (find_package)
  /Users/lama/anaconda3/envs/pyarrow-dev/lib/cmake/Arrow/ArrowConfig.cmake:99 (find_dependency)
  /Users/lama/anaconda3/envs/pyarrow-dev/lib/cmake/Arrow/ArrowConfig.cmake:122 (arrow_find_dependencies)
  CMakeLists.txt:268 (find_package)


-- Arrow version: 16.0.0
-- Found the Arrow shared library: /Users/lama/anaconda3/envs/pyarrow-dev/lib/libarrow.1600.0.0.dylib
-- Found the Arrow import library: ARROW_IMPORT_LIB-NOTFOUND
-- Found the Arrow static library: /Users/lama/anaconda3/envs/pyarrow-dev/lib/libarrow.a
-- Parquet Encryption is NOT Enabled
-- Configuring done (1.4s)
-- Generating done (0.0s)
-- Build files have been written to: /Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310
-- Finished cmake for PyArrow
-- Running cmake --build for PyArrow
cmake --build . --config release --
[  1%] Compiling Cython CXX source for lib...
[  1%] Built target lib_pyx
[  3%] Built target cython_api_headers
[ 45%] Built target arrow_python
[ 47%] Building CXX object CMakeFiles/lib.dir/lib.cpp.o
/Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/lib.cpp:75751:101: error: no member named 'MakeTensor' in 'arrow::extension::FixedShapeTensorType'
            __pyx_t_1 = arrow::py::GetResultValue<std::shared_ptr< arrow::Tensor> >(__pyx_v_c_type->MakeTensor(__pyx_v_scalar)); if (unlikely(__Pyx_ErrOccurredWithGIL())) __PYX_ERR(4, 1084, __pyx_L4_error)
                                                                                    ~~~~~~~~~~~~~~  ^
/Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/lib.cpp:118724:67: error: no member named 'ExportChunkedArray' in namespace 'arrow'
            __pyx_t_6 = __pyx_f_7pyarrow_3lib_check_status(arrow::ExportChunkedArray(__pyx_v_self->sp_chunked_array, __pyx_v_c_stream)); if (unlikely(__pyx_t_6 == ((int)-1))) __PYX_ERR(15, 1357, __pyx_L6_error)
                                                           ~~~~~~~^
/Users/lama/workspace/arrow-latest/python/build/temp.macosx-11.0-arm64-cpython-310/lib.cpp:118897:98: error: no member named 'ImportChunkedArray' in namespace 'arrow'
            __pyx_t_2 = arrow::py::GetResultValue<std::shared_ptr< arrow::ChunkedArray> >(arrow::ImportChunkedArray(__pyx_v_c_stream)); if (unlikely(__Pyx_ErrOccurredWithGIL())) __PYX_ERR(15, 1385, __pyx_L4_error)
                                                                                          ~~~~~~~^
3 errors generated.
make[2]: *** [CMakeFiles/lib.dir/build.make:93: CMakeFiles/lib.dir/lib.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:161: CMakeFiles/lib.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
error: command '/Users/lama/mambaforge/envs/pyarrow-dev/bin/cmake' failed with exit code 2

Component(s)

Python

[rok]

Could you try deleting your cpp arrow and pyarrow binaries and build them again?

[llama90]

Thank you. I am trying as you suggested, but it seems like it might take some time as I am encountering issues during the attempt.

[rok]

I usually find it helpful to follow steps here.

[llama90]

@rok

I was able to resolve it thanks to you. After cleaning the entire build environment, it worked. Thank you :)