Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arrow: Build with Zstd support #10017

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 19 additions & 6 deletions A/Arrow/build_tarballs.jl
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,9 @@
using BinaryBuilder, Pkg

name = "Arrow"
version = v"18.1.0"
# This installs Arrow 18.1.0.
# We declare this as 18.1.1 because we enabled the Zstd library, changing the package dependencies.
version = v"18.1.1"

# Collection of sources required to complete build
sources = [
Expand All @@ -17,10 +19,13 @@ script = raw"""

cd $WORKSPACE/srcdir/arrow

# Set toolchain for building external deps
for f in ${WORKSPACE}/srcdir/patches/*.patch; do
atomic_patch -p1 ${f}
done
atomic_patch -p1 ${WORKSPACE}/srcdir/patches/boost.patch
atomic_patch -p1 ${WORKSPACE}/srcdir/patches/windows.patch
if [[ $target == *mingw32* ]]; then
# This hard-codes the name and location of the zstd library and
# must not be applied on other architectures
atomic_patch -p1 ${WORKSPACE}/srcdir/patches/windows-zstd.patch
fi

cd cpp

Expand Down Expand Up @@ -48,11 +53,18 @@ CMAKE_FLAGS=(
-DARROW_WITH_SNAPPY=ON
-DARROW_WITH_UTF8PROC=OFF
-DARROW_WITH_ZLIB=ON
-DARROW_WITH_ZSTD=OFF
-DARROW_WITH_ZSTD=ON
-DPARQUET_BUILD_EXECUTABLES=OFF
-Dxsimd_SOURCE=AUTO
)

if [[ $target == *mingw32* ]]; then
# Cmake doesn't find the zstd library on Windows. It does find
# zstd, but it somehow can't determine the path to the actual
# library.
CMAKE_FLAGS+=(-DZSTD_LIB=/workspace/destdir/lib/libzstd.dll.a)
fi

cmake -B cmake-build "${CMAKE_FLAGS[@]}"
cmake --build cmake-build --parallel ${nproc}
cmake --install cmake-build
Expand All @@ -76,6 +88,7 @@ dependencies = [
Dependency("Lz4_jll"),
Dependency("Thrift_jll"; compat="0.21"),
Dependency("Zlib_jll"),
Dependency("Zstd_jll"; compat="1.5.6"),
Dependency("boost_jll"; compat="=1.79.0"),
Dependency("brotli_jll"; compat="1.1.0"),
Dependency("snappy_jll"; compat="1.2.1"),
Expand Down
13 changes: 13 additions & 0 deletions A/Arrow/bundled/patches/windows-zstd.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
Installing Arrow with Zstd fails on Windows. Reported upstream as
<https://github.com/apache/arrow/issues/45032>

--- a/cpp/cmake_modules/FindzstdAlt.cmake
+++ b/cpp/cmake_modules/FindzstdAlt.cmake
@@ -136,6 +136,7 @@
endif()
set_target_properties(${zstd_TARGET}
PROPERTIES IMPORTED_LOCATION "${ZSTD_LIB}"
+ IMPORTED_IMPLIB /workspace/destdir/bin/libzstd-1.dll
INTERFACE_INCLUDE_DIRECTORIES "${ZSTD_INCLUDE_DIR}")
message(STATUS "Zstandard library: ${ZSTD_LIB}")
message(STATUS "Zstandard include directory: ${ZSTD_INCLUDE_DIR}")