Skip to content

Commit

Permalink
Adding new shell script for cuda 12
Browse files Browse the repository at this point in the history
  • Loading branch information
jchen351 committed Nov 8, 2023
1 parent 476a8fb commit ae609f8
Show file tree
Hide file tree
Showing 2 changed files with 79 additions and 1 deletion.
78 changes: 78 additions & 0 deletions tools/ci_build/github/linux/build_linux_python_package_cuda12.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
#!/bin/bash
set -e -x

# This script invokes build.py

mkdir -p /build/dist

EXTRA_ARG=""

# Put 3.8 at the last because Ubuntu 20.04 use python 3.8 and we will upload the intermediate build files of this
# config to Azure DevOps Artifacts and download them to a Ubuntu 20.04 machine to run the tests.
PYTHON_EXES=("/opt/python/cp39-cp39/bin/python3.9" "/opt/python/cp310-cp310/bin/python3.10" "/opt/python/cp311-cp311/bin/python3.11" "/opt/python/cp38-cp38/bin/python3.8")
while getopts "d:p:x:c:" parameter_Option
do case "${parameter_Option}"

Check warning on line 14 in tools/ci_build/github/linux/build_linux_python_package_cuda12.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/build_linux_python_package_cuda12.sh#L14 <ShellCheck.SC2220>

Invalid flags are not handled. Add a *) case.
Raw output
./tools/ci_build/github/linux/build_linux_python_package_cuda12.sh:14:4: warning: Invalid flags are not handled. Add a *) case. (ShellCheck.SC2220)
in
#GPU or CPU.
d) BUILD_DEVICE=${OPTARG};;
p) PYTHON_EXES=(${OPTARG});;

Check warning on line 18 in tools/ci_build/github/linux/build_linux_python_package_cuda12.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/build_linux_python_package_cuda12.sh#L18 <ShellCheck.SC2206>

Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.
Raw output
./tools/ci_build/github/linux/build_linux_python_package_cuda12.sh:18:17: warning: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a. (ShellCheck.SC2206)
x) EXTRA_ARG=(${OPTARG});;

Check warning on line 19 in tools/ci_build/github/linux/build_linux_python_package_cuda12.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/build_linux_python_package_cuda12.sh#L19 <ShellCheck.SC2206>

Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.
Raw output
./tools/ci_build/github/linux/build_linux_python_package_cuda12.sh:19:15: warning: Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a. (ShellCheck.SC2206)
c) BUILD_CONFIG=${OPTARG};;
esac
done

BUILD_ARGS=("--build_dir" "/build" "--config" "$BUILD_CONFIG" "--update" "--build" "--skip_submodule_sync" "--parallel" "--build_wheel")

if [ "$BUILD_CONFIG" == "Debug" ]; then
CFLAGS="-ggdb3"
CXXFLAGS="-ggdb3"
else
CFLAGS="-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -O3 -pipe -Wl,--strip-all"
CXXFLAGS="-Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -O3 -pipe -Wl,--strip-all"
BUILD_ARGS+=("--enable_lto")
fi

# Depending on how the compiler has been configured when it was built, sometimes "gcc -dumpversion" shows the full version.
GCC_VERSION=$(gcc -dumpversion | cut -d . -f 1)
#-fstack-clash-protection prevents attacks based on an overlapping heap and stack.
if [ "$GCC_VERSION" -ge 8 ]; then
CFLAGS="$CFLAGS -fstack-clash-protection"
CXXFLAGS="$CXXFLAGS -fstack-clash-protection"
fi

ARCH=$(uname -m)

if [ "$ARCH" == "x86_64" ] && [ "$GCC_VERSION" -ge 9 ]; then
CFLAGS="$CFLAGS -fcf-protection"
CXXFLAGS="$CXXFLAGS -fcf-protection"
fi

echo "EXTRA_ARG:"
echo $EXTRA_ARG

Check warning on line 51 in tools/ci_build/github/linux/build_linux_python_package_cuda12.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/build_linux_python_package_cuda12.sh#L51 <ShellCheck.SC2128>

Expanding an array without an index only gives the first element.
Raw output
./tools/ci_build/github/linux/build_linux_python_package_cuda12.sh:51:6: warning: Expanding an array without an index only gives the first element. (ShellCheck.SC2128)

Check notice on line 51 in tools/ci_build/github/linux/build_linux_python_package_cuda12.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/build_linux_python_package_cuda12.sh#L51 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/build_linux_python_package_cuda12.sh:51:6: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)

if [ "$EXTRA_ARG" != "" ]; then

Check warning on line 53 in tools/ci_build/github/linux/build_linux_python_package_cuda12.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/build_linux_python_package_cuda12.sh#L53 <ShellCheck.SC2128>

Expanding an array without an index only gives the first element.
Raw output
./tools/ci_build/github/linux/build_linux_python_package_cuda12.sh:53:7: warning: Expanding an array without an index only gives the first element. (ShellCheck.SC2128)
BUILD_ARGS+=("$EXTRA_ARG")

Check warning on line 54 in tools/ci_build/github/linux/build_linux_python_package_cuda12.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/build_linux_python_package_cuda12.sh#L54 <ShellCheck.SC2128>

Expanding an array without an index only gives the first element.
Raw output
./tools/ci_build/github/linux/build_linux_python_package_cuda12.sh:54:19: warning: Expanding an array without an index only gives the first element. (ShellCheck.SC2128)
fi

if [ "$ARCH" == "x86_64" ]; then
#ARM build machines do not have the test data yet.
BUILD_ARGS+=("--enable_onnx_tests")
fi

if [ "$BUILD_DEVICE" == "GPU" ]; then
#Enable CUDA and TRT EPs.
ONNXRUNTIME_CUDA_VERSION="12.2"
BUILD_ARGS+=("--nvcc_threads=1" "--use_cuda" "--use_tensorrt" "--cuda_version=$ONNXRUNTIME_CUDA_VERSION" "--tensorrt_home=/usr" "--cuda_home=/usr/local/cuda-$ONNXRUNTIME_CUDA_VERSION" "--cudnn_home=/usr/local/cuda-$ONNXRUNTIME_CUDA_VERSION" "--cmake_extra_defines" "CMAKE_CUDA_ARCHITECTURES=52;60;61;70;75;80")
fi

export CFLAGS
export CXXFLAGS
for PYTHON_EXE in "${PYTHON_EXES[@]}"
do
rm -rf /build/$BUILD_CONFIG

Check notice on line 72 in tools/ci_build/github/linux/build_linux_python_package_cuda12.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/build_linux_python_package_cuda12.sh#L72 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/build_linux_python_package_cuda12.sh:72:17: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)
${PYTHON_EXE} /onnxruntime_src/tools/ci_build/build.py "${BUILD_ARGS[@]}"

cp /build/$BUILD_CONFIG/dist/*.whl /build/dist

Check notice on line 75 in tools/ci_build/github/linux/build_linux_python_package_cuda12.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/build_linux_python_package_cuda12.sh#L75 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/build_linux_python_package_cuda12.sh:75:13: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)
done

which ccache && ccache -sv && ccache -z
2 changes: 1 addition & 1 deletion tools/ci_build/github/linux/run_python_dockerbuild.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ docker run --rm \
-e NIGHTLY_BUILD \
-e BUILD_BUILDNUMBER \
$ADDITIONAL_DOCKER_PARAMETER \

Check notice on line 25 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L25 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:25:5: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)
$DOCKER_IMAGE tools/ci_build/github/linux/build_linux_arm64_python_package.sh -d $DEVICE -c $BUILD_CONFIG -x $BUILD_EXTR_PAR
$DOCKER_IMAGE tools/ci_build/github/linux/build_linux_python_package_cuda12.sh -d $DEVICE -c $BUILD_CONFIG -x $BUILD_EXTR_PAR

Check notice on line 26 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L26 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:26:5: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)

Check notice on line 26 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L26 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:26:87: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)

Check notice on line 26 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L26 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:26:98: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)

Check notice on line 26 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L26 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:26:115: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)

sudo rm -rf $BUILD_BINARIESDIRECTORY/$BUILD_CONFIG/onnxruntime $BUILD_BINARIESDIRECTORY/$BUILD_CONFIG/pybind11 \

Check notice on line 28 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L28 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:28:13: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)

Check notice on line 28 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L28 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:28:38: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)

Check notice on line 28 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L28 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:28:64: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)

Check notice on line 28 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L28 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:28:89: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)
$BUILD_BINARIESDIRECTORY/$BUILD_CONFIG/models $BUILD_BINARIESDIRECTORY/$BUILD_CONFIG/_deps \

Check notice on line 29 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L29 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:29:5: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)

Check notice on line 29 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L29 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:29:30: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)

Check notice on line 29 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L29 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:29:51: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)

Check notice on line 29 in tools/ci_build/github/linux/run_python_dockerbuild.sh

View workflow job for this annotation

GitHub Actions / shellcheck

[shellcheck] tools/ci_build/github/linux/run_python_dockerbuild.sh#L29 <ShellCheck.SC2086>

Double quote to prevent globbing and word splitting.
Raw output
./tools/ci_build/github/linux/run_python_dockerbuild.sh:29:76: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)
Expand Down

0 comments on commit ae609f8

Please sign in to comment.