Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TensorRT EP] New CIs to test TRT+minimal CUDA build #23028

Merged
merged 8 commits into from
Dec 19, 2024
Merged

[TensorRT EP] New CIs to test TRT+minimal CUDA build #23028

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
87e5988
test
yf711 Dec 4, 2024
82877da
onnxruntime_DISABLE_CONTRIB_OPS=ON
yf711 Dec 4, 2024
fa4dd84
adapt https://github.com/microsoft/onnxruntime/pull/19052#issuecommen…
yf711 Dec 5, 2024
5d65c67
fix
yf711 Dec 5, 2024
bef14fd
update
yf711 Dec 5, 2024
0398283
add win trt+cudaMinimal build ci
yf711 Dec 9, 2024
a95f552
update cache key
yf711 Dec 9, 2024
9188612
test build only
yf711 Dec 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions tools/ci_build/build.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,7 @@ def convert_arg_line_to_args(self, arg_line):
)

parser.add_argument("--disable_cuda_nhwc_ops", action="store_true", help="Disable CUDA NHWC ops in build.")
parser.add_argument("--enable_cuda_minimal_build", action="store_true", help="Enable CUDA minimal build.")

# Python bindings
parser.add_argument("--enable_pybind", action="store_true", help="Enable Python Bindings.")
Expand Down Expand Up @@ -1093,6 +1094,7 @@ def generate_build_tree(
"-Donnxruntime_DISABLE_FLOAT8_TYPES=" + ("ON" if disable_float8_types else "OFF"),
"-Donnxruntime_DISABLE_SPARSE_TENSORS=" + ("ON" if disable_sparse_tensors else "OFF"),
"-Donnxruntime_DISABLE_OPTIONAL_TYPE=" + ("ON" if disable_optional_type else "OFF"),
"-Donnxruntime_CUDA_MINIMAL=" + ("ON" if args.enable_cuda_minimal_build else "OFF"),
]

if args.rv64:
Expand Down
108 changes: 108 additions & 0 deletions tools/ci_build/github/azure-pipelines/linux-gpu-tensorrt-cuda-minimal-ci-pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
##### start trigger Don't edit it manually, Please do edit set-trigger-rules.py ####
### please do rerun set-trigger-rules.py ###
trigger:
branches:
include:
- main
- rel-*
paths:
exclude:
- docs/**
- README.md
- CONTRIBUTING.md
- BUILD.md
- 'js/web'
- 'onnxruntime/core/providers/js'
pr:
branches:
include:
- main
- rel-*
paths:
exclude:
- docs/**
- README.md
- CONTRIBUTING.md
- BUILD.md
- 'js/web'
- 'onnxruntime/core/providers/js'
#### end trigger ####
parameters:
- name: CudaVersion
displayName: CUDA version
type: string
default: '12.2'
values:
- 11.8
- 12.2

variables:
- template: templates/common-variables.yml
- name: docker_base_image
${{ if eq(parameters.CudaVersion, '11.8') }}:
value: onnxruntimebuildcache.azurecr.io/internal/azureml/onnxruntime/build/cuda11_x64_almalinux8_gcc11:20241120.3
${{ if eq(parameters.CudaVersion, '12.2') }}:
value: onnxruntimebuildcache.azurecr.io/internal/azureml/onnxruntime/build/cuda12_x64_ubi8_gcc12:20241120.3
- name: linux_trt_version
${{ if eq(parameters.CudaVersion, '11.8') }}:
value: ${{ variables.linux_trt_version_cuda11 }}
${{ if eq(parameters.CudaVersion, '12.2') }}:
value: ${{ variables.linux_trt_version_cuda12 }}

jobs:
- job: Linux_Build
timeoutInMinutes: 180
variables:
skipComponentGovernanceDetection: true
ALLOW_RELEASED_ONNX_OPSET_ONLY: '1'
ORT_CACHE_DIR: '$(Agent.TempDirectory)/ort/ccache'
TODAY: $[format('{0:dd}{0:MM}{0:yyyy}', pipeline.startTime)]
workspace:
clean: all
pool: onnxruntime-tensorrt-linuxbuild-T4
steps:
- task: mspremier.PostBuildCleanup.PostBuildCleanup-task.PostBuildCleanup@3
displayName: 'Clean Agent Directories'
condition: always()

- checkout: self
clean: true
submodules: none

- template: templates/get-docker-image-steps.yml
parameters:
Dockerfile: tools/ci_build/github/linux/docker/Dockerfile.manylinux2_28_cuda
Context: tools/ci_build/github/linux/docker
DockerBuildArgs: "
--network=host
--build-arg BASEIMAGE=${{ variables.docker_base_image }}
--build-arg TRT_VERSION=${{ variables.linux_trt_version }}
--build-arg BUILD_UID=$( id -u )
"
Repository: onnxruntimetensorrtcudaminimalbuild

- template: templates/linux-build-step-with-cache.yml
parameters:
WithCache: true
Today: $(TODAY)
AdditionalKey: gpu_tensorrt_cuda_minimal
CacheDir: '$(ORT_CACHE_DIR)'
BuildStep:
- task: CmdLine@2
inputs:
script: |
docker run --gpus all --rm \
--volume /data/onnx:/data/onnx:ro \
--volume $(Build.SourcesDirectory):/onnxruntime_src \
--volume $(Build.BinariesDirectory):/build \
--volume /data/models:/build/models:ro \
--volume $HOME/.onnx:/home/onnxruntimedev/.onnx \
--volume $(ORT_CACHE_DIR):/cache \
-e ALLOW_RELEASED_ONNX_OPSET_ONLY=0 \
-e NIGHTLY_BUILD \
-e BUILD_BUILDNUMBER \
-e CCACHE_DIR=/cache -w /onnxruntime_src \
onnxruntimetensorrtcudaminimalbuild tools/ci_build/github/linux/build_tensorrt_ci.sh --cuda_minimal=ON
workingDirectory: $(Build.SourcesDirectory)

- template: templates/explicitly-defined-final-tasks.yml
86 changes: 86 additions & 0 deletions tools/ci_build/github/azure-pipelines/win-gpu-tensorrt-cuda-minimal-ci-pipeline.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
##### start trigger Don't edit it manually, Please do edit set-trigger-rules.py ####
### please do rerun set-trigger-rules.py ###
trigger:
branches:
include:
- main
- rel-*
paths:
exclude:
- docs/**
- README.md
- CONTRIBUTING.md
- BUILD.md
- 'js/web'
- 'onnxruntime/core/providers/js'
pr:
branches:
include:
- main
- rel-*
paths:
exclude:
- docs/**
- README.md
- CONTRIBUTING.md
- BUILD.md
- 'js/web'
- 'onnxruntime/core/providers/js'
#### end trigger ####
parameters:
- name: CudaVersion
displayName: CUDA version
type: string
default: '12.2'
values:
- 11.8
- 12.2

variables:
- template: templates/common-variables.yml
- name: win_trt_folder
${{ if eq(parameters.CudaVersion, '11.8') }}:
value: ${{ variables.win_trt_folder_cuda11 }}
${{ if eq(parameters.CudaVersion, '12.2') }}:
value: ${{ variables.win_trt_folder_cuda12 }}

jobs:
- job: 'build'
pool: 'onnxruntime-Win2022-GPU-A10'
variables:
MsbuildArguments: '-detailedsummary -maxcpucount -consoleloggerparameters:PerformanceSummary'
EnvSetupScript: setup_env_trt.bat
skipComponentGovernanceDetection: true
TODAY: $[format('{0:dd}{0:MM}{0:yyyy}', pipeline.startTime)]
timeoutInMinutes: 150
workspace:
clean: all
steps:
- template: templates/jobs/win-ci-prebuild-steps.yml
parameters:
EnvSetupScript: $(EnvSetupScript)
DownloadCUDA: true
DownloadTRT: true
BuildArch: 'x64'
BuildConfig: RelWithDebInfo
MachinePool: 'onnxruntime-Win2022-GPU-A10'
WithCache: true
Today: $(Today)

- template: templates/jobs/win-ci-build-steps.yml
parameters:
WithCache: True
Today: $(TODAY)
AdditionalKey: "gpu_tensorrt_cuda_minimal | RelWithDebInfo"
BuildPyArguments: '--config RelWithDebInfo --parallel --use_binskim_compliant_compile_flags --build_dir $(Build.BinariesDirectory) --skip_submodule_sync --build_shared_lib --update --cmake_generator "Visual Studio 17 2022" --build_wheel --enable_onnx_tests --use_tensorrt --tensorrt_home="$(Agent.TempDirectory)\${{ variables.win_trt_folder }}" --cuda_home="$(Agent.TempDirectory)\v${{ parameters.CudaVersion }}" --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=86 --enable_cuda_minimal_build'
MsbuildArguments: $(MsbuildArguments)
BuildArch: 'x64'
Platform: 'x64'
BuildConfig: RelWithDebInfo

- task: PythonScript@0
displayName: 'Build wheel'
inputs:
scriptPath: '$(Build.SourcesDirectory)\setup.py'
arguments: 'bdist_wheel'
workingDirectory: '$(Build.BinariesDirectory)\RelWithDebInfo\RelWithDebInfo'
13 changes: 13 additions & 0 deletions tools/ci_build/github/linux/build_tensorrt_ci.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/bin/bash
set -ex
#Every cuda container has this $CUDA_VERSION env var set.
SHORT_CUDA_VERSION=$(echo $CUDA_VERSION | sed 's/\([[:digit:]]\+\.[[:digit:]]\+\)\.[[:digit:]]\+/\1/')

Check warning on line 4 in tools/ci_build/github/linux/build_tensorrt_ci.sh

View workflow job for this annotation

GitHub Actions / Optional Lint 

[shellcheck] reported by reviewdog 🐶
See if you can use ${variable//search/replace} instead.

Raw Output:
./tools/ci_build/github/linux/build_tensorrt_ci.sh:4:22: info: See if you can use ${variable//search/replace} instead. (ShellCheck.SC2001)

Check warning on line 4 in tools/ci_build/github/linux/build_tensorrt_ci.sh

View workflow job for this annotation

GitHub Actions / Optional Lint 

[shellcheck] reported by reviewdog 🐶
Double quote to prevent globbing and word splitting.

Raw Output:
./tools/ci_build/github/linux/build_tensorrt_ci.sh:4:27: info: Double quote to prevent globbing and word splitting. (ShellCheck.SC2086)

#TODO: add --update --build
BUILD_ARGS=('--config' 'Release'
Expand All @@ -21,6 +21,19 @@
"CMAKE_CUDA_ARCHITECTURES=75"
"onnxruntime_BUILD_UNIT_TESTS=ON"
"onnxruntime_ENABLE_CUDA_EP_INTERNAL_TESTS=ON")

# Parse external args
for arg in "$@"; do
case $arg in
--cuda_minimal=ON)
# Replace onnxruntime_BUILD_UNIT_TESTS=ON with OFF
BUILD_ARGS=("${BUILD_ARGS[@]/onnxruntime_BUILD_UNIT_TESTS=ON/onnxruntime_BUILD_UNIT_TESTS=OFF}")
BUILD_ARGS+=("--enable_cuda_minimal_build")
BUILD_ARGS+=("--skip_tests")
;;
esac
done

if [ -x "$(command -v ninja)" ]; then
BUILD_ARGS+=('--cmake_generator' 'Ninja')
fi
Expand Down
Loading