Failing to run Dockerfile of ONNXRuntime with TensorRT integration[Build]

[nisharyan]

Describe the issue

When trying to build a Docker image for ONNXRuntime with TensorRT integration, I'm encountering an issue related to the Python version used during the build process. The current Dockerfile and scripts are attempting to install and use Python 3.7, but the ONNXRuntime build script seems to require Python 3.8 or newer.

Urgency

Critical

Target platform

Ubuntu 22.0

Build script

# --------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
# --------------------------------------------------------------
# Dockerfile to run ONNXRuntime with TensorRT integration

# nVidia TensorRT Base Image
ARG TRT_CONTAINER_VERSION=22.12
FROM nvcr.io/nvidia/tensorrt:${TRT_CONTAINER_VERSION}-py3

ARG ONNXRUNTIME_REPO=https://github.com/Microsoft/onnxruntime
ARG ONNXRUNTIME_BRANCH=main
ARG CMAKE_CUDA_ARCHITECTURES=37;50;52;60;61;70;75;80

RUN apt-get update &&\
    apt-get install -y sudo git bash unattended-upgrades
RUN unattended-upgrade

WORKDIR /code
ENV PATH /usr/local/nvidia/bin:/usr/local/cuda/bin:/code/cmake-3.27.3-linux-x86_64/bin:/opt/miniconda/bin:${PATH}

# Prepare onnxruntime repository & build onnxruntime with TensorRT
RUN git clone --single-branch --branch ${ONNXRUNTIME_BRANCH} --recursive ${ONNXRUNTIME_REPO} onnxruntime &&\
    /bin/sh onnxruntime/dockerfiles/scripts/install_common_deps.sh &&\
    trt_version=${TRT_VERSION:0:3} &&\
    /bin/sh onnxruntime/dockerfiles/scripts/checkout_submodules.sh ${trt_version} &&\
    cd onnxruntime &&\
    /bin/sh build.sh --allow_running_as_root --parallel --build_shared_lib --cuda_home /usr/local/cuda --cudnn_home /usr/lib/x86_64-linux-gnu/ --use_tensorrt --tensorrt_home /usr/lib/x86_64-linux-gnu/ --config Release --build_wheel --skip_tests --skip_submodule_sync --cmake_extra_defines '"CMAKE_CUDA_ARCHITECTURES='${CMAKE_CUDA_ARCHITECTURES}'"' &&\
    pip install /code/onnxruntime/build/Linux/Release/dist/*.whl &&\
    cd ..

Error / output

Traceback (most recent call last):
  File "/code/onnxruntime/tools/ci_build/build.py", line 72, in <module>
    _check_python_version()
  File "/code/onnxruntime/tools/ci_build/build.py", line 60, in _check_python_version
    f"Invalid Python version. At least Python 3.{required_minor_version} is required. "
__main__.UsageError: Invalid Python version. At least Python 3.8 is required. Actual Python version: 3.7.0 (default, Jun 28 2018, 13:15:42)
[GCC 7.2.0]

ERROR: failed to solve: process "/bin/sh -c git clone --single-branch --branch ${ONNXRUNTIME_BRANCH} --recursive ${ONNXRUNTIME_REPO} onnxruntime &&    /bin/sh onnxruntime/dockerfiles/scripts/install_common_deps.sh &&    trt_version=${TRT_VERSION:0:3} &&    /bin/sh onnxruntime/dockerfiles/scripts/checkout_submodules.sh ${trt_version} &&    cd onnxruntime &&    /bin/sh build.sh --allow_running_as_root --parallel --build_shared_lib --cuda_home /usr/local/cuda --cudnn_home /usr/lib/x86_64-linux-gnu/ --use_tensorrt --tensorrt_home /usr/lib/x86_64-linux-gnu/ --config Release --build_wheel --skip_tests --skip_submodule_sync --cmake_extra_defines '\"CMAKE_CUDA_ARCHITECTURES='${CMAKE_CUDA_ARCHITECTURES}'\"' &&    pip install /code/onnxruntime/build/Linux/Release/dist/*.whl &&    cd .." did not complete successfully: exit code: 1

Visual Studio Version

No response

GCC / Compiler Version

No response

[tianleiwu]

TRT 8.6.1 was released in May 2023. Why do you use TRT container of Dec 2022?

Try replace TRT_CONTAINER_VERSION=22.12 by TRT_CONTAINER_VERSION=23.06 to see whether it is resolved.

[nisharyan]

Tried with 23.06. The same error persists

[tianleiwu]

I did a quick test. The docker has python 3.10. I can use the docker to build like the following:

git clone https://github.com/microsoft/onnxruntime
cd onnxruntime
docker run --rm -it --gpus all -v $PWD:/workspace nvcr.io/nvidia/tensorrt:23.06-py3 /bin/bash

Then run the following to build ORT in the docker:

python3 -m pip install --upgrade pip
python3 -m pip install cmake packaging
export CUDACXX=/usr/local/cuda-12.1/bin/nvcc
git config --global --add safe.directory '*'
sh build.sh --config Release  --build_shared_lib --parallel --use_cuda --cuda_version 12.1 \
            --cuda_home /usr/local/cuda-12.1 --cudnn_home /usr/lib/x86_64-linux-gnu/ --build_wheel --skip_tests \
            --use_tensorrt --tensorrt_home /usr/src/tensorrt \
            --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=OFF \
            --cmake_extra_defines CMAKE_CUDA_ARCHITECTURES=70;75;80 \
            --allow_running_as_root
python3 -m pip install build/Linux/Release/dist/onnxruntime_gpu-*.whl

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.