Skip to content

Commit

Permalink
[TensorRT EP] Update deprecated TRT api (#18834)
Browse files Browse the repository at this point in the history 
### Description
<!-- Describe your changes. -->
Update deprecated TRT api:
1.
[setMaxWorkspaceSize](https://docs.nvidia.com/deeplearning/tensorrt/api/c_api/classnvinfer1_1_1_i_builder_config.html#a8209999988ab480c60c8a905dfd2654d)(max_workspace_size_)-------->setMemoryPoolLimit(nvinfer1::MemoryPoolType::kWORKSPACE,
max_workspace_size_)
2.
[kENABLE_TACTIC_HEURISTIC](https://docs.nvidia.com/deeplearning/tensorrt/api/c_api/namespacenvinfer1.html#abdc74c40fe7a0c3d05d2caeccfbc29c1a1215692ad24465e4d9e37a8a7fce1a38)-------->supersede
by trt builder optimization level 2

Perf & warning log comparison
<html xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882"
xmlns="http://www.w3.org/TR/REC-html40">

<head>

<meta name=ProgId content=OneNote.File>
<meta name=Generator content="Microsoft OneNote 15">
</head>

<body lang=en-US style='font-family:"Microsoft YaHei";font-size:12.0pt'>
<!--StartFragment-->

<div style='direction:ltr'>


TRT EP options | User will see corresponding warning logs: | Average
inference time cost (FRCNN on A100)
-- | -- | --
trt_build_heuristics_enable\|true | [TensorRT EP]
trt_build_heuristics_enable is deprecated on TRT 8.6 onwards. Please set
builder optimization level as 2 to enable builder heuristics. | ~300ms
trt_build_heuristics_enable\|true   trt_builder_optimization_level\|2 |
[TensorRT EP] Builder heuristics are enabled automatically by builder
optimization level 2. trt_build_heuristics_enable is deprecated on TRT
8.6 onwards. | ~275ms
trt_builder_optimization_level\|2 |   | ~275ms



</div>

<!--EndFragment-->
</body>

</html>




### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
Prepare for upcoming TRT 10
  • Loading branch information
@yf711
yf711 authored Dec 18, 2023
1 parent ad476d5 commit 9426bd5
Showing 1 changed file with 18 additions and 7 deletions.
25 changes: 18 additions & 7 deletions onnxruntime/core/providers/tensorrt/tensorrt_execution_provider.cc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2506,7 +2506,7 @@ common::Status TensorrtExecutionProvider::Compile(const std::vector<FusedNodeAnd
auto trt_config = std::unique_ptr<nvinfer1::IBuilderConfig>(trt_builder->createBuilderConfig());
auto trt_parser = tensorrt_ptr::unique_pointer<nvonnxparser::IParser>(nvonnxparser::createParser(*trt_network, trt_logger));
trt_parser->parse(string_buf.data(), string_buf.size(), model_path_);
trt_config->setMaxWorkspaceSize(max_workspace_size_);
trt_config->setMemoryPoolLimit(nvinfer1::MemoryPoolType::kWORKSPACE, max_workspace_size_);

// Force Pow + Reduce ops in layer norm to run in FP32 to avoid overflow
if (fp16_enable_ && layer_norm_fp32_fallback_) {
Expand Down Expand Up @@ -2723,13 +2723,24 @@ common::Status TensorrtExecutionProvider::Compile(const std::vector<FusedNodeAnd
trt_config->setFlag(nvinfer1::BuilderFlag::kSPARSE_WEIGHTS);
LOGS_DEFAULT(VERBOSE) << "[TensorRT EP] Sparse weights are allowed";
}

// enable builder heuristics
#if NV_TENSORRT_MAJOR == 8 && NV_TENSORRT_MINOR == 5
if (build_heuristics_enable_) {
trt_config->setFlag(nvinfer1::BuilderFlag::kENABLE_TACTIC_HEURISTIC);
LOGS_DEFAULT(VERBOSE) << "[TensorRT EP] Builder heuristics are enabled";
LOGS_DEFAULT(WARNING) << "[TensorRT EP] Builder heuristics are enabled."
<< " For TRT > 8.5, trt_build_heuristics_enable is deprecated, please set builder optimization level as 2 to enable builder heuristics.";
}
#if NV_TENSORRT_MINOR > 5 && NV_TENSORRT_MAJOR >= 8
#elif NV_TENSORRT_MAJOR == 8 && NV_TENSORRT_MINOR > 5 || NV_TENSORRT_MAJOR > 8
// for TRT 8.6 onwards, heuristic-based tactic option is automatically enabled by setting builder optimization level 2
if (build_heuristics_enable_) {
if (builder_optimization_level_ == 2) {
LOGS_DEFAULT(WARNING) << "[TensorRT EP] Builder heuristics are automatically enabled by builder optimization level 2. trt_build_heuristics_enable is deprecated on TRT 8.6 onwards.";
} else {
LOGS_DEFAULT(WARNING) << "[TensorRT EP] trt_build_heuristics_enable is deprecated on TRT 8.6 onwards. Please set builder optimization level as 2 to enable builder heuristics.";
}
}
#endif

#if NV_TENSORRT_MAJOR == 8 && NV_TENSORRT_MINOR > 5 || NV_TENSORRT_MAJOR > 8
// switch optimizaion level
if (builder_optimization_level_ != 3) {
trt_config->setBuilderOptimizationLevel(builder_optimization_level_);
Expand Down Expand Up @@ -3125,7 +3136,7 @@ common::Status TensorrtExecutionProvider::Compile(const std::vector<FusedNodeAnd
trt_state->context->reset();
trt_state->engine->reset();
auto trt_config = std::unique_ptr<nvinfer1::IBuilderConfig>(trt_builder->createBuilderConfig());
trt_config->setMaxWorkspaceSize(*(trt_state->max_workspace_size_ptr));
trt_config->setMemoryPoolLimit(nvinfer1::MemoryPoolType::kWORKSPACE, *(trt_state->max_workspace_size_ptr));
for (auto trt_profile : trt_profiles) {
trt_config->addOptimizationProfile(trt_profile);
}
Expand Down Expand Up @@ -3166,7 +3177,7 @@ common::Status TensorrtExecutionProvider::Compile(const std::vector<FusedNodeAnd
trt_config->setFlag(nvinfer1::BuilderFlag::kENABLE_TACTIC_HEURISTIC);
LOGS_DEFAULT(VERBOSE) << "[TensorRT EP] Builder heuristics are enabled";
}
#if NV_TENSORRT_MINOR > 5 && NV_TENSORRT_MAJOR >= 8
#if NV_TENSORRT_MAJOR == 8 && NV_TENSORRT_MINOR > 5 || NV_TENSORRT_MAJOR > 8
// switch optimizaion level
if (trt_state->builder_optimization_level != 3) {
trt_config->setBuilderOptimizationLevel(trt_state->builder_optimization_level);
Expand Down

0 comments on commit 9426bd5

Please sign in to comment.