Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Doc] Add use_tf32 cuda provider option #19548

Merged
merged 2 commits into from
Feb 20, 2024
Merged

[Doc] Add use_tf32 cuda provider option #19548

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f1412ca
Add use_tf32 cuda provider option
tianleiwu Feb 17, 2024
c3c7f7f
comment c api
tianleiwu Feb 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion docs/execution-providers/CUDA-ExecutionProvider.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,12 +159,30 @@ Default value: 0

### enable_skip_layer_norm_strict_mode

Whether to use strict mode in SkipLayerNormalization cuda implementation. The default and recommanded setting is false.
Whether to use strict mode in SkipLayerNormalization cuda implementation. The default and recommended setting is false.
If enabled, accuracy improvement and performance drop can be expected.
This flag is only supported from the V2 version of the provider options struct when used using the C API. (sample below)

Default value: 0

### use_tf32

TF32 is a math mode available on NVIDIA GPUs since Ampere. It allows certain float32 matrix multiplications and convolutions to run much faster on tensor cores with [TensorFloat-32](https://blogs.nvidia.com/blog/tensorfloat-32-precision-format/) reduced precision: float32 inputs are rounded with 10 bits of mantissa and results are accumulated with float32 precision.

Default value: 1

TensorFloat-32 is enabled by default. Starting from ONNX Runtime 1.18, you can use this flag to disable it for an inference session.

Example python usage:

```python
providers = [("CUDAExecutionProvider", {"use_tf32": 0})]
sess_options = ort.SessionOptions()
sess = ort.InferenceSession("my_model.onnx", sess_options=sess_options, providers=providers)
```

This flag is only supported from the V2 version of the provider options struct when used using the C API. (sample below)

### gpu_external_[alloc|free|empty_cache]

gpu_external_* is used to pass external allocators.
Expand All @@ -187,6 +205,7 @@ with `onnxruntime_USE_CUDA_NHWC_OPS=ON`.
If this is enabled the EP prefers NHWC operators over NCHW. Needed transforms will be added to the model. As NVIDIA
tensor cores can only work on NHWC layout this can increase performance if the model consists of many supported
operators and does not need too many new transpose nodes. Wider operator support is planned in the future.

This flag is only supported from the V2 version of the provider options struct when used using the C API. The V2
provider options struct can be created
using [CreateCUDAProviderOptions](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a0d29cbf555aa806c050748cf8d2dc172)
Expand Down
Loading