From de4aa3cc1d7d4f29b6e83d42560961223e1b9632 Mon Sep 17 00:00:00 2001 From: nakersha Date: Mon, 25 Sep 2023 19:00:40 -0700 Subject: [PATCH 1/7] Editorial update to basic C# tutorial --- docs/tutorials/csharp/basic_csharp.md | 80 +++++++++++++-------------- 1 file changed, 39 insertions(+), 41 deletions(-) diff --git a/docs/tutorials/csharp/basic_csharp.md b/docs/tutorials/csharp/basic_csharp.md index 6b8014b8e1e7d..c02d1d8f100ee 100644 --- a/docs/tutorials/csharp/basic_csharp.md +++ b/docs/tutorials/csharp/basic_csharp.md @@ -10,64 +10,59 @@ nav_order: 1 # C# Tutorial: Basic -Here is a simple tutorial for getting started with running inference on an existing ONNX model for a given input data. -The model is typically trained using any of the well-known training frameworks and then exported into the ONNX format. +Learn how to get started with inference with the C# API. Run an ONNX model for a given input data. The model is typically trained using any of the well-known training frameworks and then exported into the ONNX format. -Note, that the following classes `NamedOnnxValue`, `DisposableNamedOnnxValue`, `FixedBufferOnnxValue` are going -to be deprecated in the future. They are not recommended for new code. +## OrtValue API -The new `OrtValue` based API is the recommended approach. The `OrtValue` API generates less garbage and is more performant. -Some scenarios indicated 4x performance improvement over the previous API and significantly less garbage. -It provides uniform access to data via `ReadOnlySpan` and `Span` structures, regardless of its location, managed or unmanaged. +The new `OrtValue` based API is the recommended approach. The `OrtValue` API generates less garbage and is more performant. Some scenarios show 4x performance improvement over the previous API and significantly less garbage. -`DenseTensor` class can still be used for multi-dimensional access to the data since the new `Span` based API feature -only a 1-D index. However, some reported a slow performance when using `DenseTensor` class multi-dimensional access. -One can then create an OrtValue on top of the tensors data. +OrtValue is a universal container that can hold different ONNX types, such as tensors, maps, and sequences. It always existed in the onnxruntime library, but was not exposed in the C# API. + +The `OrtValue` based API provides uniform access to data via `ReadOnlySpan` and `Span` structures, regardless of its location, managed or unmanaged. + +Note, that the following classes `NamedOnnxValue`, `DisposableNamedOnnxValue`, `FixedBufferOnnxValue` will be deprecated in the future. They are not recommended for new code. + +## Data shape + +`DenseTensor` class can be used for multi-dimensional access to the data since the new `Span` based API features only a 1-D index. However, some reported a slow performance when using `DenseTensor` class multi-dimensional access. One can then create an OrtValue on top of the tensors data. `ShapeUtils` class provides some help to deal with multi-dimensional indices for OrtValues. -`OrtValue` based API provides direct native memory access in a type safe manner using `ReadOnlySpan` and `Span` stack bases structures. -OrtValue is a universal container that can hold different ONNX types, such as tensors, maps, and sequences. -It always existed in the onnxruntime library, but was not exposed in the C# API. +If output shapes are known, one can pre-allocate `OrtValue` on top of the managed or unmanaged allocations and supply those OrtValues to be used as outputs. Due to this fact, the need for `IOBinding` is greatly diminished. + + +## Data types + +`OrtValues` can be created directly on top of the managed `unmanaged` [struct based blittable types](https://learn.microsoft.com/en-us/dotnet/framework/interop/blittable-and-non-blittable-types) arrays. The onnxruntime C# API allows use of managed buffers for input or output. -As before, `OrtValues` can be created directly on top of the managed `unmanaged` (struct based blittable types) arrays. -Read MS documentation on `blittable` data types. onnxruntime C# API allows use of managed buffers for input or output. +String data is represented as UTF-16 string objects in C#. It will still need to be copied and converted to UTF-8 to the native memory. However, that conversion is now more optimized and is done in a single pass without intermediate byte arrays. -If output shapes are known, one can pre-allocate `OrtValue` on top of the managed or unmanaged allocations and supply -those OrtValues to be used as outputs. Due to this fact, the need for `IOBinding` is greatly diminished. +The same applies to string `OrtValue` tensors returned as outputs. Character based API now operates on `Span`,`ReadOnlySpan`, and `ReadOnlyMemory` objects. This adds flexibility to the API and allows to avoid unnecessary copies. -String data is represented as UTF-16 string objects in C#. It will still need to be copied and converted to UTF-8 to the native -memory. However, that conversion is now more optimized and is done in a single pass without intermediate byte arrays. -The same applies to string `OrtValue` tensors returned as outputs. Character based API now operates on `Span`, -`ReadOnlySpan`, and `ReadOnlyMemory` objects. This adds flexibility to the API and allows to avoid unnecessary copies. +## Data life-cycle Except some of the above deprecated API classes, nearly all of C# API classes are `IDisposable`. -Meaning they need to be disposed after use, otherwise you will get memory leaks. -Because OrtValues are used to hold tensor data, the sizes of the leaks can be huge. They are likely -to accumulate with each `Run` call, as each inference call requires input OrtValues and returns output OrtValues. -Do not hold your breath for finalizers which are not guaranteed to ever run, and if they do, they do it -when it is too late. +Meaning they need to be disposed after use, otherwise you will get memory leaks. Because OrtValues are used to hold tensor data, the sizes of the leaks can be huge. They are likely to accumulate with each `Run` call, as each inference call requires input OrtValues and returns output OrtValues. +Do not hold your breath for finalizers which are not guaranteed to ever run, and if they do, they do it when it is too late. -This includes `SessionOptions`, `RunOptions`, `InferenceSession`, `OrtValue`. Run() calls return `IDisposableCollection` -that allows to dispose all of the containing objects in one statement or `using`. This is because these objects -own some native resource, often a native object. +This includes `SessionOptions`, `RunOptions`, `InferenceSession`, `OrtValue`. Run() calls return `IDisposableCollection` that allows to dispose all of the containing objects in one statement or `using`. This is because these objects own native resources, often a native object. Not disposing `OrtValue` that was created on top of the managed buffer would result in that buffer pinned in memory indefinitely. Such a buffer can not be garbage collected or moved in memory. -`OrtValue`s that were created on top of the native onnxruntime memory should also be disposed of promptly. -Otherwise, the native memory will not be deallocated. OrtValues returned by `Run()` usually hold native memory. +`OrtValue`s that were created on top of the native onnxruntime memory should also be disposed of promptly. Otherwise, the native memory will not be deallocated. OrtValues returned by `Run()` usually hold native memory. GC can not operate on native memory or any other native resources. The `using` statement or a block is a convenient way to ensure that the objects are disposed. -`InferenceSession` can be a long lived object and a member of another class. It eventually must also need to be disposed. -This means, the containing class also would have to be made disposable to achieve this. +`InferenceSession` can be a long lived object and a member of another class. It eventually must also need to be disposed. This means, the containing class also would have to be made disposable to achieve this. OrtValue API also provides visitor like API to walk ONNX maps and sequences. This is a more efficient way to access Onnxruntime data. -To start scoring using the model, open a session using the `InferenceSession` class, passing in the file path to the model as a parameter. +## Code example to run a model + +To start scoring using the model, create a session using the `InferenceSession` class, passing in the file path to the model as a parameter. ```cs using var session = new InferenceSession("model.onnx"); @@ -76,7 +71,6 @@ using var session = new InferenceSession("model.onnx"); Once a session is created, you can execute queries using the `Run` method of the `InferenceSession` object. ```cs - float[] sourceData; // assume your data is loaded into a flat float array long[] dimensions; // and the dimensions of the input is stored here @@ -107,11 +101,11 @@ var outputData = output_0.GetTensorDataAsSpan(); var tensorTypeAndShape = output_0.GetTensorTypeAndShape(); ``` + You can still use `Tensor` class for data manipulation if you have existing code that does it. Then create `OrtValue` on top of Tensor buffer. ```cs - // Create and manipulate the data using tensor interface DenseTensor t1 = new DenseTensor(sourceData, dimensions); @@ -127,13 +121,10 @@ using var inputOrtValue = OrtValue.CreateTensorValueFromMemory(OrtMemoryInfo.Def ``` -Here is a way to populate a string tensor. Strings can not be mapped, and must be copy/converted to native memory. -To that end we pre-allocate a native tensor of empty strings with specified dimensions, and then -set individual strings by index. +Here is a way to populate a string tensor. Strings can not be mapped, and must be copy/converted to native memory. To that end we pre-allocate a native tensor of empty strings with specified dimensions, and then set individual strings by index. ```cs - string[] strs = { "Hello", "Ort", "World" }; long[] shape = { 1, 1, 3 }; var elementsNum = ShapeUtils.GetSizeForShape(shape); @@ -144,7 +135,14 @@ for (long i = 0; i < elementsNum; ++i) { strTensor.StringTensorSetElementAt(strs[i].AsSpan(), i); } - ``` +## More examples + +[Stable Diffusion](stable-diffusion-csharp.md) +[BERT NLP example](bert-nlp-csharp-console.md) +[Run on GPU](csharp-gpu.md) +[Yolov3](yolov3_object_detection_csharp.md) +[Faster CNN](fastercnn_csharp.md) +[Resnet 50](resnet50_csharp.md) From a5996433b77e316c8005934d60a0c320dda48e6f Mon Sep 17 00:00:00 2001 From: nakersha Date: Mon, 25 Sep 2023 19:08:12 -0700 Subject: [PATCH 2/7] Bullet examples list --- docs/tutorials/csharp/basic_csharp.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/tutorials/csharp/basic_csharp.md b/docs/tutorials/csharp/basic_csharp.md index c02d1d8f100ee..dc82fa353964f 100644 --- a/docs/tutorials/csharp/basic_csharp.md +++ b/docs/tutorials/csharp/basic_csharp.md @@ -139,10 +139,10 @@ for (long i = 0; i < elementsNum; ++i) ## More examples -[Stable Diffusion](stable-diffusion-csharp.md) -[BERT NLP example](bert-nlp-csharp-console.md) -[Run on GPU](csharp-gpu.md) -[Yolov3](yolov3_object_detection_csharp.md) -[Faster CNN](fastercnn_csharp.md) -[Resnet 50](resnet50_csharp.md) +* [Stable Diffusion](stable-diffusion-csharp.md) +* [BERT NLP example](bert-nlp-csharp-console.md) +* [Run on GPU](csharp-gpu.md) +* [Yolov3](yolov3_object_detection_csharp.md) +* [Faster CNN](fastercnn_csharp.md) +* [Resnet 50](resnet50_csharp.md) From a802f5d7972abda08ec782c6989f9644a798238f Mon Sep 17 00:00:00 2001 From: nakersha Date: Mon, 25 Sep 2023 19:12:39 -0700 Subject: [PATCH 3/7] Fix link --- docs/tutorials/csharp/basic_csharp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/csharp/basic_csharp.md b/docs/tutorials/csharp/basic_csharp.md index dc82fa353964f..4106d0a73aa06 100644 --- a/docs/tutorials/csharp/basic_csharp.md +++ b/docs/tutorials/csharp/basic_csharp.md @@ -140,7 +140,7 @@ for (long i = 0; i < elementsNum; ++i) ## More examples * [Stable Diffusion](stable-diffusion-csharp.md) -* [BERT NLP example](bert-nlp-csharp-console.md) +* [BERT NLP example](bert-nlp-csharp-console-app.md) * [Run on GPU](csharp-gpu.md) * [Yolov3](yolov3_object_detection_csharp.md) * [Faster CNN](fastercnn_csharp.md) From f5c60eb84c8281755358330f8881d9d7d7aed85f Mon Sep 17 00:00:00 2001 From: nakersha Date: Mon, 25 Sep 2023 19:21:40 -0700 Subject: [PATCH 4/7] Fix another link --- docs/tutorials/csharp/basic_csharp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/csharp/basic_csharp.md b/docs/tutorials/csharp/basic_csharp.md index 4106d0a73aa06..b12a4b97a06df 100644 --- a/docs/tutorials/csharp/basic_csharp.md +++ b/docs/tutorials/csharp/basic_csharp.md @@ -143,6 +143,6 @@ for (long i = 0; i < elementsNum; ++i) * [BERT NLP example](bert-nlp-csharp-console-app.md) * [Run on GPU](csharp-gpu.md) * [Yolov3](yolov3_object_detection_csharp.md) -* [Faster CNN](fastercnn_csharp.md) +* [Faster CNN](fasterrcnn_csharp.md) * [Resnet 50](resnet50_csharp.md) From d153b294f0f4e46631710c22b95524e6a76a3509 Mon Sep 17 00:00:00 2001 From: Dmitri Smirnov Date: Thu, 28 Sep 2023 09:47:29 -0700 Subject: [PATCH 5/7] Update docs/tutorials/csharp/basic_csharp.md --- docs/tutorials/csharp/basic_csharp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/csharp/basic_csharp.md b/docs/tutorials/csharp/basic_csharp.md index b12a4b97a06df..2fbe7b049590a 100644 --- a/docs/tutorials/csharp/basic_csharp.md +++ b/docs/tutorials/csharp/basic_csharp.md @@ -41,7 +41,7 @@ The same applies to string `OrtValue` tensors returned as outputs. Character bas ## Data life-cycle -Except some of the above deprecated API classes, nearly all of C# API classes are `IDisposable`. +Except for some of the above deprecated API classes, nearly all of C# API classes are `IDisposable`. Meaning they need to be disposed after use, otherwise you will get memory leaks. Because OrtValues are used to hold tensor data, the sizes of the leaks can be huge. They are likely to accumulate with each `Run` call, as each inference call requires input OrtValues and returns output OrtValues. Do not hold your breath for finalizers which are not guaranteed to ever run, and if they do, they do it when it is too late. From ec95083653ea14e42d3de4c4986755636f689b1d Mon Sep 17 00:00:00 2001 From: Dmitri Smirnov Date: Thu, 28 Sep 2023 09:47:38 -0700 Subject: [PATCH 6/7] Update docs/tutorials/csharp/basic_csharp.md --- docs/tutorials/csharp/basic_csharp.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/tutorials/csharp/basic_csharp.md b/docs/tutorials/csharp/basic_csharp.md index 2fbe7b049590a..1aa7a55f78e42 100644 --- a/docs/tutorials/csharp/basic_csharp.md +++ b/docs/tutorials/csharp/basic_csharp.md @@ -68,7 +68,7 @@ To start scoring using the model, create a session using the `InferenceSession` using var session = new InferenceSession("model.onnx"); ``` -Once a session is created, you can execute queries using the `Run` method of the `InferenceSession` object. +Once a session is created, you can run inference using the `Run` method of the `InferenceSession` object. ```cs float[] sourceData; // assume your data is loaded into a flat float array From 9d617827c26f778bb8dd0c7706e0a0695f94eaaf Mon Sep 17 00:00:00 2001 From: Nat Kershaw Date: Wed, 11 Oct 2023 17:14:35 -0700 Subject: [PATCH 7/7] Update after review --- docs/tutorials/csharp/basic_csharp.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/tutorials/csharp/basic_csharp.md b/docs/tutorials/csharp/basic_csharp.md index 1aa7a55f78e42..ac78ae4f35b1e 100644 --- a/docs/tutorials/csharp/basic_csharp.md +++ b/docs/tutorials/csharp/basic_csharp.md @@ -10,7 +10,7 @@ nav_order: 1 # C# Tutorial: Basic -Learn how to get started with inference with the C# API. Run an ONNX model for a given input data. The model is typically trained using any of the well-known training frameworks and then exported into the ONNX format. +Learn how to get started with inference with the C# API. ## OrtValue API @@ -55,10 +55,10 @@ that buffer pinned in memory indefinitely. Such a buffer can not be garbage coll GC can not operate on native memory or any other native resources. The `using` statement or a block is a convenient way to ensure that the objects are disposed. -`InferenceSession` can be a long lived object and a member of another class. It eventually must also need to be disposed. This means, the containing class also would have to be made disposable to achieve this. +`InferenceSession` can be a long lived object and a member of another class. It eventually must be disposed. This means, the containing class also would have to be made disposable to achieve this. OrtValue API also provides visitor like API to walk ONNX maps and sequences. -This is a more efficient way to access Onnxruntime data. +This is a more efficient way to access ONNX Runtime data. ## Code example to run a model @@ -140,7 +140,7 @@ for (long i = 0; i < elementsNum; ++i) ## More examples * [Stable Diffusion](stable-diffusion-csharp.md) -* [BERT NLP example](bert-nlp-csharp-console-app.md) +* [BERT NLP](bert-nlp-csharp-console-app.md) * [Run on GPU](csharp-gpu.md) * [Yolov3](yolov3_object_detection_csharp.md) * [Faster CNN](fasterrcnn_csharp.md)