Add CANN EP #12416

wangxiyuan · 2022-08-02T02:39:50Z

Description: This PR adds Ascend CANN execution provider support.

Motivation and Context

Why is this change required? What problem does it solve?
As the info shown in the issue. CANN is the API layer for Ascend processor. Add CANN EP can allow user run onnx model on Ascend hardware via onnxruntime
The detail change:
1. Added CANN EP framework.
2. Added the basic operators to support ResNet and VGG model.
3. Added C/C++、Python API support
If it fixes an open issue, please link to the issue here.
Add Ascend backend support #11477

Author:
lijiawei [email protected]
wangxiyuan [email protected]

wangxiyuan · 2022-08-08T07:28:13Z

Any maintainer can apporve the CI workflows? Thanks

wangxiyuan · 2022-08-30T00:47:38Z

@skottmckay @edgchen1 Hi, could you please take a look at this PR? BIG thanks. I'm eager to know what should I do to push forward this code.

RyanUnderhill · 2022-08-30T01:06:55Z

include/onnxruntime/core/session/onnxruntime_c_api.h

@@ -3568,6 +3626,14 @@ ORT_API_STATUS(OrtSessionOptionsAppendExecutionProvider_CUDA, _In_ OrtSessionOpt
 */
 ORT_API_STATUS(OrtSessionOptionsAppendExecutionProvider_MIGraphX, _In_ OrtSessionOptions* options, int device_id);

+/*


This function should be deleted. The other provider ones like it were left for backwards compatibility. New provider functionality should be done only through the OrtApi as it is versioned.

RyanUnderhill · 2022-08-30T01:17:41Z

@skottmckay @edgchen1 Hi, could you please take a look at this PR? BIG thanks. I'm eager to know what should I do to push forward this code.

To other reviewers: I reviewed the shared provider library/API aspects of it, I didn't look in great detail into the provider code itself.

souptc · 2022-08-30T03:39:24Z

onnxruntime/core/providers/cann/gpu_data_transfer.cc

+}
+
+bool GPUDataTransfer::CanCopy(const OrtDevice& src_device, const OrtDevice& dst_device) const {
+  return src_device.Type() == OrtDevice::GPU || src_device.MemType() == OrtDevice::MemType::CANN_PINNED ||


GPU data transfer? and you want to reuse the OrtDevice::GPU?

souptc · 2022-08-30T03:48:05Z

onnxruntime/core/providers/cann/gpu_data_transfer.cc

+  auto& src_device = src.Location().device;
+  auto& dst_device = dst.Location().device;
+
+  if (dst_device.Type() == OrtDevice::GPU) {


are we using a tool similar to AMD's HIP to convert the ort cuda EP to this EP? THe code here looks exactly the same as cuda EP.

oh, thanks for point out this. Ascend is not GPU, it's a NPU device. We reused GPU type before which is not correct. There are three device type in onnxruntime (CPU GPU FPGA), how about add a new one called NPU? It's good for other NPU integration as well.

wangxiyuan · 2022-09-01T11:32:02Z

@souptc @RyanUnderhill @skottmckay @edgchen1 Hi, sorry to @ you again. The new commit is ready for review. All the GPU related code has been removed. Big thanks for your help.

aaronsm · 2022-09-01T22:51:55Z

onnxruntime/core/providers/cann/activation/activations.cc

@@ -0,0 +1,115 @@
+// Copyright (c) Huawei. All rights reserved.
+// Copyright (c) Huawei. All rights reserved.


duplicate copyright line

Got it. Done

FFFrog · 2022-09-05T06:54:39Z

@souptc @RyanUnderhill @skottmckay @edgchen1 Hi, sorry to @ you again. could you please take a look at this PR？

FFFrog · 2022-09-16T14:31:34Z

@souptc @RyanUnderhill @skottmckay @edgchen1 Hi, sorry to @ you again. could you please take a look at this PR？Thanks a lot.

jywu-msft · 2022-09-19T15:21:34Z

/azp run Linux CPU CI Pipeline

azure-pipelines · 2022-09-19T15:21:43Z

Azure Pipelines successfully started running 1 pipeline(s).

jywu-msft · 2022-09-19T19:51:31Z

include/onnxruntime/core/framework/allocator.h

@@ -37,6 +37,8 @@ namespace onnxruntime {
 constexpr const char* CPU = "Cpu";
 constexpr const char* CUDA = "Cuda";
 constexpr const char* CUDA_PINNED = "CudaPinned";
+constexpr const char* CANN = "Cann";
+constexpr const char* CANN_PINNED = "CannPinned";


just confirming that there really need for pinned memory type same as in cuda?

Ascend NPU provides aclrtMallocHost API to allocate pin memory on host, which may improve performance in most scenarios. So that there is a case that data need be transferred between CANN EP and CPU EP during model running. So it's required.

thanks for the explanation!

jywu-msft · 2022-09-20T15:32:46Z

/azp run Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

jywu-msft · 2022-09-20T15:32:56Z

/azp run Windows GPU TensorRT CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, onnxruntime-python-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

azure-pipelines · 2022-09-20T15:33:24Z

Azure Pipelines successfully started running 8 pipeline(s).

azure-pipelines · 2022-09-20T15:33:26Z

Azure Pipelines successfully started running 6 pipeline(s).

jywu-msft · 2022-09-20T16:14:57Z

include/onnxruntime/core/session/onnxruntime_c_api.h

@@ -413,6 +413,53 @@ typedef struct OrtCUDAProviderOptions {

 } OrtCUDAProviderOptions;

+/** \brief CANN Provider Options


Do you guys anticipate this struct changing much in the future? i.e. will you add/remove any fields here?
ORT maintains binary compatibility for C API (i.e. applications linked against libonnxruntime should expect to continue working with newer versions)
That means if you go with the struct approach for passing options, you would have to create a new version of the struct.
if that is okay, please proceed.
otherwise, consider using an opaque struct instead, so fields can be updated without impact to binary compatibility.
see comments at
https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/session/onnxruntime_c_api.h#L3232
and
https://github.com/microsoft/onnxruntime/blob/main/include/onnxruntime/core/session/onnxruntime_c_api.h#L2743
the caller does not directly access the struct.
instead, they call an interface to create the struct, update it, and release it when they're done.
e.g. CreateCannProviderOptions, UpdateCannProviderOptions, ReleaseCannOptions

In the future, this structure may change with hardware updates, so it's good to use the opaque structure instead. We'll submit a new commit later, thank you for your comments. Please wait more.

The new commit about opaque structure is ready for review.
please take a look at this, thanks a lot.

jywu-msft · 2022-09-21T14:50:25Z

onnxruntime/core/providers/cann/cann_execution_provider.h

+  std::shared_ptr<KernelRegistry> GetKernelRegistry() const override;
+  std::unique_ptr<onnxruntime::IDataTransfer> GetDataTransfer() const override;
+
+  std::vector<std::unique_ptr<ComputeCapability>> GetCapability(


FYI, you will need to make a small change to this due to #12791 which was merged yesterday

jywu-msft · 2022-09-21T14:53:52Z

A couple more follow-ups.

you'll need to fix a build break due to GetCapability() signature change caused by Update kernel matching logic: decouple from op schemas and remove kernel def hashes #12791
add some documentation. our docs are in the gh-pages branch. see https://github.com/microsoft/onnxruntime/tree/gh-pages/docs
you'll need to create a separate docs PR and make appropriate modifications to build/eps.md and add a execution-providers/CANN-ExecutionProvider.md
if you have some code examples/jupyter notebooks etc. they can go in https://github.com/microsoft/onnxruntime-inference-examples

more changes pending

wangxiyuan · 2022-09-22T02:34:01Z

A couple more follow-ups.

you'll need to fix a build break due to GetCapability() signature change caused by Update kernel matching logic: decouple from op schemas and remove kernel def hashes #12791

add some documentation. our docs are in the gh-pages branch. see https://github.com/microsoft/onnxruntime/tree/gh-pages/docs
you'll need to create a separate docs PR and make appropriate modifications to build/eps.md and add a execution-providers/CANN-ExecutionProvider.md

if you have some code examples/jupyter notebooks etc. they can go in https://github.com/microsoft/onnxruntime-inference-examples

Thanks for your suggestion. It's really helpful. We'll refresh the PR ASAP.

Docs、Example、More operator、 Performance Optimizing and so on are on our TODO list. We'll keep contributing in the future.

This PR adds Ascend CANN execution provider support. Detail: 1. Added CANN EP framework. 2. Added the basic operators to support resnet model. Co-Author: wangxiyuan <[email protected]>

Detail: 1. Adapt to the latest GetCapability() 2. Use the opaque structure instead and four functions are provided Co-Author: wangxiyuan <[email protected]>

jywu-msft · 2022-09-22T14:51:26Z

/azp run Linux CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux Nuphar CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, ONNX Runtime Web CI Pipeline, Windows CPU CI Pipeline, Windows GPU CI Pipeline

azure-pipelines · 2022-09-22T14:52:08Z

Azure Pipelines successfully started running 9 pipeline(s).

jywu-msft · 2022-09-22T19:36:27Z

/azp run Windows GPU TensorRT CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, onnxruntime-python-checks-ci-pipeline, orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-ortmodule-distributed

azure-pipelines · 2022-09-22T19:36:54Z

Azure Pipelines successfully started running 6 pipeline(s).

jywu-msft

Thanks for the contribution!

**Description**: This PR adds Ascend CANN execution provider support. **Motivation and Context** - Why is this change required? What problem does it solve? As the info shown in the issue. CANN is the API layer for Ascend processor. Add CANN EP can allow user run onnx model on Ascend hardware via onnxruntime The detail change: 1. Added CANN EP framework. 2. Added the basic operators to support ResNet and VGG model. 3. Added C/C++、Python API support - If it fixes an open issue, please link to the issue here. #11477 Author: lijiawei <[email protected]> wangxiyuan <[email protected]> Co-authored-by: FFrog <[email protected]>

justinchuby requested review from edgchen1 and skottmckay and removed request for edgchen1 August 24, 2022 01:10

RyanUnderhill reviewed Aug 30, 2022

View reviewed changes

souptc reviewed Aug 30, 2022

View reviewed changes

aaronsm reviewed Sep 1, 2022

View reviewed changes

jywu-msft reviewed Sep 19, 2022

View reviewed changes

jywu-msft reviewed Sep 20, 2022

View reviewed changes

jywu-msft reviewed Sep 21, 2022

View reviewed changes

jywu-msft previously approved these changes Sep 21, 2022

View reviewed changes

FFFrog added 2 commits September 22, 2022 16:07

Add CANN EP

3985272

This PR adds Ascend CANN execution provider support. Detail: 1. Added CANN EP framework. 2. Added the basic operators to support resnet model. Co-Author: wangxiyuan <[email protected]>

ENHANCE CANN EP

52e2d65

Detail: 1. Adapt to the latest GetCapability() 2. Use the opaque structure instead and four functions are provided Co-Author: wangxiyuan <[email protected]>

jywu-msft approved these changes Sep 22, 2022

View reviewed changes

jywu-msft merged commit 952c993 into microsoft:main Sep 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CANN EP #12416

Add CANN EP #12416

wangxiyuan commented Aug 2, 2022 •

edited

Loading

wangxiyuan commented Aug 8, 2022

wangxiyuan commented Aug 30, 2022

RyanUnderhill Aug 30, 2022

wangxiyuan Aug 31, 2022

RyanUnderhill commented Aug 30, 2022

souptc Aug 30, 2022

souptc Aug 30, 2022

wangxiyuan Aug 31, 2022

wangxiyuan commented Sep 1, 2022

aaronsm Sep 1, 2022

FFFrog Sep 2, 2022

FFFrog commented Sep 5, 2022

FFFrog commented Sep 16, 2022

jywu-msft commented Sep 19, 2022

azure-pipelines bot commented Sep 19, 2022

jywu-msft Sep 19, 2022

FFFrog Sep 20, 2022

jywu-msft Sep 20, 2022

jywu-msft commented Sep 20, 2022

jywu-msft commented Sep 20, 2022

azure-pipelines bot commented Sep 20, 2022

azure-pipelines bot commented Sep 20, 2022

jywu-msft Sep 20, 2022

FFFrog Sep 21, 2022

FFFrog Sep 22, 2022

jywu-msft Sep 21, 2022

FFFrog Sep 22, 2022

jywu-msft commented Sep 21, 2022 •

edited

Loading

wangxiyuan commented Sep 22, 2022

jywu-msft commented Sep 22, 2022

azure-pipelines bot commented Sep 22, 2022

jywu-msft commented Sep 22, 2022

azure-pipelines bot commented Sep 22, 2022

jywu-msft left a comment

		@@ -0,0 +1,115 @@
		// Copyright (c) Huawei. All rights reserved.
		// Copyright (c) Huawei. All rights reserved.

		@@ -413,6 +413,53 @@ typedef struct OrtCUDAProviderOptions {

		} OrtCUDAProviderOptions;

		/** \brief CANN Provider Options

Add CANN EP #12416

Add CANN EP #12416

Conversation

wangxiyuan commented Aug 2, 2022 • edited Loading

wangxiyuan commented Aug 8, 2022

wangxiyuan commented Aug 30, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RyanUnderhill commented Aug 30, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wangxiyuan commented Sep 1, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FFFrog commented Sep 5, 2022

FFFrog commented Sep 16, 2022

jywu-msft commented Sep 19, 2022

azure-pipelines bot commented Sep 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jywu-msft commented Sep 20, 2022

jywu-msft commented Sep 20, 2022

azure-pipelines bot commented Sep 20, 2022

azure-pipelines bot commented Sep 20, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jywu-msft commented Sep 21, 2022 • edited Loading

wangxiyuan commented Sep 22, 2022

jywu-msft commented Sep 22, 2022

azure-pipelines bot commented Sep 22, 2022

jywu-msft commented Sep 22, 2022

azure-pipelines bot commented Sep 22, 2022

jywu-msft left a comment

Choose a reason for hiding this comment

wangxiyuan commented Aug 2, 2022 •

edited

Loading

jywu-msft commented Sep 21, 2022 •

edited

Loading