[WIP]EP interface #18090

RandySheriffH · 2023-10-25T00:30:47Z

WIP...

include/onnxruntime/interface/provider/provider.h

include/onnxruntime/interface/graph/graph.h

include/onnxruntime/interface/kernel/kernel.h

include/onnxruntime/interface/common/data_types.h

…nxruntime into rashuai/EpInterface

github-advanced-security

lintrunner found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

samples/python/test_customEp2.py

+import numpy
+
+import onnxruntime
+from onnxruntime.capi import _pybind_state as C


samples/python/test_customEp2.py

+
+import onnxruntime
+from onnxruntime.capi import _pybind_state as C
+from onnxruntime.capi.onnxruntime_pybind11_state import RunOptions


samples/python/test_intreeEp.py

+import numpy
+
+import onnxruntime
+from onnxruntime.capi import _pybind_state as C


samples/python/test_intreeEp.py

+
+import onnxruntime
+from onnxruntime.capi import _pybind_state as C
+from onnxruntime.capi.onnxruntime_pybind11_state import RunOptions


samples/python/test_outTreeVariadic.py

+import numpy
+
+import onnxruntime
+from onnxruntime.capi import _pybind_state as C


samples/python/test_outTreeVariadic.py

+
+import onnxruntime
+from onnxruntime.capi import _pybind_state as C
+from onnxruntime.capi.onnxruntime_pybind11_state import RunOptions


jslhcl · 2023-10-26T20:57:45Z

include/onnxruntime/core/framework/custom_execution_provider.h

+
+/////////////////////////////////////////////////// Ep Interface ///////////////////////////////////////////////////
+
+struct IExecutionProvider {


I think there is some dup code here, especially this class, which I have moved to include/onnxruntime/interface/provider/provider.h and renamed as ExecutionProvider #Resolved

samples/python/test_customEp2.py

jslhcl · 2023-10-31T16:08:03Z

include/onnxruntime/interface/framework/kernel.h

+using ArgPtrs = std::vector<ArgPtr>;
+
+template <typename T>
+struct ITensor : public IArg {


Are you going to move this class to interface/framework/Tensor.h?

Shall we consolidate this tensor interface with the one in interface/graph/graph.h ?

jslhcl · 2023-10-31T16:14:08Z

include/onnxruntime/core/framework/op_kernel.h

 namespace onnxruntime {
 class OpKernelContext;
 }
 #endif

+#include "interface/framework/kernel.h"


No need to include kernel.h here? I don't see any class declared in kernel.h in the changes of this file

we probably shouldn't include interface/kernel here, as this file is for the internal kernel impl

jslhcl · 2023-10-31T16:14:53Z

include/onnxruntime/core/framework/op_kernel_info.h

@@ -9,6 +9,7 @@
 #include "core/framework/op_node_proto_helper.h"
 #include "core/graph/graph_viewer.h"
 #include "core/common/gsl.h"
+#include "interface/framework/kernel.h"


no need to include kernel.h here?

jslhcl · 2023-10-31T16:16:20Z

include/onnxruntime/interface/common/data_types.h

+
+namespace interface {
+
+enum TensorDataType {


I assume I need to move DataType enum in interface/graph/graph.h here?

let's use the same DataType definition for all the interface.

jslhcl · 2023-10-31T17:12:30Z

samples/customEP2/custom_ep2.cc

+struct Celu {
+  Celu(const interface::IKernelInfo&) {}
+  onnxruntime::Status Compute(onnxruntime::interface::TensorView<float>& input,
+                              onnxruntime::interface::Tensor<float>& output) {


Is this Compute function doing this algorithm of Celu?

max(0,x) + min(0,alpha*(exp(x/alpha)-1))

My understanding is that it is executing max(0, identity(input)), and alpha is not used

jslhcl · 2023-10-31T21:45:04Z

samples/customEP2/custom_ep2.cc

+struct Celu {
+  Celu(const interface::IKernelInfo&) {}
+  onnxruntime::Status Compute(onnxruntime::interface::TensorView<float>& input,
+                              onnxruntime::interface::Tensor<float>& output) {


Should we use the interface (interface::ITensor) or the implementation (interface::Tensor) as the parameter?

…optimizer_api_impl.h

RandySheriffH · 2023-11-13T22:54:31Z

include/onnxruntime/interface/framework/kernel.h

+using ArgPtr = std::unique_ptr<IArg>;
+using ArgPtrs = std::vector<ArgPtr>;
+
+struct ITensor : public IArg {


ITensor

Let internal tensor inherite from the ITensor.

…nxruntime into rashuai/EpInterface

jslhcl · 2023-11-18T14:35:55Z

onnxruntime/core/providers/openvino/openvino_execution_provider.cc

  openvino_ep::BackendManager::GetGlobalContext().onnx_model_path_name = std::string(onnx_path.begin(), onnx_path.end());
 #else
-  openvino_ep::BackendManager::GetGlobalContext().onnx_model_path_name = graph_viewer.ModelPath().ToPathString();
+  openvino_ep::BackendManager::GetGlobalContext().onnx_model_path_name = graph_viewer->ModelPath();


onnx_model_path_name is not used so remove the ModelPath() API

… existing OpenVINO EP factory

…nxruntime into rashuai/EpInterface

souptc · 2023-12-17T23:24:41Z

include/onnxruntime/interface/graph/graph.h

+#include <vector>
+#ifdef INTREE_EP
+#include "onnx/onnx_pb.h"
+#endif


can we find a way to keep the To-Model-Proto onnx API internally under core?

souptc · 2023-12-17T23:25:13Z

include/onnxruntime/interface/graph/graph.h

+/// <summary>
+/// Enum of DataTypes using standard ONNX values. Casting to/from int32_t is encouraged.
+/// </summary>
+enum class DataType : int32_t {


enum class DataType : int32_t

we should use the same enume in interface/data_types.h

souptc · 2023-12-17T23:25:39Z

include/onnxruntime/interface/graph/graph.h

+struct ModelProtoPtr {
+  const char* p;
+  size_t len;
+  size_t version;


size_t version;

what does this version means? onnx version? protobuf version? or ort version?

souptc · 2023-12-17T23:30:45Z

include/onnxruntime/interface/provider/provider.h

+//#include "core/session/onnxruntime_c_api.h"
+#include "core/framework/ortdevice.h"
+#include "core/framework/stream_handles.h"
+#include "core/framework/node_compute_info.h"


let's carefully review the included headers. if those are the headers we are going to include (with header-only implementation), let's think about how to structure the include folders to represent the depenency.

souptc · 2023-12-18T01:11:04Z

include/onnxruntime/core/common/common_no_ort_symbol.h

+#include <utility>
+#include <vector>
+
+#include "core/common/code_location.h"


code_location.h"

I think code location only used for dump call stack with windows debug build. do we need to expose it?

souptc · 2023-12-18T01:11:39Z

include/onnxruntime/core/framework/allocator_no_ort_symbol.h

+#pragma once
+
+#include "core/common/common_no_ort_symbol.h"
+#include "core/session/onnxruntime_c_api.h"


why we need CAPI?

souptc · 2023-12-18T01:13:01Z

include/onnxruntime/core/framework/allocator_no_ort_symbol.h

+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License.
+
+#pragma once


this file looks weird, it doesn't contain anything with Allocator, but named with allocator...

souptc · 2023-12-18T01:16:21Z

onnxruntime/core/providers/openvino/openvino_execution_provider.h

@@ -68,8 +69,8 @@ struct OpenVINOExecutionProviderInfo {
                                         bool enable_dynamic_shapes)
      : enable_vpu_fast_compile_(enable_vpu_fast_compile), device_id_(dev_id), num_of_threads_(num_of_threads), cache_dir_(cache_dir), num_streams_(num_streams), context_(context), enable_opencl_throttling_(enable_opencl_throttling), enable_dynamic_shapes_(enable_dynamic_shapes) {
    if (dev_type == "") {
-      LOGS_DEFAULT(INFO) << "[OpenVINO-EP]"
-                         << "No runtime device selection option provided.";
+      //LOGS_DEFAULT(INFO) << "[OpenVINO-EP]"


LOGS_DEFAULT(INFO)

we should add the logger interface

souptc · 2023-12-18T01:17:14Z

onnxruntime/core/providers/openvino/openvino_execution_provider.h

-  const void* GetExecutionHandle() const noexcept override {
-    return nullptr;
-  }
+  void RegisterKernels(interface::IKernelRegistry&) override {}


void RegisterKernels(interface::IKernelRegistry&) override {}

why need override RegisterKernels?

souptc · 2023-12-18T01:20:26Z

onnxruntime/core/providers/openvino/backend_utils.h

@@ -8,6 +8,7 @@
 #include "contexts.h"
 #include <iomanip>
 #include "ov_interface.h"
+#include "onnx/onnx_pb.h"


onnx_pb

we include onnx_pb.h for what?

souptc · 2023-12-18T01:23:10Z

onnxruntime/core/providers/openvino/backend_manager.cc

 #else
  ORT_UNUSED_PARAMETER(fused_node);
+  ORT_UNUSED_PARAMETER(subgraph);
+  return nullptr;


so for out of tree case, we just return nullptr? how could it work?

souptc · 2023-12-18T01:24:32Z

onnxruntime/core/providers/openvino/backends/basic_backend.cc

@@ -44,7 +43,7 @@ BasicBackend::BasicBackend(const ONNX_NAMESPACE::ModelProto& model_proto,
  if (IsDebugEnabled()) {
    std::string file_name = subgraph_context.subgraph_name + "_static.onnx";
    std::fstream outfile(file_name, std::ios::out | std::ios::trunc | std::ios::binary);
-    model_proto.SerializeToOstream(outfile);
+    model_proto.SerializeToOstream(&outfile);


outfile

why we need this change?

RandyShuai added 2 commits October 24, 2023 17:29

draft interface hierarchy

cdb9676

commence interface namespace

2ddea77

github-advanced-security bot found potential problems Oct 25, 2023

View reviewed changes

include/onnxruntime/interface/provider/provider.h Fixed Show fixed Hide fixed

include/onnxruntime/interface/graph/graph.h Fixed Show fixed Hide fixed

include/onnxruntime/interface/kernel/kernel.h Fixed Show fixed Hide fixed

add common folder

5fffc82

github-advanced-security bot found potential problems Oct 25, 2023

View reviewed changes

include/onnxruntime/interface/common/data_types.h Fixed Show fixed Hide fixed

jslhcl added 3 commits October 25, 2023 16:19

port code from PR 16718

ff689b1

Merge https://github.com/microsoft/onnxruntime into rashuai/EpInterface

6c92fc7

Merge branch 'rashuai/EpInterface' of https://github.com/microsoft/on…

4794e3d

…nxruntime into rashuai/EpInterface

github-advanced-security bot found potential problems Oct 25, 2023

View reviewed changes

jslhcl and others added 2 commits October 25, 2023 16:35

test_intreeEp.py works

0d4d3d3

windows fixes

858f259

jslhcl reviewed Oct 26, 2023

View reviewed changes

RandyShuai added 5 commits October 26, 2023 14:17

tune folder name

dbb9871

tune kernel hierarchy

a732a85

define ITensor

251a58b

add builder interface

e247fd8

postpone kernel creation

021405f

github-advanced-security bot found potential problems Oct 27, 2023

View reviewed changes

samples/python/test_customEp2.py Fixed Show fixed Hide fixed

RandyShuai added 2 commits October 30, 2023 15:50

fix to make customEp2 running

e433f5d

add testing model

4819293

jslhcl reviewed Oct 31, 2023

View reviewed changes

jslhcl added 3 commits November 1, 2023 17:24

customize GetCapability() and Compile() in custom_ep2 and delete ort_…

a2c247a

…optimizer_api_impl.h

test for GetCapability and Compile for custom_ep2

9b0f3b1

Subgraph API support

95e1fe6

RandySheriffH commented Nov 13, 2023

View reviewed changes

jslhcl and others added 6 commits November 15, 2023 10:21

use new graph API in openvino

e440c88

Merge branch 'rashuai/EpInterface' of https://github.com/microsoft/on…

60161f2

…nxruntime into rashuai/EpInterface

align build args between ort and out-tree custom ep dll

572d5f3

Merge branch 'rashuai/EpInterface' of https://github.com/microsoft/on…

8d0c804

…nxruntime into rashuai/EpInterface

fix wide char path for win

48dcc85

add tensorshape interface

7defe5d

jslhcl reviewed Nov 18, 2023

View reviewed changes

RandyShuai and others added 7 commits November 20, 2023 15:28

align tensor hierarchy

c04c1f2

support tensor sequence

839103a

move SubGraphDef into namespace interface and create OpenVINO EP from…

1b1c069

… existing OpenVINO EP factory

move openvino out-tree EP

5396642

update eigen checksum

539d679

openvino can be loaded and run as out-tree ep

5d450cb

Merge branch 'rashuai/EpInterface' of https://github.com/microsoft/on…

f8d6a70

…nxruntime into rashuai/EpInterface

souptc reviewed Dec 17, 2023

View reviewed changes

souptc reviewed Dec 18, 2023

View reviewed changes

jslhcl added 2 commits December 18, 2023 16:54

openVINO as out-tree EP works on Detection model

970a3f2

new member functions for EPv2

33a60f5

jslhcl force-pushed the rashuai/EpInterface branch from 01b3cee to 33a60f5 Compare December 29, 2023 18:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP]EP interface #18090

[WIP]EP interface #18090

RandySheriffH commented Oct 25, 2023

github-advanced-security bot left a comment

jslhcl Oct 26, 2023 •

edited

Loading

jslhcl Oct 31, 2023

jslhcl Oct 31, 2023

souptc Dec 18, 2023

jslhcl Oct 31, 2023

jslhcl Oct 31, 2023

souptc Nov 13, 2023

jslhcl Oct 31, 2023

jslhcl Oct 31, 2023

RandySheriffH Nov 13, 2023

jslhcl Nov 18, 2023

souptc Dec 17, 2023

souptc Dec 17, 2023

souptc Dec 17, 2023

souptc Dec 17, 2023

souptc Dec 18, 2023

souptc Dec 18, 2023

souptc Dec 18, 2023

souptc Dec 18, 2023

souptc Dec 18, 2023

souptc Dec 18, 2023

souptc Dec 18, 2023

souptc Dec 18, 2023


		/////////////////////////////////////////////////// Ep Interface ///////////////////////////////////////////////////

		struct IExecutionProvider {

[WIP]EP interface #18090

Are you sure you want to change the base?

[WIP]EP interface #18090

Conversation

RandySheriffH commented Oct 25, 2023

github-advanced-security bot left a comment

Choose a reason for hiding this comment

jslhcl Oct 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jslhcl Oct 26, 2023 •

edited

Loading