[WIP] Out-Tree EP feature #21450

jslhcl · 2024-07-23T00:36:08Z

Description

Out-Tree EP feature.

Motivation and Context

Decouple ExecutionProvider from ONNXRuntime and make it a standalone class which will benefit 3rd party EP authors to write their own EP.
Binary compatibility is required for this feature.

samples/outTreeEp/out_tree_ep.cc

samples/c_test/test.cpp

samples/outTreeEp/out_tree_ep.h

@@ -0,0 +1,35 @@
+#pragma once


onnxruntime/core/framework/provider_adapter.h

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/c_test/test.cpp

onnxruntime/core/session/inference_session.cc

jslhcl · 2024-07-23T22:11:31Z

samples/outTreeEp/out_tree_ep.cc

+#ifdef __cplusplus
+extern "C" {
+#endif
+OrtExecutionProviderFactory* RegisterCustomEp() {


return Status instead #Resolved

Do we have to do this? This function will new a factory object by invoking its constructor which has no return type

… EP as graph API is not exported by ORT. Need to put these graph API into ortapi structure

samples/outTreeEp/out_tree_ep.cc

samples/c_test/test.cpp

…roviderAdapter::Compile()

adrianlizarraga · 2024-07-30T21:24:49Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+} OrtMetaDef;
+
+typedef struct OrtIndexedSubGraph {
+  OrtMetaDef* meta_def; // TODO(leca): how to define a nested structure pointer?


Does this have to be a pointer to an OrtMetaDef? It may be simpler if this meta_def is contained by value instead. #Resolved

It looks we will check the pointer is null or not to distinguish between single node mode and fused node mode (See base class IExecutionProvider::GetCapability() which does not set this pointer and TryAssignSingleNode() which will check this pointer)

adrianlizarraga · 2024-07-30T21:34:37Z

samples/outTreeEp/out_tree_ep.cc

+
+OutTreeEp::OutTreeEp(const char* ep_type, const OutTreeEpInfo& ep_info) : info(ep_info) {
+    type = ep_type;
+    OrtExecutionProvider::GetCapability = [](const OrtExecutionProvider* this_, const OrtGraphViewer* graph, size_t* cnt, OrtIndexedSubGraph*** indexed_sub_graph) {


If I'm understanding correctly, the type of the OrtIndexedSubGraph*** indexed_sub_graph parameter is essentially asking the EP to fill out an array of pointers to OrtIndexedSubGraph objects.

Would it be simpler to change this to OrtIndexedSubgraph** indexed_sub_graph so that the EP fills out an array of OrtIndexedSubGraph objects directly? Each OrtIndexedSubgraph struct is a simple POD that can be created on the stack and copied around. It seems like it would result in less pointer tracking. #Resolved

Also, who is responsible for freeing this memory and when? If the EP allocates an array, then the EP should free it. The currently example leaks the allocations.

Edit: one possibility is to have onnxruntime call a new EP function (e.g., ReleaseOrtIndexedSubGraph()) so the the EP can free the memory. onnxruntime would call this once it is done using the indexed_sub_graph.

The problem is that we don't know how many OrtIndexedSubGraph would be before we call GetCapability() function. I will fix the leak issue in the coming commits

samples/outTreeEp/out_tree_ep.cc

onnxruntime/core/framework/provider_factory_adapter.h

@@ -0,0 +1,21 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/outTreeEp_kernel/kernel_ep.cc

@@ -0,0 +1,89 @@
+#include "kernel_ep.h"


samples/outTreeEp_kernel/kernel_ep.h

@@ -0,0 +1,36 @@
+#pragma once


include/onnxruntime/core/framework/ort_type_constraints.h

@@ -0,0 +1,15 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


onnxruntime/core/framework/ort_type_constraints.cc

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_model_wrapper.cc

@@ -0,0 +1,627 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_model_wrapper.h

@@ -0,0 +1,285 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_quant_params_wrapper.cc

@@ -0,0 +1,266 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_quant_params_wrapper.h

@@ -0,0 +1,147 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_utils.cc

@@ -0,0 +1,557 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/builder/qnn_utils.h

@@ -0,0 +1,110 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.


samples/qnnEp/qnn_execution_provider.cc

@@ -0,0 +1,54 @@
+#include "qnn_execution_provider.h"


samples/qnnEp/qnn_execution_provider.h

@@ -0,0 +1,33 @@
+#pragma once


adrianlizarraga · 2024-12-03T21:50:53Z

samples/tensorRTEp/tensorrt_execution_provider.cc

+OrtExecutionProviderFactory* RegisterCustomEp() {
+    std::unique_ptr<onnxruntime::TensorrtExecutionProviderFactory> ret = std::make_unique<onnxruntime::TensorrtExecutionProviderFactory>();
+    return ret.release();
+}


At first glance this seems to be a memory leak of the returned object. However, after some digging, it looks like ORT is freeing the memory when the EP library is unloaded. This is still an issue. Preferably, memory should not be allocated on one side of the API boundary and then deleted on the other. ORT does not know what allocator the EP library used to allocate the object, so it can't be expected to know exactly how to delete it.

This may not need a heap allocation at all. The RegisterCustomEp function could accept a pointer to a OrtExecutionProviderFactory object that was allocated by ORT, and then it could just fill out the members. Then ORT can decide how/when to free the object. The EP library only worries about filling out the function callbacks.

OrtStatus* RegisterCustomEp(OrtExecutionProviderFactory* ep_factory) { ep_factory->CreateExecutionProvider = [](/*params*/) { /* impl to create EP instance */ }; return nullptr; }

Also, I really think that this function should return an OrtStatus* so that the EP library can indicate an error with a descriptive error.

After more investigation, it seems like this implementation directly returns a OrtExecutionProviderFactory* because you want to be able to inherit from OrtExecutionProviderFactory and use polymorphism for calls to GetCapability and Compile.

Here's an alternative approach that keeps allocations on the EP side of the API boundary, returns OrtStatus* where appropriate, and still allows you to create a custom TensorRTEP class.

// tensorrt_execution_provider.cc (EP Plugin Library). #include <onnxruntime_c_api_ep.h> struct TensorRTEpFactory { // … std::string ep_name; OrtExecutionProviderFactory ort_ep_factory = {}; }; class TensorRTEp { public: static std::unique_ptr<TensorRTEp> CreateInstance(const std::string& ep_name, size_t num_ep_options, const char* const* ep_option_keys, const char* const* ep_option_vals, /*output*/ std::string& err_msg) { if (/* EP options invalid */) { err_msg = "Invalid EP options"; return nullptr; } // Create internal EP object/state and fill out ORT callbacks (only GetCapability and Compile are shown here). std::unique_ptr<TensorRTEp> ep = std::make_unique<TensorRTEp>(ep_name, /*other args*/); ep->ort_ep.GetCapability = [](const OrtExecutionProvider* ort_ep, /*params*/) { TensorRTEp* this_ = reinterpret_cast<TensorRTEp*>(ort_ep->state); /* impl */ }; ep->ort_ep.Compile = [](const OrtExecutionProvider* ort_ep, /*params*/) {/* impl */}; ep->ort_ep.state = ep->get(); return ep; } OrtExecutionProvider* GetOrtExecutionProvider() { return &this->ort_ep; } private: TensorRTEp(const std::string& ep_name, /* other params */) { /* … */ } std::string ep_name; OrtExecutionProvider ort_ep = {}; // Other state/methods here … }; static void DestroyOnDllUnload(std::unique_prt<TensorRTEpFactory>&& ep_factory) { static std::vector<std::unique_prt<TensorRTEp>> ep_factories; static std::mutex m_; std::lock_guard<std::mutex> lock(m_); ep_factories.push_back(std::move(ep_factory)); } static void DestroyOnDllUnload(std::unique<TensorRTEp> ep_instance) {/*similar implementation as above*/} #ifdef __cplusplus extern "C" { #endif // DLL ENTRY POINT OrtStatus* ORT_API_CALL RegisterCustomEp(const OrtExecutionProviderFactory** ort_ep_factory, const char* ep_name) { auto ep_factory = std::make_unique<TensorRTEpFactory>(ep_name, OrtExecutionProviderFactory{}); ep_factory->ort_ep_factory.CreateExecutionProvider = CreateExecutionProviderCallback; ep_factory->ort_ep_factory.state = ep_factor->get(); // Can optionally store custom state. *ort_ep_factory = &ep_factory->ort_ep_factory; // Update output parameter to point to our EP factory callbacks. DestroyOnDllUnload(std::move(ep_factory)); return nullptr; } #ifdef __cplusplus } #endif OrtStatus* ORT_API_CALL CreateExecutionProviderCallback(const OrtExectutionProviderFactory* ort_ep_factory, const char* const* option_keys, const char* const* option_vals, size_t num_options, const OrtExecutionProvider** ort_ep) { TensorRTEpFactory* ep_factory = reinterpret_cast<TensorRTEpFactory*>(ort_ep_factory->state); std::string err_msg; auto ep = TensorRTEp::CreateInstance(ep_factory->ep_name, option_keys, option_vals, num_options, err_msg); if (!err_msg.empty()) { /* return OrtStatus with error message */ } *ort_ep = ep->GetOrtExecutionProvider(); // Update output parameter to point to our EP callbacks. DestroyOnDllUnload(std::move(ep)); return nullptr; }

The above requires modifying OrtExecutionProviderFactory and OrtExecutionProvider to store a void* pointer to custom state.

struct OrtExecutionProviderFactory { // Same void* state; // State set by the EP plugin library. }; struct OrtExecutionProvider { // Same void* state; // State set by the EP plugin library. };

There's another alternative. In the above example, ORT receives pointers to OrtExecutionProviderFactory and OrtExecutionProvider structs that were allocated within the EP plugin library. An alternative is to just copy the structs since they’re bags of function pointers (plain-old-data). There is really no need to give ORT a pointer to these structs when we can just copy them so that ORT has its own versions. This would be achievable with the following changes:

#include <onnxruntime_c_api_ep.h> // Same as previous example #ifdef __cplusplus extern "C" { #endif // DLL ENTRY POINT OrtStatus* ORT_API_CALL RegisterCustomEp(/*out*/ OrtExecutionProviderFactory* ort_ep_factory, const char* ep_name) { auto ep_factory = std::make_unique<TensorRTEpFactory>(ep_name, OrtExecutionProviderFactory{}); ep_factory->ort_ep_factory.CreateExecutionProvider = CreateExecutionProviderCallback; ep_factory->ort_ep_factory.state = ep_factor->get(); // Can optionally store custom state. // This is a struct copy. This updates the output parameter to a copy of our EP factory callbacks. *ort_ep_factory = ep_factory->ort_ep_factory; DestroyOnDllUnload(std::move(ep_factory)); return nullptr; } #ifdef __cplusplus } #endif OrtStatus* ORT_API_CALL CreateExecutionProviderCallback(const OrtExectutionProviderFactory* ort_ep_factory, const char* const* option_keys, const char* const* option_vals, size_t num_options, /*output*/ OrtExecutionProvider* ort_ep) { TensorRTEpFactory* ep_factory = reinterpret_cast<TensorRTEpFactory*>(ort_ep_factory->state); std::string err_msg; auto ep = TensorRTEp::CreateInstance(ep_factory->ep_name, option_keys, option_vals, num_options, err_msg); if (!err_msg.empty()) { /* return OrtStatus with error message */ } *ort_ep = *(ep->GetOrtExecutionProvider()); // Struct copy. This updates the out param to a copy of our EP callbacks. DestroyOnDllUnload(std::move(ep)); return nullptr; }

chilo-ms · 2024-12-04T00:27:29Z

onnxruntime/core/session/onnxruntime_c_api_ep.cc

+    *ep_context_graph = reinterpret_cast<OrtGraphViewer*>(graph_build_viewer.release());
+  } else {
+    ::onnxruntime::GraphViewer* content_graph_viewer = reinterpret_cast<::onnxruntime::GraphViewer*>(*ep_context_graph);
+    graph_build = const_cast<::onnxruntime::Graph*>(&(content_graph_viewer->GetGraph()));


It seems using const_cast here might result in undefined behavior:
"Modifying a const object through a non-const access path and referring to a volatile object through a non-volatile glvalue results in undefined behavior." - https://en.cppreference.com/w/cpp/language/const_cast
, because later this function calls graph_build->GetOrCreateNodeArg which might modify the "constant" graph instance. #Resolved

Thanks for the comment. Rolled back the change

adrianlizarraga · 2024-12-04T00:45:15Z

samples/tensorRTEp/tensorrt_execution_provider.cc

+
+namespace onnxruntime {
+
+static const std::string tensorrtEp = "tensorrtEp";


Is this the same EP name set by the user application?

// Load the EP library and register EP-creation functions with ORT status = api->RegisterPluginExecutionProviderLibrary(L"trt_ep_lib.dll", env, "trt_ep");

Seems like this should be initialized with the name set by the user, right? Otherwise, we can have a conflict.

adrianlizarraga · 2024-12-04T01:02:32Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+  void* extra_param_for_create_state_func;
+  void* extra_param_for_compute_func;


I think this may be able to be replaced with a single void* state;. See https://github.com/microsoft/onnxruntime/pull/21450/files#r1868553280

adrianlizarraga · 2024-12-04T01:36:05Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+   *
+   * \since Version 1.xx.
+   */
+  ORT_API2_STATUS(SessionOptionsAppendPluginExecutionProvider, _In_ OrtSessionOptions* options, _In_ const char* ep_name, _In_ OrtEnv* env,


Can this can be removed? We may be able to use the existing C API function called SessionOptionsAppendExecutionProvider. The existing C API does not take an OrtEnv parameter, but we can just get the default OrtEnv since there is only one per process.

yuslepukhin · 2024-12-04T18:15:44Z

General comment, please, re-format so the lines do not exceed 120 chars limit.

yuslepukhin · 2024-12-04T18:16:57Z

include/onnxruntime/core/framework/execution_provider.h

@@ -325,6 +327,8 @@ class IExecutionProvider {
    return InlinedVector<const Node*>();
  }

+  bool IsBuiltInEp() const { return builtin_ep_; }


const

noexcept

yuslepukhin · 2024-12-04T18:18:15Z

include/onnxruntime/core/framework/ort_type_constraints.h

+#include <string>
+#include <set>
+
+struct OrtTypeConstraints {


truct OrtTypeConstraints {

Please, add documentation to all of the classes

yuslepukhin · 2024-12-04T18:19:40Z

include/onnxruntime/core/session/environment.h

 #include "core/common/common.h"
 #include "core/common/status.h"
 #include "core/platform/threadpool.h"
 #include "core/common/logging/logging.h"
 #include "core/framework/allocator.h"
+#include "core/session/onnxruntime_c_api_ep.h"



Can we forward declare the types and avoid public header inclusion in the header?

yuslepukhin · 2024-12-04T18:20:09Z

include/onnxruntime/core/session/environment.h

@@ -5,11 +5,13 @@

 #include <atomic>
 #include <memory>
+#include <unordered_set>


<unordered_set>

Please, use inlined containers

yuslepukhin · 2024-12-04T18:26:12Z

include/onnxruntime/core/session/onnxruntime_c_api.h

+   *
+   * \param[in] kernel_registry Opaque pointer of KernelRegistry object
+   * \param[in] custom_op Custom Op where the kernel compute function is defined
+   * \param[in] type_constraints


param[in] type_constraints

Given the fact that type_constraing pointer is not-const, does it mean it gets modified or the ownership is taken by the registry? Please, add this to the documentation.

yuslepukhin · 2024-12-04T18:27:31Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+  OrtExecutionProvider*(ORT_API_CALL* CreateExecutionProvider)(OrtExecutionProviderFactory* this_, const char* const* ep_option_keys, const char* const* ep_option_values, size_t option_size);
+} OrtExecutionProviderFactory;
+
+struct OrtGraphApi {


This is really a read-only graph viewer.
Suggest to name it appropriatly
Perhaps, the name should reflect the fact that this API is specifically for EP interaction.

yuslepukhin · 2024-12-04T18:29:50Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+  ONNXTensorElementDataType data_type;
+  const char* data;
+  size_t data_len;
+} OrtTensorRef;


This seems to be a repeat of the existing API such as TensorTypeAndShape. Can we re-use that part?

yuslepukhin · 2024-12-04T18:33:57Z

include/onnxruntime/core/session/onnxruntime_c_api.h

@@ -4665,7 +4671,128 @@ struct OrtApi {
                  _In_reads_(num_external_initializer_files) char* const* external_initializer_file_buffer_array,
                  _In_reads_(num_external_initializer_files) const size_t* external_initializer_file_lengths,
                  size_t num_external_initializer_files);
-};
+
+  /** \brief Create OrtDevice object.


/** \brief Create OrtDevice object.

Suggestion is to create a table that is separate from
the main API. Reasons:
Most of the clients do not need that code.
But it does affect language bindings.
For example, we now need to pad C# imported API structure, although it is unlikely we would ever need that in the C#, but if we do we can add that separate.

yuslepukhin · 2024-12-04T18:39:29Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+ORT_API2_STATUS(OrtGraph_IsConstantInitializer, const OrtGraphViewer* graph, const char* name, bool check_outer_scope, _Out_ bool* out);
+
+/** \brief Get the NodeIndex values of the graph nodes sorted in topological order
+ *


It looks like the output ptr data is const, please, specify that it is not to be freed by the client.

yuslepukhin · 2024-12-04T18:45:52Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+  size_t node_index_len;
+} OrtIndexedSubGraph;
+
+typedef struct OrtComputeContext {


Suggest adding a constructor for _cplusplus

yuslepukhin · 2024-12-04T18:46:48Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+/** \brief Gets the path of the owning model if any
+ *
+ * \param[in] graph The graph to query
+ * \param[out] model_path The path of the owning model if any


model_path The path of the owning model if any

Should this be an ORTTCHAR?

yuslepukhin · 2024-12-04T18:48:08Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+ * \param[out] out True if the graph is a subgraph
+ *
+ */
+ORT_API2_STATUS(OrtGraph_IsSubgraph, const OrtGraphViewer* graph, _Out_ bool* out);


ORT_API2_STATUS

Suggest this returns a bool

yuslepukhin · 2024-12-04T19:02:10Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+ * \remarks The caller is responsible for freeing the byte array using OrtFreeMem.
+ *
+ */
+ORT_API2_STATUS(OrtGraph_SerializeToArray, const OrtGraphViewer* graph, _Out_ void** data, _Out_ size_t* data_size);  // TODO(leca): review and discuss


oid** data

suggest this to be an array of bytes such as unsigned char

yuslepukhin · 2024-12-04T19:18:39Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+ * \param[in] onnx_model_path The file path to save to
+ *
+ */
+ORT_API2_STATUS(OrtGraph_DumpOnnxModel, const OrtGraphViewer* graph, const char* onnx_model_path);


const char* onnx_model_path)

ORTTCHAR

yuslepukhin · 2024-12-04T19:25:00Z

onnxruntime/core/framework/session_state.cc

-  auto& eps = GetExecutionProviders();
-  for (auto& ep : eps) {
-    ep->RegisterStreamHandlers(GetStreamHandleRegistryInstance(), *allocators_);
+  std::string register_resource_after = "";


= "";

redundant

yuslepukhin · 2024-12-04T19:25:11Z

onnxruntime/core/framework/session_state.cc

+  std::string register_resource_after = "";
+  IExecutionProvider* plugin_ep = nullptr;
+  for (auto& ep : execution_providers_) {
+    if (register_resource_after == "") {


== ""

.empty()

yuslepukhin · 2024-12-04T19:26:58Z

onnxruntime/core/framework/ort_type_constraints.cc

+    std::unordered_map<std::string, std::set<ONNXTensorElementDataType>>::iterator iter = type_constraints_.find(type_symbol);
+    if (iter == type_constraints_.end()) {
+        std::set<ONNXTensorElementDataType> types{type};
+        type_constraints_[type_symbol] = types;


types;

std::move()

…pCtxGraph

- Modify CMakeLists.txt for TRT EP plugin - Add "-l" for specifying EP plugin lib path for onnxruntime_perf_test

adrianlizarraga

Some memory/lifetime comments for OrtIndexedSubGraph in GetCapability()

adrianlizarraga · 2024-12-18T23:37:15Z

include/onnxruntime/core/session/onnxruntime_c_api_ep.h

+} OrtMetaDef;
+
+typedef struct OrtIndexedSubGraph {
+  OrtMetaDef* meta_def; // TODO(leca): how to define a nested structure pointer?


nit: I don't think it's necessary for this to be a separate memory allocation. I see that it was based on our internal IndexedSubGraph implementation, but in my view reducing the number of memory allocation makes things simpler. Perhaps meta_def can be stored inline and can add a boolean to indicate if the meta_def is valid.

typedef struct OrtMetaDef { bool is_valid; // ... } OrtMetaDef; typedef struct OrtIndexedSubGraph { OrtMetaDef meta_def; // ... } OrtIndexedSubGraph;

adrianlizarraga · 2024-12-19T00:00:42Z

onnxruntime/core/framework/provider_adapter.h

+  virtual std::vector<std::unique_ptr<ComputeCapability>> GetCapability(const GraphViewer& graph_viewer, const IKernelLookup& kernel_lookup) const override {
+    size_t cnt = 0;
+    OrtIndexedSubGraph** indexed_subgraph = nullptr;
+    if (ep_impl_->GetCapability) ep_impl_->GetCapability(ep_impl_, reinterpret_cast<const OrtGraphViewer*>(&graph_viewer), &cnt, &indexed_subgraph);


This is currently passing an OrtIndexedSubGraph*** to the EP's GetCapability() function and asking the EP to allocate an array of pointers to OrtIndexSubGraph objects. Because the EP is allocating the memory, we currently have a separate API to allow the EP to delete this memory.

I wonder if it would be simpler to allow ORT to pass in an OrtAllocator to the EP. The EP would use this ORT-owned allocator to allocate memory for the array. This woud remove the need for a separate C API to clean up the memory. Also, it would allow the parameter to be a OrtIndexedSubGraph** instead of OrtIndexedSubGraph***.

jslhcl added 2 commits July 17, 2024 20:39

opaque pointer for graph

0e6a80c

ORT C API RegisterOrtExecutionProviderLibrary work

c30a639

jslhcl requested review from souptc, adrianlizarraga, jywu-msft and chilo-ms July 23, 2024 00:36

github-advanced-security bot found potential problems Jul 23, 2024

View reviewed changes

ORT C-API SessionOptionsAppendOrtExecutionProvider work

7bfe57e

github-advanced-security bot found potential problems Jul 23, 2024

View reviewed changes

jslhcl commented Jul 23, 2024

View reviewed changes

onnxruntime/core/session/inference_session.cc Outdated Show resolved Hide resolved

jslhcl commented Jul 23, 2024

View reviewed changes

Test Relu with compile based EP, build work, runtime error of loading…

8e7d28d

… EP as graph API is not exported by ORT. Need to put these graph API into ortapi structure

github-advanced-security bot found potential problems Jul 26, 2024

View reviewed changes

samples/outTreeEp/out_tree_ep.cc Fixed Show fixed Hide fixed

samples/c_test/test.cpp Fixed Show fixed Hide fixed

jslhcl added 2 commits July 29, 2024 17:49

prototype works with hardcode node_compute_info's index in ExecutionP…

808bfc3

…roviderAdapter::Compile()

prototype works without hardcode

49e396c

adrianlizarraga reviewed Jul 30, 2024

View reviewed changes

samples/outTreeEp/out_tree_ep.cc Outdated Show resolved Hide resolved

jslhcl added 2 commits July 31, 2024 20:38

fix comments for Compile function

e790105

add provider_factory_adapter.h

92f529d

github-advanced-security bot found potential problems Aug 1, 2024

View reviewed changes

onnxruntime/core/framework/provider_factory_adapter.h

@@ -0,0 +1,21 @@

// Copyright (c) Microsoft Corporation. All rights reserved.

Check warning

Code scanning / lintrunner

CLANGFORMAT/format Warning

See https://clang.llvm.org/docs/ClangFormat.html.
Run lintrunner -a to apply this patch.

jslhcl added 2 commits August 5, 2024 19:27

fix crash after introducing kernel based EP

3d83ed1

kernel based EP work with type constraint check commented out

e29499a

github-advanced-security bot found potential problems Aug 6, 2024

View reviewed changes

add kernel type constraints from out tree EP

f3678c4

github-advanced-security bot found potential problems Aug 7, 2024

View reviewed changes

jslhcl added 2 commits August 7, 2024 16:27

add API ReleaseOrtTypeConstraints

ac5ae0a

introduce qnn ep

0cc78e8

github-advanced-security bot found potential problems Aug 12, 2024

View reviewed changes

more graph/node C API

740a687

adrianlizarraga reviewed Dec 3, 2024

View reviewed changes

initial commit for Graph C++ API

c8ddc73

chilo-ms reviewed Dec 4, 2024

View reviewed changes

adrianlizarraga reviewed Dec 4, 2024

View reviewed changes

yuslepukhin reviewed Dec 4, 2024

View reviewed changes

jslhcl and others added 5 commits December 4, 2024 19:42

Fix Chi's comment and rollback the change on OrtGraph_CreateOrUpdateE…

e6be85e

…pCtxGraph

Add c++ wrapper for plugin ep api (#23045)

ce76175

refine ep plugin c++ wrapper (#23050)

fefbe27

[TRT EP Plugin] Fix issues of building on Windows (#23099)

ce6630c

- Modify CMakeLists.txt for TRT EP plugin - Add "-l" for specifying EP plugin lib path for onnxruntime_perf_test

refine ep plugin c++ wrapper (#23131)

dc6674b

adrianlizarraga reviewed Dec 19, 2024

View reviewed changes

		@@ -0,0 +1,14 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,21 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,15 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,627 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,285 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,266 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,147 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,557 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.

		@@ -0,0 +1,110 @@
		// Copyright (c) Microsoft Corporation. All rights reserved.


		namespace onnxruntime {

		static const std::string tensorrtEp = "tensorrtEp";

		void* extra_param_for_create_state_func;
		void* extra_param_for_compute_func;

[WIP] Out-Tree EP feature #21450

Are you sure you want to change the base?

[WIP] Out-Tree EP feature #21450

Conversation

jslhcl commented Jul 23, 2024 • edited Loading

Description

Motivation and Context

jslhcl Jul 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrianlizarraga Jul 30, 2024 • edited by jslhcl Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrianlizarraga Jul 30, 2024 • edited by jslhcl Loading

Choose a reason for hiding this comment

adrianlizarraga Jul 30, 2024 • edited Loading

Choose a reason for hiding this comment

jslhcl Jul 31, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrianlizarraga Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

adrianlizarraga Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

chilo-ms Dec 4, 2024 • edited by jslhcl Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrianlizarraga Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuslepukhin commented Dec 4, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuslepukhin Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuslepukhin Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuslepukhin Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yuslepukhin Dec 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adrianlizarraga left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jslhcl commented Jul 23, 2024 •

edited

Loading

jslhcl Jul 23, 2024 •

edited

Loading

adrianlizarraga Jul 30, 2024 •

edited by jslhcl

Loading

adrianlizarraga Jul 30, 2024 •

edited by jslhcl

Loading

adrianlizarraga Jul 30, 2024 •

edited

Loading

jslhcl Jul 31, 2024 •

edited

Loading

adrianlizarraga Dec 4, 2024 •

edited

Loading

adrianlizarraga Dec 4, 2024 •

edited

Loading

chilo-ms Dec 4, 2024 •

edited by jslhcl

Loading

adrianlizarraga Dec 4, 2024 •

edited

Loading

yuslepukhin Dec 4, 2024 •

edited

Loading

yuslepukhin Dec 4, 2024 •

edited

Loading

yuslepukhin Dec 4, 2024 •

edited

Loading

yuslepukhin Dec 4, 2024 •

edited

Loading