Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WebNN EP] Add cache for MLContexts in the WebNNBackend #22510

Merged
merged 3 commits into from
Oct 30, 2024
Merged

[WebNN EP] Add cache for MLContexts in the WebNNBackend #22510

merged 3 commits into from
Oct 30, 2024

Conversation

egalli
Copy link
Contributor

@egalli egalli commented Oct 19, 2024

Description

This change adds a cache of MLContexts keyed by their options to the WebNNBackend. This makes is so that multiple InferenceSessions create with the same options will share the same context.

Motivation and Context

Since MLTensors are tied MLContexts, developer can't easily share tensors between InferenceSession (outside of manually an MLContext and specifying the context options). This leads strange behaviors such as,

const sessionsA = ort.InferenceSession.create(urlA, {
  executionProviders: ["webnn"],
  preferredOutputLocation: "ml-buffer",
});
const sessionsB = ort.InferenceSession.create(urlB, {
  executionProviders: ["webnn"],
});
const temp = await sessionA.run({/* arguments */});
const result = await sessionB.run({"input":temp["output"]}); // ERROR: Failed to execute 'dispatch' on 'MLContext': Invalid inputs: The context of MLGraph doesn't match the context of the MLTensor with name "input".

We encountered this behavior when updating the transformers.js version in the developer preview demos. microsoft/webnn-developer-preview#46

@egalli
[WebNN EP] Add cache for MLContexts in the WebNNBackend
3d45e5b 
### Description
This change adds a cache of `MLContext`s keyed by their options to the `WebNNBackend`. This makes is so that multiple `InferenceSession`s create with the same options will share the same context.

### Motivation and Context
Since `MLTensor`s are tied `MLContext`s, developer can't easily share tensors between `InferenceSession` (outside of manually an `MLContext` and specifying the `context` options). This leads strange behaviors such as,
```js
const sessionsA = ort.InferenceSession.create(urlA, {
  executionProviders: ["webnn"],
  preferredOutputLocation: "ml-buffer",
});
const sessionsB = ort.InferenceSession.create(urlB, {
  executionProviders: ["webnn"],
});
const temp = await sessionA.run({/* arguments */});
const result = await sessionB.run({"input":temp["output"]}); // ERROR: Failed to execute 'dispatch' on 'MLContext': Invalid inputs: The context of MLGraph doesn't match the context of the MLTensor with name "input".
```
We encountered this behavior when updating the transformers.js version in the developer preview demos. microsoft/webnn-developer-preview#46
@egalli egalli mentioned this pull request Oct 19, 2024
Closed
@egalli
Copy link
Contributor Author

egalli commented Oct 22, 2024

@fdwr, @guschmue PTAL

@egalli
Copy link
Contributor Author

egalli commented Oct 22, 2024

Also, @Honry PTAL

Honry
Honry approved these changes Oct 23, 2024
View reviewed changes
Copy link
Contributor

@Honry Honry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@fdwr
Copy link
Contributor

fdwr commented Oct 23, 2024

/azp run ONNX Runtime Web CI Pipeline,Windows GPU CI Pipeline,Linux Android Emulator QNN CI Pipeline

@fdwr
Copy link
Contributor

fdwr commented Oct 23, 2024

/azp run Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,Windows ARM64 QNN CI Pipeline,Windows CPU CI Pipeline

@fdwr
Copy link
Contributor

fdwr commented Oct 23, 2024

/azp run Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline

@fdwr
Copy link
Contributor

fdwr commented Oct 23, 2024

/azp run Windows GPU TensorRT CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,Windows x64 QNN CI Pipeline,Big Models

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 3 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 6 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 9 pipeline(s).

egalli added 2 commits October 23, 2024 10:54
@egalli
PR feedback
96e116b 
* Changed to a clearer and more robust way to compare `MLContextOption`s
@egalli
Merge remote-tracking branch 'origin/main' into fix_demo_inputs_double
5190109
fdwr
fdwr approved these changes Oct 23, 2024
View reviewed changes
Copy link
Contributor

@fdwr fdwr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@fdwr
Copy link
Contributor

fdwr commented Oct 23, 2024

/azp run ONNX Runtime Web CI Pipeline,Windows GPU CI Pipeline,Linux Android Emulator QNN CI Pipeline

@fdwr
Copy link
Contributor

fdwr commented Oct 23, 2024

/azp run Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,Windows ARM64 QNN CI Pipeline,Windows CPU CI Pipeline

@fdwr
Copy link
Contributor

fdwr commented Oct 23, 2024

/azp run Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline

@fdwr
Copy link
Contributor

fdwr commented Oct 23, 2024

/azp run Windows GPU TensorRT CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,Windows x64 QNN CI Pipeline,Big Models

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 2 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 3 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 4 pipeline(s).

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 9 pipeline(s).

@fdwr
Copy link
Contributor

fdwr commented Oct 24, 2024 •
edited
Loading

Enrico, seems like a legitimate ORT Web CI error (though I'm not seeing an obvious link from your change to a WebGPU failure):

    [webgpu]GroupQueryAttention - GroupQueryAttention PackedQKV 15
      × T[0]
        Chrome Headless 130.0.0.0 (Windows 10)
      AssertionError: tensor data should match: expected false to be true

https://dev.azure.com/onnxruntime/onnxruntime/_build/results?buildId=1534194&view=logs&j=4cf9212c-8936-5e77-cbdb-290c1c5567eb&t=f6ca5817-a7c2-5469-2496-2df89aa89689

(update) Never mind - I see the failure in #22181 too.

@fdwr fdwr requested a review from guschmue October 24, 2024 03:52
@guschmue
Copy link
Contributor

/azp run ONNX Runtime Web CI Pipeline,ONNX Runtime Web CI Pipeline (Build_web_Release build_onnxruntime_web)

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@guschmue guschmue added the ep:WebNN WebNN execution provider label Oct 30, 2024
@fdwr
Copy link
Contributor

fdwr commented Oct 30, 2024

/azp run ONNX Runtime Web CI Pipeline, ONNX Runtime Web CI Pipeline (Build_web_Release build_onnxruntime_web)

@azure-pipelines Azure Pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@fdwr fdwr merged commit df236c7 into microsoft:main Oct 30, 2024
72 checks passed
ishwar-raut1 pushed a commit to ishwar-raut1/onnxruntime that referenced this pull request Nov 19, 2024
@egalli
[WebNN EP] Add cache for MLContexts in the WebNNBackend (microsof…
bb700b1 
…t#22510)

### Description
This change adds a cache of `MLContext`s keyed by their options to the
`WebNNBackend`. This makes is so that multiple `InferenceSession`s
create with the same options will share the same context.

### Motivation and Context
Since `MLTensor`s are tied `MLContext`s, developer can't easily share
tensors between `InferenceSession` (outside of manually an `MLContext`
and specifying the `context` options). This leads strange behaviors such
as,
```js
const sessionsA = ort.InferenceSession.create(urlA, {
  executionProviders: ["webnn"],
  preferredOutputLocation: "ml-buffer",
});
const sessionsB = ort.InferenceSession.create(urlB, {
  executionProviders: ["webnn"],
});
const temp = await sessionA.run({/* arguments */});
const result = await sessionB.run({"input":temp["output"]}); // ERROR: Failed to execute 'dispatch' on 'MLContext': Invalid inputs: The context of MLGraph doesn't match the context of the MLTensor with name "input".
```
We encountered this behavior when updating the transformers.js version
in the developer preview demos. microsoft/webnn-developer-preview#46
@ibelem ibelem mentioned this pull request Dec 11, 2024
Open
ankitm3k pushed a commit to intel/onnxruntime that referenced this pull request Dec 11, 2024
@egalli @ankitm3k
[WebNN EP] Add cache for MLContexts in the WebNNBackend (microsof…
a832b81 
…t#22510)

### Description
This change adds a cache of `MLContext`s keyed by their options to the
`WebNNBackend`. This makes is so that multiple `InferenceSession`s
create with the same options will share the same context.

### Motivation and Context
Since `MLTensor`s are tied `MLContext`s, developer can't easily share
tensors between `InferenceSession` (outside of manually an `MLContext`
and specifying the `context` options). This leads strange behaviors such
as,
```js
const sessionsA = ort.InferenceSession.create(urlA, {
  executionProviders: ["webnn"],
  preferredOutputLocation: "ml-buffer",
});
const sessionsB = ort.InferenceSession.create(urlB, {
  executionProviders: ["webnn"],
});
const temp = await sessionA.run({/* arguments */});
const result = await sessionB.run({"input":temp["output"]}); // ERROR: Failed to execute 'dispatch' on 'MLContext': Invalid inputs: The context of MLGraph doesn't match the context of the MLTensor with name "input".
```
We encountered this behavior when updating the transformers.js version
in the developer preview demos. microsoft/webnn-developer-preview#46
ankitm3k pushed a commit to intel/onnxruntime that referenced this pull request Dec 11, 2024
@egalli @ankitm3k
[WebNN EP] Add cache for MLContexts in the WebNNBackend (microsof…
ef33af4 
…t#22510)

### Description
This change adds a cache of `MLContext`s keyed by their options to the
`WebNNBackend`. This makes is so that multiple `InferenceSession`s
create with the same options will share the same context.

### Motivation and Context
Since `MLTensor`s are tied `MLContext`s, developer can't easily share
tensors between `InferenceSession` (outside of manually an `MLContext`
and specifying the `context` options). This leads strange behaviors such
as,
```js
const sessionsA = ort.InferenceSession.create(urlA, {
  executionProviders: ["webnn"],
  preferredOutputLocation: "ml-buffer",
});
const sessionsB = ort.InferenceSession.create(urlB, {
  executionProviders: ["webnn"],
});
const temp = await sessionA.run({/* arguments */});
const result = await sessionB.run({"input":temp["output"]}); // ERROR: Failed to execute 'dispatch' on 'MLContext': Invalid inputs: The context of MLGraph doesn't match the context of the MLTensor with name "input".
```
We encountered this behavior when updating the transformers.js version
in the developer preview demos. microsoft/webnn-developer-preview#46
ankitm3k pushed a commit to intel/onnxruntime that referenced this pull request Dec 11, 2024
@egalli @ankitm3k
[WebNN EP] Add cache for MLContexts in the WebNNBackend (microsof…
229f8f7 
…t#22510)

### Description
This change adds a cache of `MLContext`s keyed by their options to the
`WebNNBackend`. This makes is so that multiple `InferenceSession`s
create with the same options will share the same context.

### Motivation and Context
Since `MLTensor`s are tied `MLContext`s, developer can't easily share
tensors between `InferenceSession` (outside of manually an `MLContext`
and specifying the `context` options). This leads strange behaviors such
as,
```js
const sessionsA = ort.InferenceSession.create(urlA, {
  executionProviders: ["webnn"],
  preferredOutputLocation: "ml-buffer",
});
const sessionsB = ort.InferenceSession.create(urlB, {
  executionProviders: ["webnn"],
});
const temp = await sessionA.run({/* arguments */});
const result = await sessionB.run({"input":temp["output"]}); // ERROR: Failed to execute 'dispatch' on 'MLContext': Invalid inputs: The context of MLGraph doesn't match the context of the MLTensor with name "input".
```
We encountered this behavior when updating the transformers.js version
in the developer preview demos. microsoft/webnn-developer-preview#46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Honry Honry Honry approved these changes

@fdwr fdwr fdwr approved these changes

@guschmue guschmue Awaiting requested review from guschmue

Assignees
No one assigned
Labels
ep:WebNN WebNN execution provider
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@egalli @fdwr @guschmue @Honry