[DmlEP] BeamSearch node is not supported for DmlEP?

[trajepl]

Describe the issue

Could not find an implementation for BeamSearch node even if I enable onnxruntime extensions for Dml EP.

To reproduce

Install DmlEP, run whisper example https://github.com/microsoft/Olive/tree/main/examples/whisper.

Remembered to change the ep to

Urgency

No response

Platform

Windows

OS Version

Win11

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.16.3

ONNX Runtime API

Python

Architecture

X64

Execution Provider

DirectML

Execution Provider Library Version

No response

[trajepl]

[fdwr]

com.microsoft.BeamSearch isn't supported by the DML Execution Provider (see the DML EP kernel list), but you should still get fallback to the CPU for this operator though. So why that's not happening is the question. I suspect @PatriceVignola would be more familiar with this area (noting you're using Whisper and Olive), since he's been trying a number of transformer models lately.

[trajepl]

Thanks for your answer. So sounds like it is hard to [1]run generation task in DmL-gpu if we want to insert beam search node into onnx graph. The expected behavior should fallback to CPU but the performance will be impacted.

I am the Olive contributor actually. Just want to double check if [1] is supported in Dml EP as current example in Olive for Eml use optimum for generation tasks.

[trajepl]

As for the failure of fallback, I doubt it is my local env issue. Switched to another linux machine, it worked. So I will close this issue.
Thanks! fdwr

[fdwr]

if we want to insert beam search node into onnx graph

Reading com.microsoft.BeamSearch, it doesn't sound like a very GPU-amenable operator, with subgraphs and variable size ngrams and tokens. Hopefully it's an operator than just occurs once or a few times in the graph, which avoids CPU<->GPU transfer stalls.

As for the failure of fallback ... Switched to another linux machine, it worked.

🤔 DML doesn't run on raw Linux. Did you mean using the DML EP atop WSL?

Just want to double check if [1] is supported in Dml EP as current example in Olive

Sounds like the current answer is no 😉, not until the bug is figured out. DML itself has no awareness of BeamSearch, and so it might also be a graph transformer issue outside the EP. I'm sure Pat will get to this matter :).

[trajepl]

Thanks! 👍 Yes it seems the graph transformer issue and I will reach our Pat for help if needed.

🤔 DML doesn't run on raw Linux. Did you mean using the DML EP atop WSL?

Oh, I mean the fallback logics not limited to the EP.