Can I use Microsoft.AI.MachineLearning(WinML) interfaces to enable OnnxRuntime-QNN-EP(Qualcomm NPU) or other no-DirectX EP ?

[caofx0418]

Describe the feature request

Can I use Microsoft.AI.MachineLearning(WinML) interfaces to enable OnnxRuntime-QNN-EP(Qualcomm NPU) or other no-DirectX EP ?

1. From "windows-ml/get-started", we can see 2 solutions: In-box vs NuGet
   https://learn.microsoft.com/en-us/windows/ai/windows-ml/get-started

2. In-box: OnnxRuntime as a part of Windows, we can't change. so it's no chance to use myself built OnnxRuntime with QNN-EP.
   

3. NuGet: Microsoft.ai.machinelearning.dll does not contain an embedded ONNX runtime, instead the ONNX runtime is built into a file: onnxruntime.dll.
   

   and maybe we can replace the onnxruntime.dll to myself built OnnxRuntime with QNN-EP.
   but the WinML interface: public enum LearningModelDeviceKind, only support 4 kinds: 0 Default, 1 Cpu, 2 DirectX, 3 DirectXHighPerformance, 4 DirectXMinPower.
   But there is no option for QNN-EP or other EP in the LearningModelDeviceKind enumeration.

So can I use Microsoft.AI.MachineLearning(WinML) interfaces to enable OnnxRuntime-QNN-EP(Qualcomm NPU) or other no-DirectX EP?

Describe scenario use case

Windows: Microsoft.AI.MachineLearning(WinML) interfaces to enable OnnxRuntime-QNN-EP(Qualcomm NPU) or other no-DirectX EP

[caofx0418]

@smk2007

Can you help me? Thank you.

[caofx0418]

@smk2007 @PatriceVignola @justinchuby @snnn @HectorSVC
could you please take a look this issue?
thank you

[caofx0418]

@jywu-msft @fdwr

could you please take a look this issue?
thank you

[fdwr]

Can I use WinML ... to enable QNN-EP...?

@caofx0418 Nope, WinML only supports CPU and DirectX (DML) backends.

onnxruntime/winml/lib/Api/LearningModelDevice.cpp

Lines 27 to 35 in 61e07a4

	LearningModelDevice::LearningModelDevice(winml::LearningModelDeviceKind const& deviceKind) try
	: m_deviceCache(std::make_unique<_winml::D3DDeviceCache>(deviceKind)) {
	m_deviceKind = deviceKind;
	telemetry_helper.SetLearningModelDeviceKind(static_cast<int>(deviceKind));
	m_isCpuDevice = m_deviceKind == LearningModelDeviceKind::Cpu || m_deviceKind == LearningModelDeviceKind::Default;
	if (m_isCpuDevice) {
	assert(m_deviceCache->GetD3D12Device() == nullptr);
	}
	}

[caofx0418]

Can I use WinML ... to enable QNN-EP...?

@caofx0418 Nope, WinML only supports CPU and DirectX (DML) backends.

onnxruntime/winml/lib/Api/LearningModelDevice.cpp

Lines 27 to 35 in 61e07a4

	LearningModelDevice::LearningModelDevice(winml::LearningModelDeviceKind const& deviceKind) try
	: m_deviceCache(std::make_unique<_winml::D3DDeviceCache>(deviceKind)) {
	m_deviceKind = deviceKind;
	telemetry_helper.SetLearningModelDeviceKind(static_cast<int>(deviceKind));
	m_isCpuDevice = m_deviceKind == LearningModelDeviceKind::Cpu || m_deviceKind == LearningModelDeviceKind::Default;
	if (m_isCpuDevice) {
	assert(m_deviceCache->GetD3D12Device() == nullptr);
	}
	}

Thank you for your reply!

Do WinML have plan to support other Backends in future ？

[fdwr]

Do WinML have plan to support other Backends in future ？

I defer to Sheil @smk2007 for any future WinML plans, but my wager is that WinML will only target Windows API's like DirectML rather than using lower-level device-specific driver interfaces. WinML is a thin layer over ORT though - can you call ORT directly?

[caofx0418]

Do WinML have plan to support other Backends in future ？

I defer to Sheil @smk2007 for any future WinML plans, but my wager is that WinML will only target Windows API's like DirectML rather than using lower-level device-specific driver interfaces. WinML is a thin layer over ORT though - can you call ORT directly?

Thank you for your reply!

I can call ORT C++ interface directly.
and I also want to make my EP more "Windows ML native".

[ashumish-QCOM]

Hi @caofx0418

Qualcomm NPU (QNN-EP) with ONNX Runtime should be feasible. However, WinML currently only supports CPU and DirectX (DML) backends. If you need to use the QNN-EP, you might consider calling the ONNX Runtime (ORT) directly instead of through WinML.

For those looking to integrate QNN-EP, ensure that your ONNX Runtime build includes the necessary QNN libraries and that your environment paths are correctly set. This setup has been effective for other models available from the Qualcomm AI Hub.

Thank you.