[CANN] When using onnxruntime-cann for inference, it failed to utilize the NPU for inference

[Eviltuzki]

Describe the issue

I downloaded the Llama2-7b-onnx model from Hugging Face. Using the example code, it runs normally in both GPU and CPU environments (using CPUExecutionProvider and CUDAExecutionProvider).
However, when I replace the providers with CANNExecutionProvider and run it in the NPU environment, the following issues occur:

1. FP16 cannot run and basically hangs.
2. FP32 barely runs, but only uses 100+ MB of memory, and the CPU usage reaches over 6000%.

The providers I am using only include CANNExecutionProvider. From the observations, it seems that the NPU is not being used for inference, and instead, the CPU is being used (the inference time on the NPU is even longer than on the CPU).NPU usage is as shown in the image.

To reproduce

code same as:
https://huggingface.co/alpindale/Llama-2-7b-ONNX/tree/main
I just replace providers

    providers = [
        (
            "CANNExecutionProvider",
            {
                "device_id": 0,
                "arena_extend_strategy": "kNextPowerOfTwo",
                "op_select_impl_mode": "high_performance",
                "optypelist_for_implmode": "Gelu",
                "enable_cann_graph": True
            },
        )
    ]

Urgency

No response

Platform

Linux

OS Version

openEuler

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

onnxruntime-cann 1.18.0

ONNX Runtime API

Python

Architecture

ARM64

Execution Provider

Other / Unknown

Execution Provider Library Version

Ascend-cann-toolkit_8.0.RC2

[Eviltuzki]

Supplement:
I have already tried manually compiling version 1.19.2 of onnxruntime-cann, same as 1.18.0

[github-actions]

This issue has been automatically marked as stale due to inactivity and will be closed in 30 days if no further activity occurs. If further support is needed, please provide an update and/or more details.

[lucien-ye]

Any progress? I have the same problem，when add profile to session，it shows that all operations are running on cpu.