Is sh examples/demo-apps/android/LlamaDemo/setup-with-qnn.sh supposed/equipped to generate .aar file for x86_64?

[Astuary]

🐛 Describe the bug

Hello,

I was wondering if its possible to test the QNN delegate on Android Studio emulator by building an .aar file for x86_64 ABI. I am guessing that it might not work because emulators are not equipped for QNN backend, but wanted to double check.

Versions

PyTorch version: 2.6.0.dev20241101+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 24.04 LTS (x86_64)
GCC version: (Ubuntu 13.2.0-23ubuntu4) 13.2.0
Clang version: 18.1.3 (1ubuntu1)
CMake version: version 3.30.5
Libc version: glibc-2.39

Python version: 3.10.15 (main, Oct 3 2024, 07:27:34) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-6.8.0-39-generic-x86_64-with-glibc2.39
Is CUDA available: False
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1080 Ti
Nvidia driver version: 550.90.07
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 48
On-line CPU(s) list: 0-23
Off-line CPU(s) list: 24-47
Vendor ID: GenuineIntel
Model name: Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz
CPU family: 6
Model: 85
Thread(s) per core: 1
Core(s) per socket: 12
Socket(s): 2
Stepping: 4
CPU(s) scaling MHz: 34%
CPU max MHz: 3000.0000
CPU min MHz: 0.0000
BogoMIPS: 4200.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 pti ssbd mba ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req vnmi pku ospke md_clear flush_l1d arch_capabilities
Virtualization: VT-x
L1d cache: 768 KiB (24 instances)
L1i cache: 768 KiB (24 instances)
L2 cache: 24 MiB (24 instances)
L3 cache: 33 MiB (2 instances)
NUMA node(s): 2
NUMA node0 CPU(s): 0-11
NUMA node1 CPU(s): 12-23
Vulnerability Gather data sampling: Vulnerable
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Mitigation; PTE Inversion; VMX conditional cache flushes, SMT disabled
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT disabled
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT disabled
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Mitigation; IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Mitigation; Clear CPU buffers; SMT disabled

Versions of relevant libraries:
[pip3] executorch==0.5.0a0+026fe0b
[pip3] numpy==1.26.4
[pip3] torch==2.6.0.dev20241101+cpu
[pip3] torchao==0.5.0+git0916b5b2
[pip3] torchaudio==2.5.0.dev20241101+cpu
[pip3] torchsr==1.0.4
[pip3] torchvision==0.20.0.dev20241101+cpu
[conda] executorch 0.5.0a0+026fe0b pypi_0 pypi
[conda] numpy 1.26.4 pypi_0 pypi
[conda] torch 2.6.0.dev20241101+cpu pypi_0 pypi
[conda] torchao 0.5.0 pypi_0 pypi
[conda] torchaudio 2.5.0.dev20241101+cpu pypi_0 pypi
[conda] torchsr 1.0.4 pypi_0 pypi
[conda] torchvision 0.20.0.dev20241101+cpu pypi_0 pypi

[metascroy]

cc @cccclai do you know?

[cccclai]

We actually have emulator support for QNN, but it was only tested via adb binary. See instruction in https://pytorch.org/executorch/stable/build-run-qualcomm-ai-engine-direct-backend.html#test-model-inference-on-qnn-htp-emulator

Ideally it should work with Android Studio, but may need some tweaking.

[Astuary]

Hi @cccclai

Given that tutorial, I am trying to run Llama2 on the QNN HTP emulator with this command examples/qualcomm/oss_scripts/llama2/qnn_llama_runner --model_path ../llama2.pte --tokenizer_path ../llama2_tokenizer.bin

I am getting

I 00:00:00.001227 executorch:runner.cpp:61] Creating LLaMa runner: model_path=../llama2.pte, tokenizer_path=../llama2_tokenizer.bin
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[WARNING] [Qnn ExecuTorch]: QnnDsp <W> Initializing HtpProvider

[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[WARNING] [Qnn ExecuTorch]: QnnDsp <W> Performance Estimates unsupported

[WARNING] [Qnn ExecuTorch]: QnnDsp <W> Arch 68 set by custom config is different from arch associated with SoC 57, will overwrite it to 75

[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[WARNING] [Qnn ExecuTorch]: QnnDsp <W> Performance Estimates unsupported

[WARNING] [Qnn ExecuTorch]: QnnDsp <W> Arch 68 set by custom config is different from arch associated with SoC 57, will overwrite it to 75

[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[WARNING] [Qnn ExecuTorch]: QnnDsp <W> Performance Estimates unsupported

[WARNING] [Qnn ExecuTorch]: QnnDsp <W> Arch 68 set by custom config is different from arch associated with SoC 57, will overwrite it to 75

[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 2
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[WARNING] [Qnn ExecuTorch]: QnnDsp <W> Performance Estimates unsupported

[WARNING] [Qnn ExecuTorch]: QnnDsp <W> Arch 68 set by custom config is different from arch associated with SoC 57, will overwrite it to 75

[INFO] [Qnn ExecuTorch]: Running level=3 optimization.
E 00:00:03.745815 executorch:operator_registry.cpp:185] kernel 'llama::fallback.out' not found.
E 00:00:03.745827 executorch:operator_registry.cpp:186] dtype: 6 | dim order: [
E 00:00:03.745830 executorch:operator_registry.cpp:186] 0,
E 00:00:03.745832 executorch:operator_registry.cpp:186] 1,
E 00:00:03.745837 executorch:operator_registry.cpp:186] 2,
E 00:00:03.745841 executorch:operator_registry.cpp:186] ]
E 00:00:03.745843 executorch:operator_registry.cpp:186] dtype: 6 | dim order: [
E 00:00:03.745846 executorch:operator_registry.cpp:186] 0,
E 00:00:03.745849 executorch:operator_registry.cpp:186] 1,
E 00:00:03.745853 executorch:operator_registry.cpp:186] 2,
E 00:00:03.745860 executorch:operator_registry.cpp:186] ]
E 00:00:03.745863 executorch:operator_registry.cpp:186] dtype: 6 | dim order: [
E 00:00:03.745868 executorch:operator_registry.cpp:186] 0,
E 00:00:03.745871 executorch:operator_registry.cpp:186] 1,
E 00:00:03.745875 executorch:operator_registry.cpp:186] 2,
E 00:00:03.745879 executorch:operator_registry.cpp:186] ]
E 00:00:03.745883 executorch:method.cpp:554] Missing operator: [1] llama::fallback.out
E 00:00:03.745953 executorch:operator_registry.cpp:185] kernel 'llama::fallback.out' not found.
E 00:00:03.745957 executorch:operator_registry.cpp:186] dtype: 6 | dim order: [
E 00:00:03.745960 executorch:operator_registry.cpp:186] 0,
E 00:00:03.745962 executorch:operator_registry.cpp:186] 1,
E 00:00:03.745966 executorch:operator_registry.cpp:186] 2,
E 00:00:03.745969 executorch:operator_registry.cpp:186] ]
E 00:00:03.745973 executorch:operator_registry.cpp:186] dtype: 6 | dim order: [
E 00:00:03.745975 executorch:operator_registry.cpp:186] 0,
E 00:00:03.745979 executorch:operator_registry.cpp:186] 1,
E 00:00:03.745983 executorch:operator_registry.cpp:186] 2,
E 00:00:03.745986 executorch:operator_registry.cpp:186] ]
E 00:00:03.745990 executorch:operator_registry.cpp:186] dtype: 6 | dim order: [
E 00:00:03.745993 executorch:operator_registry.cpp:186] 0,
E 00:00:03.745997 executorch:operator_registry.cpp:186] 1,
E 00:00:03.746001 executorch:operator_registry.cpp:186] 2,
E 00:00:03.746005 executorch:operator_registry.cpp:186] ]
E 00:00:03.746009 executorch:method.cpp:554] Missing operator: [1] llama::fallback.out
E 00:00:03.746075 executorch:operator_registry.cpp:185] kernel 'llama::fallback.out' not found.
E 00:00:03.746080 executorch:operator_registry.cpp:186] dtype: 6 | dim order: [
E 00:00:03.746082 executorch:operator_registry.cpp:186] 0,
E 00:00:03.746084 executorch:operator_registry.cpp:186] 1,
E 00:00:03.746087 executorch:operator_registry.cpp:186] 2,
E 00:00:03.746091 executorch:operator_registry.cpp:186] ]
E 00:00:03.746096 executorch:operator_registry.cpp:186] dtype: 6 | dim order: [
E 00:00:03.746100 executorch:operator_registry.cpp:186] 0,
E 00:00:03.746104 executorch:operator_registry.cpp:186] 1,
E 00:00:03.746109 executorch:operator_registry.cpp:186] 2,
E 00:00:03.746112 executorch:operator_registry.cpp:186] ]
E 00:00:03.746116 executorch:operator_registry.cpp:186] dtype: 6 | dim order: [
E 00:00:03.746120 executorch:operator_registry.cpp:186] 0,
E 00:00:03.746124 executorch:operator_registry.cpp:186] 1,
E 00:00:03.746128 executorch:operator_registry.cpp:186] 2,
E 00:00:03.746133 executorch:operator_registry.cpp:186] ]
E 00:00:03.746137 executorch:method.cpp:554] Missing operator: [1] llama::fallback.out
E 00:00:03.746409 executorch:method.cpp:761] There are 3 instructions don't have corresponding operator registered. See logs for details
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend parameters
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
F 00:00:04.345040 executorch:qnn_llama_runner.cpp:65] In function main(), assert failed (runner.load() == Error::Ok): Runner failed to load method
Aborted

Can you confirm if my command is correct?

[cccclai]

Ah, the command seems right to me. Looks like there is a missing operator fallback and you can refer to the lines here.

@chiwwang the llama2 runner may need the fix at some point.

In the meanwhile, I remember runner emulator is very slow...is it for testing purpose or something else?

[Astuary]

Hi @cccclai,

Yes I wanted to try the emulator because I don't have a 12GB or 16GB RAM phone for Llama2.

But I see that I should be able to run Llama3.2 (1B). Can you confirm if I should be using the runner from examples/qualcomm/qaihub_scripts/llama/llama3 or from examples/qualcomm/oss_scripts/llama3_2?

[cccclai]

examples/qualcomm/oss_scripts/llama3_2 is for 1b, but it's actively wip and we didn't try using the simulator with it. Please expect some hiccup if you'd like to try it now, otherwise maybe wait until it's in a more stable stage. In the meanwhile, the accuracy can be bad because that's also wip.

[chiwwang]

@haowhsu-quic , @shewu-quic , @winskuo-quic

[shewu-quic]

@chiwwang the llama2 runner may need the fix at some point.

Thanks for your reminder
Got it. Once we complete the llama3_2 1B enablement, we can try integrating llama2 and llama3_2."