Add documentation about exporting pte on Qualcomm backend without quantization

[DongGeun123]

The documentation for the Qualcomm backend (found here) only describes exporting to a .pte file with quantization.

Is there a way to use prequantized models provided in the Llama3 repository? For example, the XNNPACK backend supports using prequantized checkpoints (SpinQuant) from Hugging Face.

[JacobSzwejbka]

@cccclai or @larryliu0820 do you know the answer to this?

[cccclai]

The prequantized models provided in the Llama3 repository (if you meant https://ai.meta.com/blog/meta-llama-quantized-lightweight-models/) is mainly targeting for CPU instead of general NPU use cases, as it bakes in quite a bit of assumption.

@chiwwang in case you know more about QC's plan for the quantized model.

[chiwwang]

@haowhsu-quic @shewu-quic

mmmm.... I think that would be good if we can accept pre-quantized models. However, llama3.2 is special. In current situations, we need quite a few customizations on the model structure. Some of them are even hard to do by transformation, e.g., moving KV to graph I/O.

So there is still a (not small) gap at the front of us to support the quantized LLM in the official llama repo.