[Feature]: Access token embeddings

[lsorber]

What feature would you like to request?

Would it be possible to get access to the token embeddings before they are pooled? Llama-cpp-python allows this for instance by setting pooling_type=LLAMA_POOLING_TYPE_NONE [1]. Such a feature allows building custom embedding pipelines with ColBERT-like functionality.

[1] https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#embeddings

Is there any additional information you would like to provide?

No response