Skip to content

Commit

Permalink
wip
Browse files Browse the repository at this point in the history
  • Loading branch information
kenarsa committed Apr 30, 2024
1 parent 5f1b873 commit d204d9d
Show file tree
Hide file tree
Showing 2 changed files with 45 additions and 7 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ language models. picoLLM Inference Engine is:

## Models

PicoLLM Inference Engine supports the following open-weight models. You can download them from
picoLLM Inference Engine supports the following open-weight models. You can download them from
[Picovoice Console](https://console.picovoice.ai/).

- Gemma
Expand Down
50 changes: 44 additions & 6 deletions binding/python/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,17 @@

Made in Vancouver, Canada by [Picovoice](https://picovoice.ai)

picoLLM Inference Engine is a local large-language model inference engine. picoLLM Inference Engine is:
picoLLM Inference Engine is a highly accurate and cross-platform SDK optimized for running deeply compressed local large
language models. picoLLM Inference Engine is:

- Accurate; [Improves GPTQ by up to 80%]().
- Accurate; picoLLM Compression improves GPTQ by XX%.
- Private; LLM inference runs 100% locally.
- Cross-Platform:
- Linux (x86_64), macOS (x86_64, arm64), Windows (x86_64)
- Raspberry Pi (3, 4, 5)
- Runs on CPU and GPU
- Free
- Free for open-weight models

## Compatibility

- Python 3.7+
Expand All @@ -21,11 +26,44 @@ picoLLM Inference Engine is a local large-language model inference engine. picoL
pip3 install picollm
```

## Models

picoLLM Inference Engine supports the following open-weight models. You can download them from
[Picovoice Console](https://console.picovoice.ai/).

- Gemma
- `gemma-2b`
- `gemma-2b-it`
- `gemma-7b`
- `gemma-7b-it`
- Llama-2
- `llama-2-7b`
- `llama-2-7b-chat`
- `llama-2-13b`
- `llama-2-13b-chat`
- `llama-2-70b`
- `llama-2-70b-chat`
- Llama-3
- `llama-3-8b`
- `llama-3-8b-instruct`
- `llama-3-70b`
- `llama-3-70b-instruct`
- Mistral
- `mistral-7b-v0.1`
- `mistral-7b-instruct-v0.1`
- `mistral-7b-instruct-v0.2`
- Mixtral
- `mixtral-8x7b-v0.1`
- `mixtral-8x7b-instruct-v0.1`
- Phi-2

## AccessKey

picoLLM requires a valid Picovoice `AccessKey` at initialization. `AccessKey` acts as your credentials when using
picoLLM SDKs. You can get your `AccessKey` for free. Make sure to keep your `AccessKey` secret. Signup or Login to
[Picovoice Console](https://console.picovoice.ai/) to get your `AccessKey`.
AccessKey is your authentication and authorization token for deploying Picovoice SDKs, including picoLLM. Anyone who is
using Picovoice needs to have a valid AccessKey. You must keep your AccessKey secret. You would need internet
connectivity to validate your AccessKey with Picovoice license servers even though the LLM inference is running 100%
offline and completely free for open-weight models. Everyone who signs up for
[Picovoice Console](https://console.picovoice.ai/) receives a unique AccessKey.

## Usage

Expand Down

0 comments on commit d204d9d

Please sign in to comment.