Skip to content

Commit

Permalink
minor fixes
Browse files Browse the repository at this point in the history
  • Loading branch information
benironside committed Jul 5, 2024
1 parent 86923ec commit a34282f
Showing 1 changed file with 6 additions and 6 deletions.
12 changes: 6 additions & 6 deletions docs/serverless/assistant/connect-to-byo-llm.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -94,14 +94,14 @@ Once you've launched LM Studio, select a model:
For security reasons, before downloading a model, verify that it is from a trusted source. It can be helpful to review community feedback on the model (for example using a site like Hugging Face).
</DocCallOut>

1. Go to LM Studio's Search window.
2. Search for an LLM (for example, `Mixtral-8x7B-instruct`).
3. Your chosen model must include `instruct` in its name in order to work with Elastic.
4. Filter your search for "Compatibility Guess" to optimize results for your hardware. Results will be color coded:
. Go to LM Studio's Search window.
. Search for an LLM (for example, `Mixtral-8x7B-instruct`).
. Your chosen model must include `instruct` in its name in order to work with Elastic.
. Filter your search for "Compatibility Guess" to optimize results for your hardware. Results will be color coded:
.. Green means "Full GPU Possible", which yields the best results.
.. Blue means "Partial GPU offload possible", which may work.
.. Red for "Likely too large for this machine", which typically will not work.
5. Download one or more models.
. Download one or more models.

In this example we used [`TheBloke/Mixtral-8x7B-Instruct-v0.1.Q3_K_M.gguf`](https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF). It has 46.7B total parameters, a 32,000 token context window, and uses GGUF [quanitization](https://huggingface.co/docs/transformers/main/en/quantization/overview). For more information about model names and format information, refer to the following table:

Expand All @@ -119,7 +119,7 @@ After downloading a model, you'll need to load it in LM Studio using the GUI or

It is a best practice to use the GUI to download at least one model from the marketplace first, and then load/unload the model(s) using the CLI. The GUI allows you to search for models, whereas the CLI only allows you to import specific paths, but the CLI provides a good interface for loading/unloading.

Use the following commands in your CLI:
Use the following commands in your CLI.

1. Verify LM Studio: `lms`
2. Check LM Studio's status: `lms status`
Expand Down

0 comments on commit a34282f

Please sign in to comment.