From 507f5c49bec5c081e6c8baf1f060a1121349fcd5 Mon Sep 17 00:00:00 2001
From: Simon Kelly <skelly@dimagi.com>
Date: Mon, 20 May 2024 10:17:37 +0200
Subject: [PATCH] add docs for setting up ollama

---
 ai.md | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/ai.md b/ai.md
index 6640b74..2ef0b8d 100644
--- a/ai.md
+++ b/ai.md
@@ -46,8 +46,25 @@ to another model to try different options.
 For further reading, see the documentation of the [litellm Python API](https://docs.litellm.ai/docs/completion),
 and [litellm providers](https://docs.litellm.ai/docs/providers).
 
-For further reading, see the documentation of the [llm Python API](https://llm.datasette.io/en/stable/python-api.html),
-and [llm generally](https://llm.datasette.io/en/stable/index.html).
+### Running open source LLMs
+To run models like Mixtral or Llama3, you will need to run an [Ollama](https://ollama.com/) server in a separate process.
+
+1. [Download](https://ollama.com/download) and run Ollama or use the Docker [image](https://hub.docker.com/r/ollama/ollama)
+2. Download the model you want to run:
+   ```shell
+   ollama pull llama3
+   # or with docker
+   docker exec -it ollama ollama pull llama3
+   ```
+   See the [documentation](https://docs.litellm.ai/docs/providers/ollama) for the list of supported models.
+3. Update your django settings to point to the Ollama server. For example:
+   ```python
+   LLM_MODELS = {
+       "ollama_chat/llama3": {"api_base": "http://localhost:11434"},
+   }
+   DEFAULT_LLM_MODEL = "ollama_chat/llama3"
+   ```
+4. Restart your Django server.
 
 ### The Chat UI