Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

example/llm: minor refactor #227

Merged
merged 4 commits into from
Jun 14, 2024
Merged

example/llm: minor refactor #227

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d22d250
chore: update `flake.lock` so that it works
drupol Jun 14, 2024
398096a
example/llm/open-webui: update default environment variables
drupol Jun 14, 2024
9204942
example/llm/ollama: use default `dataDir` within the same directory
drupol Jun 14, 2024
222a3ea
example/llm/ollama: don't download 4GB of models upon starting
drupol Jun 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions example/llm/flake.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

18 changes: 13 additions & 5 deletions example/llm/flake.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@
# Backend service to perform inference on LLM models
ollama."ollama1" = {
enable = true;
# The models are usually huge, downloading them in every project directory can lead to a lot of duplication
dataDir = "$HOME/.services-flake/ollama1";
models = [ "llama2-uncensored" ];
# The models are usually huge, downloading them in every project
# directory can lead to a lot of duplication. Change here to a
# directory where the Ollama models can be stored and shared across
# projects.
# dataDir = "$HOME/.services-flake/ollama1";
};
# Get ChatGPT like UI, but open-source, with Open WebUI
open-webui."open-webui1" = {
Expand All @@ -33,9 +35,15 @@
inherit (pc.config.services.ollama.ollama1) host port;
in
{
OLLAMA_API_BASE_URL = "http://${host}:${toString port}";
ENABLE_OLLAMA_API = "True";
OLLAMA_BASE_URL = "http://${host}:${toString port}";
OLLAMA_API_BASE_URL = "http://${host}:${toString port}/api";
WEBUI_AUTH = "False";
};
DEVICE_TYPE = "cpu";
RAG_EMBEDDING_ENGINE = "ollama";
RAG_EMBEDDING_MODEL = "mxbai-embed-large:latest";
RAG_EMBEDDING_MODEL_AUTO_UPDATE = "True";
RAG_RERANKING_MODEL_AUTO_UPDATE = "True"; };
};
};

Expand Down
Loading