Skip to content

Commit

Permalink
docs: add numpy issue to troubleshooting
Browse files Browse the repository at this point in the history
  • Loading branch information
jaluma committed Aug 7, 2024
1 parent ca2b8da commit 1af8da2
Show file tree
Hide file tree
Showing 2 changed files with 31 additions and 1 deletion.
3 changes: 3 additions & 0 deletions fern/docs/pages/installation/installation.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,9 @@ In order for local LLM and embeddings to work, you need to download the models t
poetry run python scripts/setup
```

If you want to run PrivateGPT with Llama-CPP running using CUDA, please follow the instructions in the
[troubleshooting](#/installation/getting-started/troubleshooting#guide-for-building-llama-cpp-with-cuda-support) section.

Once installed, you can run PrivateGPT with the following command:

```bash
Expand Down
29 changes: 28 additions & 1 deletion fern/docs/pages/installation/troubleshooting.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -46,4 +46,31 @@ huggingface:
embedding:
embed_dim: 384
```
</Callout>
</Callout>

# Guide for Building llama-cpp with CUDA support

## Numpy installation issue

First, ensure you have the necessary prerequisites installed on your system.
You'll need to force a specific version of NumPy due to compatibility issues.
Run the following command to install `llama-cpp` with CUDA support:

```bash
CMAKE_ARGS='-DLLAMA_CUBLAS=on' poetry run pip install --force-reinstall --no-cache-dir llama-cpp-python numpy==1.26.0
```

Give thanks to [MarioRossiGithub](https://github.com/MarioRossiGithub) for providing the following solution.

## Out-of-memory error

If you encounter an out-of-memory error while running `llama-cpp` with CUDA, you can try the following steps to resolve the issue:
1. **Set the next environment:**
```bash
TOKENIZERS_PARALLELISM=true
```
2. **Run PrivateGPT:**
```bash
poetry run python -m privategpt
```
Give thanks to [MarioRossiGithub](https://github.com/MarioRossiGithub) for providing the following solution.

0 comments on commit 1af8da2

Please sign in to comment.