From fe32f23a7752184a620e8fd78f53a3b0e6707da3 Mon Sep 17 00:00:00 2001 From: "Jorge O. Castro" Date: Sun, 21 Jul 2024 22:04:53 -0400 Subject: [PATCH] add ai part --- _docs/ai.md | 128 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 _docs/ai.md diff --git a/_docs/ai.md b/_docs/ai.md new file mode 100644 index 0000000..7c0532a --- /dev/null +++ b/_docs/ai.md @@ -0,0 +1,128 @@ +--- +layout: page +title: AI and Machine Learning +permalink: /ai +--- + +# AI and Machine Learning + +> Note: We want Bluefin to be the best developer environment for AI enthusiasts, help wanted! + +## Local AI + +[Ollama](https://ollama.ai/) allows the running of open-source large language models, such as Llama 3, locally. It bundles model weights, configuration, and data into a single package, defined by a Modelfile, and optimizes setup and configuration details, including GPU usage. + +Bluefin-dx supports the installation of Ollama in different ways, for example by using the following `ujust` commands: + +- `ujust ollama` installs the CLI-version as a container. +- `ujust ollama-web` installs [Open Web UI](https://docs.openwebui.com/) & Ollama as a container. During the installation process, there is the choice to install either a GPU or CPU-enabled version. Additionally, installation through Homebrew (`brew install ollama`) is required. + +`systemd` does not autostart the containers; instead, the user needs to activate the script manually by using `systemctl --user start ollama` or `systemctl --user start ollama-web`. The `systemd` scripts are saved under: `~/.config/containers/systemd`. The scripts are: + +- `ollama.container` - which starts the CLI under port: 11434 +- `ollama-web.container` - which starts the Open Web UI under port: 8080 ([http://localhost:11434](http://localhost:11434)) +- `ollama.network`, the network name is set as "ollama" + +To cross-check if the containers are launched correctly, you can use `podman ps --all`. + +### Running the ollama open-webui + +`ujust ollama-web` will set up ollama with the webui as a service unit in your user space: + +- Status: `systemctl status --user ollama-web` +- Restart: `systemctl restart --user ollama-web` + +You can also `start`, `stop`, and `disable` and `enable` the service unit. Open `localhost:8080` in your browser, then make yourself an account: + +![image|539x500](upload://jfLCKBHQB7JX23NrXzscZqjWPZi.png) + +### Running ollama + +`ujust ollama` to get started! This will setup an ollama service unit in your user space. Use this one if you just want ollama without the web ui: + +- Status: `systemctl status --user ollama` +- Restart: `systemctl restart --user ollama` + +You can also `start`, `stop`, and `disable` and `enable` the service unit. + +### Desktop integration + +[Install alpaca](https://flathub.org/apps/com.jeffser.Alpaca) to use a native desktop application. + +![image|670x500](upload://l7vg1QKA0BUolxWvQhJEmcJNEnM.png) + +Se the server as `http://localhost:11434` in the settings to connect to your local setup: + +![image|670x500](upload://fxsQolGDIqaWuajIRDMOawLD4aU.png) + +### Terminal Integration + +We recommend using [mods](https://github.com/charmbracelet/mods) to use the provided ollama instance as an interface to your command line: + +![image|690x356](upload://aBT7XLxCYYUKcQu8zHTB7wvEzR8.png) + +- Install mods: `brew install mods` +- Open the config file for mods: `mods --settings` +- Modify the config file with the model you've pulled with ollama: + +![image|690x85](upload://n98BUDEuj0HGyf9uSNHQqVpOXXQ.png) + +and then make sure the ollama section in the config file matches the same thing: + +![image|690x231](upload://99fwELCEzilratywCDaNIDJLFca.png) + +Save the file and then try it: + +![image|690x314](upload://7fAOqqvgxpL2WRj46qjaOoj5fqJ.jpeg) + +## Pytorch + +Bluefin and Aurora include a NGC container that includes the latest [stable PyTorch from Nvidia](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch) on top of Ubuntu LTS. It includes `python3`, `pip3` , `pandas`, `matplotlib`, and `scikit-learn`. + +> The additional `pip` packages are commonly used but not comprehensive. Only `pip` is used and `mamba` or `conda` are not currently tested. + +### Pre-requisites + +You must already be using `bluefin-dx-nvidia` as it's meant for those GPUs and has nvidia container toolkit support. If you want to test before downloading the larger nvidia container run `nvidia-smi` +inside a regular Ubuntu box (`distrobox create --nvidia -i ubuntu`) to check that everything works correctly. + +To create the box use the following command. **Note** that this container is large (20+ GB!): + +```bash +ujust distrobox-mlbox +``` + +To enter the working environment: + +```bash +distrobox enter mlbox +``` + +Then the init-hooks will run once. After which, you should be able to run: + +```bash +nvidia-smi +``` + +To check if GPUs are seen and enter the python repo run: + +```bash +import torch;torch.cuda.is_available() +``` + +Various tests can be run to test a transformers inference or training job, or to git clone a pytorch benchmarks repo and run single or multi gpu commands: E.g. to test multi-gpu setup on two 3090s: + +```bash +git clone https://github.com/aime-team/pytorch-benchmarks.git +cd pytorch-benchmarks +python3 main.py --num_gpus 2 --compile --model bert-large-uncased --data_name squad --global_batch_size 24 +``` + +On other operating systems, use [this .ini file](https://github.com/ublue-os/bluefin/blob/730f39caae21e48fb91f00010cf0cf8d32ee44bd/dx/usr/share/ublue-os/distrobox/pytorch-nvidia.ini) and run: + +```bash +distrobox assemble create --file /path/to/your/mlbox.ini +``` + + +Doc credits: @axelquack \ No newline at end of file