This guide will walk you though the steps to get started on end-to-end flow for LlamaStack. This guide mainly focuses on getting started with building a LlamaStack distribution, and starting up a LlamaStack server. Please see our documentations on what you can do with Llama Stack, and llama-stack-apps on examples apps built with Llama Stack.
The llama CLI tool helps you setup and use the Llama toolchain & agentic systems. It should be available on your path after installing the llama-stack package.
You have two ways to install this repository:
-
Install as a package: You can install the repository directly from PyPI by running the following command:
pip install llama-stack
-
Install from source: If you prefer to install from the source code, follow these steps:
mkdir -p ~/local cd ~/local git clone [email protected]:meta-llama/llama-stack.git conda create -n stack python=3.10 conda activate stack cd llama-stack $CONDA_PREFIX/bin/pip install -e .
For what you can do with the Llama CLI, please refer to CLI Reference.
You have two ways to start up Llama stack server:
- Starting up server via docker:
We provide pre-built Docker image of Llama Stack distribution, which can be found in the following links in the distributions folder.
Note
For GPU inference, you need to set these environment variables for specifying local directory containing your model checkpoints, and enable GPU inference to start running docker container.
export LLAMA_CHECKPOINT_DIR=~/.llama
Note
~/.llama should be the path containing downloaded weights of Llama models.
To download llama models, use
llama download --model-id Llama3.1-8B-Instruct
To download and start running a pre-built docker container, you may use the following commands:
cd llama-stack/distributions/meta-reference-gpu
docker run -it -p 5000:5000 -v ~/.llama:/root/.llama -v ./run.yaml:/root/my-run.yaml --gpus=all distribution-meta-reference-gpu --yaml_config /root/my-run.yaml
Tip
Pro Tip: We may use docker compose up for starting up a distribution with remote providers (e.g. TGI) using llamastack-local-cpu. You can checkout these scripts to help you get started.
-
Build->Configure->Run Llama Stack server via conda:
You may also build a LlamaStack distribution from scratch, configure it, and start running the distribution. This is useful for developing on LlamaStack.
llama stack build- You'll be prompted to enter build information interactively.
llama stack build > Enter an unique name for identifying your Llama Stack build distribution (e.g. my-local-stack): my-local-stack > Enter the image type you want your distribution to be built with (docker or conda): conda Llama Stack is composed of several APIs working together. Let's configure the providers (implementations) you want to use for these APIs. > Enter the API provider for the inference API: (default=meta-reference): meta-reference > Enter the API provider for the safety API: (default=meta-reference): meta-reference > Enter the API provider for the agents API: (default=meta-reference): meta-reference > Enter the API provider for the memory API: (default=meta-reference): meta-reference > Enter the API provider for the telemetry API: (default=meta-reference): meta-reference > (Optional) Enter a short description for your Llama Stack distribution: Build spec configuration saved at ~/.conda/envs/llamastack-my-local-stack/my-local-stack-build.yaml You can now run `llama stack configure my-local-stack`llama stack configure- Run
llama stack configure <name>with the name you have previously defined inbuildstep.
llama stack configure <name>- You will be prompted to enter configurations for your Llama Stack
$ llama stack configure my-local-stack Configuring API `inference`... === Configuring provider `meta-reference` for API inference... Enter value for model (default: Llama3.1-8B-Instruct) (required): Do you want to configure quantization? (y/n): n Enter value for torch_seed (optional): Enter value for max_seq_len (default: 4096) (required): Enter value for max_batch_size (default: 1) (required): Configuring API `safety`... === Configuring provider `meta-reference` for API safety... Do you want to configure llama_guard_shield? (y/n): n Do you want to configure prompt_guard_shield? (y/n): n Configuring API `agents`... === Configuring provider `meta-reference` for API agents... Enter `type` for persistence_store (options: redis, sqlite, postgres) (default: sqlite): Configuring SqliteKVStoreConfig: Enter value for namespace (optional): Enter value for db_path (default: /home/xiyan/.llama/runtime/kvstore.db) (required): Configuring API `memory`... === Configuring provider `meta-reference` for API memory... > Please enter the supported memory bank type your provider has for memory: vector Configuring API `telemetry`... === Configuring provider `meta-reference` for API telemetry... > YAML configuration has been written to ~/.llama/builds/conda/my-local-stack-run.yaml. You can now run `llama stack run my-local-stack --port PORT`llama stack run- Run
llama stack run <name>with the name you have previously defined.
llama stack run my-local-stack ... > initializing model parallel with size 1 > initializing ddp with size 1 > initializing pipeline with size 1 ... Finished model load YES READY Serving POST /inference/chat_completion Serving POST /inference/completion Serving POST /inference/embeddings Serving POST /memory_banks/create Serving DELETE /memory_bank/documents/delete Serving DELETE /memory_banks/drop Serving GET /memory_bank/documents/get Serving GET /memory_banks/get Serving POST /memory_bank/insert Serving GET /memory_banks/list Serving POST /memory_bank/query Serving POST /memory_bank/update Serving POST /safety/run_shield Serving POST /agentic_system/create Serving POST /agentic_system/session/create Serving POST /agentic_system/turn/create Serving POST /agentic_system/delete Serving POST /agentic_system/session/delete Serving POST /agentic_system/session/get Serving POST /agentic_system/step/get Serving POST /agentic_system/turn/get Serving GET /telemetry/get_trace Serving POST /telemetry/log_event Listening on :::5000 INFO: Started server process [587053] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://[::]:5000 (Press CTRL+C to quit)
Once the server is setup, we can test it with a client to see the example outputs.
cd /path/to/llama-stack
conda activate <env> # any environment containing the llama-stack pip package will work
python -m llama_stack.apis.inference.client localhost 5000
This will run the chat completion client and query the distribution’s /inference/chat_completion API.
Here is an example output:
User>hello world, write me a 2 sentence poem about the moon
Assistant> Here's a 2-sentence poem about the moon:
The moon glows softly in the midnight sky,
A beacon of wonder, as it passes by.
You may also send a POST request to the server:
curl http://localhost:5000/inference/chat_completion \
-H "Content-Type: application/json" \
-d '{
"model": "Llama3.1-8B-Instruct",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Write me a 2 sentence poem about the moon"}
],
"sampling_params": {"temperature": 0.7, "seed": 42, "max_tokens": 512}
}'
Output:
{'completion_message': {'role': 'assistant',
'content': 'The moon glows softly in the midnight sky, \nA beacon of wonder, as it catches the eye.',
'stop_reason': 'out_of_tokens',
'tool_calls': []},
'logprobs': null}
Similarly you can test safety (if you configured llama-guard and/or prompt-guard shields) by:
python -m llama_stack.apis.safety.client localhost 5000
Check out our client SDKs for connecting to Llama Stack server in your preferred language, you can choose from python, node, swift, and kotlin programming languages to quickly build your applications.
You can find more example scripts with client SDKs to talk with the Llama Stack server in our llama-stack-apps repo.
Please see our Building a LLama Stack Distribution guide for more details on how to assemble your own Llama Stack Distribution.