More on the model support

[Anindyadeep]

So, I have these two models to fine-tune

1. CodeLlama-34B
2. Mistral-7B

CodeLlama derives the same architecture as of Llama. And Mistral is fairly new in the community. So, my question is can I use axolotl to fine-tune these two above models?

[winglian]

yes, CodeLlama sample configurations are here https://github.com/OpenAccess-AI-Collective/axolotl/tree/main/examples/code-llama
and Mistral configurations are provided here: https://github.com/OpenAccess-AI-Collective/axolotl/tree/main/examples/mistral

[Anindyadeep]

@winglian, thanks for the answer, it helps. However, the GPU memory requirements of the smaller model (both for fine-tuning and inference) are well written. However, I do not see the same for bigger models like 34B or 70B clearly. For example consider this:

The 34b variant does not fit on 24GB of VRAM - you will need something with +40 gb VRAM that also supports flash attention v2 - A6000 or A100 are good choices.

Here, it is saying that to use a 34 B model, I would require 40 +GB of VRAM. Is this for inference (only) or fine-tuning too? Also, does it also handle the multi-GPU training (I Believe yes, because I saw the deepspeed configs)?