Saving GGUF for Ollama: CUDA driver error: out of memory

[criogennn]

Is it possible that my video memory is sufficient for training the model but insufficient for saving it in the GGUF format? I have an RTX 3050 with 8 GB of VRAM. I receive the error "CUDA driver error: out of memory" when running:

model.save_pretrained_gguf("model", tokenizer, quantization_method="q4_k_m")

Does this error necessarily indicate a lack of memory, or could it mean something else? I would appreciate any assistance.

[danielhanchen]

@criogennn Much apologies on the delay - could you try

model.save_pretrained_gguf("model", tokenizer, quantization_method="q4_k_m", maximum_memory_usage = 0.7)