Ollama support for LLM backend

[rchan26]

Somewhat related to #65

In the past I've used Ollama for local inference for LLMs, would it be useful to add this support to the library? I'd be happy to work on this to add support to use an Ollama API endpoint for the LLM

[andimarafioti]

can ollama use APIs? There is a PR open for the API part.

[rchan26]

@andimarafioti I think the PR is only for OpenAI API. I was just suggesting to offer Ollama API support. In particular, it's possible to run a Ollama server (either locally on a slightly better machine) and then query it using the Ollama Rest API (like here).

The changes/additions would be similar to the open PR #81 but a class for Ollama API support and querying Ollama endpoints

[mattfro]

This would be great, when some people already run ollama. Like me :)

[rchan26]

happy to work on this at some point this week. there's added functionality to Ollama last week to run any GGUF model from the HF Hub too: https://huggingface.co/docs/hub/en/ollama