Using another model to run in AutoGPT

[ghost]

I have a local model running on my computer and I was wondering if I am able to use that to power AutoGPT instead of the OpenAI model. Is this possible? And if so how do I set this up?

[InvestMaestri]

Hi @Hydra84034 I have been able to run local models and redirect OpenAI requests.

I am using a program called "LM Studio" (https://lmstudio.ai/). With it you can run a local inference server and it accepts the openai formatted requests from AutoGPT.

So, all I had to do was install LM Studio, download and locally run the model I wanted, then changed the .env file to redirect the OPENAI_API_BASE_URL to local host, port 1234/v1.

`## OPENAI_API_BASE_URL - Custom url for the OpenAI API, useful for connecting to custom backends. No effect if USE_AZURE is true, leave blank to keep the default url

the following is an example:

OPENAI_API_BASE_URL=http://localhost:443/v1

OPENAI_API_BASE_URL=http://localhost:1234/v1`

The cool thing is that you can watch the server log and see the model running.

Do note that this is VERY intensive. I am running on a GTX 1080ti, Threadripper 2950x, and 64GB of ram and it takes a while.
This is not what this computer was designed to do anyways. But just so you have an idea. I'll try to test it on another computer I have that has an RTX 2080ti and a i9-9900K to see how it performs. I did have a GPU server, but it is now pretty outdated.

[gigogosya]

Hi !
Please help!
Can you tell me what I need to do to make it work?

I uncomented and set:
OPENAI_API_BASE_URL= http://localhost:1234/v1
OPENAI_API_KEY= lm-studio

and to be sure:
SMART_LLM=TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q6_K.gguf ,
FAST_LLM=TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q6_K.gguf

But it has no effect. All the time I getting:
raise APIConnectionError(request=request) from err
openai.APIConnectionError: Connection error.

[gigogosya]

I solved it with adding a new block.

Solution:

1. Run LM Studio server (ai assistant mode)

2. Open " AutoGPT-master\rnd\autogpt_server\autogpt_server\blocks\agent.py " (backup it first)

3. Add at the begining:

import openai
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1235/v1", api_key="lm-studio")

4. Replace the "base_url" and "api_key" if necessary

5. Replace "get_result" function with this:

def get_result(task: str, input_data: str) -> str:
    # Настройка клиента для работы с LM Studio / Setup
    #openai.api_base = "http://localhost:1235/v1" #<rudiment>
    #openai.api_key = "lm-studio" #<rudiment>
    # Подготовка истории сообщений (для контекста) / Preparing chat history for context
    history = [
        {"role": "system", "content": "You are an intelligent assistant. You always provide well-reasoned answers that are both correct and helpful."},
        {"role": "user", "content": f"Task: {task}"},
        {"role": "user", "content": f"Input: {input_data}"},
    ]

    # Запрос в LM Studio / LM Studio call
    #while True:
    completion = client.chat.completions.create(
        model="TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q6_K.gguf",
        messages=history,
        temperature=0.7,
        stream=True,
    )

        # Обработка потокового ответа / Processing request
    new_message = {"role": "assistant", "content": ""}
    for chunk in completion:
        if chunk.choices[0].delta.content:
            print(chunk.choices[0].delta.content, end="", flush=True)
            new_message["content"] += chunk.choices[0].delta.content
    #history.append(new_message)
    return new_message["content"]

6. Replace the "model" you use (ctrl+f, TheBloke/Mistral-7B-Instruct-v0.2-GGUF/mistral-7b-instruct-v0.2.Q6_K.gguf).

The code is not perfect, so I am not a programmer. But it works well.