Attempted Fix: Sometimes tool_calls is undefined and crashes server

[Ademsk1]

Hi

This might help with issue #20 . I'm hoping to test it for openai later on today, but I don't have access to a local language model right now, are there any models you recommend I use?

Thanks

[mishushakov]

Nice. Have you tested it?

[Ademsk1]

@mishushakov tested for the openai section, doesn't crash the server anymore :D

I haven't tested the Llama one, but I think it should work the same. I started working on one last night, so will let you know of the results later on this evening.

[mishushakov]

Very good! I will run some tests too. Looking forward to get it merged 😄

[Ademsk1]

@mishushakov Been trying to test using this model as my model, but my machine cannot handle it. What model are you using?

[mishushakov]

TinyLlama

[mishushakov]

I can test for you, no worries

[Ademsk1]

Tested this with TinyLlama and no crashes :D

that being said, there weren't any crashes without the try/catch block for the local model, because we don't reference any tool_calls or the like. Still I think it's practical to have it there just in case.

[Ademsk1]

While I have you @mishushakov , maybe you have more experience, with these models, I seem to get this error when trying to scrape data from a fairly large website.
GGML_ASSERT: ... n_tokens_all <= cparams.n_batch
Did you have this error at all? Any advice?

[mishushakov]

I believe you're hitting the input token limit. We need to think about a way of splitting websites into multiple chunks, maybe. Can you open an issue for this?