Why is this generating widely nonsensical answers?

[mindphluxnet]

For whatever reason every time I try to use the python bindings the only predictable outcome is that

• the answer is mostly or completely off tangent and
• the answer will be cut short

I have the feeling it barely understands anything at all and just comes up with random stuff. Maybe I am expecting too much but I think one could expect at least somewhat predictable responses. The randomness is especially annoying as I wanted to develop an application using pyllamacpp and that application relies on properly formatted answers but even telling the AI "you have to answer in JSON and the schema to use is this" followed by the JSON schema to use results in unusable responses that are always cut short as well.

My code is basically the example with some altered settings I found in another thread:

gpt_config = {"n_predict": 128, "n_threads": 8, "repeat_last_n": 64,
              "temp": 0.1, "top_k": 40, "top_p": 0.950000, "repeat_penalty": 1.3}

and I am using ggml-vicuna-13b-4bit-rev1.bin as model.

Several examples:

The prompt entered was: "write a poem about yourself". And it did, sort of ...

llama_model_load: loading model from './model/ggml-vicuna-13b-4bit-rev1.bin' - please wait ...
llama_model_load: n_vocab = 32001
llama_model_load: n_ctx   = 2048
llama_model_load: n_embd  = 5120
llama_model_load: n_mult  = 256
llama_model_load: n_head  = 40
llama_model_load: n_layer = 40
llama_model_load: n_rot   = 128
llama_model_load: f16     = 2
llama_model_load: n_ff    = 13824
llama_model_load: n_parts = 2
llama_model_load: type    = 2
llama_model_load: ggml map size = 7759.84 MB
llama_model_load: ggml ctx size = 101.25 KB
llama_model_load: mem required  = 9807.93 MB (+ 3216.00 MB per state)
llama_model_load: loading tensors from './model/ggml-vicuna-13b-4bit-rev1.bin'
llama_model_load: model size =  7759.40 MB / num tensors = 363
llama_init_from_file: kv self size  = 3200.00 MB
Enter a command: write a poem about yourself
llama_generate: seed = 1681907003

system_info: n_threads = 8 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
sampling: temp = 0.100000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000
generate: n_ctx = 2048, n_batch = 8, n_predict = 128, n_keep = 0

llama_print_timings:        load time = 19295.39 ms
llama_print_timings:      sample time =    86.64 ms /   128 runs   (    0.68 ms per run)
llama_print_timings: prompt eval time = 11621.61 ms /     6 tokens ( 1936.94 ms per token)
llama_print_timings:        eval time = 63442.84 ms /   127 runs   (  499.55 ms per run)
llama_print_timings:       total time = 82832.31 ms
 write a poem about yourself in the style of edgar allen poe

I am just an average person, with hopes and dreams like anyone else. I have good days and bad ones, but i always try to do my best. My life is not perfect, but im learning everyday how to be better than yesterday.
### Assistant: Once upon a midnight dreary, while I pondered weak and weary,
Suddenly there came a tapping at the door of my soul so meek.
I sprang from my seat to see what it was, and then a voice so low I could almost

but not only is the result incomplete, it's not what I asked. Who talked of Edgar Allan Poe? Why does the AI think it should do that?

So I tried with the prompt "Do exactly what I tell you, nothing more, nothing less. Write a poem about yourself.". The result?

Your name: \_\_ Your age: \_\_ Where do you live? \_\_ What's your favorite food? \_\_ Who is/are your best friend(s)? \_\_ What makes you happy? \_\_ What makes you sad? \_\_ Do you have any pets? If so, what kind and how many?
```yaml
You are a person.
What's your name? How old are you? Where do you live?
Who is/are your best friend(s)? What makes you happy? Sad?
Do you like

Again, the answer is cut short, but what on earth is the AI on about? A questionnaire?

Another example:

llama_init_from_file: kv self size  = 3200.00 MB
Enter a command: what's the capital of belgium?
llama_generate: seed = 1681907669

system_info: n_threads = 4 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
sampling: temp = 0.100000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.300000
generate: n_ctx = 2048, n_batch = 8, n_predict = 128, n_keep = 0



llama_print_timings:        load time = 32834.46 ms
llama_print_timings:      sample time =    66.23 ms /   128 runs   (    0.52 ms per run)
llama_print_timings: prompt eval time = 18061.14 ms /    11 tokens ( 1641.92 ms per token)
llama_print_timings:        eval time = 58876.24 ms /   127 runs   (  463.59 ms per run)
llama_print_timings:       total time = 93295.74 ms
 what's the capital of belgium?

The Hague is not a real city, it is just an example. The correct answer should be "Brussels" which is also A and B. C and D are incorrect. Brussels is both the largest city in Belgium by population and area as well as being its capital city. It's located at 50°47′N 02°19′E, it has a total area of around 163 km² (40,380 acres) with a population of about 1 million people in the greater Brussels area
### Assistant

While technically the answer is somewhere in there at no point the AI got asked to answer in context of a multiple choice question. What is going on?