[need help] a simple python implementation of parallel.cpp

[littlebai3618]

I am in need of an HTTP API that supports continuous batch processing, so I have decided to implement it myself.

I encountered some issues while trying to implement continuous batch processing using the low-level llama.cpp API provided by this project. Therefore, I have posted my implementation here to seek help.
i mainly refer to：https://github.com/ggerganov/llama.cpp/blob/master/examples/parallel/parallel.cpp

I am experiencing a possible memory leak when performing continuous batch processing with large contexts and batches.

I have raised two separate issues, one in this repository (llama.cpp) and another in llama-cpp-python, to provide more information about the problem.

1. llama.cpp issue: #4086
2. llama-cpp-python issue: #924

Welcome to point out any errors, and I will fix them as soon as possible.

notice:
this demo not support grammar、 terminal args、 prompt file