Add pre-generated prompts option for benchmark #1091
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
omer-demir
force-pushed
the
omerdemir/pre_generated_prompts
branch
from
f8d1bdc to
0b19f52
Compare
omer-demir
force-pushed
the
omerdemir/pre_generated_prompts
branch
from
0b19f52 to
243e7ea
Compare
| @@ -232,6 +240,9 @@ def run_benchmark(args, batch_size, prompt_length, generation_length, max_length | |||
| # use random tokens instead of generating a prompt using the model and then tokenizing it | |||
| tokens = np.random.randint(100, size=(batch_size, prompt_length)) | |||
| prompt = [tokenizer.decode(tokens[0])] * batch_size | |||
| elif args.use_prompt_set: | |||
| prompt = get_prompt_by_length(prompt_length) | |||
| tokens = tokenizer.encode_batch(prompt) | |||
There was a problem hiding this comment.
Different tokenizers can encode prompts into different prompt lengths. Some additional work is needed to get the desired prompt length. You can see an example of how to do this here.
There was a problem hiding this comment.
so basically, we will check the tokens length against requested prompt_length and we will add/trim if it is needed. Is that correct?
| @@ -83,6 +83,14 @@ def generate_prompt(model, tokenizer, prompt_length, use_graph_capture) -> str: | |||
| generator.generate_next_token() | |||
| return tokenizer.decode(generator.get_sequence(0)) | |||
|
|
|||
| # Use prompt length to get pre-defined prompt | |||
| def get_prompt_by_length(prompt_length): | |||
| json_path = "prompts.json" | |||
There was a problem hiding this comment.
Instead of uploading another copy of prompts.json, can we download it from here and save it to disk using requests or urllib instead? That way, only one location has to be updated when adding other prompts.
There was a problem hiding this comment.
would it make sense to rely on external file? If that changes, we may have problem with this. Besides, in the benchmark environment, it seems logical to use local files instead of relying on internet connectivity. What do you think?
During benchmarking, we wanted to have pre-generated prompts that have been prepared for better benchmark result. Hence, It can be handy during benchmarking. In our test, we wanted to focus only token generation and sampling on SLM.