[benchmark] optimize benchmark: counting tokenlizer tokens and error requests #1607
Conversation
| timestamps.append(time.perf_counter()) | ||
|
|
||
| first_token_latency = np.round(timestamps[1] - timestamps[0], 3) | ||
| token_latency = np.round(timestamps[-1] - timestamps[0], 3) | ||
| # assert output.pop('finish_reason') == 'length', \ | ||
| # f'Error. session_id({session_id}) request {output_seqlen} ' \ | ||
| # f'tokens, but `finish_reason` is not `length`' | ||
| total_tokens = input_seqlen + output_seqlen | ||
| tokenlizer_start = time.perf_counter() | ||
| real_output_seqlen = len(self.tokenizer(full_output).input_ids) |
There was a problem hiding this comment.
Encoding the text affects the inference performance.
I don't suggest doing that.
If the output seqlen is needed, the server can return it.
There was a problem hiding this comment.
In stream mode, this statistic is not returned.
In my tests(Qwen-72B-Chat), the tokenlizer takes only 0.027% of the whole benchmark elapsed time(tokenlizer speed: 77402.264 token/s for one concurrency), and the tokenlizer time has been removed in the final stats code.
stats = np.concatenate(stats).reshape(-1, 6)
tokenlizer_time = np.sum(stats[:, 5], axis=0) / concurrency
elapsed_time -= tokenlizer_timeThere was a problem hiding this comment.
Encoding the text affects the inference performance. I don't suggest doing that. If the output seqlen is needed, the server can return it.
lmdeploy doesn't support stream_options to get stats from server in stream mode yet.
NiuBlibing
changed the title
|
what kind of errors? |
Such as oom, account limits, etc. |
Motivation
BC-breaking (Optional)
None
Use cases (Optional)
For better benchmark.