[Feature] Response Metrics

[nathan-az]

Motivation

Response metrics are very useful for benchmarking performance of different configurations. LMDeploy could implement similar metrics to vLLM's RequestMetrics.

I think adding basic metrics like first token time, finish time, etc. should be pretty straightforward for AsyncEngine from skimming the sourcecode. I'm not sure if there are other areas where changes would be required.

If there is interest, I am happy to make a PR.

[lvhan028]

LMDeploy has a related PR: #1423
But we haven't updated it for a long time :(
Is it what you are looking for?

[nathan-az]

Yes that's exactly what I was referring to (a more comprehensive version with a new endpoint and prometheus integration 😅)!

If @AllentDan is still active I'll monitor progress on that PR :) One suggestion would also be to measure "time to first token" on a generation since it's also a good indicator of the prefill duration!