Add more detailed metrics to the LLM benchmarks

[nv-hwoo]

Few changes

1. Support both online & offline LLM benchmarks (default is online)
2. Add more detailed metrics for latencies and throughputs

Users can run offline LLM benchmark by either setting --offline flag or setting "STREAM" input tensor to false through input JSON file.

Some new metrics added are

• Max/min first token latency
• p50/p90/p95/p99 first token latency
• Avg/max/min generation latency
• p50/p90/p95/p99 generation latency
• Avg token latency
• Avg/max/min end-to-end latency
• Avg/max/min token throughput
• p50/p90/p95/p99 token throughput
• etc

[matthewkotila]

Done with my review. Looks pretty good overall. Thanks for working on this, Hyunjae 🙏