tensorrt-llm possibility?

[david-payjoy]

my use case is similar to exui. multi-gpu, 1 user, batch 1, want it to be as fast as possible. im wondering if there is a possibility / roadmap to integrate with tensorrt, but keeping the time to first token down with a conversation (so using something like the sessions) tabby or aphroditie work, but with each message it has to read the whole context again. so im trying to figure out if bringing sessions to aphroditte and silly tavern makes more sense, or bringing tensorrt-llm to exllamav2 is more feasible, or if it is too crazy an idea. thanks again for this amazing work. cheers. related on my dream board would be exllamav2 + tensorrt + speculative decoding +context/conversation caching, as I believe that would fly.

[turboderp]

It already has context caching and speculative decoding. I'm not really sure where TensorRT would fit in?

[david-payjoy]

im trying to get to what you've accomplished so far with [exui (sessions), exllamav2 (so fast!), speculative decoding(even faster!)]
plus.. parallel multi gpu (like tensorrt)

[david-payjoy]

also, seriously, amazing work, especially as a solo project