feat(ai-proxy): add streaming support and transformers

[tysoekong]

Summary

Adds "streaming" support to AI Proxy plugin.

Streaming is a mode where a client can specify "stream": true in their request, and the LLM server will stream each piece piece of the response text (usually token-by-token) as a server-sent event.

We need to capture each (batch of) event(s) in order to translate them back into our inference format, so that all providers are compatible with the same framework that our users will create on their side.

Where "streaming=false" requests proxy directly to the LLM, and look like this:

the new streaming framework captures each event, and sends the chunk back to the client, like this:

and then it exits early. Docs will describe the limitations of this (no response transformer, etc).

It will also count/estimate tokens for LLM services that have decided to not stream back the token utilisation counts when the message has completed...

Checklist

• The Pull Request has tests
• A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
• There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Can this get reviewed for code standard and design, whilst I'm writing tests and docs?

Issue reference

Fix #12680

https://konghq.atlassian.net/browse/KAG-4124

[tysoekong]

@RobSerafini @flrgh @locao It is done. This is (I really think...) the largest PR in AI plugins phase 2.

Is it okay for someone to do the quality/standards pass, whilst I'm writing the docs and the tests? Especially if big changes are suggested.

We can then meet in the middle.

[ttyS0e]

I have no idea why it thinks changelog isn't done.

[locao]

Hey @ttyS0e

I have no idea why it thinks changelog isn't done.

That's because you included:

jiras:
- "KAG-4126"

You can check the required format in this doc: https://github.com/Kong/gateway-changelog/blob/v1.0.0/README.md

[tysoekong]

@locao Yep I double-broke it myself, trying to figure out what it was. I fixed it now.

[RobSerafini]

@locao @kikito - can you help find reviewers for this PR?

[tysoekong]

I have absolutely NO IDEA where these extra 8 commits got picked up

[tysoekong]

@flrgh Fixed all coments

[tysoekong]

okay @flrgh NOW I think it all done

[flrgh]

There are more optimizations that could be made to the SSE loop in kong/llm/init.lua, but I'd rather not get into the weeds until there's good reason to.

This is looking ready to me. 👍

[team-gateway-bot]

Successfully created cherry-pick PR for master:

• https://github.com/kong/kong-ee/pull/8791