feat(plugins): ai-proxy-plugin #12323
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
chore
tysoekong
force-pushed
the
feat/ai_proxy_plugin
branch
2 times, most recently
from
a074d1c to
d9b303e
Compare
flrgh
approved these changes
Jan 9, 2024
tysoekong
force-pushed
the
feat/ai_proxy_plugin
branch
from
2fc22cd to
3b1216c
Compare
|
@fffonion Anything else for me to do here? |
tysoekong
force-pushed
the
feat/ai_proxy_plugin
branch
3 times, most recently
from
c634c5c to
7d6b50a
Compare
tysoekong
force-pushed
the
feat/ai_proxy_plugin
branch
2 times, most recently
from
4b3f822 to
949a161
Compare
locao
approved these changes
Jan 17, 2024
kong/llm/drivers/cohere.lua
Outdated
|
|
||
| if type(body) == "table" then | ||
| body_string, err = cjson.encode(body) | ||
| if err then return nil, nil, "failed to parse body to json: " .. err end |
There was a problem hiding this comment.
Suggested change
| if err then return nil, nil, "failed to parse body to json: " .. err end | |
| if err then | |
| return nil, nil, "failed to parse body to json: " .. err | |
| end |
|
|
||
| local _M = {} | ||
|
|
||
| local auth_schema = { |
There was a problem hiding this comment.
The schemas could go to their own file to keep this file cleaner.
tysoekong
force-pushed
the
feat/ai_proxy_plugin
branch
from
77b23dd to
f5be8e6
Compare
tysoekong
force-pushed
the
feat/ai_proxy_plugin
branch
from
f83b2e9 to
a25535c
Compare
flrgh
added
the
cherry-pick kong-ee
|
Successfully created cherry-pick PR for |
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This commit transforms and proxies requests to a number of AI / LLM providers and models.
It adds a plugin that takes requests in one of a few defined and standardised formats, translates them to the configured target format, and then transforms the response back into a standard format.
The request/response formats are based on OpenAI, and conform to the JSON schema in the fixtures
There is a sample API specification to describe the suported formats also
The plugin controls:
config.model.options.upstream_url)kong.logentries, which will then output via any configured logging platform e.g. file-log, kafka-log...The reasoning for trying to flatten all provider formats, is that it allows standardised manipulation of the data before and after transmission.
It currently supports
v1/chatandv1/completionstyle requests for all available providers. Each provider implementation class is in its own module underkong.plugins.ai-proxy.drivers.You can also set mode
preserveto pass through, and the plugin will perform a best-effort to conform to the chosenconfig.model.provider, based on the configured header patterns and values. This may need re-thinking, please provide input on this.This implementation only supports REST-based full text responses, but streaming support could be added based upon adoption, as with Kong >3.0 we can now process WebSocket frames.
Ideal documentation (docs.konghq.com) is being finalised internally, will be ready soon.
Checklist
changelog/unreleased/kongorskip-changeloglabel added on PR if changelog is unnecessary. README.md