-
Notifications
You must be signed in to change notification settings - Fork 605
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Gemini and Bedrock for AI Proxy #7831
Conversation
✅ Deploy Preview for kongdocs ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
## Configure Amazon Bedrock | ||
|
||
1. From the [AWS Management Console](https://aws.amazon.com/console/), navigate to Amazon Bedrock. | ||
1. Create an [Amazon Bedrock Model](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html). You will need to select a model type, configure model parameters, and deploy it. <!-- what kind of model do I want? custom? foundational? does it matter or can I choose what I want to use?--> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Model support are listed here https://docs.aws.amazon.com/bedrock/latest/userguide/models-regions.html
|
||
1. From the [AWS Management Console](https://aws.amazon.com/console/), navigate to Amazon Bedrock. | ||
1. Create an [Amazon Bedrock Model](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html). You will need to select a model type, configure model parameters, and deploy it. <!-- what kind of model do I want? custom? foundational? does it matter or can I choose what I want to use?--> | ||
1. Configure any necessary settings such as model parameters, input/output formats, and resource requirements. The configuration details can be found in the [Amazon Bedrock API Reference](https://docs.aws.amazon.com/bedrock/latest/APIReference/). <!-- are there any that we recommend configuring? If so, what guidance can we provide users so they know what to exactly put in the configuration, like example or actual values?--> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@tysoekong could you provide an example configuration that just works?
1. Create an [Amazon Bedrock Model](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html). You will need to select a model type, configure model parameters, and deploy it. <!-- what kind of model do I want? custom? foundational? does it matter or can I choose what I want to use?--> | ||
1. Configure any necessary settings such as model parameters, input/output formats, and resource requirements. The configuration details can be found in the [Amazon Bedrock API Reference](https://docs.aws.amazon.com/bedrock/latest/APIReference/). <!-- are there any that we recommend configuring? If so, what guidance can we provide users so they know what to exactly put in the configuration, like example or actual values?--> | ||
1. Navigate to the IAM (Identity and Access Management) section in the AWS Console to create or manage API keys and permissions. You will need to save your API key and endpoint URL. | ||
1. Ensure that the IAM roles and policies associated with your Bedrock models have the correct permissions to allow access and interaction. For more information, see [Amazon's IAM User Guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/). <!-- which do we recommend or need, or is this entirely up to the user?--> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this might be related to put as a link in our docs https://docs.aws.amazon.com/bedrock/latest/userguide/security_iam_service-with-iam.html
..._hub/kong-inc/ai-proxy/how-to/ai-and-machine-learning-platform-integration-guides/bedrock.md
Outdated
Show resolved
Hide resolved
…es and such Signed-off-by: Diana <[email protected]>
Signed-off-by: Diana <[email protected]>
Signed-off-by: Diana <[email protected]>
Signed-off-by: Diana <[email protected]>
17633c8
to
ae9a75e
Compare
…to AI Proxy Advanced, remove draft Bedrock guide Signed-off-by: Diana <[email protected]>
@@ -92,7 +99,8 @@ The following upstream URL patterns are used: | |||
| Anthropic | `https://api.anthropic.com:443/{route_type_path}` | | |||
| Llama2 | As defined in `config.model.options.upstream_url` | | |||
| Mistral | As defined in `config.model.options.upstream_url` | | |||
|
|||
| Amazon Bedrock | Use the specific LLM upstream URL | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Based on what we did for Gemini, I don't think this is right. We found the Gemini URL in a lua schema somewhere? But I don't remember where or which one and I think I saw the Bedrock one there as well, but couldn't remember what it was.
…tream URL formatting so it's not a link
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add an introduction, maybe something like what we have on the cohere plugin:
This guide walks you through setting up the AI Proxy Advanced plugin with [Cohere](https://cohere.com/).
For all providers, the Kong AI Proxy Advanced plugin attaches to route entities.
It can be installed into one route per operation, for example:
OpenAI chat route
Cohere chat route
Cohere completions route
Each of these AI-enabled routes must point to a null service. This service doesn’t need to map to any real upstream URL, it can point somewhere empty (for example, http://localhost:32000), because the plugin overwrites the upstream URL. This requirement will be removed in a later Kong revision.
Signed-off-by: Diana <[email protected]>
@Guaris I think this is a limitation of the plugin hub nav formatting. It pulls from the folder name, so I don't have control over it AFAIK. The same problem happens with the LLM providers section name. |
@Guaris Thanks! I added introduction paragraphs similar to the one you listed. I omitted the "doesn't need to map to a real upstream" part, since we did have to use the real upstream for Gemini. |
You can remove the AI portion out, and call it "machine learning platform integration" @cloudjumpercat |
Signed-off-by: Diana <[email protected]>
Signed-off-by: Diana <[email protected]>
Co-authored-by: Angel <[email protected]>
Ignoring the broken link checker, the Kong Community link it's failing on works for me, so I don't know why it's failing |
Description
DOCU-3863
Testing instructions
Preview link:
Checklist