Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Gemini and Bedrock for AI Proxy #7831

Merged
merged 12 commits into from
Sep 11, 2024
Merged

Conversation

cloudjumpercat
Copy link
Contributor

@cloudjumpercat cloudjumpercat commented Sep 3, 2024

Description

  • Added support for Gemini and Bedrock to tables
  • Added rough draft tutorials for Gemini and Bedrock

DOCU-3863

Testing instructions

Preview link:

Checklist

@cloudjumpercat cloudjumpercat added the review:sme Request for SME review, external to the docs team. label Sep 3, 2024
@cloudjumpercat cloudjumpercat added this to the Gateway 3.8 milestone Sep 3, 2024
@cloudjumpercat cloudjumpercat requested a review from a team as a code owner September 3, 2024 20:20
Copy link

netlify bot commented Sep 3, 2024

Deploy Preview for kongdocs ready!

Name Link
🔨 Latest commit 80df03b
🔍 Latest deploy log https://app.netlify.com/sites/kongdocs/deploys/66e1e60cc212d4000853a0df
😎 Deploy Preview https://deploy-preview-7831--kongdocs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
Lighthouse
Lighthouse
9 paths audited
Performance: 90 (🔴 down 1 from production)
Accessibility: 92 (no change from production)
Best Practices: 98 (🟢 up 8 from production)
SEO: 91 (no change from production)
PWA: -
View the detailed breakdown and full score reports

To edit notification comments on pull requests, go to your Netlify site configuration.

## Configure Amazon Bedrock

1. From the [AWS Management Console](https://aws.amazon.com/console/), navigate to Amazon Bedrock.
1. Create an [Amazon Bedrock Model](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html). You will need to select a model type, configure model parameters, and deploy it. <!-- what kind of model do I want? custom? foundational? does it matter or can I choose what I want to use?-->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


1. From the [AWS Management Console](https://aws.amazon.com/console/), navigate to Amazon Bedrock.
1. Create an [Amazon Bedrock Model](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html). You will need to select a model type, configure model parameters, and deploy it. <!-- what kind of model do I want? custom? foundational? does it matter or can I choose what I want to use?-->
1. Configure any necessary settings such as model parameters, input/output formats, and resource requirements. The configuration details can be found in the [Amazon Bedrock API Reference](https://docs.aws.amazon.com/bedrock/latest/APIReference/). <!-- are there any that we recommend configuring? If so, what guidance can we provide users so they know what to exactly put in the configuration, like example or actual values?-->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tysoekong could you provide an example configuration that just works?

1. Create an [Amazon Bedrock Model](https://docs.aws.amazon.com/bedrock/latest/userguide/what-is-bedrock.html). You will need to select a model type, configure model parameters, and deploy it. <!-- what kind of model do I want? custom? foundational? does it matter or can I choose what I want to use?-->
1. Configure any necessary settings such as model parameters, input/output formats, and resource requirements. The configuration details can be found in the [Amazon Bedrock API Reference](https://docs.aws.amazon.com/bedrock/latest/APIReference/). <!-- are there any that we recommend configuring? If so, what guidance can we provide users so they know what to exactly put in the configuration, like example or actual values?-->
1. Navigate to the IAM (Identity and Access Management) section in the AWS Console to create or manage API keys and permissions. You will need to save your API key and endpoint URL.
1. Ensure that the IAM roles and policies associated with your Bedrock models have the correct permissions to allow access and interaction. For more information, see [Amazon's IAM User Guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/). <!-- which do we recommend or need, or is this entirely up to the user?-->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cloudjumpercat cloudjumpercat force-pushed the feat/gemini-bedrock-ai-proxy branch from 17633c8 to ae9a75e Compare September 11, 2024 16:43
…to AI Proxy Advanced, remove draft Bedrock guide

Signed-off-by: Diana <[email protected]>
@@ -92,7 +99,8 @@ The following upstream URL patterns are used:
| Anthropic | `https://api.anthropic.com:443/{route_type_path}` |
| Llama2 | As defined in `config.model.options.upstream_url` |
| Mistral | As defined in `config.model.options.upstream_url` |

| Amazon Bedrock | Use the specific LLM upstream URL |
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on what we did for Gemini, I don't think this is right. We found the Gemini URL in a lua schema somewhere? But I don't remember where or which one and I think I saw the Bedrock one there as well, but couldn't remember what it was.

@cloudjumpercat cloudjumpercat added review:copyedit Request for writer review. and removed review:sme Request for SME review, external to the docs team. labels Sep 11, 2024
Copy link
Contributor

@Guaris Guaris left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add an introduction, maybe something like what we have on the cohere plugin:

This guide walks you through setting up the AI Proxy Advanced plugin with [Cohere](https://cohere.com/).

For all providers, the Kong AI Proxy Advanced plugin attaches to route entities.

It can be installed into one route per operation, for example:

OpenAI chat route
Cohere chat route
Cohere completions route
Each of these AI-enabled routes must point to a null service. This service doesn’t need to map to any real upstream URL, it can point somewhere empty (for example, http://localhost:32000), because the plugin overwrites the upstream URL. This requirement will be removed in a later Kong revision.

@Guaris
Copy link
Contributor

Guaris commented Sep 11, 2024

image
This i is lowercased as well.

@cloudjumpercat
Copy link
Contributor Author

image This i is lowercased as well.

@Guaris I think this is a limitation of the plugin hub nav formatting. It pulls from the folder name, so I don't have control over it AFAIK. The same problem happens with the LLM providers section name.

@cloudjumpercat
Copy link
Contributor Author

Can we add an introduction, maybe something like what we have on the cohere plugin:

This guide walks you through setting up the AI Proxy Advanced plugin with [Cohere](https://cohere.com/).

For all providers, the Kong AI Proxy Advanced plugin attaches to route entities.

It can be installed into one route per operation, for example:

OpenAI chat route
Cohere chat route
Cohere completions route
Each of these AI-enabled routes must point to a null service. This service doesn’t need to map to any real upstream URL, it can point somewhere empty (for example, http://localhost:32000), because the plugin overwrites the upstream URL. This requirement will be removed in a later Kong revision.

@Guaris Thanks! I added introduction paragraphs similar to the one you listed. I omitted the "doesn't need to map to a real upstream" part, since we did have to use the real upstream for Gemini.

@Guaris
Copy link
Contributor

Guaris commented Sep 11, 2024

image This i is lowercased as well.

@Guaris I think this is a limitation of the plugin hub nav formatting. It pulls from the folder name, so I don't have control over it AFAIK. The same problem happens with the LLM providers section name.

You can remove the AI portion out, and call it "machine learning platform integration" @cloudjumpercat

app/_data/docs_nav_gateway_3.8.x.yml Outdated Show resolved Hide resolved
@Guaris Guaris self-requested a review September 11, 2024 19:06
@cloudjumpercat
Copy link
Contributor Author

Ignoring the broken link checker, the Kong Community link it's failing on works for me, so I don't know why it's failing

@cloudjumpercat cloudjumpercat merged commit fd71c17 into main Sep 11, 2024
21 of 22 checks passed
@cloudjumpercat cloudjumpercat deleted the feat/gemini-bedrock-ai-proxy branch September 11, 2024 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants