[FEATURE] Extend support for Bedrock beyond Anthropic to all other models

[austintlee]

Is your feature request related to a problem?
Although it's quite possible that the current implementation of RAG works with all other models supported by Bedrock, we need to ensure it works with more than just Anthropic Claude.

What solution would you like?
A clear and concise description of what you want to happen.

What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

[ishaan-jaff]

Hi @austintlee @ylwu-amzn - I believe we can make this easier
I’m the maintainer of LiteLLM - we allow you to deploy an LLM proxy to call 100+ LLMs in 1 format - Bedrock, OpenAI, Anthropic etc https://github.com/BerriAI/litellm/tree/main/openai-proxy.

If this looks useful (we're used in production)- please let me know how we can help.

Usage

Bedrock request

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "bedrock/anthropic.claude-instant-v1",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

gpt-3.5-turbo request

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "gpt-3.5-turbo",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

claude-2 request

curl http://0.0.0.0:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
     "model": "claude-2",
     "messages": [{"role": "user", "content": "Say this is a test!"}],
     "temperature": 0.7
   }'

[austintlee]

@ishaan-jaff Yes, conceptually, something like this would be ideal. However, I am not sure about running a proxy. What I had in mind is to have this conversion layer in ml-commons itself. Are you familiar with OpenSearch? This might be a good use case for the OpenSearch Python extension (@dblock @dbwiddis). @ylwu-amzn I think of this as an all-in-one HttpConnector (but implemented in Python). It is certainly an interesting idea and I think it might be easier for people to use it if it's an extension (as in an extension of OpenSearch) as opposed to another thing people have to download and install and manage, although the quickest way to put this in the hands of the users is to have the HttpConnector talk to this proxy as-is.

[ylwu-amzn]

@ishaan-jaff Thanks for sharing this. The consistent input/output feature definitely helpful.

the quickest way to put this in the hands of the users is to have the HttpConnector talk to this proxy as-is.

Agree with @austintlee , seems this proxy could be some layer between OpenSearch HttpConnector and LLM, which adapts the HttpConnector and LLM.

The other option is we can use similar way to support consistent input/output in ml-commons with some adaption logic, but no proxy process

[ishaan-jaff]

@austintlee @ylwu-amzn how can we be most helpful here?

sounds like there are two options:

• use the litellm package for consistent LLM I/O
• use the litellm proxy server for consistent LLM I/O

[austintlee]

The first option won't work because we don't have a language binding for Python or to any language other than Java.

The second option works out of the box (in theory). Maybe you can try out a few examples and one way to contribute or let people know about it is to write a blog post?

[austintlee]

#2826 covers this.