AIGateway Support

[shaneutt]

Problem Statement

We currently have several plugins for the Kong Gateway which enable access to large language models (LLMs):

Kong/kong#12341
Kong/kong#12340
Kong/kong#12337
Kong/kong#12336

When applied to a Gateway they create our "AI Gateway" for serving and managing traffic to well-known LLM providers like OpenAI's ChatGPT.

If you're using the KGO and your goal is to try out the "AI Gateway", there's a lot of configuration that you need to do manually, and CRDs you have to learn like KongPlugin.

Proposed Solution

Create an AIGateway CRD and controller which automates this configuration and management for the user with minimal upfront configuration. Ideally the user should just be able to add their API keys to a Secret, and then the rest of the YAML can just be applied in a one-liner, resulting in an endpoint where they can access their AI Gateway in the AIGateway.status.

This should start out as an experimental and gated feature, so that we can share it with users and incorporate feedback into its development.

Additional Information

This issue originates from Aha! KIC-I-53.

Acceptance Criteria

• a new AIGateway resource is created, which composes to Gateways, Secrets, KongPlugins, e.t.c.
• a new feature gateway "ai-gateway" is added which enables this feature. This feature is clearly flagged as experimental.

Tasks

Preview Give feedback

1. https://github.com/Kong/gateway-operator/pull/1396
2. https://github.com/Kong/gateway-operator/pull/1399
3. https://github.com/Kong/gateway-operator/issues/1429
4. https://github.com/Kong/gateway-operator/issues/1430
5. https://github.com/Kong/gateway-operator/pull/1573
6. Use GenerateName for owned resources
7. Implement update/patch management for sub-resources
8. Expand watch capabilities to map AIGateways to owned resource changes
9. Implement full status handling
10. Enable real integration tests?

[shaneutt]

Met with @mflendrich and @mheap this morning. We are going to make this an OSS feature, and we're going to try for release in v1.2.0 (if that doesn't pan out for some reason, v1.3.0). The team doesn't have capacity by itself to do this during these release cycles based on current commitments, so I'm going to join the team to help with this one.

Note: This feature requires Gateway v3.6.0, which we currently expect to release February 12th, however this
shouldn't block development as the relevant features are available in current release candidates.

/cc @tysoekong

[mheap]

We don't have a confirmed date for v1.3.0, so I'd recommend pushing for v1.2.0