[Connectors][GenAI] Inference Service Kibana connector #189027

YulNaumenko · 2024-07-24T05:29:12Z

Summary

This PR adds new connector which is define integration with Elastic Inference Endpoint via Inference APIs
The lifecycle of the Inference Endpoint are managed by the connector registered handlers:

preSaveHook - create new Inference Endpoint in the connector create mode (isEdit === false) and delete+create in the connector edit mode (isEdit === true)
postSaveHook - check if the connector SO was created/updated and if not removes Inference Endpoint from preSaveHook
postDeleteHook - delete Inference Endpoint if connector was deleted.

In the Kibana Stack Management Connectors, its represented with the new card (Technical preview badge):

To simplify the future integration with AI Assistants, the Connector consists from the two main UI parts: provider selector and required provider settings, which will be always displayed

and Additional options, which contains optional provider settings and Task Type configuration:

subActions corresponds to the different taskTypes Inference API supports. Each of the task type has its own Inference Perform params. Currently added:

completion & completionStream
rerank
text_embedding
sparse_embedding

Follow up work:

Collapse/expand Additional options, when the connector flyout/modal has AI Assistant as a context (path through the extending context implementation on the connector framework level)
Add support for additional params for Completion subAction to be able to path functions
Add support for tokens usage Dashboard, when inference API will include the used tokens count in the response
Add functionality and UX for migration from existing specific AI connectors to the Inference connector with proper provider and completion task
Integrate Connector with the AI Assistants

github-actions · 2024-07-24T05:29:26Z

A documentation preview will be available soon.

🔨 Buildkite builds
📚 HTML diff
📙 Preview page

Request a new doc build by commenting

Rebuild this PR: run docs-build
Rebuild this PR and all Elastic docs: run docs-build rebuild

_{run docs-build is much faster than run docs-build rebuild. A rebuild should only be needed in rare situations.}

_{If your PR continues to fail for an unknown reason, the doc build pipeline may be broken. Elastic employees can check the pipeline status here.}

daveyholler · 2024-07-25T12:49:17Z

Some feedback from me, thanks for the video demo

@serenachou is there a link to said video demo?

serenachou · 2024-07-25T12:53:18Z

@daveyholler it's in the slack thread I added you and Julian to : https://elastic.slack.com/archives/C07DCCPGB55/p1721799374980719

leemthompo · 2024-07-25T13:49:43Z

For simplicity, once Serena's suggestions have been drafted, please re-ping and we can copyedit from there :)

YulNaumenko · 2024-08-08T02:07:15Z

The typography (labels and headings) is a little wonky in the flyout.

I cannot change it, because it's a part of the connectors framework and UX consistency.

Your buttons (Save & Test, Save) will likely need to move into the flyout footer and use the primary blue.

Buttons are also inherited from the connectors framework and there is no way to manage it's styles.
I also cannot make the Connector details group, because current "Connector settings" also belongs to the framework.

…o being able to manage inference endpoint lifecicle with hooks

leemthompo

Few copyedits per @serenachou's drafts

x-pack/plugins/stack_connectors/public/connector_types/inference/connector.tsx

x-pack/plugins/stack_connectors/public/connector_types/inference/inference.tsx

…rence-connector # Conflicts: # docs/settings/alert-action-settings.asciidoc # oas_docs/output/kibana.serverless.yaml # oas_docs/output/kibana.yaml # x-pack/plugins/actions/docs/openapi/bundled.yaml

…rence-connector

.github/CODEOWNERS

…rence-connector # Conflicts: # oas_docs/output/kibana.serverless.yaml # oas_docs/output/kibana.yaml

serenachou · 2024-09-10T12:53:12Z

@YulNaumenko will you have an updated video / demo of this based on the latest PRs you've put in?

YulNaumenko · 2024-10-10T22:04:14Z

Ya, what Yuliia said is correct - I have no idea if not having pre-defined connectors will be a problem for someone.
I think the problem from a response-ops framework side is that this is apparently a connector type that you should not pre-define, and we don’t (currently) have a way of preventing that.
I guess it depends on how obvious the connector is broken in the UX, or responds to execution requests - we'd probably want to indicate in error notifications that these can't be pre-defined. Could also log something at startup if they are found.

@pmuellr Actually we need to be able to to create pre-configured connectors for Inference connector, at least the one for Default Elastic LLM 😀
I was thinking to extend the documentation, with manual handle of Inference Endpoint lifecycle in case of pre-configured connector

pmuellr · 2024-10-10T22:14:24Z

@pmuellr Actually we need to be able to to create pre-configured connectors for Inference connector, at least the one for Default Elastic LLM 😀 I was thinking to extend the documentation, with manual handle of Inference Endpoint lifecycle in case of pre-configured connector

We could add some plugin start up code that looks for these and creates the inference, but you'd likely want this in a task so only one Kibana will run in. Or maybe the interference endpoint is idempotent and you can just call it multiple times with the same arguments?

Not clear how the inference delete would happen though.

Curious if this is something that WE provide to the customer, or something an on prem customer can do themselves. If it's something WE provide, in cloud, then the params / secrets will need to be baked in, like the email proxy is, in some cloud configuration. And it seems like some cloud startup code could also create the inference? Maybe that's too far removed from Kibana though.

Note that customers can not create preconfigured connectors in any of our hosted cloud envs (ECH, serverless), but can "on prem" offerings.

szabosteve

Left some smaller suggestions – all committable –, please take or leave them.

docs/management/connectors/action-types/inference.asciidoc

szabosteve · 2024-10-11T09:25:45Z

docs/management/connectors/action-types/inference.asciidoc

+
+[role="screenshot"]
+image::management/connectors/images/inference-connector.png[{inference} connector]
+// NOTE: This is an autogenerated screenshot. Do not edit it directly.


Is it really auto-generated or is this string a copy/paste error?

docs/management/connectors/action-types/inference.asciidoc

adcoelho · 2024-10-11T10:13:43Z

I don't think the fields are being cleared properly when changing between the different services in the form.

Screen.Recording.2024-10-11.at.12.05.00.mov

If I edit the field Model for Amazon Bedrock and then change the service to Mistral the field will be populated with the same value. Maybe that is intended but if I try to save the form I'll get a Field is required. error even though there is text there.

(This happens not just with text but with all fields with the same name across services. Model ID, API Key, etc)

adcoelho

Code-wise approved but I left some comments with bugs I found and with suggested changes to tests.

x-pack/plugins/stack_connectors/public/connector_types/inference/params.tsx

x-pack/plugins/stack_connectors/public/connector_types/inference/inference.test.tsx

adcoelho · 2024-10-11T11:25:11Z

x-pack/plugins/stack_connectors/public/connector_types/inference/params.tsx

+import { SUB_ACTION } from '../../../common/inference/constants';
+import { InferenceActionConnector, InferenceActionParams } from './types';
+
+const InferenceServiceParamsFields: React.FunctionComponent<


I wanted to test these but couldn't find them in Kibana 😅

Normally I'd create a rule and go to the actions but in this case I couldn't find the connector. How does this work?

If this is like the other gen-ai connectors, it can't be used as an alerting rule action. In theory you should be able to run it via the connector / test UX, but I think someone reported earlier this wasn't quite working.

The product uses this by interacting with the connector via the actionsClient directly. The actions plugin is used since it handles a lot of the issues that are similar to other connectors - common way to "configure" the service, all kinds of validation, checks within our axios usage for TLS/SSL / proxy / etc kind of configuration of the outgoing http connections, etc. It just can't be used as an alerting rule action.

mdefazio

One last edit that would be nice to have in:
When setting up a new elasticsearch model (ELSER or E5), is it possible to pre-populate the threads and allocations with 1 / 2 respectively so there's something in there for the user?

Not necessarily a blocker, but would be nice to have.

Update

When editing the connector, the threads/allocation fields are still available, but do not trigger the option to save. I think the expectation would be that these are either disabled / read-only, or would trigger a change. Is it correct that threads is updatable but allocations are not? (or do I have that switched)?

Co-authored-by: István Zoltán Szabó <[email protected]>

…umenko/kibana into kibana-generic-inference-connector

pmuellr

server-side Response Ops code looks good to me, looks like ux code was already reviewed by Response Ops

The comments here are mostly nits. I think we are REALLY going to want a function test where we update the config and secrets properties spec'd as objects with any allowed keys. We want to create the connector, update those properties such that we remove a property already there, and add one that's not there, to both config and secrets. Then do a GET and make sure we can still read the connector, and that it has the expected values. Then execute the connector - we want to make sure the secrets can still be decrypted - the connector will fail early if that's the case, and won't even call the connectors executor function.

pmuellr · 2024-10-11T19:52:31Z

x-pack/plugins/stack_connectors/public/connector_types/inference/get_task_types.test.ts

+
+beforeEach(() => jest.resetAllMocks());
+
+describe.skip('getTaskTypes', () => {


I assume the .skip should be removed? Or is the test not actually passing yet due to some other work not complete (on the ES side, maybe)?

Yes, when we will have a real API

pmuellr · 2024-10-11T19:55:20Z

x-pack/plugins/stack_connectors/public/connector_types/inference/helpers.ts

+  setFieldValue: (fieldName: string, value: unknown) => void
+) => {
+  const taskTypeSuffix = config.taskType ? `${config.taskType}-` : '';
+  const inferenceEndpointId = `${config.provider}-${taskTypeSuffix}${Math.random()


Sure you want to use Math.random() here, which is likely not "securely" random?

My first guess is you should create a v4 uuid, which is what we do in alerting for cases like this.

I did v4 uuid, but got feedback that it's too long and not user friendly, so was asked to use up to 8 symbols. It's not too strict regarding possible dups errors.

pmuellr · 2024-10-11T20:00:51Z

x-pack/plugins/stack_connectors/public/connector_types/inference/types.ts

+
+export type FieldsConfiguration = Record<string, ConfigProperties>;
+
+export interface Config {


The pattern I've seen folks do to access types from schema in public, is to keep the schema and it's types in server, and then import type those in the public code, right from server. You can only do import type like that, import by itself won't work (well, it will cause problems, but I think it does kinda work).

That will ensure there's a single definition for these types.

x-pack/plugins/stack_connectors/server/connector_types/inference/index.ts

pmuellr · 2024-10-11T20:07:02Z

x-pack/plugins/stack_connectors/server/connector_types/inference/index.ts

+    );
+  } catch (e) {
+    logger.warn(
+      `Failed to delete inference endpoint for task type "${taskType}" and inference id ${inferenceId}. Error: ${e.message}`


Our experience has been that there is never enough information written out for these errors :-)

It would be nice to log the status code at least, not clear if there might be other interesting bits in the error that would be useful to log.

pmuellr · 2024-10-11T20:11:24Z

x-pack/plugins/stack_connectors/server/connector_types/inference/index.ts

+        });
+        inferenceExists = true;
+      } catch (e) {
+        /* throws error if inference endpoint by id does not exist */


There are other reasons why an error could be thrown, even if it does exist. Probably would be good to specifically check if this is a "not found" error (we do this somewhere in alerting, we can look up the magic incantation) vs something else, and if it's something else, throw THAT error.

x-pack/plugins/stack_connectors/server/connector_types/inference/inference.ts

pmuellr · 2024-10-11T20:31:34Z

...st/alerting_api_integration/common/plugins/actions_simulators/server/inference_simulation.ts

+export class InferenceSimulator extends Simulator {
+  private readonly returnError: boolean;
+
+  constructor({ returnError = false, proxy }: { returnError?: boolean; proxy?: ProxyArgs }) {


For some of our simulators, we create a single simulator, and determine how to react based on the request coming in. Maybe a field is set to returnError or such, then it returns error instead of the response. May not work for this case, but thought I'd mention it.

pmuellr · 2024-10-11T20:36:29Z

...erting_api_integration/security_and_spaces/group2/tests/actions/connector_types/inference.ts

+        simulator.close();
+      });
+
+      it('should return 200 when creating the connector without a default model', async () => {


These tests should be also checking if the interference did or did not get created, right? And also testing some error conditions, I guess like the case where the same infererence would be created, and so the pre-save hook fails?

Seems like we will also want some delete tests, checking to make sure the inference gets deleted. Maybe one where the inference is ALREADY deleted, but not the connector. What happens then?

I opened a follow up ticket to add better tests coverage for this PR #196032

kibanamachine · 2024-10-13T20:39:28Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11317544203

elasticmachine · 2024-10-13T20:40:09Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: b829d6a

Failed CI Steps

FTR Configs #6

Test Failures

[job] [logs] FTR Configs #6 / Alerting builtin alertTypes es_query rule runs correctly: threshold on ungrouped hit count < > for searchSource search type

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`stackConnectors`	275	305	+30

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`integrationAssistant`	959.8KB	959.9KB	+132.0B
`securitySolution`	20.8MB	20.8MB	+264.0B
`stackConnectors`	584.1KB	681.3KB	+97.3KB
`triggersActionsUi`	1.6MB	1.6MB	+106.0B
total			+97.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`actions`	17.1KB	17.1KB	-11.0B
`stackConnectors`	56.1KB	58.3KB	+2.2KB
total			+2.2KB

Unknown metric groups

async chunk count

id	before	after	diff
`stackConnectors`	92	95	+3

ESLint disabled line counts

id	before	after	diff
`@kbn/test-suites-xpack`	726	727	+1
`stackConnectors`	119	129	+10
total			+11

miscellaneous assets size

id	before	after	diff
`stackConnectors`	0.0B	34.8KB	+34.8KB

Total ESLint disabled count

id	before	after	diff
`@kbn/test-suites-xpack`	751	752	+1
`stackConnectors`	125	135	+10
total			+11

History

💛 Build #241874 was flaky 9c20a55
💔 Build #241759 failed d3a8933
💔 Build #241655 failed 3fef5b8
💔 Build #241409 failed f75eb47
💔 Build #241389 failed 73432ee
💔 Build #241366 failed f2fe300

cc @YulNaumenko

## Summary Resolves elastic#188043 This PR adds new connector which is define integration with Elastic Inference Endpoint via [Inference APIs](https://www.elastic.co/guide/en/elasticsearch/reference/current/inference-apis.html) The lifecycle of the Inference Endpoint are managed by the connector registered handlers: - `preSaveHook` - [create](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-inference-api.html) new Inference Endpoint in the connector create mode (`isEdit === false`) and [delete](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-inference-api.html)+[create](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-inference-api.html) in the connector edit mode (`isEdit === true`) - `postSaveHook` - check if the connector SO was created/updated and if not removes Inference Endpoint from preSaveHook - `postDeleteHook` - [delete](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-inference-api.html) Inference Endpoint if connector was deleted. In the Kibana Stack Management Connectors, its represented with the new card (Technical preview badge): <img width="1261" alt="Screenshot 2024-09-27 at 2 11 12 PM" src="https://github.com/user-attachments/assets/dcbcce1f-06e7-4d08-8b77-0ba4105354f8"> To simplify the future integration with AI Assistants, the Connector consists from the two main UI parts: provider selector and required provider settings, which will be always displayed <img width="862" alt="Screenshot 2024-10-07 at 7 59 09 AM" src="https://github.com/user-attachments/assets/87bae493-c642-479e-b28f-6150354608dd"> and Additional options, which contains optional provider settings and Task Type configuration: <img width="861" alt="Screenshot 2024-10-07 at 8 00 15 AM" src="https://github.com/user-attachments/assets/2341c034-6198-4731-8ce7-e22e6c6fb20f"> subActions corresponds to the different taskTypes Inference API supports. Each of the task type has its own Inference Perform params. Currently added: - completion & completionStream - rerank - text_embedding - sparse_embedding Follow up work: 1. Collapse/expand Additional options, when the connector flyout/modal has AI Assistant as a context (path through the extending context implementation on the connector framework level) 2. Add support for additional params for Completion subAction to be able to path functions 3. Add support for tokens usage Dashboard, when inference API will include the used tokens count in the response 4. Add functionality and UX for migration from existing specific AI connectors to the Inference connector with proper provider and completion task 5. Integrate Connector with the AI Assistants --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: István Zoltán Szabó <[email protected]> Co-authored-by: Liam Thompson <[email protected]> Co-authored-by: Steph Milovic <[email protected]> (cherry picked from commit 288d41d)

kibanamachine · 2024-10-13T20:44:04Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

#196035) # Backport This will backport the following commits from `main` to `8.x`: - [[Connectors][GenAI] Inference Service Kibana connector (#189027)](#189027)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Yuliia Naumenko <[email protected]>

[Connectors][GenAI] Inference Service connector

689b966

YulNaumenko added Feature:Actions/ConnectorTypes Issues related to specific Connector Types on the Actions Framework 8.16 candidate v8.16.0 labels Jul 24, 2024

YulNaumenko self-assigned this Jul 24, 2024

Merge branch 'main' into kibana-generic-inference-connector

dda4ccc

This comment was marked as resolved.

Sign in to view

YulNaumenko added 3 commits August 8, 2024 08:47

Ui updates

7098133

added support for preSaveConnector and postDeleteConnector handlers t…

1ae31ea

…o being able to manage inference endpoint lifecicle with hooks

task types service

d14f562

leemthompo reviewed Aug 13, 2024

View reviewed changes

YulNaumenko added 4 commits August 21, 2024 12:22

Merge remote-tracking branch 'upstream/main' into kibana-generic-infe…

b47e981

…rence-connector # Conflicts: # docs/settings/alert-action-settings.asciidoc # oas_docs/output/kibana.serverless.yaml # oas_docs/output/kibana.yaml # x-pack/plugins/actions/docs/openapi/bundled.yaml

Merge remote-tracking branch 'upstream/main' into kibana-generic-infe…

952784d

…rence-connector

-

424b791

reshuffle the ui

6d0f7b4

pgayvallet reviewed Aug 30, 2024

View reviewed changes

.github/CODEOWNERS Outdated Show resolved Hide resolved

YulNaumenko added 5 commits August 30, 2024 11:53

Merge remote-tracking branch 'upstream/main' into kibana-generic-infe…

556b8a1

…rence-connector # Conflicts: # oas_docs/output/kibana.serverless.yaml # oas_docs/output/kibana.yaml

Added provider select control

62adb55

changes the form elements

9d52a13

complete inference endpoint CRUD

04fb36e

added typed schema

ab2d2b9

added subActions

080fc76

pmuellr mentioned this pull request Oct 10, 2024

[ResponseOps][Connectors] prevent or allow connectors with pre/post save/delete hooks from being used as pre-configured connectors? #195824

Open

YulNaumenko added 2 commits October 10, 2024 16:12

build fix

73432ee

-

f75eb47

szabosteve reviewed Oct 11, 2024

View reviewed changes

adcoelho approved these changes Oct 11, 2024

View reviewed changes

mdefazio self-requested a review October 11, 2024 12:09

mdefazio reviewed Oct 11, 2024

View reviewed changes

YulNaumenko and others added 3 commits October 11, 2024 07:23

Apply suggestions from code review

3fef5b8

Co-authored-by: István Zoltán Szabó <[email protected]>

tests

6553e80

Merge branch 'kibana-generic-inference-connector' of github.com:YulNa…

d3a8933

…umenko/kibana into kibana-generic-inference-connector

pmuellr approved these changes Oct 11, 2024

View reviewed changes

YulNaumenko added 4 commits October 12, 2024 19:22

-

331e0dd

Merge branch 'main' into kibana-generic-inference-connector

9c20a55

fixed update with numbers

b870e1d

Merge branch 'main' into kibana-generic-inference-connector

b829d6a

YulNaumenko enabled auto-merge (squash) October 13, 2024 19:01

YulNaumenko merged commit 288d41d into elastic:main Oct 13, 2024
38 checks passed

kibanamachine mentioned this pull request Oct 13, 2024

[8.x] [Connectors][GenAI] Inference Service Kibana connector (#189027) #196035

Merged

kilfoyle added a commit to kilfoyle/kibana that referenced this pull request Oct 23, 2024

Add setting update from Kibana elastic#189027

5efed75

This was referenced Dec 3, 2024

[Security Solution] Integrate pre-configured Kibana AI Connector to the Security GenAI powered functionality #202625

Open

[AppEx AI] Integrate pre-configured Kibana AI Connector to the O11y and Search GenAI powered functionality #202626

Open

pmuellr mentioned this pull request Jan 6, 2025

Introduce Kibana task to deploy agentless connectors for 9.0 #203973

Open

7 tasks


		beforeEach(() => jest.resetAllMocks());

		describe.skip('getTaskTypes', () => {


		export type FieldsConfiguration = Record<string, ConfigProperties>;

		export interface Config {

[Connectors][GenAI] Inference Service Kibana connector #189027

[Connectors][GenAI] Inference Service Kibana connector #189027

Conversation

YulNaumenko commented Jul 24, 2024 • edited by kibanamachine Loading

Summary

github-actions bot commented Jul 24, 2024

This comment was marked as resolved.

daveyholler commented Jul 25, 2024

serenachou commented Jul 25, 2024

leemthompo commented Jul 25, 2024

This comment was marked as resolved.

YulNaumenko commented Aug 8, 2024

This comment was marked as resolved.

leemthompo left a comment

Choose a reason for hiding this comment

serenachou commented Sep 10, 2024

YulNaumenko commented Oct 10, 2024

pmuellr commented Oct 10, 2024

szabosteve left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adcoelho commented Oct 11, 2024

adcoelho left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdefazio left a comment • edited Loading

Choose a reason for hiding this comment

pmuellr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YulNaumenko Oct 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Oct 13, 2024

elasticmachine commented Oct 13, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Async chunks

Page load bundle

async chunk count

ESLint disabled line counts

miscellaneous assets size

Total ESLint disabled count

History

kibanamachine commented Oct 13, 2024

💚 All backports created successfully

Questions ?

YulNaumenko commented Jul 24, 2024 •

edited by kibanamachine

Loading

mdefazio left a comment •

edited

Loading

YulNaumenko Oct 13, 2024 •

edited

Loading