Skip to content

Commit

Permalink
[8.x] [Obs AI Assistant] Include an AdHoc instruction about the slack…
Browse files Browse the repository at this point in the history 
… connector to avoid executing a loop (#199531) (#199860)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Obs AI Assistant] Include an AdHoc instruction about the slack
connector to avoid executing a loop
(#199531)](#199531)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Viduni
Wickramarachchi","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-12T18:28:08Z","message":"[Obs
AI Assistant] Include an AdHoc instruction about the slack connector to
avoid executing a loop (#199531)\n\nCloses
https://github.com/elastic/kibana/issues/185028\r\n\r\n##
Summary\r\n\r\n### Problem\r\nThe ticket mentions that there was an
issue with displaying results.\r\nHowever both `display results` and
`visualize query` are working as\r\nexpected based on my investigation.
More details including a video
are\r\nattached\r\n[here](https://github.com/elastic/kibana/issues/185028#issuecomment-2464879452).\r\n\r\nThe
function calling loop seems to occur when the AI Assistant is
trying\r\nto send the output to Slack via the Kibana Slack connector. In
order to\r\ndo this, the LLM invokes the function `execute_connector`.
For the Slack\r\nconnector, `id` and `params` properties are required.
However, the LLM\r\nonly populated `id` and not `params` which causes an
error when\r\nvalidated against the schema for the Slack
connector.\r\n\r\n- Sometimes, it's able to retry a few times and
successfully send the\r\noutput to Slack.\r\n- Sometimes, it goes into a
loop trying to find `params` and failing\r\nrepeatedly.\r\n\r\nAttaching
another similar issue for more context
-\r\nhttps://github.com//issues/195564\r\n\r\n_With the
solution below, my intention is to send the output to Slack in\r\none
go, without retrying the function `execute_connector`._\r\n\r\n###
Solution\r\nBased on the solutions I experimented with, seems like we
need to force\r\nthe LLM to understand what's needed for the Slack
connector. I tried a\r\nfew options here and the combination of updates
that worked are as\r\nfollows:\r\n- Appending an AdHoc instruction about
the Slack connector properties.\r\n(This gives the LLM some additional
information about the required\r\nproperties)\r\n- Updating the
`properties` attached to the connector (when passing the\r\nconnector
list to the LLM), to reflect both `id` and `params`
with\r\n`message`.\r\n\r\nWith the above change, the AI Assistant has
managed to consistently send\r\nthe output to Slack _**without any
retries**_ because of missing\r\n`params`.\r\n\r\n###
Screenshots:\r\n(all alert triggers successfully sent the output to
Slack without having\r\nto retry the function)\r\n\r\n<img
width=\"1452\"
alt=\"success-attempts-to-slack-connector\"\r\nsrc=\"https://github.com/user-attachments/assets/715a5957-2c04-4a55-868f-34abe564f6d4\">","sha":"ed4c0df9ee304ea9bc71ed30c8e5dc2dd9295a00","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v9.0.0","Team:Obs
AI
Assistant","ci:project-deploy-observability","backport:version","v8.17.0"],"title":"[Obs
AI Assistant] Include an AdHoc instruction about the slack connector to
avoid executing a
loop","number":199531,"url":"https://github.com/elastic/kibana/pull/199531","mergeCommit":{"message":"[Obs
AI Assistant] Include an AdHoc instruction about the slack connector to
avoid executing a loop (#199531)\n\nCloses
https://github.com/elastic/kibana/issues/185028\r\n\r\n##
Summary\r\n\r\n### Problem\r\nThe ticket mentions that there was an
issue with displaying results.\r\nHowever both `display results` and
`visualize query` are working as\r\nexpected based on my investigation.
More details including a video
are\r\nattached\r\n[here](https://github.com/elastic/kibana/issues/185028#issuecomment-2464879452).\r\n\r\nThe
function calling loop seems to occur when the AI Assistant is
trying\r\nto send the output to Slack via the Kibana Slack connector. In
order to\r\ndo this, the LLM invokes the function `execute_connector`.
For the Slack\r\nconnector, `id` and `params` properties are required.
However, the LLM\r\nonly populated `id` and not `params` which causes an
error when\r\nvalidated against the schema for the Slack
connector.\r\n\r\n- Sometimes, it's able to retry a few times and
successfully send the\r\noutput to Slack.\r\n- Sometimes, it goes into a
loop trying to find `params` and failing\r\nrepeatedly.\r\n\r\nAttaching
another similar issue for more context
-\r\nhttps://github.com//issues/195564\r\n\r\n_With the
solution below, my intention is to send the output to Slack in\r\none
go, without retrying the function `execute_connector`._\r\n\r\n###
Solution\r\nBased on the solutions I experimented with, seems like we
need to force\r\nthe LLM to understand what's needed for the Slack
connector. I tried a\r\nfew options here and the combination of updates
that worked are as\r\nfollows:\r\n- Appending an AdHoc instruction about
the Slack connector properties.\r\n(This gives the LLM some additional
information about the required\r\nproperties)\r\n- Updating the
`properties` attached to the connector (when passing the\r\nconnector
list to the LLM), to reflect both `id` and `params`
with\r\n`message`.\r\n\r\nWith the above change, the AI Assistant has
managed to consistently send\r\nthe output to Slack _**without any
retries**_ because of missing\r\n`params`.\r\n\r\n###
Screenshots:\r\n(all alert triggers successfully sent the output to
Slack without having\r\nto retry the function)\r\n\r\n<img
width=\"1452\"
alt=\"success-attempts-to-slack-connector\"\r\nsrc=\"https://github.com/user-attachments/assets/715a5957-2c04-4a55-868f-34abe564f6d4\">","sha":"ed4c0df9ee304ea9bc71ed30c8e5dc2dd9295a00"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199531","number":199531,"mergeCommit":{"message":"[Obs
AI Assistant] Include an AdHoc instruction about the slack connector to
avoid executing a loop (#199531)\n\nCloses
https://github.com/elastic/kibana/issues/185028\r\n\r\n##
Summary\r\n\r\n### Problem\r\nThe ticket mentions that there was an
issue with displaying results.\r\nHowever both `display results` and
`visualize query` are working as\r\nexpected based on my investigation.
More details including a video
are\r\nattached\r\n[here](https://github.com/elastic/kibana/issues/185028#issuecomment-2464879452).\r\n\r\nThe
function calling loop seems to occur when the AI Assistant is
trying\r\nto send the output to Slack via the Kibana Slack connector. In
order to\r\ndo this, the LLM invokes the function `execute_connector`.
For the Slack\r\nconnector, `id` and `params` properties are required.
However, the LLM\r\nonly populated `id` and not `params` which causes an
error when\r\nvalidated against the schema for the Slack
connector.\r\n\r\n- Sometimes, it's able to retry a few times and
successfully send the\r\noutput to Slack.\r\n- Sometimes, it goes into a
loop trying to find `params` and failing\r\nrepeatedly.\r\n\r\nAttaching
another similar issue for more context
-\r\nhttps://github.com//issues/195564\r\n\r\n_With the
solution below, my intention is to send the output to Slack in\r\none
go, without retrying the function `execute_connector`._\r\n\r\n###
Solution\r\nBased on the solutions I experimented with, seems like we
need to force\r\nthe LLM to understand what's needed for the Slack
connector. I tried a\r\nfew options here and the combination of updates
that worked are as\r\nfollows:\r\n- Appending an AdHoc instruction about
the Slack connector properties.\r\n(This gives the LLM some additional
information about the required\r\nproperties)\r\n- Updating the
`properties` attached to the connector (when passing the\r\nconnector
list to the LLM), to reflect both `id` and `params`
with\r\n`message`.\r\n\r\nWith the above change, the AI Assistant has
managed to consistently send\r\nthe output to Slack _**without any
retries**_ because of missing\r\n`params`.\r\n\r\n###
Screenshots:\r\n(all alert triggers successfully sent the output to
Slack without having\r\nto retry the function)\r\n\r\n<img
width=\"1452\"
alt=\"success-attempts-to-slack-connector\"\r\nsrc=\"https://github.com/user-attachments/assets/715a5957-2c04-4a55-868f-34abe564f6d4\">","sha":"ed4c0df9ee304ea9bc71ed30c8e5dc2dd9295a00"}},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Viduni Wickramarachchi <[email protected]>
  • Loading branch information
@kibanamachine @viduni94
kibanamachine and viduni94 authored Nov 12, 2024
1 parent 079c226 commit 8968625
Show file tree
Hide file tree
Showing 2 changed files with 34 additions and 3 deletions.
4 changes: 3 additions & 1 deletion ...s/observability_solution/observability_ai_assistant/server/functions/execute_connector.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,15 @@
import { FunctionRegistrationParameters } from '.';
import { FunctionVisibility } from '../../common';

export const EXECUTE_CONNECTOR_FUNCTION_NAME = 'execute_connector';

export function registerExecuteConnectorFunction({
functions,
resources,
}: FunctionRegistrationParameters) {
functions.registerFunction(
{
name: 'execute_connector',
name: EXECUTE_CONNECTOR_FUNCTION_NAME,
description: 'Use this function when user explicitly asks to call a kibana connector.',
visibility: FunctionVisibility.AssistantOnly,
parameters: {
Expand Down
33 changes: 31 additions & 2 deletions ...gins/observability_solution/observability_ai_assistant_app/server/rule_connector/index.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ import {
JiraParamsSchema,
PagerdutyParamsSchema,
SlackApiParamsSchema,
SlackParamsSchema,
WebhookParamsSchema,
} from '@kbn/stack-connectors-plugin/server';
import { ObservabilityAIAssistantRouteHandlerResources } from '@kbn/observability-ai-assistant-plugin/server/routes/types';
Expand All @@ -37,14 +36,26 @@ import { CompatibleJSONSchema } from '@kbn/observability-ai-assistant-plugin/com
import { AlertDetailsContextualInsightsService } from '@kbn/observability-plugin/server/services';
import { getSystemMessageFromInstructions } from '@kbn/observability-ai-assistant-plugin/server/service/util/get_system_message_from_instructions';
import { AdHocInstruction } from '@kbn/observability-ai-assistant-plugin/common/types';
import { EXECUTE_CONNECTOR_FUNCTION_NAME } from '@kbn/observability-ai-assistant-plugin/server/functions/execute_connector';
import { convertSchemaToOpenApi } from './convert_schema_to_open_api';
import { OBSERVABILITY_AI_ASSISTANT_CONNECTOR_ID } from '../../common/rule_connector';

const CONNECTOR_PRIVILEGES = ['api:observabilityAIAssistant', 'app:observabilityAIAssistant'];

const connectorParamsSchemas: Record<string, CompatibleJSONSchema> = {
'.slack': {
type: 'object',
properties: {
id: { type: 'string' },
params: {
type: 'object',
properties: {
message: { type: 'string' },
},
},
},
},
'.slack_api': convertSchemaToOpenApi(SlackApiParamsSchema),
'.slack': convertSchemaToOpenApi(SlackParamsSchema),
'.email': convertSchemaToOpenApi(EmailParamsSchema),
'.webhook': convertSchemaToOpenApi(WebhookParamsSchema),
'.jira': convertSchemaToOpenApi(JiraParamsSchema),
Expand Down Expand Up @@ -189,6 +200,24 @@ If available, include the link of the conversation at the end of your answer.`
),
};

const hasSlackConnector = !!connectorsList.filter(
(connector) => connector.actionTypeId === '.slack'
).length;

if (hasSlackConnector && functionClient.hasFunction(EXECUTE_CONNECTOR_FUNCTION_NAME)) {
const slackConnectorInstruction: AdHocInstruction = {
instruction_type: 'application_instruction',
text: dedent(
`The execute_connector function can be used to invoke Kibana connectors.
To send to the Slack connector, you need the following arguments:
- the "id" of the connector
- the "params" parameter that you will fill with the message
Please include both "id" and "params.message" in the function arguments when executing the Slack connector..`
),
};
functionClient.registerAdhocInstruction(slackConnectorInstruction);
}

const alertsContext = await getAlertsContext(
execOptions.params.rule,
execOptions.params.alerts,
Expand Down

0 comments on commit 8968625

Please sign in to comment.