Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.x] [Obs AI Assistant] Include an AdHoc instruction about the slack connector to avoid executing a loop (#199531) #199860

Merged
merged 1 commit into from
Nov 12, 2024

Conversation

kibanamachine
Copy link
Contributor

Backport

This will backport the following commits from main to 8.x:

Questions ?

Please refer to the Backport tool documentation

…ctor to avoid executing a loop (elastic#199531)

Closes elastic#185028

## Summary

### Problem
The ticket mentions that there was an issue with displaying results.
However both `display results` and `visualize query` are working as
expected based on my investigation. More details including a video are
attached
[here](elastic#185028 (comment)).

The function calling loop seems to occur when the AI Assistant is trying
to send the output to Slack via the Kibana Slack connector. In order to
do this, the LLM invokes the function `execute_connector`. For the Slack
connector, `id` and `params` properties are required. However, the LLM
only populated `id` and not `params` which causes an error when
validated against the schema for the Slack connector.

- Sometimes, it's able to retry a few times and successfully send the
output to Slack.
- Sometimes, it goes into a loop trying to find `params` and failing
repeatedly.

Attaching another similar issue for more context -
elastic#195564

_With the solution below, my intention is to send the output to Slack in
one go, without retrying the function `execute_connector`._

### Solution
Based on the solutions I experimented with, seems like we need to force
the LLM to understand what's needed for the Slack connector. I tried a
few options here and the combination of updates that worked are as
follows:
- Appending an AdHoc instruction about the Slack connector properties.
(This gives the LLM some additional information about the required
properties)
- Updating the `properties` attached to the connector (when passing the
connector list to the LLM), to reflect both `id` and `params` with
`message`.

With the above change, the AI Assistant has managed to consistently send
the output to Slack _**without any retries**_ because of missing
`params`.

### Screenshots:
(all alert triggers successfully sent the output to Slack without having
to retry the function)

<img width="1452" alt="success-attempts-to-slack-connector"
src="https://github.com/user-attachments/assets/715a5957-2c04-4a55-868f-34abe564f6d4">

(cherry picked from commit ed4c0df)
@botelastic botelastic bot added ci:project-deploy-observability Create an Observability project Team:Obs AI Assistant Observability AI Assistant labels Nov 12, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant)

Copy link
Contributor

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

  • /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
  • run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

@kibanamachine kibanamachine merged commit 8968625 into elastic:8.x Nov 12, 2024
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport ci:project-deploy-observability Create an Observability project Team:Obs AI Assistant Observability AI Assistant
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants