[Obs AI Assistant] Include an AdHoc instruction about the slack connector to avoid executing a loop #199531
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
viduni94
added
release_note:fix
Team:Obs AI Assistant
|
Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant) |
botelastic
bot
added
the
ci:project-deploy-observability
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
viduni94
force-pushed
the
avoid-loop-when-executing-connector-for-slack
branch
4 times, most recently
from
a8f747c to
d8e4cf5
Compare
dgieselaar
reviewed
Nov 12, 2024
...plugins/observability_solution/observability_ai_assistant_app/server/rule_connector/index.ts
Show resolved
Hide resolved
dgieselaar
approved these changes
Nov 12, 2024
dgieselaar
reviewed
Nov 12, 2024
...plugins/observability_solution/observability_ai_assistant_app/server/rule_connector/index.ts
Outdated
Show resolved
Hide resolved
viduni94
force-pushed
the
avoid-loop-when-executing-connector-for-slack
branch
from
09e9c4c to
98ccc0a
Compare
viduni94
force-pushed
the
avoid-loop-when-executing-connector-for-slack
branch
from
98ccc0a to
525cd3e
Compare
💚 Build Succeeded
Metrics [docs]
History
cc @viduni94 |
|
Starting backport for target branches: 8.x |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 12, 2024
…ctor to avoid executing a loop (elastic#199531) Closes elastic#185028 ## Summary ### Problem The ticket mentions that there was an issue with displaying results. However both `display results` and `visualize query` are working as expected based on my investigation. More details including a video are attached [here](elastic#185028 (comment)). The function calling loop seems to occur when the AI Assistant is trying to send the output to Slack via the Kibana Slack connector. In order to do this, the LLM invokes the function `execute_connector`. For the Slack connector, `id` and `params` properties are required. However, the LLM only populated `id` and not `params` which causes an error when validated against the schema for the Slack connector. - Sometimes, it's able to retry a few times and successfully send the output to Slack. - Sometimes, it goes into a loop trying to find `params` and failing repeatedly. Attaching another similar issue for more context - elastic#195564 _With the solution below, my intention is to send the output to Slack in one go, without retrying the function `execute_connector`._ ### Solution Based on the solutions I experimented with, seems like we need to force the LLM to understand what's needed for the Slack connector. I tried a few options here and the combination of updates that worked are as follows: - Appending an AdHoc instruction about the Slack connector properties. (This gives the LLM some additional information about the required properties) - Updating the `properties` attached to the connector (when passing the connector list to the LLM), to reflect both `id` and `params` with `message`. With the above change, the AI Assistant has managed to consistently send the output to Slack _**without any retries**_ because of missing `params`. ### Screenshots: (all alert triggers successfully sent the output to Slack without having to retry the function) <img width="1452" alt="success-attempts-to-slack-connector" src="https://github.com/user-attachments/assets/715a5957-2c04-4a55-868f-34abe564f6d4"> (cherry picked from commit ed4c0df)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Nov 12, 2024
… connector to avoid executing a loop (#199531) (#199860) # Backport This will backport the following commits from `main` to `8.x`: - [[Obs AI Assistant] Include an AdHoc instruction about the slack connector to avoid executing a loop (#199531)](#199531) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Viduni Wickramarachchi","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-12T18:28:08Z","message":"[Obs AI Assistant] Include an AdHoc instruction about the slack connector to avoid executing a loop (#199531)\n\nCloses https://github.com/elastic/kibana/issues/185028\r\n\r\n## Summary\r\n\r\n### Problem\r\nThe ticket mentions that there was an issue with displaying results.\r\nHowever both `display results` and `visualize query` are working as\r\nexpected based on my investigation. More details including a video are\r\nattached\r\n[here](https://github.com/elastic/kibana/issues/185028#issuecomment-2464879452).\r\n\r\nThe function calling loop seems to occur when the AI Assistant is trying\r\nto send the output to Slack via the Kibana Slack connector. In order to\r\ndo this, the LLM invokes the function `execute_connector`. For the Slack\r\nconnector, `id` and `params` properties are required. However, the LLM\r\nonly populated `id` and not `params` which causes an error when\r\nvalidated against the schema for the Slack connector.\r\n\r\n- Sometimes, it's able to retry a few times and successfully send the\r\noutput to Slack.\r\n- Sometimes, it goes into a loop trying to find `params` and failing\r\nrepeatedly.\r\n\r\nAttaching another similar issue for more context -\r\nhttps://github.com//issues/195564\r\n\r\n_With the solution below, my intention is to send the output to Slack in\r\none go, without retrying the function `execute_connector`._\r\n\r\n### Solution\r\nBased on the solutions I experimented with, seems like we need to force\r\nthe LLM to understand what's needed for the Slack connector. I tried a\r\nfew options here and the combination of updates that worked are as\r\nfollows:\r\n- Appending an AdHoc instruction about the Slack connector properties.\r\n(This gives the LLM some additional information about the required\r\nproperties)\r\n- Updating the `properties` attached to the connector (when passing the\r\nconnector list to the LLM), to reflect both `id` and `params` with\r\n`message`.\r\n\r\nWith the above change, the AI Assistant has managed to consistently send\r\nthe output to Slack _**without any retries**_ because of missing\r\n`params`.\r\n\r\n### Screenshots:\r\n(all alert triggers successfully sent the output to Slack without having\r\nto retry the function)\r\n\r\n<img width=\"1452\" alt=\"success-attempts-to-slack-connector\"\r\nsrc=\"https://github.com/user-attachments/assets/715a5957-2c04-4a55-868f-34abe564f6d4\">","sha":"ed4c0df9ee304ea9bc71ed30c8e5dc2dd9295a00","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v9.0.0","Team:Obs AI Assistant","ci:project-deploy-observability","backport:version","v8.17.0"],"title":"[Obs AI Assistant] Include an AdHoc instruction about the slack connector to avoid executing a loop","number":199531,"url":"https://github.com/elastic/kibana/pull/199531","mergeCommit":{"message":"[Obs AI Assistant] Include an AdHoc instruction about the slack connector to avoid executing a loop (#199531)\n\nCloses https://github.com/elastic/kibana/issues/185028\r\n\r\n## Summary\r\n\r\n### Problem\r\nThe ticket mentions that there was an issue with displaying results.\r\nHowever both `display results` and `visualize query` are working as\r\nexpected based on my investigation. More details including a video are\r\nattached\r\n[here](https://github.com/elastic/kibana/issues/185028#issuecomment-2464879452).\r\n\r\nThe function calling loop seems to occur when the AI Assistant is trying\r\nto send the output to Slack via the Kibana Slack connector. In order to\r\ndo this, the LLM invokes the function `execute_connector`. For the Slack\r\nconnector, `id` and `params` properties are required. However, the LLM\r\nonly populated `id` and not `params` which causes an error when\r\nvalidated against the schema for the Slack connector.\r\n\r\n- Sometimes, it's able to retry a few times and successfully send the\r\noutput to Slack.\r\n- Sometimes, it goes into a loop trying to find `params` and failing\r\nrepeatedly.\r\n\r\nAttaching another similar issue for more context -\r\nhttps://github.com//issues/195564\r\n\r\n_With the solution below, my intention is to send the output to Slack in\r\none go, without retrying the function `execute_connector`._\r\n\r\n### Solution\r\nBased on the solutions I experimented with, seems like we need to force\r\nthe LLM to understand what's needed for the Slack connector. I tried a\r\nfew options here and the combination of updates that worked are as\r\nfollows:\r\n- Appending an AdHoc instruction about the Slack connector properties.\r\n(This gives the LLM some additional information about the required\r\nproperties)\r\n- Updating the `properties` attached to the connector (when passing the\r\nconnector list to the LLM), to reflect both `id` and `params` with\r\n`message`.\r\n\r\nWith the above change, the AI Assistant has managed to consistently send\r\nthe output to Slack _**without any retries**_ because of missing\r\n`params`.\r\n\r\n### Screenshots:\r\n(all alert triggers successfully sent the output to Slack without having\r\nto retry the function)\r\n\r\n<img width=\"1452\" alt=\"success-attempts-to-slack-connector\"\r\nsrc=\"https://github.com/user-attachments/assets/715a5957-2c04-4a55-868f-34abe564f6d4\">","sha":"ed4c0df9ee304ea9bc71ed30c8e5dc2dd9295a00"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199531","number":199531,"mergeCommit":{"message":"[Obs AI Assistant] Include an AdHoc instruction about the slack connector to avoid executing a loop (#199531)\n\nCloses https://github.com/elastic/kibana/issues/185028\r\n\r\n## Summary\r\n\r\n### Problem\r\nThe ticket mentions that there was an issue with displaying results.\r\nHowever both `display results` and `visualize query` are working as\r\nexpected based on my investigation. More details including a video are\r\nattached\r\n[here](https://github.com/elastic/kibana/issues/185028#issuecomment-2464879452).\r\n\r\nThe function calling loop seems to occur when the AI Assistant is trying\r\nto send the output to Slack via the Kibana Slack connector. In order to\r\ndo this, the LLM invokes the function `execute_connector`. For the Slack\r\nconnector, `id` and `params` properties are required. However, the LLM\r\nonly populated `id` and not `params` which causes an error when\r\nvalidated against the schema for the Slack connector.\r\n\r\n- Sometimes, it's able to retry a few times and successfully send the\r\noutput to Slack.\r\n- Sometimes, it goes into a loop trying to find `params` and failing\r\nrepeatedly.\r\n\r\nAttaching another similar issue for more context -\r\nhttps://github.com//issues/195564\r\n\r\n_With the solution below, my intention is to send the output to Slack in\r\none go, without retrying the function `execute_connector`._\r\n\r\n### Solution\r\nBased on the solutions I experimented with, seems like we need to force\r\nthe LLM to understand what's needed for the Slack connector. I tried a\r\nfew options here and the combination of updates that worked are as\r\nfollows:\r\n- Appending an AdHoc instruction about the Slack connector properties.\r\n(This gives the LLM some additional information about the required\r\nproperties)\r\n- Updating the `properties` attached to the connector (when passing the\r\nconnector list to the LLM), to reflect both `id` and `params` with\r\n`message`.\r\n\r\nWith the above change, the AI Assistant has managed to consistently send\r\nthe output to Slack _**without any retries**_ because of missing\r\n`params`.\r\n\r\n### Screenshots:\r\n(all alert triggers successfully sent the output to Slack without having\r\nto retry the function)\r\n\r\n<img width=\"1452\" alt=\"success-attempts-to-slack-connector\"\r\nsrc=\"https://github.com/user-attachments/assets/715a5957-2c04-4a55-868f-34abe564f6d4\">","sha":"ed4c0df9ee304ea9bc71ed30c8e5dc2dd9295a00"}},{"branch":"8.x","label":"v8.17.0","branchLabelMappingKey":"^v8.17.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Viduni Wickramarachchi <[email protected]>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Nov 18, 2024
…ctor to avoid executing a loop (elastic#199531) Closes elastic#185028 ## Summary ### Problem The ticket mentions that there was an issue with displaying results. However both `display results` and `visualize query` are working as expected based on my investigation. More details including a video are attached [here](elastic#185028 (comment)). The function calling loop seems to occur when the AI Assistant is trying to send the output to Slack via the Kibana Slack connector. In order to do this, the LLM invokes the function `execute_connector`. For the Slack connector, `id` and `params` properties are required. However, the LLM only populated `id` and not `params` which causes an error when validated against the schema for the Slack connector. - Sometimes, it's able to retry a few times and successfully send the output to Slack. - Sometimes, it goes into a loop trying to find `params` and failing repeatedly. Attaching another similar issue for more context - elastic#195564 _With the solution below, my intention is to send the output to Slack in one go, without retrying the function `execute_connector`._ ### Solution Based on the solutions I experimented with, seems like we need to force the LLM to understand what's needed for the Slack connector. I tried a few options here and the combination of updates that worked are as follows: - Appending an AdHoc instruction about the Slack connector properties. (This gives the LLM some additional information about the required properties) - Updating the `properties` attached to the connector (when passing the connector list to the LLM), to reflect both `id` and `params` with `message`. With the above change, the AI Assistant has managed to consistently send the output to Slack _**without any retries**_ because of missing `params`. ### Screenshots: (all alert triggers successfully sent the output to Slack without having to retry the function) <img width="1452" alt="success-attempts-to-slack-connector" src="https://github.com/user-attachments/assets/715a5957-2c04-4a55-868f-34abe564f6d4">
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Nov 18, 2024
…ctor to avoid executing a loop (elastic#199531) Closes elastic#185028 ## Summary ### Problem The ticket mentions that there was an issue with displaying results. However both `display results` and `visualize query` are working as expected based on my investigation. More details including a video are attached [here](elastic#185028 (comment)). The function calling loop seems to occur when the AI Assistant is trying to send the output to Slack via the Kibana Slack connector. In order to do this, the LLM invokes the function `execute_connector`. For the Slack connector, `id` and `params` properties are required. However, the LLM only populated `id` and not `params` which causes an error when validated against the schema for the Slack connector. - Sometimes, it's able to retry a few times and successfully send the output to Slack. - Sometimes, it goes into a loop trying to find `params` and failing repeatedly. Attaching another similar issue for more context - elastic#195564 _With the solution below, my intention is to send the output to Slack in one go, without retrying the function `execute_connector`._ ### Solution Based on the solutions I experimented with, seems like we need to force the LLM to understand what's needed for the Slack connector. I tried a few options here and the combination of updates that worked are as follows: - Appending an AdHoc instruction about the Slack connector properties. (This gives the LLM some additional information about the required properties) - Updating the `properties` attached to the connector (when passing the connector list to the LLM), to reflect both `id` and `params` with `message`. With the above change, the AI Assistant has managed to consistently send the output to Slack _**without any retries**_ because of missing `params`. ### Screenshots: (all alert triggers successfully sent the output to Slack without having to retry the function) <img width="1452" alt="success-attempts-to-slack-connector" src="https://github.com/user-attachments/assets/715a5957-2c04-4a55-868f-34abe564f6d4">
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #185028
Summary
Problem
The ticket mentions that there was an issue with displaying results. However both
display resultsandvisualize queryare working as expected based on my investigation. More details including a video are attached here.The function calling loop seems to occur when the AI Assistant is trying to send the output to Slack via the Kibana Slack connector. In order to do this, the LLM invokes the function
execute_connector. For the Slack connector,idandparamsproperties are required. However, the LLM only populatedidand notparamswhich causes an error when validated against the schema for the Slack connector.paramsand failing repeatedly.Attaching another similar issue for more context - #195564
With the solution below, my intention is to send the output to Slack in one go, without retrying the function
execute_connector.Solution
Based on the solutions I experimented with, seems like we need to force the LLM to understand what's needed for the Slack connector. I tried a few options here and the combination of updates that worked are as follows:
propertiesattached to the connector (when passing the connector list to the LLM), to reflect bothidandparamswithmessage.With the above change, the AI Assistant has managed to consistently send the output to Slack without any retries because of missing
params.Screenshots:
(all alert triggers successfully sent the output to Slack without having to retry the function)