[Security Assistant] V2 Knowledge Base Settings feedback and fixes #194354
+686
−143
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…elds for indexEntries, and plumbs dataViews for autocomplete
spong
added
release_note:skip
spong
commented
Oct 3, 2024
Comment on lines
+76
to
+77
| const { data: kbStatus, isFetched } = useKnowledgeBaseStatus({ http, resource: ESQL_RESOURCE }); | ||
| const isKbSetup = isKnowledgeBaseSetup(kbStatus); |
There was a problem hiding this comment.
@patrykkopycinski, this may need to be updated once #184885 is merged
spong
commented
Oct 3, 2024
| } from './nodes/translations'; | ||
|
|
||
| export const formatPrompt = (prompt: string, additionalPrompt?: string) => | ||
| ChatPromptTemplate.fromMessages([ | ||
| ['system', additionalPrompt ? `${prompt}\n\n${additionalPrompt}` : prompt], | ||
| ['placeholder', '{knowledge_history}'], |
There was a problem hiding this comment.
This is not behind the feature flag as it is fine if we return no knowledge history before this feature is fully enabled. Behavior will be just the same as if there is no knowledge history once enabled.
spong
commented
Oct 3, 2024
Comment on lines
+44
to
+53
| const knowledgeHistory = await kbDataClient?.getRequiredKnowledgeBaseDocumentEntries(); | ||
|
|
||
| const agentOutcome = await agentRunnable.withConfig({ tags: [AGENT_NODE_TAG] }).invoke( | ||
| { | ||
| ...state, | ||
| knowledge_history: `${KNOWLEDGE_HISTORY_PREFIX}\n${ | ||
| knowledgeHistory?.length | ||
| ? JSON.stringify(knowledgeHistory.map((e) => e.text)) | ||
| : NO_KNOWLEDGE_HISTORY | ||
| }`, |
There was a problem hiding this comment.
nit: I had considered pulling this out to it's own AddKnowledgeHistory node (similar to how we fetch chat_history and add to graph state). This would probably be good cleanup
spong
commented
Oct 3, 2024
| @@ -492,7 +539,7 @@ export class AIAssistantKnowledgeBaseDataClient extends AIAssistantDataClient { | |||
| page: 1, | |||
| sortField: 'created_at', | |||
| sortOrder: 'asc', | |||
| filter: `${userFilter}${` AND type:index`}`, // TODO: Support global tools (no user filter), and filter by space as well | |||
| filter: `${userFilter} AND type:index`, | |||
There was a problem hiding this comment.
Note: Removed TODO as the new userFilter covers this case (and it always only queries for the current space).
spong
commented
Oct 3, 2024
Comment on lines
+420
to
+422
| `getKnowledgeBaseDocuments() - Similarity Search returned [${JSON.stringify( | ||
| results.length | ||
| )}] results` |
There was a problem hiding this comment.
Just cleaning up console noise
spong
commented
Oct 3, 2024
Comment on lines
74
to
77
| // TODO: Plumb pipeline for update operations so that embeddings are regenerated on update | ||
| // Changing to `final_pipeline` should work as well according to the docs, see `pipeline` param: | ||
| // https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update-by-query.html#docs-update-by-query-api-query-params | ||
| pipeline: '.kibana-elastic-ai-assistant-ingest-pipeline-knowledge-base', |
There was a problem hiding this comment.
Note: Removing this as it did not work as a fix. Passing pipeline here didn't regenerate embeddings, which should've worked. Perhaps our painless scripting for managing patch operations is getting in the way here? O11y has this working with just an index() operation, so we may have to do that.
Also note that using a final_pipeline didn't work, and caused ELSER/mapping errors on KB initialization, so that's not a solution either.
💚 Build Succeeded
Metrics [docs]Module Count
Async chunks
History
To update your PR or re-run it, just comment with: cc @spong |
# Conflicts: # x-pack/packages/kbn-elastic-assistant/impl/knowledge_base/knowledge_base_settings_management/use_knowledge_base_table.tsx # x-pack/plugins/elastic_assistant/server/lib/langchain/graphs/default_assistant_graph/nodes/run_agent.ts # x-pack/plugins/elastic_assistant/server/lib/langchain/graphs/default_assistant_graph/nodes/translations.ts # x-pack/plugins/elastic_assistant/server/lib/langchain/graphs/default_assistant_graph/prompts.ts
💚 Build Succeeded
Metrics [docs]Module Count
Async chunks
History
cc @spong |
patrykkopycinski
approved these changes
Oct 9, 2024
|
Starting backport for target branches: 8.x |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 9, 2024
…lastic#194354) ## Summary This PR is a follow up to elastic#192665 and addresses a bunch of feedback and fixes including: - [X] Adds support for updating/editing entries - [X] Fixes initial loading experience of the KB Settings Setup/Table - [X] Fixes two bugs where `semantic_text` and `text` must be declared for `IndexEntries` to work - [X] Add new Settings Context Menu items for KB and Alerts - [X] Add support for `required` entries in initial prompt * See [this trace](https://smith.langchain.com/public/84a17a31-8ce8-4bd9-911e-38a854484dd8/r) for included knowledge. Note that the KnowledgeBaseRetrievalTool was not selected. * Note: All prompts were updated to include the `{knowledge_history}` placeholder, and _not behind the feature flag_, as this will just be the empty case until the feature flag is enabled. TODO (in this or follow-up PR): - [ ] Add suggestions to `index` and `fields` inputs - [ ] Adds URL deeplinking to securityAssistantManagement - [ ] Fix bug where updating entry does not re-create embeddings (see [comment](elastic#194354 (comment))) - [ ] Fix loading indicators when adding/editing entries - [ ] API integration tests for update API (@e40pud) ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials * Docs being tracked in elastic/security-docs#5337 for when feature flag is enabled - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Patryk Kopycinski <[email protected]> (cherry picked from commit 7df3672)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Oct 9, 2024
…xes (#194354) (#195644) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Assistant] V2 Knowledge Base Settings feedback and fixes (#194354)](#194354) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Garrett Spong","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-09T16:17:47Z","message":"[Security Assistant] V2 Knowledge Base Settings feedback and fixes (#194354)\n\n## Summary\r\n\r\nThis PR is a follow up to #192665 and addresses a bunch of feedback and\r\nfixes including:\r\n\r\n- [X] Adds support for updating/editing entries\r\n- [X] Fixes initial loading experience of the KB Settings Setup/Table\r\n- [X] Fixes two bugs where `semantic_text` and `text` must be declared\r\nfor `IndexEntries` to work\r\n- [X] Add new Settings Context Menu items for KB and Alerts\r\n - [X] Add support for `required` entries in initial prompt\r\n* See [this\r\ntrace](https://smith.langchain.com/public/84a17a31-8ce8-4bd9-911e-38a854484dd8/r)\r\nfor included knowledge. Note that the KnowledgeBaseRetrievalTool was not\r\nselected.\r\n* Note: All prompts were updated to include the `{knowledge_history}`\r\nplaceholder, and _not behind the feature flag_, as this will just be the\r\nempty case until the feature flag is enabled.\r\n\r\nTODO (in this or follow-up PR):\r\n - [ ] Add suggestions to `index` and `fields` inputs\r\n - [ ] Adds URL deeplinking to securityAssistantManagement\r\n- [ ] Fix bug where updating entry does not re-create embeddings (see\r\n[comment](https://github.com/elastic/kibana/pull/194354#discussion_r1786475496))\r\n - [ ] Fix loading indicators when adding/editing entries\r\n - [ ] API integration tests for update API (@e40pud)\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [X] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n* Docs being tracked in\r\nhttps://github.com/elastic/security-docs/issues/5337 for when feature\r\nflag is enabled\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>\r\nCo-authored-by: Patryk Kopycinski <[email protected]>","sha":"7df36721923159f45bc4fdbd26f76b20ad84249a","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Feature:Security Assistant","Team:Security Generative AI","v8.16.0","backport:version"],"title":"[Security Assistant] V2 Knowledge Base Settings feedback and fixes","number":194354,"url":"https://github.com/elastic/kibana/pull/194354","mergeCommit":{"message":"[Security Assistant] V2 Knowledge Base Settings feedback and fixes (#194354)\n\n## Summary\r\n\r\nThis PR is a follow up to #192665 and addresses a bunch of feedback and\r\nfixes including:\r\n\r\n- [X] Adds support for updating/editing entries\r\n- [X] Fixes initial loading experience of the KB Settings Setup/Table\r\n- [X] Fixes two bugs where `semantic_text` and `text` must be declared\r\nfor `IndexEntries` to work\r\n- [X] Add new Settings Context Menu items for KB and Alerts\r\n - [X] Add support for `required` entries in initial prompt\r\n* See [this\r\ntrace](https://smith.langchain.com/public/84a17a31-8ce8-4bd9-911e-38a854484dd8/r)\r\nfor included knowledge. Note that the KnowledgeBaseRetrievalTool was not\r\nselected.\r\n* Note: All prompts were updated to include the `{knowledge_history}`\r\nplaceholder, and _not behind the feature flag_, as this will just be the\r\nempty case until the feature flag is enabled.\r\n\r\nTODO (in this or follow-up PR):\r\n - [ ] Add suggestions to `index` and `fields` inputs\r\n - [ ] Adds URL deeplinking to securityAssistantManagement\r\n- [ ] Fix bug where updating entry does not re-create embeddings (see\r\n[comment](https://github.com/elastic/kibana/pull/194354#discussion_r1786475496))\r\n - [ ] Fix loading indicators when adding/editing entries\r\n - [ ] API integration tests for update API (@e40pud)\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [X] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n* Docs being tracked in\r\nhttps://github.com/elastic/security-docs/issues/5337 for when feature\r\nflag is enabled\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>\r\nCo-authored-by: Patryk Kopycinski <[email protected]>","sha":"7df36721923159f45bc4fdbd26f76b20ad84249a"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/194354","number":194354,"mergeCommit":{"message":"[Security Assistant] V2 Knowledge Base Settings feedback and fixes (#194354)\n\n## Summary\r\n\r\nThis PR is a follow up to #192665 and addresses a bunch of feedback and\r\nfixes including:\r\n\r\n- [X] Adds support for updating/editing entries\r\n- [X] Fixes initial loading experience of the KB Settings Setup/Table\r\n- [X] Fixes two bugs where `semantic_text` and `text` must be declared\r\nfor `IndexEntries` to work\r\n- [X] Add new Settings Context Menu items for KB and Alerts\r\n - [X] Add support for `required` entries in initial prompt\r\n* See [this\r\ntrace](https://smith.langchain.com/public/84a17a31-8ce8-4bd9-911e-38a854484dd8/r)\r\nfor included knowledge. Note that the KnowledgeBaseRetrievalTool was not\r\nselected.\r\n* Note: All prompts were updated to include the `{knowledge_history}`\r\nplaceholder, and _not behind the feature flag_, as this will just be the\r\nempty case until the feature flag is enabled.\r\n\r\nTODO (in this or follow-up PR):\r\n - [ ] Add suggestions to `index` and `fields` inputs\r\n - [ ] Adds URL deeplinking to securityAssistantManagement\r\n- [ ] Fix bug where updating entry does not re-create embeddings (see\r\n[comment](https://github.com/elastic/kibana/pull/194354#discussion_r1786475496))\r\n - [ ] Fix loading indicators when adding/editing entries\r\n - [ ] API integration tests for update API (@e40pud)\r\n\r\n\r\n### Checklist\r\n\r\nDelete any items that are not applicable to this PR.\r\n\r\n- [X] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n* Docs being tracked in\r\nhttps://github.com/elastic/security-docs/issues/5337 for when feature\r\nflag is enabled\r\n- [ ] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>\r\nCo-authored-by: Patryk Kopycinski <[email protected]>","sha":"7df36721923159f45bc4fdbd26f76b20ad84249a"}},{"branch":"8.x","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Garrett Spong <[email protected]>
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR is a follow up to #192665 and addresses a bunch of feedback and fixes including:
semantic_textandtextmust be declared forIndexEntriesto workrequiredentries in initial prompt{knowledge_history}placeholder, and not behind the feature flag, as this will just be the empty case until the feature flag is enabled.TODO (in this or follow-up PR):
indexandfieldsinputsChecklist
Delete any items that are not applicable to this PR.