Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Assistant] Knowledge base switch to use semantic_text #197007

Merged
merged 55 commits into from
Oct 30, 2024
Merged
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
1a42e3b
[Security Assistant] Fix KB output fields
patrykkopycinski Oct 16, 2024
6bbd037
update tests
patrykkopycinski Oct 16, 2024
e97b94b
Merge branch 'main' into fix/kb-output-fields
patrykkopycinski Oct 16, 2024
4ba1f33
Merge branch 'main' into fix/kb-output-fields
patrykkopycinski Oct 16, 2024
a628e39
Merge branch 'main' into fix/kb-output-fields
patrykkopycinski Oct 17, 2024
bbad040
revert cog wheel
patrykkopycinski Oct 18, 2024
bfe0693
fix nav
patrykkopycinski Oct 18, 2024
0098227
fix tests
patrykkopycinski Oct 18, 2024
7ff4a2e
cleanup
patrykkopycinski Oct 18, 2024
feee0ce
Migrate to use semantic_text
patrykkopycinski Oct 21, 2024
e3d801b
fix
patrykkopycinski Oct 21, 2024
55e8c55
[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…
kibanamachine Oct 21, 2024
8cfbf61
WIP
patrykkopycinski Oct 22, 2024
09476b3
Merge branch 'fix/kb-semantic-text' of github.com:patrykkopycinski/ki…
patrykkopycinski Oct 22, 2024
c2b829e
Merge branch 'main' of github.com:elastic/kibana into fix/kb-semantic…
patrykkopycinski Oct 22, 2024
cffe767
fix
patrykkopycinski Oct 22, 2024
e251ca2
fix
patrykkopycinski Oct 22, 2024
033cc57
fix
patrykkopycinski Oct 22, 2024
82e4ca3
filter out tools with missing indices
patrykkopycinski Oct 22, 2024
70ef308
fix
patrykkopycinski Oct 22, 2024
68a1f66
[CI] Auto-commit changed files from 'node scripts/notice'
kibanamachine Oct 22, 2024
185f44d
test
patrykkopycinski Oct 23, 2024
0e85e29
test
patrykkopycinski Oct 23, 2024
fc84417
cleanup
patrykkopycinski Oct 23, 2024
07d4339
Merge branch 'main' into fix/kb-semantic-text
patrykkopycinski Oct 23, 2024
d1ac1b6
test
patrykkopycinski Oct 23, 2024
3786057
test
patrykkopycinski Oct 23, 2024
dba8f0b
cleanup
patrykkopycinski Oct 23, 2024
060f287
fix
patrykkopycinski Oct 23, 2024
d329ca7
fix
patrykkopycinski Oct 23, 2024
6f2bd73
flags
patrykkopycinski Oct 23, 2024
4e28c67
fix
patrykkopycinski Oct 24, 2024
461eb79
test
patrykkopycinski Oct 24, 2024
b4c7e2c
Merge branch 'main' of github.com:elastic/kibana into fix/kb-semantic…
patrykkopycinski Oct 24, 2024
59226a5
fix
patrykkopycinski Oct 24, 2024
7ba0000
Merge branch 'main' of github.com:elastic/kibana into fix/serverless-kb
patrykkopycinski Oct 24, 2024
701a54a
fix
patrykkopycinski Oct 25, 2024
574f834
fix
patrykkopycinski Oct 25, 2024
d2aa921
fix
patrykkopycinski Oct 28, 2024
211586a
fix
patrykkopycinski Oct 28, 2024
f86beae
PR comments
patrykkopycinski Oct 28, 2024
3fdb27f
PR comments
patrykkopycinski Oct 28, 2024
90cf431
[CI] Auto-commit changed files from 'yarn openapi:generate'
kibanamachine Oct 28, 2024
ae220af
update test
patrykkopycinski Oct 29, 2024
9217d9a
[CI] Auto-commit changed files from 'yarn openapi:generate'
kibanamachine Oct 29, 2024
a68026a
Merge branch 'main' into fix/serverless-kb
patrykkopycinski Oct 29, 2024
dec121d
fix
patrykkopycinski Oct 29, 2024
7095357
Merge branch 'fix/serverless-kb' of github.com:patrykkopycinski/kiban…
patrykkopycinski Oct 29, 2024
f06ada1
switch to use assistant elser id
patrykkopycinski Oct 29, 2024
9a84bf1
revert FF
patrykkopycinski Oct 29, 2024
811f540
Merge branch 'main' into fix/kb-semantic-text
patrykkopycinski Oct 29, 2024
5efe3ac
Merge branch 'main' into fix/kb-semantic-text
peluja1012 Oct 30, 2024
7c8025a
Merge branch 'main' into fix/kb-semantic-text
patrykkopycinski Oct 30, 2024
b11e196
tests
patrykkopycinski Oct 30, 2024
4412d4d
fix kbretrievaltool with flag off
patrykkopycinski Oct 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix
patrykkopycinski committed Oct 22, 2024
commit 033cc5717ed7c689ebea46fd825778381288b94f
Original file line number Diff line number Diff line change
@@ -175,7 +175,7 @@ export class AIAssistantKnowledgeBaseDataClient extends AIAssistantDataClient {
});
}
} catch (error) {
this.options.logger.error(`Error creating ELSER inference endpoint '${elserId}':\n${error}`);
this.options.logger.error(`Error deploying ELSER model '${elserId}':\n${error}`);
throw new Error(`Error deploying ELSER model '${elserId}':\n${error}`);
}
};
Original file line number Diff line number Diff line change
@@ -147,8 +147,6 @@ export class AIAssistantService {
fieldMap,
});

console.error('CREATE DATA STREAM', resource, this.v2KnowledgeBaseEnabled);

newDataStream.setIndexTemplate({
name: this.resourceNames.indexTemplate[resource],
componentTemplateRefs: [this.resourceNames.componentTemplate[resource]],
Original file line number Diff line number Diff line change
@@ -25,10 +25,12 @@ describe('getVectorSearchQuery', () => {
filter: undefined,
must: [
{
semantic: {
field: 'semantic_text',
query:
'Generate an ES|QL query that will count the number of connections made to external IP addresses, broken down by user. If the count is greater than 100 for a specific user, add a new field called follow_up that contains a value of true, otherwise, it should contain false. The user names should also be enriched with their respective group names.',
text_expansion: {
'vector.tokens': {
model_id: '.elser_model_2',
model_text:
'Generate an ES|QL query that will count the number of connections made to external IP addresses, broken down by user. If the count is greater than 100 for a specific user, add a new field called follow_up that contains a value of true, otherwise, it should contain false. The user names should also be enriched with their respective group names.',
},
},
},
],
@@ -50,10 +52,12 @@ describe('getVectorSearchQuery', () => {
filter: undefined,
must: [
{
semantic: {
field: 'semantic_text',
query:
'Generate an ES|QL query that will count the number of connections made to external IP addresses, broken down by user. If the count is greater than 100 for a specific user, add a new field called follow_up that contains a value of true, otherwise, it should contain false. The user names should also be enriched with their respective group names.',
text_expansion: {
'vector.tokens': {
model_id: '.elser_model_2',
model_text:
'Generate an ES|QL query that will count the number of connections made to external IP addresses, broken down by user. If the count is greater than 100 for a specific user, add a new field called follow_up that contains a value of true, otherwise, it should contain false. The user names should also be enriched with their respective group names.',
},
},
},
],
Original file line number Diff line number Diff line change
@@ -37,9 +37,11 @@ export const getVectorSearchQuery = ({
must_not: [...mustNotTerms],
must: [
{
semantic: {
field: 'semantic_text',
query,
text_expansion: {
'vector.tokens': {
model_id: modelId,
model_text: query,
},
},
},
],
Original file line number Diff line number Diff line change
@@ -20,7 +20,7 @@ import { isV2KnowledgeBaseEnabled } from '../helpers';

// Since we're awaiting on ELSER setup, this could take a bit (especially if ML needs to autoscale)
// Consider just returning if attempt was successful, and switch to client polling
const ROUTE_HANDLER_TIMEOUT = 20 * 60 * 1000; // 10 * 60 seconds = 20 minutes
const ROUTE_HANDLER_TIMEOUT = 10 * 60 * 1000; // 10 * 60 seconds = 20 minutes
patrykkopycinski marked this conversation as resolved.
Show resolved Hide resolved

/**
* Load Knowledge Base index, pipeline, and resources (collection of documents)