[Auto Import] Improve the ECS mapping extraction logic #195167
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ilyannn
added
Feature:AutomaticImport
Team:Security-Scalability
ilyannn
changed the title
ilyannn
changed the title
|
Pinging @elastic/security-scalability (Team:Security-Scalability) |
ilyannn
changed the title
|
@elasticmachine merge upstream |
💚 Build Succeeded
Metrics [docs]
History
To update your PR or re-run it, just comment with: cc @ilyannn |
bhapas
reviewed
Oct 7, 2024
x-pack/plugins/integration_assistant/server/graphs/ecs/validate.ts
Outdated
Show resolved
Hide resolved
Comment on lines
+115
to
+118
| // If the value is an array, iterate through items and process them. | ||
| for (const item of value) { | ||
| if (typeof item === 'object' && item !== null) { | ||
| extractECSMapping(path, item, output); |
There was a problem hiding this comment.
Can try something like. Just a suggestion feel free to pick whatever you like to
value
.filter((item) => (typeof item === 'object' && item !== null))
.forEach((item) => extractECSMapping(path, item, output));There was a problem hiding this comment.
I agree it can be rewritten to be cleaner, but let's keep it like this because this makes it clear I just moved the lines around.
ilyannn
added
backport:prev-major
|
Tested with ai_teleport_202410072334-1.0.0.zip |
|
@elasticmachine merge upstream |
|
@elasticmachine merge upstream |
💚 Build Succeeded
The CI Stats report is too large to be displayed here, check out the CI build annotation for this information. History
cc @ilyannn |
|
Starting backport for target branches: 8.15, 8.x |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 9, 2024
## Release Notes Automatic Import is more forgiving if an LLM returns an ECS mapping in a slightly unexpected format. ## Summary When implementing elastic#194386 an issue has been encountered where Claude returns the field name `date_format` instead of expected `date_formats` and the ECS chain breaks down. We add this case as a test to `x-pack/plugins/integration_assistant/server/graphs/ecs/validate.test`. Without the changes in this PR the list returned by `findInvalidEcsFields` is ``` [ 'Reserved ECS field mapping identified for event.created : ai_postgres_202410050058.logs.column1.target', 'Invalid ECS field mapping identified for 0.9 : ai_postgres_202410050058.logs.column1.confidence, ai_postgres_202410050058.logs.column5.confidence', 'Invalid ECS field mapping identified for date : ai_postgres_202410050058.logs.column1.type, ai_postgres_202410050058.logs.column9.type', 'Invalid ECS field mapping identified for 0.95 : ai_postgres_202410050058.logs.column12.confidence', 'Invalid ECS field mapping identified for string : ai_postgres_202410050058.logs.column12.type, ai_postgres_202410050058.logs.column14.type, ai_postgres_202410050058.logs.column24.type, ai_postgres_202410050058.logs.column5.type, ai_postgres_202410050058.logs.column3.type, ai_postgres_202410050058.logs.column2.type', 'Invalid ECS field mapping identified for 0.8 : ai_postgres_202410050058.logs.column9.confidence, ai_postgres_202410050058.logs.column3.confidence', 'Invalid ECS field mapping identified for 0.7 : ai_postgres_202410050058.logs.column14.confidence, ai_postgres_202410050058.logs.column2.confidence', 'Invalid ECS field mapping identified for 0.85 : ai_postgres_202410050058.logs.column24.confidence' ] ``` while with these changes the result does not contain any `Invalid ECS field` messages. The key changes are in the `processMapping` function: 1. We made function more forgiving in regards to the input, accepting `date_format` in lieu of `date_formats`. 2. We have removed the collection of "other paths", that is, the reverse index for simple values like `0.8`. The latter change generally limits the impact of any other format issues in the ECS mapping in the future. Additionally, the function has been renamed to `extractECSMapping`, its output type validated, and documentation has been added. --------- Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 637d796)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Oct 9, 2024
## Release Notes Automatic Import is more forgiving if an LLM returns an ECS mapping in a slightly unexpected format. ## Summary When implementing elastic#194386 an issue has been encountered where Claude returns the field name `date_format` instead of expected `date_formats` and the ECS chain breaks down. We add this case as a test to `x-pack/plugins/integration_assistant/server/graphs/ecs/validate.test`. Without the changes in this PR the list returned by `findInvalidEcsFields` is ``` [ 'Reserved ECS field mapping identified for event.created : ai_postgres_202410050058.logs.column1.target', 'Invalid ECS field mapping identified for 0.9 : ai_postgres_202410050058.logs.column1.confidence, ai_postgres_202410050058.logs.column5.confidence', 'Invalid ECS field mapping identified for date : ai_postgres_202410050058.logs.column1.type, ai_postgres_202410050058.logs.column9.type', 'Invalid ECS field mapping identified for 0.95 : ai_postgres_202410050058.logs.column12.confidence', 'Invalid ECS field mapping identified for string : ai_postgres_202410050058.logs.column12.type, ai_postgres_202410050058.logs.column14.type, ai_postgres_202410050058.logs.column24.type, ai_postgres_202410050058.logs.column5.type, ai_postgres_202410050058.logs.column3.type, ai_postgres_202410050058.logs.column2.type', 'Invalid ECS field mapping identified for 0.8 : ai_postgres_202410050058.logs.column9.confidence, ai_postgres_202410050058.logs.column3.confidence', 'Invalid ECS field mapping identified for 0.7 : ai_postgres_202410050058.logs.column14.confidence, ai_postgres_202410050058.logs.column2.confidence', 'Invalid ECS field mapping identified for 0.85 : ai_postgres_202410050058.logs.column24.confidence' ] ``` while with these changes the result does not contain any `Invalid ECS field` messages. The key changes are in the `processMapping` function: 1. We made function more forgiving in regards to the input, accepting `date_format` in lieu of `date_formats`. 2. We have removed the collection of "other paths", that is, the reverse index for simple values like `0.8`. The latter change generally limits the impact of any other format issues in the ECS mapping in the future. Additionally, the function has been renamed to `extractECSMapping`, its output type validated, and documentation has been added. --------- Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 637d796)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
2 tasks
kibanamachine
added a commit
that referenced
this pull request
Oct 9, 2024
… (#195585) # Backport This will backport the following commits from `main` to `8.15`: - [[Auto Import] Improve the ECS mapping extraction logic (#195167)](#195167) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ilya Nikokoshev","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-09T12:24:00Z","message":"[Auto Import] Improve the ECS mapping extraction logic (#195167)\n\n## Release Notes\r\n\r\nAutomatic Import is more forgiving if an LLM returns an ECS mapping in a\r\nslightly unexpected format.\r\n\r\n## Summary\r\n\r\nWhen implementing #194386 an issue\r\nhas been encountered where Claude returns the field name `date_format`\r\ninstead of expected `date_formats` and the ECS chain breaks down.\r\n\r\nWe add this case as a test to\r\n`x-pack/plugins/integration_assistant/server/graphs/ecs/validate.test`.\r\n\r\nWithout the changes in this PR the list returned by\r\n`findInvalidEcsFields` is\r\n\r\n```\r\n [\r\n 'Reserved ECS field mapping identified for event.created : ai_postgres_202410050058.logs.column1.target',\r\n 'Invalid ECS field mapping identified for 0.9 : ai_postgres_202410050058.logs.column1.confidence, ai_postgres_202410050058.logs.column5.confidence',\r\n 'Invalid ECS field mapping identified for date : ai_postgres_202410050058.logs.column1.type, ai_postgres_202410050058.logs.column9.type',\r\n 'Invalid ECS field mapping identified for 0.95 : ai_postgres_202410050058.logs.column12.confidence',\r\n 'Invalid ECS field mapping identified for string : ai_postgres_202410050058.logs.column12.type, ai_postgres_202410050058.logs.column14.type, ai_postgres_202410050058.logs.column24.type, ai_postgres_202410050058.logs.column5.type, ai_postgres_202410050058.logs.column3.type, ai_postgres_202410050058.logs.column2.type',\r\n 'Invalid ECS field mapping identified for 0.8 : ai_postgres_202410050058.logs.column9.confidence, ai_postgres_202410050058.logs.column3.confidence',\r\n 'Invalid ECS field mapping identified for 0.7 : ai_postgres_202410050058.logs.column14.confidence, ai_postgres_202410050058.logs.column2.confidence',\r\n 'Invalid ECS field mapping identified for 0.85 : ai_postgres_202410050058.logs.column24.confidence'\r\n ]\r\n```\r\n\r\nwhile with these changes the result does not contain any `Invalid ECS field` messages.\r\n\r\nThe key changes are in the `processMapping` function:\r\n\r\n1. We made function more forgiving in regards to the input, accepting\r\n`date_format` in lieu of `date_formats`.\r\n2. We have removed the collection of \"other paths\", that is, the reverse\r\nindex for simple values like `0.8`.\r\n\r\nThe latter change generally limits the impact of any other format issues\r\nin the ECS mapping in the future.\r\n\r\nAdditionally, the function has been renamed to `extractECSMapping`, its\r\noutput type validated, and documentation has been added.\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"637d796071f067f8cab37165dd8f80111251ae81","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Auto Import] Improve the ECS mapping extraction logic","number":195167,"url":"https://github.com/elastic/kibana/pull/195167","mergeCommit":{"message":"[Auto Import] Improve the ECS mapping extraction logic (#195167)\n\n## Release Notes\r\n\r\nAutomatic Import is more forgiving if an LLM returns an ECS mapping in a\r\nslightly unexpected format.\r\n\r\n## Summary\r\n\r\nWhen implementing #194386 an issue\r\nhas been encountered where Claude returns the field name `date_format`\r\ninstead of expected `date_formats` and the ECS chain breaks down.\r\n\r\nWe add this case as a test to\r\n`x-pack/plugins/integration_assistant/server/graphs/ecs/validate.test`.\r\n\r\nWithout the changes in this PR the list returned by\r\n`findInvalidEcsFields` is\r\n\r\n```\r\n [\r\n 'Reserved ECS field mapping identified for event.created : ai_postgres_202410050058.logs.column1.target',\r\n 'Invalid ECS field mapping identified for 0.9 : ai_postgres_202410050058.logs.column1.confidence, ai_postgres_202410050058.logs.column5.confidence',\r\n 'Invalid ECS field mapping identified for date : ai_postgres_202410050058.logs.column1.type, ai_postgres_202410050058.logs.column9.type',\r\n 'Invalid ECS field mapping identified for 0.95 : ai_postgres_202410050058.logs.column12.confidence',\r\n 'Invalid ECS field mapping identified for string : ai_postgres_202410050058.logs.column12.type, ai_postgres_202410050058.logs.column14.type, ai_postgres_202410050058.logs.column24.type, ai_postgres_202410050058.logs.column5.type, ai_postgres_202410050058.logs.column3.type, ai_postgres_202410050058.logs.column2.type',\r\n 'Invalid ECS field mapping identified for 0.8 : ai_postgres_202410050058.logs.column9.confidence, ai_postgres_202410050058.logs.column3.confidence',\r\n 'Invalid ECS field mapping identified for 0.7 : ai_postgres_202410050058.logs.column14.confidence, ai_postgres_202410050058.logs.column2.confidence',\r\n 'Invalid ECS field mapping identified for 0.85 : ai_postgres_202410050058.logs.column24.confidence'\r\n ]\r\n```\r\n\r\nwhile with these changes the result does not contain any `Invalid ECS field` messages.\r\n\r\nThe key changes are in the `processMapping` function:\r\n\r\n1. We made function more forgiving in regards to the input, accepting\r\n`date_format` in lieu of `date_formats`.\r\n2. We have removed the collection of \"other paths\", that is, the reverse\r\nindex for simple values like `0.8`.\r\n\r\nThe latter change generally limits the impact of any other format issues\r\nin the ECS mapping in the future.\r\n\r\nAdditionally, the function has been renamed to `extractECSMapping`, its\r\noutput type validated, and documentation has been added.\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"637d796071f067f8cab37165dd8f80111251ae81"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/195167","number":195167,"mergeCommit":{"message":"[Auto Import] Improve the ECS mapping extraction logic (#195167)\n\n## Release Notes\r\n\r\nAutomatic Import is more forgiving if an LLM returns an ECS mapping in a\r\nslightly unexpected format.\r\n\r\n## Summary\r\n\r\nWhen implementing #194386 an issue\r\nhas been encountered where Claude returns the field name `date_format`\r\ninstead of expected `date_formats` and the ECS chain breaks down.\r\n\r\nWe add this case as a test to\r\n`x-pack/plugins/integration_assistant/server/graphs/ecs/validate.test`.\r\n\r\nWithout the changes in this PR the list returned by\r\n`findInvalidEcsFields` is\r\n\r\n```\r\n [\r\n 'Reserved ECS field mapping identified for event.created : ai_postgres_202410050058.logs.column1.target',\r\n 'Invalid ECS field mapping identified for 0.9 : ai_postgres_202410050058.logs.column1.confidence, ai_postgres_202410050058.logs.column5.confidence',\r\n 'Invalid ECS field mapping identified for date : ai_postgres_202410050058.logs.column1.type, ai_postgres_202410050058.logs.column9.type',\r\n 'Invalid ECS field mapping identified for 0.95 : ai_postgres_202410050058.logs.column12.confidence',\r\n 'Invalid ECS field mapping identified for string : ai_postgres_202410050058.logs.column12.type, ai_postgres_202410050058.logs.column14.type, ai_postgres_202410050058.logs.column24.type, ai_postgres_202410050058.logs.column5.type, ai_postgres_202410050058.logs.column3.type, ai_postgres_202410050058.logs.column2.type',\r\n 'Invalid ECS field mapping identified for 0.8 : ai_postgres_202410050058.logs.column9.confidence, ai_postgres_202410050058.logs.column3.confidence',\r\n 'Invalid ECS field mapping identified for 0.7 : ai_postgres_202410050058.logs.column14.confidence, ai_postgres_202410050058.logs.column2.confidence',\r\n 'Invalid ECS field mapping identified for 0.85 : ai_postgres_202410050058.logs.column24.confidence'\r\n ]\r\n```\r\n\r\nwhile with these changes the result does not contain any `Invalid ECS field` messages.\r\n\r\nThe key changes are in the `processMapping` function:\r\n\r\n1. We made function more forgiving in regards to the input, accepting\r\n`date_format` in lieu of `date_formats`.\r\n2. We have removed the collection of \"other paths\", that is, the reverse\r\nindex for simple values like `0.8`.\r\n\r\nThe latter change generally limits the impact of any other format issues\r\nin the ECS mapping in the future.\r\n\r\nAdditionally, the function has been renamed to `extractECSMapping`, its\r\noutput type validated, and documentation has been added.\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"637d796071f067f8cab37165dd8f80111251ae81"}}]}] BACKPORT--> Co-authored-by: Ilya Nikokoshev <[email protected]>
kibanamachine
added a commit
that referenced
this pull request
Oct 9, 2024
#195586) # Backport This will backport the following commits from `main` to `8.x`: - [[Auto Import] Improve the ECS mapping extraction logic (#195167)](#195167) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Ilya Nikokoshev","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-10-09T12:24:00Z","message":"[Auto Import] Improve the ECS mapping extraction logic (#195167)\n\n## Release Notes\r\n\r\nAutomatic Import is more forgiving if an LLM returns an ECS mapping in a\r\nslightly unexpected format.\r\n\r\n## Summary\r\n\r\nWhen implementing #194386 an issue\r\nhas been encountered where Claude returns the field name `date_format`\r\ninstead of expected `date_formats` and the ECS chain breaks down.\r\n\r\nWe add this case as a test to\r\n`x-pack/plugins/integration_assistant/server/graphs/ecs/validate.test`.\r\n\r\nWithout the changes in this PR the list returned by\r\n`findInvalidEcsFields` is\r\n\r\n```\r\n [\r\n 'Reserved ECS field mapping identified for event.created : ai_postgres_202410050058.logs.column1.target',\r\n 'Invalid ECS field mapping identified for 0.9 : ai_postgres_202410050058.logs.column1.confidence, ai_postgres_202410050058.logs.column5.confidence',\r\n 'Invalid ECS field mapping identified for date : ai_postgres_202410050058.logs.column1.type, ai_postgres_202410050058.logs.column9.type',\r\n 'Invalid ECS field mapping identified for 0.95 : ai_postgres_202410050058.logs.column12.confidence',\r\n 'Invalid ECS field mapping identified for string : ai_postgres_202410050058.logs.column12.type, ai_postgres_202410050058.logs.column14.type, ai_postgres_202410050058.logs.column24.type, ai_postgres_202410050058.logs.column5.type, ai_postgres_202410050058.logs.column3.type, ai_postgres_202410050058.logs.column2.type',\r\n 'Invalid ECS field mapping identified for 0.8 : ai_postgres_202410050058.logs.column9.confidence, ai_postgres_202410050058.logs.column3.confidence',\r\n 'Invalid ECS field mapping identified for 0.7 : ai_postgres_202410050058.logs.column14.confidence, ai_postgres_202410050058.logs.column2.confidence',\r\n 'Invalid ECS field mapping identified for 0.85 : ai_postgres_202410050058.logs.column24.confidence'\r\n ]\r\n```\r\n\r\nwhile with these changes the result does not contain any `Invalid ECS field` messages.\r\n\r\nThe key changes are in the `processMapping` function:\r\n\r\n1. We made function more forgiving in regards to the input, accepting\r\n`date_format` in lieu of `date_formats`.\r\n2. We have removed the collection of \"other paths\", that is, the reverse\r\nindex for simple values like `0.8`.\r\n\r\nThe latter change generally limits the impact of any other format issues\r\nin the ECS mapping in the future.\r\n\r\nAdditionally, the function has been renamed to `extractECSMapping`, its\r\noutput type validated, and documentation has been added.\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"637d796071f067f8cab37165dd8f80111251ae81","branchLabelMapping":{"^v9.0.0$":"main","^v8.16.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v9.0.0","backport:prev-major","Team:Security-Scalability","Feature:AutomaticImport"],"title":"[Auto Import] Improve the ECS mapping extraction logic","number":195167,"url":"https://github.com/elastic/kibana/pull/195167","mergeCommit":{"message":"[Auto Import] Improve the ECS mapping extraction logic (#195167)\n\n## Release Notes\r\n\r\nAutomatic Import is more forgiving if an LLM returns an ECS mapping in a\r\nslightly unexpected format.\r\n\r\n## Summary\r\n\r\nWhen implementing #194386 an issue\r\nhas been encountered where Claude returns the field name `date_format`\r\ninstead of expected `date_formats` and the ECS chain breaks down.\r\n\r\nWe add this case as a test to\r\n`x-pack/plugins/integration_assistant/server/graphs/ecs/validate.test`.\r\n\r\nWithout the changes in this PR the list returned by\r\n`findInvalidEcsFields` is\r\n\r\n```\r\n [\r\n 'Reserved ECS field mapping identified for event.created : ai_postgres_202410050058.logs.column1.target',\r\n 'Invalid ECS field mapping identified for 0.9 : ai_postgres_202410050058.logs.column1.confidence, ai_postgres_202410050058.logs.column5.confidence',\r\n 'Invalid ECS field mapping identified for date : ai_postgres_202410050058.logs.column1.type, ai_postgres_202410050058.logs.column9.type',\r\n 'Invalid ECS field mapping identified for 0.95 : ai_postgres_202410050058.logs.column12.confidence',\r\n 'Invalid ECS field mapping identified for string : ai_postgres_202410050058.logs.column12.type, ai_postgres_202410050058.logs.column14.type, ai_postgres_202410050058.logs.column24.type, ai_postgres_202410050058.logs.column5.type, ai_postgres_202410050058.logs.column3.type, ai_postgres_202410050058.logs.column2.type',\r\n 'Invalid ECS field mapping identified for 0.8 : ai_postgres_202410050058.logs.column9.confidence, ai_postgres_202410050058.logs.column3.confidence',\r\n 'Invalid ECS field mapping identified for 0.7 : ai_postgres_202410050058.logs.column14.confidence, ai_postgres_202410050058.logs.column2.confidence',\r\n 'Invalid ECS field mapping identified for 0.85 : ai_postgres_202410050058.logs.column24.confidence'\r\n ]\r\n```\r\n\r\nwhile with these changes the result does not contain any `Invalid ECS field` messages.\r\n\r\nThe key changes are in the `processMapping` function:\r\n\r\n1. We made function more forgiving in regards to the input, accepting\r\n`date_format` in lieu of `date_formats`.\r\n2. We have removed the collection of \"other paths\", that is, the reverse\r\nindex for simple values like `0.8`.\r\n\r\nThe latter change generally limits the impact of any other format issues\r\nin the ECS mapping in the future.\r\n\r\nAdditionally, the function has been renamed to `extractECSMapping`, its\r\noutput type validated, and documentation has been added.\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"637d796071f067f8cab37165dd8f80111251ae81"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/195167","number":195167,"mergeCommit":{"message":"[Auto Import] Improve the ECS mapping extraction logic (#195167)\n\n## Release Notes\r\n\r\nAutomatic Import is more forgiving if an LLM returns an ECS mapping in a\r\nslightly unexpected format.\r\n\r\n## Summary\r\n\r\nWhen implementing #194386 an issue\r\nhas been encountered where Claude returns the field name `date_format`\r\ninstead of expected `date_formats` and the ECS chain breaks down.\r\n\r\nWe add this case as a test to\r\n`x-pack/plugins/integration_assistant/server/graphs/ecs/validate.test`.\r\n\r\nWithout the changes in this PR the list returned by\r\n`findInvalidEcsFields` is\r\n\r\n```\r\n [\r\n 'Reserved ECS field mapping identified for event.created : ai_postgres_202410050058.logs.column1.target',\r\n 'Invalid ECS field mapping identified for 0.9 : ai_postgres_202410050058.logs.column1.confidence, ai_postgres_202410050058.logs.column5.confidence',\r\n 'Invalid ECS field mapping identified for date : ai_postgres_202410050058.logs.column1.type, ai_postgres_202410050058.logs.column9.type',\r\n 'Invalid ECS field mapping identified for 0.95 : ai_postgres_202410050058.logs.column12.confidence',\r\n 'Invalid ECS field mapping identified for string : ai_postgres_202410050058.logs.column12.type, ai_postgres_202410050058.logs.column14.type, ai_postgres_202410050058.logs.column24.type, ai_postgres_202410050058.logs.column5.type, ai_postgres_202410050058.logs.column3.type, ai_postgres_202410050058.logs.column2.type',\r\n 'Invalid ECS field mapping identified for 0.8 : ai_postgres_202410050058.logs.column9.confidence, ai_postgres_202410050058.logs.column3.confidence',\r\n 'Invalid ECS field mapping identified for 0.7 : ai_postgres_202410050058.logs.column14.confidence, ai_postgres_202410050058.logs.column2.confidence',\r\n 'Invalid ECS field mapping identified for 0.85 : ai_postgres_202410050058.logs.column24.confidence'\r\n ]\r\n```\r\n\r\nwhile with these changes the result does not contain any `Invalid ECS field` messages.\r\n\r\nThe key changes are in the `processMapping` function:\r\n\r\n1. We made function more forgiving in regards to the input, accepting\r\n`date_format` in lieu of `date_formats`.\r\n2. We have removed the collection of \"other paths\", that is, the reverse\r\nindex for simple values like `0.8`.\r\n\r\nThe latter change generally limits the impact of any other format issues\r\nin the ECS mapping in the future.\r\n\r\nAdditionally, the function has been renamed to `extractECSMapping`, its\r\noutput type validated, and documentation has been added.\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"637d796071f067f8cab37165dd8f80111251ae81"}}]}] BACKPORT--> Co-authored-by: Ilya Nikokoshev <[email protected]>
|
This PR didn't make it into the latest BC of v8.15.3. Updating the labels. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Release Notes
Automatic Import is more forgiving if an LLM returns an ECS mapping in a slightly unexpected format.
Summary
When implementing #194386 an issue has been encountered where Claude returns the field name
date_formatinstead of expecteddate_formatsand the ECS chain breaks down.We add this case as a test to
x-pack/plugins/integration_assistant/server/graphs/ecs/validate.test. Without the changes in this PR the list returned byfindInvalidEcsFieldsiswhile with these changes the result does not contain any
Invalid ECS fieldmessages.The key changes are in the
processMappingfunction:date_formatin lieu ofdate_formats.0.8.The latter change generally limits the impact of any other format issues in the ECS mapping in the future.
Additionally, the function has been renamed to
extractECSMapping, its output type validated, and documentation has been added.Checklist