[Obs AI Assistant] Evaluation framework #173010
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
dgieselaar
added
release_note:skip
…ieselaar/kibana into obs-ai-assistant-evaluation-framework
…ieselaar/kibana into obs-ai-assistant-evaluation-framework
CoenWarmer
approved these changes
Dec 13, 2023
|
@elasticmachine merge upstream |
…ieselaar/kibana into obs-ai-assistant-evaluation-framework
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]
History
To update your PR or re-run it, just comment with: |
| ], | ||
| }; | ||
|
|
||
| const sortedEvaluationFunctions = sortBy(evaluationFunctions, 'fileName', 'name'); |
There was a problem hiding this comment.
I'm a little bit hesitant to sort the test within the file, why not keep them in their order of declaration?
I haven't seen any other test framework sort test cases
There was a problem hiding this comment.
We use exports so no order is guaranteed. Ideally we have something similar to describe and it where the order of the statements decides order of execution, but need to figure out how we can use something like Mocha for this.
There was a problem hiding this comment.
Right, then it makes a lot of sense to sort them until then
|
|
||
| async function getKibanaUrl({ kibana, log }: { kibana: string; log: ToolingLog }) { | ||
| try { | ||
| const isCI = process.env.CI?.toLowerCase() === 'true'; |
There was a problem hiding this comment.
Thought this wouldn't run on CI?
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Dec 13, 2023
Adds an evaluation framework for the Observability AI Assistant. For more details, [see the README](https://github.com/dgieselaar/kibana/blob/obs-ai-assistant-evaluation-framework/x-pack/plugins/observability_ai_assistant/scripts/evaluation/README.md). --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit c36410f)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Dec 13, 2023
# Backport This will backport the following commits from `main` to `8.12`: - [[Obs AI Assistant] Evaluation framework (#173010)](#173010) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Dario Gieselaar","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-12-13T14:43:59Z","message":"[Obs AI Assistant] Evaluation framework (#173010)\n\nAdds an evaluation framework for the Observability AI Assistant. For\r\nmore details, [see the\r\nREADME](https://github.com/dgieselaar/kibana/blob/obs-ai-assistant-evaluation-framework/x-pack/plugins/observability_ai_assistant/scripts/evaluation/README.md).\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"c36410f78ee0103cb9c6dd056b1921c1eeec1e58","branchLabelMapping":{"^v8.13.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v8.12.0","v8.12.1","v8.13.0"],"number":173010,"url":"https://github.com/elastic/kibana/pull/173010","mergeCommit":{"message":"[Obs AI Assistant] Evaluation framework (#173010)\n\nAdds an evaluation framework for the Observability AI Assistant. For\r\nmore details, [see the\r\nREADME](https://github.com/dgieselaar/kibana/blob/obs-ai-assistant-evaluation-framework/x-pack/plugins/observability_ai_assistant/scripts/evaluation/README.md).\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"c36410f78ee0103cb9c6dd056b1921c1eeec1e58"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.12","label":"v8.12.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.13.0","labelRegex":"^v8.13.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/173010","number":173010,"mergeCommit":{"message":"[Obs AI Assistant] Evaluation framework (#173010)\n\nAdds an evaluation framework for the Observability AI Assistant. For\r\nmore details, [see the\r\nREADME](https://github.com/dgieselaar/kibana/blob/obs-ai-assistant-evaluation-framework/x-pack/plugins/observability_ai_assistant/scripts/evaluation/README.md).\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"c36410f78ee0103cb9c6dd056b1921c1eeec1e58"}}]}] BACKPORT--> Co-authored-by: Dario Gieselaar <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds an evaluation framework for the Observability AI Assistant. For more details, see the README.