[ES|QL] implicit casting changes #182989
Merged
[ES|QL] implicit casting changes #182989
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/ci |
drewdaemon
commented
May 8, 2024
Comment on lines
-1126
to
-1150
| // Implicit casting of literal values tests | ||
| testErrorsAndWarnings(`from a_index | where numberField ${op} stringField`, [ | ||
| `Argument of [${op}] must be [number], found value [stringField] type [string]`, | ||
| ]); | ||
| testErrorsAndWarnings(`from a_index | where stringField ${op} numberField`, [ | ||
| `Argument of [${op}] must be [number], found value [stringField] type [string]`, | ||
| ]); | ||
| testErrorsAndWarnings(`from a_index | where numberField ${op} "2022"`, []); | ||
|
|
||
| testErrorsAndWarnings(`from a_index | where dateField ${op} stringField`, [ | ||
| `Argument of [${op}] must be [string], found value [dateField] type [date]`, | ||
| ]); | ||
| testErrorsAndWarnings(`from a_index | where stringField ${op} dateField`, [ | ||
| `Argument of [${op}] must be [string], found value [dateField] type [date]`, | ||
| ]); | ||
| testErrorsAndWarnings(`from a_index | where dateField ${op} "2022"`, []); | ||
|
|
||
| // Check that the implicit cast doesn't apply for fields | ||
| testErrorsAndWarnings(`from a_index | where stringField ${op} 0`, [ | ||
| `Argument of [${op}] must be [number], found value [stringField] type [string]`, | ||
| ]); | ||
| testErrorsAndWarnings(`from a_index | where stringField ${op} now()`, [ | ||
| `Argument of [${op}] must be [string], found value [now()] type [date]`, | ||
| ]); |
There was a problem hiding this comment.
These felt superfluous, especially because I think the tests will soon be reorganized around functions more than commands (see #182390) and this is just a duplication of some tests that already exist in the from block
|
/ci |
drewdaemon
added
backport:prev-minor
5 tasks
|
Pinging @elastic/kibana-esql (Team:ESQL) |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Page load bundle
History
To update your PR or re-run it, just comment with: |
|
Ok the latest snapshot has the changes now! 👍 |
stratoula
approved these changes
May 9, 2024
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
May 9, 2024
## Summary This is a follow-on from elastic/elasticsearch#107859. Two main changes to our client-side validation code. 1. comparison operators like `==` and `>=` now support implicit casting from strings for `version`, `ip`, and `boolean` types 2. `in` and `not in` operators support implicit casting from strings for `version`, `ip`, `boolean`, and `date` constants. To address this quickly, I have disabled type-checking for array values (e.g. `in ( version_field, "1.2.3", "2.3.1" )`) because our parameter typing system does not currently support arrays of mixed types which it will need to in order to re-enable checking while allowing string literals. I have added this to elastic#177699 ### A note on testing These implicit casting changes may not be on the latest Elasticsearch snapshot. Instead, check out the `8.14` branch in a local Elasticsearch repo and run `yarn es source --source-path='path/to/elasticsearch'` See [these tests](https://github.com/elastic/kibana/pull/182989/files#diff-89c4af0faedcf80d51cfb19ae397a5898c7293055b19284a94cb3f6a7cd4d071R1727-R1763) for a set of good use cases to try. `to_<type>` functions can be used if the sample data doesn't contain a field of the type you want to test. ### A note on string to date casting In elastic#182856 we started accepting string literals for any function arguments that are dates. By the same logic, `"2022" - 1 day` would work, so our validator doesn't complain about it. However, it currently fails at the Elasticsearch level. In a discussion with Fang, we determined that this is a valid use case, so I have created elastic/elasticsearch#108432 and determined not to tighten the client-side validation since this seems more like a casting "hole" that will soon be filled in on the ES side (though not in 8.14). ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or (cherry picked from commit c3e1027)
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
May 9, 2024
# Backport This will backport the following commits from `main` to `8.14`: - [[ES|QL] implicit casting changes (#182989)](#182989) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Drew Tate","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-05-09T05:51:18Z","message":"[ES|QL] implicit casting changes (#182989)\n\n## Summary\r\n\r\nThis is a follow-on from\r\nhttps://github.com/elastic/elasticsearch/pull/107859.\r\n\r\nTwo main changes to our client-side validation code.\r\n1. comparison operators like `==` and `>=` now support implicit casting\r\nfrom strings for `version`, `ip`, and `boolean` types\r\n\r\n2. `in` and `not in` operators support implicit casting from strings for\r\n`version`, `ip`, `boolean`, and `date` constants. To address this\r\nquickly, I have disabled type-checking for array values (e.g. `in (\r\nversion_field, \"1.2.3\", \"2.3.1\" )`) because our parameter typing system\r\ndoes not currently support arrays of mixed types which it will need to\r\nin order to re-enable checking while allowing string literals. I have\r\nadded this to https://github.com/elastic/kibana/issues/177699\r\n\r\n### A note on testing\r\n\r\nThese implicit casting changes may not be on the latest Elasticsearch\r\nsnapshot. Instead, check out the `8.14` branch in a local Elasticsearch\r\nrepo and run `yarn es source --source-path='path/to/elasticsearch'`\r\n\r\nSee [these\r\ntests](https://github.com/elastic/kibana/pull/182989/files#diff-89c4af0faedcf80d51cfb19ae397a5898c7293055b19284a94cb3f6a7cd4d071R1727-R1763)\r\nfor a set of good use cases to try. `to_<type>` functions can be used if\r\nthe sample data doesn't contain a field of the type you want to test.\r\n\r\n### A note on string to date casting\r\n\r\nIn #182856 we started accepting\r\nstring literals for any function arguments that are dates.\r\n\r\nBy the same logic, `\"2022\" - 1 day` would work, so our validator doesn't\r\ncomplain about it. However, it currently fails at the Elasticsearch\r\nlevel.\r\n\r\nIn a discussion with Fang, we determined that this is a\r\nvalid use case, so I have created\r\nhttps://github.com/elastic/elasticsearch/issues/108432 and determined\r\nnot to tighten the client-side validation since this seems more like a\r\ncasting \"hole\" that will soon be filled in on the ES side (though not in\r\n8.14).\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or","sha":"c3e1027b2d5da9361251e3af3304098717193ded","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:enhancement","backport:prev-minor","Feature:ES|QL","v8.14.0","Team:ESQL","v8.15.0"],"title":"[ES|QL] implicit casting changes","number":182989,"url":"https://github.com/elastic/kibana/pull/182989","mergeCommit":{"message":"[ES|QL] implicit casting changes (#182989)\n\n## Summary\r\n\r\nThis is a follow-on from\r\nhttps://github.com/elastic/elasticsearch/pull/107859.\r\n\r\nTwo main changes to our client-side validation code.\r\n1. comparison operators like `==` and `>=` now support implicit casting\r\nfrom strings for `version`, `ip`, and `boolean` types\r\n\r\n2. `in` and `not in` operators support implicit casting from strings for\r\n`version`, `ip`, `boolean`, and `date` constants. To address this\r\nquickly, I have disabled type-checking for array values (e.g. `in (\r\nversion_field, \"1.2.3\", \"2.3.1\" )`) because our parameter typing system\r\ndoes not currently support arrays of mixed types which it will need to\r\nin order to re-enable checking while allowing string literals. I have\r\nadded this to https://github.com/elastic/kibana/issues/177699\r\n\r\n### A note on testing\r\n\r\nThese implicit casting changes may not be on the latest Elasticsearch\r\nsnapshot. Instead, check out the `8.14` branch in a local Elasticsearch\r\nrepo and run `yarn es source --source-path='path/to/elasticsearch'`\r\n\r\nSee [these\r\ntests](https://github.com/elastic/kibana/pull/182989/files#diff-89c4af0faedcf80d51cfb19ae397a5898c7293055b19284a94cb3f6a7cd4d071R1727-R1763)\r\nfor a set of good use cases to try. `to_<type>` functions can be used if\r\nthe sample data doesn't contain a field of the type you want to test.\r\n\r\n### A note on string to date casting\r\n\r\nIn #182856 we started accepting\r\nstring literals for any function arguments that are dates.\r\n\r\nBy the same logic, `\"2022\" - 1 day` would work, so our validator doesn't\r\ncomplain about it. However, it currently fails at the Elasticsearch\r\nlevel.\r\n\r\nIn a discussion with Fang, we determined that this is a\r\nvalid use case, so I have created\r\nhttps://github.com/elastic/elasticsearch/issues/108432 and determined\r\nnot to tighten the client-side validation since this seems more like a\r\ncasting \"hole\" that will soon be filled in on the ES side (though not in\r\n8.14).\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or","sha":"c3e1027b2d5da9361251e3af3304098717193ded"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/182989","number":182989,"mergeCommit":{"message":"[ES|QL] implicit casting changes (#182989)\n\n## Summary\r\n\r\nThis is a follow-on from\r\nhttps://github.com/elastic/elasticsearch/pull/107859.\r\n\r\nTwo main changes to our client-side validation code.\r\n1. comparison operators like `==` and `>=` now support implicit casting\r\nfrom strings for `version`, `ip`, and `boolean` types\r\n\r\n2. `in` and `not in` operators support implicit casting from strings for\r\n`version`, `ip`, `boolean`, and `date` constants. To address this\r\nquickly, I have disabled type-checking for array values (e.g. `in (\r\nversion_field, \"1.2.3\", \"2.3.1\" )`) because our parameter typing system\r\ndoes not currently support arrays of mixed types which it will need to\r\nin order to re-enable checking while allowing string literals. I have\r\nadded this to https://github.com/elastic/kibana/issues/177699\r\n\r\n### A note on testing\r\n\r\nThese implicit casting changes may not be on the latest Elasticsearch\r\nsnapshot. Instead, check out the `8.14` branch in a local Elasticsearch\r\nrepo and run `yarn es source --source-path='path/to/elasticsearch'`\r\n\r\nSee [these\r\ntests](https://github.com/elastic/kibana/pull/182989/files#diff-89c4af0faedcf80d51cfb19ae397a5898c7293055b19284a94cb3f6a7cd4d071R1727-R1763)\r\nfor a set of good use cases to try. `to_<type>` functions can be used if\r\nthe sample data doesn't contain a field of the type you want to test.\r\n\r\n### A note on string to date casting\r\n\r\nIn #182856 we started accepting\r\nstring literals for any function arguments that are dates.\r\n\r\nBy the same logic, `\"2022\" - 1 day` would work, so our validator doesn't\r\ncomplain about it. However, it currently fails at the Elasticsearch\r\nlevel.\r\n\r\nIn a discussion with Fang, we determined that this is a\r\nvalid use case, so I have created\r\nhttps://github.com/elastic/elasticsearch/issues/108432 and determined\r\nnot to tighten the client-side validation since this seems more like a\r\ncasting \"hole\" that will soon be filled in on the ES side (though not in\r\n8.14).\r\n\r\n\r\n\r\n### Checklist\r\n\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or","sha":"c3e1027b2d5da9361251e3af3304098717193ded"}}]}] BACKPORT--> Co-authored-by: Drew Tate <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This is a follow-on from elastic/elasticsearch#107859.
Two main changes to our client-side validation code.
comparison operators like
==and>=now support implicit casting from strings forversion,ip, andbooleantypesinandnot inoperators support implicit casting from strings forversion,ip,boolean, anddateconstants. To address this quickly, I have disabled type-checking for array values (e.g.in ( version_field, "1.2.3", "2.3.1" )) because our parameter typing system does not currently support arrays of mixed types which it will need to in order to re-enable checking while allowing string literals. I have added this to [ES|QL] Client side validation enhancements #177699A note on testing
These implicit casting changes may not be on the latest Elasticsearch snapshot. Instead, check out the
8.14branch in a local Elasticsearch repo and runyarn es source --source-path='path/to/elasticsearch'See these tests for a set of good use cases to try.
to_<type>functions can be used if the sample data doesn't contain a field of the type you want to test.A note on string to date casting
In #182856 we started accepting string literals for any function arguments that are dates.
By the same logic,
"2022" - 1 daywould work, so our validator doesn't complain about it. However, it currently fails at the Elasticsearch level.In a discussion with @fang-xing-esql , we determined that this is a valid use case, so I have created elastic/elasticsearch#108432 and determined not to tighten the client-side validation since this seems more like a casting "hole" that will soon be filled in on the ES side (though not in 8.14).
Checklist