Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ESQL Add esql hash function #117989

Merged
merged 44 commits into from
Dec 18, 2024
Merged
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
39a2d17
test case
idegtiarenko Dec 4, 2024
a9ff454
dummy implementation
idegtiarenko Dec 4, 2024
5101864
simple implementation
idegtiarenko Dec 4, 2024
fabaabb
attempt to reuse MessageDigest
idegtiarenko Dec 4, 2024
33d5473
unit tests
idegtiarenko Dec 4, 2024
06a322a
update description
idegtiarenko Dec 4, 2024
2c83925
replace getBytes usage
idegtiarenko Dec 4, 2024
644874f
introduce capability
idegtiarenko Dec 4, 2024
b324f39
Update docs/changelog/117989.yaml
idegtiarenko Dec 4, 2024
514e1db
update functions counter
idegtiarenko Dec 4, 2024
052c247
fix required capability
idegtiarenko Dec 4, 2024
48d4dd1
Merge branch 'main' into add_esql_hash
idegtiarenko Dec 4, 2024
1adfc89
additional test cases
idegtiarenko Dec 5, 2024
633a044
update spec
idegtiarenko Dec 5, 2024
f063055
move function definition
idegtiarenko Dec 5, 2024
062967f
update
idegtiarenko Dec 5, 2024
62f97c3
enable by default
idegtiarenko Dec 6, 2024
923f453
handle error cases
idegtiarenko Dec 6, 2024
9863fcc
make formatting a bit cheaper
idegtiarenko Dec 6, 2024
6f08e29
Merge branch 'main' into add_esql_hash
idegtiarenko Dec 6, 2024
a22e2ad
fix yaml test
idegtiarenko Dec 6, 2024
bb8f73b
update corresponding test
idegtiarenko Dec 6, 2024
e03a7b3
Merge branch 'main' into add_esql_hash
idegtiarenko Dec 6, 2024
2506a55
reorder
idegtiarenko Dec 6, 2024
dce7cd0
update docs
idegtiarenko Dec 6, 2024
8d76d16
Merge branch 'main' into add_esql_hash
idegtiarenko Dec 6, 2024
9d3a44f
Merge branch 'main' into add_esql_hash
idegtiarenko Dec 9, 2024
f471900
Merge branch 'main' into add_esql_hash
idegtiarenko Dec 10, 2024
ed4f01f
use non lookup index in test
idegtiarenko Dec 11, 2024
c2811b7
Merge branch 'main' into add_esql_hash
idegtiarenko Dec 11, 2024
99ad357
rename alg -> algorithm
idegtiarenko Dec 12, 2024
3abf208
more hash test cases
idegtiarenko Dec 12, 2024
0983786
cover folded literal case
idegtiarenko Dec 13, 2024
7b5a3ec
Merge branch 'main' into add_esql_hash
idegtiarenko Dec 13, 2024
08eaee6
fix test
idegtiarenko Dec 13, 2024
bfab16b
additional cases
idegtiarenko Dec 13, 2024
8dfca90
Merge branch 'main' into add_esql_hash
idegtiarenko Dec 13, 2024
be91871
update function count
idegtiarenko Dec 13, 2024
cd2df96
Merge branch 'main' into add_esql_hash
idegtiarenko Dec 16, 2024
fb86d18
fix order
idegtiarenko Dec 16, 2024
edcf755
randomize input
idegtiarenko Dec 16, 2024
2c8f541
merge test scenario creation
idegtiarenko Dec 16, 2024
5ac4ae6
Merge branch 'main' into add_esql_hash
idegtiarenko Dec 18, 2024
c72e947
cleanup tests
idegtiarenko Dec 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/changelog/117989.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pr: 117989
summary: ESQL Add esql hash function
area: ES|QL
type: enhancement
issues: []
5 changes: 5 additions & 0 deletions docs/reference/esql/functions/description/hash.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

82 changes: 82 additions & 0 deletions docs/reference/esql/functions/kibana/definition/hash.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

7 changes: 7 additions & 0 deletions docs/reference/esql/functions/kibana/docs/hash.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

14 changes: 14 additions & 0 deletions docs/reference/esql/functions/layout/hash.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions docs/reference/esql/functions/parameters/hash.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions docs/reference/esql/functions/signature/hash.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 2 additions & 0 deletions docs/reference/esql/functions/string-functions.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
* <<esql-concat>>
* <<esql-ends_with>>
* <<esql-from_base64>>
* <<esql-hash>>
* <<esql-left>>
* <<esql-length>>
* <<esql-locate>>
Expand All @@ -37,6 +38,7 @@ include::layout/byte_length.asciidoc[]
include::layout/concat.asciidoc[]
include::layout/ends_with.asciidoc[]
include::layout/from_base64.asciidoc[]
include::layout/hash.asciidoc[]
include::layout/left.asciidoc[]
include::layout/length.asciidoc[]
include::layout/locate.asciidoc[]
Expand Down
12 changes: 12 additions & 0 deletions docs/reference/esql/functions/types/hash.asciidoc

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

105 changes: 105 additions & 0 deletions x-pack/plugin/esql/qa/testFixtures/src/main/resources/hash.csv-spec
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
hash
idegtiarenko marked this conversation as resolved.
Show resolved Hide resolved
required_capability: hash_function

FROM sample_data
| WHERE message != "Connection error"
| EVAL md5 = hash("md5", message), sha256 = hash("sha256", message)
| KEEP message, md5, sha256;
ignoreOrder:true

message:keyword | md5:keyword | sha256:keyword
Connected to 10.1.0.1 | abd7d1ce2bb636842a29246b3512dcae | 6d8372129ad78770f7185554dd39864749a62690216460752d6c075fa38ad85c
Connected to 10.1.0.2 | 8f8f1cb60832d153f5b9ec6dc828b93f | b0db24720f15857091b3c99f4c4833586d0ea3229911b8777efb8d917cf27e9a
Connected to 10.1.0.3 | 912b6dc13503165a15de43304bb77c78 | 75b0480188db8acc4d5cc666a51227eb2bc5b989cd8ca912609f33e0846eff57
Disconnected | ef70e46fd3bbc21e3e1f0b6815e750c0 | 04dfac3671b494ad53fcd152f7a14511bfb35747278aad8ce254a0d6e4ba4718
;


hashOfConvertedType
required_capability: hash_function

FROM sample_data
| WHERE message != "Connection error"
| EVAL input = event_duration::STRING, md5 = hash("md5", input), sha256 = hash("sha256", input)
| KEEP message, input, md5, sha256;
ignoreOrder:true

message:keyword | input:keyword | md5:keyword | sha256:keyword
Connected to 10.1.0.1 | 1756467 | c4fc1c57ee9b1d2b2023b70c8c167b54 | 8376a50a7ba7e6bd1bf9ad0c32d27d2f49fd0fa422573f98f239e21048b078f3
Connected to 10.1.0.2 | 2764889 | 8e8cf005e11a7b5df1d9478a4715a444 | 1031f2bef8eaecbf47319505422300b27ea1f7c38b6717d41332325062f9a56a
Connected to 10.1.0.3 | 3450233 | 09f2c64f5a55e9edf8ffbad336b561d8 | f77d7545769c4ecc85092f4f0b7ec8c20f467e4beb15fe67ca29f9aa8e9a6900
Disconnected | 1232382 | 6beac1485638d51e13c2c53990a2f611 | 9a03c1274a3ebb6c1cb85d170ce0a6fdb9d2232724e06b9f5e7cb9274af3cad6
;


hashOfEmptyInput
required_capability: hash_function

ROW input="" | EVAL md5 = hash("md5", input), sha256 = hash("sha256", input);

input:keyword | md5:keyword | sha256:keyword
| d41d8cd98f00b204e9800998ecf8427e | e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
;

hashOfNullInput
required_capability: hash_function

ROW input=null::STRING | EVAL md5 = hash("md5", input), sha256 = hash("sha256", input);

input:keyword | md5:keyword | sha256:keyword
null | null | null
;


hashWithNullAlgorithm
required_capability: hash_function

ROW input="input" | EVAL hash = hash(null, input);

input:keyword | hash:keyword
input | null
;


hashWithMv
required_capability: hash_function

ROW input=["foo", "bar"] | mv_expand input | EVAL md5 = hash("md5", input), sha256 = hash("sha256", input);

input:keyword | md5:keyword | sha256:keyword
foo | acbd18db4cc2f85cedef654fccc4a4d8 | 2c26b46b68ffc68ff99b453c1d30413413422d706483bfa0f98a5e886266e7ae
bar | 37b51d194a7513e45b56f6524f2d51f2 | fcde2b2edba56bf408601fb721fe9b5c338d10ee429ea04fae5511b68fbf8fb9
;


hashWithNestedFunctions
required_capability: hash_function

ROW input=["foo", "bar"] | EVAL hash = concat(hash("md5", mv_concat(input, "-")), "-", hash("sha256", mv_concat(input, "-")));

input:keyword | hash:keyword
["foo", "bar"] | e5f9ec048d1dbe19c70f720e002f9cb1-7d89c4f517e3bd4b5e8e76687937005b602ea00c5cba3e25ef1fc6575a55103e
;


hashWithConvertedTypes
required_capability: hash_function

ROW input=42 | EVAL md5 = hash("md5", input::STRING), sha256 = hash("sha256", to_string(input));

input:integer | md5:keyword | sha256:keyword
42 | a1d0c6e83f027327d8461063f4ac58a6 | 73475cb40a568e8da8a045ced110137e159f890ac4da883b6b17dc651b3a8049
;


hashWithStats
required_capability: hash_function

FROM sample_data
| EVAL md5="md5"
| STATS count = count(*) by hash(md5, message)
| WHERE count > 1;

count:long | hash(md5, message):keyword
3 | 2e92ae79ff32b37fee4368a594792183
;
idegtiarenko marked this conversation as resolved.
Show resolved Hide resolved
Loading