Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updating Ip fields to use doc_values to search #11508

Merged
merged 12 commits into from
Jan 30, 2024
Merged

Updating Ip fields to use doc_values to search #11508

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
68983cf
Updating Ip fields to use doc_values to search
harshavamsi Dec 7, 2023
c34fed9
Fix IP tests
harshavamsi Dec 8, 2023
6364147
Fix skip to allow yaml test to pass on main
harshavamsi Dec 8, 2023
4a3aff8
Update tests to use existing test file
harshavamsi Jan 8, 2024
8e661c0
Changing skip version to match bwc
harshavamsi Jan 10, 2024
3ae54b6
Using exact match instead of range
harshavamsi Jan 19, 2024
89876e0
Spotless
harshavamsi Jan 19, 2024
3d6f4cc
Fix IP field tests
harshavamsi Jan 19, 2024
a84588e
Fix spotless + precommit failure
harshavamsi Jan 19, 2024
d455bee
Get point out of query and into value
harshavamsi Jan 22, 2024
468313a
Fix term tests
harshavamsi Jan 22, 2024
407b723
Add skip test logic to only doc_values test
harshavamsi Jan 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
- Ensure Jackson default maximums introduced in 2.16.0 do not conflict with OpenSearch settings ([#11890](https://github.com/opensearch-project/OpenSearch/pull/11890))
- Extract cluster management for integration tests into JUnit test rule out of OpenSearchIntegTestCase ([#11877](https://github.com/opensearch-project/OpenSearch/pull/11877)), ([#12000](https://github.com/opensearch-project/OpenSearch/pull/12000))
- Workaround for https://bugs.openjdk.org/browse/JDK-8323659 regression, introduced in JDK-21.0.2 ([#11968](https://github.com/opensearch-project/OpenSearch/pull/11968))
- Updates IpField to be searchable when only `doc_values` are enabled ([#11508](https://github.com/opensearch-project/OpenSearch/pull/11508))

### Deprecated

Expand Down
136 changes: 115 additions & 21 deletions rest-api-spec/src/main/resources/rest-api-spec/test/search/340_doc_values_field.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,3 @@
setup:
- skip:
features: [ "headers" ]
version: " - 2.11.99"
reason: "searching with only doc_values was added in 2.12.0"
---
"search on fields with both index and doc_values enabled":
- do:
Expand Down Expand Up @@ -47,18 +42,22 @@ setup:
type: unsigned_long
index: true
doc_values: true
ip_field:
type: ip
index: true
doc_values: true

- do:
bulk:
index: test-iodvq
refresh: true
body:
- '{"index": {"_index": "test-iodvq", "_id": "1" }}'
- '{ "some_keyword": "ingesting some random keyword data", "byte": 120, "double": 100.0, "float": "800.0", "half_float": "400.0", "integer": 1290, "long": 13456, "short": 150, "unsigned_long": 10223372036854775800 }'
- '{ "some_keyword": "ingesting some random keyword data", "byte": 120, "double": 100.0, "float": "800.0", "half_float": "400.0", "integer": 1290, "long": 13456, "short": 150, "unsigned_long": 10223372036854775800, "ip_field": "192.168.0.1" }'
- '{ "index": { "_index": "test-iodvq", "_id": "2" }}'
- '{ "some_keyword": "400", "byte": 121, "double": 101.0, "float": "801.0", "half_float": "401.0", "integer": 1291, "long": 13457, "short": 151, "unsigned_long": 10223372036854775801 }'
- '{ "some_keyword": "400", "byte": 121, "double": 101.0, "float": "801.0", "half_float": "401.0", "integer": 1291, "long": 13457, "short": 151, "unsigned_long": 10223372036854775801, "ip_field": "192.168.0.2" }'
- '{ "index": { "_index": "test-iodvq", "_id": "3" } }'
- '{ "some_keyword": "5", "byte": 122, "double": 102.0, "float": "802.0", "half_float": "402.0", "integer": 1292, "long": 13458, "short": 152, "unsigned_long": 10223372036854775802 }'
- '{ "some_keyword": "5", "byte": 122, "double": 102.0, "float": "802.0", "half_float": "402.0", "integer": 1292, "long": 13458, "short": 152, "unsigned_long": 10223372036854775802, "ip_field": "192.168.0.3" }'

- do:
search:
Expand Down Expand Up @@ -162,7 +161,6 @@ setup:

- match: { hits.total: 1 }


- do:
search:
rest_total_hits_as_int: true
Expand All @@ -174,6 +172,16 @@ setup:

- match: { hits.total: 1 }

- do:
search:
rest_total_hits_as_int: true
index: test-iodvq
body:
query:
term:
ip_field: "192.168.0.1"

- match: {hits.total: 1}

- do:
search:
Expand All @@ -186,7 +194,6 @@ setup:

- match: { hits.total: 2 }


- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -264,6 +271,17 @@ setup:

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
index: test-iodvq
body:
query:
terms:
ip_field: ["192.168.0.1", "192.168.0.2"]

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -384,6 +402,19 @@ setup:

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
index: test-iodvq
body:
query:
range:
ip_field:
gte: "192.168.0.1"
lte: "192.168.0.2"

- match: { hits.total: 2 }

---
"search on fields with only index enabled":
- do:
Expand Down Expand Up @@ -428,18 +459,22 @@ setup:
type: unsigned_long
index: true
doc_values: false
ip_field:
type: ip
index: true
doc_values: false

- do:
bulk:
index: test-index
refresh: true
body:
- '{"index": {"_index": "test-index", "_id": "1" }}'
- '{ "some_keyword": "ingesting some random keyword data", "byte": 120, "double": 100.0, "float": "800.0", "half_float": "400.0", "integer": 1290, "long": 13456, "short": 150, "unsigned_long": 10223372036854775800 }'
- '{ "some_keyword": "ingesting some random keyword data", "byte": 120, "double": 100.0, "float": "800.0", "half_float": "400.0", "integer": 1290, "long": 13456, "short": 150, "unsigned_long": 10223372036854775800, "ip_field": "192.168.0.1" }'
- '{ "index": { "_index": "test-index", "_id": "2" }}'
- '{ "some_keyword": "400", "byte": 121, "double": 101.0, "float": "801.0", "half_float": "401.0", "integer": 1291, "long": 13457, "short": 151, "unsigned_long": 10223372036854775801 }'
- '{ "some_keyword": "400", "byte": 121, "double": 101.0, "float": "801.0", "half_float": "401.0", "integer": 1291, "long": 13457, "short": 151, "unsigned_long": 10223372036854775801, "ip_field": "192.168.0.2" }'
- '{ "index": { "_index": "test-index", "_id": "3" } }'
- '{ "some_keyword": "5", "byte": 122, "double": 102.0, "float": "802.0", "half_float": "402.0", "integer": 1292, "long": 13458, "short": 152, "unsigned_long": 10223372036854775802 }'
- '{ "some_keyword": "5", "byte": 122, "double": 102.0, "float": "802.0", "half_float": "402.0", "integer": 1292, "long": 13458, "short": 152, "unsigned_long": 10223372036854775802, "ip_field": "192.168.0.3" }'

- do:
search:
Expand All @@ -465,7 +500,6 @@ setup:

- match: { hits.total: 2 }


- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -555,6 +589,16 @@ setup:

- match: { hits.total: 1 }

- do:
search:
rest_total_hits_as_int: true
index: test-index
body:
query:
term:
ip_field: "192.168.0.1"

- match: {hits.total: 1}

- do:
search:
Expand All @@ -567,7 +611,6 @@ setup:

- match: { hits.total: 2 }


- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -645,6 +688,17 @@ setup:

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
index: test-index
body:
query:
terms:
ip_field: ["192.168.0.1", "192.168.0.2"]

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -765,8 +819,24 @@ setup:

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
index: test-index
body:
query:
range:
ip_field:
gte: "192.168.0.1"
lte: "192.168.0.2"

- match: { hits.total: 2 }
---
"search on fields with only doc_values enabled":
- skip:
features: [ "headers" ]
version: " - 2.99.99"
reason: "searching with only doc_values was added in 3.0.0"
- do:
indices.create:
index: test-doc-values
Expand Down Expand Up @@ -809,18 +879,22 @@ setup:
type: unsigned_long
index: false
doc_values: true
ip_field:
type: ip
index: false
doc_values: true

- do:
bulk:
index: test-doc-values
refresh: true
body:
- '{"index": {"_index": "test-doc-values", "_id": "1" }}'
- '{ "some_keyword": "ingesting some random keyword data", "byte": 120, "double": 100.0, "float": "800.0", "half_float": "400.0", "integer": 1290, "long": 13456, "short": 150, "unsigned_long": 10223372036854775800 }'
- '{ "some_keyword": "ingesting some random keyword data", "byte": 120, "double": 100.0, "float": "800.0", "half_float": "400.0", "integer": 1290, "long": 13456, "short": 150, "unsigned_long": 10223372036854775800, "ip_field": "192.168.0.1" }'
- '{ "index": { "_index": "test-doc-values", "_id": "2" }}'
- '{ "some_keyword": "400", "byte": 121, "double": 101.0, "float": "801.0", "half_float": "401.0", "integer": 1291, "long": 13457, "short": 151, "unsigned_long": 10223372036854775801 }'
- '{ "some_keyword": "400", "byte": 121, "double": 101.0, "float": "801.0", "half_float": "401.0", "integer": 1291, "long": 13457, "short": 151, "unsigned_long": 10223372036854775801, "ip_field": "192.168.0.2" }'
- '{ "index": { "_index": "test-doc-values", "_id": "3" } }'
- '{ "some_keyword": "5", "byte": 122, "double": 102.0, "float": "802.0", "half_float": "402.0", "integer": 1292, "long": 13458, "short": 152, "unsigned_long": 10223372036854775802 }'
- '{ "some_keyword": "5", "byte": 122, "double": 102.0, "float": "802.0", "half_float": "402.0", "integer": 1292, "long": 13458, "short": 152, "unsigned_long": 10223372036854775802, "ip_field": "192.168.0.3" }'

- do:
search:
Expand All @@ -846,7 +920,6 @@ setup:

- match: { hits.total: 2 }


- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -924,7 +997,6 @@ setup:

- match: { hits.total: 1 }


- do:
search:
rest_total_hits_as_int: true
Expand All @@ -936,6 +1008,16 @@ setup:

- match: { hits.total: 1 }

- do:
search:
rest_total_hits_as_int: true
index: test-doc-values
body:
query:
term:
ip_field: "192.168.0.3"

- match: { hits.total: 1 }

- do:
search:
Expand All @@ -948,7 +1030,6 @@ setup:

- match: { hits.total: 2 }


- do:
search:
rest_total_hits_as_int: true
Expand Down Expand Up @@ -1145,3 +1226,16 @@ setup:
}

- match: { hits.total: 2 }

- do:
search:
rest_total_hits_as_int: true
index: test-doc-values
body:
query:
range:
ip_field:
gte: "192.168.0.1"
lte: "192.168.0.2"

- match: { hits.total: 2 }
Loading
Loading