#3145 Add IP Address Data Type #3175

currantw · 2024-11-29T18:55:00Z

Signed-off-by: currantw [email protected]

Description

#3145 Add IP Address Data Type

Related Issues

Check List

New functionality includes testing.
New functionality has been documented.
New functionality has javadoc added.
New functionality has a user manual doc added.
~~API changes companion pull request created.~~
Commits are signed per the DCO using --signoff.
~~Public documentation issue/PR created.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

YANG-DB · 2024-12-06T23:24:53Z

@currantw can we test an IP range query as well?
With operations such as < and in range

currantw · 2024-12-10T18:52:09Z

Woops! Didn't mean to close this... 😬

YANG-DB · 2024-12-10T20:27:26Z

@currantw can u plz resolve conflicts ?

currantw · 2024-12-11T03:45:15Z

@currantw can we test an IP range query as well? With operations such as < and in range

I have added a test to SortCommandIT for testing sorting of the IP address. There are also unit tests in ExprIpValueTest that test compareTo.

Let me know which additional tests you think may be needed. I didn't find any existing integration tests for comparisons -- perhaps this is something that should be added?

I'm not sure exactly what you mean by "in range" operations: can you give an example query?

YANG-DB · 2024-12-11T23:48:21Z

@currantw LGTM 👍
@penghuo @dai-chen @acarbonetto could u plz review ?

YANG-DB · 2024-12-11T23:49:18Z

@currantw can u plz check the failed CI plz ?

Signed-off-by: currantw <[email protected]>

…g unused sorting syntax. Signed-off-by: currantw <[email protected]>

Signed-off-by: currantw <[email protected]>

…s` test data. Signed-off-by: currantw <[email protected]>

Signed-off-by: currantw <[email protected]>

YANG-DB · 2024-12-13T20:12:05Z

ppl/src/main/antlr/OpenSearchPPLParser.g4

-   : (PLUS | MINUS)? sortFieldExpression
-   ;
-
-sortFieldExpression


@currantw was this sortFieldExpression used in a query anywhere ?

it is used in sortField, which is used by Trendline and SORT BY. @currantw we sure we want to include these changes?

acarbonetto · 2024-12-10T00:24:18Z

core/src/main/java/org/opensearch/sql/data/model/ExprIpValue.java

+public class ExprIpValue extends AbstractExprValue {
+  private final IPAddress value;
+
+  public ExprIpValue(String s) {


nit:

Suggested change

public ExprIpValue(String s) {

public ExprIpValue(String addressStr) {

acarbonetto · 2024-12-17T02:03:00Z

core/src/main/java/org/opensearch/sql/expression/ip/IPFunctions.java

    }

-    return ExprValueUtils.booleanValue(range.contains(address));
+    return ExprValueUtils.LITERAL_TRUE;


nit:

return (IPUtils.compare(address, range.getLower()) < 0) || (IPUtils.compare(address, range.getUpper()) > 0) ? ExprValueUtils.LITERAL_FALSE : ExprValueUtils.LITERAL_TRUE;

acarbonetto · 2024-12-17T02:03:45Z

core/src/main/java/org/opensearch/sql/expression/operator/convert/TypeCastOperators.java

+  private static DefaultFunctionResolver castToIp() {
+    return FunctionDSL.define(
+        BuiltinFunctionName.CAST_TO_IP.getName(),
+        impl(nullMissingHandling((v) -> v), IP, IP),


ordering matters - make sure to but the naive cast last

acarbonetto · 2024-12-17T02:34:23Z

...h/src/test/java/org/opensearch/sql/opensearch/data/value/OpenSearchExprValueFactoryTest.java

  }

+  private static final double TOLERANCE = 1E-5;


why did you move this?

acarbonetto · 2024-12-17T02:35:49Z

...src/test/java/org/opensearch/sql/opensearch/storage/script/filter/lucene/RangeQueryTest.java

-  void test_non_date_field_type() {
-    String dateString = "2021-11-08";
+  void test_string_field_type() {
+    String dateString = "STRING";


acarbonetto · 2024-12-17T02:36:07Z

ppl/src/main/antlr/OpenSearchPPLLexer.g4

-AUTO:                               'AUTO';
-STR:                                'STR';
-IP:                                 'IP';
-NUM:                                'NUM';


acarbonetto · 2024-12-17T02:39:25Z

ppl/src/main/antlr/OpenSearchPPLParser.g4

-   : (PLUS | MINUS)? sortFieldExpression
-   ;
-
-sortFieldExpression


it is used in sortField, which is used by Trendline and SORT BY. @currantw we sure we want to include these changes?

currantw mentioned this pull request Dec 2, 2024

[FEATURE] Add Support for IP Data Type #3145

Open

currantw marked this pull request as ready for review December 6, 2024 18:09

currantw requested review from ps48, kavithacm, derek-ho, joshuali925, dai-chen, YANG-DB, mengweieric, vmmusings, Swiddis, penghuo, seankao-az, MaxKsyunz, Yury-Fridlyand, anirudha, forestmvey, acarbonetto, GumpacG, ykmr1224, LantaoJin and noCharger as code owners December 6, 2024 18:09

YANG-DB previously approved these changes Dec 6, 2024

View reviewed changes

currantw closed this Dec 10, 2024

currantw reopened this Dec 10, 2024

currantw dismissed YANG-DB’s stale review via 201697b December 11, 2024 02:52

currantw force-pushed the opensearch-sql-3145 branch from cd9f94b to 201697b Compare December 11, 2024 02:52

YANG-DB previously approved these changes Dec 11, 2024

View reviewed changes

currantw dismissed YANG-DB’s stale review via 3f9e428 December 13, 2024 00:33

currantw force-pushed the opensearch-sql-3145 branch from 9cbc60b to 3f9e428 Compare December 13, 2024 00:33

currantw added 18 commits December 12, 2024 23:24

Add ExprIpValue and IP data type

e2cfd1f

Signed-off-by: currantw <[email protected]>

Add support for casting (cast(field_name to ip)) and remove existin…

8a00e0d

…g unused sorting syntax. Signed-off-by: currantw <[email protected]>

Update comparison logic to compare in IPv6

62be5e0

Signed-off-by: currantw <[email protected]>

Fix bug casting to IP

e83db18

Signed-off-by: currantw <[email protected]>

Fix failing tests

a56c4ff

Signed-off-by: currantw <[email protected]>

Assert that comparison only valid if same type, update tests accordingly

2c1f7da

Signed-off-by: currantw <[email protected]>

Add additional tests to increase code coverage

0da521e

Signed-off-by: currantw <[email protected]>

Integrate cidrmatch changes

3179f4b

Signed-off-by: currantw <[email protected]>

Remove OpenSearchIPType data type

a64ad9a

Signed-off-by: currantw <[email protected]>

Fix more failing tests

8eaf4dc

Signed-off-by: currantw <[email protected]>

Minor cleanup

5025ade

Signed-off-by: currantw <[email protected]>

Add new tests for IP data type to SortCommandIT, and update `weblog…

a58478f

…s` test data. Signed-off-by: currantw <[email protected]>

Fixing IT test failure.

2c91dd6

Signed-off-by: currantw <[email protected]>

Spotless and update test to sort in SQL

79bfc20

Signed-off-by: currantw <[email protected]>

Fix broken link

f40193c

Signed-off-by: currantw <[email protected]>

Fix failing code coverage

3926879

Signed-off-by: currantw <[email protected]>

Fix failing doctest

5be1be5

Signed-off-by: currantw <[email protected]>

Fix failing ip.rst doctest

b45114a

Signed-off-by: currantw <[email protected]>

currantw force-pushed the opensearch-sql-3145 branch from 3f9e428 to b45114a Compare December 13, 2024 07:26

currantw added 2 commits December 13, 2024 08:46

Fix test failure due to merge.

c166cdf

Signed-off-by: currantw <[email protected]>

Fix spotless

fd20fd0

Signed-off-by: currantw <[email protected]>

YANG-DB reviewed Dec 13, 2024

View reviewed changes

YANG-DB approved these changes Dec 13, 2024

View reviewed changes

acarbonetto reviewed Dec 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#3145 Add IP Address Data Type #3175

#3145 Add IP Address Data Type #3175

currantw commented Nov 29, 2024 •

edited

Loading

YANG-DB commented Dec 6, 2024

currantw commented Dec 10, 2024

YANG-DB commented Dec 10, 2024

currantw commented Dec 11, 2024

YANG-DB commented Dec 11, 2024

YANG-DB commented Dec 11, 2024

YANG-DB Dec 13, 2024

acarbonetto Dec 17, 2024

acarbonetto Dec 10, 2024

acarbonetto Dec 17, 2024

acarbonetto Dec 17, 2024

acarbonetto Dec 17, 2024

acarbonetto Dec 17, 2024

acarbonetto Dec 17, 2024

acarbonetto Dec 17, 2024

	public ExprIpValue(String s) {
	public ExprIpValue(String addressStr) {

#3145 Add IP Address Data Type #3175

Are you sure you want to change the base?

#3145 Add IP Address Data Type #3175

Conversation

currantw commented Nov 29, 2024 • edited Loading

Description

Related Issues

Check List

YANG-DB commented Dec 6, 2024

currantw commented Dec 10, 2024

YANG-DB commented Dec 10, 2024

currantw commented Dec 11, 2024

YANG-DB commented Dec 11, 2024

YANG-DB commented Dec 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

currantw commented Nov 29, 2024 •

edited

Loading