-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
#3145 Add IP Address Data Type #3175
Open
currantw
wants to merge
20
commits into
opensearch-project:main
Choose a base branch
from
Bit-Quill:opensearch-sql-3145
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
e2cfd1f
Add `ExprIpValue` and `IP` data type
currantw 8a00e0d
Add support for casting (`cast(field_name to ip)`) and remove existin…
currantw 62be5e0
Update comparison logic to compare in IPv6
currantw e83db18
Fix bug casting to IP
currantw a56c4ff
Fix failing tests
currantw 2c1f7da
Assert that comparison only valid if same type, update tests accordingly
currantw 0da521e
Add additional tests to increase code coverage
currantw 3179f4b
Integrate `cidrmatch` changes
currantw a64ad9a
Remove `OpenSearchIPType` data type
currantw 8eaf4dc
Fix more failing tests
currantw 5025ade
Minor cleanup
currantw a58478f
Add new tests for IP data type to `SortCommandIT`, and update `weblog…
currantw 2c91dd6
Fixing IT test failure.
currantw 79bfc20
Spotless and update test to sort in SQL
currantw f40193c
Fix broken link
currantw 3926879
Fix failing code coverage
currantw 5be1be5
Fix failing doctest
currantw b45114a
Fix failing `ip.rst` doctest
currantw c166cdf
Fix test failure due to merge.
currantw fd20fd0
Fix spotless
currantw File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
50 changes: 50 additions & 0 deletions
50
core/src/main/java/org/opensearch/sql/data/model/ExprIpValue.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,50 @@ | ||
/* | ||
* Copyright OpenSearch Contributors | ||
* SPDX-License-Identifier: Apache-2.0 | ||
*/ | ||
|
||
package org.opensearch.sql.data.model; | ||
|
||
import inet.ipaddr.IPAddress; | ||
import org.opensearch.sql.data.type.ExprCoreType; | ||
import org.opensearch.sql.data.type.ExprType; | ||
import org.opensearch.sql.utils.IPUtils; | ||
|
||
/** Expression IP Address Value. */ | ||
public class ExprIpValue extends AbstractExprValue { | ||
private final IPAddress value; | ||
|
||
public ExprIpValue(String s) { | ||
value = IPUtils.toAddress(s); | ||
} | ||
|
||
@Override | ||
public String value() { | ||
return value.toCanonicalString(); | ||
} | ||
|
||
@Override | ||
public ExprType type() { | ||
return ExprCoreType.IP; | ||
} | ||
|
||
@Override | ||
public int compare(ExprValue other) { | ||
return IPUtils.compare(value, ((ExprIpValue) other).value); | ||
} | ||
|
||
@Override | ||
public boolean equal(ExprValue other) { | ||
return compare(other) == 0; | ||
} | ||
|
||
@Override | ||
public String toString() { | ||
return String.format("IP %s", value()); | ||
} | ||
|
||
@Override | ||
public IPAddress ipValue() { | ||
return value; | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,21 +6,21 @@ | |
package org.opensearch.sql.expression.ip; | ||
|
||
import static org.opensearch.sql.data.type.ExprCoreType.BOOLEAN; | ||
import static org.opensearch.sql.data.type.ExprCoreType.IP; | ||
import static org.opensearch.sql.data.type.ExprCoreType.STRING; | ||
import static org.opensearch.sql.expression.function.FunctionDSL.define; | ||
import static org.opensearch.sql.expression.function.FunctionDSL.impl; | ||
import static org.opensearch.sql.expression.function.FunctionDSL.nullMissingHandling; | ||
|
||
import inet.ipaddr.AddressStringException; | ||
import inet.ipaddr.IPAddressString; | ||
import inet.ipaddr.IPAddressStringParameters; | ||
import inet.ipaddr.IPAddress; | ||
import lombok.experimental.UtilityClass; | ||
import org.opensearch.sql.data.model.ExprValue; | ||
import org.opensearch.sql.data.model.ExprValueUtils; | ||
import org.opensearch.sql.exception.SemanticCheckException; | ||
import org.opensearch.sql.expression.function.BuiltinFunctionName; | ||
import org.opensearch.sql.expression.function.BuiltinFunctionRepository; | ||
import org.opensearch.sql.expression.function.DefaultFunctionResolver; | ||
import org.opensearch.sql.utils.IPUtils; | ||
|
||
/** Utility class that defines and registers IP functions. */ | ||
@UtilityClass | ||
|
@@ -31,75 +31,35 @@ public void register(BuiltinFunctionRepository repository) { | |
} | ||
|
||
private DefaultFunctionResolver cidrmatch() { | ||
|
||
// TODO #3145: Add support for IP address data type. | ||
return define( | ||
BuiltinFunctionName.CIDRMATCH.getName(), | ||
impl(nullMissingHandling(IPFunctions::exprCidrMatch), BOOLEAN, STRING, STRING)); | ||
impl(nullMissingHandling(IPFunctions::exprCidrMatch), BOOLEAN, IP, STRING)); | ||
} | ||
|
||
/** | ||
* Returns whether the given IP address is within the specified inclusive CIDR IP address range. | ||
* Supports both IPv4 and IPv6 addresses. | ||
* | ||
* @param addressExprValue IP address as a string (e.g. "198.51.100.14" or | ||
* "2001:0db8::ff00:42:8329"). | ||
* @param rangeExprValue IP address range in CIDR notation as a string (e.g. "198.51.100.0/24" or | ||
* @param addressExprValue IP address (e.g. "198.51.100.14" or "2001:0db8::ff00:42:8329"). | ||
* @param rangeExprValue IP address range string in CIDR notation (e.g. "198.51.100.0/24" or | ||
* "2001:0db8::/32") | ||
* @return true if the address is in the range; otherwise false. | ||
* @throws SemanticCheckException if the address or range is not valid, or if they do not use the | ||
* same version (IPv4 or IPv6). | ||
*/ | ||
private ExprValue exprCidrMatch(ExprValue addressExprValue, ExprValue rangeExprValue) { | ||
|
||
// TODO #3145: Update to support IP address data type. | ||
String addressString = addressExprValue.stringValue(); | ||
String rangeString = rangeExprValue.stringValue(); | ||
|
||
final IPAddressStringParameters validationOptions = | ||
new IPAddressStringParameters.Builder() | ||
.allowEmpty(false) | ||
.setEmptyAsLoopback(false) | ||
.allow_inet_aton(false) | ||
.allowSingleSegment(false) | ||
.toParams(); | ||
|
||
// Get and validate IP address. | ||
IPAddressString address = | ||
new IPAddressString(addressExprValue.stringValue(), validationOptions); | ||
|
||
try { | ||
address.validate(); | ||
} catch (AddressStringException e) { | ||
String msg = | ||
String.format( | ||
"IP address '%s' is not valid. Error details: %s", addressString, e.getMessage()); | ||
throw new SemanticCheckException(msg, e); | ||
} | ||
|
||
// Get and validate CIDR IP address range. | ||
IPAddressString range = new IPAddressString(rangeExprValue.stringValue(), validationOptions); | ||
IPAddress address = addressExprValue.ipValue(); | ||
IPAddress range = IPUtils.toRange(rangeExprValue.stringValue()); | ||
|
||
try { | ||
range.validate(); | ||
} catch (AddressStringException e) { | ||
String msg = | ||
String.format( | ||
"CIDR IP address range '%s' is not valid. Error details: %s", | ||
rangeString, e.getMessage()); | ||
throw new SemanticCheckException(msg, e); | ||
if (IPUtils.compare(address, range.getLower()) < 0) { | ||
return ExprValueUtils.LITERAL_FALSE; | ||
} | ||
|
||
// Address and range must use the same IP version (IPv4 or IPv6). | ||
if (address.isIPv4() ^ range.isIPv4()) { | ||
String msg = | ||
String.format( | ||
"IP address '%s' and CIDR IP address range '%s' are not compatible. Both must be" | ||
+ " either IPv4 or IPv6.", | ||
addressString, rangeString); | ||
throw new SemanticCheckException(msg); | ||
if (IPUtils.compare(address, range.getUpper()) > 0) { | ||
return ExprValueUtils.LITERAL_FALSE; | ||
} | ||
|
||
return ExprValueUtils.booleanValue(range.contains(address)); | ||
return ExprValueUtils.LITERAL_TRUE; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit:
|
||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -11,6 +11,7 @@ | |
import static org.opensearch.sql.data.type.ExprCoreType.DOUBLE; | ||
import static org.opensearch.sql.data.type.ExprCoreType.FLOAT; | ||
import static org.opensearch.sql.data.type.ExprCoreType.INTEGER; | ||
import static org.opensearch.sql.data.type.ExprCoreType.IP; | ||
import static org.opensearch.sql.data.type.ExprCoreType.LONG; | ||
import static org.opensearch.sql.data.type.ExprCoreType.SHORT; | ||
import static org.opensearch.sql.data.type.ExprCoreType.STRING; | ||
|
@@ -31,6 +32,7 @@ | |
import org.opensearch.sql.data.model.ExprDoubleValue; | ||
import org.opensearch.sql.data.model.ExprFloatValue; | ||
import org.opensearch.sql.data.model.ExprIntegerValue; | ||
import org.opensearch.sql.data.model.ExprIpValue; | ||
import org.opensearch.sql.data.model.ExprLongValue; | ||
import org.opensearch.sql.data.model.ExprShortValue; | ||
import org.opensearch.sql.data.model.ExprStringValue; | ||
|
@@ -54,6 +56,7 @@ public static void register(BuiltinFunctionRepository repository) { | |
repository.register(castToFloat()); | ||
repository.register(castToDouble()); | ||
repository.register(castToBoolean()); | ||
repository.register(castToIp()); | ||
repository.register(castToDate()); | ||
repository.register(castToTime()); | ||
repository.register(castToTimestamp()); | ||
|
@@ -173,6 +176,13 @@ private static DefaultFunctionResolver castToBoolean() { | |
impl(nullMissingHandling((v) -> v), BOOLEAN, BOOLEAN)); | ||
} | ||
|
||
private static DefaultFunctionResolver castToIp() { | ||
return FunctionDSL.define( | ||
BuiltinFunctionName.CAST_TO_IP.getName(), | ||
impl(nullMissingHandling((v) -> v), IP, IP), | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. ordering matters - make sure to but the naive cast last |
||
impl(nullMissingHandling((v) -> new ExprIpValue(v.stringValue())), IP, STRING)); | ||
} | ||
|
||
private static DefaultFunctionResolver castToDate() { | ||
return FunctionDSL.define( | ||
BuiltinFunctionName.CAST_TO_DATE.getName(), | ||
|
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: