Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/main' into mapping_context_bit…
Browse files Browse the repository at this point in the history
…setproducer
  • Loading branch information
jimczi committed Jun 4, 2024
2 parents a30f686 + 4d91f0f commit d7d2a54
Show file tree
Hide file tree
Showing 10 changed files with 18 additions and 286 deletions.
8 changes: 8 additions & 0 deletions docs/changelog/107377.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,11 @@ summary: Add support for the 'Enterprise' database to the geoip processor
area: Ingest Node
type: enhancement
issues: []
highlight:
title: "Preview: Support for the 'Anonymous IP' and 'Enterprise' databases in the geoip processor"
body: |-
As a Technical Preview, the {ref}/geoip-processor.html[`geoip`] processor can now use the commercial
https://www.maxmind.com/en/solutions/geoip2-enterprise-product-suite/enterprise-database[GeoIP2 'Enterprise']
and
https://www.maxmind.com/en/solutions/geoip2-enterprise-product-suite/anonymous-ip-database[GeoIP2 'Anonymous IP']
databases from MaxMind.
23 changes: 0 additions & 23 deletions docs/changelog/108639.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,26 +3,3 @@ summary: Add support for the 'Domain' database to the geoip processor
area: Ingest Node
type: enhancement
issues: []
highlight:
title: Add support for the 'Domain' database to the geoip processor
body: |-
Follow on to #107287 and #107377
Adds support for the ['GeoIP2
Domain'](https://dev.maxmind.com/geoip/docs/databases/domain) database
from MaxMind to the `geoip` processor.
The `geoip` processor will automatically download the [various
'GeoLite2'
databases](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data),
but the 'GeoIP2 Domain' database is not a 'GeoLite2' database -- it's a
commercial database available to those with a suitable license from
MaxMind.
The support that is being added for it in this PR is in line with the
support that we already have for MaxMind's 'GeoIP2 City' and 'GeoIP2
Country' databases -- that is, one would need to arrange their own
download management via some custom endpoint or otherwise arrange for
the relevant file(s) to be in the `$ES_CONFIG/ingest-geoip` directory on
the nodes of the cluster.
notable: true
24 changes: 0 additions & 24 deletions docs/changelog/108651.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,27 +3,3 @@ summary: Add support for the 'ISP' database to the geoip processor
area: Ingest Node
type: enhancement
issues: []
highlight:
title: Add support for the 'ISP' database to the geoip processor
body: |-
Follow on to https://github.com/elastic/elasticsearch/pull/107287,
https://github.com/elastic/elasticsearch/pull/107377, and
https://github.com/elastic/elasticsearch/pull/108639
Adds support for the ['GeoIP2
ISP'](https://dev.maxmind.com/geoip/docs/databases/isp) database from
MaxMind to the geoip processor.
The geoip processor will automatically download the [various 'GeoLite2'
databases](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data),
but the 'GeoIP2 ISP' database is not a 'GeoLite2' database -- it's a
commercial database available to those with a suitable license from
MaxMind.
The support that is being added for it in this PR is in line with the
support that we already have for MaxMind's 'GeoIP2 City' and 'GeoIP2
Country' databases -- that is, one would need to arrange their own
download management via some custom endpoint or otherwise arrange for
the relevant file(s) to be in the $ES_CONFIG/ingest-geoip directory on
the nodes of the cluster.
notable: true
28 changes: 7 additions & 21 deletions docs/changelog/108683.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,11 @@ area: Ingest Node
type: enhancement
issues: []
highlight:
title: Add support for the 'Connection Type' database to the geoip processor
title: "Preview: Support for the 'Connection Type, 'Domain', and 'ISP' databases in the geoip processor"
body: |-
Follow on to #107287, #107377, #108639, and #108651
Adds support for the ['GeoIP2 Connection
Type'](https://dev.maxmind.com/geoip/docs/databases/connection-type)
database from MaxMind to the `geoip` processor.
The `geoip` processor will automatically download the [various
'GeoLite2'
databases](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data),
but the 'GeoIP2 Connection Type' database is not a 'GeoLite2' database
-- it's a commercial database available to those with a suitable license
from MaxMind.
The support that is being added for it in this PR is in line with the
support that we already have for MaxMind's 'GeoIP2 City' and 'GeoIP2
Country' databases -- that is, one would need to arrange their own
download management via some custom endpoint or otherwise arrange for
the relevant file(s) to be in the `$ES_CONFIG/ingest-geoip` directory on
the nodes of the cluster.
notable: true
As a Technical Preview, the {ref}/geoip-processor.html[`geoip`] processor can now use the commercial
https://dev.maxmind.com/geoip/docs/databases/connection-type[GeoIP2 'Connection Type'],
https://dev.maxmind.com/geoip/docs/databases/domain[GeoIP2 'Domain'],
and
https://dev.maxmind.com/geoip/docs/databases/isp[GeoIP2 'ISP']
databases from MaxMind.
1 change: 1 addition & 0 deletions docs/reference/ingest/processors/geoip.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ depend on what has been found and which properties were configured in `propertie
`residential_proxy`, `domain`, `isp`, `isp_organization`, `mobile_country_code`, `mobile_network_code`, `user_type`, and
`connection_type`. The fields actually added depend on what has been found and which properties were configured in `properties`.

preview::["Do not use the GeoIP2 Anonymous IP, GeoIP2 Connection Type, GeoIP2 Domain, GeoIP2 ISP, and GeoIP2 Enterprise databases in production environments. This functionality is in technical preview and may be changed or removed in a future release. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features."]

Here is an example that uses the default city database and adds the geographical information to the `geoip` field based on the `ip` field:

Expand Down
3 changes: 2 additions & 1 deletion x-pack/plugin/core/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -191,5 +191,6 @@ tasks.register('testAutomatonPatterns', Test) {

tasks.named('test').configure {
exclude '**/AutomatonPatternsTests.class'
dependsOn testAutomatonPatterns //to ensure testAutomatonPatterns are run with the test task
}

tasks.named("check").configure { dependsOn "testAutomatonPatterns" }
Original file line number Diff line number Diff line change
Expand Up @@ -8,24 +8,10 @@
package org.elasticsearch.xpack.esql.core.analyzer;

import org.elasticsearch.xpack.esql.core.expression.Attribute;
import org.elasticsearch.xpack.esql.core.expression.Expression;
import org.elasticsearch.xpack.esql.core.expression.FieldAttribute;
import org.elasticsearch.xpack.esql.core.expression.Literal;
import org.elasticsearch.xpack.esql.core.expression.UnresolvedAttribute;
import org.elasticsearch.xpack.esql.core.expression.function.Function;
import org.elasticsearch.xpack.esql.core.expression.function.FunctionDefinition;
import org.elasticsearch.xpack.esql.core.expression.function.FunctionRegistry;
import org.elasticsearch.xpack.esql.core.expression.function.UnresolvedFunction;
import org.elasticsearch.xpack.esql.core.expression.predicate.logical.BinaryLogic;
import org.elasticsearch.xpack.esql.core.expression.predicate.operator.comparison.Equals;
import org.elasticsearch.xpack.esql.core.plan.logical.Filter;
import org.elasticsearch.xpack.esql.core.plan.logical.LogicalPlan;
import org.elasticsearch.xpack.esql.core.rule.ParameterizedRule;
import org.elasticsearch.xpack.esql.core.rule.Rule;
import org.elasticsearch.xpack.esql.core.session.Configuration;
import org.elasticsearch.xpack.esql.core.type.DataTypes;
import org.elasticsearch.xpack.esql.core.type.InvalidMappedField;
import org.elasticsearch.xpack.esql.core.type.UnsupportedEsField;

import java.util.ArrayList;
import java.util.Collection;
Expand All @@ -34,48 +20,10 @@
import java.util.function.Predicate;
import java.util.function.Supplier;

import static java.util.Arrays.asList;
import static java.util.Collections.singletonList;
import static org.elasticsearch.xpack.esql.core.type.DataTypes.BOOLEAN;

public final class AnalyzerRules {

public static class AddMissingEqualsToBoolField extends AnalyzerRule<Filter> {

@Override
protected LogicalPlan rule(Filter filter) {
if (filter.resolved() == false) {
return filter;
}
// check the condition itself
Expression condition = replaceRawBoolFieldWithEquals(filter.condition());
// otherwise look for binary logic
if (condition == filter.condition()) {
condition = condition.transformUp(
BinaryLogic.class,
b -> b.replaceChildren(asList(replaceRawBoolFieldWithEquals(b.left()), replaceRawBoolFieldWithEquals(b.right())))
);
}

if (condition != filter.condition()) {
filter = filter.with(condition);
}
return filter;
}

private static Expression replaceRawBoolFieldWithEquals(Expression e) {
if (e instanceof FieldAttribute && e.dataType() == BOOLEAN) {
e = new Equals(e.source(), e, Literal.of(e, Boolean.TRUE));
}
return e;
}

@Override
protected boolean skipResolved() {
return false;
}
}

public abstract static class AnalyzerRule<SubPlan extends LogicalPlan> extends Rule<SubPlan, LogicalPlan> {

// transformUp (post-order) - that is first children and then the node
Expand Down Expand Up @@ -123,24 +71,6 @@ protected LogicalPlan rule(LogicalPlan plan) {
protected abstract LogicalPlan doRule(LogicalPlan plan);
}

public static Function resolveFunction(UnresolvedFunction uf, Configuration configuration, FunctionRegistry functionRegistry) {
Function f = null;
if (uf.analyzed()) {
f = uf;
} else if (uf.childrenResolved() == false) {
f = uf;
} else {
String functionName = functionRegistry.resolveAlias(uf.name());
if (functionRegistry.functionExists(functionName) == false) {
f = uf.missing(functionName, functionRegistry.listFunctions());
} else {
FunctionDefinition def = functionRegistry.resolveFunction(functionName);
f = uf.buildResolved(configuration, def);
}
}
return f;
}

public static List<Attribute> maybeResolveAgainstList(
UnresolvedAttribute u,
Collection<Attribute> attrList,
Expand Down Expand Up @@ -218,42 +148,4 @@ public static List<Attribute> maybeResolveAgainstList(
)
);
}

public static Attribute handleSpecialFields(UnresolvedAttribute u, Attribute named, boolean allowCompound) {
// if it's a object/compound type, keep it unresolved with a nice error message
if (named instanceof FieldAttribute fa) {

// incompatible mappings
if (fa.field() instanceof InvalidMappedField imf) {
named = u.withUnresolvedMessage("Cannot use field [" + fa.name() + "] due to ambiguities being " + imf.errorMessage());
}
// unsupported types
else if (DataTypes.isUnsupported(fa.dataType())) {
UnsupportedEsField unsupportedField = (UnsupportedEsField) fa.field();
if (unsupportedField.hasInherited()) {
named = u.withUnresolvedMessage(
"Cannot use field ["
+ fa.name()
+ "] with unsupported type ["
+ unsupportedField.getOriginalType()
+ "] in hierarchy (field ["
+ unsupportedField.getInherited()
+ "])"
);
} else {
named = u.withUnresolvedMessage(
"Cannot use field [" + fa.name() + "] with unsupported type [" + unsupportedField.getOriginalType() + "]"
);
}
}
// compound fields
else if (allowCompound == false && DataTypes.isPrimitive(fa.dataType()) == false) {
named = u.withUnresolvedMessage(
"Cannot use field [" + fa.name() + "] type [" + fa.dataType().typeName() + "] only its subfields"
);
}
}
// make sure to copy the resolved attribute with the proper location
return named.withLocation(u.source());
}
}

This file was deleted.

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,4 @@ org.elasticsearch.xpack.esql.core.plan.logical.Aggregate @ use @org.elasticsearc
org.elasticsearch.xpack.esql.core.plan.logical.EsRelation @ use @org.elasticsearch.xpack.esql.plan.logical.EsRelation instead
org.elasticsearch.xpack.esql.core.plan.logical.Project @ use @org.elasticsearch.xpack.esql.plan.logical.Project instead
org.elasticsearch.xpack.esql.core.plan.logical.UnresolvedRelation @ use @org.elasticsearch.xpack.esql.plan.logical.UnresolvedRelation instead
org.elasticsearch.xpack.esql.core.analyzer.PreAnalyzer @ use @org.elasticsearch.xpack.esql.analysis.PreAnalyzer instead
org.elasticsearch.xpack.esql.core.expression.function.Functions @ use @org.elasticsearch.xpack.esql.expression.function.Functions instead

0 comments on commit d7d2a54

Please sign in to comment.