Skip to content

Commit

Permalink
Update 23.0.x (#31)
Browse files Browse the repository at this point in the history
* Buffer and coverage 22.0.x (#25)

* Jts upgrade 21.2 (#19)

* Additional Functionality (#13)

* Added intersection and collection support

* ST_Collect and ST_CollectAggregate now work alongside each other

* Added additional ST* functionality

* Minor name fixes

* Improved collection ser/de performance

* Import fixes

* Readme fixes

* Readme fixes

* Additional Functionality (#13) (#14)

* Added intersection and collection support

* ST_Collect and ST_CollectAggregate now work alongside each other

* Added additional ST* functionality

* Minor name fixes

* Improved collection ser/de performance

* Import fixes

* Readme fixes

* Readme fixes

* Work on angle functionality and proj4 transform

* Added missing files

* Added ST_Angle implementations

* Added Azimuth

* Added ST_Union Aggregate variant

* Some fixes following manual tests

* Jts upgrade 21.1 (#18)

* Additional Functionality (#13)

* Added intersection and collection support

* ST_Collect and ST_CollectAggregate now work alongside each other

* Added additional ST* functionality

* Minor name fixes

* Improved collection ser/de performance

* Import fixes

* Readme fixes

* Readme fixes

* Work on angle functionality and proj4 transform

* Added missing files

* Added ST_Angle implementations

* Added Azimuth

* Added ST_Union Aggregate variant

* Some fixes following manual tests

* Bad version fix

* 21.1.x add tests (#20)

* Dremio 21.2.x upgrade

* Updated WIKI link

* Additional Functionality (#13)

* Added intersection and collection support

* ST_Collect and ST_CollectAggregate now work alongside each other

* Added additional ST* functionality

* Minor name fixes

* Improved collection ser/de performance

* Import fixes

* Readme fixes

* Readme fixes

* Additional Functionality (#13) (#15)

* Added intersection and collection support

* ST_Collect and ST_CollectAggregate now work alongside each other

* Added additional ST* functionality

* Minor name fixes

* Improved collection ser/de performance

* Import fixes

* Readme fixes

* Readme fixes

* Work on angle functionality and proj4 transform

* Added missing files

* Added ST_Angle implementations

* Added Azimuth

* Added ST_Union Aggregate variant

* Some fixes following manual tests

* Readme Update

* version bump, updated readme

* ST_X tests foundations laid down

* Checkstyle IDEA

* Made improvement with test coverage

* A lot more work on tests
Better code formatting

* Some more relation tests cleanup

* Added detekt style checks and fixed CI

* Missing detekt plugin configuration

* Added output tests

* Add Geometry Input Tests

* Add Geometry input testing

* Added WKB Input tests

* Measurement functions testing

* Added overlay function tests

* Some more overlay function tests

* Added codecov

* Added GH hosted runners

* Coverall migration

* Coverall migration

* Coverall migration

* Coverall migration

* Coverall migration

* Added some more tests

* Added some ST_Transform test functionality

* Additional transformation tests

* Somme additional transformation tests

* Print evaluated geom

* Done scaling geometry values

* Done scaling geometry values

* Improve collection coverage

* Remove printing

* Added ST_Buffer tests and new variant

* Removed new buffer variant

* Added coveralls badge

Co-authored-by: Idan Sheinberg <[email protected]>

* Add tests 21.2.x (#21)

* Updated WIKI link

* Additional Functionality (#13)

* Added intersection and collection support

* ST_Collect and ST_CollectAggregate now work alongside each other

* Added additional ST* functionality

* Minor name fixes

* Improved collection ser/de performance

* Import fixes

* Readme fixes

* Readme fixes

* Additional Functionality (#13) (#14)

* Added intersection and collection support

* ST_Collect and ST_CollectAggregate now work alongside each other

* Added additional ST* functionality

* Minor name fixes

* Improved collection ser/de performance

* Import fixes

* Readme fixes

* Readme fixes

* Work on angle functionality and proj4 transform

* Added missing files

* Added ST_Angle implementations

* Added Azimuth

* Added ST_Union Aggregate variant

* Some fixes following manual tests

* Jts upgrade 21.1 (#18)

* Additional Functionality (#13)

* Added intersection and collection support

* ST_Collect and ST_CollectAggregate now work alongside each other

* Added additional ST* functionality

* Minor name fixes

* Improved collection ser/de performance

* Import fixes

* Readme fixes

* Readme fixes

* Work on angle functionality and proj4 transform

* Added missing files

* Added ST_Angle implementations

* Added Azimuth

* Added ST_Union Aggregate variant

* Some fixes following manual tests

* Bad version fix

* Readme Update

* version bump, updated readme

* ST_X tests foundations laid down

* Checkstyle IDEA

* Made improvement with test coverage

* A lot more work on tests
Better code formatting

* Some more relation tests cleanup

* Added detekt style checks and fixed CI

* Missing detekt plugin configuration

* Added output tests

* Add Geometry Input Tests

* Add Geometry input testing

* Added WKB Input tests

* Measurement functions testing

* Added overlay function tests

* Some more overlay function tests

* Added codecov

* Added GH hosted runners

* Coverall migration

* Coverall migration

* Coverall migration

* Coverall migration

* Coverall migration

* Added some more tests

* Added some ST_Transform test functionality

* Additional transformation tests

* Somme additional transformation tests

* Print evaluated geom

* Done scaling geometry values

* Done scaling geometry values

* Improve collection coverage

* Remove printing

* Added ST_Buffer tests and new variant

* Removed new buffer variant

* Added coveralls badge

Co-authored-by: Idan Sheinberg <[email protected]>

* Done with aggregate testing

* Done with aggregate testing

* Formatting fixes

* Dremio compat fixes

* Lot's of work on buffer parsing

* Code now compiles

* Added ST_Buffer tests

* Added ST_Buffer tests

* Added ST_Buffer tests

* Coverage targets

* Version alignment

* Version alignment

* Readme fix

Co-authored-by: Idan Sheinberg <[email protected]>

* Proj4 version bump
checkstyle lower requirements
Kotest/kotlin upgrades
Added ST_GeometryN, ST_GeomFromEWKT and ST_NumGeometries support

* Added ST_NumGeometries tests

* Detket fixes

* Improve test coverage

* Improve test coverage for new functionality
Detekt upgrade

* Improve test coverage for new functionality
Detekt upgrade

* Remove debug logging for polygon transformation

* Added ST_AsEWKT

* Dependency fixes

* Moved I/O function and buffer population to work with internal null handling

* Improved test coverage

* Detekt fixes

* Added more tests.
Fixed ST_Envelope
Moved more functions to use internal null handling

* Detekt fixes

* Added ST_IsClosed
More functions moved to internal null handling

* Fixed function name

* Fixed ST_Accessor behavior

* Ported functional fixes

* Added missing files

* minor fix

Co-authored-by: Idan Sheinberg <[email protected]>
  • Loading branch information
sheinbergon and 4msheinbergon authored Dec 16, 2022
1 parent 9151727 commit d39df57
Show file tree
Hide file tree
Showing 45 changed files with 924 additions and 112 deletions.
2 changes: 1 addition & 1 deletion .idea/kotlinc.xml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

16 changes: 7 additions & 9 deletions detekt.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ complexity:
active: false
threshold: 10
includeStaticDeclarations: false
ComplexMethod:
CyclomaticComplexMethod:
active: true
threshold: 10
ignoreSingleWhenExpression: true
Expand Down Expand Up @@ -246,7 +246,6 @@ formatting:
ParameterListWrapping:
active: true
autoCorrect: true
indentSize: 4
SpacingAroundColon:
active: true
autoCorrect: true
Expand Down Expand Up @@ -367,8 +366,6 @@ performance:

potential-bugs:
active: true
DuplicateCaseInWhenExpression:
active: true
EqualsAlwaysReturnsTrueOrFalse:
active: false
EqualsWithHashCodeExist:
Expand All @@ -388,8 +385,6 @@ potential-bugs:
excludes: "**/test/**,**/androidTest/**,**/*.Test.kt,**/*.Spec.kt,**/*.Spek.kt"
ignoreAnnotated: [ ]
ignoreOnClassesPattern: ""
MissingWhenCase:
active: false
UnconditionalJumpStatementInLoop:
active: false
UnreachableCode:
Expand All @@ -409,7 +404,8 @@ style:
active: false
DataClassContainsFunctions:
active: false
conversionFunctionPrefix: 'to'
conversionFunctionPrefix:
- 'to'
DataClassShouldBeImmutable:
active: false
DestructuringDeclarationWithTooManyEntries:
Expand All @@ -435,7 +431,8 @@ style:
FunctionOnlyReturningConstant:
active: false
ignoreOverridableFunction: true
excludedFunctions: 'describeContents'
excludedFunctions:
- 'describeContents'
LoopWithTooManyJumpStatements:
active: false
maxJumpCount: 1
Expand Down Expand Up @@ -485,7 +482,8 @@ style:
ReturnCount:
active: true
max: 2
excludedFunctions: "equals"
excludedFunctions:
- "equals"
excludeLabeled: false
excludeReturnFromLambda: true
SafeCast:
Expand Down
23 changes: 17 additions & 6 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -8,20 +8,20 @@
<checkstyle.version>9.3</checkstyle.version>
<dremio.version>23.0.1-202210141019030815-c1de8bcc</dremio.version>
<dremio-arrow.version>9.0.0-20220922165652-3a8c29faef-dremio</dremio-arrow.version>
<proj4j.version>1.1.5</proj4j.version>
<proj4j.version>1.2.0</proj4j.version>
<jts-core.version>1.19.0</jts-core.version>
<jts-io-common.version>1.19.0</jts-io-common.version>
<commons-io.version>2.11.0</commons-io.version>
<junit.version>4.13.2</junit.version>
<junit-jupiter.version>5.9.0</junit-jupiter.version>
<kotlin.version>1.7.20</kotlin.version>
<kotest.version>5.5.3</kotest.version>
<junit-jupiter.version>5.9.1</junit-jupiter.version>
<kotlin.version>1.7.21</kotlin.version>
<kotest.version>5.5.4</kotest.version>
<joda.version>2.9</joda.version>
<detekt.version>1.21.0</detekt.version>
<detekt.version>1.22.0</detekt.version>
<carrotsearch.version>0.7.0</carrotsearch.version>
<arrow-memory-netty.version>9.0.0</arrow-memory-netty.version>
</properties>
<version>0.7.0${artifact.version.suffix}</version>
<version>0.7.2${artifact.version.suffix}</version>
<name>dremio-udf-gis</name>
<description>GIS UDF extensions for Dremio</description>
<url>https://github.com/sheinbergon/dremio-udf-gis</url>
Expand Down Expand Up @@ -133,6 +133,11 @@
<artifactId>proj4j</artifactId>
<version>${proj4j.version}</version>
</dependency>
<dependency>
<groupId>org.locationtech.proj4j</groupId>
<artifactId>proj4j-epsg</artifactId>
<version>${proj4j.version}</version>
</dependency>
<dependency>
<groupId>org.locationtech.jts</groupId>
<artifactId>jts-core</artifactId>
Expand All @@ -155,6 +160,12 @@
<version>${commons-io.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>${commons-io.version}</version>
<scope>provided</scope>
</dependency>
<!-- Test Dependencies -->
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
Expand Down
55 changes: 55 additions & 0 deletions src/main/java/org/sheinbergon/dremio/udf/gis/STAsEWKT.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
* <p>
* http://www.apache.org/licenses/LICENSE-2.0
* <p>
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.sheinbergon.dremio.udf.gis;

import com.dremio.exec.expr.SimpleFunction;
import com.dremio.exec.expr.annotations.FunctionTemplate;
import com.dremio.exec.expr.annotations.Output;
import com.dremio.exec.expr.annotations.Param;

import javax.inject.Inject;

@FunctionTemplate(
name = "ST_AsEWKT",
scope = FunctionTemplate.FunctionScope.SIMPLE,
nulls = FunctionTemplate.NullHandling.INTERNAL)
public class STAsEWKT implements SimpleFunction {

@Param
org.apache.arrow.vector.holders.NullableVarBinaryHolder binaryInput;

@Output
org.apache.arrow.vector.holders.NullableVarCharHolder ewktOutput;

@Inject
org.apache.arrow.memory.ArrowBuf buffer;

public void setup() {
}

public void eval() {
if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isHolderSet(binaryInput)) {
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toEWKT(geom);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, ewktOutput);
} else {
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(ewktOutput);
}
}
}
16 changes: 9 additions & 7 deletions src/main/java/org/sheinbergon/dremio/udf/gis/STAsGeoJson.java
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,12 @@
import com.dremio.exec.expr.annotations.Output;
import com.dremio.exec.expr.annotations.Param;



import javax.inject.Inject;

@FunctionTemplate(
name = "ST_AsGeoJSON",
scope = FunctionTemplate.FunctionScope.SIMPLE,
nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
nulls = FunctionTemplate.NullHandling.INTERNAL)
public class STAsGeoJson implements SimpleFunction {

@Param
Expand All @@ -45,9 +43,13 @@ public void setup() {
}

public void eval() {
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeoJson(geom);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, geoJsonOutput);
if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isHolderSet(binaryInput)) {
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeoJson(geom);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, geoJsonOutput);
} else {
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(geoJsonOutput);
}
}
}
14 changes: 9 additions & 5 deletions src/main/java/org/sheinbergon/dremio/udf/gis/STAsText.java
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@
@FunctionTemplate(
name = "ST_AsText",
scope = FunctionTemplate.FunctionScope.SIMPLE,
nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
nulls = FunctionTemplate.NullHandling.INTERNAL)
public class STAsText implements SimpleFunction {

@Param
Expand All @@ -45,9 +45,13 @@ public void setup() {
}

public void eval() {
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toText(geom);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, wktOutput);
if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isHolderSet(binaryInput)) {
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toText(geom);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, wktOutput);
} else {
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(wktOutput);
}
}
}
20 changes: 12 additions & 8 deletions src/main/java/org/sheinbergon/dremio/udf/gis/STCentroid.java
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
@FunctionTemplate(
name = "ST_Centroid",
scope = FunctionTemplate.FunctionScope.SIMPLE,
nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
nulls = FunctionTemplate.NullHandling.INTERNAL)
public class STCentroid implements SimpleFunction {
@Param
org.apache.arrow.vector.holders.NullableVarBinaryHolder binaryInput;
Expand All @@ -43,12 +43,16 @@ public void setup() {
}

public void eval() {
org.locationtech.jts.geom.GeometryFactory factory = new GeometryFactory();
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
org.locationtech.jts.algorithm.Centroid centroid = new org.locationtech.jts.algorithm.Centroid(geom);
org.locationtech.jts.geom.Point point = factory.createPoint(centroid.getCentroid());
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(point);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, binaryOutput);
if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isHolderSet(binaryInput)) {
org.locationtech.jts.geom.GeometryFactory factory = new GeometryFactory();
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
org.locationtech.jts.algorithm.Centroid centroid = new org.locationtech.jts.algorithm.Centroid(geom);
org.locationtech.jts.geom.Point point = factory.createPoint(centroid.getCentroid());
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(point);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, binaryOutput);
} else {
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(binaryOutput);
}
}
}
18 changes: 11 additions & 7 deletions src/main/java/org/sheinbergon/dremio/udf/gis/STConvexHull.java
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
@FunctionTemplate(
name = "ST_ConvexHull",
scope = FunctionTemplate.FunctionScope.SIMPLE,
nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
nulls = FunctionTemplate.NullHandling.INTERNAL)
public class STConvexHull implements SimpleFunction {
@Param
org.apache.arrow.vector.holders.NullableVarBinaryHolder binaryInput;
Expand All @@ -42,11 +42,15 @@ public void setup() {
}

public void eval() {
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
org.locationtech.jts.algorithm.ConvexHull convexHull = new org.locationtech.jts.algorithm.ConvexHull(geom);
org.locationtech.jts.geom.Geometry hull = convexHull.getConvexHull();
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(hull);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, binaryOutput);
if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isHolderSet(binaryInput)) {
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
org.locationtech.jts.algorithm.ConvexHull convexHull = new org.locationtech.jts.algorithm.ConvexHull(geom);
org.locationtech.jts.geom.Geometry hull = convexHull.getConvexHull();
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(hull);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, binaryOutput);
} else {
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(binaryOutput);
}
}
}
22 changes: 8 additions & 14 deletions src/main/java/org/sheinbergon/dremio/udf/gis/STEnvelope.java
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,12 @@
import com.dremio.exec.expr.annotations.Output;
import com.dremio.exec.expr.annotations.Param;



import javax.inject.Inject;

@FunctionTemplate(
name = "ST_Envelope",
scope = FunctionTemplate.FunctionScope.SIMPLE,
nulls = FunctionTemplate.NullHandling.NULL_IF_NULL)
nulls = FunctionTemplate.NullHandling.INTERNAL)
public class STEnvelope implements SimpleFunction {
@Param
org.apache.arrow.vector.holders.NullableVarBinaryHolder binaryInput;
Expand All @@ -44,18 +42,14 @@ public void setup() {
}

public void eval() {
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
org.locationtech.jts.geom.Geometry enveloped = envelope(geom);
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(enveloped);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, binaryOutput);
}

private org.locationtech.jts.geom.Geometry envelope(final org.locationtech.jts.geom.Geometry geometry) {
if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isAPoint(geometry)) {
return geometry;
if (org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.isHolderSet(binaryInput)) {
org.locationtech.jts.geom.Geometry geom = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toGeometry(binaryInput);
org.locationtech.jts.geom.Geometry envelope = geom.getEnvelope();
byte[] bytes = org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.toBinary(envelope);
buffer = buffer.reallocIfNeeded(bytes.length);
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.populate(bytes, buffer, binaryOutput);
} else {
return geometry.getEnvelope();
org.sheinbergon.dremio.udf.gis.util.GeometryHelpers.markHolderNotSet(binaryOutput);
}
}
}
Loading

0 comments on commit d39df57

Please sign in to comment.