Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNOW-969794 Arrow native structured type #1649

Merged
merged 51 commits into from
Mar 22, 2024
Merged
Show file tree
Hide file tree
Changes from 45 commits
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
a9551fe
SNOW-970859 Implement getObject for structured types for JSON response
sfc-gh-pfus Feb 1, 2024
17f18bd
SNOW-974576 Fixes in FieldsMetadata test
sfc-gh-pmotacki Feb 5, 2024
52cdbad
SNOW-974576 Add JsonConverters in ArrowResultSet for struct
sfc-gh-pmotacki Feb 5, 2024
fa4cda5
SNOW-974576 ConditionalIgnore dla Structured Type tests
sfc-gh-pmotacki Feb 6, 2024
988d4ed
SNOW-970859 Fix of snowflake types mapping.
sfc-gh-pmotacki Feb 7, 2024
3a04312
SNOW-970859 Fix of snowflake types mapping.
sfc-gh-pmotacki Feb 12, 2024
0e5f191
SNOW-970859 Fix of snowflake types mapping.
sfc-gh-pmotacki Feb 12, 2024
63ba15c
SNOW-970859 Fix of snowflake types mapping.
sfc-gh-pmotacki Feb 16, 2024
2cfa1e9
Add native arrow structured types support
sfc-gh-dheyman Feb 24, 2024
7f0ed85
Fixed conflicts
sfc-gh-dheyman Mar 7, 2024
d474ce9
Conflicts resolved
sfc-gh-dheyman Mar 8, 2024
c262c01
Add time-related converter
sfc-gh-dheyman Mar 12, 2024
2e2074d
structured types arrow Timestamp refactor
sfc-gh-dheyman Mar 12, 2024
0682a13
Add timestamp into account when parsing NTZ
sfc-gh-dheyman Mar 14, 2024
cc8ca5f
COnflicts resolved
sfc-gh-dheyman Mar 14, 2024
c26c1a4
Change ResultSetStructuredTypesLatesIt to JSON
sfc-gh-dheyman Mar 14, 2024
f739b7e
Add structured types format check in tests
sfc-gh-dheyman Mar 14, 2024
17f4599
reformatted
sfc-gh-dheyman Mar 14, 2024
b32a858
Fix checkstyle violations
sfc-gh-dheyman Mar 14, 2024
06610c6
Fixed formatting
sfc-gh-dheyman Mar 14, 2024
99b89f1
Removed duplicate honorClientTZForTimestampNTZ assign
sfc-gh-dheyman Mar 14, 2024
6aa73f4
Code review suggestions
sfc-gh-dheyman Mar 14, 2024
e123d8a
Extracted BaseSQlInput
sfc-gh-dheyman Mar 19, 2024
34d4b45
Add copyright
sfc-gh-dheyman Mar 19, 2024
cb1fc73
CR suggestions
sfc-gh-dheyman Mar 19, 2024
da8f52e
reformat
sfc-gh-dheyman Mar 19, 2024
73b0afc
Fix TZ converter
sfc-gh-dheyman Mar 20, 2024
5c52a98
Reverted fromToString removal
sfc-gh-dheyman Mar 21, 2024
defb067
Add timestamp formatter test
sfc-gh-dheyman Mar 21, 2024
34ad6fc
Add explicit timezone to format util test
sfc-gh-dheyman Mar 21, 2024
a90eb2e
Add internal api annotations to according classes
sfc-gh-dheyman Mar 21, 2024
5d0ed00
CR suggestions
sfc-gh-dheyman Mar 21, 2024
1b843dc
Reformat
sfc-gh-dheyman Mar 21, 2024
58a803e
a
sfc-gh-dheyman Mar 21, 2024
e211ee2
a
sfc-gh-dheyman Mar 21, 2024
7ab78af
e
sfc-gh-dheyman Mar 21, 2024
295554f
a
sfc-gh-dheyman Mar 21, 2024
a90f7e7
Change ignore timezone
sfc-gh-dheyman Mar 21, 2024
7ba81d5
Change timezone to CET in test
sfc-gh-dheyman Mar 21, 2024
cbac5c0
a
sfc-gh-dheyman Mar 21, 2024
bbe6ff8
a
sfc-gh-dheyman Mar 21, 2024
86e343d
a
sfc-gh-dheyman Mar 21, 2024
2267266
a
sfc-gh-dheyman Mar 21, 2024
862cf18
a
sfc-gh-dheyman Mar 21, 2024
3d38fc6
Add timezone printing
sfc-gh-dheyman Mar 21, 2024
740d900
CR suggestions
sfc-gh-dheyman Mar 22, 2024
348cfee
a
sfc-gh-dheyman Mar 22, 2024
0afe368
Conflicts resolved
sfc-gh-dheyman Mar 22, 2024
014b8ff
Checkstyle fixed
sfc-gh-dheyman Mar 22, 2024
5c230df
Fix columnIndex
sfc-gh-dheyman Mar 22, 2024
b0bef52
Fix OBJECT converter handling
sfc-gh-dheyman Mar 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
215 changes: 215 additions & 0 deletions src/main/java/net/snowflake/client/core/ArrowSqlInput.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
/*
* Copyright (c) 2012-2024 Snowflake Computing Inc. All rights reserved.
*/

package net.snowflake.client.core;

import static net.snowflake.client.jdbc.SnowflakeUtil.mapExceptions;

import java.math.BigDecimal;
import java.sql.Date;
import java.sql.SQLData;
import java.sql.SQLException;
import java.sql.Time;
import java.sql.Timestamp;
import java.util.Iterator;
import java.util.List;
import java.util.TimeZone;
import net.snowflake.client.core.json.Converters;
sfc-gh-pfus marked this conversation as resolved.
Show resolved Hide resolved
import net.snowflake.client.core.structs.SQLDataCreationHelper;
import net.snowflake.client.jdbc.FieldMetadata;
import net.snowflake.client.util.ThrowingBiFunction;
import org.apache.arrow.vector.util.JsonStringHashMap;

@SnowflakeJdbcInternalApi
public class ArrowSqlInput extends BaseSqlInput {

private final Iterator<Object> structuredTypeFields;
private int currentIndex = 0;

public ArrowSqlInput(
JsonStringHashMap<String, Object> input,
SFBaseSession session,
Converters converters,
List<FieldMetadata> fields) {
super(session, converters, fields);
this.structuredTypeFields = input.values().iterator();
}

@Override
public String readString() throws SQLException {
sfc-gh-pfus marked this conversation as resolved.
Show resolved Hide resolved
return withNextValue(
((value, fieldMetadata) -> {
int columnType = ColumnTypeHelper.getColumnType(fieldMetadata.getType(), session);
int columnSubType = fieldMetadata.getType();
int scale = fieldMetadata.getScale();
return mapExceptions(
() ->
converters
.getStringConverter()
.getString(value, columnType, columnSubType, scale));
}));
}

@Override
public boolean readBoolean() throws SQLException {
return withNextValue(
(value, fieldMetadata) -> {
int columnType = ColumnTypeHelper.getColumnType(fieldMetadata.getType(), session);
return mapExceptions(
() -> converters.getBooleanConverter().getBoolean(value, columnType));
});
}

@Override
public byte readByte() throws SQLException {
return withNextValue(
(value, fieldMetadata) ->
mapExceptions(() -> converters.getNumberConverter().getByte(value)));
}

@Override
public short readShort() throws SQLException {
return withNextValue(
(value, fieldMetadata) -> {
int columnType = ColumnTypeHelper.getColumnType(fieldMetadata.getType(), session);
return mapExceptions(() -> converters.getNumberConverter().getShort(value, columnType));
});
}

@Override
public int readInt() throws SQLException {
return withNextValue(
(value, fieldMetadata) -> {
int columnType = ColumnTypeHelper.getColumnType(fieldMetadata.getType(), session);
return mapExceptions(() -> converters.getNumberConverter().getInt(value, columnType));
});
}

@Override
public long readLong() throws SQLException {
return withNextValue(
(value, fieldMetadata) -> {
int columnType = ColumnTypeHelper.getColumnType(fieldMetadata.getType(), session);
return mapExceptions(() -> converters.getNumberConverter().getLong(value, columnType));
});
}

@Override
public float readFloat() throws SQLException {
return withNextValue(
(value, fieldMetadata) -> {
int columnType = ColumnTypeHelper.getColumnType(fieldMetadata.getType(), session);
return mapExceptions(() -> converters.getNumberConverter().getFloat(value, columnType));
});
}

@Override
public double readDouble() throws SQLException {
return withNextValue(
(value, fieldMetadata) -> {
int columnType = ColumnTypeHelper.getColumnType(fieldMetadata.getType(), session);
return mapExceptions(() -> converters.getNumberConverter().getDouble(value, columnType));
});
}

@Override
public BigDecimal readBigDecimal() throws SQLException {
return withNextValue(
(value, fieldMetadata) -> {
int columnType = ColumnTypeHelper.getColumnType(fieldMetadata.getType(), session);
return mapExceptions(
() -> converters.getNumberConverter().getBigDecimal(value, columnType));
});
}

@Override
public byte[] readBytes() throws SQLException {
return withNextValue(
(value, fieldMetadata) -> {
int columnType = ColumnTypeHelper.getColumnType(fieldMetadata.getType(), session);
int columnSubType = fieldMetadata.getType();
int scale = fieldMetadata.getScale();
return mapExceptions(
() ->
converters.getBytesConverter().getBytes(value, columnType, columnSubType, scale));
});
}

@Override
public Date readDate() throws SQLException {
return withNextValue(
(value, fieldMetadata) ->
mapExceptions(
() ->
converters
.getStructuredTypeDateTimeConverter()
.getDate((int) value, TimeZone.getDefault())));
}

@Override
public Time readTime() throws SQLException {
return withNextValue(
(value, fieldMetadata) ->
mapExceptions(
() -> {
int scale = fieldMetadata.getScale();
return converters
.getStructuredTypeDateTimeConverter()
.getTime((long) value, scale);
}));
}

@Override
public Timestamp readTimestamp(TimeZone tz) throws SQLException {
return withNextValue(
(value, fieldMetadata) -> {
if (value == null) {
return null;
}
int scale = fieldMetadata.getScale();
return mapExceptions(
() ->
converters
.getStructuredTypeDateTimeConverter()
.getTimestamp(
(JsonStringHashMap<String, Object>) value,
fieldMetadata.getBase(),
tz,
scale));
});
}

@Override
public Object readObject() throws SQLException {
return withNextValue(
(value, fieldMetadata) -> {
if (!(value instanceof JsonStringHashMap)) {
throw new SQLException(
"Invalid value passed to 'readObject()', expected Map; got: " + value.getClass());
}
return value;
});
}

@Override
public <T> T readObject(Class<T> type) throws SQLException {
return withNextValue(
(value, fieldMetadata) -> {
SQLData instance = (SQLData) SQLDataCreationHelper.create(type);
instance.readSQL(
new ArrowSqlInput(
(JsonStringHashMap<String, Object>) value,
session,
converters,
fieldMetadata.getFields()),
null);
return (T) instance;
});
}

private <T> T withNextValue(ThrowingBiFunction<Object, FieldMetadata, T, SQLException> action)
throws SQLException {
return action.apply(structuredTypeFields.next(), fields.get(currentIndex++));
}
}
106 changes: 106 additions & 0 deletions src/main/java/net/snowflake/client/core/BaseSqlInput.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
/*
* Copyright (c) 2012-2024 Snowflake Computing Inc. All rights reserved.
*/

package net.snowflake.client.core;

import java.io.InputStream;
import java.io.Reader;
import java.net.URL;
import java.sql.Array;
import java.sql.Blob;
import java.sql.Clob;
import java.sql.NClob;
import java.sql.Ref;
import java.sql.RowId;
import java.sql.SQLException;
import java.sql.SQLXML;
import java.sql.Timestamp;
import java.util.List;
import net.snowflake.client.core.json.Converters;
import net.snowflake.client.jdbc.FieldMetadata;
import net.snowflake.client.jdbc.SnowflakeLoggedFeatureNotSupportedException;

@SnowflakeJdbcInternalApi
public abstract class BaseSqlInput implements SFSqlInput {

protected final SFBaseSession session;
protected final Converters converters;
protected final List<FieldMetadata> fields;

protected BaseSqlInput(SFBaseSession session, Converters converters, List<FieldMetadata> fields) {
this.session = session;
this.converters = converters;
this.fields = fields;
}

@Override
public Timestamp readTimestamp() throws SQLException {
return readTimestamp(null);
}

@Override
public Reader readCharacterStream() throws SQLException {
throw new SnowflakeLoggedFeatureNotSupportedException(session, "readCharacterStream");
}

@Override
public InputStream readAsciiStream() throws SQLException {
throw new SnowflakeLoggedFeatureNotSupportedException(session, "readAsciiStream");
}

@Override
public InputStream readBinaryStream() throws SQLException {
throw new SnowflakeLoggedFeatureNotSupportedException(session, "readBinaryStream");
}

@Override
public Ref readRef() throws SQLException {
throw new SnowflakeLoggedFeatureNotSupportedException(session, "readRef");
}

@Override
public Blob readBlob() throws SQLException {
throw new SnowflakeLoggedFeatureNotSupportedException(session, "readBlob");
}

@Override
public Clob readClob() throws SQLException {
throw new SnowflakeLoggedFeatureNotSupportedException(session, "readClob");
}

@Override
public Array readArray() throws SQLException {
throw new SnowflakeLoggedFeatureNotSupportedException(session, "readArray");
}

@Override
public boolean wasNull() throws SQLException {
return false; // nulls are not allowed in structure types
}

@Override
public URL readURL() throws SQLException {
throw new SnowflakeLoggedFeatureNotSupportedException(session, "readCharacterStream");
}

@Override
public NClob readNClob() throws SQLException {
throw new SnowflakeLoggedFeatureNotSupportedException(session, "readNClob");
}

@Override
public String readNString() throws SQLException {
throw new SnowflakeLoggedFeatureNotSupportedException(session, "readNString");
}

@Override
public SQLXML readSQLXML() throws SQLException {
throw new SnowflakeLoggedFeatureNotSupportedException(session, "readSQLXML");
}

@Override
public RowId readRowId() throws SQLException {
throw new SnowflakeLoggedFeatureNotSupportedException(session, "readRowId");
}
}
Loading
Loading