-
Notifications
You must be signed in to change notification settings - Fork 57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SNOW-1757554 Support quoted object fields name #869
Conversation
src/test/java/net/snowflake/ingest/streaming/internal/datatypes/IcebergStructuredIT.java
Show resolved
Hide resolved
6dc18a1
to
ba48f45
Compare
src/main/java/net/snowflake/ingest/streaming/internal/ParquetRowBuffer.java
Outdated
Show resolved
Hide resolved
src/main/java/net/snowflake/ingest/utils/IcebergDataTypeParser.java
Outdated
Show resolved
Hide resolved
src/main/java/net/snowflake/ingest/utils/IcebergDataTypeParser.java
Outdated
Show resolved
Hide resolved
src/main/java/net/snowflake/ingest/utils/IcebergDataTypeParser.java
Outdated
Show resolved
Hide resolved
src/main/java/net/snowflake/ingest/utils/IcebergDataTypeParser.java
Outdated
Show resolved
Hide resolved
src/main/java/net/snowflake/ingest/utils/IcebergDataTypeParser.java
Outdated
Show resolved
Hide resolved
} | ||
if (parquetType.getId() != null) { | ||
builder.id(parquetType.getId().intValue()); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
else ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The middle layer of Iceberg list/map does not have id. We don't want to set it.
src/main/java/net/snowflake/ingest/streaming/internal/IcebergParquetValueParser.java
Outdated
Show resolved
Hide resolved
src/test/java/net/snowflake/ingest/streaming/internal/datatypes/IcebergStructuredIT.java
Show resolved
Hide resolved
de23571
to
d939c5a
Compare
d939c5a
to
af46e60
Compare
for (String subColumn : subColumnFinder.getSubColumns(columnName)) { | ||
RowBufferStats stats = statsMap.get(subColumn); | ||
for (String subColumnId : | ||
subColumnFinder.getSubColumns(fieldIndex.get(columnName).type.getId())) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fieldIndex.get(columnName) can return null, lets handle that and throw explicitly? OK in your next PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The columnName
is from Set.difference(fieldIndex.keySet(), otherSet)
. I think this should be safe.
This PR aims to support quoted object fields name for Iceberg table ingestion. The PR include following change:
("a.a" int, a object(a int))
.fieldId
as key instead to avoid string construction. Keep a map offieldId -> dotPath
insubcolumnFinder
for logging purpose.