SNOW-1748333 Fix Iceberg decimal type schema parser #864
+77
−6
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
src/main/java/net/snowflake/ingest/streaming/internal/ParquetRowBuffer.java
Show resolved
Hide resolved
src/main/java/net/snowflake/ingest/utils/IcebergDataTypeParser.java
Outdated
Show resolved
Hide resolved
src/test/java/net/snowflake/ingest/streaming/internal/datatypes/IcebergNumericTypesIT.java
Outdated
Show resolved
Hide resolved
| } | ||
| list.add( | ||
| new BigDecimal( | ||
| RandomStringUtils.randomNumeric(intPart) |
There was a problem hiding this comment.
this makes it impossible to reproduce failures locally. Why do we need this randomNumeric when we already generated a random number for intpart and floatpart using the seeded RNG in lines 412 and 413?
There was a problem hiding this comment.
The length of decimal is greater than int/long.max, thus we need string generator for this. Switch to generator with consistent seed.
sfc-gh-alhuang
force-pushed
the
alhuang-iceberg-data-type-parser
branch
from
f0286a1 to
7787aa5
Compare
src/test/java/net/snowflake/ingest/streaming/internal/datatypes/IcebergNumericTypesIT.java
Outdated
Show resolved
Hide resolved
| bigDecimals_38_10); | ||
| } | ||
|
|
||
| private static List<Object> randomBigDecimal(int count, int precision, int scale) { |
There was a problem hiding this comment.
not sure i fully understand what this is trying to do (comments will help?). Lets get this merged in to unblock KC folks and I'll follow up with you offline on this method.
There was a problem hiding this comment.
Added comments
sfc-gh-hmadan
approved these changes
Oct 21, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR addresses the issue where the scanner encounters an error when scanning delta-encoded decimal values with a precision greater than 18.
Changes
sfVermetadata in Iceberg mode: This prevents the scanner from applying optimizations specific to Snowflake-written Parquet files, which was the root cause of the issue with scanning.