Integration data type test for new table format #850

sfc-gh-alhuang · 2024-10-02T23:37:05Z

Created data type ITs for Iceberg table and verified through local VM. This test should be disabled until the sever side change is release.
Bug fix for FileColumnProperties. The Iceberg mode will fill in default value for corresponding type now if all values in the column are null.
Created AbstractDataTypeTest.testIcebergIngestAndQuery for multi row testing.

src/test/java/net/snowflake/ingest/streaming/internal/datatypes/AbstractDataTypeTest.java

src/test/java/net/snowflake/ingest/streaming/internal/datatypes/IcebergDataTypeTest.java

sfc-gh-hmadan · 2024-10-03T18:24:26Z

src/test/java/net/snowflake/ingest/streaming/internal/datatypes/IcebergDataTypeTest.java

+
+  @Test
+  public void testBoolean() throws Exception {
+    testIcebergIngestion("boolean", true, new BooleanProvider());


nit: whenever you have time in the next few weeks, lets introduce a class called ITTestcaseBuilder that has methods withDataType, withInputValue, withExpectedValue, withVerificationQuery.

THe provider business is funky as it only supports doing exact value verification with a where value = {expectedValue}. I'd much rather write the sql filter directly in the testcase instead of have code generate it, for flexibility.

Added to jira

src/test/java/net/snowflake/ingest/streaming/internal/datatypes/IcebergDataTypeTest.java

sfc-gh-hmadan · 2024-10-03T18:32:11Z

src/test/java/net/snowflake/ingest/streaming/internal/datatypes/AbstractDataTypeTest.java

+    return tableName;
+  }
+
+  protected String createIcebergTable(String dataType) throws SQLException {


also add an argument for boolean nullable and we can make sure the sdk does proper nullability checking for columns marked non-nullable.

Added nullable test.

I see nullable was removed, why so?

Removed and use <datatype> not null in datatype testing. By doing this we can set nullable on any sub-columns on structured data type.

sfc-gh-hmadan · 2024-10-03T18:33:05Z

src/test/java/net/snowflake/ingest/streaming/internal/datatypes/IcebergDataTypeTest.java

+        Assert.assertThrows(
+            SFException.class,
+            () -> testIcebergIngestion("boolean", new Object(), true, new BooleanProvider()));
+    Assert.assertEquals(ErrorCode.INVALID_FORMAT_ROW.getMessageCode(), ex.getVendorCode());


(dupe) lets add nullability tests too - sending in null should work for every column, but when its a non-null column then the sdk should throw.

Added nullable test for non structured data type. Will fix structured data type null count in next PR.

src/test/java/net/snowflake/ingest/streaming/internal/datatypes/IcebergDataTypeTest.java

sfc-gh-hmadan · 2024-10-08T04:36:42Z

src/main/java/net/snowflake/ingest/streaming/internal/RowBufferStats.java

+  /** Whether the column is an Iceberg column */
+  private final boolean isIcebergColumn;
+  /** Primitive type of the column, only used for Iceberg columns */
+  private final PrimitiveType primitiveType;


pl update the PR description to explain there's a bug fix in this PR too.

src/main/java/net/snowflake/ingest/streaming/internal/AbstractRowBuffer.java

src/main/java/net/snowflake/ingest/streaming/internal/RowBufferStats.java

src/test/java/net/snowflake/ingest/streaming/internal/datatypes/IcebergDataTypeIT.java

@sfc-gh-alhuang

1. Remove hidden polymorphic behavior of RowBufferStats, 2. remove isIceberg 3. Refactoring in FileColumnProperties - @sfc-gh-alhuang For strings, i see that there is a truncateUp for the Max stat and truncateDown for min stat. Please reivew this part carefully and compare with what it used to be before we started making changes, want to make sure we aren't regressing FDNs on this front.

sfc-gh-alhuang · 2024-10-08T20:28:13Z

@sfc-gh-hmadan Confirmed that the string min/max values for Iceberg table EP info also apply the 32 bytes round down/up truncate as FDN did.

sfc-gh-alhuang added 5 commits October 2, 2024 22:22

tmp

6188e57

tmp

a25d652

tmp

fbb468e

structured data type verified

06d34e6

code style

c0d69b4

sfc-gh-alhuang requested review from sfc-gh-tzhang and a team as code owners October 2, 2024 23:37

sfc-gh-alhuang requested a review from sfc-gh-hmadan October 3, 2024 17:18