Code refactor for Iceberg support #792

sfc-gh-alhuang · 2024-07-11T22:04:01Z

The Iceberg support PR on feature branch includes some refactor. Merge the refactor code to reduce conflict when merging feature branch in the future.

This PR includes following changes

Split StreamingIngestStage into StorageManager and StreamingIngestStorage. The StorageManager is responsible for managing storage. This is useful for streaming to Iceberg which a StorageManager could manage multiple storages.
Add SnowflakeServiceClient to handle all api calls to server. As we are going to add some endpoints for Iceberg table streaming, having an all-in-one place api caller makes development easier.

* Add parameters & disable interleaving mode

… & configure response (#773)

Co-authored-by: Hitesh Madan <[email protected]>

src/main/java/net/snowflake/ingest/streaming/internal/ClientConfigureRequest.java

sfc-gh-hmadan · 2024-07-12T17:30:40Z

src/main/java/net/snowflake/ingest/streaming/internal/StorageManager.java

+ * @param <T> The type of chunk data
+ * @param <TLocation> the type of location that's being managed (internal stage / external volume)
+ */
+interface StorageManager<T, TLocation> {


nit: IStorageManager, since its an interface? (similarly for all other interfaces?)
I don't think this is something we've really followed consistently everywhere else, but check with Toby on whether this is something we want to start.

yeah, IStorageManager is better

sfc-gh-hmadan · 2024-07-12T22:10:56Z

...ain/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java

@@ -228,8 +225,14 @@ public class SnowflakeStreamingIngestClientInternal<T> implements SnowflakeStrea
      this.setupMetricsForClient();
    }



Please create a JIRA to track passing this snowflakeServiceClient all the way into OAuthClient, which also does RPC with the snowflake service.

Jira created.

...ain/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java

sfc-gh-hmadan · 2024-07-12T22:16:29Z

...ain/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java

-      }
+      DropChannelRequestInternal dropChannelRequest =
+          new DropChannelRequestInternal(
+              this.storageManager.getClientPrefix() + "_" + counter.getAndIncrement(),


nit: should getNextRequestId kind of behavior be in the storage manager perhaps? Can be a future PR, just curious what you think @sfc-gh-tzhang ?

Makes sense to me, then counter can be moved to storageManager as well

src/main/java/net/snowflake/ingest/streaming/internal/SnowflakeServiceClient.java

...ain/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java

src/main/java/net/snowflake/ingest/streaming/internal/OpenChannelRequestInternal.java

src/main/java/net/snowflake/ingest/streaming/internal/OpenChannelResponse.java

src/main/java/net/snowflake/ingest/streaming/internal/DropChannelRequestInternal.java

sfc-gh-hmadan · 2024-07-12T23:02:40Z

src/main/java/net/snowflake/ingest/streaming/OpenChannelRequest.java

@@ -161,10 +157,6 @@ public String getOffsetToken() {
    return this.offsetToken;
  }


please undo these changes. you're changing public API surface of a released SDK.

@sfc-gh-tzhang do we have CI tools to catch changes to public API surface? If not lets add a backlog item to catch accidental API surface area changes before a release?

Undo changes.

sfc-gh-hmadan · 2024-07-12T23:04:36Z

src/main/java/net/snowflake/ingest/utils/ErrorCode.java

@@ -26,7 +26,7 @@ public enum ErrorCode {
  INVALID_ENCRYPTED_KEY("0018"),
  INVALID_DATA_IN_CHUNK("0019"),
  IO_ERROR("0020"),


It looks like error code actually gets exposed to customers via SFException.getErrorCode().
Thus this is a breaking change, please don't rename an error code.

@sfc-gh-tzhang pl confirm?

yeah, I tagged you in another thread, please don't modify existing error codes, just add new ones, see #794 (comment)

Reverted change.

src/main/java/net/snowflake/ingest/streaming/internal/StreamingIngestStorage.java

src/main/java/net/snowflake/ingest/streaming/internal/FileLocationInfo.java

sfc-gh-tzhang

Left some comments, otherwise LGTM, thanks!

sfc-gh-tzhang · 2024-07-16T22:18:32Z

...ain/java/net/snowflake/ingest/streaming/internal/SnowflakeStreamingIngestClientInternal.java

-      }
+      DropChannelRequestInternal dropChannelRequest =
+          new DropChannelRequestInternal(
+              this.storageManager.getClientPrefix() + "_" + counter.getAndIncrement(),


Makes sense to me, then counter can be moved to storageManager as well

sfc-gh-tzhang · 2024-07-16T22:22:56Z

src/main/java/net/snowflake/ingest/streaming/internal/StorageManager.java

+ * @param <T> The type of chunk data
+ * @param <TLocation> the type of location that's being managed (internal stage / external volume)
+ */
+interface StorageManager<T, TLocation> {


yeah, IStorageManager is better

sfc-gh-tzhang · 2024-07-16T22:23:10Z

src/main/java/net/snowflake/ingest/streaming/internal/StreamingIngestRequest.java

+ * The StreamingIngestRequest interface is a marker interface used for type safety in the {@link
+ * SnowflakeServiceClient} for streaming ingest API request.
+ */
+interface StreamingIngestRequest {


IStreamingIngestRequest?

sfc-gh-tzhang · 2024-07-16T22:25:12Z

src/main/java/net/snowflake/ingest/utils/ErrorCode.java

@@ -26,7 +26,7 @@ public enum ErrorCode {
  INVALID_ENCRYPTED_KEY("0018"),
  INVALID_DATA_IN_CHUNK("0019"),
  IO_ERROR("0020"),


yeah, I tagged you in another thread, please don't modify existing error codes, just add new ones, see #794 (comment)

Co-authored-by: Hitesh Madan <[email protected]>

sfc-gh-alhuang and others added 6 commits June 10, 2024 17:31

SNOW-1437885 Disable blob interleaving when in Iceberg mode (#763)

c6dfbf3

* Add parameters & disable interleaving mode

SNOW-1483230 Disable blob encryption / Add class for storage metadata…

a9aa682

… & configure response (#773)

SNOW-1497358 Support multiple storage for Iceberg mode (#783)

eac448a

Co-authored-by: Hitesh Madan <[email protected]>

merge

e3f9d65

post merge

6eb0fff

eliminate iceberg logic

c1d7116

sfc-gh-alhuang marked this pull request as ready for review July 11, 2024 22:14

sfc-gh-alhuang requested review from sfc-gh-tzhang and a team as code owners July 11, 2024 22:14

sfc-gh-alhuang requested a review from sfc-gh-hmadan July 11, 2024 22:14