SNOW-914666 Adds MAX_CLIENT_LAG configuration option #586
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We want to expose a knob that gives users the ability to control when data is ingested. This has a material difference on the size of blobs generated and can result in fewer smaller-sized blobs which in turn affects query performance. The trade-off is higher ingest latencies. We have decided to expose this in the form of an optional `MAX_CLIENT_LAG` option that accepts inputs as the following: - `number second` (ex: `1 second`) - `number seconds` (ex: `2 seconds`) - `number minute` (ex: `1 minute`) - `number minutes` (ex: `2 minutes`) By default we use 1 second as the maximum client lag which is the current behavior of the SDK. Note that this dictates when a flush is triggered to cloud storage. Depending on your connection to cloud storage and cloud storage tail latencies a blob persist may take longer than expected. Therefore, it is helpful to think of this parameter as a target, rather than an absolute number. @test Adds tests to `ParameterProviderTest`
sfc-gh-tjones
requested review from
sfc-gh-tzhang and
a team
as code owners
sfc-gh-asen
reviewed
Sep 13, 2023
src/main/java/net/snowflake/ingest/utils/ParameterProvider.java
Outdated
Show resolved
Hide resolved
src/main/java/net/snowflake/ingest/utils/ParameterProvider.java
Outdated
Show resolved
Hide resolved
src/main/java/net/snowflake/ingest/utils/ParameterProvider.java
Outdated
Show resolved
Hide resolved
src/test/java/net/snowflake/ingest/streaming/internal/ParameterProviderTest.java
Show resolved
Hide resolved
|
What's the max lag that can be set ? |
We probably should have a minimum and maximum. 10 minutes seems reasonable as an upper limit, what about for a lower limit? |
@sfc-gh-xhuang added minimum of 1 second, max of 10 minutes. PTAL |
sfc-gh-asen
approved these changes
Sep 13, 2023
src/main/java/net/snowflake/ingest/utils/ParameterProvider.java
Outdated
Show resolved
Hide resolved
src/main/java/net/snowflake/ingest/utils/ParameterProvider.java
Outdated
Show resolved
Hide resolved
src/main/java/net/snowflake/ingest/utils/ParameterProvider.java
Outdated
Show resolved
Hide resolved
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We want to expose a knob that gives users the ability to control when data is ingested. This has a material difference on the size of blobs generated and can result in fewer smaller-sized blobs which in turn affects query performance. The trade-off is higher ingest latencies. We have decided to expose this in the form of an optional
MAX_CLIENT_LAGoption that accepts inputs as the following:number second(ex:1 second)number seconds(ex:2 seconds)number minute(ex:1 minute)number minutes(ex:2 minutes)By default we use 1 second as the maximum client lag which is the current behavior of the SDK.
Note that this dictates when a flush is triggered to cloud storage. Depending on your connection to cloud storage and cloud storage tail latencies a blob persist may take longer than expected. Therefore, it is helpful to think of this parameter as a target, rather than an absolute number.
@test Adds tests to
ParameterProviderTest