-
Notifications
You must be signed in to change notification settings - Fork 57
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SNOW-1821246 Add parameter for blob upload timeout #916
Conversation
src/main/java/net/snowflake/ingest/utils/ParameterProvider.java
Outdated
Show resolved
Hide resolved
src/main/java/net/snowflake/ingest/utils/ParameterProvider.java
Outdated
Show resolved
Hide resolved
RegisterService<StubChunkData> rs = new RegisterService<>(null, true); | ||
SnowflakeStreamingIngestClientInternal<StubChunkData> client = | ||
Mockito.mock(SnowflakeStreamingIngestClientInternal.class); | ||
ParameterProvider parameterProvider = Mockito.mock(ParameterProvider.class); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
curious why do we need to mock the parameter provider, can we not do away with this unnecessary mocking (not for this PR) ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We need this as RegisterServiceTest
is reading blob upload timeout from parameter provider.
Want to make sure we understand the upload logic correctly, the total wait time is NOT 5 seconds, the total time is BLOB_UPLOAD_TIMEOUT_IN_SEC * BLOB_UPLOAD_MAX_RETRY_COUNT which is 120s, and currently the BLOB_UPLOAD_MAX_RETRY_COUNT is already configurable. Ideally we do not want to exposing more and more configurable parameters even though they're not documented. And please make sure you have read the comment in the upload function
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please see my other comment, thanks
Closed this PR as the customer can configure max retry count. |
The original blob upload timeout is hard coded as 5 seconds. This might not be feasible for Iceberg table streaming as the 30 seconds client lag in Iceberg streaming might create larger file which requires larger blob upload timeout. Make the
BLOB_UPLOAD_TIMEOUT
constant to configurable parameter and keep the default timeout of FDN table streaming as 5 seconds while have a 20 seconds default for Iceberg table streaming.