-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Retry Wrapper | WIP #19
base: develop
Are you sure you want to change the base?
Conversation
@@ -65,8 +66,8 @@ public final class SalesforceBulkUtil { | |||
* @throws AsyncApiException if there is an issue creating the job | |||
*/ | |||
|
|||
public static JobInfo createJob(BulkConnection bulkConnection, String sObject, OperationEnum operationEnum, | |||
@Nullable String externalIdField, | |||
public static JobInfo createJob(BulkConnectionRetryWrapper bulkConnection, String sObject, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rename the field to bulkConnectionRetryWrapper
@@ -164,7 +165,10 @@ public static List<SalesforceSplit> getSplits( | |||
bulkConnection.addHeader(SalesforceSourceConstants.HEADER_ENABLE_PK_CHUNK, | |||
String.join(";", chunkHeaderValues)); | |||
} | |||
List<SalesforceSplit> querySplits = SalesforceSplitUtil.getQuerySplits(query, bulkConnection, | |||
BulkConnectionRetryWrapper bulkConnectionRetryWrapper = new BulkConnectionRetryWrapper(bulkConnection, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than passing the bulkConnection here, write a method in SalesforceSplitUtil that returns RetryWrapper
} | ||
|
||
public JobInfo createJob(JobInfo jobInfo) throws AsyncApiException { | ||
if (!retryOnBackendError) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rather than adding if condition for each method, can we have a retry policy that dont try after failure. E.g maxAttempts = 1
try { | ||
return createBatchFromStream(query, job); | ||
} catch (AsyncApiException e) { | ||
throw new SalesforceQueryExecutionException(e.getMessage()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SalesforceQueryExecutionException will only be thrown if error code is among the four which we were using previously, write a method that will handle AsyncAPIException and throw error only if code is among those and call that from all catches
As per new guidelines please get design doc approved before reviewing the PR. 😄 ✌️ |
Generating diff for branch: [feat/bulk-api-retry] against [develop]
Default Title
Jira : PLUGIN-0000
Description
Default Description
UI Field
Salesforce-batchsink.json
Docs
Code change
BulkConnectionRetryWrapper.java
SalesforceBulkUtil.java
SalesforceOutputFormatProvider.java
SalesforceSinkConfig.java
SalesforceBatchMultiSource.java
SalesforceBatchSource.java
SalesforceBulkRecordReader.java
SalesforceSplitUtil.java
Unit Tests