retry fuctionality for all bulk apis #18
Conversation
pom.xml
Outdated
| @@ -66,7 +66,7 @@ | |||
| <spark2.version>2.1.3</spark2.version> | |||
| <hydrator.version>2.10.0</hydrator.version> | |||
| <commons.version>3.8.1</commons.version> | |||
| <salesforce.api.version>53.0.0</salesforce.api.version> | |||
| <salesforce.api.version>61.0.0</salesforce.api.version> | |||
There was a problem hiding this comment.
raise a separate PR just for this version upgrade.
| @@ -68,6 +69,7 @@ public final class SalesforceBulkUtil { | |||
| public static JobInfo createJob(BulkConnection bulkConnection, String sObject, OperationEnum operationEnum, | |||
| @Nullable String externalIdField, | |||
| ConcurrencyMode concurrencyMode, ContentType contentType) throws AsyncApiException { | |||
| BulkConnectionRetryWrapper bulkConnectionRetryWrapper = new BulkConnectionRetryWrapper(bulkConnection); | |||
There was a problem hiding this comment.
Instead of creating object of RetryWrapper, create it once in getBulkConnection method and rename it and return wrapper instead of bulkConnection. Retry criteria fields also fetch from config instead of hard coding as given now
There was a problem hiding this comment.
Retry criteria fields also fetch from config instead of hard coding as given now
Which config ?
There are no fields for retry
| return (InputStream) inputStream; | ||
| } | ||
|
|
||
| public static RetryPolicy<Object> getRetryPolicy(Long initialRetryDuration, Long maxRetryDuration, |
There was a problem hiding this comment.
Use the one from SplitUtil class and call it once while creating retry wrapper. If retryDisabled, handle it in this class constructor.
| import io.cdap.plugin.salesforce.plugin.source.batch.util.SalesforceQueryExecutionException; | ||
| import io.cdap.plugin.salesforce.plugin.source.batch.util.SalesforceSourceConstants; | ||
| import io.cdap.plugin.salesforce.plugin.source.batch.util.SalesforceSplitUtil; | ||
|
|
||
| import org.apache.commons.csv.CSVFormat; | ||
| import org.apache.commons.csv.CSVParser; | ||
| import org.apache.commons.csv.CSVRecord; |
There was a problem hiding this comment.
Some of the methods are still not called using retry wrapper. E.g. line 208. move those also to retry wrapper class
| .get(() -> bulkConnection.updateJob(jobInfo)); | ||
| } | ||
|
|
||
| public BatchInfoList getBatchInfoList(String jobId) throws AsyncApiException { |
There was a problem hiding this comment.
Add try catch to each method and in catch add a method that handles asyncapiException and handle retry as retry will be happen based on SalesforceQueryExecutionException.
|
Ref : #19 |
@psainics We need to move these classes on top of develop branch instead of 1.6. I am adding few suggestions for you to do.