-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add data chunk and task result enums and dtos #2442
Open
inv-jishnu
wants to merge
5
commits into
master
Choose a base branch
from
feat/data-loader/import-utils-dtos
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
bf31a01
Data chunk and task result enums and dtos
inv-jishnu 57cd330
Spotless applied
inv-jishnu 7a39564
Changes
inv-jishnu be4583c
Added transaction batch dtos
inv-jishnu ca71bff
Merge branch 'master' into feat/data-loader/import-utils-dtos
ypeckstadt File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
14 changes: 14 additions & 0 deletions
14
...ore/src/main/java/com/scalar/db/dataloader/core/dataimport/datachunk/ImportDataChunk.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
package com.scalar.db.dataloader.core.dataimport.datachunk; | ||
|
||
import java.util.List; | ||
import lombok.Builder; | ||
import lombok.Data; | ||
|
||
/** * Import data chunk data */ | ||
@Data | ||
@Builder | ||
public class ImportDataChunk { | ||
|
||
int dataChunkId; | ||
List<ImportRow> sourceData; | ||
} |
41 changes: 41 additions & 0 deletions
41
...c/main/java/com/scalar/db/dataloader/core/dataimport/datachunk/ImportDataChunkStatus.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,41 @@ | ||
package com.scalar.db.dataloader.core.dataimport.datachunk; | ||
|
||
import com.fasterxml.jackson.annotation.JsonProperty; | ||
import com.fasterxml.jackson.databind.annotation.JsonDeserialize; | ||
import java.time.Instant; | ||
import lombok.Builder; | ||
import lombok.Data; | ||
|
||
/** * A DTO to store import data chunk details */ | ||
@Data | ||
@Builder | ||
@JsonDeserialize(builder = ImportDataChunkStatus.ImportDataChunkStatusBuilder.class) | ||
public class ImportDataChunkStatus { | ||
|
||
@JsonProperty("dataChunkId") | ||
private final int dataChunkId; | ||
|
||
@JsonProperty("startTime") | ||
private final Instant startTime; | ||
|
||
@JsonProperty("endTime") | ||
private final Instant endTime; | ||
|
||
@JsonProperty("totalRecords") | ||
private final int totalRecords; | ||
|
||
@JsonProperty("successCount") | ||
private final int successCount; | ||
|
||
@JsonProperty("failureCount") | ||
private final int failureCount; | ||
|
||
@JsonProperty("batchCount") | ||
private final int batchCount; | ||
|
||
@JsonProperty("totalDurationInMilliSeconds") | ||
private final int totalDurationInMilliSeconds; | ||
|
||
@JsonProperty("status") | ||
private final ImportDataChunkStatusState status; | ||
} |
8 changes: 8 additions & 0 deletions
8
...n/java/com/scalar/db/dataloader/core/dataimport/datachunk/ImportDataChunkStatusState.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
package com.scalar.db.dataloader.core.dataimport.datachunk; | ||
|
||
/** * Status of the import data chunk which during the import process */ | ||
public enum ImportDataChunkStatusState { | ||
START, | ||
IN_PROGRESS, | ||
COMPLETE | ||
} |
11 changes: 11 additions & 0 deletions
11
...ader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/datachunk/ImportRow.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
package com.scalar.db.dataloader.core.dataimport.datachunk; | ||
|
||
import com.fasterxml.jackson.databind.JsonNode; | ||
import lombok.Value; | ||
|
||
/** Stores data related to a single row on import file */ | ||
@Value | ||
public class ImportRow { | ||
int rowNumber; | ||
JsonNode sourceData; | ||
} |
7 changes: 7 additions & 0 deletions
7
...er/core/src/main/java/com/scalar/db/dataloader/core/dataimport/task/ImportTaskAction.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
package com.scalar.db.dataloader.core.dataimport.task; | ||
|
||
/** * Import task actions */ | ||
public enum ImportTaskAction { | ||
INSERT, | ||
UPDATE, | ||
} |
12 changes: 12 additions & 0 deletions
12
...rc/main/java/com/scalar/db/dataloader/core/dataimport/task/result/ImportResultStatus.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
package com.scalar.db.dataloader.core.dataimport.task.result; | ||
|
||
public enum ImportResultStatus { | ||
SUCCESS, | ||
PARTIAL_SUCCESS, | ||
FAILURE, | ||
VALIDATION_FAILED, | ||
RETRIEVAL_FAILED, | ||
MAPPING_FAILED, | ||
TIMEOUT, | ||
CANCELLED | ||
} |
19 changes: 19 additions & 0 deletions
19
...rc/main/java/com/scalar/db/dataloader/core/dataimport/task/result/ImportTargetResult.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,19 @@ | ||
package com.scalar.db.dataloader.core.dataimport.task.result; | ||
|
||
import com.fasterxml.jackson.databind.JsonNode; | ||
import com.scalar.db.dataloader.core.dataimport.task.ImportTaskAction; | ||
import java.util.List; | ||
import lombok.Builder; | ||
import lombok.Value; | ||
|
||
@Builder | ||
@Value | ||
public class ImportTargetResult { | ||
String namespace; | ||
String tableName; | ||
ImportTaskAction importAction; | ||
List<String> errors; | ||
boolean dataMapped; | ||
JsonNode importedRecord; | ||
ImportTargetResultStatus status; | ||
} |
12 changes: 12 additions & 0 deletions
12
...n/java/com/scalar/db/dataloader/core/dataimport/task/result/ImportTargetResultStatus.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
package com.scalar.db.dataloader.core.dataimport.task.result; | ||
|
||
public enum ImportTargetResultStatus { | ||
VALIDATION_FAILED, | ||
RETRIEVAL_FAILED, | ||
MISSING_COLUMNS, | ||
DATA_ALREADY_EXISTS, | ||
DATA_NOT_FOUND, | ||
SAVE_FAILED, | ||
SAVED, | ||
ABORTED | ||
} |
25 changes: 25 additions & 0 deletions
25
.../src/main/java/com/scalar/db/dataloader/core/dataimport/task/result/ImportTaskResult.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,25 @@ | ||
package com.scalar.db.dataloader.core.dataimport.task.result; | ||
|
||
import com.fasterxml.jackson.annotation.JsonProperty; | ||
import com.fasterxml.jackson.databind.JsonNode; | ||
import com.fasterxml.jackson.databind.annotation.JsonDeserialize; | ||
import java.util.List; | ||
import lombok.Builder; | ||
import lombok.Value; | ||
|
||
@Builder | ||
@Value | ||
@JsonDeserialize(builder = ImportTaskResult.ImportTaskResultBuilder.class) | ||
public class ImportTaskResult { | ||
@JsonProperty("rowNumber") | ||
int rowNumber; | ||
|
||
@JsonProperty("targets") | ||
List<ImportTargetResult> targets; | ||
|
||
@JsonProperty("rawRecord") | ||
JsonNode rawRecord; | ||
|
||
@JsonProperty("dataChunkId") | ||
int dataChunkId; | ||
} |
14 changes: 14 additions & 0 deletions
14
...ava/com/scalar/db/dataloader/core/dataimport/transactionbatch/ImportTransactionBatch.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
package com.scalar.db.dataloader.core.dataimport.transactionbatch; | ||
|
||
import com.scalar.db.dataloader.core.dataimport.datachunk.ImportRow; | ||
import java.util.List; | ||
import lombok.Builder; | ||
import lombok.Value; | ||
|
||
/** Transaction batch details */ | ||
@Builder | ||
@Value | ||
public class ImportTransactionBatch { | ||
int transactionBatchId; | ||
List<ImportRow> sourceData; | ||
} |
32 changes: 32 additions & 0 deletions
32
...m/scalar/db/dataloader/core/dataimport/transactionbatch/ImportTransactionBatchResult.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
package com.scalar.db.dataloader.core.dataimport.transactionbatch; | ||
|
||
import com.fasterxml.jackson.annotation.JsonProperty; | ||
import com.fasterxml.jackson.databind.annotation.JsonDeserialize; | ||
import com.scalar.db.dataloader.core.dataimport.task.result.ImportTaskResult; | ||
import java.util.List; | ||
import lombok.Builder; | ||
import lombok.Value; | ||
|
||
/** Transaction batch result */ | ||
@Builder | ||
@Value | ||
@JsonDeserialize(builder = ImportTransactionBatchResult.ImportTransactionBatchResultBuilder.class) | ||
public class ImportTransactionBatchResult { | ||
@JsonProperty("dataChunkId") | ||
int dataChunkId; | ||
|
||
@JsonProperty("transactionBatchId") | ||
int transactionBatchId; | ||
|
||
@JsonProperty("transactionId") | ||
String transactionId; | ||
|
||
@JsonProperty("records") | ||
List<ImportTaskResult> records; | ||
|
||
@JsonProperty("errors") | ||
List<String> errors; | ||
|
||
@JsonProperty("success") | ||
boolean success; | ||
} |
18 changes: 18 additions & 0 deletions
18
...m/scalar/db/dataloader/core/dataimport/transactionbatch/ImportTransactionBatchStatus.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,18 @@ | ||
package com.scalar.db.dataloader.core.dataimport.transactionbatch; | ||
|
||
import com.scalar.db.dataloader.core.dataimport.task.result.ImportTaskResult; | ||
import java.util.List; | ||
import lombok.Builder; | ||
import lombok.Value; | ||
|
||
/** Batch status details */ | ||
@Builder | ||
@Value | ||
public class ImportTransactionBatchStatus { | ||
int dataChunkId; | ||
int transactionBatchId; | ||
String transactionId; | ||
List<ImportTaskResult> records; | ||
List<String> errors; | ||
boolean success; | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Honestly speaking, it's hard to me to get the relationship of these classes and how to use them from this PR. It would be great if there is a simple diagram or something to show the relationship.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, my previous comment might be unclear. I just read the design document again. The definition of the following terms are still unclear to me, so adding comments about the definitions of xxxxx would be enough:
batch
(I guessImportTransactionBatch
is related)task
(I guessImportTaskAction
andImportTaskResult
are related)