Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Persistence Library: Dry run mode support for Bulk Load and Capture Data Errors #2642

Merged
merged 39 commits into from
Mar 13, 2024
Merged

Persistence Library: Dry run mode support for Bulk Load and Capture Data Errors #2642

merged 39 commits into from
Mar 13, 2024

Conversation

prasar-ashutosh
Copy link
Contributor

@prasar-ashutosh prasar-ashutosh commented Feb 20, 2024 •
edited
Loading

What type of PR is this?

  • Improvement

What does this PR do / why is it needed ?

  1. Dry run mode support: Ability to run ingest in dry run which runs validation on the data without actually loading the data to target table
    1.1 Supported Formats: CSV, AVRO, JSON
    1.2 Supported Sink: H2, Snowflake
    1.2 Supported Error Categories: Data type Conversion , Null Check , File not found, parsing error, others
  2. Capture Fail on Duplicates Data errors
  3. Capture Data Version errors
  4. Bug Fix: Append a hash to the temporary table names to provide concurrent safety

Which issue(s) this PR fixes:

Fixes #

Other notes for reviewers:

Does this PR introduce a user-facing change?

NO

@prasar-ashutosh prasar-ashutosh requested a review from a team as a code owner February 20, 2024 07:13
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 20, 2024 •
edited
Loading

Test Results

     748 files  +  2       748 suites  +2   1h 4m 41s ⏱️ -34s
12 285 tests +28  12 120 ✔️ +28  165 💤 ±0  0 ❌ ±0 
15 318 runs  +28  15 143 ✔️ +28  175 💤 ±0  0 ❌ ±0 

Results for commit bea2326. ± Comparison against base commit 3037b89.

♻️ This comment has been updated with latest results.

kumuwu
kumuwu reviewed Feb 23, 2024
View reviewed changes
.../legend/engine/persistence/components/ingestmode/versioning/DeriveDataErrorsLogicalPlan.java Outdated Show resolved Hide resolved
...an/src/main/java/org/finos/legend/engine/persistence/components/planner/BulkLoadPlanner.java Outdated Show resolved Hide resolved
...gical-plan/src/main/java/org/finos/legend/engine/persistence/components/planner/Planner.java Outdated Show resolved Hide resolved
...inos/legend/engine/persistence/components/relational/bigquery/executor/BigQueryExecutor.java Outdated Show resolved Hide resolved
...rg/finos/legend/engine/persistence/components/relational/api/RelationalIngestorAbstract.java Show resolved Hide resolved
...relational/snowflake/logicalplan/datasets/SnowflakeStagedFilesDatasetPropertiesAbstract.java Outdated Show resolved Hide resolved
...e/persistence/components/relational/snowflake/sqldom/schemaops/statements/CopyStatement.java Outdated Show resolved Hide resolved
kumuwu and others added 7 commits February 27, 2024 16:48
@kumuwu
Implement bulk load dry run generic approach
1cc2276
@prasar-ashutosh
Changes for Adding hash to temp tables
b2e43a5
@kumuwu
Change logical try_cast to cast in non safe-cast flow + refactor
313e0df
@kumuwu
Merge branch 'dry_run_mode_sf' of https://github.com/prasar-ashutosh/…
f4ef2f0 
…legend-engine into zhlizh-bulk-load-dry-run
@kumuwu
Add cleanup operations for dry run
ca4aef5
@prasar-ashutosh
Merge remote-tracking branch 'origin/master' into dry_run_mode_sf
a3e1e58 
# Conflicts:
#	legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-physical-plan/src/main/java/org/finos/legend/engine/persistence/components/executor/Executor.java
#	legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/main/java/org/finos/legend/engine/persistence/components/relational/bigquery/executor/BigQueryExecutor.java
#	legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-core/src/main/java/org/finos/legend/engine/persistence/components/relational/RelationalSink.java
#	legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-core/src/main/java/org/finos/legend/engine/persistence/components/relational/api/RelationalIngestorAbstract.java
#	legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-core/src/main/java/org/finos/legend/engine/persistence/components/relational/executor/RelationalExecutor.java
@prasar-ashutosh
Rebase with master
d8bff93
@prasar-ashutosh prasar-ashutosh changed the title Persistence Library: Dry run mode support for Snowflake While loading CSV files Persistence Library: Dry run mode support Feb 28, 2024
@prasar-ashutosh prasar-ashutosh changed the title Persistence Library: Dry run mode support Persistence Library: Dry run mode support for CSV loading in Snowflake Feb 28, 2024
kumuwu and others added 16 commits March 4, 2024 15:59
@kumuwu
Fix snowflake datatype check - to check column by column and other fixes
3a5f626
@kumuwu
Address comments
c4252eb
@kumuwu
Fix checkstyle
d5e7af2
@kumuwu
Fix snowflake datatype check
ea16748
@prasar-ashutosh
Merge pull request #16 from kumuwu/zhlizh-bulk-load-dry-run
8f82dfe 
Implement bulk load dry run generic approach
@prasar-ashutosh
Clean up Batch Error
e7a0c8f
@kumuwu
Add error category
d07e0da
@kumuwu
Merge branch 'dry_run_mode_sf' of https://github.com/prasar-ashutosh/…
ce51c12 
…legend-engine into zhlizh-bulk-load-dry-run
@prasar-ashutosh
Clean up interface for Data Error
1d2497e
@kumuwu
Fix test
39664d9
@kumuwu
Add exception handling and adapt to new DataError model
e880452
@prasar-ashutosh
Handle Data Errors in Duplicates and Data version error cases
3852239
@kumuwu
Fix tests
342ccfc
@prasar-ashutosh
Provided an interface for init Datasets to pass the ingestRunId
78ed3a9
@prasar-ashutosh
Clean up the ingest run id generation
40c6df6
@prasar-ashutosh
Fix regex in H2 Sink to catch the exceptions
bea2326
@prasar-ashutosh prasar-ashutosh changed the title Persistence Library: Dry run mode support for CSV loading in Snowflake Persistence Library: Dry run mode support for Bulk Load Mar 13, 2024
@prasar-ashutosh prasar-ashutosh changed the title Persistence Library: Dry run mode support for Bulk Load Persistence Library: Dry run mode support for Bulk Load and Capture Merge Errors Mar 13, 2024
@prasar-ashutosh prasar-ashutosh changed the title Persistence Library: Dry run mode support for Bulk Load and Capture Merge Errors Persistence Library: Dry run mode support for Bulk Load and Capture Data Errors Mar 13, 2024
@epsstan epsstan merged commit 20de9ff into finos:master Mar 13, 2024
3 checks passed
lisaknbur pushed a commit to goldmansachs/legend-engine that referenced this pull request Mar 15, 2024
@prasar-ashutosh @kumuwu @lisaknbur
Persistence Library: Dry run mode support for Bulk Load and Capture D…
9bf028c 
…ata Errors (finos#2642)

Co-authored-by: kumuwu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kumuwu kumuwu kumuwu left review comments

@epsstan epsstan epsstan approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@prasar-ashutosh @epsstan @kumuwu