Persistence Component: Bulk Load Support for Big Query #2294

kumuwu · 2023-09-22T07:26:59Z

What type of PR is this?

Improvement

What does this PR do / why is it needed ?

Added support for bulk load for Big Query
Added unit tests and end-to-end tests with Big Query
Modified H2 digest MD5 algorithm to provide consistency
Modified capabilities interface
Implement bulk load batch id as ever increasing integer and changed the original batch id to event id
Fixed bug in Snowflake bulk load stats: return stats also when the status is failed

Which issue(s) this PR fixes:

Fixes #

Other notes for reviewers:

Does this PR introduce a user-facing change?

…FROM/VALID_TRHOUGH field if the name matches with source specified VALID_FROM/VALID_TRHOUGH fields

…vided validity fields

…o zhlizh-bulk-load

…-ashutosh/legend-engine into zhlizh-bulk-load

… zhlizh-bulk-load # Conflicts: # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/logicalplan/datasets/StagedFilesDatasetProperties.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/planner/BulkLoadPlanner.java

… zhlizh-big-query-bulk-load # Conflicts: # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/common/FileFormat.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/planner/BulkLoadPlanner.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-core/src/main/java/org/finos/legend/engine/persistence/components/relational/sqldom/common/Clause.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/main/java/org/finos/legend/engine/persistence/components/relational/h2/logicalplan/datasets/H2StagedFilesDatasetPropertiesAbstract.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/bulkload/BulkLoadTest.java

# Conflicts: # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/planner/BulkLoadPlanner.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/bulkload/BulkLoadTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/resources/data/bulk-load/expected/expected_table1.csv # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/resources/data/bulk-load/expected/expected_table2.csv # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/resources/data/bulk-load/expected/expected_table3.csv # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/resources/data/bulk-load/expected/expected_table4.csv

…legend-engine into zhlizh-big-query-bulk-load # Conflicts: # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/planner/BulkLoadPlanner.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/bulkload/BulkLoadTest.java

… zhlizh-big-query-bulk-load # Conflicts: # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/planner/BulkLoadPlanner.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/bulkload/BulkLoadTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/resources/data/bulk-load/expected/expected_table3.csv # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/resources/data/bulk-load/expected/expected_table4.csv

github-actions · 2023-09-22T09:38:38Z

Test Results

    693 files +    693     693 suites +693 58m 39s ⏱️ + 58m 39s
11 446 tests +11 446 11 062 ✔️ +11 062 384 💤 +384 0 ❌ ±0
14 338 runs +14 338 13 870 ✔️ +13 870 468 💤 +468 0 ❌ ±0

Results for commit f611912. ± Comparison against base commit a8d72b3.

♻️ This comment has been updated with latest results.

...an/src/main/java/org/finos/legend/engine/persistence/components/planner/BulkLoadPlanner.java

...gical-plan/src/main/java/org/finos/legend/engine/persistence/components/util/Capability.java

...ava/org/finos/legend/engine/persistence/components/logicalplan/values/DigestUdfAbstract.java

...in/java/org/finos/legend/engine/persistence/components/relational/bigquery/BigQuerySink.java

...rsistence/components/relational/bigquery/sql/visitor/StagedFilesDatasetReferenceVisitor.java

...an/src/main/java/org/finos/legend/engine/persistence/components/planner/BulkLoadPlanner.java

...c/main/java/org/finos/legend/engine/persistence/components/common/CsvFileFormatAbstract.java

...rsistence/components/relational/bigquery/sql/visitor/StagedFilesDatasetReferenceVisitor.java

… zhlizh-big-query-bulk-load

* Add PK validation in bulk load * Resolve conflict * Remove unnecessary delete * Introduce bulk load batch id and bulk load task id * Rename variable

… zhlizh-big-query-bulk-load

...cal-plan/src/main/java/org/finos/legend/engine/persistence/components/common/FileFormat.java

...inos/legend/engine/persistence/components/relational/bigquery/executor/BigQueryExecutor.java

prasar-ashutosh and others added 30 commits August 18, 2023 14:52

Bug Fix: Bitemporal milestoning Derive Main schema removes the VALID_…

130c0c2

…FROM/VALID_TRHOUGH field if the name matches with source specified VALID_FROM/VALID_TRHOUGH fields

Bug Fix: Bitemporal milestoning Schema Evolution must ignore user pro…

b25832b

…vided validity fields

Adding code for concurrent safety feature

f4d6186

Adding test for Multi Ingest Mode with concurrent Safety

f43bd95

Adding tests for concurrent safety

f712980

Code Clean up

8973f91

Bug Fix: Bitemporal temp tables must be deleted after usage

97d93c9

Update readme and code review comments

5b6e979

Fix typo

2d45a3a

Fix typos in readme

30c64f4

Bug Fix: Empty Batch Handling in Unitemp Snapshot

3263040

Bug Fix: Code review comments

50faeae

Implement StagedFilesDatasetReference

7d29ffd

Implement StagedFilesSelection

136793f

Support for Empty Batch Handling in Unitemporal Snapshot

bb9b3ad

Support for FailEmptyBatch strategy in Unitemporal Snapshot

871461b

Merge remote-tracking branch 'Ashutosh/concurrent_safety_support' int…

ddc2bdd

…o zhlizh-bulk-load

Enrich datasets to add additionalDatasetproperties every where

fd35802

Add tests for Empty Data handling

6bd3eef

Support ICEBERG_TABLE_2022 for Iceberg tables

a0bccd4

Implement bulk load for h2

16aece8

Merge branch 'concurrent_safety_support' of https://github.com/prasar…

96fa3b9

…-ashutosh/legend-engine into zhlizh-bulk-load

Add missing datasetAdditionalProperties in BigQueryHelper

738f9c8

Implement executor flow and add end-to-end test

e82af33

Add test

9753521

Add support for digest udf in h2 and add tests

7cc25fd

Clean up

a3c7f74

Add file format and validation for file format and add tests

a1bb51a

Add missing statement

edeef49

kumuwu and others added 10 commits September 19, 2023 11:25

Resolve conflicts

3e69548

Add digest udf and more tests

97a7eb0

Fix digest problem

4e28794

Change H2 digest algo

d804bd2

Fix tests

106de03

Fix typo

2e630df

kumuwu requested a review from a team as a code owner September 22, 2023 07:27

kumuwu changed the title ~~Persistence Component: Bulk Load Support forBig Query~~ Persistence Component: Bulk Load Support for Big Query Sep 22, 2023

prasar-ashutosh reviewed Sep 26, 2023

View reviewed changes

kumuwu and others added 9 commits September 26, 2023 16:33

Refactor file format and load options

31b2199

Refactor dataset, selection, reference logic

b948e02

Fix other comments

6611f90

Fix big query bulk load ingestor flow and add end-to-end tests

33f9e88

Merge branch 'master' of https://github.com/kumuwu/legend-engine into…

17e9349

… zhlizh-big-query-bulk-load

Add rows with error handling and test

e1a5f5d

Address comments

577082c

Bulk Load Batch ID and Task ID & PK Validation (#11)

8d7316a

* Add PK validation in bulk load * Resolve conflict * Remove unnecessary delete * Introduce bulk load batch id and bulk load task id * Rename variable

Merge branch 'master' of https://github.com/kumuwu/legend-engine into…

9ecee88

… zhlizh-big-query-bulk-load

akphi previously approved these changes Oct 16, 2023

View reviewed changes

rafaelbey reviewed Oct 19, 2023

View reviewed changes

...cal-plan/src/main/java/org/finos/legend/engine/persistence/components/common/FileFormat.java Outdated Show resolved Hide resolved

...inos/legend/engine/persistence/components/relational/bigquery/executor/BigQueryExecutor.java Outdated Show resolved Hide resolved

Address Code Review Comments

f611912

prasar-ashutosh dismissed akphi’s stale review via f611912 October 19, 2023 03:40

rafaelbey approved these changes Oct 19, 2023

View reviewed changes

rafaelbey merged commit bc6cc1a into finos:master Oct 19, 2023
3 checks passed

kumuwu deleted the zhlizh-big-query-bulk-load branch January 19, 2024 03:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Persistence Component: Bulk Load Support for Big Query #2294

Persistence Component: Bulk Load Support for Big Query #2294

kumuwu commented Sep 22, 2023 •

edited

Loading

github-actions bot commented Sep 22, 2023 •

edited

Loading

Persistence Component: Bulk Load Support for Big Query #2294

Persistence Component: Bulk Load Support for Big Query #2294

Conversation

kumuwu commented Sep 22, 2023 • edited Loading

What type of PR is this?

What does this PR do / why is it needed ?

Which issue(s) this PR fixes:

Other notes for reviewers:

Does this PR introduce a user-facing change?

github-actions bot commented Sep 22, 2023 • edited Loading

Test Results

kumuwu commented Sep 22, 2023 •

edited

Loading

github-actions bot commented Sep 22, 2023 •

edited

Loading