Persistence Component: Append Only Digest Generation and Other Fixes #2451

kumuwu · 2023-11-10T07:39:51Z

What type of PR is this?

Improvement
Bug Fix

What does this PR do / why is it needed ?

Implemented DigestGenStrategy for Append Only, rewrote original digest field to UserProvidedDigestGenStrategy and added a new way to get digest - UDFBasedDigestGenStrategy
Implemented FilteredDataset which allows user to provide any condition to filter their staging data
Updated Snowflake create iceberg table syntax
Updated Snowflake bulk load syntax to support copy options and file path/pattern
Bug fix: Fixed persistence test runner digest not being added for Append Only mode
Bug fix: BigQuery PARSE_DATETIME format updated to include microseconds
Bug fix: Implemented BigQuery's own version of drop table to ignore keywords like CASCADE which is not supported
Bug fix: Fixed data type conversion issue in BigQueryHelper
Bug fix: Fixed show commands quoting issue for database name and schema name
Bug Fix: Fixed Memsql reference table issue

Which issue(s) this PR fixes:

Fixes #

Other notes for reviewers:

Does this PR introduce a user-facing change?

…FROM/VALID_TRHOUGH field if the name matches with source specified VALID_FROM/VALID_TRHOUGH fields

…vided validity fields

…o zhlizh-bulk-load

…-ashutosh/legend-engine into zhlizh-bulk-load

… zhlizh-bulk-load # Conflicts: # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/logicalplan/datasets/StagedFilesDatasetProperties.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/planner/BulkLoadPlanner.java

…st interface

github-actions · 2023-11-10T08:24:22Z

Test Results

    705 files +    2     705 suites +2 1h 2m 44s ⏱️ + 1m 23s
11 733 tests +  97 11 319 ✔️ +  87 414 💤 +10 0 ❌ ±0
14 672 runs +138 14 166 ✔️ +120 506 💤 +18 0 ❌ ±0

Results for commit abdd853. ± Comparison against base commit 84c62a1.

♻️ This comment has been updated with latest results.

… zhlizh-append-only-digest # Conflicts: # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/ingestmode/AppendOnlyAbstract.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/ingestmode/DeriveMainDatasetSchemaFromStaging.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/ingestmode/IngestModeCaseConverter.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/ingestmode/IngestModeVisitors.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-logical-plan/src/main/java/org/finos/legend/engine/persistence/components/planner/AppendOnlyPlanner.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-ansi/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/nontemporal/AppendOnlyTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/AppendOnlyTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/BigQueryTestArtifacts.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/BitemporalDeltaSourceSpecifiesFromAndThroughTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/BitemporalDeltaSourceSpecifiesFromTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/BulkLoadTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/IngestModeTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/NontemporalDeltaTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/NontemporalSnapshotTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/UnitemporalDeltaBatchIdDateTimeBasedTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/UnitemporalDeltaDateTimeBasedTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/UnitemporalSnapshotBatchIdDateTimeBasedTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/UnitemporalSnapshotDateTimeBasedTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-bigquery/src/test/java/org/finos/legend/engine/persistence/components/util/BulkLoadDatasetUtilsBigQueryTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/java/org/finos/legend/engine/persistence/components/BaseTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/java/org/finos/legend/engine/persistence/components/TestUtils.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/nontemporal/AppendOnlyTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/java/org/finos/legend/engine/persistence/components/logicalplan/operations/SchemaEvolutionTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-h2/src/test/java/org/finos/legend/engine/persistence/components/versioning/TestDedupAndVersioning.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-memsql/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/AppendOnlyTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-test/src/test/java/org/finos/legend/engine/persistence/components/scenarios/AppendOnlyScenarios.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-test/src/test/java/org/finos/legend/engine/persistence/components/testcases/ingestmode/nontemporal/AppendOnlyTestCases.java # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-test/src/test/java/org/finos/legend/engine/persistence/components/testcases/ingestmode/nontemporal/derivation/AppendOnlyBasedDerivationTest.java # legend-engine-xts-persistence/legend-engine-xt-persistence-test-runner/src/main/java/org/finos/legend/engine/testable/persistence/mapper/AppendOnlyMapper.java

… zhlizh-append-only-digest

.../main/java/org/finos/legend/engine/persistence/components/ingestmode/IngestModeVisitors.java

...n/java/org/finos/legend/engine/persistence/components/ingestmode/digest/DigestGenerator.java

.../src/main/java/org/finos/legend/engine/persistence/components/planner/AppendOnlyPlanner.java

...gend/engine/persistence/components/relational/snowflake/sqldom/common/IcebergProperties.java

...legend/engine/persistence/components/relational/sqldom/schemaops/statements/ShowCommand.java

…/legend-engine into zhlizh-append-only-digest # Conflicts: # legend-engine-xts-persistence/legend-engine-xt-persistence-component/legend-engine-xt-persistence-component-relational-memsql/src/test/java/org/finos/legend/engine/persistence/components/ingestmode/MemsqlTestArtifacts.java

...an/src/main/java/org/finos/legend/engine/persistence/components/planner/BulkLoadPlanner.java

…inos#2451)

prasar-ashutosh and others added 30 commits August 18, 2023 14:52

Bug Fix: Bitemporal milestoning Derive Main schema removes the VALID_…

130c0c2

…FROM/VALID_TRHOUGH field if the name matches with source specified VALID_FROM/VALID_TRHOUGH fields

Bug Fix: Bitemporal milestoning Schema Evolution must ignore user pro…

b25832b

…vided validity fields

Adding code for concurrent safety feature

f4d6186

Adding test for Multi Ingest Mode with concurrent Safety

f43bd95

Adding tests for concurrent safety

f712980

Code Clean up

8973f91

Bug Fix: Bitemporal temp tables must be deleted after usage

97d93c9

Update readme and code review comments

5b6e979

Fix typo

2d45a3a

Fix typos in readme

30c64f4

Bug Fix: Empty Batch Handling in Unitemp Snapshot

3263040

Bug Fix: Code review comments

50faeae

Implement StagedFilesDatasetReference

7d29ffd

Implement StagedFilesSelection

136793f

Support for Empty Batch Handling in Unitemporal Snapshot

bb9b3ad

Support for FailEmptyBatch strategy in Unitemporal Snapshot

871461b

Merge remote-tracking branch 'Ashutosh/concurrent_safety_support' int…

ddc2bdd

…o zhlizh-bulk-load

Enrich datasets to add additionalDatasetproperties every where

fd35802

Add tests for Empty Data handling

6bd3eef

Support ICEBERG_TABLE_2022 for Iceberg tables

a0bccd4

Implement bulk load for h2

16aece8

Merge branch 'concurrent_safety_support' of https://github.com/prasar…

96fa3b9

…-ashutosh/legend-engine into zhlizh-bulk-load

Add missing datasetAdditionalProperties in BigQueryHelper

738f9c8

Implement executor flow and add end-to-end test

e82af33

Add test

9753521

Add support for digest udf in h2 and add tests

7cc25fd

Clean up

a3c7f74

Add file format and validation for file format and add tests

a1bb51a

Add missing statement

edeef49

kumuwu added 3 commits November 9, 2023 14:32

Add tests for other sinks + fix bugs in big query

be220be

Fix show commands not being properly quoted

93b3fd2

Implement UserProvidedDigestGenStrategy and change Append Only's dige…

758bac6

…st interface

kumuwu requested a review from a team as a code owner November 10, 2023 07:39

kumuwu marked this pull request as draft November 10, 2023 07:40

kumuwu added 7 commits November 10, 2023 16:32

Fix persistence test runner

37b5429

Fix persistence test runner

c482858

Fix show commands not being properly quoted

9eb704b

Fix persistence test runner

1cc0c1a

Implement load options for snowflake bulk load

75830f6

Update snowflake create iceberg table syntax

d38f116

kumuwu marked this pull request as ready for review November 16, 2023 06:26

kumuwu changed the title ~~[Draft] Persistence Component: Append Only Digest Generation~~ Persistence Component: Append Only Digest Generation Nov 16, 2023

kumuwu changed the title ~~Persistence Component: Append Only Digest Generation~~ Persistence Component: Append Only Digest Generation and Other Fixes Nov 16, 2023

Merge branch 'master' of https://github.com/kumuwu/legend-engine into…

9975d56

… zhlizh-append-only-digest

prasar-ashutosh reviewed Nov 21, 2023

View reviewed changes

kumuwu and others added 6 commits November 22, 2023 14:09

Address comments

95a68aa

Memsql Bug Fix for Sharded Table

70c0eb6

Implement FilteredDataset and add SQL tests

892acf6

Add h2 tests

06cf988

Support for filepaths and filepatterns in Snowflake Load

c2ac933

kumuwu commented Nov 28, 2023

View reviewed changes

...an/src/main/java/org/finos/legend/engine/persistence/components/planner/BulkLoadPlanner.java Outdated Show resolved Hide resolved

Address Code Review Comments

abdd853

epsstan approved these changes Dec 4, 2023

View reviewed changes

epsstan merged commit 27c7583 into finos:master Dec 4, 2023
4 checks passed

AFine-gs pushed a commit to AFine-gs/legend-engine that referenced this pull request Jan 9, 2024

Persistence Component: Append Only Digest Generation and Other Fixes (f…

a8fe2ce

…inos#2451)

kumuwu deleted the zhlizh-append-only-digest branch April 8, 2024 05:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Persistence Component: Append Only Digest Generation and Other Fixes #2451

Persistence Component: Append Only Digest Generation and Other Fixes #2451

kumuwu commented Nov 10, 2023 •

edited

Loading

github-actions bot commented Nov 10, 2023 •

edited

Loading

Persistence Component: Append Only Digest Generation and Other Fixes #2451

Persistence Component: Append Only Digest Generation and Other Fixes #2451

Conversation

kumuwu commented Nov 10, 2023 • edited Loading

What type of PR is this?

What does this PR do / why is it needed ?

Which issue(s) this PR fixes:

Other notes for reviewers:

Does this PR introduce a user-facing change?

github-actions bot commented Nov 10, 2023 • edited Loading

Test Results

kumuwu commented Nov 10, 2023 •

edited

Loading

github-actions bot commented Nov 10, 2023 •

edited

Loading