Migrates Pipelines crawled during the assessment phase #2778

pritishpai · 2024-10-01T20:49:18Z

Changes

Added PipelineMigrator to help with migration of DLT pipelines

Linked issues

Adds #3098

Functionality

added relevant user documentation
added a new workflow

Tests

manually tested
added unit tests
added integration tests
verified on staging environment (screenshot attached)

github-actions · 2024-10-29T18:08:51Z

❌ 53/55 passed, 2 failed, 4 skipped, 7h23m52s total

❌ test_all_grants_for_other_objects: AssertionError: assert {'MODIFY', 'SELECT'} == set() (13m21.016s)

AssertionError: assert {'MODIFY', 'SELECT'} == set()
  
  Extra items in the left set:
  'SELECT'
  'MODIFY'
  
  Full diff:
  - set()
  + {
  +     'MODIFY',
  +     'SELECT',
  + }
[gw4] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.grants] fetching grants inventory
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/grants.py", line 236, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 344, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 268, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 478, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sc85a`.`grants` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.grants] crawling new set of snapshot data for grants
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.tables] fetching tables inventory
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/tables.py", line 411, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 344, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 268, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 478, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sc85a`.`tables` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.tables] crawling new set of snapshot data for tables
20:05 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_s9uw6] listing tables and views
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.tables] found 0 new records for tables
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.udfs] fetching udfs inventory
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/udfs.py", line 63, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 344, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 268, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 478, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sc85a`.`udfs` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.udfs] crawling new set of snapshot data for udfs
20:05 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_s9uw6] listing udfs
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.udfs] found 0 new records for udfs
20:15 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-dbb555ae0c9e4579ae9cc9970222d73e-391a3f11ec63a478-00]
20:16 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANY FILE : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-15d3ef22ee2693679f19ad92b65c2ac0-3bc28fdf1a58bbcc-00]
20:16 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.grants] found 3 new records for grants
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.grants] fetching grants inventory
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/grants.py", line 236, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 344, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 268, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 478, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sc85a`.`grants` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.grants] crawling new set of snapshot data for grants
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.tables] fetching tables inventory
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/tables.py", line 411, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 344, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 268, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 478, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sc85a`.`tables` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.tables] crawling new set of snapshot data for tables
20:05 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_s9uw6] listing tables and views
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.tables] found 0 new records for tables
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.udfs] fetching udfs inventory
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] Inventory table not found
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/framework/crawlers.py", line 152, in _snapshot
    cached_results = list(fetcher())
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/hive_metastore/udfs.py", line 63, in _try_fetch
    for row in self._fetch(f"SELECT * FROM {escape_sql_identifier(self.full_name)}"):
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 344, in fetch_all
    execute_response = self.execute(
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 268, in execute
    self._raise_if_needed(status)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/labs/lsql/core.py", line 478, in _raise_if_needed
    raise NotFound(error_message)
databricks.sdk.errors.platform.NotFound: [TABLE_OR_VIEW_NOT_FOUND] The table or view `hive_metastore`.`dummy_sc85a`.`udfs` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog.
To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 1 pos 14
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.udfs] crawling new set of snapshot data for udfs
20:05 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_s9uw6] listing udfs
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.udfs] found 0 new records for udfs
20:15 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-dbb555ae0c9e4579ae9cc9970222d73e-391a3f11ec63a478-00]
20:16 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANY FILE : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-15d3ef22ee2693679f19ad92b65c2ac0-3bc28fdf1a58bbcc-00]
20:16 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sc85a.grants] found 3 new records for grants
[gw4] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

❌ test_all_grant_types: AssertionError: assert {('CATALOG', ...dummy_tmr5h')} == {('ANONYMOUS ..._firae'), ...} (13m34.716s)

AssertionError: assert {('CATALOG', ...dummy_tmr5h')} == {('ANONYMOUS ..._firae'), ...}
  
  Extra items in the right set:
  ('ANONYMOUS FUNCTION', None)
  ('ANY FILE', None)
  
  Full diff:
    {
  -     (
  -         'ANONYMOUS FUNCTION',
  -         None,
  -     ),
  -     (
  -         'ANY FILE',
  -         None,
  -     ),
        (
            'CATALOG',
            'hive_metastore',
        ),
        (
            'DATABASE',
            'hive_metastore.dummy_sn1co',
        ),
        (
            'TABLE',
            'hive_metastore.dummy_sn1co.dummy_txfjs',
        ),
        (
            'UDF',
            'hive_metastore.dummy_sn1co.dummy_firae',
        ),
        (
            'VIEW',
            'hive_metastore.dummy_sn1co.dummy_tmr5h',
        ),
    }
20:05 INFO [databricks.labs.ucx.install] Creating ucx schemas...
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
20:05 INFO [databricks.labs.ucx.install] Creating ucx schemas...
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.grants] fetching grants inventory
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.grants] crawling new set of snapshot data for grants
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.tables] fetching tables inventory
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.tables] crawling new set of snapshot data for tables
20:05 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_sn1co] listing tables and views
20:05 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_sn1co.dummy_tmr5h] fetching table metadata
20:05 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_sn1co.dummy_txfjs] fetching table metadata
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.tables] found 2 new records for tables
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.udfs] fetching udfs inventory
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.udfs] crawling new set of snapshot data for udfs
20:05 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_sn1co] listing udfs
20:05 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_sn1co.dummy_firae] fetching udf metadata
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.udfs] found 1 new records for udfs
20:08 WARNING [databricks.labs.ucx.hive_metastore.grants] Schema hive_metastore.dummy_sonpp no longer existed
20:16 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANY FILE : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-3b28bb6ed63ff707c24eea541dbe4ed6-e0c157e515c47483-00]
20:17 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-46e1a53c6d66275eceeb6cd82032fdca-649c3b36ab6167e3-00]
20:17 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.grants] found 100 new records for grants
20:05 INFO [databricks.labs.ucx.install] Creating ucx schemas...
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.grants] fetching grants inventory
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.grants] crawling new set of snapshot data for grants
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.tables] fetching tables inventory
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.tables] crawling new set of snapshot data for tables
20:05 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_sn1co] listing tables and views
20:05 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_sn1co.dummy_tmr5h] fetching table metadata
20:05 DEBUG [databricks.labs.ucx.hive_metastore.tables] [hive_metastore.dummy_sn1co.dummy_txfjs] fetching table metadata
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.tables] found 2 new records for tables
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.udfs] fetching udfs inventory
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.udfs] crawling new set of snapshot data for udfs
20:05 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_sn1co] listing udfs
20:05 DEBUG [databricks.labs.ucx.hive_metastore.udfs] [hive_metastore.dummy_sn1co.dummy_firae] fetching udf metadata
20:05 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.udfs] found 1 new records for udfs
20:08 WARNING [databricks.labs.ucx.hive_metastore.grants] Schema hive_metastore.dummy_sonpp no longer existed
20:16 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANY FILE : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-3b28bb6ed63ff707c24eea541dbe4ed6-e0c157e515c47483-00]
20:17 ERROR [databricks.labs.ucx.hive_metastore.grants] Couldn't fetch grants for object ANONYMOUS FUNCTION : TEMPORARILY_UNAVAILABLE: The service at /api/2.0/sql-acl/get-permissions is taking too long to process your request. Please try again later or try a faster operation. [TraceId: 00-46e1a53c6d66275eceeb6cd82032fdca-649c3b36ab6167e3-00]
20:17 DEBUG [databricks.labs.ucx.framework.crawlers] [hive_metastore.dummy_sf6vp.grants] found 100 new records for grants
[gw3] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

_{Running from acceptance #7363}

nfx

make integration tests less flaky
add cli command and document it in readme
check if we can avoid creating a CSV mapping

src/databricks/labs/ucx/hive_metastore/pipelines_migrate.py

tests/integration/hive_metastore/test_pipeline_migrate.py

src/databricks/labs/ucx/hive_metastore/pipelines_migrate.py

nfx · 2024-10-31T13:13:52Z

src/databricks/labs/ucx/hive_metastore/pipelines_migrate.py

+
+    def _migrate_pipelines(self):
+        # get pipelines to migrate
+        pipelines_to_migrate = self._pm.get_pipelines_to_migrate(self._pc)


why do we need a csv mapping for this one? we're copying pipelines, so there's less risk, right?

can you rather rename the old pipeline with [OLD] suffix and stop it. then start the new one.

also i'm not seeing the updates of pipeline ids into jobs that trigger them via pipeline task.

CSV mapping is to allow custom values for the optional parameters of clone API call: target_schema_name and target_pipeline_name. If doing a default 'as-is' migration, can directly migrate with default values. The csv will also allow for cleanup if the user does not want to migrate some pipelines.

Maybe we can:

Migrate all pipelines to one catalog.schema given input parameters from the cli. It is less flexible than a mapping, however, I do not know if it needs to be this flexible for pipelines.

Cleanup using a skip marker

src/databricks/labs/ucx/hive_metastore/workflows.py

JCZuurmond

Added some pointers

src/databricks/labs/ucx/hive_metastore/pipelines_migrate.py

JCZuurmond · 2024-11-01T08:27:26Z

src/databricks/labs/ucx/hive_metastore/pipelines_migrate.py

+        self._pipeline_crawler = pipelines_crawler
+        self._pipeline_mapping = pipeline_mapping
+
+    def migrate_pipelines(self) -> None:


This method looks redundant

src/databricks/labs/ucx/hive_metastore/pipelines_migrate.py

pritishpai force-pushed the feat/dlt_migrate branch from dfe9b45 to 96ccebf Compare October 11, 2024 15:37

pritishpai force-pushed the feat/dlt_migrate branch from 738abc5 to 0858c5c Compare October 20, 2024 16:07

nfx linked an issue Oct 23, 2024 that may be closed by this pull request

[EPIC]: Migrate Delta Live Table code #2065

Open

6 tasks

pritishpai force-pushed the feat/dlt_migrate branch from 1ab4210 to f28c72f Compare October 29, 2024 14:56

pritishpai marked this pull request as ready for review October 29, 2024 16:32

pritishpai requested a review from a team as a code owner October 29, 2024 16:32

pritishpai had a problem deploying to account-admin October 29, 2024 16:32 — with GitHub Actions Failure

pritishpai removed a link to an issue Oct 30, 2024

[EPIC]: Migrate Delta Live Table code #2065

Open

6 tasks

pritishpai linked an issue Oct 30, 2024 that may be closed by this pull request

[FEATURE]: Add PipelineMigrator that uses the DLT Migration (clone) API #3098

Open

1 task

pritishpai force-pushed the feat/dlt_migrate branch from 7dd9c06 to 71d98d0 Compare October 30, 2024 16:19

pritishpai had a problem deploying to account-admin October 30, 2024 16:19 — with GitHub Actions Error

pritishpai force-pushed the feat/dlt_migrate branch from 71d98d0 to e12dbd0 Compare October 30, 2024 16:37

pritishpai temporarily deployed to account-admin October 30, 2024 16:37 — with GitHub Actions Inactive

pritishpai force-pushed the feat/dlt_migrate branch from e12dbd0 to 28da733 Compare October 30, 2024 17:55

pritishpai temporarily deployed to account-admin October 30, 2024 17:55 — with GitHub Actions Inactive

pritishpai had a problem deploying to account-admin October 30, 2024 19:02 — with GitHub Actions Error

pritishpai temporarily deployed to account-admin October 30, 2024 19:13 — with GitHub Actions Inactive

pritishpai temporarily deployed to account-admin October 30, 2024 23:41 — with GitHub Actions Inactive

pritishpai requested review from nfx and JCZuurmond October 31, 2024 12:37

nfx requested changes Oct 31, 2024

View reviewed changes

pritishpai had a problem deploying to account-admin October 31, 2024 14:06 — with GitHub Actions Error

pritishpai had a problem deploying to account-admin October 31, 2024 14:16 — with GitHub Actions Error

pritishpai temporarily deployed to account-admin October 31, 2024 14:45 — with GitHub Actions Inactive

JCZuurmond reviewed Nov 1, 2024

View reviewed changes

pritishpai had a problem deploying to account-admin November 1, 2024 14:46 — with GitHub Actions Error

pritishpai had a problem deploying to account-admin November 1, 2024 14:56 — with GitHub Actions Error

pritishpai had a problem deploying to account-admin November 1, 2024 15:00 — with GitHub Actions Error

pritishpai had a problem deploying to account-admin November 1, 2024 15:07 — with GitHub Actions Error

pritishpai added 28 commits November 13, 2024 15:04

full names

178f46d

consolidate tests to 1 pipeline

fb11644

use context

d06a68b

add return types

a231bfb

isort

ddb1646

fmt

9a4c68b

fix check

450e6e7

restructure ifelse

dbb4f07

remove dunder methods

273721a

remove from workflows will be using as a cli command instead

61c0e59

add cli command

f4abd0a

add cached property for pipeline migrator

a4286e3

remove pipeline migration from workflows

5eb2e9e

migrate without mapping WIP

2476ecd

add skipping pipelines logic

678bcb7

older pipeline name update

c69f88e

ignore already migrated pipelines

e915c7e

add skip pipeline to integration test

2bf3c96

update jobs tasks with new pipeline ids

e438b81

cleanup and fmt

a862eca

make PR green

bf1199e

re-introduce unit test

429a8dd

fmt

bed57c2

add jobs mock

a093037

more cov

6e547b3

max cov

96e5eac

add README for cli

0b02bf0

hyperlinks and add to ToC

1709ea2

pritishpai force-pushed the feat/dlt_migrate branch from 8fee71e to 1709ea2 Compare November 13, 2024 20:04

pritishpai had a problem deploying to account-admin November 13, 2024 20:04 — with GitHub Actions Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrates Pipelines crawled during the assessment phase #2778

Migrates Pipelines crawled during the assessment phase #2778

pritishpai commented Oct 1, 2024 •

edited

Loading

github-actions bot commented Oct 29, 2024 •

edited

Loading

nfx left a comment

nfx Oct 31, 2024

pritishpai Oct 31, 2024

JCZuurmond Nov 1, 2024

JCZuurmond left a comment

JCZuurmond Nov 1, 2024

Migrates Pipelines crawled during the assessment phase #2778

Are you sure you want to change the base?

Migrates Pipelines crawled during the assessment phase #2778

Conversation

pritishpai commented Oct 1, 2024 • edited Loading

Changes

Linked issues

Functionality

Tests

github-actions bot commented Oct 29, 2024 • edited Loading

nfx left a comment

Choose a reason for hiding this comment

nfx Oct 31, 2024

Choose a reason for hiding this comment

pritishpai Oct 31, 2024

Choose a reason for hiding this comment

JCZuurmond Nov 1, 2024

Choose a reason for hiding this comment

JCZuurmond left a comment

Choose a reason for hiding this comment

JCZuurmond Nov 1, 2024

Choose a reason for hiding this comment

pritishpai commented Oct 1, 2024 •

edited

Loading

github-actions bot commented Oct 29, 2024 •

edited

Loading