fix(ingest/teradata): Teradata speed up changes #9059
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
ingestion
asikowitz
reviewed
Oct 20, 2023
Comment on lines
+350
to
+741
| setattr( # noqa: B010 | ||
| inspector, | ||
| "get_table_names", | ||
| lambda schema: [ | ||
| i.name | ||
| for i in filter( | ||
| lambda t: t.object_type != "View", self._tables_cache[schema] | ||
| ) | ||
| ], | ||
| ) | ||
| yield from super().loop_tables(inspector, schema, sql_config) |
There was a problem hiding this comment.
This seems kinda hacky lol, do we need to make our SQL common source more extensible?
Comment on lines
234
to
239
| WHERE DatabaseName NOT IN ('All', 'Crashdumps', 'DBC', 'dbcmngr', | ||
| 'Default', 'External_AP', 'EXTUSER', 'LockLogShredder', 'PUBLIC', | ||
| 'Sys_Calendar', 'SysAdmin', 'SYSBAR', 'SYSJDBC', 'SYSLIB', | ||
| 'SystemFe', 'SYSUDTLIB', 'SYSUIF', 'TD_SERVER_DB', 'TDStats', | ||
| 'TD_SYSGPL', 'TD_SYSXML', 'TDMaps', 'TDPUSER', 'TDQCD', | ||
| 'tdwm', 'SQLJ', 'TD_SYSFNLIB', 'SYSSPATIAL') |
There was a problem hiding this comment.
Would be nice to reuse this list
|
|
||
| use_cached_metadata: bool = Field( | ||
| default=True, | ||
| description="Whether to use cached metadata. This reduce the number of queries to the database but requires to have cached metadata.", |
There was a problem hiding this comment.
Suggested change
| description="Whether to use cached metadata. This reduce the number of queries to the database but requires to have cached metadata.", | |
| description="Whether to use cached metadata. This reduces the number of queries to the database but requires storing all tables in memory.", |
Comment on lines
283
to
596
| # self.loop_tables = self.cached_loop_tables | ||
| # self.loop_views = self.cached_loop_views | ||
| # self.get_table_properties = self.cached_get_table_properties |
There was a problem hiding this comment.
Suggested change
| # self.loop_tables = self.cached_loop_tables | |
| # self.loop_views = self.cached_loop_views | |
| # self.get_table_properties = self.cached_get_table_properties |
Comment on lines
287
to
288
| # self._get_columns = lambda dataset_name, inspector, schema, table: [] | ||
| # self._get_foreign_keys = lambda dataset_name, inspector, schema, table: [] |
There was a problem hiding this comment.
Suggested change
| # self._get_columns = lambda dataset_name, inspector, schema, table: [] | |
| # self._get_foreign_keys = lambda dataset_name, inspector, schema, table: [] |
Comment on lines
+327
to
+710
| # url = self.config.get_sql_alchemy_url(current_db=db) | ||
| # with create_engine(url, **self.config.options).connect() as conn: | ||
| # inspector = inspect(conn) |
There was a problem hiding this comment.
Suggested change
| # url = self.config.get_sql_alchemy_url(current_db=db) | |
| # with create_engine(url, **self.config.options).connect() as conn: | |
| # inspector = inspect(conn) |
treff7es
commented
Oct 24, 2023
| self.report.num_queries_parsed += 1 | ||
| if self.report.num_queries_parsed % 1000 == 0: | ||
| logger.info(f"Parsed {self.report.num_queries_parsed} queries") | ||
|
|
||
| print(entry.query) |
There was a problem hiding this comment.
You left this print here, I don't know if it was intentional
maggiehays
added
the
hacktoberfest-accepted
treff7es
force-pushed
the
teradata_speed_up
branch
from
0b2ce01
to
ef454ee
Compare
Fixing empty lineage
treff7es
force-pushed
the
teradata_speed_up
branch
from
ef454ee
to
4fb4d2a
Compare
asikowitz
approved these changes
Nov 22, 2023
|
|
||
| if self.config.use_cached_metadata: | ||
| if self.config.include_tables or self.config.include_views: |
There was a problem hiding this comment.
Do we not make any of these queries if include_tables and include_views are false?
| ) -> Iterable[MetadataWorkUnit]: | ||
| result = sqlglot_lineage( | ||
| sql=query, | ||
| # With this clever hack we can make the query parser to not fail on queries with CASESPECIFIC | ||
| sql=query.replace("(NOT CASESPECIFIC)", ""), |
There was a problem hiding this comment.
I actually have a whole host of query parsing hacks to add, somewhere, lol
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist