Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(ingest/tableau): graceful handling of get all datasources failure… #8406

Merged

Conversation

mayurinehate
Copy link
Collaborator

…, add fallback

Also adds TableauSourceReport to track failures when parsing tableau response.

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@mayurinehate mayurinehate force-pushed the tableau_handle_datasources_exc branch from 652fed1 to 47ea73c Compare July 12, 2023 06:26
@mayurinehate mayurinehate requested review from hsheth2, treff7es and asikowitz and removed request for treff7es July 12, 2023 06:28
@github-actions github-actions bot added the ingestion PR or Issue related to the ingestion of metadata label Jul 12, 2023
Copy link
Collaborator

@asikowitz asikowitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple refactor requests but LGTM

)
continue
self.datasource_project_map[ds.id] = ds.project_id
self.report.get_all_datasources_query_failed = False
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a no-op. Either way we should set the default to None or just remove this statement

@@ -1304,6 +1331,30 @@ def _get_published_datasource_project_luid(self, ds):

return None

def _query_published_datasource_for_project_luid(self, ds: dict) -> None:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we pass in ds[LUID] rather than ds here?

Copy link
Collaborator Author

@mayurinehate mayurinehate Jul 13, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

addressed in followup PR

Comment on lines +1674 to +1676
container_key = self._get_datasource_container_key(
datasource, workbook, is_embedded_ds
)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@anshbansal anshbansal merged commit 60b9623 into datahub-project:master Jul 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ingestion PR or Issue related to the ingestion of metadata
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants