Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(integration/prefect): prefect block to emit prefect pipeline #8414

Conversation

shubhamjagtap639
Copy link
Contributor

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added ingestion PR or Issue related to the ingestion of metadata docs Issues and Improvements to docs devops PR or Issue related to DataHub backend & deployment labels Jul 13, 2023
Copy link
Collaborator

@hsheth2 hsheth2 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Things to fix here:

  • there's conflicts on some files
  • it seems like a fair bit of it was copied from the airflow connector / prefect template repo, whether or not it was actually necessary (tox.ini, mkdocs, extra things in setup.py and setup.cfg, etc). let's remove those extraneous things

Other thing to look into: the airflow plugin is able to inspect the SQL executed by certain tasks in order to automatically extract lineage. Can we do something similar here? Might even be ok if they need to manually call datahub_emitter.add_sql(...) or something

"task_key": "__main__.transform",
"tags": ["etl flow task"],
}
mock_extract_task_run_json: Dict = {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all of this data should go in the test file where it is used, not conftest.py

@anshbansal
Copy link
Collaborator

Closing in favor of #10643

@anshbansal anshbansal closed this Jun 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
devops PR or Issue related to DataHub backend & deployment docs Issues and Improvements to docs ingestion PR or Issue related to the ingestion of metadata
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants