-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(integration/prefect): prefect block to emit prefect pipeline #8414
feat(integration/prefect): prefect block to emit prefect pipeline #8414
Conversation
…amjagtap639/datahub into prefect-source-integration
…amjagtap639/datahub into prefect-source-integration
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Things to fix here:
- there's conflicts on some files
- it seems like a fair bit of it was copied from the airflow connector / prefect template repo, whether or not it was actually necessary (tox.ini, mkdocs, extra things in setup.py and setup.cfg, etc). let's remove those extraneous things
Other thing to look into: the airflow plugin is able to inspect the SQL executed by certain tasks in order to automatically extract lineage. Can we do something similar here? Might even be ok if they need to manually call datahub_emitter.add_sql(...)
or something
"task_key": "__main__.transform", | ||
"tags": ["etl flow task"], | ||
} | ||
mock_extract_task_run_json: Dict = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
all of this data should go in the test file where it is used, not conftest.py
…into prefect-source-integration
Closing in favor of #10643 |
Checklist