-
Notifications
You must be signed in to change notification settings - Fork 161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor dbt project parsing and Airflow DAG generation #360
Conversation
👷 Deploy Preview for amazing-pothos-a3bca0 processing.
|
Codecov ReportPatch coverage:
Additional details and impacted files@@ Coverage Diff @@
## main #360 +/- ##
==========================================
+ Coverage 88.94% 90.70% +1.75%
==========================================
Files 39 44 +5
Lines 1303 1484 +181
==========================================
+ Hits 1159 1346 +187
+ Misses 144 138 -6
☔ View full report in Codecov by Sentry. |
37bdcc5
to
35ed763
Compare
3200346
to
4358fd7
Compare
Summary of improvements: - Unify where tests dependencies are declared (hatch) - Unify command to run integration tests (hatch) - Have a single way of defining connection for tests (environment variable) - Clean up CI workflow to be compatible with the cleanup - Improve documentation on how to run integration tests locally
…e of the DbtGraph class
…ng the custom dbt project parser
35a58f1
to
1ed023f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
approving conditional on unit tests passing
This is a major refactor related to how we render
dbt
projects in Airflow.Some of the changes:
DbtDag
andDbtTaskGroup
dbt
projects from the actual Airflow DAG generationdbt ls
manifest.json
Breaking changes:
from cosmos.dag import DbtDag
is nowfrom cosmos.airflow.dag import DbtDag
orfrom cosmos import DbtDag
from cosmos.dag import DbtTaskGroup
is nowfrom cosmos.airflow.dag import DbtTaskGroup
orfrom cosmos import DbtTaskGroup
select
andexclude
arguments, passed toDbtDag
orDbtTaskGroup
used to be dictionaries, now they are lists - so we are compatible with the representation in dbtDISCLAIMER: This PR is in draft mode because I'm improving the following:
(features)
DbtToAirflowConverter
classdbt ls
expects the user to have a validprofiles.yml
- we can extract the logic from the local executor. It feels we could do this in a follow-up PR(test features we lack integration tests)
(good practices)
Closes: #308