-
Notifications
You must be signed in to change notification settings - Fork 161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support persisting the LoadMode.VIRTUALENV
directory
#1079
Commits on Jul 29, 2024
-
Configuration menu - View commit details
-
Copy full SHA for cdc2582 - Browse repository at this point
Copy the full SHA cdc2582View commit details -
Configuration menu - View commit details
-
Copy full SHA for a2e4022 - Browse repository at this point
Copy the full SHA a2e4022View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6ac0225 - Browse repository at this point
Copy the full SHA 6ac0225View commit details -
Configuration menu - View commit details
-
Copy full SHA for bf3d1f1 - Browse repository at this point
Copy the full SHA bf3d1f1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5715d16 - Browse repository at this point
Copy the full SHA 5715d16View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4971c4e - Browse repository at this point
Copy the full SHA 4971c4eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 522e7ac - Browse repository at this point
Copy the full SHA 522e7acView commit details -
Configuration menu - View commit details
-
Copy full SHA for 905f602 - Browse repository at this point
Copy the full SHA 905f602View commit details -
Configuration menu - View commit details
-
Copy full SHA for ef245ba - Browse repository at this point
Copy the full SHA ef245baView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2fcc707 - Browse repository at this point
Copy the full SHA 2fcc707View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0a58f34 - Browse repository at this point
Copy the full SHA 0a58f34View commit details -
Configuration menu - View commit details
-
Copy full SHA for 55f785e - Browse repository at this point
Copy the full SHA 55f785eView commit details -
Configuration menu - View commit details
-
Copy full SHA for cd47ddc - Browse repository at this point
Copy the full SHA cd47ddcView commit details -
Support
ProjectConfig.dbt_project_path = None
& different paths for…… Rendering and Execution (#634) This MR finishes the work that was started in #605 to add full support for ProjectConfig.dbt_project_path = None, and implements #568. Within this PR, several things have been updated: 1 - Added project_path fields to RenderConfig and ExecutionConfig 2 - Simplified the consumption of RenderConfig in the dbtGraph class 3 - added option to configure different dbt executables for Rendering vs Execution. Closes: #568
Configuration menu - View commit details
-
Copy full SHA for ad54b59 - Browse repository at this point
Copy the full SHA ad54b59View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8e86ce2 - Browse repository at this point
Copy the full SHA 8e86ce2View commit details -
Fix reusing config accross TaskGroups/DAGs (#664)
If execution_config was reused, Cosmos 1.2.2 would raise: ``` astronomer-cosmos/dags/basic_cosmos_task_group.py Traceback (most recent call last): File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dagbag.py", line 343, in parse loader.exec_module(new_module) File "<frozen importlib._bootstrap_external>", line 848, in exec_module File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 74, in <module> basic_cosmos_task_group() File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/venv-38/lib/python3.8/site-packages/airflow/models/dag.py", line 3817, in factory f(**f_kwargs) File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/dags/basic_cosmos_task_group.py", line 54, in basic_cosmos_task_group orders = DbtTaskGroup( File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/airflow/task_group.py", line 26, in __init__ DbtToAirflowConverter.__init__(self, *args, **specific_kwargs(**kwargs)) File "/Users/tati/Code/cosmos-clean/astronomer-cosmos/cosmos/converter.py", line 113, in __init__ raise CosmosValueError( cosmos.exceptions.CosmosValueError: ProjectConfig.dbt_project_path is mutually exclusive with RenderConfig.dbt_project_path and ExecutionConfig.dbt_project_path.If using RenderConfig.dbt_project_path or ExecutionConfig.dbt_project_path, ProjectConfig.dbt_project_path should be None ``` This has been raised by an Astro customer and our field engineer, who tried to run: https://github.com/astronomer/cosmos-demo
Configuration menu - View commit details
-
Copy full SHA for e384177 - Browse repository at this point
Copy the full SHA e384177View commit details -
Configuration menu - View commit details
-
Copy full SHA for d0eeae0 - Browse repository at this point
Copy the full SHA d0eeae0View commit details -
Configuration menu - View commit details
-
Copy full SHA for bebe6d4 - Browse repository at this point
Copy the full SHA bebe6d4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 801a6dc - Browse repository at this point
Copy the full SHA 801a6dcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 521ce5a - Browse repository at this point
Copy the full SHA 521ce5aView commit details -
Configuration menu - View commit details
-
Copy full SHA for bd5aa76 - Browse repository at this point
Copy the full SHA bd5aa76View commit details -
Configuration menu - View commit details
-
Copy full SHA for 376616b - Browse repository at this point
Copy the full SHA 376616bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 738ee73 - Browse repository at this point
Copy the full SHA 738ee73View commit details -
Configuration menu - View commit details
-
Copy full SHA for 610e5a4 - Browse repository at this point
Copy the full SHA 610e5a4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7aabbf6 - Browse repository at this point
Copy the full SHA 7aabbf6View commit details -
Configuration menu - View commit details
-
Copy full SHA for c33dc5b - Browse repository at this point
Copy the full SHA c33dc5bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2949862 - Browse repository at this point
Copy the full SHA 2949862View commit details -
Configuration menu - View commit details
-
Copy full SHA for 490855a - Browse repository at this point
Copy the full SHA 490855aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 551fd28 - Browse repository at this point
Copy the full SHA 551fd28View commit details -
Configuration menu - View commit details
-
Copy full SHA for 16b83bb - Browse repository at this point
Copy the full SHA 16b83bbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4fe457e - Browse repository at this point
Copy the full SHA 4fe457eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4068edc - Browse repository at this point
Copy the full SHA 4068edcView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0e17d30 - Browse repository at this point
Copy the full SHA 0e17d30View commit details
Commits on Aug 14, 2024
-
Configuration menu - View commit details
-
Copy full SHA for d9c7cc4 - Browse repository at this point
Copy the full SHA d9c7cc4View commit details -
⬆ [pre-commit.ci] pre-commit autoupdate (#1125)
<!--pre-commit.ci start--> updates: - [github.com/asottile/pyupgrade: v3.16.0 → v3.17.0](asottile/pyupgrade@v3.16.0...v3.17.0) - [github.com/astral-sh/ruff-pre-commit: v0.5.4 → v0.5.5](astral-sh/ruff-pre-commit@v0.5.4...v0.5.5) <!--pre-commit.ci end--> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for 0042e76 - Browse repository at this point
Copy the full SHA 0042e76View commit details -
Fix import handling by lazy loading hooks introduced in PR #1109 (#1132)
Making an update to #1109, which introduced module-level imports of optional dependencies. This is inappropriate as it will break if the user does not have them installed, and indeed the user really does not need them installed if they are not relying on them directly. This PR lazy-loads the imports so that it does not impact users who do not need them. In the upath library, `az:`, `adl:`, `abfs:` and `abfss:` are also all valid schemes, albeit Airflow only references the latter 3 in the code: https://github.com/apache/airflow/blob/e3824eaaba7eada9a807f7a2f9f89d977a210e15/airflow/providers/microsoft/azure/fs/adls.py#L29, so `adl:`, `abfs:` and `abfss:` also have been added to the list of schemes supported.
Configuration menu - View commit details
-
Copy full SHA for 63a2bf4 - Browse repository at this point
Copy the full SHA 63a2bf4View commit details -
⬆ [pre-commit.ci] pre-commit autoupdate (#1144)
<!--pre-commit.ci start--> updates: - [github.com/astral-sh/ruff-pre-commit: v0.5.5 → v0.5.6](astral-sh/ruff-pre-commit@v0.5.5...v0.5.6) - [github.com/psf/black: 24.4.2 → 24.8.0](psf/black@24.4.2...24.8.0) - [github.com/pre-commit/mirrors-mypy: v1.11.0 → v1.11.1](pre-commit/mirrors-mypy@v1.11.0...v1.11.1) <!--pre-commit.ci end--> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for 31041fe - Browse repository at this point
Copy the full SHA 31041feView commit details -
Cache package-lock.yml file (#1086)
This PR aims to cache the package-lock.yml in `cache_dir/dbt_project` Since dbt version 1.7.0, executing the dbt deps command results in the generation of a package-lock.yml file. This file pins the dependencies and their versions for the dbt project. dbt uses this file to install packages, ensuring predictable and consistent package installations across environments. - This feature is enabled only if the user checks in package-lock.yml in their dbt project. Also, I'm assuming if `package-lock.yml` their dbt-core version is >= 1.7.0 since this feature is available for only dbt >= 1.7.0 - package-lock.yml also contains the sha1_hash of the packages. This is used to check if the cached package-lock.yml is outdated or not in this PR - The cached `package-lock.yml` is finally copied from from cached path to the tmp project and used - To update dependencies or versions, it is expected that the user will manually update their package-lock.yml in the dbt project using the dbt deps command. closes: #930
Configuration menu - View commit details
-
Copy full SHA for 0f0d93a - Browse repository at this point
Copy the full SHA 0f0d93aView commit details -
⬆ [pre-commit.ci] pre-commit autoupdate (#1154)
<!--pre-commit.ci start--> updates: - [github.com/astral-sh/ruff-pre-commit: v0.5.6 → v0.5.7](astral-sh/ruff-pre-commit@v0.5.6...v0.5.7) <!--pre-commit.ci end--> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Configuration menu - View commit details
-
Copy full SHA for f333714 - Browse repository at this point
Copy the full SHA f333714View commit details -
Add default source nodes rendering (#1107)
Re-Opening of PR #661 This PR features a new way of rendering source nodes: - Check freshness for sources with freshness checks - Source tests - Empty operators for nodes without tests or freshness. One of the main limitations I found while using the `custom_callback` functions on source nodes to check freshness is that nodes were being created on 100% of sources but not all of them required freshness checks, this made workers waste compute time. I'm adding a new variable into the DbtNode class called has_freshness which would be True for sources with freshness checks and False for any other resource type. If this feature is enabled with the option `ALL`: All sources with the has_freshness == False will be rendered as Empty Operators, to keep the dbt's behavior of showing sources as suggested in issue #630 <!-- Add a brief but complete description of the change. --> A new rendered template field is included too: `freshness` which is the sources.json generated by dbt when running `dbt source freshness` This adds a new node type (source), which changes some tests behavior. This PR also updates the dev dbt project jaffle_shop to include source nodes when enabled. ![image](https://github.com/user-attachments/assets/e972ac58-8741-4c13-9905-e78775f9cc80) As seen in the image, source nodes with freshness checks are rendered with a blue color, while the ones rendered as EmptyOperator show a white/light green color Closes: #630 Closes: #572 Closes: #875 <!-- If this PR closes an issue, you can use a keyword to auto-close. --> <!-- i.e. "closes #0000" --> This won't be a breaking change since the default behavior will still be ignoring this new feature. That can be changed with the new RenderConfig variable called `source_rendering_behavior`. Co-authored-by: Pankaj <[email protected]> Co-authored-by: Pankaj Singh <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for f1a1273 - Browse repository at this point
Copy the full SHA f1a1273View commit details -
Configuration menu - View commit details
-
Copy full SHA for eb983d6 - Browse repository at this point
Copy the full SHA eb983d6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 5da4395 - Browse repository at this point
Copy the full SHA 5da4395View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7d0c31a - Browse repository at this point
Copy the full SHA 7d0c31aView commit details -
Configuration menu - View commit details
-
Copy full SHA for 31bbd19 - Browse repository at this point
Copy the full SHA 31bbd19View commit details
Commits on Aug 15, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 8621e49 - Browse repository at this point
Copy the full SHA 8621e49View commit details -
Configuration menu - View commit details
-
Copy full SHA for c48f1ca - Browse repository at this point
Copy the full SHA c48f1caView commit details
Commits on Aug 16, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 0f7fcf4 - Browse repository at this point
Copy the full SHA 0f7fcf4View commit details -
Configuration menu - View commit details
-
Copy full SHA for a712c27 - Browse repository at this point
Copy the full SHA a712c27View commit details -
Configuration menu - View commit details
-
Copy full SHA for 43cb07d - Browse repository at this point
Copy the full SHA 43cb07dView commit details -
Configuration menu - View commit details
-
Copy full SHA for b465b2a - Browse repository at this point
Copy the full SHA b465b2aView commit details -
Configuration menu - View commit details
-
Copy full SHA for bf35988 - Browse repository at this point
Copy the full SHA bf35988View commit details -
Configuration menu - View commit details
-
Copy full SHA for 41fe21c - Browse repository at this point
Copy the full SHA 41fe21cView commit details -
Configuration menu - View commit details
-
Copy full SHA for b6813af - Browse repository at this point
Copy the full SHA b6813afView commit details -
Configuration menu - View commit details
-
Copy full SHA for 0ff5c7a - Browse repository at this point
Copy the full SHA 0ff5c7aView commit details -
Configuration menu - View commit details
-
Copy full SHA for f4d0c3a - Browse repository at this point
Copy the full SHA f4d0c3aView commit details