Skip to content

Commit

Permalink
Fix using the full_refresh argument in projects that contain tests
Browse files Browse the repository at this point in the history
Stop forwarding the `full_refresh` argument to dbt commands that do not support it.

Before this change, if users attempted to use:
```
operator_args={"full_refresh": True},
```

In dbt projects with nodes and other resources that do not support the `--full-refresh` flag, the `DbtTaskGroup` or `DbtDag` would fail, causing:

```
airflow.exceptions.AirflowException: Invalid arguments were passed to
DbtTestLocalOperator (task_id: test). Invalid arguments were:
**kwargs: {'full_refresh': True}
```

This issue was originally reported  in Slack:
https://apache-airflow.slack.com/archives/C059CC42E9W/p1696857998959909

Closes: #589

Co-authored-by: Tatiana Al-Chueyr <[email protected]>
  • Loading branch information
EgorSemenov and tatiana authored Oct 13, 2023
1 parent e80abd7 commit a426dd2
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 1 deletion.
1 change: 1 addition & 0 deletions cosmos/operators/local.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,7 @@ def __init__(
self.compiled_sql = ""
self.should_store_compiled_sql = should_store_compiled_sql
self.openlineage_events_completes: list[RunEvent] = []
kwargs.pop("full_refresh", None) # usage of this param should be implemented in child classes
super().__init__(**kwargs)

@cached_property
Expand Down
5 changes: 4 additions & 1 deletion dev/dags/basic_cosmos_dag.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,10 @@
DBT_ROOT_PATH / "jaffle_shop",
),
profile_config=profile_config,
operator_args={"install_deps": True},
operator_args={
"install_deps": True, # install any necessary dependencies before running any dbt command
"full_refresh": True, # used only in dbt commands that support this flag
},
# normal dag parameters
schedule_interval="@daily",
start_date=datetime(2023, 1, 1),
Expand Down
1 change: 1 addition & 0 deletions tests/operators/test_local.py
Original file line number Diff line number Diff line change
Expand Up @@ -332,6 +332,7 @@ def test_store_compiled_sql() -> None:
[
(DbtSeedLocalOperator, {"full_refresh": True}, {"context": {}, "cmd_flags": ["--full-refresh"]}),
(DbtRunLocalOperator, {"full_refresh": True}, {"context": {}, "cmd_flags": ["--full-refresh"]}),
(DbtTestLocalOperator, {"full_refresh": True}, {"context": {}}),
(
DbtRunOperationLocalOperator,
{"args": {"days": 7, "dry_run": True}, "macro_name": "bla"},
Expand Down

0 comments on commit a426dd2

Please sign in to comment.