Skip to content

Commit

Permalink
Do not block Cosmos if openlineage-common raises a jinja exception (#626
Browse files Browse the repository at this point in the history
)

Before this change, Cosmos failed to run if there is an issue in the
openlineage-common parsing of the dbt project due to a jinja2 exception,
which did not happen when running the dbt command by itself:

#612 (comment)

```
  File "/usr/local/lib/python3.11/site-packages/cosmos/operators/local.py", line 268, in calculate_openlineage_events_completes
    openlineage_processor = DbtLocalArtifactProcessor(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 57, in __init__
    dbt_project = self.load_yaml_with_jinja(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 157, in load_yaml_with_jinja
    return self.render_values_jinja(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 180, in render_values_jinja
    parsed_dict[key] = cls.render_values_jinja(
                       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 180, in render_values_jinja
    parsed_dict[key] = cls.render_values_jinja(
                       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 180, in render_values_jinja
    parsed_dict[key] = cls.render_values_jinja(
                       ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openlineage/common/provider/dbt/local.py", line 190, in render_values_jinja
    return environment.from_string(value).render()  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1301, in render
    self.environment.handle_exception()
  File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 936, in handle_exception
    raise rewrite_traceback_stack(source=source)
  File "<template>", line 1, in top-level template code
jinja2.exceptions.UndefinedError: 'run_started_at' is undefined
```
Closes: #612
Relates to: OpenLineage/OpenLineage#2212
  • Loading branch information
tatiana authored Oct 25, 2023
1 parent ad7dcf0 commit 4c65f25
Showing 1 changed file with 2 additions and 1 deletion.
3 changes: 2 additions & 1 deletion cosmos/operators/local.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import warnings

import airflow
import jinja2
import yaml
from airflow import DAG
from airflow.compat.functools import cached_property
Expand Down Expand Up @@ -283,7 +284,7 @@ def calculate_openlineage_events_completes(
try:
events = openlineage_processor.parse()
self.openlineage_events_completes = events.completes
except (FileNotFoundError, NotImplementedError, ValueError, KeyError):
except (FileNotFoundError, NotImplementedError, ValueError, KeyError, jinja2.exceptions.UndefinedError):
logger.debug("Unable to parse OpenLineage events", stack_info=True)

def get_datasets(self, source: Literal["inputs", "outputs"]) -> list[Dataset]:
Expand Down

0 comments on commit 4c65f25

Please sign in to comment.