Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report fails to generate if exposure owners is string #1732

Open
arthurcw opened this issue Oct 26, 2024 · 1 comment
Open

Report fails to generate if exposure owners is string #1732

arthurcw opened this issue Oct 26, 2024 · 1 comment
Labels
Bug Something isn't working Triage 👀

Comments

@arthurcw
Copy link

arthurcw commented Oct 26, 2024

Describe the bug
Could not generate the report - Error: validation error for ExposureSchema. It's similar to #827

To Reproduce
Configure the exposure owner to:

owner:
  name: abcdefg

Expected behavior
edr report or edr send-report to run without errors

Screenshots

INFO — Running with edr=0.16.1
INFO — Installing packages for edr internal dbt package...
INFO — Running dbt command --log-format json deps --project-dir /opt/venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
INFO — edr (0.16.1) and Elementary's dbt package (0.16.0) are compatible.
INFO — Elementary's database and schema: '"abc.efg"'
INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /opt/venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 7, "invocations_per_test": 720}} --project-dir /opt/venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
 INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /opt/venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /opt/venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /opt/venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
ERROR — Could not generate the report - Error: 1 validation error for ExposureSchema
INFO - [base] owners
INFO - [base]   value is not a valid list (type=type_error.list)
INFO - [base] Please reach out to our community for help with this issue.
INFO - [base] Traceback (most recent call last):
INFO - [base]   File "/opt/venv/lib/python3.11/site-packages/elementary/monitor/api/report/report.py", line 72, in get_report_data
INFO - [base]     exposures = models_api.get_exposures(upstream_node_ids=lineage_node_ids)
INFO - [base]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO - [base]   File "/opt/venv/lib/python3.11/site-packages/elementary/monitor/api/models/models.py", line 158, in get_exposures
INFO - [base]     exposures_results = self.models_fetcher.get_exposures()
INFO - [base]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO - [base]   File "/opt/venv/lib/python3.11/site-packages/elementary/monitor/fetchers/models/models.py", line 72, in get_exposures
INFO - [base]     exposures = [ExposureSchema(**exposure) for exposure in exposures]
INFO - [base]                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO - [base]   File "/opt/venv/lib/python3.11/site-packages/elementary/monitor/fetchers/models/models.py", line 72, in <listcomp>
INFO - [base]     exposures = [ExposureSchema(**exposure) for exposure in exposures]
INFO - [base]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
INFO - [base]   File "/opt/venv/lib/python3.11/site-packages/pydantic/v1/main.py", line 341, in __init__
INFO - [base]     raise validation_error
INFO - [base] pydantic.v1.error_wrappers.ValidationError: 1 validation error for ExposureSchema
INFO - [base] owners
INFO - [base]   value is not a valid list (type=type_error.list)

Environment (please complete the following information):

  • Elementary CLI (edr) version: 0.16.1
  • Elementary dbt package version: 0.16.0
  • dbt version you're using: 1.8.6
  • Data warehouse: BigQuery
  • Infrastructure details: prod

Additional context
I think we need updates to the _load_var_to_list method here

  1. At this step, expsoure["owners"] = "abcdefg"
  2. When running load_owners method in ArtifactSchema, it calls the cls._load_var_to_list(owners)method
  3. _load_var_to_list will return abcdefg and not a list

Would you be willing to contribute a fix for this issue?
Yes. I think either need to adjust how the owners is returned from self.models_fetcher.get_exposures() or adjust __load_var_to_list method.

@arthurcw arthurcw added Bug Something isn't working Triage 👀 labels Oct 26, 2024
@haritamar
Copy link
Collaborator

Thanks for reporting @arthurcw !
Interesting - I'd expect _load_var_to_list to work correctly since try_load_json would return None and we'd return a list with a single string as a result.

Though admittedly the code is confusing so maybe there's a bug there. In any case if you have a fix please feel free to submit a PR.

Thanks!
Itamar

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Something isn't working Triage 👀
Projects
None yet
Development

No branches or pull requests

2 participants