-
Notifications
You must be signed in to change notification settings - Fork 166
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add docs a for exposure generic tests #1154
Add docs a for exposure generic tests #1154
Conversation
ELE-1703 Create generic test for exposure stability
Definition of done: Create a generic test defined per model. The test iterates on the defined exposures and looks for exposures that depend on the model, and checks the columns described in the exposure match the columns in the model (name and data type). |
👋 @erikzaadi |
docs/guides/add-exposure-tests.mdx
Outdated
## Exposure validation dbt tests | ||
|
||
Elementary dbt package includes **exposure validation tests, implemented as [dbt tests](https://docs.getdbt.com/docs/building-a-dbt-project/tests)**. | ||
These tests can detect missing columns and column data types for defined exposures. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@erikzaadi
This is a technical description of the test.
I think it's better to describe here the use case it solves.
"These tests will fail if there are column changes that break downstream exposures, such as BI dashboards."
Also, it's a good place to say that Elementary Cloud automatically generates exposures, maps the lineage, and adds the test to the models that the exposures depend on.
83deb76
to
a098d3c
Compare
|
||
``` | ||
|
||
We recommend adding a tag to the tests so you could execute these in a dedicated run using the selection parameter `--select tag:elementary`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we have this recommendation in all the test types?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
docs/guides/add-exposure-tests.mdx
Outdated
data_type: "numeric" | ||
``` | ||
|
||
Then in your module schema, add the elementary exposure tests: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it's important to point out that the test needs to be set up for every model used in the exposure.
Also you might wanna explain a little on how the test works, mainly to communicate why no other setup is needed except this test :)
docs/guides/add-exposure-tests.mdx
Outdated
source: ref('returned_orders') | ||
- name: "customer_id" | ||
data_type: "numeric" | ||
source: ref('orders') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure it's possible, but it will be nice if we could do something like:
referenced_columns:
ref('orders'):
- name: "order_id"
data_type: "numeric"
- name: "customer_id"
data_type: "numeric"
ref('customers'):
- ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not sure, and it'll make the "simple" workflow (AKA no need to set the source property since there's only one exposure dependency) obsolute, which IMHO is a shame, it's prolly our 80 vs 20 scenario
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yep I agree. but maybe we can rename "name" and "source" to "column_name" and "model_name"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pushed, only difference is node
instead of model_name
as per our conversation and @haritamar 's comment on the main PR
c32c7fd
to
579ed26
Compare
5aca7de
to
2e00e6c
Compare
2e00e6c
to
a366ba4
Compare
Depends on elementary-data/dbt-data-reliability#530
Added docs for the new exposure validation tests: