diff --git a/website/docs/best-practices/how-we-build-our-metrics/semantic-layer-4-build-metrics.md b/website/docs/best-practices/how-we-build-our-metrics/semantic-layer-4-build-metrics.md index cd0efdc9e64..8b112f2f828 100644 --- a/website/docs/best-practices/how-we-build-our-metrics/semantic-layer-4-build-metrics.md +++ b/website/docs/best-practices/how-we-build-our-metrics/semantic-layer-4-build-metrics.md @@ -34,8 +34,15 @@ metrics: ## Query your metric -- It's best practice any time we're updating our semantic layer code to run a `dbt parse && mf validate-configs`. +Use [MetricFlow commands](/docs/build/metricflow-commands#metricflow) for metric validation or queries during development, and apply the following conventions based on your environment: + +- For dbt Cloud, use the `dbt sl` prefix before the command (such as, `dbt sl parse` or `dbt sl query`). +- For dbt Core, use the `mf` prefix (such as `mf validate-configs` or `mf query)`. + +Follow these best practices when updating your semantic layer code, using the `mf` command as an example (replace `mf` with `dbt sl` if you're using dbt Cloud): + +- It's best practice any time we're updating our semantic layer code to run `dbt parse` if using dbt Cloud or `dbt parse && mf validate-configs` if using dbt Core, to validate your configs. - If everything passes, we can start querying this metric with `mf query`! - `mf query` is not how you would use the tool in production, that's handled by the dbt Cloud Semantic Layer's features. It's available for testing results of various metric queries in development, exactly as we're using it now. - Try `mf query --metrics revenue --group-by metric_time__day` and see a preview of the data come back. -- Note the structure of the above query. We select the metric(s) we want and the dimensions to group them by — we use dunders (double underscores e.g.`metric_time__[time bucket]`) to designate time dimensions or other non-unique dimensions that need a specified entity path to resolve (e.g. if you have a orders location dimension and a employee location dimension both named 'location' you would need dunders to specify `orders__location` or `employee__location`). +- Note the structure of the above query. We select the metric(s) we want and the dimensions to group them by — we use dunders (double underscores e.g.`metric_time__[time bucket]`) to designate time dimensions or other non-unique dimensions that need a specified entity path to resolve (e.g. if you have an orders location dimension and an employee location dimension both named 'location' you would need dunders to specify `orders__location` or `employee__location`). diff --git a/website/docs/docs/build/metricflow-commands.md b/website/docs/docs/build/metricflow-commands.md index 2386dab4ba2..fd120591900 100644 --- a/website/docs/docs/build/metricflow-commands.md +++ b/website/docs/docs/build/metricflow-commands.md @@ -180,9 +180,9 @@ Options: ### Validate-configs This command performs validations against the defined semantic model configurations: + ```bash -dbt sl validate-configs # In dbt Cloud mf validate-configs # In dbt Core @@ -206,20 +206,18 @@ Options: ### Health checks This command performs a health check against the data platform you provided in the configs: + ```bash -dbt sl health-checks #in dbt Cloud - mf health-checks #in dbt Core ``` ### Tutorial Follow the dedicated MetricFlow tutorial to help you get started: + ```bash -dbt sl tutorial # In dbt Cloud - mf tutorial # In dbt Core ``` @@ -522,7 +520,7 @@ mf query --metrics revenue --group-by metric_time__month # In dbt Core To add a dimension filter to a where filter, you have to indicate that the filter item is part of your model and use a template wrapper: {{Dimension('primary_entity__dimension_name')}}. -Here's an example query: dbt sl query --metrics order_total --group-by metric_time --where "{{Dimension('order_id__is_food_order')}} = True".

Before using the template wrapper, however, you will need to set up your terminal to escape curly braces for the filter template to work. +Here's an example query: dbt sl query --metrics order_total --group-by metric_time --where "{{Dimension('order_id__is_food_order')}} = True".

Before using the template wrapper, however, set up your terminal to escape curly braces for the filter template to work.
How to set up your terminal to escape curly braces? diff --git a/website/docs/docs/core/connect-data-platform/bigquery-setup.md b/website/docs/docs/core/connect-data-platform/bigquery-setup.md index a630cbb0bd3..96eafadea3b 100644 --- a/website/docs/docs/core/connect-data-platform/bigquery-setup.md +++ b/website/docs/docs/core/connect-data-platform/bigquery-setup.md @@ -455,6 +455,7 @@ my-profile: dataproc_region: us-central1 submission_method: serverless dataproc_batch: + batch_id: MY_CUSTOM_BATCH_ID # Supported in v1.7+ environment_config: execution_config: service_account: dbt@abc-123.iam.gserviceaccount.com diff --git a/website/docs/docs/dbt-versions/release-notes/02-Nov-2023/job-notifications-rn.md b/website/docs/docs/dbt-versions/release-notes/02-Nov-2023/job-notifications-rn.md new file mode 100644 index 00000000000..660129513d7 --- /dev/null +++ b/website/docs/docs/dbt-versions/release-notes/02-Nov-2023/job-notifications-rn.md @@ -0,0 +1,17 @@ +--- +title: "Enhancement: Email and Slack job notifications" +description: "November 2023: New quality-of-life improvements for setting up and administering email and Slack job notifications" +sidebar_label: "Enhancement: Job notifications" +sidebar_position: 10 +tags: [Nov-2023] +--- + +There are new quality-of-life improvements in dbt Cloud for email and Slack notifications about your jobs: + +- You can add external email addresses and send job notifications to them. External emails can be: + - Addresses that are outside of your dbt Cloud account + - Third-party integration addresses for configuring notifications to services like Microsoft Teams or PagerDuty +- You can configure notifications for multiple Slack channels. Previously, you could only configure one Slack channel. +- Any account admin can now edit slack notifications, not just the person who created them. + +To learn more, check out [Job notifications](/docs/deploy/job-notifications). \ No newline at end of file diff --git a/website/docs/reference/configs-and-properties.md b/website/docs/reference/configs-and-properties.md index c2ad5b77629..8a557c762ed 100644 --- a/website/docs/reference/configs-and-properties.md +++ b/website/docs/reference/configs-and-properties.md @@ -11,7 +11,7 @@ A rule of thumb: properties declare things _about_ your project resources; confi For example, you can use resource **properties** to: * Describe models, snapshots, seed files, and their columns -- Assert "truths" about a model, in the form of [tests](/docs/build/tests), e.g. "this `id` column is unique" +* Assert "truths" about a model, in the form of [tests](/docs/build/tests), e.g. "this `id` column is unique" * Define pointers to existing tables that contain raw data, in the form of [sources](/docs/build/sources), and assert the expected "freshness" of this raw data * Define official downstream uses of your data models, in the form of [exposures](/docs/build/exposures) @@ -35,11 +35,11 @@ dbt prioritizes configurations in order of specificity, from most specificity to Note - Generic tests work a little differently when it comes to specificity. See [test configs](/reference/test-configs). -Within the project file, configurations are also applied hierarchically. The most-specific config always "wins": In the project file, configurations applied to a `marketing` subdirectory will take precedence over configurations applied to the entire `jaffle_shop` project. To apply a configuration to a model, or directory of models, define the resource path as nested dictionary keys. +Within the project file, configurations are also applied hierarchically. The most specific config always "wins": In the project file, configurations applied to a `marketing` subdirectory will take precedence over configurations applied to the entire `jaffle_shop` project. To apply a configuration to a model, or directory of models, define the resource path as nested dictionary keys. ### Combining configs -Most configurations are "clobbered" when applied hierarchically. Whenever a more-specific value is available, it will completely replace the less-specific value. Note that a few configs have different merge behavior: +Most configurations are "clobbered" when applied hierarchically. Whenever a more specific value is available, it will completely replace the less specific value. Note that a few configs have different merge behavior: - [`tags`](tags) are additive. If a model has some tags configured in `dbt_project.yml`, and more tags applied in its `.sql` file, the final set of tags will include all of them. - [`meta`](/reference/resource-configs/meta) dictionaries are merged (a more specific key-value pair replaces a less specific value with the same key) - [`pre-hook` and `post-hook`](/reference/resource-configs/pre-hook-post-hook) are also additive. @@ -67,12 +67,14 @@ Previous versions of the docs referred to these as `schema.yml` files — we've dbt has the ability to define node configs in `.yml` files, in addition to `config()` blocks and `dbt_project.yml`. But the reverse isn't always true: there are some things in `.yml` files that can _only_ be defined there. Certain properties are special, because: + - They have a unique Jinja rendering context - They create new project resources - They don't make sense as hierarchical configuration - They're older properties that haven't yet been redefined as configs These properties are: + - [`description`](/reference/resource-properties/description) - [`tests`](/reference/resource-properties/tests) - [`docs`](/reference/resource-configs/docs) @@ -202,3 +204,4 @@ Runtime Error ``` This error occurred because a semicolon (`;`) was accidentally used instead of a colon (`:`) after the `description` field. To resolve issues like this, find the `.yml` file referenced in the error message and fix any syntax errors present in the file. There are online YAML validators that can be helpful here, but please be mindful of submitting sensitive information to third-party applications! + diff --git a/website/docs/reference/dbt_project.yml.md b/website/docs/reference/dbt_project.yml.md index 9bd85d0d5dd..caf501c27ab 100644 --- a/website/docs/reference/dbt_project.yml.md +++ b/website/docs/reference/dbt_project.yml.md @@ -1,5 +1,5 @@ -Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. It also contains important information that tells dbt how to operate on your project. +Every [dbt project](/docs/build/projects) needs a `dbt_project.yml` file — this is how dbt knows a directory is a dbt project. It also contains important information that tells dbt how to operate your project. @@ -96,6 +96,7 @@ vars: + ```yml diff --git a/website/docs/reference/project-configs/version.md b/website/docs/reference/project-configs/version.md index 1c947412fcd..890ad8542a7 100644 --- a/website/docs/reference/project-configs/version.md +++ b/website/docs/reference/project-configs/version.md @@ -8,15 +8,17 @@ import VersionsCallout from '/snippets/_version-callout.md'; - -dbt projects have two distinct types of the `version` tags. This field has a different meaning depending on its location. +dbt projects have two distinct types of `version` tags. This field has a different meaning depending on its location. ## `dbt_project.yml` versions -The version tag in a `dbt_project` file represents the version of your dbt project. Starting in version 1.5, `version` in the `dbt_project.yml` is an *optional parameter*. If specified, the version must be in a [semantic version](https://semver.org/) format, e.g. `1.0.0`. The default value if not specified is `None`. +The version tag in a `dbt_project` file represents the version of your dbt project. + +Starting in dbt version 1.5, `version` in the `dbt_project.yml` is an *optional parameter*. If used, the version must be in a [semantic version](https://semver.org/) format, such as `1.0.0`. The default value is `None` if not specified. For users on dbt version 1.4 or lower, this tag is required, though it isn't currently used meaningfully by dbt. For more on Core versions, see [About dbt Core versions](/docs/dbt-versions/core). + ```yml @@ -29,9 +31,9 @@ version: version A version tag in a `.yml` property file provides the control tag, which informs how dbt processes property files. -Starting from version 1.5, dbt will no longer require this configuration in your resource `.yml` files. If you want to know more about why this tag was previously required, you can refer to the [property file FAQs](reference/configs-and-properties#faqs). +Starting from version 1.5, dbt will no longer require this configuration in your resource `.yml` files. If you want to know more about why this tag was previously required, you can refer to the [FAQs](#faqs). For users on dbt version 1.4 or lower, this tag is required, -For more on property files, see their general [documentation](reference/configs-and-properties#where-can-i-define-properties) on the same page. +For more on property files, see their general [documentation](/reference/configs-and-properties#where-can-i-define-properties) on the same page. +## FAQS - - - - -dbt projects have two distinct types of `version` tags. This field has a different meaning depending on its location. - -## `dbt_project.yml` versions - -The version tag in a `dbt_project` file represents the version of your dbt project and **is a required parameter**. However, it isn't currently used in a meaningful way by dbt. The version must follow a [semantic version](https://semver.org/) format, such as 1.0.0. For more information about dbt Core versions, refer to [About dbt Core versions](/docs/dbt-versions/core). - - -```yml -version: version -``` - - - -## `.yml` property file versions - -A version tag in a `.yml` property file provides the control tag, which informs how dbt processes property files. For more on why we require this tag, see property file [FAQs](reference/configs-and-properties#faqs). - -For more on property files, see their general [documentation](reference/configs-and-properties#where-can-i-define-properties) on the same page. - - - -```yml -version: 2 # Only 2 is accepted by current and recent versions of dbt. - -models: - ... -``` - - - - + diff --git a/website/docs/reference/resource-configs/docs.md b/website/docs/reference/resource-configs/docs.md index 0ccd21d7504..d300979a826 100644 --- a/website/docs/reference/resource-configs/docs.md +++ b/website/docs/reference/resource-configs/docs.md @@ -17,10 +17,12 @@ default_value: {show: true} { label: 'Macros', value: 'macros', }, ] }> + + ```yml version: 2 @@ -29,7 +31,6 @@ models: docs: show: true | false node_color: "black" - ``` @@ -53,9 +54,7 @@ seeds: - name: seed_name docs: show: true | false - ``` - @@ -71,9 +70,7 @@ snapshots: - name: snapshot_name docs: show: true | false - ``` - @@ -90,7 +87,6 @@ analyses: docs: show: true | false ``` - @@ -110,9 +106,7 @@ macros: - name: macro_name docs: show: true | false - ``` - Also refer to [macro properties](/reference/macro-properties). diff --git a/website/docs/reference/resource-configs/store_failures.md b/website/docs/reference/resource-configs/store_failures.md index 6c71cdb9296..2c596d1cf3e 100644 --- a/website/docs/reference/resource-configs/store_failures.md +++ b/website/docs/reference/resource-configs/store_failures.md @@ -10,7 +10,7 @@ Optionally set a test to always or never store its failures in the database. - If specified as `true` or `false`, the `store_failures` config will take precedence over the presence or absence of the `--store-failures` flag. - If the `store_failures` config is `none` or omitted, the resource will use the value of the `--store-failures` flag. -- When true, `store_failures` save all the record(s) that failed the test only if [limit](/reference/resource-configs/limit) is not set or if there are fewer records than the limit. `store_failures` are saved in a new table with the name of the test. By default, `store_failures` use a schema named `dbt_test__audit`, but, you can configure the schema to a different value. +- When true, `store_failures` save all the record(s) that failed the test only if [limit](/reference/resource-configs/limit) is not set or if there are fewer records than the limit. `store_failures` are saved in a new table with the name of the test. By default, `store_failures` use a schema named `dbt_test__audit`, but, you can [configure](/reference/resource-configs/schema#tests) the schema to a different value. This logic is encoded in the [`should_store_failures()`](https://github.com/dbt-labs/dbt-core/blob/98c015b7754779793e44e056905614296c6e4527/core/dbt/include/global_project/macros/materializations/helpers.sql#L77) macro. diff --git a/website/docs/reference/source-properties.md b/website/docs/reference/source-properties.md index d20ef5f2877..1fc7a4aef55 100644 --- a/website/docs/reference/source-properties.md +++ b/website/docs/reference/source-properties.md @@ -1,5 +1,5 @@ --- -title: "About source properties" +title: "Source properties" description: "Learn how to use source properties in dbt." ---