Skip to content

Commit

Permalink
Project flags, for global config defaults + behavior change migrations (
Browse files Browse the repository at this point in the history
#4814)

resolves #4366
resolves #4709
resolves #4787

## What are you changing in this pull request and why?

- Reframe "global configs" → "flags"
- Settable in `dbt_project.yml` instead of `profiles.yml`
- Subcategory of flags for legacy behaviors / behavior change migrations

<details>
<summary>

### Previews
</summary>

Most important changes:
-
[website/docs/reference/global-configs/about-global-configs.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/global-configs/about-global-configs)
-
[website/docs/reference/global-configs/project-flags.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/global-configs/project-flags)
-
[website/docs/reference/dbt_project.yml.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/dbt_project.yml)
-
[website/docs/reference/global-configs/legacy-behaviors.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/global-configs/legacy-behaviors)
-
[website/docs/docs/core/connect-data-platform/profiles.yml.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/docs/core/connect-data-platform/profiles.yml)

Other new pages:
-
[website/docs/reference/global-configs/record-timing-info.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/global-configs/record-timing-info)
-
[website/docs/reference/global-configs/indirect-selection.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/global-configs/indirect-selection)

Other significant edits:
-
[website/docs/reference/global-configs/logs.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/global-configs/logs)
-
[website/docs/reference/global-configs/parsing.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/global-configs/parsing)
-
[website/docs/reference/global-configs/version-compatibility.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/global-configs/version-compatibility)
-
[website/docs/reference/global-configs/json-artifacts.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/global-configs/json-artifacts)
- website/sidebars.js
- website/vercel.json
- website/dbt-versions.js

Edited links:
-
[website/docs/docs/dbt-versions/core-upgrade/08-upgrading-to-v1.2.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/docs/dbt-versions/core-upgrade/upgrading-to-v1.2)
-
[website/docs/docs/dbt-versions/core-upgrade/10-upgrading-to-v1.0.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/docs/dbt-versions/core-upgrade/upgrading-to-v1.0)
-
[website/docs/docs/dbt-cloud-apis/sl-manifest.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/docs/dbt-cloud-apis/sl-manifest)
-
[website/docs/reference/artifacts/dbt-artifacts.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/artifacts/dbt-artifacts)
-
[website/docs/reference/events-logging.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/events-logging)
-
[website/docs/reference/parsing.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/parsing)
-
[website/docs/reference/project-configs/clean-targets.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/project-configs/clean-targets)
-
[website/docs/reference/resource-configs/severity.md](https://docs-getdbt-com-git-jerco-flags-behavior-changes-dbt-labs.vercel.app/reference/resource-configs/severity)

Deleted:
- website/docs/reference/global-cli-flags.md
- website/docs/reference/global-configs/yaml-configurations.md
- website/docs/reference/project-configs/log-path.md
- website/docs/reference/project-configs/target-path.md

Minor edits (moved and re-titled):
- website/docs/reference/global-configs/command-line-options.md

Other:
- .github/pull_request_template.md

</details>

## Checklist

Adding or removing pages (delete if not applicable):
- [x] Add/remove page in `website/sidebars.js`
- [x] Provide a unique filename for new pages
- [x] Add an entry for deleted pages in `website/vercel.json`
- [x] Run link testing locally with `npm run build` to update the links
that point to deleted pages
  • Loading branch information
matthewshaver authored Feb 27, 2024
2 parents 27747f7 + 40066f8 commit 8e2e363
Show file tree
Hide file tree
Showing 28 changed files with 381 additions and 259 deletions.
2 changes: 1 addition & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,5 +19,5 @@ Uncomment when publishing docs for a prerelease version of dbt:
Adding or removing pages (delete if not applicable):
- [ ] Add/remove page in `website/sidebars.js`
- [ ] Provide a unique filename for new pages
- [ ] Add an entry for deleted pages in `website/static/_redirects`
- [ ] Add an entry for deleted pages in `website/vercel.json`
- [ ] Run link testing locally with `npm run build` to update the links that point to deleted pages
4 changes: 4 additions & 0 deletions website/dbt-versions.js
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,10 @@ exports.versions = [
]

exports.versionedPages = [
{
"page": "reference/global-configs/indirect-selection",
"firstVersion": "1.8",
},
{
"page": "reference/resource-configs/store_failures_as",
"firstVersion": "1.7",
Expand Down
42 changes: 41 additions & 1 deletion website/docs/docs/core/connect-data-platform/profiles.yml.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,14 @@ If you're using dbt Cloud, you can [connect to your data platform](/docs/cloud/c

This section identifies the parts of your `profiles.yml` that aren't specific to a particular data platform. For specific connection details, refer to the relevant page for your data platform.

<VersionBlock lastVersion="1.7">

:::warning Global configs

Starting in dbt v1.8, global configs have been deprecated from the `profiles.yml` file and should be configured in the [`dbt_project.yml`](/reference/dbt_project.yml) file instead.

:::

<File name='profiles.yml'>

```yml
Expand All @@ -28,6 +36,7 @@ This section identifies the parts of your `profiles.yml` that aren't specific to
[debug](/reference/global-configs/logs#log-level): <true | false>
[version_check](/reference/global-configs/version-compatibility): <true | false>
[fail_fast](/reference/global-configs/failing-fast): <true | false>
[indirect_selection](/reference/global-configs/indirect-selection): <eager | cautious | buildable | empty>
[use_experimental_parser](/reference/global-configs/parsing): <true | false>
[static_parser](/reference/global-configs/parsing): <true | false>
[cache_selected_only](/reference/global-configs/cache): <true | false>
Expand All @@ -54,6 +63,37 @@ This section identifies the parts of your `profiles.yml` that aren't specific to
</File>
</VersionBlock>
<VersionBlock firstVersion="1.8">
<File name='profiles.yml'>
```yml

<profile-name>:
target: <target-name> # this is the default target
outputs:
<target-name>:
type: <bigquery | postgres | redshift | snowflake | other>
schema: <schema_identifier>
threads: <natural_number>

### database-specific connection details
...

<target-name>: # additional targets
...

<profile-name>: # additional profiles
...

```
</File>
</VersionBlock>
## User config
You can set default values of global configs for all projects that you run using your local machine. See the docs on [global configs](/reference/global-configs/about-global-configs) for details.
You can set default values of global configs for all projects that you run using your local machine. Refer to [About global configs](/reference/global-configs/about-global-configs) for details.
2 changes: 1 addition & 1 deletion website/docs/docs/dbt-cloud-apis/sl-manifest.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ dbt creates an [artifact](/reference/artifacts/dbt-artifacts) file called the _S

By using the semantic manifest produced by dbt Core, MetricFlow will instantiate a data flow plan and generate SQL from Semantic Layer query requests. It's a valuable reference that you can use to understand the structure and details of your data models.

Similar to the [`manifest.json` file](/reference/artifacts/manifest-json), the `semantic_manifest.json` file also lives in the [target directory](/reference/project-configs/target-path) of your dbt project where dbt stores various artifacts (such as compiled models and tests) generated during the execution of your project.
Similar to the [`manifest.json` file](/reference/artifacts/manifest-json), the `semantic_manifest.json` file also lives in the [target directory](/reference/global-configs/json-artifacts) of your dbt project where dbt stores various artifacts (such as compiled models and tests) generated during the execution of your project.

## How it's produced

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ See GitHub discussion [dbt-labs/dbt-core#5468](https://github.com/dbt-labs/dbt-c
- **[Metrics](/docs/build/build-metrics-intro)** now support an `expression` type (metrics-on-metrics), as well as a `metric()` function to use when referencing metrics from within models, macros, or `expression`-type metrics. For more information on how to use expression metrics, check out the [**`dbt_metrics` package**](https://github.com/dbt-labs/dbt_metrics)
- **[dbt-Jinja functions](/reference/dbt-jinja-functions)** now include the [`itertools` Python module](/reference/dbt-jinja-functions/modules#itertools), as well as the [set](/reference/dbt-jinja-functions/set) and [zip](/reference/dbt-jinja-functions/zip) functions.
- **[Node selection](/reference/node-selection/syntax)** includes a [file selection method](/reference/node-selection/methods#the-file-method) (`-s model.sql`), and [yaml selector](/reference/node-selection/yaml-selectors) inheritance.
- **[Global configs](/reference/global-configs/about-global-configs)** now include CLI flag and environment variable settings for [`target-path`](/reference/project-configs/target-path) and [`log-path`](/reference/project-configs/log-path), which can be used to override the values set in `dbt_project.yml`
- **[Global configs](/reference/global-configs/about-global-configs)** now include CLI flag and environment variable settings for [`target-path`](/reference/global-configs/json-artifacts) and [`log-path`](/reference/global-configs/logs), which can be used to override the values set in `dbt_project.yml`

### Specific adapters

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ Several under-the-hood changes from past minor versions, tagged with deprecation
- Add [metrics](/docs/build/build-metrics-intro), a new node type
- [Generic tests](/best-practices/writing-custom-generic-tests) can be defined in `tests/generic` (new), in addition to `macros/` (as before)
- [Parsing](/reference/parsing): partial parsing and static parsing have been turned on by default.
- [Global configs](/reference/global-configs/about-global-configs) have been standardized. Related updates to [global CLI flags](/reference/global-cli-flags) and [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml).
- [Global configs](/reference/global-configs/about-global-configs) have been standardized. Related updates to [global CLI flags](/reference/global-configs/about-global-configs) and [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml).
- [The `init` command](/reference/commands/init) has a whole new look and feel. It's no longer just for first-time users.
- Add `result:<status>` subselectors for smarter reruns when dbt models have errors and tests fail. See examples: [Pro-tips for Workflows](/best-practices/best-practice-workflows#pro-tips-for-workflows)
- Secret-prefixed [env vars](/reference/dbt-jinja-functions/env_var) are now allowed only in `profiles.yml` + `packages.yml`
2 changes: 1 addition & 1 deletion website/docs/reference/artifacts/dbt-artifacts.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Most dbt commands (and corresponding RPC methods) produce artifacts:

## Where are artifacts produced?

By default, artifacts are written to the `/target` directory of your dbt project. You can configure the location using the [`target-path`](/reference/project-configs/target-path#configuration).
By default, artifacts are written to the `/target` directory of your dbt project. You can configure the location using the [`target-path` flag](/reference/global-configs/json-artifacts).

## Common metadata

Expand Down
15 changes: 9 additions & 6 deletions website/docs/reference/dbt_project.yml.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ The following example is a list of all available configurations in the `dbt_proj
[docs-paths](/reference/project-configs/docs-paths): [directorypath]
[asset-paths](/reference/project-configs/asset-paths): [directorypath]

[target-path](/reference/project-configs/target-path): directorypath
[log-path](/reference/project-configs/log-path): directorypath
[target-path](/reference/global-configs/json-artifacts): directorypath
[log-path](/reference/global-configs/logs): directorypath
[packages-install-path](/reference/project-configs/packages-install-path): directorypath

[clean-targets](/reference/project-configs/clean-targets): [directorypath]
Expand All @@ -52,6 +52,9 @@ The following example is a list of all available configurations in the `dbt_proj

[require-dbt-version](/reference/project-configs/require-dbt-version): version-range | [version-range]

[flags](/reference/global-configs/project-flags):
[<global-configs>](/reference/global-configs/project-flags)

[dbt-cloud](/docs/cloud/cloud-cli-installation):
[project-id](/docs/cloud/configure-cloud-cli#configure-the-dbt-cloud-cli): project_id # Required
[defer-env-id](/docs/cloud/about-cloud-develop-defer#defer-in-dbt-cloud-cli): environment_id # Optional
Expand Down Expand Up @@ -120,8 +123,8 @@ vars:
[docs-paths](/reference/project-configs/docs-paths): [directorypath]
[asset-paths](/reference/project-configs/asset-paths): [directorypath]

[target-path](/reference/project-configs/target-path): directorypath
[log-path](/reference/project-configs/log-path): directorypath
[target-path](/reference/global-configs/json-artifacts): directorypath
[log-path](/reference/global-configs/logs): directorypath
[packages-install-path](/reference/project-configs/packages-install-path): directorypath

[clean-targets](/reference/project-configs/clean-targets): [directorypath]
Expand Down Expand Up @@ -193,8 +196,8 @@ vars:
[docs-paths](/reference/project-configs/docs-paths): [directorypath]
[asset-paths](/reference/project-configs/asset-paths): [directorypath]

[target-path](/reference/project-configs/target-path): directorypath
[log-path](/reference/project-configs/log-path): directorypath
[target-path](/reference/global-configs/json-artifacts): directorypath
[log-path](/reference/global-configs/logs): directorypath
[packages-install-path](/reference/project-configs/packages-install-path): directorypath

[clean-targets](/reference/project-configs/clean-targets): [directorypath]
Expand Down
2 changes: 1 addition & 1 deletion website/docs/reference/events-logging.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ title: "Events and logs"

As dbt runs, it generates events. The most common way to see those events is as log messages, written in real time to two places:
- The command line terminal (`stdout`), to provide interactive feedback while running dbt.
- The debug log file (`logs/dbt.log`), to enable detailed [debugging of errors](/guides/debug-errors) when they occur. The text-formatted log messages in this file include all `DEBUG`-level events, as well as contextual information, such as log level and thread name. The location of this file can be configured via [the `log_path` config](/reference/project-configs/log-path).
- The debug log file (`logs/dbt.log`), to enable detailed [debugging of errors](/guides/debug-errors) when they occur. The text-formatted log messages in this file include all `DEBUG`-level events, as well as contextual information, such as log level and thread name. The location of this file can be configured via [the `log-path` flag](/reference/global-configs/logs).

<File name='CLI'>

Expand Down
26 changes: 0 additions & 26 deletions website/docs/reference/global-cli-flags.md

This file was deleted.

Loading

0 comments on commit 8e2e363

Please sign in to comment.