diff --git a/website/docs/docs/build/incremental-microbatch.md b/website/docs/docs/build/incremental-microbatch.md index e1c39e6ae47..46dfa795f1d 100644 --- a/website/docs/docs/build/incremental-microbatch.md +++ b/website/docs/docs/build/incremental-microbatch.md @@ -20,7 +20,7 @@ Refer to [Supported incremental strategies by adapter](/docs/build/incremental-s Incremental models in dbt are a [materialization](/docs/build/materializations) designed to efficiently update your data warehouse tables by only transforming and loading _new or changed data_ since the last run. Instead of reprocessing an entire dataset every time, incremental models process a smaller number of rows, and then append, update, or replace those rows in the existing table. This can significantly reduce the time and resources required for your data transformations. -Microbatch incremental models make it possible to process transformations on very large time-series datasets with efficiency and resiliency. When dbt runs a microbatch model — whether for the first time, during incremental runs, or in specified backfills — it will split the processing into multiple queries (or "batches"), based on the `event_time` and `batch_size` you configure. +Microbatch incremental models make it possible to process transformations on very large time-series datasets with efficiency and resiliency. When dbt runs a microbatch model — whether for the first time, during incremental runs, or in specified backfills — it will split the processing into multiple queries (or "batches"), based on the [`event_time`](/reference/resource-configs/event-time) and `batch_size` you configure. Each "batch" corresponds to a single bounded time period (by default, a single day of data). Where other incremental strategies operate only on "old" and "new" data, microbatch models treat every batch as an atomic unit that can be built or replaced on its own. Each batch is independent and . This is a powerful abstraction that makes it possible for dbt to run batches separately — in the future, concurrently — and to retry them independently. @@ -48,7 +48,7 @@ We run the `sessions` model on October 1, 2024, and then again on October 2. It -The `event_time` for the `sessions` model is set to `session_start`, which marks the beginning of a user’s session on the website. This setting allows dbt to combine multiple page views (each tracked by their own `page_view_start` timestamps) into a single session. This way, `session_start` differentiates the timing of individual page views from the broader timeframe of the entire user session. +The [`event_time`](/reference/resource-configs/event-time) for the `sessions` model is set to `session_start`, which marks the beginning of a user’s session on the website. This setting allows dbt to combine multiple page views (each tracked by their own `page_view_start` timestamps) into a single session. This way, `session_start` differentiates the timing of individual page views from the broader timeframe of the entire user session. @@ -162,7 +162,7 @@ Several configurations are relevant to microbatch models, and some are required: | Config | Type | Description | Default | |----------|------|---------------|---------| -| `event_time` | Column (required) | The column indicating "at what time did the row occur." Required for your microbatch model and any direct parents that should be filtered. | N/A | +| [`event_time`](/reference/resource-configs/event-time) | Column (required) | The column indicating "at what time did the row occur." Required for your microbatch model and any direct parents that should be filtered. | N/A | | `begin` | Date (required) | The "beginning of time" for the microbatch model. This is the starting point for any initial or full-refresh builds. For example, a daily-grain microbatch model run on `2024-10-01` with `begin = '2023-10-01` will process 366 batches (it's a leap year!) plus the batch for "today." | N/A | | `batch_size` | String (required) | The granularity of your batches. Supported values are `hour`, `day`, `month`, and `year` | N/A | | `lookback` | Integer (optional) | Process X batches prior to the latest bookmark to capture late-arriving records. | `1` | diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index 6759a026e02..9fbfa139e47 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -19,6 +19,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo \* The official release date for this new format of release notes is May 15th, 2024. Historical release notes for prior dates may not reflect all available features released earlier this year or their tenancy availability. ## November 2024 +- **New**: Use the `event_time` configuration to specify "at what time did the row occur." This configuration is required for [Incremental microbatch](/docs/build/incremental-microbatch) and can be added to ensure you're comparing overlapping times in [Advanced CI's compare changes](/docs/deploy/advanced-ci). Available in dbt Cloud Versionless and dbt Core v1.9 and higher. Refer to [event_time](/reference/resource-configs/event-time) for more information. - **Fix**: This update improves [dbt Semantic Layer Tableau integration](/docs/cloud-integrations/semantic-layer/tableau) making query parsing more reliable. Some key fixes include: - Error messages for unsupported joins between saved queries and ALL tables. - Improved handling of queries when multiple tables are selected in a data source. @@ -27,6 +28,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo - **Enhancement**: The dbt Semantic Layer supports creating new credentials for users who don't have permissions to create service tokens. In the **Credentials & service tokens** side panel, the **+Add Service Token** option is unavailable for those users who don't have permission. Instead, the side panel displays a message indicating that the user doesn't have permission to create a service token and should contact their administration. Refer to [Set up dbt Semantic Layer](/docs/use-dbt-semantic-layer/setup-sl) for more details. ## October 2024 + Documentation for new features and functionality announced at Coalesce 2024: diff --git a/website/docs/docs/deploy/advanced-ci.md b/website/docs/docs/deploy/advanced-ci.md index 7a36688402b..b5caad33e81 100644 --- a/website/docs/docs/deploy/advanced-ci.md +++ b/website/docs/docs/deploy/advanced-ci.md @@ -36,6 +36,16 @@ dbt reports the comparison differences in: +### Optimizing comparisons + +When an [`event_time`](/reference/resource-configs/event-time) column is specified on your model, compare changes can optimize comparisons by using only the overlapping timeframe (meaning the timeframe exists in both the CI and production environment), helping you avoid incorrect row-count changes and return results faster. + +This is useful in scenarios like: +- **Subset of data in CI** — When CI builds only a [subset of data](/best-practices/best-practice-workflows#limit-the-data-processed-when-in-development) (like the most recent 7 days), compare changes would interpret the excluded data as "deleted rows." Configuring `event_time` allows you to avoid this issue by limiting comparisons to the overlapping timeframe, preventing false alerts about data deletions that are just filtered out in CI. +- **Fresher data in CI than in production** — When your CI job includes fresher data than production (because it has run more recently), compare changes would flag the additional rows as "new" data, even though they’re just fresher data in CI. With `event_time` configured, the comparison only includes the shared timeframe and correctly reflects actual changes in the data. + + + ## About the cached data After [comparing changes](#compare-changes), dbt Cloud stores a cache of no more than 100 records for each modified model for preview purposes. By caching this data, you can view the examples of changed data without rerunning the comparison against the data warehouse every time (optimizing for lower compute costs). To display the changes, dbt Cloud uses a cached version of a sample of the data records. These data records are queried from the database using the connection configuration (such as user, role, service account, and so on) that's set in the CI job's environment. diff --git a/website/docs/reference/model-configs.md b/website/docs/reference/model-configs.md index 65133dcb25a..9508cf68ceb 100644 --- a/website/docs/reference/model-configs.md +++ b/website/docs/reference/model-configs.md @@ -104,6 +104,8 @@ models: + + ```yaml models: [](/reference/resource-configs/resource-path): @@ -121,7 +123,29 @@ models: [+](/reference/resource-configs/plus-prefix)[contract](/reference/resource-configs/contract): {} ``` + + + + +```yaml +models: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false + [+](/reference/resource-configs/plus-prefix)[tags](/reference/resource-configs/tags): | [] + [+](/reference/resource-configs/plus-prefix)[pre-hook](/reference/resource-configs/pre-hook-post-hook): | [] + [+](/reference/resource-configs/plus-prefix)[post-hook](/reference/resource-configs/pre-hook-post-hook): | [] + [+](/reference/resource-configs/plus-prefix)[database](/reference/resource-configs/database): + [+](/reference/resource-configs/plus-prefix)[schema](/reference/resource-properties/schema): + [+](/reference/resource-configs/plus-prefix)[alias](/reference/resource-configs/alias): + [+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): + [+](/reference/resource-configs/plus-prefix)[full_refresh](/reference/resource-configs/full_refresh): + [+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta): {} + [+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {} + [+](/reference/resource-configs/plus-prefix)[contract](/reference/resource-configs/contract): {} + [+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field +``` + @@ -131,6 +155,8 @@ models: + + ```yaml version: 2 @@ -150,17 +176,63 @@ models: [grants](/reference/resource-configs/grants): {} [contract](/reference/resource-configs/contract): {} ``` + - + - +```yaml +version: 2 +models: + - name: [] + config: + [enabled](/reference/resource-configs/enabled): true | false + [tags](/reference/resource-configs/tags): | [] + [pre_hook](/reference/resource-configs/pre-hook-post-hook): | [] + [post_hook](/reference/resource-configs/pre-hook-post-hook): | [] + [database](/reference/resource-configs/database): + [schema](/reference/resource-properties/schema): + [alias](/reference/resource-configs/alias): + [persist_docs](/reference/resource-configs/persist_docs): + [full_refresh](/reference/resource-configs/full_refresh): + [meta](/reference/resource-configs/meta): {} + [grants](/reference/resource-configs/grants): {} + [contract](/reference/resource-configs/contract): {} + [event_time](/reference/resource-configs/event-time): my_time_field +``` + + + + + + +```jinja + +{{ config( + [enabled](/reference/resource-configs/enabled)=true | false, + [tags](/reference/resource-configs/tags)="" | [""], + [pre_hook](/reference/resource-configs/pre-hook-post-hook)="" | [""], + [post_hook](/reference/resource-configs/pre-hook-post-hook)="" | [""], + [database](/reference/resource-configs/database)="", + [schema](/reference/resource-properties/schema)="", + [alias](/reference/resource-configs/alias)="", + [persist_docs](/reference/resource-configs/persist_docs)={}, + [meta](/reference/resource-configs/meta)={}, + [grants](/reference/resource-configs/grants)={}, + [contract](/reference/resource-configs/contract)={} +) }} + +``` + + + + ```jinja {{ config( @@ -175,9 +247,11 @@ models: [meta](/reference/resource-configs/meta)={}, [grants](/reference/resource-configs/grants)={}, [contract](/reference/resource-configs/contract)={} + [event_time](/reference/resource-configs/event-time): my_time_field ) }} ``` + diff --git a/website/docs/reference/resource-configs/event-time.md b/website/docs/reference/resource-configs/event-time.md new file mode 100644 index 00000000000..d8c0c0e0472 --- /dev/null +++ b/website/docs/reference/resource-configs/event-time.md @@ -0,0 +1,284 @@ +--- +title: "event_time" +id: "event-time" +sidebar_label: "event_time" +resource_types: [models, seeds, source] +description: "dbt uses event_time to understand when an event occurred. When defined, event_time enables microbatch incremental models and more refined comparison of datasets during Advanced CI." +datatype: string +--- + +Available in dbt Cloud Versionless and dbt Core v1.9 and higher. + + + + + + +```yml +models: + [resource-path:](/reference/resource-configs/resource-path) + +event_time: my_time_field +``` + + + + + +```yml +models: + - name: model_name + [config](/reference/resource-properties/config): + event_time: my_time_field +``` + + + + +```sql +{{ config( + event_time='my_time_field' +) }} +``` + + + + + + + + + +```yml +seeds: + [resource-path:](/reference/resource-configs/resource-path) + +event_time: my_time_field +``` + + + + +```yml +seeds: + - name: seed_name + [config](/reference/resource-properties/config): + event_time: my_time_field +``` + + + + + + + + +```yml +snapshots: + [resource-path:](/reference/resource-configs/resource-path) + +event_time: my_time_field +``` + + + + + +```yml +snapshots: + - name: snapshot_name + [config](/reference/resource-properties/config): + event_time: my_time_field +``` + + + + + + + +```sql + +{{ config( + event_time: 'my_time_field' +) }} +``` + + + + +import SnapshotYaml from '/snippets/_snapshot-yaml-spec.md'; + + + + + + + + + + + + +```yml +sources: + [resource-path:](/reference/resource-configs/resource-path) + +event_time: my_time_field +``` + + + + +```yml +sources: + - name: source_name + [config](/reference/resource-properties/config): + event_time: my_time_field +``` + + + + + +## Definition + +Set the `event_time` to the name of the field that represents the timestamp of the event -- "at what time did the row occur" -- as opposed to an event ingestion date. You can configure `event_time` for a [model](/docs/build/models), [seed](/docs/build/seeds), or [source](/docs/build/sources) in your `dbt_project.yml` file, property YAML file, or config block. + +Here are some examples of good and bad `event_time` columns: + +- ✅ Good: + - `account_created_at` — This represents the specific time when an account was created, making it a fixed event in time. + - `session_began_at` — This captures the exact timestamp when a user session started, which won’t change and directly ties to the event. + +- ❌ Bad: + + - `_fivetran_synced` — This isn't the time that the event happened, it's the time that the event was ingested. + - `last_updated_at` — This isn't a good use case as this will keep changing over time. + +`event_time` is required for [Incremental microbatch](/docs/build/incremental-microbatch) and highly recommended for [Advanced CI's compare changes](/docs/deploy/advanced-ci#optimizing-comparisons) in CI/CD workflows, where it ensures the same time-slice of data is correctly compared between your CI and production environments. + +## Examples + + + + + +Here's an example in the `dbt_project.yml` file: + + + +```yml +models: + my_project: + user_sessions: + +event_time: session_start_time +``` + + +Example in a properties YAML file: + + + +```yml +models: + - name: user_sessions + config: + event_time: session_start_time +``` + + + +Example in sql model config block: + + + +```sql +{{ config( + event_time='session_start_time' +) }} +``` + + + +This setup sets `session_start_time` as the `event_time` for the `user_sessions` model. + + + + +Here's an example in the `dbt_project.yml` file: + + + +```yml +seeds: + my_project: + my_seed: + +event_time: record_timestamp +``` + + + +Example in a seed properties YAML: + + + +```yml +seeds: + - name: my_seed + config: + event_time: record_timestamp +``` + + +This setup sets `record_timestamp` as the `event_time` for `my_seed`. + + + + + +Here's an example in the `dbt_project.yml` file: + + + +```yml +snapshots: + my_project: + my_snapshot: + +event_time: record_timestamp +``` + + + +Example in a snapshot properties YAML: + + + +```yml +snapshots: + - name: my_snapshot + config: + event_time: record_timestamp +``` + + +This setup sets `record_timestamp` as the `event_time` for `my_snapshot`. + + + + + +Here's an example of source properties YAML file: + + + +```yml +sources: + - name: source_name + tables: + - name: table_name + config: + event_time: event_timestamp +``` + + +This setup sets `event_timestamp` as the `event_time` for the specified source table. + + + diff --git a/website/docs/reference/seed-configs.md b/website/docs/reference/seed-configs.md index 5d5c39071d6..a18f1fc28f7 100644 --- a/website/docs/reference/seed-configs.md +++ b/website/docs/reference/seed-configs.md @@ -79,6 +79,8 @@ seeds: + + ```yaml seeds: [](/reference/resource-configs/resource-path): @@ -95,7 +97,28 @@ seeds: [+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {} ``` + + + + +```yaml +seeds: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false + [+](/reference/resource-configs/plus-prefix)[tags](/reference/resource-configs/tags): | [] + [+](/reference/resource-configs/plus-prefix)[pre-hook](/reference/resource-configs/pre-hook-post-hook): | [] + [+](/reference/resource-configs/plus-prefix)[post-hook](/reference/resource-configs/pre-hook-post-hook): | [] + [+](/reference/resource-configs/plus-prefix)[database](/reference/resource-configs/database): + [+](/reference/resource-configs/plus-prefix)[schema](/reference/resource-properties/schema): + [+](/reference/resource-configs/plus-prefix)[alias](/reference/resource-configs/alias): + [+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): + [+](/reference/resource-configs/plus-prefix)[full_refresh](/reference/resource-configs/full_refresh): + [+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta): {} + [+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {} + [+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field +``` + @@ -105,6 +128,8 @@ seeds: + + ```yaml version: 2 @@ -122,13 +147,36 @@ seeds: [full_refresh](/reference/resource-configs/full_refresh): [meta](/reference/resource-configs/meta): {} [grants](/reference/resource-configs/grants): {} + [event_time](/reference/resource-configs/event-time): my_time_field + +``` + + + + +```yaml +version: 2 +seeds: + - name: [] + config: + [enabled](/reference/resource-configs/enabled): true | false + [tags](/reference/resource-configs/tags): | [] + [pre_hook](/reference/resource-configs/pre-hook-post-hook): | [] + [post_hook](/reference/resource-configs/pre-hook-post-hook): | [] + [database](/reference/resource-configs/database): + [schema](/reference/resource-properties/schema): + [alias](/reference/resource-configs/alias): + [persist_docs](/reference/resource-configs/persist_docs): + [full_refresh](/reference/resource-configs/full_refresh): + [meta](/reference/resource-configs/meta): {} + [grants](/reference/resource-configs/grants): {} ``` + - ## Configuring seeds diff --git a/website/docs/reference/snapshot-configs.md b/website/docs/reference/snapshot-configs.md index 2054caf725b..7b3c0f8e5b1 100644 --- a/website/docs/reference/snapshot-configs.md +++ b/website/docs/reference/snapshot-configs.md @@ -168,6 +168,24 @@ Configurations can be applied to snapshots using the [YAML syntax](/docs/build/s + + +```yaml +snapshots: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false + [+](/reference/resource-configs/plus-prefix)[tags](/reference/resource-configs/tags): | [] + [+](/reference/resource-configs/plus-prefix)[alias](/reference/resource-configs/alias): + [+](/reference/resource-configs/plus-prefix)[pre-hook](/reference/resource-configs/pre-hook-post-hook): | [] + [+](/reference/resource-configs/plus-prefix)[post-hook](/reference/resource-configs/pre-hook-post-hook): | [] + [+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): {} + [+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {} + [+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field +``` + + + + ```yaml snapshots: [](/reference/resource-configs/resource-path): @@ -179,6 +197,7 @@ snapshots: [+](/reference/resource-configs/plus-prefix)[persist_docs](/reference/resource-configs/persist_docs): {} [+](/reference/resource-configs/plus-prefix)[grants](/reference/resource-configs/grants): {} ``` + @@ -225,6 +244,7 @@ snapshots: [post_hook](/reference/resource-configs/pre-hook-post-hook): | [] [persist_docs](/reference/resource-configs/persist_docs): {} [grants](/reference/resource-configs/grants): {} + [event_time](/reference/resource-configs/event-time): my_time_field ``` @@ -292,7 +312,6 @@ The following examples demonstrate how to configure snapshots using the `dbt_pro ```yml - snapshots: +unique_key: id ``` @@ -307,7 +326,6 @@ The following examples demonstrate how to configure snapshots using the `dbt_pro ```yml - snapshots: jaffle_shop: +unique_key: id diff --git a/website/docs/reference/source-configs.md b/website/docs/reference/source-configs.md index 64dda8bffde..c5264e82fc7 100644 --- a/website/docs/reference/source-configs.md +++ b/website/docs/reference/source-configs.md @@ -8,7 +8,17 @@ import ConfigGeneral from '/snippets/_config-description-general.md'; ## Available configurations -Sources only support one configuration, [`enabled`](/reference/resource-configs/enabled). + + +Sources supports [`enabled`](/reference/resource-configs/enabled) and [`meta`](/reference/resource-configs/meta). + + + + + +Sources configurations support [`enabled`](/reference/resource-configs/enabled), [`event_time`](/reference/resource-configs/event-time), and [`meta`](/reference/resource-configs/meta) + + ### General configurations @@ -27,12 +37,29 @@ Sources only support one configuration, [`enabled`](/reference/resource-configs/ + + ```yaml sources: [](/reference/resource-configs/resource-path): [+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false + [+](/reference/resource-configs/plus-prefix)[event_time](/reference/resource-configs/event-time): my_time_field + [+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta): + key: value ``` + + + + +```yaml +sources: + [](/reference/resource-configs/resource-path): + [+](/reference/resource-configs/plus-prefix)[enabled](/reference/resource-configs/enabled): true | false + [+](/reference/resource-configs/plus-prefix)[meta](/reference/resource-configs/meta): + key: value +``` + @@ -43,6 +70,8 @@ sources: + + ```yaml version: 2 @@ -50,12 +79,37 @@ sources: - name: [] [config](/reference/resource-properties/config): [enabled](/reference/resource-configs/enabled): true | false + [event_time](/reference/resource-configs/event-time): my_time_field + [meta](/reference/resource-configs/meta): {} + tables: - name: [] [config](/reference/resource-properties/config): [enabled](/reference/resource-configs/enabled): true | false + [event_time](/reference/resource-configs/event-time): my_time_field + [meta](/reference/resource-configs/meta): {} ``` + + + + +```yaml +version: 2 + +sources: + - name: [] + [config](/reference/resource-properties/config): + [enabled](/reference/resource-configs/enabled): true | false + [meta](/reference/resource-configs/meta): {} + tables: + - name: [] + [config](/reference/resource-properties/config): + [enabled](/reference/resource-configs/enabled): true | false + [meta](/reference/resource-configs/meta): {} + +``` + @@ -74,6 +128,8 @@ You can disable sources imported from a package to prevent them from rendering i + + ```yaml sources: your_project_name: @@ -81,11 +137,34 @@ You can disable sources imported from a package to prevent them from rendering i source_name: source_table_name: +enabled: false + +event_time: my_time_field ``` + + + + + ```yaml + sources: + your_project_name: + subdirectory_name: + source_name: + source_table_name: + +enabled: false + ``` + ### Examples + +The following examples show how to configure sources in your dbt project. + +— [Disable all sources imported from a package](#disable-all-sources-imported-from-a-package)
+— [Conditionally enable a single source](#conditionally-enable-a-single-source)
+— [Disable a single source from a package](#disable-a-single-source-from-a-package)
+— [Configure a source with an `event_time`](#configure-a-source-with-an-event_time)
+— [Configure meta to a source](#configure-meta-to-a-source)
+ #### Disable all sources imported from a package To apply a configuration to all sources included from a [package](/docs/build/packages), state your configuration under the [project name](/reference/project-configs/name.md) in the @@ -172,6 +251,53 @@ sources: +#### Configure a source with an `event_time` + + + +Configuring an [`event_time`](/reference/resource-configs/event-time) for a source is only available in dbt Cloud Versionless or dbt Core versions 1.9 and later. + + + + + +To configure a source with an `event_time`, specify the `event_time` field in the source configuration. This field is used to represent the actual timestamp of the event, rather than something like a loading date. + +For example, if you had a source table called `clickstream` in the `events` source, you can use the timestamp for each event in the `event_timestamp` column as follows: + + + +```yaml +sources: + events: + clickstream: + +event_time: event_timestamp +``` + + +In this example, the `event_time` is set to `event_timestamp`, which has the exact time each clickstream event happened. +Not only is this required for the [incremental microbatching strategy](/docs/build/incremental-microbatch), but when you compare data across [CI and production](/docs/deploy/advanced-ci#speeding-up-comparisons) environments, dbt will use `event_timestamp` to filter and match data by this event-based timeframe, ensuring that only overlapping timeframes are compared. + + + +#### Configure meta to a source + +Use the `meta` field to assign metadata information to sources. This is useful for tracking additional context, documentation, logging, and more. + +For example, you can add `meta` information to a `clickstream` source to include information about the data source system: + + + +```yaml +sources: + events: + clickstream: + +meta: + source_system: "Google analytics" + data_owner: "marketing_team" +``` + + ## Example source configuration The following is a valid source configuration for a project with: * `name: jaffle_shop` diff --git a/website/sidebars.js b/website/sidebars.js index b5134479837..4b3248403f2 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -926,6 +926,7 @@ const sidebarSettings = { "reference/resource-configs/alias", "reference/resource-configs/database", "reference/resource-configs/enabled", + "reference/resource-configs/event-time", "reference/resource-configs/full_refresh", "reference/resource-configs/contract", "reference/resource-configs/grants", diff --git a/website/static/img/docs/deploy/apples_to_apples.png b/website/static/img/docs/deploy/apples_to_apples.png new file mode 100644 index 00000000000..b1216b6eeb2 Binary files /dev/null and b/website/static/img/docs/deploy/apples_to_apples.png differ