Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clarify SL test/query language #3945

Merged
merged 21 commits into from
Aug 24, 2023
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
7b97f58
clarify test/query behavior for cloud/cli
mirnawong1 Aug 17, 2023
4e94588
fix spacing
mirnawong1 Aug 17, 2023
c3307ba
Merge branch 'current' into mwong-clarify-sl
mirnawong1 Aug 17, 2023
a916ea2
fix broken link
mirnawong1 Aug 17, 2023
ee82078
Merge branch 'mwong-clarify-sl' of https://github.com/dbt-labs/docs.g…
mirnawong1 Aug 17, 2023
e1d11b1
Merge branch 'current' into mwong-clarify-sl
mirnawong1 Aug 22, 2023
8e07ce9
Merge branch 'current' into mwong-clarify-sl
mirnawong1 Aug 22, 2023
61bb513
Merge branch 'current' into mwong-clarify-sl
mirnawong1 Aug 23, 2023
4fcd1dc
Update website/docs/docs/build/sl-getting-started.md
mirnawong1 Aug 24, 2023
5990360
Update website/snippets/_v2-sl-prerequisites.md
mirnawong1 Aug 24, 2023
5e975bb
Merge branch 'current' into mwong-clarify-sl
mirnawong1 Aug 24, 2023
7aae342
Update sidebars.js
mirnawong1 Aug 24, 2023
ce56bec
Update website/docs/docs/build/sl-getting-started.md
mirnawong1 Aug 24, 2023
52c480e
Update website/docs/docs/build/sl-getting-started.md
mirnawong1 Aug 24, 2023
2565b4d
Merge branch 'current' into mwong-clarify-sl
mirnawong1 Aug 24, 2023
0710886
Update website/docs/docs/build/sl-getting-started.md
mirnawong1 Aug 24, 2023
1862041
Update website/docs/docs/build/sl-getting-started.md
mirnawong1 Aug 24, 2023
702a9dd
Update website/docs/docs/build/sl-getting-started.md
mirnawong1 Aug 24, 2023
1d89f48
Update website/snippets/_sl-test-and-query-metrics.md
mirnawong1 Aug 24, 2023
a7d9734
Update website/snippets/_sl-test-and-query-metrics.md
mirnawong1 Aug 24, 2023
6ac27f3
Update website/snippets/_sl-test-and-query-metrics.md
mirnawong1 Aug 24, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 17 additions & 19 deletions website/docs/docs/build/sl-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,38 +14,34 @@ import DefineMetrics from '/snippets/_sl-define-metrics.md';
import ConfigMetric from '/snippets/_sl-configure-metricflow.md';
import TestQuery from '/snippets/_sl-test-and-query-metrics.md';

This getting started page presents a sample workflow to help you create your first metrics. It uses the [Jaffle shop example project](https://github.com/dbt-labs/jaffle-sl-template) as the project data source and is available for you to use. If you prefer, you can create semantic models and metrics for your own dbt project.
This getting started page presents a sample workflow to help you create your first metrics in dbt Cloud or the command-line interface (CLI). It uses the [Jaffle shop example project](https://github.com/dbt-labs/jaffle-sl-template) as the project data source and is available for you to use. If you prefer, you can create semantic models and metrics for your own dbt project.

To fully experience the power of a universal dbt Semantic Layer, take the following steps:
This guide will walk you through how to:

- [Create a semantic model](#create-a-semantic-model) with MetricFlow
- [Define metrics](#define-metrics) with MetricFlow
- [Test and query metrics locally](#test-and-query-metrics) with MetricFlow
- [Create a semantic model](#create-a-semantic-model) using MetricFlow
- [Define metrics](#define-metrics) using MetricFlow
- [Test and query metrics locally](#test-and-query-metrics) using MetricFlow
- [Run a production job](#run-a-production-job) in dbt Cloud
- [Set up dbt Semantic Layer](#set-up-dbt-semantic-layer) in dbt Cloud
- [Connect to and query the API](#connect-and-query-api) with dbt Cloud

To fully experience the power of a universal [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl), you'll need a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

## Prerequisites

- Have an understanding of key concepts in [MetricFlow](/docs/build/about-metricflow), which powers the revamped dbt Semantic Layer.
- Have both your production and development environments running dbt version 1.6 or higher. Refer to [upgrade in dbt Cloud](/docs/dbt-versions/upgrade-core-in-cloud) for more info.
- Use MetricFlow with Snowflake, BigQuery, Databricks, Postgres (CLI only), or Redshift. (dbt Cloud Postgres support coming soon)
- A successful run in the environment where your Semantic Layer is configured
- To query with dbt Cloud:
* Have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment, hosted in North America (Other regions coming soon)
* Set up the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and the [Semantic Layer API](/docs/dbt-cloud-apis/sl-api-overview) in the integrated tool to import metric definitions.
- To query locally:
* dbt Core or Developer accounts must manually install the [MetricFlow CLI](/docs/build/metricflow-cli) to test or query their metrics. After installing the package, make sure you run at least one model. <br />

- Use Snowflake, BigQuery, Databricks, Postgres (CLI only), or Redshift. (dbt Cloud Postgres support coming soon)
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved
- Create a successful run in the environment where you configure the Semantic Layer.
- **Note:** Semantic Layer currently supports the Deployment environment for querying. (_development querying experience coming soon_)
- Set up the [Semantic Layer API](/docs/dbt-cloud-apis/sl-api-overview) in the integrated tool to import metric definitions.
- **Note:** dbt Core or Developer accounts can only query data manually using the [MetricFlow CLI](/docs/build/metricflow-cli) and SQL. To dynamically query metrics using external tools, you must have a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account with access to the Semantic Layer API.<br />
- Understand [MetricFlow's](/docs/build/about-metricflow) key concepts, which powers the revamped dbt Semantic Layer.

:::tip
New to dbt or metrics? Try our [Jaffle shop example project](https://github.com/dbt-labs/jaffle-sl-template) to help you get started!
:::

## Install MetricFlow

<InstallMetricFlow />

## Create a semantic model

<CreateModel />
Expand All @@ -64,7 +60,7 @@ New to dbt or metrics? Try our [Jaffle shop example project](https://github.com/

## Run a production job

Before you begin, you must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment, hosted in North America.
Before you begin, you must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment, hosted in North America (cloud.getdbt.com login).
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

Once you’ve defined metrics in your dbt project, you can perform a job run in your dbt Cloud deployment environment to materialize your metrics. Only the deployment environment is supported for the dbt Semantic Layer at this moment.

Expand All @@ -82,7 +78,9 @@ import SlSetUp from '/snippets/_new-sl-setup.md';

## Connect and query API

You can query your metrics in a JDBC-enabled tool or use existing first-class integrations with the dbt Semantic Layer. In order to do so, you must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment, hosted in North America.
You can query your metrics in a JDBC-enabled tool or use existing first-class integrations with the dbt Semantic Layer.

Before you begin, you must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment, hosted in North America (cloud.getdbt.com login).

- <span>To learn how to use the JDBC API and what tools you can query it with, refer to the <a href="https://docs.getdbt.com/docs/dbt-cloud-apis/sl-api-overview" target="_self">{frontMatter.meta.api_name}</a></span>.<br />

Expand Down
16 changes: 7 additions & 9 deletions website/docs/docs/use-dbt-semantic-layer/quickstart-sl.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@ MetricFlow is a powerful component within the dbt Semantic Layer that helps user

Use this guide to fully experience the power of a universal dbt Semantic Layer. Here are the following steps you'll take:

- [Create a semantic model](#create-a-semantic-model) with MetricFlow
- [Define metrics](#define-metrics) with MetricFlow
- [Test and query metrics locally](#test-and-query-metrics) with MetricFlow
- [Create a semantic model](#create-a-semantic-model) in dbt Cloud using MetricFlow
- [Define metrics](#define-metrics) in dbt Cloud using MetricFlow
- [Test and query metrics locally](#test-and-query-metrics) using MetricFlow
- [Run a production job](#run-a-production-job) in dbt Cloud
- [Set up dbt Semantic Layer](#setup) in dbt Cloud
- [Connect and query API](#connect-and-query-api) with dbt Cloud
Expand All @@ -44,10 +44,6 @@ import SetUp from '/snippets/_v2-sl-prerequisites.md';
New to dbt or metrics? Try our [Jaffle shop example project](https://github.com/dbt-labs/jaffle-sl-template) to help you get started!
:::

## Install MetricFlow

<InstallMetricFlow />

## Create a semantic model

<CreateModel />
Expand All @@ -62,7 +58,7 @@ New to dbt or metrics? Try our [Jaffle shop example project](https://github.com/

## Run a production job

Once you’ve defined metrics in your dbt project, you can perform a job run in your deployment environment to materialize your metrics. The deployment environment is only supported for the dbt Semantic Layer at this moment.
Once you’ve defined metrics in your dbt project, you can perform a job run in your deployment environment in dbt Cloud to materialize your metrics. The deployment environment is only supported for the dbt Semantic Layer at this moment.

1. Go to **Deploy** in the navigation header
2. Select **Jobs** to re-run the job with the most recent code in the deployment environment.
Expand All @@ -88,7 +84,9 @@ import SlSetUp from '/snippets/_new-sl-setup.md';

## Connect and query API

You can query your metrics in a JDBC-enabled tool or use existing first-class integrations with the dbt Semantic Layer. In order to do so, you must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment, hosted in North America.
You can query your metrics in a JDBC-enabled tool or use existing first-class integrations with the dbt Semantic Layer.

Before you begin, you must have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment, hosted in North America (cloud.getdbt.com login).

- <span>To learn how to use the JDBC API and what tools you can query it with, refer to the <a href="https://docs.getdbt.com/docs/dbt-cloud-apis/sl-api-overview" target="_self">{frontMatter.meta.api_name}</a></span>.<br />

Expand Down
2 changes: 1 addition & 1 deletion website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -251,6 +251,7 @@ const sidebarSettings = {
link: { type: "doc", id: "docs/build/build-metrics-intro" },
collapsed: true,
items: [
"docs/build/sl-getting-started",
{
type: "category",
label: "About MetricFlow",
Expand All @@ -262,7 +263,6 @@ const sidebarSettings = {
"docs/build/metricflow-cli",
]
},
"docs/build/sl-getting-started",
{
type: "category",
label: "Semantic models",
Expand Down
2 changes: 1 addition & 1 deletion website/snippets/_sl-create-semanticmodel.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
The following steps will walk you through setting up semantic models in your dbt project, which consist of [entities](/docs/build/entities), [dimensions](/docs/build/dimensions), and [measures](/docs/build/measures).
The following steps will walk you through setting up semantic models, which you can do with the dbt Cloud IDE or the CLI. Semantic models consist of [entities](/docs/build/entities), [dimensions](/docs/build/dimensions), and [measures](/docs/build/measures).

We highly recommend you read the overview of what a [semantic model](/docs/build/semantic-models) is before getting started. If you're working in the [Jaffle shop example](https://github.com/dbt-labs/jaffle-sl-template), delete the `orders.yml` config or delete the .yml extension so it's ignored during parsing. **We'll be rebuilding it step by step in this example.**

Expand Down
4 changes: 3 additions & 1 deletion website/snippets/_sl-define-metrics.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
Now that you've created your first semantic model, it's time to define your first metric! MetricFlow supports different metric types like [simple](/docs/build/simple), [ratio](/docs/build/ratio), [cumulative](/docs/build/cumulative), and [derived](/docs/build/derived). It's recommended that you read the [metrics overview docs](/docs/build/metrics-overview) before getting started.
Now that you've created your first semantic model, it's time to define your first metric! You can define metrics with the dbt Cloud IDE or CLI.

MetricFlow supports different metric types like [simple](/docs/build/simple), [ratio](/docs/build/ratio), [cumulative](/docs/build/cumulative), and [derived](/docs/build/derived). It's recommended that you read the [metrics overview docs](/docs/build/metrics-overview) before getting started.

1. You can define metrics in the same YAML files as your semantic models or create a new file. If you want to create your metrics in a new file, create another directory called `/models/metrics`. The file structure for metrics can become more complex from here if you need to further organize your metrics, for example, by data source or business line.

Expand Down
5 changes: 3 additions & 2 deletions website/snippets/_sl-install-metricflow.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
Before you begin, install the [MetricFlow CLI](/docs/build/metricflow-cli) as an extension of a dbt adapter from PyPI. The MetricFlow CLI is compatible with Python versions 3.8, 3.9, 3.10 and 3.11
Install the [MetricFlow CLI](/docs/build/metricflow-cli) as an extension of a dbt adapter from PyPI. The MetricFlow CLI is compatible with Python versions 3.8, 3.9, 3.10 and 3.11

Use pip install `metricflow` and your [dbt adapter](/docs/supported-data-platforms):

- Create or activate your virtual environment. `python -m venv venv` or `source your-venv/bin/activate`
- Run `pip install "dbt-metricflow[your_adapter_name]"`
- You must specify `[your_adapter_name]`. For example, run `pip install "dbt-metricflow[snowflake]"` if you use a Snowflake adapter.
- You must specify `[your_adapter_name]`.
- For example, run `pip install "dbt-metricflow[snowflake]"` if you use a Snowflake adapter.
20 changes: 15 additions & 5 deletions website/snippets/_sl-test-and-query-metrics.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,23 @@
This section will explain how you can locally test and query metrics. Before you begin, refer to [MetricFlow CLI](/docs/build/metricflow-cli) for instructions on installing it and a reference for the CLI commands.
:::important Testing and querying metrics in the dbt Cloud IDE not yet supported
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

:::important
For public beta, testing or querying metrics in the dbt Cloud IDE isn't supported yet, however support is coming soon.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

- dbt Cloud Team or Enterprise &mdash; For public beta, querying metrics in the dbt Cloud IDE isn't yet supported (Coming soon). You'll still be able to run semantic validation on your metrics in the IDE to ensure they are defined correctly. You can also use the MetricFlow CLI to test and query metrics locally. Alternatively, you can test using SQL client tools like DataGrip, DBeaver, or RazorSQL.
You can use the **Preview** or **Compile** buttons in the IDE to run semantic validations and make sure your metrics are defined. And you can [dynamically query metrics](#connect-and-query-api) with integrated tools on a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) plan using the [Semantic Layer API](/docs/dbt-cloud-apis/sl-api-overview).
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

You can currently only test metrics locally using the MetricFlow CLI. Alternatively, you can test using SQL client tools like DataGrip, DBeaver, or RazorSQL.
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

- dbt Core or Developer plan &mdash; Users can only test and query metrics manually using the CLI, but won't be able to use the dbt Semantic Layer to dynamically query metrics.
:::

**Query and commit your metrics using the CLI:**
This section will explain how you can test and query metrics locally using the MetricFlow CLI (dbt Cloud IDE support coming soon).

Before you begin, you'll need to install the [MetricFlow CLI](/docs/build/metricflow-cli) package and make sure you run at least one model.
### Install MetricFlow

import InstallMetricFlow from '/snippets/_sl-install-metricflow.md';
mirnawong1 marked this conversation as resolved.
Show resolved Hide resolved

<InstallMetricFlow />

### Query and commit your metrics using the CLI

MetricFlow needs a `semantic_manifest.json` in order to build a semantic graph. To generate a semantic_manifest.json artifact run `dbt parse`. This will create the file in your `/target` directory. If you're working from the Jaffle shop example, run `dbt seed && dbt run` before preceding to ensure the data exists in your warehouse.

Expand Down
11 changes: 5 additions & 6 deletions website/snippets/_v2-sl-prerequisites.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,12 @@
To use the Semantic Layer, you must:

- Have a dbt Cloud Team or Enterprise [multi-tenant](/docs/cloud/about-cloud/regions-ip-addresses) deployment, hosted in North America.
- Have both your production and development environments running dbt version 1.6 or higher.
- Have both your production and development environments running dbt version 1.6 or higher. Refer to [upgrade in dbt Cloud](/docs/dbt-versions/upgrade-core-in-cloud) for more info.
- Use Snowflake, BigQuery, Databricks, or Redshift (dbt Cloud Postgres support coming soon).
- Create a successful run in the environment where you configure the Semantic Layer. <br/>
**Note:** Semantic Layer currently supports the Deployment environment. (_development experience coming soon_)
- Install the [MetricFlow CLI](/docs/build/metricflow-cli). After installing the package, make sure you run at least one model.
- Set up the [Semantic Layer API](/docs/dbt-cloud-apis/sl-api-overview) in the integrated tool to import metric definitions. <br />
**Note:** Developer accounts can only query data manually using the [MetricFlow CLI](/docs/build/metricflow-cli) and SQL. To dynamically query metrics using external tools, you must have a dbt Cloud Team or Enterprise account with access to the Semantic Layer API.<br />
- Create a successful run in the environment where you configure the Semantic Layer.
- **Note:** Semantic Layer currently supports the Deployment environment for querying. (_development querying experience coming soon_)
- Set up the [Semantic Layer API](/docs/dbt-cloud-apis/sl-api-overview) in the integrated tool to import metric definitions.
- **Note:** dbt Core or Developer accounts can only query data manually using the [MetricFlow CLI](/docs/build/metricflow-cli) and SQL. To dynamically query metrics using external tools, you must have a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account with access to the Semantic Layer API.<br />
- Understand [MetricFlow's](/docs/build/about-metricflow) key concepts, which powers the revamped dbt Semantic Layer.


Expand Down