Skip to content

Commit

Permalink
Merge branch 'current' into sr-doc
Browse files Browse the repository at this point in the history
  • Loading branch information
dataders authored Oct 25, 2023
2 parents bad298d + 05edcf9 commit e9b0615
Show file tree
Hide file tree
Showing 20 changed files with 1,368 additions and 100 deletions.
20 changes: 2 additions & 18 deletions website/docs/docs/build/about-metricflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,24 +37,6 @@ MetricFlow abides by these principles:
- **Simplicity with gradual complexity:** Approach MetricFlow using familiar data modeling concepts.
- **Performance and efficiency**: Optimize performance while supporting centralized data engineering and distributed logic ownership.

<!--MetricFlow is a SQL query generation engine that helps you create metrics by constructing appropriate queries for different granularities and dimensions that are useful for various business applications.
- It uses YAML files to define a semantic graph, which maps language to data. This graph consists of , which serve as data entry points, and [, which are functions used to create new quantitative indicators.
- MetricFlow is a () and available on dbt versions 1.6 and higher.
- MetricFlow, as a part of the dbt Semantic Layer, allows organizations to define company metrics logic through YAML abstractions, as described in the following sections.
- To query metrics dimensions, dimension values, and validate your configurations; install MetricFlow in
MetricFlow has the following principles:
- **Flexible, but complete** &mdash; Ability to create any metric on any data model by defining logic in flexible abstractions.
- **Don't Repeat Yourself (DRY)** &mdash; Avoid repetition by allowing metric definitions to be enabled whenever possible.
- **Simple with progressive complexity** &mdash; Make MetricFlow approachable by relying on known concepts and structures in data modeling.
- **Performant and efficient** &mdash; Allow for performance optimizations in centralized data engineering while still enabling distributed definition and ownership of logic.
-->

### Semantic graph

We're introducing a new concept: a "semantic graph". It's the relationship between semantic models and YAML configurations that creates a data landscape for building metrics. You can think of it like a map, where tables are like locations, and the connections between them (edges) are like roads. Although it's under the hood, the semantic graph is a subset of the <Term id="dag" />, and you can see the semantic models as nodes on the DAG.
Expand All @@ -73,6 +55,8 @@ For a semantic model, there are three main pieces of metadata:
* [Dimensions](/docs/build/dimensions) &mdash; These are the ways you want to group or slice/dice your metrics.
* [Measures](/docs/build/measures) &mdash; The aggregation functions that give you a numeric result and can be used to create your metrics.

<Lightbox src="/img/docs/dbt-cloud/semantic-layer/semantic_foundation.jpg" width="70%" title="A semantic model is made up of different components: Entities, Measures, and Dimensions."/>

### Metrics

Metrics, which is a key concept, are functions that combine measures, constraints, or other mathematical functions to define new quantitative indicators. MetricFlow uses measures and various aggregation types, such as average, sum, and count distinct, to create metrics. Dimensions add context to metrics and without them, a metric is simply a number for all time. You can define metrics in the same YAML files as your semantic models, or create a new file.
Expand Down
18 changes: 11 additions & 7 deletions website/docs/docs/build/metricflow-commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ sidebar_label: "MetricFlow commands"
tags: [Metrics, Semantic Layer]
---

Once you define metrics in your dbt project, you can query metrics, dimensions, dimension values, and validate your configs using the MetricFlow commands.
Once you define metrics in your dbt project, you can query metrics, dimensions, and dimension values, and validate your configs using the MetricFlow commands.

MetricFlow allows you to define and query metrics in your dbt project in the [dbt Cloud CLI](/docs/cloud/cloud-cli-installation), [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud), or [dbt Core](/docs/core/installation). To experience the power of the universal [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and dynamically query those metrics in downstream tools, you'll need a dbt Cloud [Team or Enterprise](https://www.getdbt.com/pricing/) account.

MetricFlow is compatible with Python versions 3.8, 3.9, 3.10 and 3.11.
MetricFlow is compatible with Python versions 3.8, 3.9, 3.10, and 3.11.


## MetricFlow
Expand All @@ -25,7 +25,7 @@ MetricFlow is a dbt package that allows you to define and query metrics in your

MetricFlow commands are embedded in the dbt Cloud CLI, which means you can immediately run them once you install the dbt Cloud CLI.

A benefit to using the dbt Cloud CLI or dbt Cloud IDE is that you won't need to manage versioning &mdash; your dbt Cloud account will automatically manage the versioning for you.
A benefit to using the dbt Cloud is that you won't need to manage versioning &mdash; your dbt Cloud account will automatically manage the versioning.

</TabItem>

Expand All @@ -35,6 +35,8 @@ A benefit to using the dbt Cloud CLI or dbt Cloud IDE is that you won't need to
You can create metrics using MetricFlow in the dbt Cloud IDE. However, support for running MetricFlow commands in the IDE will be available soon.
:::

A benefit to using the dbt Cloud is that you won't need to manage versioning &mdash; your dbt Cloud account will automatically manage the versioning.

</TabItem>

<TabItem value="core" label="dbt Core">
Expand All @@ -44,13 +46,13 @@ You can create metrics using MetricFlow in the dbt Cloud IDE. However, support f

Use the dbt Cloud CLI for the experience in defining and querying metrics in your dbt project on dbt Cloud or dbt Core with MetricFlow.

A benefit to using the dbt Cloud CLI or dbt Cloud IDE is that you won't need to manage versioning your dbt Cloud account will automatically manage the versioning for you.
A benefit to using the dbt Cloud is that you won't need to manage versioning &mdash; your dbt Cloud account will automatically manage the versioning.
:::


You can install [MetricFlow](https://github.com/dbt-labs/metricflow#getting-started) from [PyPI](https://pypi.org/project/dbt-metricflow/). You need to use `pip` to instal MetricFlow on Windows or Linux operating systems:
You can install [MetricFlow](https://github.com/dbt-labs/metricflow#getting-started) from [PyPI](https://pypi.org/project/dbt-metricflow/). You need to use `pip` to install MetricFlow on Windows or Linux operating systems:

1. Create or activate your virtual environment`python -m venv venv`
1. Create or activate your virtual environment `python -m venv venv`
2. Run `pip install dbt-metricflow`
* You can install MetricFlow using PyPI as an extension of your dbt adapter in the command line. To install the adapter, run `pip install "dbt-metricflow[your_adapter_name]"` and add the adapter name at the end of the command. For example, for a Snowflake adapter run `pip install "dbt-metricflow[snowflake]"`

Expand All @@ -60,6 +62,7 @@ You can install [MetricFlow](https://github.com/dbt-labs/metricflow#getting-star

</Tabs>

Something to note, MetricFlow `mf` commands return an error if you have a Metafont latex package installed. To run `mf` commands, uninstall the package.

## MetricFlow commands

Expand All @@ -77,7 +80,7 @@ Use the `dbt sl` prefix before the command name to execute them in dbt Cloud. Fo
- [`list entities`](#list-entities) &mdash; Lists all unique entities.
- [`query`](#query) &mdash; Query metrics and dimensions you want to see in the command line interface. Refer to [query examples](#query-examples) to help you get started.

<!--below commands aren't support in dbt cloud yet
<!--below commands aren't supported in dbt cloud yet
- [`validate-configs`](#validate-configs) &mdash; Validates semantic model configurations.
- [`health-checks`](#health-checks) &mdash; Performs data platform health check.
- [`tutorial`](#tutorial) &mdash; Dedicated MetricFlow tutorial to help get you started.
Expand Down Expand Up @@ -552,3 +555,4 @@ Keep in mind that modifying your shell configuration files can have an impact on
</details>
</details>
2 changes: 2 additions & 0 deletions website/docs/docs/build/semantic-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ Semantic models are the foundation for data definition in MetricFlow, which powe
- Configure semantic models in a YAML file within your dbt project directory.
- Organize them under a `metrics:` folder or within project sources as needed.

<Lightbox src="/img/docs/dbt-cloud/semantic-layer/semantic_foundation.jpg" width="70%" title="A semantic model is made up of different components: Entities, Measures, and Dimensions."/>

Semantic models have 6 components and this page explains the definitions with some examples:

| Component | Description | Type |
Expand Down
1 change: 1 addition & 0 deletions website/docs/docs/build/sl-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,3 +94,4 @@ The dbt Semantic Layer is proprietary, however, some components of the dbt Seman
- [Build your metrics](/docs/build/build-metrics-intro)
- [Get started with the dbt Semantic Layer](/docs/use-dbt-semantic-layer/quickstart-sl)
- [Available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations)
- Demo on [how to define and query metrics with MetricFlow](https://www.loom.com/share/60a76f6034b0441788d73638808e92ac?sid=861a94ac-25eb-4fd8-a310-58e159950f5a)
102 changes: 71 additions & 31 deletions website/docs/docs/cloud/cloud-cli-installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,38 +23,61 @@ dbt commands are run against dbt Cloud's infrastructure and benefit from:


## Prerequisites
The dbt Cloud CLI is available in all [deployment regions](/docs/cloud/about-cloud/regions-ip-addresses) and and for both multi-tenant and single-tenant accounts (Azure single-tenant not supported at this time).
The dbt Cloud CLI is available in all [deployment regions](/docs/cloud/about-cloud/regions-ip-addresses) and for both multi-tenant and single-tenant accounts (Azure single-tenant not supported at this time).

You must be on dbt version 1.5 or higher. Refer to [dbt Cloud versions](/docs/dbt-versions/upgrade-core-in-cloud) to upgrade.
- Ensure you are using dbt version 1.5 or higher. Refer to [dbt Cloud versions](/docs/dbt-versions/upgrade-core-in-cloud) to upgrade.
- Avoid using SSH tunneling for [Postgres and Redshift](/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb) connections.
- Avoid using [PrivateLink](/docs/cloud/secure/about-privatelink).

## Install dbt Cloud CLI

You can install the dbt Cloud CLI on the command line by using one of these methods:
You can install the dbt Cloud CLI on the command line by using one of these methods.

<details>
<summary>View a video tutorial for a step-by-step guide to installing</summary>

<LoomVideo id="dd80828306c5432a996d4580135041b6?sid=fe1895b7-1281-4e42-9968-5f7d11768000"/>

</details>

<Tabs queryString="install">

<TabItem value="brew" label="macOS (brew)">

Before you begin, make sure you have [Homebrew installed](http://brew.sh/) in your code editor or command line terminal. Refer to the [FAQs](#faqs) if your operating system runs into path conflicts.


1. Run the following command to verify that there is no conflict with a dbt Core installation on your system:
1. Run the following command to verify that you don't already have dbt Core installed:

```bash
which dbt
```
- This should return a `dbt not found`. If the dbt help text appears, use `pip uninstall dbt` to deactivate dbt Core from your machine.
- This should return a `dbt not found`. If the dbt help text appears, use `pip uninstall dbt` to remove dbt Core from your machine. <br />

2. Install the dbt Cloud CLI with Homebrew:

```bash
brew untap dbt-labs/dbt
brew tap dbt-labs/dbt-cli
brew install dbt
```
- First, remove the dbt-labs tap, the separate repository for packages, from Homebrew. This prevents Homebrew from installing packages from that repository:
```bash
brew untap dbt-labs/dbt
- Then run `brew tap` to add and install the dbt Cloud CLI as a package:
```bash
brew tap dbt-labs/dbt-cli
```
- Lastly, install the dbt Cloud CLI with Homebrew:
```bash
brew install dbt
```

3. Verify your installation by running `dbt --help` in the command line. If you see the following output, your installation is correct:
```bash
The dbt Cloud CLI - an ELT tool for running SQL transformations and data models in dbt Cloud...
```
If you don't see this output, check that you've deactivated pyenv or venv and don't have a global dbt version installed.
* Note that you no longer need to run the `dbt deps` command when your environment starts. This step was previously required during initialization. However, you should still run `dbt deps` if you make any changes to your `packages.yml` file.
3. Verify the installation by running `dbt --help` from the command line. If the help text doesn't indicate that you're using the dbt Cloud CLI, make sure you've deactivated your pyenv or venv and don't have a version of dbt globally installed.
* You don't have to run the `dbt deps` command when your environment starts. Previously, you had to do it during initialization. However, you'll still need to run `dbt deps` if you make changes to your `packages.yml` file.
4. After you've verified the installation, [configure](/docs/cloud/configure-cloud-cli) the dbt Cloud CLI for your dbt Cloud project and use it to run [dbt commands](/reference/dbt-commands) similar to dbt Core. For example, execute `dbt compile` to compile a project using dbt Cloud and validate your models and tests.
* If you're using the dbt Cloud CLI, you can connect to your data platform directly in the dbt Cloud interface and don't need a [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml) file locally on your machine.
</TabItem>
Expand All @@ -73,8 +96,17 @@ Advanced users can configure multiple projects to use the same dbt Cloud CLI by
Note that if you are using VS Code, you must restart it to pick up modified environment variables.
:::
3. Verify the installation by running `./dbt --help` from the command line. If the help text doesn't indicate that you're using the dbt Cloud CLI, make sure you've deactivated your pyenv or venv and don't have a version of dbt globally installed.
* You don't have to run the `dbt deps` command when your environment starts. Previously, you had to do it during initialization. However, you'll still need to run `dbt deps` if you make changes to your `packages.yml` file.
3. Verify your installation by running `./dbt --help` in the command line. If you see the following output, your installation is correct:
```bash
The dbt Cloud CLI - an ELT tool for running SQL transformations and data models in dbt Cloud...
```
If you don't see this output, check that you've deactivated pyenv or venv and don't have a global dbt version installed.
* Note that you no longer need to run the `dbt deps` command when your environment starts. This step was previously required during initialization. However, you should still run `dbt deps` if you make any changes to your `packages.yml` file.
4. After installation, [configure](/docs/cloud/configure-cloud-cli) the dbt Cloud CLI for your dbt Cloud project and use it to run [dbt commands](/reference/dbt-commands) similar to dbt Core. For example, execute `dbt compile`, to compile a project using dbt Cloud and confirm that it works.
* If you're using the dbt Cloud CLI, you can connect to your data platform directly in the dbt Cloud interface and don't need a [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml) file locally on your machine.
</TabItem>
Expand All @@ -97,23 +129,36 @@ Advanced users can configure multiple projects to use the same Cloud CLI executa
:::
3. Verify the installation by running `./dbt --help` from the command line. If the help text doesn't indicate that you're using the dbt Cloud CLI, make sure you've deactivated your pyenv or venv and don't have a version of dbt globally installed.
* You don't have to run the `dbt deps` command when your environment starts. Previously, you had to do it during initialization. However, you'll still need to run `dbt deps` if you make changes to your `packages.yml` file.
3. Verify your installation by running `./dbt --help` in the command line. If you see the following output, your installation is correct:
```bash
The dbt Cloud CLI - an ELT tool for running SQL transformations and data models in dbt Cloud...
```
If you don't see this output, check that you've deactivated pyenv or venv and don't have a global dbt version installed.
* Note that you no longer need to run the `dbt deps` command when your environment starts. This step was previously required during initialization. However, you should still run `dbt deps` if you make any changes to your `packages.yml` file.
4. After installation, [configure](/docs/cloud/configure-cloud-cli) the dbt Cloud CLI for your dbt Cloud project and use it to run [dbt commands](/reference/dbt-commands) similar to dbt Core. For example, execute `dbt compile`, to compile a project using dbt Cloud and confirm that it works.
* If you're using the dbt Cloud CLI, you can connect to your data platform directly in the dbt Cloud interface and don't need a [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml) file locally on your machine.
</TabItem>
<TabItem value="pip" label="Existing dbt Core users (pip)">
:::info Use native packages or a virtual environment to avoid overriding dbt Core
Installing the dbt Cloud CLI with pip replaces dbt Core. This change can be avoided by using the native install method and configuring your PATH or by creating a new virtual environment.
:::info Use native packages or a virtual environment to prevent dbt Core conflicts
To prevent overwriting dbt Core, avoid installing the dbt Cloud CLI with pip. Instead, consider using the native installation method and configuring your PATH or create a new virtual environment.
Otherwise, to switch back to dbt Core, uninstall the dbt Cloud CLI and follow the dbt Core installation instructions.
If you've already installed the dbt Cloud CLI and need to switch back to dbt Core, uninstall the dbt Cloud CLI, and follow the dbt Core installation instructions.
You can also have both dbt Cloud CLI and dbt Core installed simultaneously. To avoid conflicts, alias the dbt Cloud CLI as `dbt-cloud`. For more details, check the [FAQs](#faqs) if your operating system experiences path conflicts.
:::
:::
Before installing the dbt Cloud CLI, make sure you have Python installed and your virtual environment venv or pyenv . If you already have a Python environment configured, you can skip to the [pip installation step](#install-dbt-cloud-cli-in-pip).
### Install a virtual environment
We recommend using virtual environments (venv) to namespace `cloud-cli`.
Expand Down Expand Up @@ -156,6 +201,9 @@ We recommend using virtual environments (venv) to namespace `cloud-cli`.
pip3 install dbt-core==VERSION
```
4. After you've verified the installation, [configure](/docs/cloud/configure-cloud-cli) the dbt Cloud CLI for your dbt Cloud project and use it to run [dbt commands](/reference/dbt-commands) similar to dbt Core. For example, execute `dbt compile` to compile a project using dbt Cloud and validate your models and tests.
* If you're using the dbt Cloud CLI, you can connect to your data platform directly in the dbt Cloud interface and don't need a [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml) file locally on your machine.
</TabItem>
</Tabs>
Expand Down Expand Up @@ -198,16 +246,6 @@ To update:
</Tabs>
## Next steps
After installation, you can [configure](/docs/cloud/configure-cloud-cli) the dbt Cloud CLI for your dbt Cloud project and use it to run [dbt commands](/reference/dbt-commands) similar to dbt Core.
For example, you can execute `dbt compile` to compile a project using dbt Cloud.
Note, that if you're using the dbt Cloud CLI, you can connect to your data platform directly in the dbt Cloud interface and don't need a [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml) file.
## FAQs
<details>
Expand All @@ -231,3 +269,5 @@ If you have dbt Core installed locally, either:
You can always uninstall the dbt Cloud CLI to return to using dbt Core.
</details>
Loading

0 comments on commit e9b0615

Please sign in to comment.