From bed4612e9075c61a887c75c9f0ee5cecf53a8a1d Mon Sep 17 00:00:00 2001 From: Tatiana Al-Chueyr Date: Thu, 14 Sep 2023 11:00:58 +0100 Subject: [PATCH] Address CR feedback --- docs/getting_started/dbt-airflow-concepts.rst | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/docs/getting_started/dbt-airflow-concepts.rst b/docs/getting_started/dbt-airflow-concepts.rst index 6afc1b1fb..8dfe00582 100644 --- a/docs/getting_started/dbt-airflow-concepts.rst +++ b/docs/getting_started/dbt-airflow-concepts.rst @@ -17,14 +17,17 @@ who may be new to Airflow or dbt and are considering to use Cosmos. | DAG | Workflow | Pipeline (Direct Acyclic Graph) that contains a group of steps | Airflow expects upstream tasks to have passed to run downstream tasks. | https://airflow.apache.org/docs/apache-airflow/2.7.1/core-concepts/dags.html | | | | | dbt can run a subset of tasks assuming upstream tasks were run. | https://docs.getdbt.com/docs/introduction | +----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+ -| Task | Node | Step within a pipeline (DAG or workflow) | In dbt, users write mostly SQL and YML to define the steps of a pipeline. | https://docs.getdbt.com/reference/node-selection/syntax | -| | | | Airflow expects steps to be written in Python. | https://airflow.apache.org/docs/apache-airflow/2.7.1/core-concepts/tasks.html | +| Task | Node | Step within a pipeline (DAG or workflow) | In dbt, these are usually transformations that run on a remote database. | https://docs.getdbt.com/reference/node-selection/syntax | +| | | | In Airflow, steps can be anything, running locally in Airflow or remotely. | https://airflow.apache.org/docs/apache-airflow/2.7.1/core-concepts/tasks.html | ++----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+ +| Language | Language | Programming or declarative language used to define pipelines and steps. | In dbt, users write SQL, YML and Python to define the steps of a pipeline. | https://docs.getdbt.com/docs/introduction#dbt-optimizes-your-workflow | +| | | | Airflow expects steps and pipelines are written in Python. | https://airflow.apache.org/docs/apache-airflow/stable/public-airflow-interface.html | +----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+ | Variables | Variables | Key-value configuration that can be used in steps and avoids hard-coded values | | https://docs.getdbt.com/docs/build/project-variables | | | | | | https://airflow.apache.org/docs/apache-airflow/2.7.1/core-concepts/variables.html | +----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+ | Templating | Macros | Jinja templating used to access variables, configuration and reference steps | dbt encourages using jinja templating for control structures (if and for). | https://docs.getdbt.com/docs/build/jinja-macros | -| | | | Airflow usage is limited to variables, macros and filters | https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html | +| | | | Native in Airflow/Python, used to define variables, macros and filters. | https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html | +----------------+--------------+---------------------------------------------------------------------------------+-----------------------------------------------------------------------------+--------------------------------------------------------------------------------------+ | Connection | Profile | Configuration to connect to databases or other services | | https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html | | | | | | https://docs.getdbt.com/docs/core/connect-data-platform/connection-profiles |