Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create dbt converter for Airbyte Amplitude source connector #47

Merged
merged 14 commits into from
Aug 13, 2024
88 changes: 82 additions & 6 deletions connectors/source_amplitude/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,85 @@
# Airbyte source_amplitude dbt Package
# Amplitude Support Airbyte dbt Package

This package contains dbt models for Airbyte source_amplitude source.
---

What it includes:
- This package contains dbt models to work with Airbyte Amplitude Support connector.
- The package is compatible with latest version of Airbyte Amplitude Support connector.
- Currently, it is limited to creating transformations compatible with [Fivetran's modeling dbt package](https://github.com/fivetran/dbt_amplitude/tree/main).
- In the future, specific models will be applied directly to Airbyte connector output. If you have an idea or want to propose an analytical model for this source, please refer to the contributing guide, which explains how to propose a new transformation model.
- This package was tested with BigQuery, Snowflake, and Postgres data warehouses.

* A complete source description
* ERD model for the source
* Diagram documentation for the source
---

## 🎯 Intructions how to use

### Airbyte dbt Package

For now Airbyte dbt packages aren't versioned. You must configure using git and subdirectory. For now there isn't any transformation model directly applied to this package. But you can generate docs and tests with dbt.

Create the following files:

**`dbt_project.yml`**

```yaml
vars:
using_fivetran_model: False
airbyte_database: "airbyte_db_default"
airbyte_schema: "airbyte_dbt_amplitude_support"
```

**`packages.yml`**

```yaml
packages:
- git: "https://github.com/airbytehq/airbyte-dbt-models.git"
subdirectory: "connectors/source_amplitude"
```

After you can run `dbt tests` or `dbt docs generate` to have a preview of Airbyte output data.

### Fivetran Amplitude Modeling dbt package

This package transforms Airbyte connector output data, making it compatible with Fivetran's Amplitude dbt package. You can check the analytical models Fivetran creates [here](https://github.com/fivetran/dbt_amplitude/tree/main?tab=readme-ov-file#-what-does-this-dbt-package-do). The link also provides information about how the package works and what is configurable.

Create the require files to use Airbyte and Fivetran dbt packages:

**`packages.yml`**

```yaml
packages:
- git: "https://github.com/airbytehq/airbyte-dbt-models.git"
subdirectory: "connectors/source_amplitude_support"

- package: fivetran/amplitude
version: [">=0.16.0", "<0.17.0"]
```

This is a default variable definition you must configure to have the models created.
At the moment this package doesn't support (schedules, domains, user tags, ticket form history and organization tags) for that reason keep those variables set to `False`.
Variables starting with the prefix `amplitude_..._identifier` represent the names of tables generated by the Airbyte connector. If you configured your sync with this prefix, ensure you edit it accordingly.

**`dbt_project.yml`**

```yaml
vars:
# Required by Airbyte dbt model
using_fivetran_model: True
airbyte_database: "airbyte_db_default"
airbyte_schema: "airbyte_dbt_amplitude"

# Required by Fivetran dbt model
amplitude_database: "airbyte_db_default"
amplitude_schema: "airbyte_dbt_amplitude"

amplitude_event_identifier: "event"
amplitude_event_type_identifier: "events_list"
```

After run `dbt run`, you can see the models being created.

---

## :package: Package Maintenance

- This package is maintained by the Airbyte Community.
- You can contribute any time please read the Contributing Guidelines or enter the Airbyte Slack Channel `#airbyte-dbt-packages`
4 changes: 2 additions & 2 deletions connectors/source_amplitude/dbt_project.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ config-version: 2

version: 0.1.0

profile: airbyte
profile: integration_tests

model-paths:
- models
Expand All @@ -20,7 +20,7 @@ clean-targets:
- logs

require-dbt-version:
- '>=1.0.0'
- ">=1.0.0"
- <2.0.0

models:
Expand Down
46 changes: 46 additions & 0 deletions connectors/source_amplitude/integration_tests/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: integration_test_amplitude

config-version: 2

version: 0.1.0

profile: integration_tests

model-paths:
- models

macro-paths:
- macros

target-path: target

clean-targets:
- target
- dbt_modules
- logs

require-dbt-version:
- ">=1.0.0"
- <2.0.0

models:
airbyte_dbt_source_amplitude:
materialized: view
+schema: dbt_source_amplitude
staging:
materialized: view
tmp:
materialized: view

vars:
# Required by Airbyte dbt model
using_fivetran_model: True
airbyte_database: "airbyte_db_default"
airbyte_schema: "dbt_source_amplitude"

# Required by Fivetran dbt model
amplitude_database: "airbyte_db_default"
amplitude_schema: "dbt_source_amplitude"

amplitude_event_identifier: "event"
amplitude_event_type_identifier: "events_list"
13 changes: 13 additions & 0 deletions connectors/source_amplitude/integration_tests/package-lock.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
packages:
- local: ../
- package: fivetran/amplitude
version: 0.4.0
- package: fivetran/amplitude_source
version: 0.3.0
- package: fivetran/fivetran_utils
version: 0.4.10
- package: dbt-labs/spark_utils
version: 0.3.0
- package: dbt-labs/dbt_utils
version: 1.2.0
sha1_hash: 45535d64b566fe5c1fee01eda2715d9cda256b6c
5 changes: 5 additions & 0 deletions connectors/source_amplitude/integration_tests/packages.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
packages:
- local: ../

- package: fivetran/amplitude
version: [">=0.4.0", "<0.5.0"]
1 change: 1 addition & 0 deletions connectors/source_amplitude/integration_tests/vars
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{airbyte_database: $AB_DB, amplitude_database: $AB_DB}
Loading
Loading