diff --git a/connectors/source_tiktok_marketing/README.md b/connectors/source_tiktok_marketing/README.md index 1e0eabe4..a72831d5 100644 --- a/connectors/source_tiktok_marketing/README.md +++ b/connectors/source_tiktok_marketing/README.md @@ -1,9 +1,93 @@ -# Airbyte source_tiktok_marketing dbt Package +# Tiktok Marketing Airbyte dbt Package -This package contains dbt models for Airbyte source_tiktok_marketing source. +--- + +> [!WARNING] +> This model was tested and only work with Bigquery. What it includes: -* A complete source description -* ERD model for the source -* Diagram documentation for the source +- This package contains dbt models to work with Airbyte Tiktok Marketing connector. +- The package is compatible with latest version of Airbyte Tiktok Marketing connector. +- Currently, it is limited to creating transformations compatible with [Fivetran's modeling dbt package](https://github.com/fivetran/dbt_tiktok_ads/tree/main). +- In the future, specific models will be applied directly to Airbyte connector output. If you have an idea or want to propose an analytical model for this source, please refer to the contributing guide, which explains how to propose a new transformation model. +- This package was tested with BigQuery, Snowflake, and Postgres data warehouses. + +--- + +## 🎯 Intructions how to use + +### Airbyte dbt Package + +For now Airbyte dbt packages aren't versioned. You must configure using git and subdirectory. For now there isn't any transformation model directly applied to this package. But you can generate docs and tests with dbt. + +Create the following files: + +**`dbt_project.yml`** + +```yaml +vars: + using_fivetran_model: False + airbyte_database: "airbyte_db_default" + airbyte_schema: "airbyte_dbt_tiktok_marketing" +``` + +**`packages.yml`** + +```yaml +packages: + - git: "https://github.com/airbytehq/airbyte-dbt-models.git" + subdirectory: "connectors/source_tiktok_marketing" +``` + +After you can run `dbt tests` or `dbt docs generate` to have a preview of Airbyte output data. + +### Fivetran Tiktok Marketing Modeling dbt package + +This package transforms Airbyte connector output data, making it compatible with Fivetran's Tiktok Marketing dbt package. You can check the analytical models Fivetran creates [here](https://github.com/fivetran/dbt_tiktok_ads/tree/main?tab=readme-ov-file#-what-does-this-dbt-package-do). The link also provides information about how the package works and what is configurable. + +Create the require files to use Airbyte and Fivetran dbt packages: + +**`packages.yml`** + +```yaml +packages: + - git: "https://github.com/airbytehq/airbyte-dbt-models.git" + subdirectory: "connectors/source_tiktok_marketing" + + - package: fivetran/tiktok_ads + version: [">=0.5.0", "<0.6.0"] +``` + +This is a default variable definition you must configure to have the models created. + +**`dbt_project.yml`** + +```yaml +vars: + # Required by Airbyte dbt model + using_fivetran_model: True + airbyte_database: "airbyte_db_default" + airbyte_schema: "airbyte_dbt_tiktok_marketing" + + # Required by Fivetran dbt model + tiktok_ads_database: "airbyte_db_default" + tiktok_ads_schema: "airbyte_dbt_tiktok_marketing" + + tiktok_ads_adgroup_history_identifier: "ad_group_history" + tiktok_ads_adgroup_report_hourly_identifier: "ad_group_report_hourly" + tiktok_ads_ad_history_identifier: "ad_history" + tiktok_ads_ad_report_hourly_identifier: "ad_report_hourly" + tiktok_ads_advertiser_identifier: "advertiser" + tiktok_ads_campaign_history_identifier: "campaign_history" + tiktok_ads_campaign_report_hourly_identifier: "campaign_report_hourly" +``` + +After run `dbt run`, you can see the models being created. + +--- + +## :package: Package Maintenance + +- This package is maintained by the Airbyte Community. +- You can contribute any time please read the Contributing Guidelines or enter the Airbyte Slack Channel `#airbyte-dbt-packages` \ No newline at end of file diff --git a/connectors/source_tiktok_marketing/dbt_project.yml b/connectors/source_tiktok_marketing/dbt_project.yml index cf75a719..4a41f800 100644 --- a/connectors/source_tiktok_marketing/dbt_project.yml +++ b/connectors/source_tiktok_marketing/dbt_project.yml @@ -4,7 +4,7 @@ config-version: 2 version: 0.1.0 -profile: airbyte +profile: integration_tests model-paths: - models @@ -20,7 +20,7 @@ clean-targets: - logs require-dbt-version: - - '>=1.0.0' + - ">=1.0.0" - <2.0.0 models: diff --git a/connectors/source_tiktok_marketing/integration_tests/dbt_project.yml b/connectors/source_tiktok_marketing/integration_tests/dbt_project.yml new file mode 100644 index 00000000..c73ef2ce --- /dev/null +++ b/connectors/source_tiktok_marketing/integration_tests/dbt_project.yml @@ -0,0 +1,51 @@ +name: integration_test_tiktok_marketing + +config-version: 2 + +version: 0.1.0 + +profile: integration_tests + +model-paths: + - models + +macro-paths: + - macros + +target-path: target + +clean-targets: + - target + - dbt_modules + - logs + +require-dbt-version: + - ">=1.0.0" + - <2.0.0 + +models: + airbyte_dbt_source_tiktok_marketing: + materialized: view + +schema: dbt_source_tiktok_marketing + staging: + materialized: view + tmp: + materialized: view + +vars: + # Required by Airbyte dbt model + using_fivetran_model: True + airbyte_database: "airbyte_db_default" + airbyte_schema: "airbyte_dbt_source_tiktok_marketing" + + # Required by Fivetran dbt model + tiktok_ads_database: "airbyte_db_default" + tiktok_ads_schema: "airbyte_dbt_source_tiktok_marketing" + + tiktok_ads_adgroup_history_identifier: "ad_group_history" + tiktok_ads_adgroup_report_hourly_identifier: "ad_group_report_hourly" + tiktok_ads_ad_history_identifier: "ad_history" + tiktok_ads_ad_report_hourly_identifier: "ad_report_hourly" + tiktok_ads_advertiser_identifier: "advertiser" + tiktok_ads_campaign_history_identifier: "campaign_history" + tiktok_ads_campaign_report_hourly_identifier: "campaign_report_hourly" diff --git a/connectors/source_tiktok_marketing/integration_tests/package-lock.yml b/connectors/source_tiktok_marketing/integration_tests/package-lock.yml new file mode 100644 index 00000000..647a0f5e --- /dev/null +++ b/connectors/source_tiktok_marketing/integration_tests/package-lock.yml @@ -0,0 +1,13 @@ +packages: + - local: ../ + - package: fivetran/tiktok_ads + version: 0.5.0 + - package: fivetran/tiktok_ads_source + version: 0.5.2 + - package: fivetran/fivetran_utils + version: 0.4.10 + - package: dbt-labs/spark_utils + version: 0.3.0 + - package: dbt-labs/dbt_utils + version: 1.2.0 +sha1_hash: 42ec6d0b088b381a14f44945a2410c46d5049e4f diff --git a/connectors/source_tiktok_marketing/integration_tests/packages.yml b/connectors/source_tiktok_marketing/integration_tests/packages.yml new file mode 100644 index 00000000..cec807c5 --- /dev/null +++ b/connectors/source_tiktok_marketing/integration_tests/packages.yml @@ -0,0 +1,5 @@ +packages: + - local: ../ + + - package: fivetran/tiktok_ads + version: [">=0.5.0", "<0.6.0"] diff --git a/connectors/source_tiktok_marketing/integration_tests/vars b/connectors/source_tiktok_marketing/integration_tests/vars new file mode 100644 index 00000000..729d94e7 --- /dev/null +++ b/connectors/source_tiktok_marketing/integration_tests/vars @@ -0,0 +1 @@ +{airbyte_database: $AB_DB, tiktok_marketing_database: $AB_DB} \ No newline at end of file diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/ad_group_history.sql b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_group_history.sql new file mode 100644 index 00000000..314f4311 --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_group_history.sql @@ -0,0 +1,89 @@ + +{% if target.type == "postgres" %} + + WITH TMP AS ( + SELECT + action_category_ids AS action_categories, + action_days, + adgroup_id, + adgroup_name, + advertiser_id, + age_groups AS age, --FIX + age_groups, + audience_type, + budget, + campaign_id, + category_id AS category, + campaign_name AS display_name, + frequency, + frequency_schedule, + gender, + interest_category_ids AS interest_category_v_2, + app_download_url AS landing_page_url, + languages, + modify_time AS updated_at + FROM + {{ source('source_tiktok_marketing', 'ad_groups') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "bigquery" %} + + WITH TMP AS ( + SELECT + action_category_ids AS action_categories, + action_days, + adgroup_id, + adgroup_name, + advertiser_id, + age_groups AS age, --FIX + age_groups, + audience_type, + budget, + campaign_id, + category_id AS category, + campaign_name AS display_name, + frequency, + frequency_schedule, + gender, + interest_category_ids AS interest_category_v_2, + app_download_url AS landing_page_url, + languages, + modify_time AS updated_at + FROM + {{ source('source_tiktok_marketing', 'ad_groups') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "snowflake" %} + + WITH TMP AS ( + SELECT + action_category_ids AS action_categories, + action_days, + adgroup_id, + adgroup_name, + advertiser_id, + age_groups AS age, --FIX + age_groups, + audience_type, + budget, + campaign_id, + category_id AS category, + campaign_name AS display_name, + frequency, + frequency_schedule, + gender, + interest_category_ids AS interest_category_v_2, + app_download_url AS landing_page_url, + languages, + modify_time AS updated_at + FROM + {{ source('source_tiktok_marketing', 'ad_groups') }} + ) + + SELECT * FROM TMP + +{%endif%} diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/ad_group_history.yml b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_group_history.yml new file mode 100644 index 00000000..2d0c38a3 --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_group_history.yml @@ -0,0 +1,49 @@ +version: 2 + +models: + - name: ad_group_history + schema: "{{ var('airbyte_schema', target.schema) }}" + database: "{{ var('airbyte_database', target.database) }}" + identifier: "{{ var('tiktok_ads_adgroup_history_identifier', 'ad_group_history') }}" + description: All fields and field values associated with ad_group_history. + config: + +enabled: "{{ var('using_fivetran_model', False) }}" + columns: + - name: action_categories + description: The IDs of the action categories associated with the ad group + - name: action_days + description: The number of days the action has been performed + - name: adgroup_id + description: The unique identifier of the ad group + - name: adgroup_name + description: The name of the ad group + - name: advertiser_id + description: The unique identifier of the advertiser + - name: age + description: The targeted age groups for the ad group + - name: age_groups + description: The targeted age groups for the ad group + - name: audience_type + description: The type of audience being targeted + - name: budget + description: The allocated budget for the ad group + - name: campaign_id + description: The unique identifier of the campaign + - name: category + description: The ID of the category for the ad group + - name: display_name + description: The name of the campaign + - name: frequency + description: The frequency of ad display + - name: frequency_schedule + description: The schedule for frequency capping + - name: gender + description: The targeted gender for the ad group + - name: interest_category_v_2 + description: The IDs of interest categories for targeting + - name: landing_page_url + description: The URL for downloading the associated app + - name: languages + description: The targeted languages for the ad group + - name: updated_at + description: The timestamp for when the ad group was last modified \ No newline at end of file diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/ad_group_report_hourly.sql b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_group_report_hourly.sql new file mode 100644 index 00000000..e3534a35 --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_group_report_hourly.sql @@ -0,0 +1,107 @@ + +{% if target.type == "postgres" %} + + WITH TMP AS ( + SELECT + adgroup_id AS ad_group_id, + metrics->>'average_video_play' AS average_video_play, + metrics->>'average_video_play_per_user' AS average_video_play_per_user, + metrics->>'clicks' AS clicks, + metrics->>'comments' AS comments, + metrics->>'conversion' AS conversion, + metrics->>'conversion_rate' AS conversion_rate, + metrics->>'cost_per_conversion' AS cost_per_conversion, + metrics->>'cpc' AS cpc, + metrics->>'cpm' AS cpm, + metrics->>'ctr' AS ctr, + metrics->>'follows' AS follows, + metrics->>'impressions' AS impressions, + metrics->>'likes' AS likes, + metrics->>'profile_visits' AS profile_visits, + metrics->>'reach' AS reach, + metrics->>'shares' AS shares, + metrics->>'spend' AS spend, + stat_time_hour, + metrics->>'video_play_actions' AS video_play_actions, + metrics->>'video_views_p_25' AS video_views_p_25, + metrics->>'video_views_p_50' AS video_views_p_50, + metrics->>'video_views_p_75' AS video_views_p_75, + metrics->>'video_watched_2_s' AS video_watched_2_s, + metrics->>'video_watched_6_s' AS video_watched_6_s + FROM + {{ source('source_tiktok_marketing', 'ad_groups_reports_hourly') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "bigquery" %} + + WITH TMP AS ( + SELECT + adgroup_id AS ad_group_id, + CAST(JSON_VALUE(metrics, '$.average_video_play') as float64) AS average_video_play, + CAST(JSON_VALUE(metrics, '$.average_video_play_per_user') as float64) AS average_video_play_per_user, + CAST(JSON_VALUE(metrics, '$.clicks') as float64) AS clicks, + CAST(JSON_VALUE(metrics, '$.comments') as float64) AS comments, + CAST(JSON_VALUE(metrics, '$.conversion') as float64) AS conversion, + CAST(JSON_VALUE(metrics, '$.conversion_rate') as float64) AS conversion_rate, + CAST(JSON_VALUE(metrics, '$.cost_per_conversion') as float64) AS cost_per_conversion, + CAST(JSON_VALUE(metrics, '$.cpc') as float64) AS cpc, + CAST(JSON_VALUE(metrics, '$.cpm') as float64) AS cpm, + CAST(JSON_VALUE(metrics, '$.ctr') as float64) AS ctr, + CAST(JSON_VALUE(metrics, '$.follows') as float64) AS follows, + CAST(JSON_VALUE(metrics, '$.impressions') as float64) AS impressions, + CAST(JSON_VALUE(metrics, '$.likes') as float64) AS likes, + CAST(JSON_VALUE(metrics, '$.profile_visits') as float64) AS profile_visits, + CAST(JSON_VALUE(metrics, '$.reach') as float64) AS reach, + CAST(JSON_VALUE(metrics, '$.shares') as float64) AS shares, + CAST(JSON_VALUE(metrics, '$.spend') as float64) AS spend, + stat_time_hour, + CAST(JSON_VALUE(metrics, '$.video_play_actions') as float64) AS video_play_actions, + CAST(JSON_VALUE(metrics, '$.video_views_p_25') as float64) AS video_views_p_25, + CAST(JSON_VALUE(metrics, '$.video_views_p_50') as float64) aS video_views_p_50, + CAST(JSON_VALUE(metrics, '$.video_views_p_75') as float64) AS video_views_p_75, + CAST(JSON_VALUE(metrics, '$.video_watched_2_s') as float64) AS video_watched_2_s, + CAST(JSON_VALUE(metrics, '$.video_watched_6_s') as float64) AS video_watched_6_s + FROM + {{ source('source_tiktok_marketing', 'ad_groups_reports_hourly') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "snowflake" %} + + WITH TMP AS ( + SELECT + adgroup_id AS ad_group_id, + metrics:"average_video_play"::string AS average_video_play, + metrics:"average_video_play_per_user"::string AS average_video_play_per_user, + metrics:"clicks"::string AS clicks, + metrics:"comments"::string AS comments, + metrics:"conversion"::string AS conversion, + metrics:"conversion_rate"::string AS conversion_rate, + metrics:"cost_per_conversion"::string AS cost_per_conversion, + metrics:"cpc"::string AS cpc, + metrics:"cpm"::string AS cpm, + metrics:"ctr"::string AS ctr, + metrics:"follows"::string AS follows, + metrics:"impressions"::string AS impressions, + metrics:"likes"::string AS likes, + metrics:"profile_visits"::string AS profile_visits, + metrics:"reach"::string AS reach, + metrics:"shares"::string AS shares, + metrics:"spend"::string AS spend, + stat_time_hour, + metrics:"video_play_actions"::string AS video_play_actions, + metrics:"video_views_p_25"::string AS video_views_p_25, + metrics:"video_views_p_50"::string AS video_views_p_50, + metrics:"video_views_p_75"::string AS video_views_p_75, + metrics:"video_watched_2_s"::string AS video_watched_2_s, + metrics:"video_watched_6_s"::string AS video_watched_6_s + FROM + {{ source('source_tiktok_marketing', 'ad_groups_reports_hourly') }} + ) + + SELECT * FROM TMP + +{%endif%} diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/ad_group_report_hourly.yml b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_group_report_hourly.yml new file mode 100644 index 00000000..3ba572db --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_group_report_hourly.yml @@ -0,0 +1,62 @@ +version: 2 + +models: + - name: ad_group_report_hourly + schema: "{{ var('airbyte_schema', target.schema) }}" + database: "{{ var('airbyte_database', target.database) }}" + identifier: "{{ var('tiktok_ads_adgroup_report_hourly_identifier', 'ad_group_report_hourly') }}" + description: All fields and field values associated with ad_group_report_hourly. + config: + +enabled: "{{ var('using_fivetran_model', False) }}" + columns: + - name: ad_group_id + description: The unique identifier for an ad group. + - name: average_video_play + description: The average time your video was played per single video view, including any time spent replaying the video. + - name: average_video_play_per_user + description: The average time per user your video was played per single video view, including any time spent replaying the video. + - name: clicks + description: The number of clicks on your ads. + - name: comments + description: The number of comments your video creative received within 1 day of a user seeing a paid ad. + - name: conversion + description: The number of times your ad achieved an outcome, based on the secondary goal you selected. As one campaign may have a number of different secondary goals, this statistic is not supported for campaigns. Please go to ad groups or ads to view. (The total count is calculated based on the time each ad impression occurred.) + - name: conversion_rate + description: The percentage of results you received out of all the clicks of your ads. (The total count is calculated based on the time each ad impression occurred.) + - name: cost_per_conversion + description: The average amount of money you've spent on a conversion. (The total count is calculated based on the time each ad impression occurred.) + - name: cpc + description: The average amount of money you've spent on a click. + - name: cpm + description: The average amount of money you've spent per 1,000 impressions. + - name: ctr + description: The percentage of times people saw your ad and performed a click. + - name: follows + description: The number of new followers that were gained within 1 day of a user seeing a paid ad. This metric is only for Boosted TikToks. + - name: impressions + description: The number of times your ads were on screen. + - name: likes + description: The number of likes your video creative received within 1 day of a user seeing a paid ad. + - name: profile_visits + description: The number of profile visits the ad drove during the campaign. This metric is only for Boosted TikToks. + - name: reach + description: The number of unique users who saw your ads at least once. This metric is estimated. + - name: shares + description: The number of shares your video creative received within 1 day of a user seeing a paid ad. + - name: spend + description: The estimated total amount of money you've spent on your campaign, ad group or ad during its schedule. + - name: stat_time_hour + description: The hour of the day for which the statistical data is recorded. + - name: video_play_actions + description: The number of times your video starts to play. Replays will not be counted. + - name: video_views_p_25 + description: The number of times your video was played at 25% of its length. Replays will not be counted. + - name: video_views_p_50 + description: The number of times your video was played at 50% of its length. Replays will not be counted. + - name: video_views_p_75 + description: The number of times your video was played at 75% of its length. Replays will not be counted. + - name: video_watched_2_s + description: The number of times your video played for at least 2 seconds. Replays will not be counted. + - name: video_watched_6_s + description: The number of times your video played for at least 6 seconds, or completely played. Replays will not be counted. + \ No newline at end of file diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/ad_history.sql b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_history.sql new file mode 100644 index 00000000..30af5c4d --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_history.sql @@ -0,0 +1,69 @@ + +{% if target.type == "postgres" %} + + WITH TMP AS ( + SELECT + + ad_id, + ad_name, + adgroup_id, + advertiser_id, + call_to_action, + campaign_id, + click_tracking_url, + impression_tracking_url, + landing_page_url, + modify_time AS updated_at + + FROM + {{ source('source_tiktok_marketing', 'ads') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "bigquery" %} + + WITH TMP AS ( + SELECT + + ad_id, + ad_name, + adgroup_id, + advertiser_id, + call_to_action, + campaign_id, + click_tracking_url, + impression_tracking_url, + landing_page_url, + modify_time AS updated_at + + FROM + {{ source('source_tiktok_marketing', 'ads') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "snowflake" %} + + WITH TMP AS ( + SELECT + + ad_id, + ad_name, + adgroup_id, + advertiser_id, + call_to_action, + campaign_id, + click_tracking_url, + impression_tracking_url, + landing_page_url, + modify_time AS updated_at + + FROM + {{ source('source_tiktok_marketing', 'ads') }} + ) + + SELECT * FROM TMP + +{%endif%} + \ No newline at end of file diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/ad_history.yml b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_history.yml new file mode 100644 index 00000000..b2fdba7d --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_history.yml @@ -0,0 +1,31 @@ +version: 2 + +models: + - name: ad_history + schema: "{{ var('airbyte_schema', target.schema) }}" + database: "{{ var('airbyte_database', target.database) }}" + identifier: "{{ var('tiktok_ads_ad_history_identifier', 'ad_history') }}" + description: All fields and field values associated with ad_history. + config: + +enabled: "{{ var('using_fivetran_model', False) }}" + columns: + - name: ad_id + description: The unique identifier of the ad + - name: ad_name + description: The name of the ad + - name: adgroup_id + description: The unique identifier of the ad group + - name: advertiser_id + description: The unique identifier of the advertiser + - name: call_to_action + description: The call-to-action text for the ad + - name: campaign_id + description: The unique identifier of the campaign + - name: click_tracking_url + description: The URL for tracking ad clicks + - name: impression_tracking_url + description: The URL for tracking ad impressions + - name: landing_page_url + description: The URL of the landing page for the ad + - name: updated_at + description: The timestamp when the ad was last modified diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/ad_report_hourly.sql b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_report_hourly.sql new file mode 100644 index 00000000..a44286a0 --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_report_hourly.sql @@ -0,0 +1,107 @@ + +{% if target.type == "postgres" %} + + WITH TMP AS ( + SELECT + ad_id, + metrics->>'average_video_play' AS average_video_play, + metrics->>'average_video_play_per_user' AS average_video_play_per_user, + metrics->>'clicks' AS clicks, + metrics->>'comments' AS comments, + metrics->>'conversion' AS conversion, + metrics->>'conversion_rate' AS conversion_rate, + metrics->>'cost_per_conversion' AS cost_per_conversion, + metrics->>'cpc' AS cpc, + metrics->>'cpm' AS cpm, + metrics->>'ctr' AS ctr, + metrics->>'follows' AS follows, + metrics->>'impressions' AS impressions, + metrics->>'likes' AS likes, + metrics->>'profile_visits' AS profile_visits, + metrics->>'reach' AS reach, + metrics->>'shares' AS shares, + metrics->>'spend' AS spend, + stat_time_hour, + metrics->>'video_play_actions' AS video_play_actions, + metrics->>'video_views_p_25' AS video_views_p_25, + metrics->>'video_views_p_50' AS video_views_p_50, + metrics->>'video_views_p_75' AS video_views_p_75, + metrics->>'video_watched_2_s' AS video_watched_2_s, + metrics->>'video_watched_6_s' AS video_watched_6_s + FROM + {{ source('source_tiktok_marketing', 'ads_reports_hourly') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "bigquery" %} + + WITH TMP AS ( + SELECT + ad_id, + cast(JSON_VALUE(metrics, '$.average_video_play') as float64) AS average_video_play, + CAST(JSON_VALUE(metrics, '$.average_video_play_per_user') as float64) AS average_video_play_per_user, + CAST(JSON_VALUE(metrics, '$.clicks') as float64) AS clicks, + cast(JSON_VALUE(metrics, '$.comments') as float64) AS comments, + CAST(JSON_VALUE(metrics, '$.conversion') as float64) AS conversion, + CAST(JSON_VALUE(metrics, '$.conversion_rate') as float64) AS conversion_rate, + CAST(JSON_VALUE(metrics, '$.cost_per_conversion') as float64) AS cost_per_conversion, + CAST(JSON_VALUE(metrics, '$.cpc') as float64) AS cpc, + CAST(JSON_VALUE(metrics, '$.cpm') as float64) AS cpm, + CAST(JSON_VALUE(metrics, '$.ctr') as float64) AS ctr, + CAST(JSON_VALUE(metrics, '$.follows') as float64) AS follows, + CAST(JSON_VALUE(metrics, '$.impressions') as float64) AS impressions, + CAST(JSON_VALUE(metrics, '$.likes') as float64) AS likes, + CAST(JSON_VALUE(metrics, '$.profile_visits') as float64) AS profile_visits, + CAST(JSON_VALUE(metrics, '$.reach') as float64) AS reach, + CAST(JSON_VALUE(metrics, '$.shares') as float64) AS shares, + CAST(JSON_VALUE(metrics, '$.spend') as float64) AS spend, + stat_time_hour, + CAST(JSON_VALUE(metrics, '$.video_play_actions') as float64) AS video_play_actions, + CAST(JSON_VALUE(metrics, '$.video_views_p_25') as float64) AS video_views_p_25, + CAST(JSON_VALUE(metrics, '$.video_views_p_50') as float64) AS video_views_p_50, + CAST(JSON_VALUE(metrics, '$.video_views_p_75') as float64) AS video_views_p_75, + CAST(JSON_VALUE(metrics, '$.video_watched_2_s') as float64) AS video_watched_2_s, + CAST(JSON_VALUE(metrics, '$.video_watched_6_s') as float64) AS video_watched_6_s + FROM + {{ source('source_tiktok_marketing', 'ads_reports_hourly') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "snowflake" %} + + WITH TMP AS ( + SELECT + ad_id, + metrics:"average_video_play"::string AS average_video_play, + metrics:"average_video_play_per_user"::string AS average_video_play_per_user, + metrics:"clicks"::string AS clicks, + metrics:"comments"::string AS comments, + metrics:"conversion"::string AS conversion, + metrics:"conversion_rate"::string AS conversion_rate, + metrics:"cost_per_conversion"::string AS cost_per_conversion, + metrics:"cpc"::string AS cpc, + metrics:"cpm"::string AS cpm, + metrics:"ctr"::string AS ctr, + metrics:"follows"::int AS follows, + metrics:"impressions"::int AS impressions, + metrics:"likes"::int AS likes, + metrics:"profile_visits":: AS profile_visits, + metrics:"reach"::int AS reach, + metrics:"shares"::int AS shares, + metrics:"spend"::int AS spend, + stat_time_hour, + metrics:"video_play_actions"::int AS video_play_actions, + metrics:"video_views_p_25"::int AS video_views_p_25, + metrics:"video_views_p_50"::int AS video_views_p_50, + metrics:"video_views_p_75"::int AS video_views_p_75, + metrics:"video_watched_2_s"::int AS video_watched_2_s, + metrics:"video_watched_6_s"::int AS video_watched_6_s + FROM + {{ source('source_tiktok_marketing', 'ads_reports_hourly') }} + ) + + SELECT * FROM TMP + +{%endif%} diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/ad_report_hourly.yml b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_report_hourly.yml new file mode 100644 index 00000000..5f565062 --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/ad_report_hourly.yml @@ -0,0 +1,62 @@ +version: 2 + +models: + - name: ad_report_hourly + schema: "{{ var('airbyte_schema', target.schema) }}" + database: "{{ var('airbyte_database', target.database) }}" + identifier: "{{ var('tiktok_ads_ad_report_hourly_identifier', 'ad_report_hourly') }}" + description: All fields and field values associated with ad_report_hourly. + config: + +enabled: "{{ var('using_fivetran_model', False) }}" + columns: + - name: ad_id + description: The unique identifier for an ad. + - name: average_video_play + description: The average time your video was played per single video view, including any time spent replaying the video. + - name: average_video_play_per_user + description: The average time per user your video was played per single video view, including any time spent replaying the video. + - name: clicks + description: The number of clicks on your ad. + - name: comments + description: The number of comments your video creative received within 1 day of a user seeing a paid ad. + - name: conversion + description: The number of times your ad achieved an outcome, based on the secondary goal you selected. As one campaign may have a number of different secondary goals, this statistic is not supported for campaigns. Please go to ad groups or ads to view. (The total count is calculated based on the time each ad impression occurred.) + - name: conversion_rate + description: The percentage of results you received out of all the clicks of your ads. (The total count is calculated based on the time each ad impression occurred.) + - name: cost_per_conversion + description: The average amount of money you've spent on a conversion. (The total count is calculated based on the time each ad impression occurred.) + - name: cpc + description: The average amount of money you've spent on a click. + - name: cpm + description: The average amount of money you've spent per 1,000 impressions. + - name: ctr + description: The percentage of times people saw your ad and performed a click. + - name: follows + description: The number of new followers that were gained within 1 day of a user seeing a paid ad. This metric is only for Boosted TikToks. + - name: impressions + description: The number of times your ads were on screen. + - name: likes + description: The number of likes your video creative received within 1 day of a user seeing a paid ad. + - name: profile_visits + description: The number of profile visits the ad drove during the campaign. This metric is only for Boosted TikToks. + - name: reach + description: The number of unique users who saw your ads at least once. This metric is estimated. + - name: shares + description: The number of shares your video creative received within 1 day of a user seeing a paid ad. + - name: spend + description: The estimated total amount of money you've spent on your campaign, ad group or ad during its schedule. + - name: stat_time_hour + description: The hour of the day for which the statistical data is recorded. + - name: video_play_actions + description: The number of times your video starts to play. Replays will not be counted. + - name: video_views_p_25 + description: The number of times your video was played at 25% of its length. Replays will not be counted. + - name: video_views_p_50 + description: The number of times your video was played at 50% of its length. Replays will not be counted. + - name: video_views_p_75 + description: The number of times your video was played at 75% of its length. Replays will not be counted. + - name: video_watched_2_s + description: The number of times your video played for at least 2 seconds. Replays will not be counted. + - name: video_watched_6_s + description: The number of times your video played for at least 6 seconds, or completely played. Replays will not be counted. + \ No newline at end of file diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/advertiser.sql b/connectors/source_tiktok_marketing/models/fivetran_converter/advertiser.sql new file mode 100644 index 00000000..5a38e2e6 --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/advertiser.sql @@ -0,0 +1,89 @@ + +{% if target.type == "postgres" %} + + WITH TMP AS ( + SELECT + + address, + advertiser_id AS id, + balance, + cellphone_number, + cellphone_number AS phone_number, + company, + contacter, + country, + currency, + description, + email, + industry, + language, + name, + telephone_number AS telephone, + telephone_number, + timezone + + FROM + {{ source('source_tiktok_marketing', 'advertisers') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "bigquery" %} + + WITH TMP AS ( + SELECT + + address, + advertiser_id AS id, + balance, + cellphone_number, + cellphone_number AS phone_number, + company, + contacter, + country, + currency, + description, + email, + industry, + language, + name, + telephone_number AS telephone, + telephone_number, + timezone + + FROM + {{ source('source_tiktok_marketing', 'advertisers') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "snowflake" %} + + WITH TMP AS ( + SELECT + + address, + advertiser_id AS id, + balance, + cellphone_number, + cellphone_number AS phone_number, + company, + contacter, + country, + currency, + description, + email, + industry, + language, + name, + telephone_number AS telephone, + telephone_number, + timezone + + FROM + {{ source('source_tiktok_marketing', 'advertisers') }} + ) + + SELECT * FROM TMP + +{%endif%} diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/advertiser.yml b/connectors/source_tiktok_marketing/models/fivetran_converter/advertiser.yml new file mode 100644 index 00000000..1244443f --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/advertiser.yml @@ -0,0 +1,45 @@ +version: 2 + +models: + - name: advertiser + schema: "{{ var('airbyte_schema', target.schema) }}" + database: "{{ var('airbyte_database', target.database) }}" + identifier: "{{ var('tiktok_ads_advertiser_identifier', 'advertiser') }}" + description: All fields and field values associated with advertiser. + config: + +enabled: "{{ var('using_fivetran_model', False) }}" + columns: + - name: address + description: The physical address of the advertiser. + - name: id + description: Unique identifier for the advertiser. + - name: balance + description: The current balance in the advertiser's account. + - name: cellphone_number + description: The cellphone number of the advertiser. + - name: phone_number + description: The cellphone number of the advertiser. + - name: company + description: The name of the company associated with the advertiser. + - name: contacter + description: The contact person for the advertiser. + - name: country + description: The country where the advertiser is located. + - name: currency + description: The currency used for transactions in the account. + - name: description + description: A brief description or bio of the advertiser or company. + - name: email + description: The email address associated with the advertiser. + - name: industry + description: The industry or sector the advertiser operates in. + - name: language + description: The preferred language of communication for the advertiser. + - name: name + description: The name of the advertiser or company. + - name: telephone + description: The telephone number of the advertiser. + - name: telephone_number + description: The telephone number of the advertiser. + - name: timezone + description: The timezone setting for the advertiser's activities. diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/campaign_history.sql b/connectors/source_tiktok_marketing/models/fivetran_converter/campaign_history.sql new file mode 100644 index 00000000..ed037b90 --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/campaign_history.sql @@ -0,0 +1,56 @@ + +{% if target.type == "postgres" %} + + WITH TMP AS ( + SELECT + + advertiser_id, + campaign_id, + campaign_name, + campaign_type, + split_test_variable, + modify_time AS updated_at + + FROM + {{ source('source_tiktok_marketing', 'campaigns') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "bigquery" %} + + WITH TMP AS ( + SELECT + + advertiser_id, + campaign_id, + campaign_name, + campaign_type, + split_test_variable, + modify_time AS updated_at + + FROM + {{ source('source_tiktok_marketing', 'campaigns') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "snowflake" %} + + WITH TMP AS ( + SELECT + + advertiser_id, + campaign_id, + campaign_name, + campaign_type, + split_test_variable, + modify_time AS updated_at + + FROM + {{ source('source_tiktok_marketing', 'campaigns') }} + ) + + SELECT * FROM TMP + +{%endif%} diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/campaign_history.yml b/connectors/source_tiktok_marketing/models/fivetran_converter/campaign_history.yml new file mode 100644 index 00000000..02f310cd --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/campaign_history.yml @@ -0,0 +1,23 @@ +version: 2 + +models: + - name: campaign_history + schema: "{{ var('airbyte_schema', target.schema) }}" + database: "{{ var('airbyte_database', target.database) }}" + identifier: "{{ var('tiktok_ads_campaign_history_identifier', 'campaign_history') }}" + description: All fields and field values associated with campaign_history. + config: + +enabled: "{{ var('using_fivetran_model', False) }}" + columns: + - name: advertiser_id + description: The unique identifier of the advertiser associated with the campaign + - name: campaign_id + description: The unique identifier of the campaign + - name: campaign_name + description: Name of the campaign for easy identification + - name: campaign_type + description: Type of campaign (e.g., awareness, conversion) + - name: split_test_variable + description: Variable being tested in a split test campaign + - name: updated_at + description: Timestamp when the campaign was last modified \ No newline at end of file diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/campaign_report_hourly.sql b/connectors/source_tiktok_marketing/models/fivetran_converter/campaign_report_hourly.sql new file mode 100644 index 00000000..74ef8c30 --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/campaign_report_hourly.sql @@ -0,0 +1,107 @@ + +{% if target.type == "postgres" %} + + WITH TMP AS ( + SELECT + campaign_id, + metrics->>'average_video_play' AS average_video_play, + metrics->>'average_video_play_per_user' AS average_video_play_per_user, + metrics->>'clicks' AS clicks, + metrics->>'comments' AS comments, + metrics->>'conversion' AS conversion, + metrics->>'conversion_rate' AS conversion_rate, + metrics->>'cost_per_conversion' AS cost_per_conversion, + metrics->>'cpc' AS cpc, + metrics->>'cpm' AS cpm, + metrics->>'ctr' AS ctr, + metrics->>'follows' AS follows, + metrics->>'impressions' AS impressions, + metrics->>'likes' AS likes, + metrics->>'profile_visits' AS profile_visits, + metrics->>'reach' AS reach, + metrics->>'shares' AS shares, + metrics->>'spend' AS spend, + stat_time_hour, + metrics->>'video_play_actions' AS video_play_actions, + metrics->>'video_views_p_25' AS video_views_p_25, + metrics->>'video_views_p_50' AS video_views_p_50, + metrics->>'video_views_p_75' AS video_views_p_75, + metrics->>'video_watched_2_s' AS video_watched_2_s, + metrics->>'video_watched_6_s' AS video_watched_6_s + FROM + {{ source('source_tiktok_marketing', 'campaigns_reports_hourly') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "bigquery" %} + + WITH TMP AS ( + SELECT + campaign_id, + CAST(JSON_VALUE(metrics, '$.average_video_play') as float64) AS average_video_play, + CAST(JSON_VALUE(metrics, '$.average_video_play_per_user') as float64) AS average_video_play_per_user, + CAST(JSON_VALUE(metrics, '$.clicks') as float64) AS clicks, + CAST(JSON_VALUE(metrics, '$.comments') as float64) AS comments, + CAST(JSON_VALUE(metrics, '$.conversion') as float64) AS conversion, + CAST(JSON_VALUE(metrics, '$.conversion_rate') as float64) AS conversion_rate, + CAST(JSON_VALUE(metrics, '$.cost_per_conversion') as float64) AS cost_per_conversion, + CAST(JSON_VALUE(metrics, '$.cpc') as float64) AS cpc, + CAST(JSON_VALUE(metrics, '$.cpm') as float64) AS cpm, + CAST(JSON_VALUE(metrics, '$.ctr') as float64) AS ctr, + CAST(JSON_VALUE(metrics, '$.follows') as float64) AS follows, + CAST(JSON_VALUE(metrics, '$.impressions') as float64) AS impressions, + CAST(JSON_VALUE(metrics, '$.likes') as float64) AS likes, + CAST(JSON_VALUE(metrics, '$.profile_visits') as float64) AS profile_visits, + CAST(JSON_VALUE(metrics, '$.reach') as float64) AS reach, + CAST(JSON_VALUE(metrics, '$.shares') as float64) AS shares, + CAST(JSON_VALUE(metrics, '$.spend') as float64) AS spend, + stat_time_hour, + CAST(JSON_VALUE(metrics, '$.video_play_actions') as float64) AS video_play_actions, + CAST(JSON_VALUE(metrics, '$.video_views_p_25') as float64) AS video_views_p_25, + CAST(JSON_VALUE(metrics, '$.video_views_p_50') as float64) AS video_views_p_50, + CAST(JSON_VALUE(metrics, '$.video_views_p_75') as float64) AS video_views_p_75, + CAST(JSON_VALUE(metrics, '$.video_watched_2_s') as float64) AS video_watched_2_s, + CAST(JSON_VALUE(metrics, '$.video_watched_6_s') as float64) AS video_watched_6_s + FROM + {{ source('source_tiktok_marketing', 'campaigns_reports_hourly') }} + ) + + SELECT * FROM TMP + +{% elif target.type == "snowflake" %} + + WITH TMP AS ( + SELECT + campaign_id, + metrics:"average_video_play"::string AS average_video_play, + metrics:"average_video_play_per_user"::string AS average_video_play_per_user, + metrics:"clicks"::string AS clicks, + metrics:"comments"::string AS comments, + metrics:"conversion"::string AS conversion, + metrics:"conversion_rate"::string AS conversion_rate, + metrics:"cost_per_conversion"::string AS cost_per_conversion, + metrics:"cpc"::string AS cpc, + metrics:"cpm"::string AS cpm, + metrics:"ctr"::string AS ctr, + metrics:"follows"::string AS follows, + metrics:"impressions"::string AS impressions, + metrics:"likes"::string AS likes, + metrics:"profile_visits"::string AS profile_visits, + metrics:"reach"::string AS reach, + metrics:"shares"::string AS shares, + metrics:"spend"::string AS spend, + stat_time_hour, + metrics:"video_play_actions"::string AS video_play_actions, + metrics:"video_views_p_25"::string AS video_views_p_25, + metrics:"video_views_p_50"::string AS video_views_p_50, + metrics:"video_views_p_75"::string AS video_views_p_75, + metrics:"video_watched_2_s"::string AS video_watched_2_s, + metrics:"video_watched_6_s"::string AS video_watched_6_s + FROM + {{ source('source_tiktok_marketing', 'campaigns_reports_hourly') }} + ) + + SELECT * FROM TMP + +{%endif%} diff --git a/connectors/source_tiktok_marketing/models/fivetran_converter/campaign_report_hourly.yml b/connectors/source_tiktok_marketing/models/fivetran_converter/campaign_report_hourly.yml new file mode 100644 index 00000000..e9375a05 --- /dev/null +++ b/connectors/source_tiktok_marketing/models/fivetran_converter/campaign_report_hourly.yml @@ -0,0 +1,62 @@ +version: 2 + +models: + - name: campaign_report_hourly + schema: "{{ var('airbyte_schema', target.schema) }}" + database: "{{ var('airbyte_database', target.database) }}" + identifier: "{{ var('tiktok_ads_campaign_report_hourly_identifier', 'campaign_report_hourly') }}" + description: All fields and field values associated with campaign_report_hourly. + config: + +enabled: "{{ var('using_fivetran_model', False) }}" + columns: + - name: campaign_id + description: The unique identifier for a campaign. + - name: average_video_play + description: The average time your video was played per single video view, including any time spent replaying the video. + - name: average_video_play_per_user + description: The average time per user your video was played per single video view, including any time spent replaying the video. + - name: clicks + description: The number of clicks on your ad. + - name: comments + description: The number of comments your video creative received within 1 day of a user seeing a paid ad. + - name: conversion + description: The number of times your ad achieved an outcome, based on the secondary goal you selected. As one campaign may have a number of different secondary goals, this statistic is not supported for campaigns. Please go to ad groups or ads to view. (The total count is calculated based on the time each ad impression occurred.) + - name: conversion_rate + description: The percentage of results you received out of all the clicks of your ads. (The total count is calculated based on the time each ad impression occurred.) + - name: cost_per_conversion + description: The average amount of money you've spent on a conversion. (The total count is calculated based on the time each ad impression occurred.) + - name: cpc + description: The average amount of money you've spent on a click. + - name: cpm + description: The average amount of money you've spent per 1,000 impressions. + - name: ctr + description: The percentage of times people saw your ad and performed a click. + - name: follows + description: The number of new followers that were gained within 1 day of a user seeing a paid ad. This metric is only for Boosted TikToks. + - name: impressions + description: The number of times your ads were on screen. + - name: likes + description: The number of likes your video creative received within 1 day of a user seeing a paid ad. + - name: profile_visits + description: The number of profile visits the ad drove during the campaign. This metric is only for Boosted TikToks. + - name: reach + description: The number of unique users who saw your ads at least once. This metric is estimated. + - name: shares + description: The number of shares your video creative received within 1 day of a user seeing a paid ad. + - name: spend + description: The estimated total amount of money you've spent on your campaign, ad group or ad during its schedule. + - name: stat_time_hour + description: The hour of the day for which the statistical data is recorded. + - name: video_play_actions + description: The number of times your video starts to play. Replays will not be counted. + - name: video_views_p_25 + description: The number of times your video was played at 25% of its length. Replays will not be counted. + - name: video_views_p_50 + description: The number of times your video was played at 50% of its length. Replays will not be counted. + - name: video_views_p_75 + description: The number of times your video was played at 75% of its length. Replays will not be counted. + - name: video_watched_2_s + description: The number of times your video played for at least 2 seconds. Replays will not be counted. + - name: video_watched_6_s + description: The number of times your video played for at least 6 seconds, or completely played. Replays will not be counted. + \ No newline at end of file