bug/spine-empty-lead-update (#52)

* bug/spine-empty-lead-update * add tests and changelog * regen docs * Update CHANGELOG.md Co-authored-by: Joe Markiewicz <[email protected]> --------- Co-authored-by: Joe Markiewicz <[email protected]>
fivetran · Jun 17, 2024 · 614880a · 614880a
1 parent abdc7e7
commit 614880a
Show file tree

Hide file tree

Showing 12 changed files with 137 additions and 90 deletions.
diff --git a/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md b/.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md
@@ -4,48 +4,27 @@
 **This PR will result in the following new package version:**
 <!--- Please add details around your decision for breaking vs non-breaking version upgrade. If this is a breaking change, were backwards-compatible options explored? -->
 
-**Please detail what change(s) this PR introduces and any additional information that should be known during the review of this PR:**
+**Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:**
+<!--- Copy/paste the CHANGELOG for this version below. -->
 
 ## PR Checklist
 ### Basic Validation
 Please acknowledge that you have successfully performed the following commands locally:
-- [ ] dbt compile
-- [ ] dbt run –full-refresh
-- [ ] dbt run
-- [ ] dbt test
-- [ ] dbt run –vars (if applicable)
+- [ ] dbt run –full-refresh && dbt test
+- [ ] dbt run (if incremental models are present) && dbt test
 
 Before marking this PR as "ready for review" the following have been applied:
-- [ ] The appropriate issue has been linked and tagged
-- [ ] You are assigned to the corresponding issue and this PR
+- [ ] The appropriate issue has been linked, tagged, and properly assigned
+- [ ] All necessary documentation and version upgrades have been applied
+    <!--- Be sure to update the package version in the dbt_project.yml, integration_tests/dbt_project.yml, and README if necessary. -->
+- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
 - [ ] BuildKite integration tests are passing
+- [ ] Detailed validation steps have been provided below
 
 ### Detailed Validation
-Please acknowledge that the following validation checks have been performed prior to marking this PR as "ready for review":
-- [ ] You have validated these changes and assure this PR will address the respective Issue/Feature.
-- [ ] You are reasonably confident these changes will not impact any other components of this package or any dependent packages.
-- [ ] You have provided details below around the validation steps performed to gain confidence in these changes.
+Please share any and all of your validation steps:
 <!--- Provide the steps you took to validate your changes below. -->
 
-### Standard Updates
-Please acknowledge that your PR contains the following standard updates:
-- Package versioning has been appropriately indexed in the following locations:
-    - [ ] indexed within dbt_project.yml
-    - [ ] indexed within integration_tests/dbt_project.yml
-- [ ] CHANGELOG has individual entries for each respective change in this PR
-    <!--- If there is a parallel upstream change, remember to reference the corresponding CHANGELOG as an individual entry.  -->
-- [ ] README updates have been applied (if applicable)
-    <!--- Remember to check the following README locations for common updates. →
-        <!--- Suggested install range (needed for breaking changes) →
-        <!--- Dependency matrix is appropriately updated (if applicable) →
-        <!--- New variable documentation (if applicable) -->
-- [ ] DECISIONLOG updates have been updated (if applicable)
-- [ ] Appropriate yml documentation has been added (if applicable)
-
-### dbt Docs
-Please acknowledge that after the above were all completed the below were applied to your branch:
-- [ ] docs were regenerated (unless this PR does not include any code or yml updates)
-
 ### If you had to summarize this PR in an emoji, which would it be?
 <!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
-:dancer:
+:dancer:
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,13 @@
+# dbt_salesforce v1.0.2
+[PR #52](https://github.com/fivetran/dbt_salesforce/pull/52) includes the following updates:
+
+## Bug fixes
+- Updated model `int_salesforce__date_spine` to accommodate when the Salesforce `lead` object exists but has no records. In this case, the model now defaults to a range of one-month from the current date.
+
+## Under the hood
+- Updated structure of model `int_salesforce__date_spine` for improved performance and maintainability.
+- Updated maintainer pull request template.
+
 # dbt_salesforce v1.0.1
 [PR #48](https://github.com/fivetran/dbt_salesforce/pull/48) includes the following updates:
 

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,6 +1,6 @@
 config-version: 2
 name: 'salesforce'
-version: '1.0.1'
+version: '1.0.2'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   salesforce:

diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/index.html b/docs/index.html
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/docs/run_results.json b/docs/run_results.json
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,12 +1,13 @@
 name: 'salesforce_integration_tests'
-version: '1.0.1'
+version: '1.0.2'
 config-version: 2
 
 profile: 'integration_tests'
 
 models:
   salesforce:
     materialized: table
+    +schema: "salesforce_{{ var('directed_schema','dev') }}"
 
 vars:
   # enable history models when generating docs!

diff --git a/integration_tests/tests/consistency/consistency_daily_activity.sql b/integration_tests/tests/consistency/consistency_daily_activity.sql
@@ -0,0 +1,32 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test ensures the daily_activity end model matches the prior version
+with prod as (
+    select *
+    from {{ target.schema }}_salesforce_prod.salesforce__daily_activity
+),
+
+dev as (
+    select *
+    from {{ target.schema }}_salesforce_dev.salesforce__daily_activity
+),
+
+final as (
+    -- test will fail if any rows from prod are not found in dev
+    (select * from prod
+    except distinct
+    select * from dev)
+
+    union all -- union since we only care if rows are produced
+
+    -- test will fail if any rows from dev are not found in prod
+    (select * from dev
+    except distinct
+    select * from prod)
+)
+
+select *
+from final
diff --git a/integration_tests/tests/consistency/consistency_daily_activity_count.sql b/integration_tests/tests/consistency/consistency_daily_activity_count.sql
@@ -0,0 +1,21 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test is to make sure the rows counts are the same between versions
+with prod as (
+    select count(*) as prod_rows
+    from {{ target.schema }}_salesforce_prod.salesforce__daily_activity
+),
+
+dev as (
+    select count(*) as dev_rows
+    from {{ target.schema }}_salesforce_dev.salesforce__daily_activity
+)
+
+-- test will return values and fail if the row counts don't match
+select *
+from prod
+join dev
+    on prod.prod_rows != dev.dev_rows
diff --git a/integration_tests/tests/integrity/integrity_daily_activity.sql b/integration_tests/tests/integrity/integrity_daily_activity.sql
@@ -0,0 +1,21 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test is to make sure there is no fanout between the spine and the daily_activity
+with spine as (
+    select count(*) as spine_count
+    from {{ target.schema }}_salesforce_dev.int_salesforce__date_spine
+),
+
+daily_activity as (
+    select count(*) as daily_activity_count
+    from {{ target.schema }}_salesforce_dev.salesforce__daily_activity
+)
+
+-- test will return values and fail if the row counts don't match
+select *
+from spine
+join daily_activity
+    on spine.spine_count != daily_activity.daily_activity_count
diff --git a/models/salesforce/intermediate/int_salesforce__date_spine.sql b/models/salesforce/intermediate/int_salesforce__date_spine.sql
@@ -1,68 +1,51 @@
-
 with spine as (
 
     {% if execute %}
-    {% set first_date_query %}
-        {% if var('salesforce__lead_enabled', True) %}
-            select  min( created_date ) as min_date from {{ source('salesforce', 'lead') }}
-        {% else %}
-            select  coalesce(min( created_date ), '2015-01-01') as min_date from {{ source('salesforce', 'opportunity') }}
-        {% endif %}   
-
-    {% endset %}
-    {% set first_date = run_query(first_date_query).columns[0][0]|string %}
-
-        {% if target.type == 'postgres' %}
-            {% set first_date_adjust = "cast('" ~ first_date[0:10] ~ "' as date)" %}
 
+    {%- set first_date_query %}
+        select 
+            coalesce(
+                min(cast(created_date as date)), 
+                cast({{ dbt.dateadd("month", -1, "current_date") }} as date)
+                ) as min_date
+        {% if var('salesforce__lead_enabled', True) %}
+            from {{ source('salesforce', 'lead') }}
         {% else %}
-            {% set first_date_adjust = "'" ~ first_date[0:10] ~ "'" %}
+            from {{ source('salesforce', 'opportunity') }}
+        {% endif %}  
+    {% endset -%}
 
-        {% endif %}
+    {%- set first_date = dbt_utils.get_single_value(first_date_query) %}
 
-    {% else %} {% set first_date_adjust = "'2015-01-01'" %}
-    {% endif %}
-
-    {% if execute %}
     {% set last_date_query %}
+        select 
+            coalesce(
+                greatest(max(cast(created_date as date)), cast(current_date as date)),
+                cast(current_date as date)
+                ) as max_date
         {% if var('salesforce__lead_enabled', True) %}
-            select  max( created_date ) as max_date from {{ source('salesforce', 'lead') }}
+            from {{ source('salesforce', 'lead') }}
         {% else %}
-        select  coalesce(max( created_date ), '2024-01-01') as max_date from {{ source('salesforce', 'opportunity') }}
-        {% endif %}
-
-    {% endset %}
+            from {{ source('salesforce', 'opportunity') }}
+        {% endif %}  
+    {% endset -%}
 
-    {% set current_date_query %}
-        select current_date
-    {% endset %}
-
-    {% if run_query(current_date_query).columns[0][0]|string < run_query(last_date_query).columns[0][0]|string %}
-
-    {% set last_date = run_query(last_date_query).columns[0][0]|string %}
-
-    {% else %} {% set last_date = run_query(current_date_query).columns[0][0]|string %}
-    {% endif %}
-
-    {% if target.type == 'postgres' %}
-        {% set last_date_adjust = "cast('" ~ last_date[0:10] ~ "' as date)" %}
+    {%- set last_date = dbt_utils.get_single_value(last_date_query) %}
 
     {% else %}
-        {% set last_date_adjust = "'" ~ last_date[0:10] ~ "'" %}
 
-    {% endif %}
+    {% set first_date = 'dbt.dateadd("month", -1, "current_date")' %}
+    {% set last_date = 'dbt.current_timestamp_backcompat()' %}
+
     {% endif %}
 
     {{ dbt_utils.date_spine(
         datepart="day",
-        start_date=first_date_adjust,
-        end_date=dbt.dateadd("day", 1, last_date_adjust)
+        start_date="cast('" ~ first_date ~ "' as date)",
+        end_date=dbt.dateadd("day", 1, "cast('" ~ last_date  ~ "' as date)")
         )
     }}
 )
 
-select 
-
-distinct(date_day)
-
+select * 
 from spine