Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use lead for end-date in spine + document empty opportunity table solution #48

Merged
merged 9 commits into from
Feb 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/workflows/auto-release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
name: 'auto release'
on:
pull_request:
types:
- closed
branches:
- main

jobs:
call-workflow-passing-data:
if: github.event.pull_request.merged
uses: fivetran/dbt_package_automations/.github/workflows/auto-release.yml@main
secrets: inherit
26 changes: 26 additions & 0 deletions .quickstart/quickstart.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
database_key: salesforce_database
schema_key: salesforce_schema

dbt_versions: ">=1.3.0 <2.0.0"

table_variables:
salesforce__user_role_enabled:
- user_role
salesforce__lead_enabled:
- lead
salesforce__event_enabled:
- event
salesforce__task_enabled:
- task
salesforce__opportunity_line_item_enabled:
- opportunity_line_item
salesforce__order_enabled:
- order
salesforce__product_2_enabled:
- product_2

destination_configurations:
databricks:
dispatch:
- macro_namespace: dbt_utils
search_order: [ 'spark_utils', 'dbt_utils' ]
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,16 @@
# dbt_salesforce v1.0.1
[PR #48](https://github.com/fivetran/dbt_salesforce/pull/48) includes the following updates:

## Bug Fix
- Aligns the `last_date_query` logic in the `int_salesforce__date_spine` model with the `first_date_query` logic. This ensures that users with empty `opportunity` tables will have a valid end-date (based on `lead` instead of `opportunity`) for the `salesforce__daily_activity` end model.
- Also adds coalesce-logic to `int_salesforce__date_spine` to ensure a succesful run without `lead` data.

## Documentation
- Documents how users without an `opportunity` table can still have the package run successfully for them. See [README](https://github.com/fivetran/dbt_salesforce?tab=readme-ov-file#working-without-an-opportunity-table) for details.

## Under the Hood
- Included auto-releaser GitHub Actions workflow to automate future releases.

# 🚨 Notice for Quickstart Data Model Users Only 🚨
Please note that this data model will now create a new transformation for **all** your Salesforce objects (tables) to replicate and include the relevant Salesforce formula fields. With the addition of formula fields, your transformation schema will change to <connector_schema> + `_quickstart`, rather than inheriting the schema from your connector. Please make sure you adjust downstream queries accordingly. If you wish to disable any of these new transformations you may remove them within the UI.

Expand Down
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,20 @@ vars:
```
The corresponding metrics from the disabled tables will not populate in the downstream models.

### Working without an `OPPORTUNITY` Table
If you do not have the `OPPORTUNITY` table, there is no variable to turn off opportunity-related transformations, as this table is largely the backbone of the Salesforce package.

However, you may still find value in this package without opportunity data, specifically in the `salesforce__contact_enhanced`, `salesforce__daily_activity`, `salesforce__account_daily_history` and `salesforce__contact_daily_history` (if using History Mode) end models.

For this use case, to ensure the package runs successfully, we recommend leveraging this [Fivetran Feature](https://fivetran.com/docs/using-fivetran/features#syncingemptytablesandcolumns) to create an empty `opportunity` table. To do so, follow these steps:
1. Navigate to your Salesforce connector in the "Connectors" tab within the Fivetran UI.
2. Click on the "Schema" tab.
3. Scroll down to `Opportunity` and click on its checkbox to add it into your schema.
4. Click "Save Changes" in the upper righthand corner of the screen.
5. Either click "Resync" for the `Opportunity` table specifically or wait for your next connector-level sync.

> Note that all other end models (`salesforce__opportunity_enhanced`, `salesforce__opportunity_line_item_enhanced`, `salesforce__manager_performance`, `salesforce__owner_performance`, `salesforce__sales_snapshot`, and `salesforce__opportunity_daily_history`) will still materialize after a blanket `dbt run` but will be largely empty/null.

## (Optional) Step 4: Utilizing Salesforce History Mode records
If you have Salesforce [History Mode](https://fivetran.com/docs/getting-started/feature/history-mode) enabled for your connector, we now include support for the `account`, `contact`, and `opportunity` tables directly. These staging models from our `dbt_salesforce_source` package flow into our daily history models. This will allow you access to your historical data for these tables while taking advantage of incremental loads to help with compute.

Expand Down
2 changes: 1 addition & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
config-version: 2
name: 'salesforce'
version: '1.0.0'
version: '1.0.1'
require-dbt-version: [">=1.3.0", "<2.0.0"]
models:
salesforce:
Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions docs/index.html

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/run_results.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion integration_tests/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'salesforce_integration_tests'
version: '1.0.0'
version: '1.0.1'
config-version: 2

profile: 'integration_tests'
Expand Down
9 changes: 7 additions & 2 deletions models/salesforce/intermediate/int_salesforce__date_spine.sql
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ with spine as (
{% if var('salesforce__lead_enabled', True) %}
select min( created_date ) as min_date from {{ source('salesforce', 'lead') }}
{% else %}
select min( created_date ) as min_date from {{ source('salesforce', 'opportunity') }}
select coalesce(min( created_date ), '2015-01-01') as min_date from {{ source('salesforce', 'opportunity') }}
{% endif %}

{% endset %}
Expand All @@ -25,7 +25,12 @@ with spine as (

{% if execute %}
{% set last_date_query %}
select max( created_date ) as max_date from {{ source('salesforce', 'opportunity') }}
{% if var('salesforce__lead_enabled', True) %}
select max( created_date ) as max_date from {{ source('salesforce', 'lead') }}
{% else %}
select coalesce(max( created_date ), '2024-01-01') as max_date from {{ source('salesforce', 'opportunity') }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we consider changing the default to just be the current date instead of hard coding 2024-01-01

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah because of this line, we end up choosing the current date anyways!

but perhaps this whole snippet is ultimately kinda roundabout and unnecessary -- is the last_date ever going to be greater than the current date, or should we just automatically choose the current_date? changing it entirely might be venturing out of scope i suppose though

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh okay I see now. Yeah this is a bit roundabout and not entirely necessary. I think we can open a FR to clean up this logic in the future to make it more streamlined. I would agree that this is a bit out of scope for the final hours of the sprint.

This looks good to go!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FR made! #49

{% endif %}

{% endset %}

{% set current_date_query %}
Expand Down
Loading