v0.15.0 dbt_hubspot_source

PR #126 includes the following updates:

🚨 Breaking Changes 🚨

• Added field _fivetran_end to macro get_ticket_property_history_columns() to ensure the column is available for use downstream.
  • While this should not affect most users, this will add a new column _fivetran_end to stg_hubspot__ticket_property_history if you do not have _fivetran_end in your source TICKET_PROPERTY_HISTORY table.

Documentation

• Update documentation to include _fivetran_end under model stg_hubspot__ticket_property_history.

Under the hood

• Updated the maintainer PR template to the current format.
• Included auto-releaser GitHub Actions workflow to automate future releases.
• Added integration testing for Databricks SQL Warehouse.

Full Changelog: v0.14.1...v0.15.0

v0.14.1 dbt_hubspot_source

PR #125 includes the following updates:

Features

• Adds the stg_hubspot__merged_deal model. Downstream, this model is used to remove stale deals and aggregate them to the deal they have been merged into. This is disabled by default as not every user has the merged_deal table.
  • To enable this model and filter out merged deals downstream, make sure the following is set to true in your dbt_project.yml: hubspot_sales_enabled, hubspot_deal_enabled, and hubspot_merged_deal_enabled. See Step 4 of the README for more details.

Full Changelog: v0.14.0...v0.14.1

v0.14.0 dbt_hubspot_source

PR #122 includes the following updates:

Features

• Added the following staging models, along with documentation and tests:

  • stg_hubspot__property
  • stg_hubspot__property_option
  • These tables can be disabled by setting hubspot_property_enabled: False in your dbt_project.yml vars. See Step 4 of the README for more details.

• When including a passthrough property_hs_* column, you now have the option to include the corresponding, human-readable label in the staging models.

  • The above-mentioned property tables are required for this feature. If you do not have them and have to disable them, unfortunately you will not be able to use this feature.
  • See the Adding property label section of the README for instructions on how to enable this feature!
  • We recommend being selective with the label columns you add. As you add more label columns, your run time will increase due to the underlying logic requirements.
  • This update applies to models:
    • stg_hubspot__company
    • stg_hubspot__contact
    • stg_hubspot__deal
    • stg_hubspot__ticket

Bug fixes

• Updated macro remove_duplicate_and_prefix_from_columns to accommodate incoming custom column names containing characters such as - or $ that are not permitted. The resulting column name will have these characters removed or replaced in its stg_* model.
• Removed extra comma from stg_hubspot__ticket, which was causing compilation issues when passing through all columns.

Full Changelog: v0.13.0...v0.14.0

v0.13.0 dbt_hubspot_source

🚨 Breaking Changes 🚨

• The created_at and closed_at fields in the below mentioned staging models have been renamed to created_date and closed_date respectively to be consistent with the source data. Additionally, this will ensure there are no duplicate column errors when passing through all property_* columns, which could potentially conflict with property_created_at or property_closed_at (PR #119).
  • stg_hubspot__company
  • stg_hubspot__contact
  • stg_hubspot__deal
  • stg_hubspot__ticket

Features

• Addition of the following variables to allow the disabling of the *_property_history models if they are not being leveraged. All variables are true by default (PR #119).
  • hubspot_company_property_history_enabled
  • hubspot_contact_property_history_enabled
  • hubspot_deal_property_history_enabled

Under the Hood

• All stg_hubspot__*_tmp models have been updated to leverage the dbt_utils.star() macro. This ensures if the source dimension changes there is no potential for a mismatch in columns error that is commonly seen in Snowflake destinations (PR #119).
• Updates to the seed files and seed file configurations for the package integration tests to ensure updates are properly tested (PR #119).
• V3 of the Hubspot Service endpoint contains the field _fivetran_deleted in the TICKET table, whereas V2 has a field called is_deleted. Previously, the package only looked at is_deleted. Now, it will fold in _fivetran_deleted and coalesce it with is_deleted to more fully represent is_ticket_deleted (PR #120).

Full Changelog: v0.12.0...v0.13.0

Parallel Transform Package Release

v0.12.0 dbt_hubspot_source

🚨 Breaking Changes 🚨

• The following models now use a custom macro to remove the property_hs_ prefix in staging columns, while also preventing duplicates. If de-prefixed columns match existing ones (e.g., property_hs_meeting_outcome vs. meeting_outcome), the macro favors the property_hs_field, aligning with the latest HubSpot API update. (PR #115)
  • stg_hubspot__engagement_call
  • stg_hubspot__engagement_company
  • stg_hubspot__engagement_contact
  • stg_hubspot__engagement_deal
  • stg_hubspot__engagement_email
  • stg_hubspot__engagement_meeting
  • stg_hubspot__engagement_note
  • stg_hubspot__engagement_task
  • stg_hubspot__ticket
  • stg_hubspot__ticket_company
  • stg_hubspot__ticket_contact
  • stg_hubspot__ticket_deal
  • stg_hubspot__ticket_engagement
  • stg_hubspot__ticket_property_history

Feature Updates

• A new macro remove_duplicate_and_prefix_from_columns has been included which expands off the fivetran_utils.remove_prefix_columns macro by removing any duplicate columns that result from the prefix removal. (PR #115 and PR #114)

Contributors

• @greg-finley (PR #114)

Full Changelog: v0.11.0...v0.12.0

v0.11.0 dbt_hubspot_source

PR #112 includes the following updates:

🚨 Breaking Changes 🚨

• Following the May 2023 connector update the HubSpot connector now syncs the below parent and child tables from the new v3 API. As a result the dependent fields and field names from the downstream staging models have changed depending on the fields available in your HubSpot data. Now the respective staging models will sync the required fields for the dbt_hubspot downstream transformations and all of your property_hs_* fields. Please be aware that the property_hs_* will be truncated from the field name in the staging and downstream models. The impacted sources (and relevant staging models) are below:

  - `ENGAGEMENT`
    - `ENGAGEMENT_CALL`
    - `ENGAGEMENT_COMPANY`
    - `ENGAGEMENT_CONTACT`
    - `ENGAGEMENT_DEAL`
    - `ENGAGEMENT_EMAIL`
    - `ENGAGEMENT_MEETING`
    - `ENGAGEMENT_NOTE`
    - `ENGAGEMENT_TASK`
  - `TICKET`
    - `TICKET_COMPANY`
    - `TICKET_CONTACT`
    - `TICKET_DEAL`
    - `TICKET_ENGAGEMENT`
    - `TICKET_PROPERTY_HISTORY`

• Please note that while these changes are breaking, the package has been updated to ensure backwards compatibility with the pre HubSpot v3 API updates. As a result, you may see some null fields which are artifacts of the pre v3 API HubSpot version. Be sure to inspect the relevant field descriptions for an understanding of which fields remain for backwards compatibility purposes. These fields will be removed once all HubSpot connectors are upgraded to the v3 API.

Documentation Updates

• As new fields were added in the v3 API updates, and old fields were removed, the documentation was updated to reflect the v3 API consistent fields. Please take note if you are still using the pre v3 API, you will find the above mentioned source (and respective staging) models no longer have complete field documentation coverage.

Full Changelog: v0.10.0...v0.11.0

v0.10.0 dbt_hubspot_source

This release of dbt_hubspot_source includes:

🚨 Breaking Changes 🚨

• In the May 2023 connector update type_id was added to sources DEAL_COMPANY and DEAL_CONTACT as a part of the composite primary key for these tables. This column has been adding to the corresponding staging models. (PR #109)
• Updated tests for these tables with type_id as part of the primary key. (PR #109)
• Also resulting from the connector update, columns updated_at and created_at have been added to the following sources and their corresponding staging models: (PR #109)
  • DEAL_PIPELINE
  • DEAL_PIPELINE_STAGE
  • TICKET_PIPELINE
  • TICKET_PIPELINE_STAGE
• Updated docs with these changes and a little housekeeping. (PR #109)

Feature Updates

• Updated README to include the variable hubspot_owner_enabled. (PR #109)

🚘 Under the Hood

• Updated seed data for testing newly added columns. (PR #109)

Full Changelog: v0.9.1...v0.10.0

v0.9.1 dbt_hubspot_source

This release of dbt_hubspot_source includes:

Feature Updates

• A new variable was added hubspot_using_all_email_events to allow package users to remove filtered email events from the stg_hubspot__email_event staging model as well as the relevant downstream reporting models. This is crucial for HubSpot users who greatly take advantage of marking events as filtered in order to provide accurate reporting. (PR #104)
  • The hubspot_using_all_email_events variable is true by default. Set the variable to false to filter out specified email events in your staging and downstream models.

Under the Hood

• The email_event_data.csv seed file was updated to include events that are listed as true for filtered_events. This is to effectively test the above mentioned feature update. (PR #104)
• Included hubspot_using_all_email_events: false as a variable declared in the final run_models.sh step to ensure our integration tests gain coverage over this new feature and variable. (PR #104)
• Incorporated the new fivetran_utils.drop_schemas_automation macro into the end of each Buildkite integration test job. (PR #103)
• Updated the pull request templates. (PR #103)

Full Changelog: v0.9.0...v0.9.1

v0.9.0 dbt_hubspot_source

🚨 Breaking Changes 🚨

In November 2022, the Fivetran Hubspot connector switched to v3 of the Hubspot CRM API, which deprecated the CONTACT_MERGE_AUDIT table and stored merged contacts in a field in the CONTACT table. This has not been rolled out to BigQuery warehouses yet. BigQuery connectors with the CONTACT_MERGE_AUDIT table enabled will continue to sync this table until the new CONTACT.property_hs_calculated_merged_vids field and API version becomes available to them.

This release introduces breaking changes around how contacts are merged in order to align with the above connector changes. It is, however, backwards-compatible.

PR #98 applies the following changes:

• Updates logic around the recently deprecated CONTACT_MERGE_AUDIT table.
  • The package now brings in the new property_hs_calculated_merged_vids field (and removes the property_hs_ prefix) for all customers, including those on BigQuery (the field will just be null).
  • Backwards-compatibility: the package will only reference the old CONTACT_MERGE_AUDIT table and create stg_hubspot__contact_merge_audit if hubspot_contact_merge_audit_enabled is explicitly set to true in your root dbt_project.yml file.

Bug Fixes

• The CONTACT, COMPANY, DEAL, and TICKET staging models have been updated to ensure users enabling the hubspot__pass_through_all_columns will have all required columns. (PR #100)

Under the Hood

• Updates seed data to test new merging paradigm. (PR #98)
• Ensures that all timestamp fields are explicitly cast as timestamps without timezone, as recent API changes also introduced inconsistent timestamp formats. (PR #98)
• Creation of the get_macro_columns macro to help perform a check when enabling the hubspot__pass_through_all_columns to ensure the required fields are captured regardless of their existence in the source table. (PR #100)
• Creation of the all_passthrough_column_check macro to help ensure that the operation to bring in all fields for the CONTACT, COMPANY, DEAL, and TICKET staging models is performed only if additional fields from the required are present in the source. (PR #100)

See the transform package CHANGELOG for updates made to end models in dbt_hubspot v0.9.0.

v0.8.0...v0.9.0

dbt_hubspot_source 0.8.0

Happy International Thank You Day -- THANK YOU all for your continued contributions and making our packages better everyday! 🥳

🚨 Breaking Changes 🚨:

PR #96 incorporates the following updates:

• The is_deleted field has been renamed within the below models:

  • stg_hubspot__company ( is_company_deleted)
  • stg_hubspot__deal (is_deal_deleted)
  • stg_hubspot__ticket (is_ticket_deleted)

• The _fivetran_deleted field has been renamed within the below models:

  • stg_husbpot__contact_list_member (is_contact_list_member_deleted)
  • stg_hubspot__contact_list (is_contact_list_deleted)
  • stg_hubspot__contact (is_contact_deleted)
  • stg_hubspot__deal_pipeline_stage (is_deal_pipeline_stage_deleted)
  • stg_hubspot__deal_pipeline (is_deal_pipeline_deleted)
  • stg_hubspot__ticket_pipeline_stage (is_ticket_pipeline_stage_deleted)
  • stg_hubspot__ticket_pipeline (is_ticket_pipeline_deleted)

• Filtering deleted records with the methods where not coalesce(is_deleted, false) or where not coalesce(_fivetran_deleted, false) has been removed from the above models. Instead, the new is_<model>_deleted flags will now offer customers flexibility to filter models for deleted records as necessary.

Under the Hood

• stg*.yml documentation has been updated such that all is_<model>_deleted fields point to the is_deleted definition rather than respective is_deleted and _fivetran_deleted since the two fields are equivalent and in order to maintain consistency. (PR #96).