Merge pull request #143 from fivetran/bugfix/filter-deleted-accounts

Filter out deleted accounts

CHANGELOG.md

@@ -1,3 +1,23 @@
+# dbt_quickbooks v0.16.0
+[PR #143](https://github.com/fivetran/dbt_quickbooks/pull/143) introduces the following updates:
+
+## Upstream Source Package Updates
+- Deleted records have been previously been brought into the `dbt_quickbooks` package. To ensure accuracy in reporting, the following updates were made in the [v0.11.0 release](https://github.com/fivetran/dbt_quickbooks_source/releases/tag/v0.11.0) of `dbt_quickbooks_source`:
+- We introduced the `_fivetran_deleted` field to filter out deleted records from the following staging models:
+  - `stg_quickbooks__account`
+  - `stg_quickbooks__bundle`
+  - `stg_quickbooks__customer`
+  - `stg_quickbooks__department`
+  - `stg_quickbooks__item`
+  - `stg_quickbooks__vendor`
+- Since filtering out deleted records that were previously being counted impact all output models, this is being treated as a breaking change.
+
+## Documentation update
+- Added the `_fivetran_deleted` field to the above corresponding seed files in integration tests.
+
+## Under The Hood
+- Updated the `consistency_*_amounts` tests to include the `converted_amount` comparisons. They were previously commented out due to introducing multicurrency support in a previous release that would have led to test failures, but can now be brought in to properly validate these changes.
+
 # dbt_quickbooks v0.15.0
 [PR #142](https://github.com/fivetran/dbt_quickbooks/pull/142) introduces the following updates:
 

README.md

@@ -73,7 +73,7 @@ Include the following QuickBooks package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/quickbooks
-    version: [">=0.15.0", "<0.16.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.16.0", "<0.17.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 Do NOT include the `quickbooks_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
@@ -163,9 +163,7 @@ These are our recommended best practices to follow with your seed file (you can
 - REQUIRED: Every row should have a non-null `ordinal` and `cash_flow_type` column value. 
 - REQUIRED: In each row of the seed file, only populate **ONE** of the `account_class`, `account_type`, `account_sub_type`, and `account_number` columns to avoid duplicated ordinals and cash flow types and test failures. This should also make the logic cleaner in defining which account value takes precedence in the ordering hierarchy. 
 - In `cash_flow_statement_type_ordinal_example`, we recommend creating ordinals for each `cash_flow_type` value available (the default types are `Cash or Cash Equivalents`, `Operating`, `Investing`, `Financing` as per best financial practices, but you can configure as you like in your seed file) to make sure each cash flow statement type can be easily ordered. Then you can create any additional customization as needed with the more specific account fields to order even further.   
-- In `cash_flow_statement_type_ordinal_example`, the `report` field should always be `Cash Flow`.  
-
-We'd love for you to share your experiences with the cash flow seed file with us [in the Fivetran community user group](https://community.fivetran.com/t5/user-group-for-dbt/gh-p/dbt-user-group) so we can make these model and seed configurations even better for you in the future!.
+- In `cash_flow_statement_type_ordinal_example`, the `report` field should always be `Cash Flow`.
 
 ### Customize the account ordering of your financial models. 
 [The current default numbering for ordinals](https://github.com/fivetran/dbt_quickbooks/blob/main/models/quickbooks__general_ledger_by_period.sql#L44-L50) is based on best practices for balance sheets and profit-and-loss statements in accounting. You can see these ordinals in action in the `quickbooks__general_ledger_by_period`, `quickbooks__balance_sheet` and `quickbooks__profit_and_loss` models. The ordinals are assigned off of the `account_class` values.
@@ -189,8 +187,6 @@ These are our recommended best practices to follow with your seed file ([you can
 - Fill out the `report` field as either `Balance Sheet` if the particular row belongs in `quickbooks__balance_sheet`, or `Profit and Loss` for `quickbooks__profit_and_loss`.
 - We recommend ordering the `ordinal` for each report separately in the seed, i.e. have ordinals for `quickbooks__balance_sheet` and `quickbooks__profit_and_loss` start at 1 each, to make your reporting more clean.
 
-We'd love for you to share your experiences with the ordinal seed file with us [in the Fivetran community user group](https://community.fivetran.com/t5/user-group-for-dbt/gh-p/dbt-user-group) so we can make these model and seed configurations even better for you in the future.
-
 #### Changing the Build Schema
 By default this package will build the QuickBooks staging models within a schema titled (<target_schema> + `_quickbooks_staging`), QuickBooks intermediate (particularly the double entry) models within a schema titled (<target_schema> + `_quickbooks_intermediate`), and QuickBooks final models within a schema titled (<target_schema> + `_quickbooks`) in your target database. If this is not where you would like your modeled QuickBooks data to be written to, add the following configuration to your `dbt_project.yml` file:
 
@@ -230,7 +226,7 @@ This dbt package is dependent on the following dbt packages. These dependencies
 ```yml
 packages:
     - package: fivetran/quickbooks_source
-      version: [">=0.10.0", "<0.11.0"]
+      version: [">=0.11.0", "<0.12.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]
@@ -253,4 +249,4 @@ This dbt package takes an opinionated stance on how to define the ordering and c
 
 ## Are there any resources available?
 - If you have questions or want to reach out for help, see the [GitHub Issue](https://github.com/fivetran/dbt_quickbooks/issues/new/choose) section to find the right avenue of support for you.
-- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
+- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).

dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 name: 'quickbooks'
 
-version: '0.15.0'
+version: '0.16.0'
 
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 

integration_tests/ci/sample.profiles.yml

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: quickbooks_integration_tests_15
+      schema: quickbooks_integration_tests_18
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: quickbooks_integration_tests_15
+      schema: quickbooks_integration_tests_16
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: quickbooks_integration_tests_15
+      schema: quickbooks_integration_tests_18
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: quickbooks_integration_tests_15
+      schema: quickbooks_integration_tests_18
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: quickbooks_integration_tests_15
+      schema: quickbooks_integration_tests_18
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

integration_tests/dbt_project.yml

@@ -1,6 +1,6 @@
 name: 'quickbooks_integration_tests'
 
-version: '0.15.0'
+version: '0.16.0'
 
 profile: 'integration_tests'
 config-version: 2
@@ -21,7 +21,7 @@ models:
 
 vars:
   quickbooks_source:
-    quickbooks_schema: quickbooks_integration_tests_15
+    quickbooks_schema: quickbooks_integration_tests_18
     quickbooks_account_identifier: "account_data"
     quickbooks_address_identifier: "address_data"
     quickbooks_bill_line_identifier: "bill_line_data"
@@ -66,8 +66,8 @@ vars:
 
   # For validation testing.  
   # This below vars configuration is needed for the `general_ledger_amounts_match` macro.
-  # using_bill: true
-  # using_credit_card_payment_txn: true
+  using_bill: true
+  using_credit_card_payment_txn: true
 
 
 seeds:

integration_tests/seeds/account_data.csv

@@ -1,12 +1,12 @@
-id,_fivetran_synced,account_number,account_sub_type,account_type,active,balance,balance_with_sub_accounts,classification,created_at,currency_id,description,fully_qualified_name,name,parent_account_id,sub_account,sync_token,tax_code_id,updated_at
-52,2020-06-23 3:58:54,,d225490f65343b9016d6db95c0b146d8,Expense,TRUE,41493,41493,,2020-06-18 14:09:07,USD,,a8dd055b9b91809ca74ddb5a5f357173,9b3a3362b3b1b8055be9d27401cd1be9,51,TRUE,0,,2020-06-18 14:09:07
-51,2020-06-23 3:58:54,,d225490f65343b9016d6db95c0b146d8,Expense,TRUE,45528,45528,,2020-06-18 14:08:47,USD,,82be779191266b6ef0597be0c7f60d6d,82be779191266b6ef0597be0c7f60d6d,,FALSE,0,,2020-06-18 14:09:07
-40,2020-12-09 19:02:02,,195917574edc9b6bbeb5be9785b6a479,Bank,TRUE,93234,93234,Asset,2020-06-05 16:30:39,USD,,dd6e03342a9e50cf278448f99e66d998,dd6e03342a9e50cf278448f99e66d998,,FALSE,0,,2020-12-09 19:01:03
-39,2020-12-09 19:16:58,,195917574edc9b6bbeb5be9785b6a479,Bank,TRUE,17735,17735,Asset,2020-05-28 13:09:16,USD,,6d69cd92d636a07b52a62b37e1f5201a,6d69cd92d636a07b52a62b37e1f5201a,,FALSE,1,,2020-12-09 19:02:26
-28,2020-12-17 20:32:00,,195917574edc9b6bbeb5be9785b6a479,Bank,TRUE,77166,77166,Asset,2020-04-30 17:27:58,USD,,ee791126ad0eff462e26b36f4a97fce0,ee791126ad0eff462e26b36f4a97fce0,,FALSE,2,,2020-12-17 20:17:44
-54,2020-12-08 21:01:59,,195917574edc9b6bbeb5be9785b6a479,Bank,FALSE,23231,23231,Asset,2020-07-08 18:08:40,USD,,245178490d20e65c6626c6af75f62215,245178490d20e65c6626c6af75f62215,,FALSE,1,,2020-12-08 20:58:43
-33,2020-06-23 3:58:54,,4d604dd8f008145471dc845683399189,Other Current Asset,TRUE,80922,80922,Asset,2020-04-30 17:29:35,USD,,07bd344f6acd4e8ed41ae67f8d8b4aee,07bd344f6acd4e8ed41ae67f8d8b4aee,,FALSE,0,,2020-04-30 17:29:35
-34,2020-12-17 19:31:57,6674,263b6c4c47a488aab804c8b8c12b4f76,Bank,TRUE,83287,83287,Asset,2020-04-30 17:36:45,USD,,939ee17c7427610c55083b8623cde590,939ee17c7427610c55083b8623cde590,,FALSE,0,,2020-12-17 19:21:21
-57,2020-12-02 15:31:58,,100fb8196b919896e083af2939bbffb2,Other Current Asset,TRUE,84550,84550,Asset,2020-07-13 19:15:58,USD,,5270c9c9f9382af307c28a838e89c57d,5270c9c9f9382af307c28a838e89c57d,,FALSE,0,,2020-12-02 15:19:27
-60,2020-11-09 22:47:02,,9337c3fde59c076b0d3ba902fa18e4e8,Other Current Asset,TRUE,65627,65627,Asset,2020-11-09 22:33:35,USD,,4d41392cafa229d9dffd274c9bf6d278,4d41392cafa229d9dffd274c9bf6d278,,FALSE,0,,2020-11-09 22:33:35
-75,2020-11-09 22:47:02,,DiscountsRefundsGiven,Discount,TRUE,6,6,,2020-11-09 22:33:35,USD,,4d41392cafa229d9dffd274c9bf6d278,4d41392cafa229d9dffd274c9bf6d278,,FALSE,0,,2020-11-09 22:33:35
+id,_fivetran_synced,account_number,account_sub_type,account_type,active,balance,balance_with_sub_accounts,classification,created_at,currency_id,description,fully_qualified_name,name,parent_account_id,sub_account,sync_token,tax_code_id,updated_at,_fivetran_deleted
+52,2020-06-23 3:58:54,,d225490f65343b9016d6db95c0b146d8,Expense,TRUE,41493,41493,,2020-06-18 14:09:07,USD,,a8dd055b9b91809ca74ddb5a5f357173,9b3a3362b3b1b8055be9d27401cd1be9,51,TRUE,0,,2020-06-18 14:09:07,FALSE
+51,2020-06-23 3:58:54,,d225490f65343b9016d6db95c0b146d8,Expense,TRUE,45528,45528,,2020-06-18 14:08:47,USD,,82be779191266b6ef0597be0c7f60d6d,82be779191266b6ef0597be0c7f60d6d,,FALSE,0,,2020-06-18 14:09:07,FALSE
+40,2020-12-09 19:02:02,,195917574edc9b6bbeb5be9785b6a479,Bank,TRUE,93234,93234,Asset,2020-06-05 16:30:39,USD,,dd6e03342a9e50cf278448f99e66d998,dd6e03342a9e50cf278448f99e66d998,,FALSE,0,,2020-12-09 19:01:03,FALSE
+39,2020-12-09 19:16:58,,195917574edc9b6bbeb5be9785b6a479,Bank,TRUE,17735,17735,Asset,2020-05-28 13:09:16,USD,,6d69cd92d636a07b52a62b37e1f5201a,6d69cd92d636a07b52a62b37e1f5201a,,FALSE,1,,2020-12-09 19:02:26,FALSE
+28,2020-12-17 20:32:00,,195917574edc9b6bbeb5be9785b6a479,Bank,TRUE,77166,77166,Asset,2020-04-30 17:27:58,USD,,ee791126ad0eff462e26b36f4a97fce0,ee791126ad0eff462e26b36f4a97fce0,,FALSE,2,,2020-12-17 20:17:44,FALSE
+54,2020-12-08 21:01:59,,195917574edc9b6bbeb5be9785b6a479,Bank,FALSE,23231,23231,Asset,2020-07-08 18:08:40,USD,,245178490d20e65c6626c6af75f62215,245178490d20e65c6626c6af75f62215,,FALSE,1,,2020-12-08 20:58:43,FALSE
+33,2020-06-23 3:58:54,,4d604dd8f008145471dc845683399189,Other Current Asset,TRUE,80922,80922,Asset,2020-04-30 17:29:35,USD,,07bd344f6acd4e8ed41ae67f8d8b4aee,07bd344f6acd4e8ed41ae67f8d8b4aee,,FALSE,0,,2020-04-30 17:29:35,FALSE
+34,2020-12-17 19:31:57,6674,263b6c4c47a488aab804c8b8c12b4f76,Bank,TRUE,83287,83287,Asset,2020-04-30 17:36:45,USD,,939ee17c7427610c55083b8623cde590,939ee17c7427610c55083b8623cde590,,FALSE,0,,2020-12-17 19:21:21,FALSE
+57,2020-12-02 15:31:58,,100fb8196b919896e083af2939bbffb2,Other Current Asset,TRUE,84550,84550,Asset,2020-07-13 19:15:58,USD,,5270c9c9f9382af307c28a838e89c57d,5270c9c9f9382af307c28a838e89c57d,,FALSE,0,,2020-12-02 15:19:27,FALSE
+60,2020-11-09 22:47:02,,9337c3fde59c076b0d3ba902fa18e4e8,Other Current Asset,TRUE,65627,65627,Asset,2020-11-09 22:33:35,USD,,4d41392cafa229d9dffd274c9bf6d278,4d41392cafa229d9dffd274c9bf6d278,,FALSE,0,,2020-11-09 22:33:35,TRUE
+75,2020-11-09 22:47:02,,DiscountsRefundsGiven,Discount,TRUE,6,6,,2020-11-09 22:33:35,USD,,4d41392cafa229d9dffd274c9bf6d278,4d41392cafa229d9dffd274c9bf6d278,,FALSE,0,,2020-11-09 22:33:35,FALSE

integration_tests/seeds/bundle_data.csv

@@ -1,11 +1,11 @@
-id,_fivetran_synced,active,created_at,description,fully_qualified_name,name,print_grouped_items,purchase_cost,sync_token,taxable,unit_price,updated_at
-19,2020-04-07 2:01:57,FALSE,2018-06-13 18:11:46,f69f149c065f9ba26db109327236d5cf,ded3af31c45f3f147bd29c5712d26867,ded3af31c45f3f147bd29c5712d26867,TRUE,0,1,FALSE,0,2019-01-15 19:16:42
-308,2020-10-06 21:15:35,FALSE,2020-10-05 18:12:56,,323c5927c3d020f010a685027d9953e6,323c5927c3d020f010a685027d9953e6,FALSE,0,2,FALSE,0,2020-10-06 20:51:43
-149,2020-04-07 2:01:57,FALSE,2019-09-24 21:53:03,,6a35b856f545b3f10fe168733755db80,6a35b856f545b3f10fe168733755db80,FALSE,0,7,FALSE,0,2019-09-30 3:12:01
-314,2020-10-06 19:16:08,TRUE,2020-10-05 18:21:41,,6d6314f5d29eb469be5607ab4dc43eb9,6d6314f5d29eb469be5607ab4dc43eb9,FALSE,0,1,FALSE,0,2020-10-06 18:46:23
-311,2020-10-05 20:16:13,TRUE,2020-10-05 18:18:28,,03bbca4dc28a8e558eb6115b79ad4b45,03bbca4dc28a8e558eb6115b79ad4b45,FALSE,0,1,FALSE,0,2020-10-05 19:22:50
-323,2020-10-05 20:16:13,TRUE,2020-10-05 18:34:02,,5884cbfcd93a0b9c97e91135c395e8fe,5884cbfcd93a0b9c97e91135c395e8fe,FALSE,0,1,FALSE,0,2020-10-05 19:18:55
-296,2020-10-05 20:16:13,TRUE,2020-10-05 17:51:13,,15500c470b4f3ed71ab9fbdde2115ad4,15500c470b4f3ed71ab9fbdde2115ad4,FALSE,0,1,FALSE,0,2020-10-05 19:30:13
-299,2020-10-05 20:16:13,TRUE,2020-10-05 17:59:13,,9fae5687972112018aaefea9c5334dc5,9fae5687972112018aaefea9c5334dc5,FALSE,0,1,FALSE,0,2020-10-05 19:29:03
-320,2020-10-05 19:15:37,TRUE,2020-10-05 18:30:16,,d1e02e5b96bd4d756b2974d1075ffd00,d1e02e5b96bd4d756b2974d1075ffd00,FALSE,0,1,FALSE,0,2020-10-05 19:14:19
-305,2020-10-05 20:16:13,TRUE,2020-10-05 18:10:06,,8154dbb4a94945236fed5a263f68ea8f,8154dbb4a94945236fed5a263f68ea8f,FALSE,0,1,FALSE,0,2020-10-05 19:26:36
+id,_fivetran_synced,active,created_at,description,fully_qualified_name,name,print_grouped_items,purchase_cost,sync_token,taxable,unit_price,updated_at,_fivetran_deleted
+19,2020-04-07 2:01:57,FALSE,2018-06-13 18:11:46,f69f149c065f9ba26db109327236d5cf,ded3af31c45f3f147bd29c5712d26867,ded3af31c45f3f147bd29c5712d26867,TRUE,0,1,FALSE,0,2019-01-15 19:16:42,FALSE
+308,2020-10-06 21:15:35,FALSE,2020-10-05 18:12:56,,323c5927c3d020f010a685027d9953e6,323c5927c3d020f010a685027d9953e6,FALSE,0,2,FALSE,0,2020-10-06 20:51:43,FALSE
+149,2020-04-07 2:01:57,FALSE,2019-09-24 21:53:03,,6a35b856f545b3f10fe168733755db80,6a35b856f545b3f10fe168733755db80,FALSE,0,7,FALSE,0,2019-09-30 3:12:01,FALSE
+314,2020-10-06 19:16:08,TRUE,2020-10-05 18:21:41,,6d6314f5d29eb469be5607ab4dc43eb9,6d6314f5d29eb469be5607ab4dc43eb9,FALSE,0,1,FALSE,0,2020-10-06 18:46:23,FALSE
+311,2020-10-05 20:16:13,TRUE,2020-10-05 18:18:28,,03bbca4dc28a8e558eb6115b79ad4b45,03bbca4dc28a8e558eb6115b79ad4b45,FALSE,0,1,FALSE,0,2020-10-05 19:22:50,FALSE
+323,2020-10-05 20:16:13,TRUE,2020-10-05 18:34:02,,5884cbfcd93a0b9c97e91135c395e8fe,5884cbfcd93a0b9c97e91135c395e8fe,FALSE,0,1,FALSE,0,2020-10-05 19:18:55,FALSE
+296,2020-10-05 20:16:13,TRUE,2020-10-05 17:51:13,,15500c470b4f3ed71ab9fbdde2115ad4,15500c470b4f3ed71ab9fbdde2115ad4,FALSE,0,1,FALSE,0,2020-10-05 19:30:13,FALSE
+299,2020-10-05 20:16:13,TRUE,2020-10-05 17:59:13,,9fae5687972112018aaefea9c5334dc5,9fae5687972112018aaefea9c5334dc5,FALSE,0,1,FALSE,0,2020-10-05 19:29:03,FALSE
+320,2020-10-05 19:15:37,TRUE,2020-10-05 18:30:16,,d1e02e5b96bd4d756b2974d1075ffd00,d1e02e5b96bd4d756b2974d1075ffd00,FALSE,0,1,FALSE,0,2020-10-05 19:14:19,FALSE
+305,2020-10-05 20:16:13,TRUE,2020-10-05 18:10:06,,8154dbb4a94945236fed5a263f68ea8f,8154dbb4a94945236fed5a263f68ea8f,FALSE,0,1,FALSE,0,2020-10-05 19:26:36,TRUE