Merge pull request #1 from danflippo/source_constraints

Merging branch that adds constraints on sources

README.md

@@ -18,11 +18,21 @@ When you add this package, dbt will automatically begin to create unique keys fo
 
 ### Disabling automatic constraint generation
 
-The `dbt_constraints_enabled` variable can be set to `false` in your project to disable automatic constraint generation.
+The `dbt_constraints_enabled` variable can be set to `false` in your project to disable automatic constraint generation. By default dbt Constraints only creates constraints on models. To allow constraints on sources, you can set `dbt_constraints_sources_enabled` to `true`. The package will verify that you have sufficient database privileges to create constraints on sources.
 
 ```yml
 vars:
-  dbt_constraints_enabled: false
+  # The package can be temporarily disabled using this variable
+  dbt_constraints_enabled: true
+
+  # The package can also add constraints on sources if you have sufficient privileges
+  dbt_constraints_sources_enabled: false
+
+  # You can also be specific on which constraints are enabled for sources
+  # You must also enable dbt_constraints_sources_enabled above
+  dbt_constraints_sources_pk_enabled: true
+  dbt_constraints_sources_uk_enabled: true
+  dbt_constraints_sources_fk_enabled: true
 ```
 
 ## Installation
@@ -85,23 +95,25 @@ packages:
 
 * The package's macros depend on the results and graph object schemas of dbt >=1.0.0
 
-* The package currently only includes macros for creating constraints in Snowflake and PostgreSQL. To add support for other databases, it is necessary to implement the following five macros with the appropriate DDL & SQL for your database. Pull requests to contribute support for other databases are welcome. See the snowflake__create_constraints.sql and postgres__create_constraints.sql files as examples.
+* The package currently only includes macros for creating constraints in Snowflake and PostgreSQL. To add support for other databases, it is necessary to implement the following seven macros with the appropriate DDL & SQL for your database. Pull requests to contribute support for other databases are welcome. See the snowflake__create_constraints.sql and postgres__create_constraints.sql files as examples.
 
 ```
-<ADAPTER_NAME>__create_primary_key(table_model, column_names, quote_columns=false)
-<ADAPTER_NAME>__create_unique_key(table_model, column_names, quote_columns=false)
-<ADAPTER_NAME>__create_foreign_key(test_model, pk_model, pk_column_names, fk_model, fk_column_names, quote_columns=false)
+<ADAPTER_NAME>__create_primary_key(table_model, column_names, verify_permissions, quote_columns=false)
+<ADAPTER_NAME>__create_unique_key(table_model, column_names, verify_permissions, quote_columns=false)
+<ADAPTER_NAME>__create_foreign_key(pk_model, pk_column_names, fk_model, fk_column_names, verify_permissions, quote_columns=false)
 <ADAPTER_NAME>__unique_constraint_exists(table_relation, column_names)
 <ADAPTER_NAME>__foreign_key_exists(table_relation, column_names)
+<ADAPTER_NAME>__have_references_priv(table_relation, verify_permissions)
+<ADAPTER_NAME>__have_ownership_priv(table_relation, verify_permissions)
 ```
 
 ## dbt_constraints Limitations
 
 Generally, if you don't meet a requirement, tests are still executed but the constraint is skipped rather than producing an error.
 
-- All models involved in a constraint must be materialized as table, incremental, or snapshot
+- All models involved in a constraint must be materialized as table, incremental, or snapshot.
 
-- Constraints will not be created on sources, only models. You can use the PK/UK/FK tests with sources but constraints won't be generated.
+-  If source constraints are enabled, the source must be a table. You must also have the `OWNERSHIP` table privilege to add a constraint. For foreign keys you also need the `REFERENCES` privilege on the parent table with the primary or unique key.
 
 - All columns on constraints must be individual column names, not expressions. You can reference columns on a model that come from an expression.
 

TROUBLESHOOTING.md

@@ -0,0 +1,49 @@
+# Troubleshooting dbt Constraints
+
+As it executes, dbt Constraints will log a number of messages when it cannot create a constraint. 
+A list of the messages is provided below with additional details on how to address the message.
+
+## Messages that are displayed on the front-end of the command line interface and in the dbt log
+
+```
+Skipping primary/unique key because a physical column name was not found on the table: {TABLE NAME} {TABLE COLUMNS}
+Skipping foreign key because a physical column was not found on the pk table: {PK TABLE NAME} {PK TABLE COLUMNS}
+Skipping foreign key because a physical column was not found on the fk table: {FK TABLE NAME} {FK TABLE COLUMNS}
+```
+- These error messages typically occur when a column is misspelled or if the test uses an expression instead of a column name. 
+- One solution can be adding the expression as an additional column in your model so that you can reference it in your constraint.
+
+
+```
+"Skipping foreign key because a we couldn't find the child table: model={FK TABLE NAME} or source={PK TABLE NAME}
+```
+- This is only expected to occur when a foreign key constraint is made with a source and dbt Constraints can't parse the reference to a source table.
+- The package is looking for something that looks like: `source("source_name", "table_name")` or `source('source_name', 'table_name')`
+- You may need to replace any dynamic variables with strings for the source name or table name.
+
+
+```
+Skipping {CONSTRAINT NAME} because of insufficient privileges: {FK TABLE NAME} referencing {PK TABLE NAME}
+Skipping {CONSTRAINT NAME} because of insufficient privileges: {TABLE NAME}
+```
+- You must have OWNERSHIP on the child FK table and you must have REFERENCES on the parent PK table.
+- For primary keys and unique keys, you need ownership on the table.
+- These errors most frequently apply to sources.
+- This can also indicate that one of your tables is actually a view
+
+
+```
+Skipping {CONSTRAINT NAME} because a PK/UK was not found on the PK table: {PK TABLE NAME} {PK TABLE COLUMNS}
+```
+- You either need to manually create a primary key/unique key or you need to add a test to the parent table and allow the package to create the constraint.
+- The package creates constraints in the order of primary keys, unique keys, foreign keys to allow parent constraints to be referenced by foreign keys.
+
+
+## Messages that are only displayed in the dbt log:
+
+```
+Skipping {CONSTRAINT NAME} because PK/UK already exists: {TABLE NAME} {TABLE COLUMNS}
+Skipping {CONSTRAINT NAME} because FK already exists: {FK TABLE NAME} {FK TABLE COLUMNS}
+```
+- Indicates duplicate constraints or that a constraint was already added to an incremental / snapshot table on a previous run
+- Typically not an issue

dbt_project.yml

@@ -24,3 +24,12 @@ on-run-end:
 vars:
   # The package can be temporarily disabled using this variable
   dbt_constraints_enabled: true
+
+  # The package can also add constraints on sources if you have sufficient privileges
+  dbt_constraints_sources_enabled: false
+
+  # You can also be specific on which constraints are enabled for sources
+  # You must also enable dbt_constraints_sources_enabled above
+  dbt_constraints_sources_pk_enabled: true
+  dbt_constraints_sources_uk_enabled: true
+  dbt_constraints_sources_fk_enabled: true

docs/catalog.json

@@ -1 +1 @@
-{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/catalog/v1.json", "dbt_version": "1.0.4", "generated_at": "2022-04-15T00:12:22.303029Z", "invocation_id": "4e679f95-191b-4f16-a8c0-c24292e8cdb0", "env": {}}, "nodes": {}, "sources": {}, "errors": null}
+{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/catalog/v1.json", "dbt_version": "1.0.4", "generated_at": "2022-04-22T21:39:49.991421Z", "invocation_id": "ce25464d-8bb3-48c4-aecd-e2013012be59", "env": {}}, "nodes": {}, "sources": {}, "errors": null}

docs/run_results.json

@@ -1 +1 @@
-{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/run-results/v4.json", "dbt_version": "1.0.4", "generated_at": "2022-04-15T00:12:22.294829Z", "invocation_id": "4e679f95-191b-4f16-a8c0-c24292e8cdb0", "env": {}}, "results": [{"status": "success", "timing": [{"name": "compile", "started_at": "2022-04-15T00:12:22.259463Z", "completed_at": "2022-04-15T00:12:22.292643Z"}, {"name": "execute", "started_at": "2022-04-15T00:12:22.293042Z", "completed_at": "2022-04-15T00:12:22.293069Z"}], "thread_id": "Thread-1", "execution_time": 0.03456306457519531, "adapter_response": {}, "message": null, "failures": null, "unique_id": "operation.dbt_constraints.dbt_constraints-on-run-end-0"}], "elapsed_time": 0.041356801986694336, "args": {"write_json": true, "use_colors": true, "printer_width": 80, "version_check": true, "partial_parse": true, "static_parser": true, "profiles_dir": "/Users/dflippo/.dbt", "send_anonymous_usage_stats": true, "event_buffer_size": 100000, "compile": true, "which": "generate", "rpc_method": "docs.generate", "indirect_selection": "eager"}}
+{"metadata": {"dbt_schema_version": "https://schemas.getdbt.com/dbt/run-results/v4.json", "dbt_version": "1.0.4", "generated_at": "2022-04-22T21:39:49.983008Z", "invocation_id": "ce25464d-8bb3-48c4-aecd-e2013012be59", "env": {}}, "results": [{"status": "success", "timing": [{"name": "compile", "started_at": "2022-04-22T21:39:49.936114Z", "completed_at": "2022-04-22T21:39:49.980945Z"}, {"name": "execute", "started_at": "2022-04-22T21:39:49.981331Z", "completed_at": "2022-04-22T21:39:49.981355Z"}], "thread_id": "Thread-1", "execution_time": 0.04632973670959473, "adapter_response": {}, "message": null, "failures": null, "unique_id": "operation.dbt_constraints.dbt_constraints-on-run-end-0"}], "elapsed_time": 0.1201019287109375, "args": {"write_json": true, "use_colors": true, "printer_width": 80, "version_check": true, "partial_parse": true, "static_parser": true, "profiles_dir": "/Users/dflippo/.dbt", "send_anonymous_usage_stats": true, "event_buffer_size": 100000, "compile": true, "which": "generate", "rpc_method": "docs.generate", "indirect_selection": "eager"}}

integration_tests/dbt_project.yml

@@ -30,7 +30,18 @@ clean-targets:         # directories to be removed by `dbt clean`
 
 # Global variables
 vars:
+  # The package can be temporarily disabled using this variable
   dbt_constraints_enabled: true
 
+  # The package can also add constraints on sources if you have sufficient privileges
+  dbt_constraints_sources_enabled: true
+
+  # You can also be specific on which constraints are enabled for sources
+  # You must also enable dbt_constraints_sources_enabled above
+  dbt_constraints_sources_pk_enabled: true
+  dbt_constraints_sources_uk_enabled: true
+  dbt_constraints_sources_fk_enabled: true
+
+
 models:
   +materialized: table

integration_tests/models/schema.yml

@@ -195,6 +195,15 @@ models:
 
   - name: dim_part_supplier
     description: "Multi column UK"
+    columns:
+      - name: ps_suppkey
+        description: "Part of compound primary key for this table"
+        tests:
+          - not_null
+          # Testing FK to a source
+          - relationships:
+              to: source('tpc_h', 'supplier')
+              field: s_suppkey
     tests:
       - dbt_constraints.unique_key:
           column_names: