-
Notifications
You must be signed in to change notification settings - Fork 72
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added masking_strategy_override at field level #5446
Merged
Merged
Changes from all commits
Commits
Show all changes
42 commits
Select commit
Hold shift + click to select a range
892f807
added masking_strategy_override at field level
Linker44 cf068a3
added testing
Linker44 605138f
run static checks
Linker44 a64592e
add requirements.txt
Linker44 a704b40
fix broken tests
Linker44 f9e6544
fix more broken tests
Linker44 41e76a0
implicit empty configuration on null_rewrite
Linker44 17732e5
fixed typing errors
Linker44 148e80a
added testing for db integrations
Linker44 90735bb
added validation for dataset creation and update
Linker44 8d40bc4
run static_checks
Linker44 1fcaa08
improve testing and move validations
Linker44 488718b
improve router_factory redability
Linker44 957b485
Merge branch 'main' into field_level_masking
Linker44 dcfc835
fix: edit fideslang requirement
Linker44 51fe052
Merge branch 'main' into field_level_masking
Linker44 7d24f82
change commit in requirements.txt
Linker44 648cd18
fix using wrong commit hash in requirements.txt
Linker44 aecd1c2
fixed comments
Linker44 f509649
Merge branch 'main' into field_level_masking
Linker44 0a98510
update requirements.txt to new fideslang release
Linker44 63ea347
corrected variable names, deleted repeated filter_data_category function
Linker44 f48b1cc
improved readability and added docstring to validate_masking_strategies
Linker44 57654ad
added testing for masking override put and happy path for update
Linker44 47711ea
added changelog
Linker44 294d964
Merge branch 'main' into field_level_masking
Linker44 dbf10d9
Update tests/ops/service/connectors/test_queryconfig.py
galvana 8e14528
remove failing log
Linker44 430a494
fixed bigquery test
Linker44 de55a34
added log warning for when applying field-level override
Linker44 b88d34f
Merge branch 'main' into field_level_masking
Linker44 a4d79de
Merge branch 'main' into field_level_masking
galvana 6885d77
Adding verbose logging and enabling traceback
galvana f61eabe
Adding more logging
galvana d8c454c
Adding timeout to external tests
galvana 03525e0
Adding timeout to admin DB actions
galvana 5c16eaa
Removing yield from setup_ctl_db fixture
galvana f91ae25
Merge branch 'main' into field_level_masking
galvana 1d0382c
Updating fixture
galvana 91018d3
Added changelog change
Linker44 893c4cd
Merge branch 'main' into field_level_masking
Linker44 b315095
revert nox pytest ctl command change
Linker44 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
252 changes: 252 additions & 0 deletions
252
data/dataset/example_field_masking_override_test_dataset.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,252 @@ | ||
| dataset: | ||
| - fides_key: field_masking_override_test_dataset | ||
| name: Field Masking Override Test Dataset | ||
| description: Example of a dataset containing masking strategy override at the field-level. | ||
| collections: | ||
| - name: address | ||
| fields: | ||
| - name: city | ||
| data_categories: [user.contact.address.city] | ||
| - name: house | ||
| data_categories: [user.contact.address.street] | ||
| - name: id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| primary_key: True | ||
| - name: state | ||
| data_categories: [user.contact.address.state] | ||
| - name: street | ||
| data_categories: [user.contact.address.street] | ||
| - name: zip | ||
| data_categories: [user.contact.address.postal_code] | ||
|
|
||
| - name: customer | ||
| fields: | ||
| - name: address_id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| references: | ||
| - dataset: field_masking_override_test_dataset | ||
| field: address.id | ||
| direction: to | ||
| - name: created | ||
| data_categories: [system.operations] | ||
| - name: email | ||
| data_categories: [user.contact.email] | ||
| fides_meta: | ||
| identity: email | ||
| data_type: string | ||
| - name: id | ||
| data_categories: [user.unique_id] | ||
| fides_meta: | ||
| primary_key: True | ||
| - name: name | ||
| data_categories: [user.name] | ||
| fides_meta: | ||
| data_type: string | ||
| length: 40 | ||
| masking_strategy_override: | ||
| strategy: random_string_rewrite | ||
| configuration: | ||
| length: 5 | ||
| format_preservation: | ||
| suffix: "@example.com" | ||
| - name: address | ||
| fields: | ||
| - name: city | ||
| data_categories: [user.contact.address.city] | ||
| - name: house | ||
| data_categories: [user.contact.address.street] | ||
| fides_meta: | ||
| data_type: string | ||
| masking_strategy_override: | ||
| strategy: string_rewrite | ||
| configuration: | ||
| rewrite_value: "1234" | ||
| format_preservation: | ||
| suffix: "-test" | ||
| - name: state | ||
| data_categories: [user.contact.address.state] | ||
| masking_strategy_override: | ||
| strategy: null_rewrite | ||
| - name: street | ||
| data_categories: [user.contact.address.street] | ||
| - name: zip | ||
| data_categories: [user.contact.address.postal_code] | ||
|
|
||
| - name: employee | ||
| fields: | ||
| - name: address_id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| references: | ||
| - dataset: field_masking_override_test_dataset | ||
| field: address.id | ||
| direction: to | ||
| - name: email | ||
| data_categories: [user.contact.email] | ||
| fides_meta: | ||
| identity: email | ||
| data_type: string | ||
| - name: id | ||
| data_categories: [user.unique_id] | ||
| fides_meta: | ||
| primary_key: True | ||
| - name: name | ||
| data_categories: [user.name] | ||
| fides_meta: | ||
| data_type: string | ||
|
|
||
| - name: login | ||
| fields: | ||
| - name: customer_id | ||
| data_categories: [user.unique_id] | ||
| fides_meta: | ||
| references: | ||
| - dataset: field_masking_override_test_dataset | ||
| field: customer.id | ||
| direction: from | ||
| - name: id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| primary_key: True | ||
| - name: time | ||
| data_categories: [user.sensor] | ||
|
|
||
| - name: orders | ||
| fields: | ||
| - name: customer_id | ||
| data_categories: [user.unique_id] | ||
| fides_meta: | ||
| references: | ||
| - dataset: field_masking_override_test_dataset | ||
| field: customer.id | ||
| direction: from | ||
| - name: id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| primary_key: True | ||
| - name: shipping_address_id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| references: | ||
| - dataset: field_masking_override_test_dataset | ||
| field: address.id | ||
| direction: to | ||
|
|
||
| # order_item | ||
| - name: order_item | ||
| fields: | ||
| - name: order_id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| references: | ||
| - dataset: field_masking_override_test_dataset | ||
| field: orders.id | ||
| direction: from | ||
| - name: product_id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| references: | ||
| - dataset: field_masking_override_test_dataset | ||
| field: product.id | ||
| direction: to | ||
| - name: quantity | ||
| data_categories: [system.operations] | ||
|
|
||
| - name: payment_card | ||
| fields: | ||
| - name: billing_address_id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| references: | ||
| - dataset: field_masking_override_test_dataset | ||
| field: address.id | ||
| direction: to | ||
| - name: ccn | ||
| data_categories: [user.financial.bank_account] | ||
| - name: code | ||
| data_categories: [user.financial] | ||
| - name: customer_id | ||
| data_categories: [user.unique_id] | ||
| fides_meta: | ||
| references: | ||
| - dataset: field_masking_override_test_dataset | ||
| field: customer.id | ||
| direction: from | ||
| - name: id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| primary_key: True | ||
| - name: name | ||
| data_categories: [user.financial] | ||
| - name: preferred | ||
| data_categories: [user] | ||
|
|
||
| - name: product | ||
| fields: | ||
| - name: id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| primary_key: True | ||
| - name: name | ||
| data_categories: [system.operations] | ||
| - name: price | ||
| data_categories: [system.operations] | ||
|
|
||
| - name: report | ||
| fields: | ||
| - name: email | ||
| data_categories: [user.contact.email] | ||
| fides_meta: | ||
| identity: email | ||
| data_type: string | ||
| - name: id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| primary_key: True | ||
| - name: month | ||
| data_categories: [system.operations] | ||
| - name: name | ||
| data_categories: [system.operations] | ||
| - name: total_visits | ||
| data_categories: [system.operations] | ||
| - name: year | ||
| data_categories: [system.operations] | ||
|
|
||
| - name: service_request | ||
| fields: | ||
| - name: alt_email | ||
| data_categories: [user.contact.email] | ||
| fides_meta: | ||
| identity: email | ||
| data_type: string | ||
| - name: closed | ||
| data_categories: [system.operations] | ||
| - name: email | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| identity: email | ||
| data_type: string | ||
| - name: employee_id | ||
| data_categories: [user.unique_id] | ||
| fides_meta: | ||
| references: | ||
| - dataset: field_masking_override_test_dataset | ||
| field: employee.id | ||
| direction: from | ||
| - name: id | ||
| data_categories: [system.operations] | ||
| fides_meta: | ||
| primary_key: True | ||
| - name: opened | ||
| data_categories: [system.operations] | ||
| - name: visit | ||
| fields: | ||
| - name: email | ||
| data_categories: [user.contact.email] | ||
| fides_meta: | ||
| identity: email | ||
| data_type: string | ||
| - name: last_visit | ||
| data_categories: [system.operations] |
43 changes: 43 additions & 0 deletions
43
data/dataset/postgres_example_invalid_masking_strategy_override_dataset.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| dataset: | ||
| - fides_key: postgres_example_invalid_masking_strategy_override | ||
| name: Postgres Example Invalid Masking Strategy Override Test Dataset | ||
| description: Example of a Postgres dataset containing an invalid masking startegy override | ||
| collections: | ||
| - name: customer | ||
| fields: | ||
| - name: created | ||
| data_categories: [system.operations] | ||
| - name: email | ||
| data_categories: [user.contact.email] | ||
| fides_meta: | ||
| identity: email | ||
| data_type: string | ||
| - name: id | ||
| data_categories: [user.unique_id] | ||
| fides_meta: | ||
| primary_key: True | ||
| - name: name | ||
| data_categories: [user.name] | ||
| fides_meta: | ||
| data_type: string | ||
| length: 40 | ||
|
|
||
| - name: employee | ||
| fields: | ||
| - name: email | ||
| data_categories: [user.contact.email] | ||
| fides_meta: | ||
| identity: email | ||
| data_type: string | ||
| - name: id | ||
| data_categories: [user.unique_id] | ||
| fides_meta: | ||
| primary_key: True | ||
| - name: name | ||
| data_categories: [user.name] | ||
| fides_meta: | ||
| data_type: string | ||
| masking_strategy_override: | ||
| strategy: hash | ||
| configuration: | ||
| algorithm: SHA-256 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -71,4 +71,4 @@ twilio==7.15.0 | |
| typing-extensions==4.12.2 | ||
| validators==0.20.0 | ||
| versioneer==0.19 | ||
| fideslang==3.0.8 | ||
| fideslang==3.0.9 | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
dataset creation and update happens in many places so this validation gets repeated all over. this is one of those cases where a service would come in handy.