Add configurable presets for name prefixes, tags, etc. #1490

lennartkats-db · 2024-06-12T15:31:04Z

Changes

This adds configurable transformations based on the transformations currently seen in mode: development.

Example databricks.yml showcasing how some transformations:

bundle:
  name: my_bundle

targets:
  dev:
    presets:
      prefix: "myprefix_"          # prefix all resource names with myprefix_
      pipelines_development: true  # set development to true by default for pipelines
      trigger_pause_status: PAUSED # set pause_status to PAUSED by default for all triggers and schedules
      jobs_max_concurrent_runs: 10 # set max_concurrent runs to 10 by default for all jobs
      tags:
        dev: true

Tests

Existing process_target_mode tests that were adapted to use this new code
Unit tests specific for the new mutator
Unit tests for config loading and merging
Manual e2e testing

codecov-commenter · 2024-06-14T12:37:19Z

Codecov Report

Attention: Patch coverage is 75.00000% with 48 lines in your changes missing coverage. Please review.

Project coverage is 53.69%. Comparing base (e22dd8a) to head (4dc5f41).
Report is 142 commits behind head on main.

Files	Patch %	Lines
bundle/config/mutator/process_target_mode.go	68.57%	16 Missing and 6 partials ⚠️
bundle/config/mutator/apply_transforms.go	84.82%	12 Missing and 5 partials ⚠️
bundle/config/root.go	16.66%	4 Missing and 1 partial ⚠️
bundle/config/transforms.go	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1490      +/-   ##
==========================================
+ Coverage   52.25%   53.69%   +1.43%     
==========================================
  Files         317      353      +36     
  Lines       18004    20406    +2402     
==========================================
+ Hits         9408    10956    +1548     
- Misses       7903     8642     +739     
- Partials      693      808     +115

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bundle/config/mutator/apply_transforms_test.go

bundle/config/mutator/apply_transforms.go

bundle/config/mutator/apply_transforms_test.go

bundle/config/mutator/process_target_mode.go

bundle/config/root.go

pietern · 2024-06-18T06:53:33Z

Thanks, will be great to have this.

bundle/config/mutator/apply_transforms.go

bundle/config/mutator/apply_transforms_test.go

bundle/config/mutator/process_target_mode.go

bundle/config/mutator/process_target_mode_test.go

bundle/config/root.go

bundle/config/transforms.go

This cherry-picks from #1490 to address an issue that came up in #1511. The function `dyn.SetByPath` requires intermediate values to be present. If they are not, it returns an error that it cannot index a map. This is not an issue on main where the intermediate maps are always created, even if they are not present in the dynamic configuration tree. As of #1511 we'll no longer populate empty maps for empty structs if they are not explictly set (i.e. a non-nil pointer). We can write a bool pointer to avoid this issue altogether.

## Changes This cherry-picks from #1490 to address an issue that came up in #1511. The function `dyn.SetByPath` requires intermediate values to be present. If they are not, it returns an error that it cannot index a map. This is not an issue on main, where the intermediate maps are always created, even if they are not present in the dynamic configuration tree. As of #1511, we'll no longer populate empty maps for empty structs if they are not explicitly set (i.e., a non-nil pointer). This change writes a bool pointer to avoid this issue altogether. ## Tests Unit tests pass.

lennartkats-db · 2024-07-19T19:55:57Z

@pietern PTAL, I took another pass here

.vscode/settings.json

bundle/config/mutator/apply_presets_test.go

bundle/config/mutator/process_target_mode.go

bundle/tests/presets_test.go

lennartkats-db · 2024-08-16T07:30:29Z

When ready, I'd like to first merge #1590. cc @pietern

## Changes In #1490 we regressed and started using the development mode prefix for UC schemas regardless of the mode of the bundle target. This PR fixes the regression and adds a regression test ## Tests Failing integration tests pass now.

CLI: * Added filtering flags for cluster list commands ([#1703](#1703)). Bundles: * Remove reference to "dbt" in the default-sql template ([#1696](#1696)). * Pause continuous pipelines when 'mode: development' is used ([#1590](#1590)). * Add configurable presets for name prefixes, tags, etc. ([#1490](#1490)). * Report all empty resources present in error diagnostic ([#1685](#1685)). * Improves detection of PyPI package names in environment dependencies ([#1699](#1699)). * [DAB] Add support for requirements libraries in Job Tasks ([#1543](#1543)). * Add paths field to bundle sync configuration ([#1694](#1694)). Internal: * Add `import` option for PyDABs ([#1693](#1693)). * Make fileset take optional list of paths to list ([#1684](#1684)). * Pass through paths argument to libs/sync ([#1689](#1689)). * Correctly mark package names with versions as remote libraries ([#1697](#1697)). * Share test initializer in common helper function ([#1695](#1695)). * Make `pydabs/venv_path` optional ([#1687](#1687)). * Use API mocks for duplicate path errors in workspace files extensions client ([#1690](#1690)). * Fix prefix preset used for UC schemas ([#1704](#1704)).

lennartkats-db added 10 commits June 8, 2024 18:40

Initial draft of customizable transformation

e5ac74d

Refactor to one function per transformer

6eaee84

WIP

601c32b

Remove enabled fields

b16c18c

Add config merging & test

901097a

e2e fixes

a815f30

Cleanup

13630dd

Use PAUSED/UNPAUSED instead of a boolean

e3b0435

Rename to transform for now

10a1ffc

Merge remote-tracking branch 'databricks/main' into cp-mutator-settings

405e202

lennartkats-db changed the title ~~[WIP] Add configurable transformations for name prefixes, tags, etc.~~ Add configurable transformations for name prefixes, tags, etc. Jun 12, 2024

lennartkats-db marked this pull request as ready for review June 12, 2024 16:16

lennartkats-db requested review from andrewnester, pietern and shreyas-goenka June 12, 2024 16:17

lennartkats-db mentioned this pull request Jun 12, 2024

Pause quality monitors when "mode: development" is used #1481

Merged

1 task

Cleanup

7323d02

Add stricter validations

4dc5f41

pietern reviewed Jun 18, 2024

View reviewed changes

lennartkats-db mentioned this pull request Jun 18, 2024

Customize the prefix assigned to resources deployed in development mode #1084

Closed

Cleanup

82e1d49

lennartkats-db force-pushed the cp-mutator-settings branch from 6fd240d to 82e1d49 Compare June 19, 2024 08:33

Simply mutations, no need for dyn here

6d75e84

pietern reviewed Jun 20, 2024

View reviewed changes

andrewnester mentioned this pull request Jun 20, 2024

[BUG] Bundle in dev mode uses service principal client ID in the Job Name #1509

Open

pietern mentioned this pull request Jun 21, 2024

Set bool pointer to disable lock #1516

Merged

lennartkats-db force-pushed the cp-mutator-settings branch 2 times, most recently from c001314 to 40f3bb4 Compare July 19, 2024 19:39

lennartkats-db added 3 commits July 19, 2024 21:40

Address reviewer comments, fix names

b1427b3

Merge remote-tracking branch 'databricks/main' into cp-mutator-settings

f2553ff

Fix regression in main

fb902c9

andrewnester mentioned this pull request Aug 1, 2024

Alter bundle asset development mode tags for jobs and pipelines #773

Open

pietern reviewed Aug 12, 2024

View reviewed changes

Merge remote-tracking branch 'databricks/main' into cp-mutator-settings

6159c3c

lennartkats-db force-pushed the cp-mutator-settings branch from 61704c8 to 6159c3c Compare August 14, 2024 14:53

Use bundle.Apply() for tests

b4564f2

lennartkats-db requested a review from pietern August 14, 2024 15:00

lennartkats-db added 2 commits August 16, 2024 09:27

Add assertion

70d8988

Merge remote-tracking branch 'databricks/main' into cp-mutator-settings

a073f84

lennartkats-db enabled auto-merge August 16, 2024 07:29

lennartkats-db disabled auto-merge August 16, 2024 07:30

pietern approved these changes Aug 19, 2024

View reviewed changes

andrewnester approved these changes Aug 19, 2024

View reviewed changes

Merge remote-tracking branch 'databricks/main' into cp-mutator-settings

b9e3278

lennartkats-db enabled auto-merge August 19, 2024 18:17

lennartkats-db added this pull request to the merge queue Aug 19, 2024

Merged via the queue into databricks:main with commit 78d0ac5 Aug 19, 2024
5 checks passed

lennartkats-db deleted the cp-mutator-settings branch August 19, 2024 18:26

shreyas-goenka mentioned this pull request Aug 21, 2024

Fix prefix preset used for UC schemas #1704

Merged

andrewnester mentioned this pull request Aug 21, 2024

[Release] Release v0.227.0 #1705

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add configurable presets for name prefixes, tags, etc. #1490

Add configurable presets for name prefixes, tags, etc. #1490

lennartkats-db commented Jun 12, 2024 •

edited

Loading

codecov-commenter commented Jun 14, 2024 •

edited

Loading

pietern commented Jun 18, 2024

lennartkats-db commented Jul 19, 2024

lennartkats-db commented Aug 16, 2024 •

edited

Loading

Add configurable presets for name prefixes, tags, etc. #1490

Add configurable presets for name prefixes, tags, etc. #1490

Conversation

lennartkats-db commented Jun 12, 2024 • edited Loading

Changes

Tests

codecov-commenter commented Jun 14, 2024 • edited Loading

Codecov Report

pietern commented Jun 18, 2024

lennartkats-db commented Jul 19, 2024

lennartkats-db commented Aug 16, 2024 • edited Loading

lennartkats-db commented Jun 12, 2024 •

edited

Loading

codecov-commenter commented Jun 14, 2024 •

edited

Loading

lennartkats-db commented Aug 16, 2024 •

edited

Loading