Pause quality monitors when "mode: development" is used #1481
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1481 +/- ##
==========================================
+ Coverage 52.25% 53.58% +1.33%
==========================================
Files 317 351 +34
Lines 18004 20281 +2277
==========================================
+ Hits 9408 10868 +1460
- Misses 7903 8614 +711
- Partials 693 799 +106 ☔ View full report in Codecov by Sentry. |
|
@vperiyasamy to confirm behavior from Monitoring side |
|
Related PR, generalizing the "mode: development" behavior: #1490 |
vperiyasamy
suggested changes
Jun 12, 2024
There was a problem hiding this comment.
We actually don't support pausing the monitor schedules directly, the schedule status is a read-only field that we exposed because we were considering building an "auto pausing" feature which would detect successive failures. Is it required for us to have this behavior? If so, we can delete the schedule to "pause" it.
|
@aravind-segu @arpitjasa-db @vperiyasamy I updated the PR to remove the schedule instead for now, please take another look. I also updated the config test so it actually works out of the box on staging in order to verify this change. |
vperiyasamy
approved these changes
Jun 17, 2024
| @@ -24,14 +31,14 @@ targets: | |||
| quality_monitors: | |||
| my_monitor: | |||
| table_name: "main.test.staging" | |||
| output_schema_name: "staging" | |||
| output_schema_name: "main.test" | |||
|
|
|||
| production: | |||
| resources: | |||
| quality_monitors: | |||
| my_monitor: | |||
There was a problem hiding this comment.
The original name wasn't valid according the the API. It requires something of the format catalog.schema. Actually, that seems like something to reflect on https://docs.databricks.com/api/workspace/qualitymonitors/create?
pietern
reviewed
Jun 18, 2024
pietern
approved these changes
Jun 18, 2024
arpitjasa-db
approved these changes
Jun 18, 2024
pietern
added a commit
that referenced
this pull request
Jun 26, 2024
CLI: * Add link to documentation for Homebrew installation to README ([#1505](#1505)). * Fix `databricks configure` to use `DATABRICKS_CONFIG_FILE` environment variable if exists as config file ([#1325](#1325)). Bundles: The Terraform upgrade to v1.48.0 includes a fix for library order not being respected. * Fix conditional in query in `default-sql` template ([#1479](#1479)). * Remove user credentials specified in the Git origin URL ([#1494](#1494)). * Serialize dynamic value for `bundle validate` output ([#1499](#1499)). * Override variables with lookup value even if values has default value set ([#1504](#1504)). * Pause quality monitors when "mode: development" is used ([#1481](#1481)). * Return `fs.ModeDir` for Git folders in the workspace ([#1521](#1521)). * Upgrade TF provider to 1.48.0 ([#1527](#1527)). * Added support for complex variables ([#1467](#1467)). Internal: * Add randIntn function ([#1475](#1475)). * Avoid multiple file tree traversals on bundle deploy ([#1493](#1493)). * Clean up unused code ([#1502](#1502)). * Use `dyn.InvalidValue` to indicate absence ([#1507](#1507)). * Add ApplyPythonMutator ([#1430](#1430)). * Set bool pointer to disable lock ([#1516](#1516)). * Allow the any type to be set to nil in `convert.FromTyped` ([#1518](#1518)). * Properly deal with nil values in `convert.FromTyped` ([#1511](#1511)). * Return `dyn.InvalidValue` instead of `dyn.NilValue` when errors happen ([#1514](#1514)). * PythonMutator: replace stdin/stdout with files ([#1512](#1512)). * Add context type and value to path rewriting ([#1525](#1525)). API Changes: * Added schedule CRUD commands to `databricks lakeview`. * Added subscription CRUD commands to `databricks lakeview`. * Added `databricks apps start` command. OpenAPI commit 7437dabb9dadee402c1fc060df4c1ce8cc5369f0 (2024-06-24) Dependency updates: * Bump golang.org/x/text from 0.15.0 to 0.16.0 ([#1482](#1482)). * Bump golang.org/x/term from 0.20.0 to 0.21.0 ([#1483](#1483)). * Bump golang.org/x/mod from 0.17.0 to 0.18.0 ([#1484](#1484)). * Bump golang.org/x/oauth2 from 0.20.0 to 0.21.0 ([#1485](#1485)). * Bump github.com/briandowns/spinner from 1.23.0 to 1.23.1 ([#1495](#1495)). * Bump github.com/spf13/cobra from 1.8.0 to 1.8.1 ([#1496](#1496)). * Bump github.com/databricks/databricks-sdk-go from 0.42.0 to 0.43.0 ([#1522](#1522)).
Merged
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jun 26, 2024
CLI: * Add link to documentation for Homebrew installation to README ([#1505](#1505)). * Fix `databricks configure` to use `DATABRICKS_CONFIG_FILE` environment variable if exists as config file ([#1325](#1325)). Bundles: The Terraform upgrade to v1.48.0 includes a fix for library order not being respected. * Fix conditional in query in `default-sql` template ([#1479](#1479)). * Remove user credentials specified in the Git origin URL ([#1494](#1494)). * Serialize dynamic value for `bundle validate` output ([#1499](#1499)). * Override variables with lookup value even if values has default value set ([#1504](#1504)). * Pause quality monitors when "mode: development" is used ([#1481](#1481)). * Return `fs.ModeDir` for Git folders in the workspace ([#1521](#1521)). * Upgrade TF provider to 1.48.0 ([#1527](#1527)). * Added support for complex variables ([#1467](#1467)). Internal: * Add randIntn function ([#1475](#1475)). * Avoid multiple file tree traversals on bundle deploy ([#1493](#1493)). * Clean up unused code ([#1502](#1502)). * Use `dyn.InvalidValue` to indicate absence ([#1507](#1507)). * Add ApplyPythonMutator ([#1430](#1430)). * Set bool pointer to disable lock ([#1516](#1516)). * Allow the any type to be set to nil in `convert.FromTyped` ([#1518](#1518)). * Properly deal with nil values in `convert.FromTyped` ([#1511](#1511)). * Return `dyn.InvalidValue` instead of `dyn.NilValue` when errors happen ([#1514](#1514)). * PythonMutator: replace stdin/stdout with files ([#1512](#1512)). * Add context type and value to path rewriting ([#1525](#1525)). API Changes: * Added schedule CRUD commands to `databricks lakeview`. * Added subscription CRUD commands to `databricks lakeview`. * Added `databricks apps start` command. OpenAPI commit 7437dabb9dadee402c1fc060df4c1ce8cc5369f0 (2024-06-24) Dependency updates: * Bump golang.org/x/text from 0.15.0 to 0.16.0 ([#1482](#1482)). * Bump golang.org/x/term from 0.20.0 to 0.21.0 ([#1483](#1483)). * Bump golang.org/x/mod from 0.17.0 to 0.18.0 ([#1484](#1484)). * Bump golang.org/x/oauth2 from 0.20.0 to 0.21.0 ([#1485](#1485)). * Bump github.com/briandowns/spinner from 1.23.0 to 1.23.1 ([#1495](#1495)). * Bump github.com/spf13/cobra from 1.8.0 to 1.8.1 ([#1496](#1496)). * Bump github.com/databricks/databricks-sdk-go from 0.42.0 to 0.43.0 ([#1522](#1522)).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Similar to scheduled jobs, quality monitors should be paused when in development mode (in line with the behavior for scheduled jobs). @aravind-segu @arpitjasa-db please take a look and verify this behavior.
Tests
Unit tests