Avoid multiple file tree traversals on bundle deploy #1493
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
shreyas-goenka
requested review from
pietern and
andrewnester
and removed request for
pietern
lennartkats-db
approved these changes
Jun 14, 2024
pietern
added a commit
that referenced
this pull request
Jun 26, 2024
CLI: * Add link to documentation for Homebrew installation to README ([#1505](#1505)). * Fix `databricks configure` to use `DATABRICKS_CONFIG_FILE` environment variable if exists as config file ([#1325](#1325)). Bundles: The Terraform upgrade to v1.48.0 includes a fix for library order not being respected. * Fix conditional in query in `default-sql` template ([#1479](#1479)). * Remove user credentials specified in the Git origin URL ([#1494](#1494)). * Serialize dynamic value for `bundle validate` output ([#1499](#1499)). * Override variables with lookup value even if values has default value set ([#1504](#1504)). * Pause quality monitors when "mode: development" is used ([#1481](#1481)). * Return `fs.ModeDir` for Git folders in the workspace ([#1521](#1521)). * Upgrade TF provider to 1.48.0 ([#1527](#1527)). * Added support for complex variables ([#1467](#1467)). Internal: * Add randIntn function ([#1475](#1475)). * Avoid multiple file tree traversals on bundle deploy ([#1493](#1493)). * Clean up unused code ([#1502](#1502)). * Use `dyn.InvalidValue` to indicate absence ([#1507](#1507)). * Add ApplyPythonMutator ([#1430](#1430)). * Set bool pointer to disable lock ([#1516](#1516)). * Allow the any type to be set to nil in `convert.FromTyped` ([#1518](#1518)). * Properly deal with nil values in `convert.FromTyped` ([#1511](#1511)). * Return `dyn.InvalidValue` instead of `dyn.NilValue` when errors happen ([#1514](#1514)). * PythonMutator: replace stdin/stdout with files ([#1512](#1512)). * Add context type and value to path rewriting ([#1525](#1525)). API Changes: * Added schedule CRUD commands to `databricks lakeview`. * Added subscription CRUD commands to `databricks lakeview`. * Added `databricks apps start` command. OpenAPI commit 7437dabb9dadee402c1fc060df4c1ce8cc5369f0 (2024-06-24) Dependency updates: * Bump golang.org/x/text from 0.15.0 to 0.16.0 ([#1482](#1482)). * Bump golang.org/x/term from 0.20.0 to 0.21.0 ([#1483](#1483)). * Bump golang.org/x/mod from 0.17.0 to 0.18.0 ([#1484](#1484)). * Bump golang.org/x/oauth2 from 0.20.0 to 0.21.0 ([#1485](#1485)). * Bump github.com/briandowns/spinner from 1.23.0 to 1.23.1 ([#1495](#1495)). * Bump github.com/spf13/cobra from 1.8.0 to 1.8.1 ([#1496](#1496)). * Bump github.com/databricks/databricks-sdk-go from 0.42.0 to 0.43.0 ([#1522](#1522)).
Merged
github-merge-queue bot
pushed a commit
that referenced
this pull request
Jun 26, 2024
CLI: * Add link to documentation for Homebrew installation to README ([#1505](#1505)). * Fix `databricks configure` to use `DATABRICKS_CONFIG_FILE` environment variable if exists as config file ([#1325](#1325)). Bundles: The Terraform upgrade to v1.48.0 includes a fix for library order not being respected. * Fix conditional in query in `default-sql` template ([#1479](#1479)). * Remove user credentials specified in the Git origin URL ([#1494](#1494)). * Serialize dynamic value for `bundle validate` output ([#1499](#1499)). * Override variables with lookup value even if values has default value set ([#1504](#1504)). * Pause quality monitors when "mode: development" is used ([#1481](#1481)). * Return `fs.ModeDir` for Git folders in the workspace ([#1521](#1521)). * Upgrade TF provider to 1.48.0 ([#1527](#1527)). * Added support for complex variables ([#1467](#1467)). Internal: * Add randIntn function ([#1475](#1475)). * Avoid multiple file tree traversals on bundle deploy ([#1493](#1493)). * Clean up unused code ([#1502](#1502)). * Use `dyn.InvalidValue` to indicate absence ([#1507](#1507)). * Add ApplyPythonMutator ([#1430](#1430)). * Set bool pointer to disable lock ([#1516](#1516)). * Allow the any type to be set to nil in `convert.FromTyped` ([#1518](#1518)). * Properly deal with nil values in `convert.FromTyped` ([#1511](#1511)). * Return `dyn.InvalidValue` instead of `dyn.NilValue` when errors happen ([#1514](#1514)). * PythonMutator: replace stdin/stdout with files ([#1512](#1512)). * Add context type and value to path rewriting ([#1525](#1525)). API Changes: * Added schedule CRUD commands to `databricks lakeview`. * Added subscription CRUD commands to `databricks lakeview`. * Added `databricks apps start` command. OpenAPI commit 7437dabb9dadee402c1fc060df4c1ce8cc5369f0 (2024-06-24) Dependency updates: * Bump golang.org/x/text from 0.15.0 to 0.16.0 ([#1482](#1482)). * Bump golang.org/x/term from 0.20.0 to 0.21.0 ([#1483](#1483)). * Bump golang.org/x/mod from 0.17.0 to 0.18.0 ([#1484](#1484)). * Bump golang.org/x/oauth2 from 0.20.0 to 0.21.0 ([#1485](#1485)). * Bump github.com/briandowns/spinner from 1.23.0 to 1.23.1 ([#1495](#1495)). * Bump github.com/spf13/cobra from 1.8.0 to 1.8.1 ([#1496](#1496)). * Bump github.com/databricks/databricks-sdk-go from 0.42.0 to 0.43.0 ([#1522](#1522)).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
To run bundle deploy from DBR we use an abstraction over the workspace import / export APIs to create a
filer.Filerand abstract the file system. Walking the file tree in such a filer is expensive and requires multiple API calls. This PR remove the two duplicate file tree walks that happen by caching the result.