[Feature] Added `no_wait` option for clusters to skip waiting to start on cluster creation #3953

andrewnester · 2024-08-27T11:08:45Z

Changes

Added no_wait option for clusters to skip waiting to start on cluster creation

Tests

make test run locally
relevant change in docs/ folder
covered with integration tests in internal/acceptance
relevant acceptance tests are passing
using Go SDK

Manually run the following configuration to confirm everything works

data "databricks_node_type" "smallest" {
  local_disk = true
}

data "databricks_spark_version" "latest_lts" {
  long_term_support = true
}

resource "databricks_cluster" "shared_autoscaling" {
  cluster_name            = "Andrew Nester TF Test"
  spark_version           = data.databricks_spark_version.latest_lts.id
  node_type_id            = data.databricks_node_type.smallest.id
  autotermination_minutes = 20
  autoscale {
    min_workers = 1
    max_workers = 50
  }
  no_wait = true

  library {
    pypi {
        package = "fbprophet==0.6"
    }
  }
}

… creation

pietern · 2024-08-28T11:42:28Z

clusters/resource_cluster.go

@@ -437,6 +445,20 @@ func resourceClusterCreate(ctx context.Context, d *schema.ResourceData, c *commo
 		}); err != nil {
 			return err
 		}
+	}


Can you confirm that library installation on a not-yet-started cluster works?

Yes, it does work correctly

Could you please check? It's not working for library installation.

pietern

@mgyucht Could you TAL at this?

docs/resources/cluster.md

Co-authored-by: Pieter Noordhuis <[email protected]>

mgyucht

LGTM, but one question about the no_wait behavior.

mgyucht · 2024-09-02T11:11:21Z

docs/resources/cluster.md

@@ -51,6 +51,7 @@ resource "databricks_cluster" "shared_autoscaling" {
 * `custom_tags` - (Optional) Additional tags for cluster resources. Databricks will tag all cluster resources (e.g., AWS EC2 instances and EBS volumes) with these tags in addition to `default_tags`. If a custom cluster tag has the same name as a default cluster tag, the custom tag is prefixed with an `x_` when it is propagated.
 * `spark_conf` - (Optional) Map with key-value pairs to fine-tune Spark clusters, where you can provide custom [Spark configuration properties](https://spark.apache.org/docs/latest/configuration.html) in a cluster configuration.
 * `is_pinned` - (Optional) boolean value specifying if the cluster is pinned (not pinned by default). You must be a Databricks administrator to use this.  The pinned clusters' maximum number is [limited to 100](https://docs.databricks.com/clusters/clusters-manage.html#pin-a-cluster), so `apply` may fail if you have more than that (this number may change over time, so check Databricks documentation for actual number).
+* `no_wait` - (Optional) boolean value specifying if the provider doesn't need to wait for the cluster to be started when creating it. Defaults to false.


Should we do the same for updates?

Discussed offline: decided not to add the flag for update for now because default behaviour for update at the moment is already no_wait and changing default + introducing flag might be too complicated. We can follow up on this later if needed

docs/resources/cluster.md

### New Features and Improvements * Added `no_wait` option for clusters to skip waiting to start on cluster creation ([#3953](#3953)). * Introduced Plugin Framework ([#3920](#3920)). ### Bug Fixes * Add suppress diff for `azure_attributes.spot_bid_max_price` in `databricks_instance_pool` ([#3970](#3970)). * Correctly send workload_type fields in `databricks_cluster` to allow users to disable usage in certain contexts ([#3972](#3972)). * Fix `TestAccClusterResource_WorkloadType` ([#3989](#3989)). * Fix `databricks_sql_table` treatment of properties ([#3925](#3925)). * Force send fields for settings resources ([#3978](#3978)). * Handle cluster deletion in `databricks_library` read ([#3909](#3909)). * Make subscriptions optional for SqlAlertTask ([#3983](#3983)). ### Documentation * Add troubleshooting guide for Provider "registry.terraform.io/databricks/databricks" planned an invalid value ([#3961](#3961)). * Adopt official naming of Mosaic AI Vector Search ([#3971](#3971)). * Document Terraform 1.0 as minimum version ([#3952](#3952)). * Mention Salesforce as supported type in `databricks_connection` ([#3949](#3949)). * Reimplement Azure Databricks deployment guide to use VNet injection & NPIP ([#3986](#3986)). * Resolves [#3127](#3127): Remove deprecated account_id field from mws_credentials resource ([#3974](#3974)). * Small Grammar Corrections in Docs ([#4006](#4006)). * Update `databricks_vector_search_index` docs to match latest SDK ([#4008](#4008)). * Update aws_unity_catalog_assume_role_policy.md ([#3968](#3968)). * Update documentation regarding authentication with Azure-managed Service Principal using GITHUB OIDC ([#3932](#3932)). * Update metastore_assignment.md to properly reflect possible usage ([#3967](#3967)). * Update minimum supported terraform version to 1.1.5 ([#3965](#3965)). * Update resources diagram to include newer resources ([#3962](#3962)). * Update workspace_binding import command ([#3944](#3944)). * fix possible values for `securable_type` in `databricks_workspace_binding` ([#3942](#3942)). ### Internal Changes * Add `AddPlanModifer` method for AttributeBuilder ([#4009](#4009)). * Add integration tests for volumes and quality monitor plugin framework ([#3975](#3975)). * Add support for `computed` tag in TfSDK Structs ([#4005](#4005)). * Added `databricks_quality_monitor` resource and `databricks_volumes` data source to plugin framework ([#3958](#3958)). * Allow vector search tests to fail ([#3959](#3959)). * Make test utils public and move integration test for quality monitor ([#3993](#3993)). * Migrate Share resource to Go SDK ([#3916](#3916)). * Migrate imports for terraform plugin framework + update init test provider factory ([#3943](#3943)). * Move volumes test next to plugin framework data source ([#3995](#3995)). * Refactor provider and related packages ([#3940](#3940)). * Support import in acceptance test + adding import state for quality monitor ([#3994](#3994)). ### Dependency Updates * Bump github.com/hashicorp/hcl/v2 from 2.21.0 to 2.22.0 ([#3948](#3948)). * Update Go SDK to 0.46.0 ([#4007](#4007)). ### Exporter * Don't generate instance pools if the pool name is empty ([#3960](#3960)).

### New Features and Improvements * Add support for filters in `databricks_clusters` data source ([#4014](#4014)). * Added `no_wait` option for clusters to skip waiting to start on cluster creation ([#3953](#3953)). * Introduced Plugin Framework ([#3920](#3920)). * Library plugin framework migration ([#3979](#3979)). ### Bug Fixes * Add suppress diff for `azure_attributes.spot_bid_max_price` in `databricks_instance_pool` ([#3970](#3970)). * Correctly send workload_type fields in `databricks_cluster` to allow users to disable usage in certain contexts ([#3972](#3972)). * Fix `TestAccClusterResource_WorkloadType` ([#3989](#3989)). * Fix `databricks_sql_table` treatment of properties ([#3925](#3925)). * Force send fields for settings resources ([#3978](#3978)). * Handle cluster deletion in `databricks_library` read ([#3909](#3909)). * Make subscriptions optional for SqlAlertTask ([#3983](#3983)). * Permanently delete `ERROR` and `TERMINATED` state clusters if their creation fails ([#4021](#4021)). ### Documentation * Add troubleshooting guide for Provider "registry.terraform.io/databricks/databricks" planned an invalid value ([#3961](#3961)). * Adopt official naming of Mosaic AI Vector Search ([#3971](#3971)). * Document Terraform 1.0 as minimum version ([#3952](#3952)). * Mention Salesforce as supported type in `databricks_connection` ([#3949](#3949)). * Reimplement Azure Databricks deployment guide to use VNet injection & NPIP ([#3986](#3986)). * Resolves [#3127](#3127): Remove deprecated account_id field from mws_credentials resource ([#3974](#3974)). * Small Grammar Corrections in Docs ([#4006](#4006)). * Update `databricks_vector_search_index` docs to match latest SDK ([#4008](#4008)). * Update aws_unity_catalog_assume_role_policy.md ([#3968](#3968)). * Update documentation regarding authentication with Azure-managed Service Principal using GITHUB OIDC ([#3932](#3932)). * Update metastore_assignment.md to properly reflect possible usage ([#3967](#3967)). * Update minimum supported terraform version to 1.1.5 ([#3965](#3965)). * Update resources diagram to include newer resources ([#3962](#3962)). * Update workspace_binding import command ([#3944](#3944)). * fix possible values for `securable_type` in `databricks_workspace_binding` ([#3942](#3942)). ### Internal Changes * Add `AddPlanModifer` method for AttributeBuilder ([#4009](#4009)). * Add integration tests for volumes and quality monitor plugin framework ([#3975](#3975)). * Add support for `computed` tag in TfSDK Structs ([#4005](#4005)). * Added `databricks_quality_monitor` resource and `databricks_volumes` data source to plugin framework ([#3958](#3958)). * Allow vector search tests to fail ([#3959](#3959)). * Clean up comments in library resource ([#4015](#4015)). * Fix irregularities in plugin framework converter function errors ([#4010](#4010)). * Make test utils public and move integration test for quality monitor ([#3993](#3993)). * Migrate Share resource to Go SDK ([#3916](#3916)). * Migrate `databricks_cluster` data source to plugin framework ([#3988](#3988)). * Migrate imports for terraform plugin framework + update init test provider factory ([#3943](#3943)). * Move volumes test next to plugin framework data source ([#3995](#3995)). * Refactor provider and related packages ([#3940](#3940)). * Support import in acceptance test + adding import state for quality monitor ([#3994](#3994)). ### Dependency Updates * Bump github.com/hashicorp/hcl/v2 from 2.21.0 to 2.22.0 ([#3948](#3948)). * Update Go SDK to 0.46.0 ([#4007](#4007)). ### Exporter * Don't generate instance pools if the pool name is empty ([#3960](#3960)).

### New Features and Improvements * Add support for filters in `databricks_clusters` data source ([#4014](#4014)). * Added `no_wait` option for clusters to skip waiting to start on cluster creation ([#3953](#3953)). * Introduced Plugin Framework ([#3920](#3920)). ### Bug Fixes * Add suppress diff for `azure_attributes.spot_bid_max_price` in `databricks_instance_pool` ([#3970](#3970)). * Correctly send workload_type fields in `databricks_cluster` to allow users to disable usage in certain contexts ([#3972](#3972)). * Fix `databricks_sql_table` treatment of properties ([#3925](#3925)). * Force send fields for settings resources ([#3978](#3978)). * Handle cluster deletion in `databricks_library` read ([#3909](#3909)). * Make subscriptions optional for SqlAlertTask ([#3983](#3983)). * Permanently delete `ERROR` and `TERMINATED` state clusters if their creation fails ([#4021](#4021)). ### Documentation * Add troubleshooting guide for Provider "registry.terraform.io/databricks/databricks" planned an invalid value ([#3961](#3961)). * Adopt official naming of Mosaic AI Vector Search ([#3971](#3971)). * Document Terraform 1.0 as minimum version ([#3952](#3952)). * Mention Salesforce as supported type in `databricks_connection` ([#3949](#3949)). * Reimplement Azure Databricks deployment guide to use VNet injection & NPIP ([#3986](#3986)). * Resolves [#3127](#3127): Remove deprecated account_id field from mws_credentials resource ([#3974](#3974)). * Small Grammar Corrections in Docs ([#4006](#4006)). * Update `databricks_vector_search_index` docs to match latest SDK ([#4008](#4008)). * Update aws_unity_catalog_assume_role_policy.md ([#3968](#3968)). * Update documentation regarding authentication with Azure-managed Service Principal using GITHUB OIDC ([#3932](#3932)). * Update metastore_assignment.md to properly reflect possible usage ([#3967](#3967)). * Update minimum supported terraform version to 1.1.5 ([#3965](#3965)). * Update resources diagram to include newer resources ([#3962](#3962)). * Update workspace_binding import command ([#3944](#3944)). * fix possible values for `securable_type` in `databricks_workspace_binding` ([#3942](#3942)). ### Internal Changes * Add `AddPlanModifer` method for AttributeBuilder ([#4009](#4009)). * Add integration tests for volumes and quality monitor plugin framework ([#3975](#3975)). * Add support for `computed` tag in TfSDK Structs ([#4005](#4005)). * Added `databricks_quality_monitor` resource and `databricks_volumes` data source to plugin framework ([#3958](#3958)). * Allow vector search tests to fail ([#3959](#3959)). * Clean up comments in library resource ([#4015](#4015)). * Fix irregularities in plugin framework converter function errors ([#4010](#4010)). * Make test utils public and move integration test for quality monitor ([#3993](#3993)). * Migrate Share resource to Go SDK ([#3916](#3916)). * Migrate `databricks_cluster` data source to plugin framework ([#3988](#3988)). * Migrate imports for terraform plugin framework + update init test provider factory ([#3943](#3943)). * Move volumes test next to plugin framework data source ([#3995](#3995)). * Refactor provider and related packages ([#3940](#3940)). * Support import in acceptance test + adding import state for quality monitor ([#3994](#3994)). * Library plugin framework migration ([#3979](#3979)). * Fix `TestAccClusterResource_WorkloadType` ([#3989](#3989)). ### Dependency Updates * Bump github.com/hashicorp/hcl/v2 from 2.21.0 to 2.22.0 ([#3948](#3948)). * Update Go SDK to 0.46.0 ([#4007](#4007)). ### Exporter * Don't generate instance pools if the pool name is empty ([#3960](#3960)). * Expand list of non-interactive clusters ([#4023](#4023)). * Ignore databricks_artifact_allowlist with zero artifact_matcher blocks ([#4019](#4019)). ## [Release] Release v1.51.0 ### Breaking Changes With this release, only protocol version 6 will be supported which is compatible with terraform CLI version 1.1.5 and later. If you are using an older version of the terraform CLI, please upgrade it to use this and further releases of Databricks terraform provider. ### New Features and Improvements * Automatically create `parent_path` folder when creating `databricks_dashboard resource` if it doesn't exist ([#3778](#3778)). ### Bug Fixes * Fixed logging for underlying Go SDK ([#3917](#3917)). * Remove not necessary field in `databricks_job` schema ([#3907](#3907)). ### Internal Changes * Add AttributeBuilder for Plugin Framework schema ([#3922](#3922)). * Add CustomizableSchema for Plugin Framework ([#3927](#3927)). * Add StructToSchema for Plugin Framework ([#3928](#3928)). * Add codegen template and generated files for tfsdk structs ([#3911](#3911)). * Add converter functions and tests for plugin framework ([#3914](#3914)). * Added support to use protocol version 6 provider server for SDK plugin ([#3862](#3862)). * Bump Go SDK to v0.45.0 ([#3933](#3933)). * Change name with the aliases in codegen template ([#3936](#3936)). * Update jd version from latest to 1.8.1 ([#3915](#3915)). * Upgrade `staticcheck` to v0.5.1 to get Go 1.23 support ([#3931](#3931)). * OPENAPI_SHA check ([#3935](#3935)). * Use generic error for missing clusters ([#3938](#3938)) ### Exporter * Better support for notebooks with /Workspace path ([#3901](#3901)). * Improve exporting of DLT and test coverage ([#3898](#3898)).

Added no_wait option for clusters to skip waiting to start on cluster…

512e055

… creation

andrewnester requested review from mgyucht, tanmay-db and hectorcast-db August 27, 2024 11:08

andrewnester requested review from a team as code owners August 27, 2024 11:08

andrewnester changed the title ~~Added no_wait option for clusters to skip waiting to start on cluster creation~~ [Feature] Added no_wait option for clusters to skip waiting to start on cluster creation Aug 27, 2024

pietern reviewed Aug 28, 2024

View reviewed changes

andrewnester requested a review from pietern August 29, 2024 14:53

pietern approved these changes Sep 2, 2024

View reviewed changes

docs/resources/cluster.md Outdated Show resolved Hide resolved

Update docs/resources/cluster.md

a060fd2

Co-authored-by: Pieter Noordhuis <[email protected]>

mgyucht reviewed Sep 2, 2024

View reviewed changes

added integ test

928e8e0

andrewnester requested a review from mgyucht September 2, 2024 12:36

mgyucht approved these changes Sep 2, 2024

View reviewed changes

mgyucht reviewed Sep 2, 2024

View reviewed changes

docs/resources/cluster.md Outdated Show resolved Hide resolved

Update docs/resources/cluster.md

20cbc41

mgyucht enabled auto-merge September 2, 2024 12:56

mgyucht added this pull request to the merge queue Sep 2, 2024

Merged via the queue into main with commit e4c36c8 Sep 2, 2024
7 checks passed

mgyucht deleted the feature/clusters-no-wait branch September 2, 2024 13:02

hectorcast-db mentioned this pull request Sep 12, 2024

[Release] Release v1.52.0 #4013

Closed

tanmay-db mentioned this pull request Sep 16, 2024

[Release] Release v1.52.0 #4022

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Added `no_wait` option for clusters to skip waiting to start on cluster creation #3953

[Feature] Added `no_wait` option for clusters to skip waiting to start on cluster creation #3953

andrewnester commented Aug 27, 2024 •

edited

Loading

pietern Aug 28, 2024

andrewnester Aug 29, 2024

luisalves85 Oct 16, 2024

pietern left a comment

mgyucht left a comment

mgyucht Sep 2, 2024

andrewnester Sep 2, 2024

[Feature] Added no_wait option for clusters to skip waiting to start on cluster creation #3953

[Feature] Added no_wait option for clusters to skip waiting to start on cluster creation #3953

Conversation

andrewnester commented Aug 27, 2024 • edited Loading

Changes

Tests

pietern Aug 28, 2024

Choose a reason for hiding this comment

andrewnester Aug 29, 2024

Choose a reason for hiding this comment

luisalves85 Oct 16, 2024

Choose a reason for hiding this comment

pietern left a comment

Choose a reason for hiding this comment

mgyucht left a comment

Choose a reason for hiding this comment

mgyucht Sep 2, 2024

Choose a reason for hiding this comment

andrewnester Sep 2, 2024

Choose a reason for hiding this comment

[Feature] Added `no_wait` option for clusters to skip waiting to start on cluster creation #3953

[Feature] Added `no_wait` option for clusters to skip waiting to start on cluster creation #3953

andrewnester commented Aug 27, 2024 •

edited

Loading