Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add databricks_function resource #4189

Open
wants to merge 12 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
112 changes: 112 additions & 0 deletions docs/resources/function.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
---
subcategory: "Unity Catalog"
---
# databricks_function Resource

-> This resource source can only be used with a workspace-level provider.

Creates a [User-Defined Function (UDF)](https://docs.databricks.com/en/udf/unity-catalog.html) in Unity Catalog. UDFs can be defined using SQL, or external languages (e.g., Python) and are stored within [Unity Catalog schemas](../resources/schema.md).

## Example Usage

### SQL-based function:

```hcl
resource "databricks_catalog" "sandbox" {
name = "sandbox_example"
comment = "Catalog managed by Terraform"
}

resource "databricks_schema" "functions" {
catalog_name = databricks_catalog.sandbox.name
name = "functions_example"
comment = "Schema managed by Terraform"
}

resource "databricks_function" "calculate_bmi" {
name = "calculate_bmi"
catalog_name = databricks_catalog.sandbox.name
schema_name = databricks_schema.functions.name
input_params = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be easier for people to specify input_param as separate blocks, like we do in other resources

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although doc says that input_params is a block, and then there are parameters inside: https://docs.databricks.com/api/workspace/functions/create#input_params

{
name = "weight"
type = "DOUBLE"
},
{
name = "height"
type = "DOUBLE"
}
]
data_type = "DOUBLE"
routine_body = "SQL"
routine_definition = "weight / (height * height)"
language = "SQL"
is_deterministic = true
sql_data_access = "CONTAINS_SQL"
security_type = "DEFINER"
}
```

### Python-based function:

```hcl
resource "databricks_function" "calculate_bmi_py" {
name = "calculate_bmi_py"
catalog_name = databricks_catalog.sandbox.name
schema_name = databricks_schema.functions.name
input_params = [
{
name = "weight_kg"
type = "DOUBLE"
},
{
name = "height_m"
type = "DOUBLE"
}
]
data_type = "DOUBLE"
routine_body = "EXTERNAL"
routine_definition = "return weight_kg / (height_m ** 2)"
language = "Python"
is_deterministic = false
sql_data_access = "NO_SQL"
security_type = "DEFINER"
}
```

## Argument Reference

The following arguments are supported:

* `name` - (Required) The name of the function.
* `catalog_name` - (Required) The name of the parent [databricks_catalog](../resources/catalog.md).
* `schema_name` - (Required) The name of [databricks_schema](../resources/schema.md) where the function will reside.
* `input_params` - (Required) A list of objects specifying the input parameters for the function.
* `name` - (Required) The name of the parameter.
* `type` - (Required) The data type of the parameter (e.g., `DOUBLE`, `INT`, etc.).
* `data_type` - (Required) The return data type of the function (e.g., `DOUBLE`).
* `routine_body` - (Required) Specifies the body type of the function, either `SQL` for SQL-based functions or `EXTERNAL` for functions in external languages.
dgomez04 marked this conversation as resolved.
Show resolved Hide resolved
* `routine_definition` - (Required) The actual definition of the function, expressed in SQL or the specified external language.
* `language` - (Required) The language of the function, e.g., `SQL` or `Python`.
* `is_deterministic`- (Optional, `bool`) Whether the function is deterministic. Default is `true`.
* `sql_data_Access`- (Optional) The SQL data access level for the function. Possible values are:
* `CONTAINS_SQL` - The function contains SQL statements.
* `READS_SQL_DATA` - The function reads SQL data but does not modify it.
* `NO_SQL` - The function does not contain SQL.
* `security_type` - (Optional) The security type of the function, generally `DEFINER`.

## Attribute Reference

In addition to all arguments above, the following attributes are exported:
* `full_name` - Full name of the function in the form of `catalog_name.schema_name.function_name`.
* `created_at` - The time when this function was created, in epoch milliseconds.
* `created_by` - The username of the function's creator.
* `updated_at` - The time when this function was last updated, in epoch milliseconds.
* `updated_by` - The username of the last user to modify the function.

## Related Resources

The following resources are used in the same context:

* [databricks_schema](./schema.md) to get information about a single schema
* Data source [databricks_functions](../data-sources/functions.md) to get a list of functions under a specified location.
1 change: 1 addition & 0 deletions internal/providers/pluginfw/pluginfw_rollout_utils.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ var migratedDataSources = []func() datasource.DataSource{
var pluginFwOnlyResources = []func() resource.Resource{
// TODO Add resources here
sharing.ResourceShare, // Using the staging name (with pluginframework suffix)
catalog.ResourceFunction,
}

// List of data sources that have been onboarded to the plugin framework - not migrated from sdkv2.
Expand Down
226 changes: 226 additions & 0 deletions internal/providers/pluginfw/resources/catalog/resource_function.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,226 @@
package catalog

import (
"context"
"fmt"
"time"

"github.com/databricks/databricks-sdk-go"
"github.com/databricks/databricks-sdk-go/apierr"
"github.com/databricks/databricks-sdk-go/retries"
"github.com/databricks/databricks-sdk-go/service/catalog"
"github.com/databricks/terraform-provider-databricks/common"
pluginfwcommon "github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/common"
pluginfwcontext "github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/context"
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/converters"
"github.com/databricks/terraform-provider-databricks/internal/providers/pluginfw/tfschema"
"github.com/databricks/terraform-provider-databricks/internal/service/catalog_tf"
"github.com/hashicorp/terraform-plugin-framework/diag"
"github.com/hashicorp/terraform-plugin-framework/path"
"github.com/hashicorp/terraform-plugin-framework/resource"
"github.com/hashicorp/terraform-plugin-framework/resource/schema"
)

const resourceName = "function"

var _ resource.ResourceWithConfigure = &FunctionResource{}

func ResourceFunction() resource.Resource {
return &FunctionResource{}
}

func waitForFunction(ctx context.Context, w *databricks.WorkspaceClient, funcInfo *catalog.FunctionInfo) diag.Diagnostics {
const timeout = 5 * time.Minute
dgomez04 marked this conversation as resolved.
Show resolved Hide resolved

result, err := retries.Poll[catalog.FunctionInfo](ctx, timeout, func() (*catalog.FunctionInfo, *retries.Err) {
attempt, err := w.Functions.GetByName(ctx, funcInfo.FullName)
if err != nil {
if apierr.IsMissing(err) {
return nil, retries.Continue(fmt.Errorf("function %s is not yet available", funcInfo.FullName))
}
return nil, retries.Halt(fmt.Errorf("failed to get function: %s", err))
}
return attempt, nil
})

if err != nil {
return diag.Diagnostics{diag.NewErrorDiagnostic("failed to create function", err.Error())}
}

*funcInfo = *result
return nil
}
dgomez04 marked this conversation as resolved.
Show resolved Hide resolved

type FunctionResource struct {
Client *common.DatabricksClient
}

func (r *FunctionResource) Metadata(ctx context.Context, req resource.MetadataRequest, resp *resource.MetadataResponse) {
resp.TypeName = pluginfwcommon.GetDatabricksProductionName(resourceName)
}

func (r *FunctionResource) Schema(ctx context.Context, req resource.SchemaRequest, resp *resource.SchemaResponse) {
attrs, blocks := tfschema.ResourceStructToSchemaMap(catalog_tf.FunctionInfo{}, func(c tfschema.CustomizableSchema) tfschema.CustomizableSchema {
c.SetRequired("name")
c.SetRequired("catalog_name")
c.SetRequired("schema_name")
c.SetRequired("input_params")
c.SetRequired("data_type")
c.SetRequired("routine_body")
c.SetRequired("routine_defintion")
c.SetRequired("language")
dgomez04 marked this conversation as resolved.
Show resolved Hide resolved

c.SetReadOnly("full_name")
c.SetReadOnly("created_at")
c.SetReadOnly("created_by")
c.SetReadOnly("updated_at")
c.SetReadOnly("updated_by")

return c
})

resp.Schema = schema.Schema{
Description: "Terraform schema for Databricks Function",
Attributes: attrs,
Blocks: blocks,
}
}

func (r *FunctionResource) Configure(ctx context.Context, req resource.ConfigureRequest, resp *resource.ConfigureResponse) {
if r.Client == nil && req.ProviderData != nil {
r.Client = pluginfwcommon.ConfigureResource(req, resp)
}
}

func (r *FunctionResource) ImportState(ctx context.Context, req resource.ImportStateRequest, resp *resource.ImportStateResponse) {
resource.ImportStatePassthroughID(ctx, path.Root("full_name"), req, resp)
}

func (r *FunctionResource) Create(ctx context.Context, req resource.CreateRequest, resp *resource.CreateResponse) {
ctx = pluginfwcontext.SetUserAgentInResourceContext(ctx, resourceName)
w, diags := r.Client.GetWorkspaceClient()
resp.Diagnostics.Append(diags...)
if resp.Diagnostics.HasError() {
return
}

var planFunc catalog_tf.FunctionInfo
resp.Diagnostics.Append(req.Plan.Get(ctx, &planFunc)...)
if resp.Diagnostics.HasError() {
return
}

var createReq catalog.CreateFunctionRequest

resp.Diagnostics.Append(converters.TfSdkToGoSdkStruct(ctx, planFunc, &createReq)...)
if resp.Diagnostics.HasError() {
return
}

funcInfo, err := w.Functions.Create(ctx, createReq)
if err != nil {
resp.Diagnostics.AddError("failed to create function", err.Error())
dgomez04 marked this conversation as resolved.
Show resolved Hide resolved
}

resp.Diagnostics.Append(waitForFunction(ctx, w, funcInfo)...)
if resp.Diagnostics.HasError() {
return
}

resp.Diagnostics.Append(converters.GoSdkToTfSdkStruct(ctx, funcInfo, &planFunc)...)
if resp.Diagnostics.HasError() {
return
}

resp.Diagnostics.Append(resp.State.Set(ctx, planFunc)...)
}

func (r *FunctionResource) Update(ctx context.Context, req resource.UpdateRequest, resp *resource.UpdateResponse) {
ctx = pluginfwcontext.SetUserAgentInResourceContext(ctx, resourceName)
w, diags := r.Client.GetWorkspaceClient()
resp.Diagnostics.Append(diags...)
if resp.Diagnostics.HasError() {
return
}

var planFunc catalog_tf.FunctionInfo
resp.Diagnostics.Append(req.Plan.Get(ctx, &planFunc)...)
if resp.Diagnostics.HasError() {
return
}

var updateReq catalog.UpdateFunction

resp.Diagnostics.Append(converters.TfSdkToGoSdkStruct(ctx, planFunc, &updateReq)...)
if resp.Diagnostics.HasError() {
return
}

funcInfo, err := w.Functions.Update(ctx, updateReq)
dgomez04 marked this conversation as resolved.
Show resolved Hide resolved
if err != nil {
resp.Diagnostics.AddError("failed to update function", err.Error())
dgomez04 marked this conversation as resolved.
Show resolved Hide resolved
}

resp.Diagnostics.Append(converters.GoSdkToTfSdkStruct(ctx, funcInfo, &planFunc)...)
if resp.Diagnostics.HasError() {
return
}

resp.Diagnostics.Append(resp.State.Set(ctx, planFunc)...)
}

func (r *FunctionResource) Read(ctx context.Context, req resource.ReadRequest, resp *resource.ReadResponse) {
ctx = pluginfwcontext.SetUserAgentInResourceContext(ctx, resourceName)

w, diags := r.Client.GetWorkspaceClient()
resp.Diagnostics.Append(diags...)
if resp.Diagnostics.HasError() {
return
}

var stateFunc catalog_tf.FunctionInfo

resp.Diagnostics.Append(req.State.Get(ctx, &stateFunc)...)
if resp.Diagnostics.HasError() {
return
}

funcName := stateFunc.Name.ValueString()

funcInfo, err := w.Functions.GetByName(ctx, funcName)
if err != nil {
if apierr.IsMissing(err) {
resp.State.RemoveResource(ctx)
return
}
resp.Diagnostics.AddError("failed to get function", err.Error())
return
}

resp.Diagnostics.Append(converters.GoSdkToTfSdkStruct(ctx, funcInfo, &stateFunc)...)
if resp.Diagnostics.HasError() {
return
}

resp.Diagnostics.Append(resp.State.Set(ctx, stateFunc)...)
}

func (r *FunctionResource) Delete(ctx context.Context, req resource.DeleteRequest, resp *resource.DeleteResponse) {
ctx = pluginfwcontext.SetUserAgentInResourceContext(ctx, resourceName)
w, diags := r.Client.GetWorkspaceClient()
resp.Diagnostics.Append(diags...)
if resp.Diagnostics.HasError() {
return
}

var deleteReq catalog_tf.DeleteFunctionRequest
resp.Diagnostics.Append(req.State.GetAttribute(ctx, path.Root("full_name"), &deleteReq.Name)...)
if resp.Diagnostics.HasError() {
return
}

err := w.Functions.DeleteByName(ctx, deleteReq.Name.ValueString())
if err != nil && !apierr.IsMissing(err) {
resp.Diagnostics.AddError("failed to delete function", err.Error())
}
}
Loading
Loading