Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add vertexai/featureonlinestore resource #9632

Merged
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
97 changes: 94 additions & 3 deletions mmv1/products/vertexai/FeatureOnlineStore.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ self_link: projects/{{project}}/locations/{{region}}/featureOnlineStores/{{name}
update_verb: :PATCH
update_mask: true
references: !ruby/object:Api::Resource::ReferenceLinks
guides:
'Official Documentation': 'https://cloud.google.com/vertex-ai/docs'
api: https://cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.featureOnlineStores
async: !ruby/object:Api::OpAsync
operation: !ruby/object:Api::OpAsync::Operation
Expand All @@ -45,19 +47,47 @@ examples:
primary_resource_id: feature_online_store
vars:
name: example_feature_online_store
ignore_read_extra:
- force_destroy
- !ruby/object:Provider::Terraform::Examples
name: vertex_ai_featureonlinestore_with_beta_fields_optimized
primary_resource_id: featureonlinestore
vars:
name: example_feature_online_store_optimized
ignore_read_extra:
- force_destroy
min_version: beta
- !ruby/object:Provider::Terraform::Examples
name: vertex_ai_featureonlinestore_with_beta_fields_bigtable
primary_resource_id: featureonlinestore
vars:
name: example_feature_online_store_beta_bigtable
ignore_read_extra:
- force_destroy
min_version: beta
custom_code: !ruby/object:Provider::Terraform::CustomCode
pre_delete: templates/terraform/pre_delete/vertex_ai_force_delete.go.erb
virtual_fields:
- !ruby/object:Api::Type::Boolean
name: force_destroy
description:
If set to true, any FeatureViews and Features for this FeatureOnlineStore will also be deleted.
default_value: false
parameters:
- !ruby/object:Api::Type::String
name: region
description: The region of feature online store. eg us-central1
url_param_only: true
immutable: true
required: true
default_from_api: true
roaks3 marked this conversation as resolved.
Show resolved Hide resolved
properties:
- !ruby/object:Api::Type::String
name: name
description: The resource name of the Feature Online Store.
description: The resource name of the Feature Online Store. This value may be up to 60 characters, and
valid characters are [a-z0-9_]. The first character cannot be a number.
immutable: true
required: true
url_param_only: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know if the configuration for name needs to change as you've suggested here? My understanding is that the previous config was working because the name can be provided in the request body, and it is returned in the response body, so it is arguably a simpler approach.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've done this so that it is consistent with the legacy FeatureStore which has similar url structure and json request object. If the existing one is simpler and easy to understand, I can change it too. Pls let me know.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The precedence for vertexai is a bit fragmented unfortunately, with Endpoint being the primary exception to the pattern you mention. However, since most of the other resources do use url_param_only and pattern, I think it makes sense to keep them here.

However, are you sure we should remove required? What happens if a name is not specified?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct! The REST API documentation doesn't explicitly mention it as required field but when I try to run without this field specified, it throws an error. So, updated to include required : true now

pattern: projects/{{project}}/locations/{{region}}/featureOnlineStores/{{name}}
custom_flatten: templates/terraform/custom_flatten/name_from_self_link.erb
- !ruby/object:Api::Type::String
name: createTime
kkram01 marked this conversation as resolved.
Show resolved Hide resolved
Expand Down Expand Up @@ -103,3 +133,64 @@ properties:
A percentage of the cluster's CPU capacity. Can be from 10% to 80%.
When a cluster's CPU utilization exceeds the target that you have set,
Bigtable immediately adds nodes to the cluster. When CPU utilization is substantially lower than the target, Bigtable removes nodes. If not set will default to 50%.
- !ruby/object:Api::Type::NestedObject
name: optimized
description:
Contains settings for the Optimized store that will be created to serve featureValues for all FeatureViews under this FeatureOnlineStore
conflicts:
- embeddingManagement
exactly_one_of:
kkram01 marked this conversation as resolved.
Show resolved Hide resolved
- bigtable
- optimized
min_version: beta
properties:
[] # Meant to be an empty object with no properties - see here : https://cloud.google.com/vertex-ai/docs/reference/rest/v1beta1/projects.locations.featureOnlineStores#Optimized
# The fields below are necessary to include the "Optimized" transformation in the payload
send_empty_value: true
allow_empty_object: true

- !ruby/object:Api::Type::NestedObject
name: dedicatedServingEndpoint
description: |
The dedicated serving endpoint for this FeatureOnlineStore, which is different from common vertex service endpoint. Only need to set when you choose Optimized storage type or enable EmbeddingManagement. Will use public endpoint by default.
default_from_api: true
min_version: beta
properties:
- !ruby/object:Api::Type::String
name: publicEndpointDomainName
description: |
populated with the domain name to use for this FeatureOnlineStore
kkram01 marked this conversation as resolved.
Show resolved Hide resolved
output: true
- !ruby/object:Api::Type::String
name: serviceAttachment
description: |
name of the service attachment resource. Populated if private service connect is enabled and after FeatureViewSync is created.
output: true
- !ruby/object:Api::Type::NestedObject
name: privateServiceConnectConfig
description: |
Private service connect config.
properties:
- !ruby/object:Api::Type::Boolean
name: enablePrivateServiceConnect
required: true
description: |
If set to true, customers will use private service connection to send request. Otherwise, the connection will set to public endpoint.
- !ruby/object:Api::Type::Array
name: projectAllowlist
description: |
A list of Projects from which the forwarding rule will target the service attachment.
item_type: Api::Type::String
- !ruby/object:Api::Type::NestedObject
name: embeddingManagement
description: |
The settings for embedding management in FeatureOnlineStore. Embedding management can only be used with BigTable.
conflicts:
- optimized
min_version: beta
properties:
- !ruby/object:Api::Type::Boolean
name: enabled
description: |
Enable embedding management.
immutable: true
Original file line number Diff line number Diff line change
@@ -1,15 +1,14 @@
resource google_vertex_ai_feature_online_store "<%= ctx[:primary_resource_id] %>" {
name = "<%= ctx[:vars]['name'] %>"
region = "us-central1"
labels = {
label-one = "value-one"
}

bigtable {
auto_scaling {
min_node_count = 1
max_node_count = 2
cpu_utilization_target = 60
}
resource "google_vertex_ai_feature_online_store" "<%= ctx[:primary_resource_id] %>" {
name = "<%= ctx[:vars]['name'] %>"
labels = {
foo = "bar"
}
region = "us-central1"
bigtable {
auto_scaling {
min_node_count = 1
max_node_count = 3
cpu_utilization_target = 50
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
resource "google_vertex_ai_feature_online_store" "<%= ctx[:primary_resource_id] %>" {
provider = google-beta
name = "<%= ctx[:vars]['name'] %>"
labels = {
foo = "bar"
}
region = "us-central1"
bigtable {
auto_scaling {
min_node_count = 1
max_node_count = 2
cpu_utilization_target = 80
}
}
embedding_management {
enabled = true
}
force_destroy = true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably be removed, this will show up in our docs.

Follow-up question to confirm: when this config is run in our tests, I assume it will still be deleted without force_destroy because it is not using any feature views or features. Is that correct, or will there be leftover resources?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow

  1. Can we have force_destroy = true as set for one of the examples ? This would also mean that we'd be addressing the missing test report that is getting generated for force_destroy field and also an example of the usage of the field too.
  2. Yes, It'll still be deleted. I've run the tests manually in my test env and the resources got deleted as expected at the end of the test in both force_destroy = true and false cases.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, if you feel it is better to include for this example I think that makes sense, and can always be adjusted later. Thanks for checking that resources get deleted!

}

data "google_project" "project" {
provider = google-beta
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
resource "google_vertex_ai_feature_online_store" "<%= ctx[:primary_resource_id] %>" {
provider = google-beta
name = "<%= ctx[:vars]['name'] %>"
labels = {
foo = "bar"
}
region = "us-central1"
optimized {}
dedicated_serving_endpoint {
private_service_connect_config {
enable_private_service_connect = true
project_allowlist = [data.google_project.project.number]
}
}
}

data "google_project" "project" {
provider = google-beta
}


Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
##FeatureOnlineStore With BigTable
kkram01 marked this conversation as resolved.
Show resolved Hide resolved
kkram01 marked this conversation as resolved.
Show resolved Hide resolved
resource "google_vertex_ai_feature_online_store" "<%= ctx[:primary_resource_id] %>" {
name = "<%= ctx[:vars]['name'] %>"
labels = {
foo = "bar"
}
region = "us-central1"
bigtable {
auto_scaling {
min_node_count = 1
max_node_count = 3
cpu_utilization_target = 50
}
}
force_destroy = true
kkram01 marked this conversation as resolved.
Show resolved Hide resolved
}
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ func TestAccVertexAIFeatureOnlineStore_updated(t *testing.T) {
acctest.VcrTest(t, resource.TestCase{
PreCheck: func() { acctest.AccTestPreCheck(t) },
ProtoV5ProviderFactories: acctest.ProtoV5ProviderFactories(t),
CheckDestroy: testAccCheckVertexAIFeatureOnlineStoreDestroyProducer(t),
kkram01 marked this conversation as resolved.
Show resolved Hide resolved
Steps: []resource.TestStep{
{
Config: testAccVertexAIFeatureOnlineStore_basic(context),
Expand All @@ -27,7 +26,7 @@ func TestAccVertexAIFeatureOnlineStore_updated(t *testing.T) {
ResourceName: "google_vertex_ai_feature_online_store.feature_online_store",
ImportState: true,
ImportStateVerify: true,
ImportStateVerifyIgnore: []string{"etag", "region", "labels", "terraform_labels"},
ImportStateVerifyIgnore: []string{"name", "etag", "region", "force_destroy", "labels", "terraform_labels"},
},
{
Config: testAccVertexAIFeatureOnlineStore_updated(context),
Expand All @@ -36,7 +35,7 @@ func TestAccVertexAIFeatureOnlineStore_updated(t *testing.T) {
ResourceName: "google_vertex_ai_feature_online_store.feature_online_store",
ImportState: true,
ImportStateVerify: true,
ImportStateVerifyIgnore: []string{"etag", "region", "labels", "terraform_labels"},
ImportStateVerifyIgnore: []string{"name", "etag", "region", "force_destroy", "labels", "terraform_labels"},
},
},
})
Expand All @@ -58,6 +57,8 @@ resource google_vertex_ai_feature_online_store "feature_online_store" {
cpu_utilization_target = 60
}
}
}
force_destroy = true
}
`, context)
}
Expand All @@ -78,6 +79,8 @@ resource google_vertex_ai_feature_online_store "feature_online_store" {
max_node_count = 3
}
}
}
force_destroy = true
}
`, context)
}