Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: split consumer group for kafka connect worker v. connector #744

Merged
merged 1 commit into from
Dec 20, 2024

Conversation

fillyD
Copy link
Contributor

@fillyD fillyD commented Dec 20, 2024

No description provided.

@fillyD fillyD requested review from a team as code owners December 20, 2024 19:52
@uw-infra
Copy link

uw-infra commented Dec 20, 2024

Terraform run output for

Cluster: dev-aws
Module: pubsub/kafka-shared-msk-dev-enablement
Path: dev-aws/kafka-shared-msk/dev-enablement
Commit ID: e7535e3c1a90f4df0031ea24dfcd83be3864ad77
✅ Run Status: Ok, Run Summary: Plan: 1 to add, 3 to change, 0 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # kafka_topic.connect_configs will be updated in-place
  ~ resource "kafka_topic" "connect_configs" {
      ~ config             = {
          - "retention.ms"     = "1" -> null
            # (2 unchanged elements hidden)
        }
        id                 = "dev-enablement.connect-configs"
        name               = "dev-enablement.connect-configs"
        # (2 unchanged attributes hidden)
    }

  # kafka_topic.connect_offsets will be updated in-place
  ~ resource "kafka_topic" "connect_offsets" {
      ~ config             = {
          - "retention.ms"     = "1" -> null
            # (2 unchanged elements hidden)
        }
        id                 = "dev-enablement.connect-offsets"
        name               = "dev-enablement.connect-offsets"
        # (2 unchanged attributes hidden)
    }

  # kafka_topic.connect_status will be updated in-place
  ~ resource "kafka_topic" "connect_status" {
      ~ config             = {
          - "retention.ms"     = "1" -> null
            # (2 unchanged elements hidden)
        }
        id                 = "dev-enablement.connect-status"
        name               = "dev-enablement.connect-status"
        # (2 unchanged attributes hidden)
    }

  # module.kafka_connect_full_internal_topics.kafka_acl.group_acl["dev-enablement.kafka-connect-worker-group"] will be created
  + resource "kafka_acl" "group_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=dev-enablement/kafka-connect"
      + id                           = (known after apply)
      + resource_name                = "dev-enablement.kafka-connect-worker-group"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Group"
    }

Plan: 1 to add, 3 to change, 0 to destroy.

To manually trigger plan again please post @terraform-applier plan dev-aws/kafka-shared-msk/dev-enablement as comment.

@uw-infra
Copy link

uw-infra commented Dec 20, 2024

Terraform run output for

Cluster: dev-aws
Module: pubsub/kafka-shared-msk-account-identity
Path: dev-aws/kafka-shared-msk/account-identity
Commit ID: e7535e3c1a90f4df0031ea24dfcd83be3864ad77
✅ Run Status: Ok, Run Summary: Plan: 1 to add, 0 to change, 1 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # module.kafka_connect_read_legacy_account_events.kafka_acl.group_acl["dev-enablement.kafka-connect-consumer-group"] will be created
  + resource "kafka_acl" "group_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=dev-enablement/kafka-connect"
      + id                           = (known after apply)
      + resource_name                = "dev-enablement.kafka-connect-consumer-group"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Group"
    }

  # module.kafka_connect_read_legacy_account_events.kafka_acl.group_acl["dev-enablement.kafka-connect-group"] will be destroyed
  # (because key ["dev-enablement.kafka-connect-group"] is not in for_each map)
  - resource "kafka_acl" "group_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Read" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=dev-enablement/kafka-connect" -> null
      - id                           = "User:CN=dev-enablement/kafka-connect|*|Read|Allow|Group|dev-enablement.kafka-connect-group|Literal" -> null
      - resource_name                = "dev-enablement.kafka-connect-group" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Group" -> null
    }

Plan: 1 to add, 0 to change, 1 to destroy.

To manually trigger plan again please post @terraform-applier plan dev-aws/kafka-shared-msk/account-identity as comment.

@uw-infra
Copy link

uw-infra commented Dec 20, 2024

Terraform run output for

Cluster: dev-aws
Module: pubsub/kafka-shared-msk-customer-proposition
Path: dev-aws/kafka-shared-msk/customer-proposition
Commit ID: e7535e3c1a90f4df0031ea24dfcd83be3864ad77
✅ Run Status: Ok, Run Summary: Plan: 1 to add, 0 to change, 1 to destroy.
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # module.kafka_connect_read_service_status_events.kafka_acl.group_acl["dev-enablement.kafka-connect-consumer-group"] will be created
  + resource "kafka_acl" "group_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=dev-enablement/kafka-connect"
      + id                           = (known after apply)
      + resource_name                = "dev-enablement.kafka-connect-consumer-group"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Group"
    }

  # module.kafka_connect_read_service_status_events.kafka_acl.group_acl["dev-enablement.kafka-connect-group"] will be destroyed
  # (because key ["dev-enablement.kafka-connect-group"] is not in for_each map)
  - resource "kafka_acl" "group_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Read" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=dev-enablement/kafka-connect" -> null
      - id                           = "User:CN=dev-enablement/kafka-connect|*|Read|Allow|Group|dev-enablement.kafka-connect-group|Literal" -> null
      - resource_name                = "dev-enablement.kafka-connect-group" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Group" -> null
    }

Plan: 1 to add, 0 to change, 1 to destroy.

To manually trigger plan again please post @terraform-applier plan dev-aws/kafka-shared-msk/customer-proposition as comment.

@fillyD fillyD merged commit 79bcf9b into main Dec 20, 2024
2 checks passed
@fillyD fillyD deleted the fix/split-kc-cg branch December 20, 2024 20:00
@uw-infra
Copy link

Terraform run output for

Cluster: dev-aws
Module: pubsub/kafka-shared-msk-dev-enablement
Path: dev-aws/kafka-shared-msk/dev-enablement
Commit ID: 79bcf9b5e87f4e0d9984939d5cd848201318d9c9
✅ Run Status: Ok, Run Summary: Apply complete! Resources: 1 added, 3 changed, 0 destroyed
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place

Terraform will perform the following actions:

  # kafka_topic.connect_configs will be updated in-place
  ~ resource "kafka_topic" "connect_configs" {
      ~ config             = {
          - "retention.ms"     = "1" -> null
            # (2 unchanged elements hidden)
        }
        id                 = "dev-enablement.connect-configs"
        name               = "dev-enablement.connect-configs"
        # (2 unchanged attributes hidden)
    }

  # kafka_topic.connect_offsets will be updated in-place
  ~ resource "kafka_topic" "connect_offsets" {
      ~ config             = {
          - "retention.ms"     = "1" -> null
            # (2 unchanged elements hidden)
        }
        id                 = "dev-enablement.connect-offsets"
        name               = "dev-enablement.connect-offsets"
        # (2 unchanged attributes hidden)
    }

  # kafka_topic.connect_status will be updated in-place
  ~ resource "kafka_topic" "connect_status" {
      ~ config             = {
          - "retention.ms"     = "1" -> null
            # (2 unchanged elements hidden)
        }
        id                 = "dev-enablement.connect-status"
        name               = "dev-enablement.connect-status"
        # (2 unchanged attributes hidden)
    }

  # module.kafka_connect_full_internal_topics.kafka_acl.group_acl["dev-enablement.kafka-connect-worker-group"] will be created
  + resource "kafka_acl" "group_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=dev-enablement/kafka-connect"
      + id                           = (known after apply)
      + resource_name                = "dev-enablement.kafka-connect-worker-group"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Group"
    }

Plan: 1 to add, 3 to change, 0 to destroy.
module.kafka_connect_full_internal_topics.kafka_acl.group_acl["dev-enablement.kafka-connect-worker-group"]: Creating...
kafka_topic.connect_configs: Modifying... [id=dev-enablement.connect-configs]
kafka_topic.connect_status: Modifying... [id=dev-enablement.connect-status]
kafka_topic.connect_offsets: Modifying... [id=dev-enablement.connect-offsets]
module.kafka_connect_full_internal_topics.kafka_acl.group_acl["dev-enablement.kafka-connect-worker-group"]: Creation complete after 0s [id=User:CN=dev-enablement/kafka-connect|*|Read|Allow|Group|dev-enablement.kafka-connect-worker-group|Literal]
kafka_topic.connect_status: Modifications complete after 1s [id=dev-enablement.connect-status]
kafka_topic.connect_configs: Modifications complete after 1s [id=dev-enablement.connect-configs]
kafka_topic.connect_offsets: Modifications complete after 1s [id=dev-enablement.connect-offsets]

Warning: Argument is deprecated

  with provider["registry.terraform.io/mongey/kafka"],
  on __env.tf line 12, in provider "kafka":
  12: provider "kafka" {

This parameter is now deprecated and will be removed in a later release,
please use `client_key` instead.

(and one more similar warning elsewhere)

Apply complete! Resources: 1 added, 3 changed, 0 destroyed.

To manually trigger plan again please post @terraform-applier plan dev-aws/kafka-shared-msk/dev-enablement as comment.

@uw-infra
Copy link

Terraform run output for

Cluster: dev-aws
Module: pubsub/kafka-shared-msk-customer-proposition
Path: dev-aws/kafka-shared-msk/customer-proposition
Commit ID: 79bcf9b5e87f4e0d9984939d5cd848201318d9c9
✅ Run Status: Ok, Run Summary: Apply complete! Resources: 1 added, 0 changed, 1 destroyed
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # module.kafka_connect_read_service_status_events.kafka_acl.group_acl["dev-enablement.kafka-connect-consumer-group"] will be created
  + resource "kafka_acl" "group_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=dev-enablement/kafka-connect"
      + id                           = (known after apply)
      + resource_name                = "dev-enablement.kafka-connect-consumer-group"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Group"
    }

  # module.kafka_connect_read_service_status_events.kafka_acl.group_acl["dev-enablement.kafka-connect-group"] will be destroyed
  # (because key ["dev-enablement.kafka-connect-group"] is not in for_each map)
  - resource "kafka_acl" "group_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Read" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=dev-enablement/kafka-connect" -> null
      - id                           = "User:CN=dev-enablement/kafka-connect|*|Read|Allow|Group|dev-enablement.kafka-connect-group|Literal" -> null
      - resource_name                = "dev-enablement.kafka-connect-group" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Group" -> null
    }

Plan: 1 to add, 0 to change, 1 to destroy.
module.kafka_connect_read_service_status_events.kafka_acl.group_acl["dev-enablement.kafka-connect-group"]: Destroying... [id=User:CN=dev-enablement/kafka-connect|*|Read|Allow|Group|dev-enablement.kafka-connect-group|Literal]
module.kafka_connect_read_service_status_events.kafka_acl.group_acl["dev-enablement.kafka-connect-consumer-group"]: Creating...
module.kafka_connect_read_service_status_events.kafka_acl.group_acl["dev-enablement.kafka-connect-consumer-group"]: Creation complete after 0s [id=User:CN=dev-enablement/kafka-connect|*|Read|Allow|Group|dev-enablement.kafka-connect-consumer-group|Literal]
module.kafka_connect_read_service_status_events.kafka_acl.group_acl["dev-enablement.kafka-connect-group"]: Destruction complete after 0s

Warning: Argument is deprecated

  with provider["registry.terraform.io/mongey/kafka"],
  on __env.tf line 12, in provider "kafka":
  12: provider "kafka" {

This parameter is now deprecated and will be removed in a later release,
please use `client_cert` instead.

(and one more similar warning elsewhere)

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

To manually trigger plan again please post @terraform-applier plan dev-aws/kafka-shared-msk/customer-proposition as comment.

@uw-infra
Copy link

Terraform run output for

Cluster: dev-aws
Module: pubsub/kafka-shared-msk-account-identity
Path: dev-aws/kafka-shared-msk/account-identity
Commit ID: 79bcf9b5e87f4e0d9984939d5cd848201318d9c9
✅ Run Status: Ok, Run Summary: Apply complete! Resources: 1 added, 0 changed, 0 destroyed
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.kafka_connect_read_legacy_account_events.kafka_acl.group_acl["dev-enablement.kafka-connect-consumer-group"] will be created
  + resource "kafka_acl" "group_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=dev-enablement/kafka-connect"
      + id                           = (known after apply)
      + resource_name                = "dev-enablement.kafka-connect-consumer-group"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Group"
    }

Plan: 1 to add, 0 to change, 0 to destroy.
module.kafka_connect_read_legacy_account_events.kafka_acl.group_acl["dev-enablement.kafka-connect-consumer-group"]: Creating...
module.kafka_connect_read_legacy_account_events.kafka_acl.group_acl["dev-enablement.kafka-connect-consumer-group"]: Creation complete after 0s [id=User:CN=dev-enablement/kafka-connect|*|Read|Allow|Group|dev-enablement.kafka-connect-consumer-group|Literal]

Warning: Argument is deprecated

  with provider["registry.terraform.io/mongey/kafka"],
  on __env.tf line 12, in provider "kafka":
  12: provider "kafka" {

This parameter is now deprecated and will be removed in a later release,
please use `client_key` instead.

(and one more similar warning elsewhere)

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

To manually trigger plan again please post @terraform-applier plan dev-aws/kafka-shared-msk/account-identity as comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants