Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update resources for throughput experiment #716

Merged
merged 1 commit into from
Nov 28, 2024
Merged

Update resources for throughput experiment #716

merged 1 commit into from
Nov 28, 2024

Conversation

matthewhughes-uw
Copy link
Contributor

  • Drop old ACLs: these only existed because the tls-app module used to create quotas and we didn't want those on the topic, the module no longer creates those quotas, so just rely on the module
  • Unify consumer requirements into a single module: so the single deployment can consume from one topic and write to another

Ticket: DENA-1069

@matthewhughes-uw
Update resources for throughput experiment
663ff25 
* Drop old ACLs: these only existed because the `tls-app` module used to
  create quotas and we didn't want those on the topic, the module
  no longer creates those quotas, so just rely on the module
* Unify consumer requirements into a single module: so the single
  deployment can consume from one topic and write to another

Ticket: DENA-1069
@matthewhughes-uw matthewhughes-uw requested a review from a team as a code owner November 28, 2024 15:03
@linear Linear
Copy link

linear bot commented Nov 28, 2024

DENA-1069 Add throughput test

@uw-infra
Copy link

uw-infra commented Nov 28, 2024 •
edited
Loading

Terraform run output for

Cluster: dev-aws
Module: pubsub/kafka-shared-msk-pubsub
Path: dev-aws/kafka-shared-msk/pubsub
Commit ID: 663ff257f3e824aa5676be1896bcee6354515eed
✅ Run Status: Ok, Run Summary: Plan: 3 to add, 0 to change, 3 to destroy. 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # kafka_acl.throughput_consumer_group_acl will be destroyed
  # (because kafka_acl.throughput_consumer_group_acl is not in configuration)
  - resource "kafka_acl" "throughput_consumer_group_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Read" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=pubsub/throughput-test-consumer" -> null
      - id                           = "User:CN=pubsub/throughput-test-consumer|*|Read|Allow|Group|pubsub.consume-throughput|Literal" -> null
      - resource_name                = "pubsub.consume-throughput" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Group" -> null
    }

  # kafka_acl.throughput_consumer_topic_acl will be destroyed
  # (because kafka_acl.throughput_consumer_topic_acl is not in configuration)
  - resource "kafka_acl" "throughput_consumer_topic_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Read" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=pubsub/throughput-test-consumer" -> null
      - id                           = "User:CN=pubsub/throughput-test-consumer|*|Read|Allow|Topic|pubsub.throughput-test|Literal" -> null
      - resource_name                = "pubsub.throughput-test" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Topic" -> null
    }

  # module.example_producer_throughput_out.kafka_acl.producer_acl["pubsub.throughput-test-out"] will be destroyed
  # (because kafka_acl.producer_acl is not in configuration)
  - resource "kafka_acl" "producer_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Write" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=pubsub/example-throughput-producer-out" -> null
      - id                           = "User:CN=pubsub/example-throughput-producer-out|*|Write|Allow|Topic|pubsub.throughput-test-out|Literal" -> null
      - resource_name                = "pubsub.throughput-test-out" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Topic" -> null
    }

  # module.throughput_test_consumer.kafka_acl.group_acl["pubsub.throughput-test-consumer"] will be created
  + resource "kafka_acl" "group_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=pubsub/throughput-test-consumer"
      + id                           = (known after apply)
      + resource_name                = "pubsub.throughput-test-consumer"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Group"
    }

  # module.throughput_test_consumer.kafka_acl.producer_acl["pubsub.throughput-test-out"] will be created
  + resource "kafka_acl" "producer_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Write"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=pubsub/throughput-test-consumer"
      + id                           = (known after apply)
      + resource_name                = "pubsub.throughput-test-out"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

  # module.throughput_test_consumer.kafka_acl.topic_acl["pubsub.throughput-test"] will be created
  + resource "kafka_acl" "topic_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=pubsub/throughput-test-consumer"
      + id                           = (known after apply)
      + resource_name                = "pubsub.throughput-test"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

Plan: 3 to add, 0 to change, 3 to destroy.

To manually trigger plan again please post @terraform-applier plan dev-aws/kafka-shared-msk/pubsub as comment.

1 similar comment
@uw-infra
Copy link

uw-infra commented Nov 28, 2024 •
edited
Loading

Terraform run output for

Cluster: dev-aws
Module: pubsub/kafka-shared-msk-pubsub
Path: dev-aws/kafka-shared-msk/pubsub
Commit ID: 663ff257f3e824aa5676be1896bcee6354515eed
✅ Run Status: Ok, Run Summary: Plan: 3 to add, 0 to change, 3 to destroy. 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # kafka_acl.throughput_consumer_group_acl will be destroyed
  # (because kafka_acl.throughput_consumer_group_acl is not in configuration)
  - resource "kafka_acl" "throughput_consumer_group_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Read" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=pubsub/throughput-test-consumer" -> null
      - id                           = "User:CN=pubsub/throughput-test-consumer|*|Read|Allow|Group|pubsub.consume-throughput|Literal" -> null
      - resource_name                = "pubsub.consume-throughput" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Group" -> null
    }

  # kafka_acl.throughput_consumer_topic_acl will be destroyed
  # (because kafka_acl.throughput_consumer_topic_acl is not in configuration)
  - resource "kafka_acl" "throughput_consumer_topic_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Read" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=pubsub/throughput-test-consumer" -> null
      - id                           = "User:CN=pubsub/throughput-test-consumer|*|Read|Allow|Topic|pubsub.throughput-test|Literal" -> null
      - resource_name                = "pubsub.throughput-test" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Topic" -> null
    }

  # module.example_producer_throughput_out.kafka_acl.producer_acl["pubsub.throughput-test-out"] will be destroyed
  # (because kafka_acl.producer_acl is not in configuration)
  - resource "kafka_acl" "producer_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Write" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=pubsub/example-throughput-producer-out" -> null
      - id                           = "User:CN=pubsub/example-throughput-producer-out|*|Write|Allow|Topic|pubsub.throughput-test-out|Literal" -> null
      - resource_name                = "pubsub.throughput-test-out" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Topic" -> null
    }

  # module.throughput_test_consumer.kafka_acl.group_acl["pubsub.throughput-test-consumer"] will be created
  + resource "kafka_acl" "group_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=pubsub/throughput-test-consumer"
      + id                           = (known after apply)
      + resource_name                = "pubsub.throughput-test-consumer"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Group"
    }

  # module.throughput_test_consumer.kafka_acl.producer_acl["pubsub.throughput-test-out"] will be created
  + resource "kafka_acl" "producer_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Write"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=pubsub/throughput-test-consumer"
      + id                           = (known after apply)
      + resource_name                = "pubsub.throughput-test-out"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

  # module.throughput_test_consumer.kafka_acl.topic_acl["pubsub.throughput-test"] will be created
  + resource "kafka_acl" "topic_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=pubsub/throughput-test-consumer"
      + id                           = (known after apply)
      + resource_name                = "pubsub.throughput-test"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

Plan: 3 to add, 0 to change, 3 to destroy.

To manually trigger plan again please post @terraform-applier plan dev-aws/kafka-shared-msk/pubsub as comment.

@matthewhughes-uw matthewhughes-uw merged commit 73a05e2 into main Nov 28, 2024
2 checks passed
@matthewhughes-uw matthewhughes-uw deleted the mhughes-DENA-1069-update-throughput-roles branch November 28, 2024 15:14
@uw-infra
Copy link

Terraform run output for

Cluster: dev-aws
Module: pubsub/kafka-shared-msk-pubsub
Path: dev-aws/kafka-shared-msk/pubsub
Commit ID: 73a05e28c49920380e5909f8da2091d2f41e2e90
✅ Run Status: Ok, Run Summary: Apply complete! Resources: 3 added, 0 changed, 3 destroyed 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # kafka_acl.throughput_consumer_group_acl will be destroyed
  # (because kafka_acl.throughput_consumer_group_acl is not in configuration)
  - resource "kafka_acl" "throughput_consumer_group_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Read" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=pubsub/throughput-test-consumer" -> null
      - id                           = "User:CN=pubsub/throughput-test-consumer|*|Read|Allow|Group|pubsub.consume-throughput|Literal" -> null
      - resource_name                = "pubsub.consume-throughput" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Group" -> null
    }

  # kafka_acl.throughput_consumer_topic_acl will be destroyed
  # (because kafka_acl.throughput_consumer_topic_acl is not in configuration)
  - resource "kafka_acl" "throughput_consumer_topic_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Read" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=pubsub/throughput-test-consumer" -> null
      - id                           = "User:CN=pubsub/throughput-test-consumer|*|Read|Allow|Topic|pubsub.throughput-test|Literal" -> null
      - resource_name                = "pubsub.throughput-test" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Topic" -> null
    }

  # module.example_producer_throughput_out.kafka_acl.producer_acl["pubsub.throughput-test-out"] will be destroyed
  # (because kafka_acl.producer_acl is not in configuration)
  - resource "kafka_acl" "producer_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Write" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=pubsub/example-throughput-producer-out" -> null
      - id                           = "User:CN=pubsub/example-throughput-producer-out|*|Write|Allow|Topic|pubsub.throughput-test-out|Literal" -> null
      - resource_name                = "pubsub.throughput-test-out" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Topic" -> null
    }

  # module.throughput_test_consumer.kafka_acl.group_acl["pubsub.throughput-test-consumer"] will be created
  + resource "kafka_acl" "group_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=pubsub/throughput-test-consumer"
      + id                           = (known after apply)
      + resource_name                = "pubsub.throughput-test-consumer"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Group"
    }

  # module.throughput_test_consumer.kafka_acl.producer_acl["pubsub.throughput-test-out"] will be created
  + resource "kafka_acl" "producer_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Write"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=pubsub/throughput-test-consumer"
      + id                           = (known after apply)
      + resource_name                = "pubsub.throughput-test-out"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

  # module.throughput_test_consumer.kafka_acl.topic_acl["pubsub.throughput-test"] will be created
  + resource "kafka_acl" "topic_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=pubsub/throughput-test-consumer"
      + id                           = (known after apply)
      + resource_name                = "pubsub.throughput-test"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

Plan: 3 to add, 0 to change, 3 to destroy.
kafka_acl.throughput_consumer_topic_acl: Destroying... [id=User:CN=pubsub/throughput-test-consumer|*|Read|Allow|Topic|pubsub.throughput-test|Literal]
module.example_producer_throughput_out.kafka_acl.producer_acl["pubsub.throughput-test-out"]: Destroying... [id=User:CN=pubsub/example-throughput-producer-out|*|Write|Allow|Topic|pubsub.throughput-test-out|Literal]
kafka_acl.throughput_consumer_group_acl: Destroying... [id=User:CN=pubsub/throughput-test-consumer|*|Read|Allow|Group|pubsub.consume-throughput|Literal]
module.throughput_test_consumer.kafka_acl.group_acl["pubsub.throughput-test-consumer"]: Creating...
module.throughput_test_consumer.kafka_acl.topic_acl["pubsub.throughput-test"]: Creating...
module.throughput_test_consumer.kafka_acl.producer_acl["pubsub.throughput-test-out"]: Creating...
kafka_acl.throughput_consumer_group_acl: Destruction complete after 0s
module.example_producer_throughput_out.kafka_acl.producer_acl["pubsub.throughput-test-out"]: Destruction complete after 0s
kafka_acl.throughput_consumer_topic_acl: Destruction complete after 0s
module.throughput_test_consumer.kafka_acl.producer_acl["pubsub.throughput-test-out"]: Creation complete after 0s [id=User:CN=pubsub/throughput-test-consumer|*|Write|Allow|Topic|pubsub.throughput-test-out|Literal]
module.throughput_test_consumer.kafka_acl.topic_acl["pubsub.throughput-test"]: Creation complete after 0s [id=User:CN=pubsub/throughput-test-consumer|*|Read|Allow|Topic|pubsub.throughput-test|Literal]
module.throughput_test_consumer.kafka_acl.group_acl["pubsub.throughput-test-consumer"]: Creation complete after 0s [id=User:CN=pubsub/throughput-test-consumer|*|Read|Allow|Group|pubsub.throughput-test-consumer|Literal]

Warning: Argument is deprecated

  with provider["registry.terraform.io/mongey/kafka"],
  on __env.tf line 12, in provider "kafka":
  12: provider "kafka" {

This parameter is now deprecated and will be removed in a later release,
please use `client_cert` instead.

(and one more similar warning elsewhere)

Apply complete! Resources: 3 added, 0 changed, 3 destroyed.

To manually trigger plan again please post @terraform-applier plan dev-aws/kafka-shared-msk/pubsub as comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sbuliarca sbuliarca sbuliarca approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@matthewhughes-uw @uw-infra @sbuliarca