BUG: Unable to delete some topics in MSK serverless

[gpad]

Issue submitter TODO list

• I've looked up my issue in FAQ
• I've searched for an already existing issues here
• I've tried running master-labeled docker image and the issue still persists there
• I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

Some topics (automatically created by connectors) are impossible to delete because it's not possible to select them.

It's possible to delete manually via CLI:

bin/kafka-topics.sh \
    --bootstrap-server "boot-blahblah.kafka-serverless.eu-west-1.amazonaws.com:9098" \
    --command-config /etc/kafkaui/dynamic_config.yaml \
    --delete \
    --topic __amazon_msk_connect_status_xyzycy_ead17ca5-edbd-4d55-b356-ce70e84e0fe4-3

Expected behavior

Select the connectors and delete them

Your installation details

56fa824 v0.7.1
Intalled by aws marketplace

auth:
type: "LOGIN_FORM"
spring:
security:
user:
name: name
password: password
kafka:
clusters:

• bootstrapServers: boot-blabla.kafka-serverless.eu-west-1.amazonaws.com:9098
  name: demo-2
  properties:
  security.protocol: SASL_SSL
  sasl.mechanism: AWS_MSK_IAM
  sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
  sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler
  readOnly: false
  rrrbac:
  roles: []
  webclient: {}

Steps to reproduce

Create a MSK cluster
Create connector
Destroy a connector
you will find some topics like __amazon_msk_connect these topics are not selectable

Screenshots

Logs

No response

Additional context

No response

[github-actions]

Hello there gpad! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀

[adamshugar]

I'm having the same issue with MSK provisioned (non-serverless). My hunch is that Kafka UI automatically labels any topic starting with __ as internal and prevents you from deleting it in the UI for "safety" (you can also delete __consumer_offsets using the CLI, but you usually probably shouldn't). So it might not be a bug so much as an expected but suboptimal behavior.

In this case though, it's necessary to delete hanging topics storing metadata about deleted MSK connectors. MSK doesn't auto delete topics for deleted connectors, and MSK requires you to make a brand new connector every time you want to apply a config changes to the connector, which means these dead topics accumulate fast. This can quickly lead to not being able to apply config changes to the entire cluster at all see here, which just happened to me.

[rekhasuresh16]

@adamshugar Is there a way to clean up these topics manually by specifying some connector configs? We tried using offset.storage.topic mentioned here But, issue is still the same.

[Haarolean]

These topics can't be selected due to their names starting with an underscore, having one is considered them being internal topics in kafka. This can be adjusted by changing the kafka_internalTopicPrefix configuration property to something different from an underscore.

Please note that this repo is not maintained (#4255), will be happy to asnwer any other questions here: https://github.com/kafbat/kafka-ui