kafka in kraft mode update from 0.40-3.7.0 to 0.43-3.8.0 fails to update metadata version #10672

Tchirana · 2024-10-02T19:43:41Z

Tchirana
Oct 2, 2024

Hey hello,
I update the operator to 0.43.0 with sucess and after that updated the version in the kafka manifest to 3.8.0. I have 2 clusters. One got updated perfectly:

  Resource Version:    1857375050
    Version:     3.8.0
  Kafka Metadata Version:  3.8-IV0
  Kafka Version:  3.8.0
  Operator Last Successful Version:  0.43.0```

but the other one is having issues on the metadata part:

```API Version:  kafka.strimzi.io/v1beta2
  Resource Version:    1857413602
    Version:     3.8.0
    Message:               Failed to update metadata version to 3.8
  Kafka Metadata Version:  3.7-IV4
  Kafka Version:  3.8.0
  Operator Last Successful Version:  0.43.0```

logs in operator pod:

``` INFO  KRaftMetadataManager:165 - Reconciliation #203(timer) Kafka(kafka/xxx): Updating metadata version from 3.7-IV4 to 3.8-IV0
WARN  KRaftMetadataManager:124 - Reconciliation #203(timer) Kafka(kafka/xxx): Failed to update metadata version to 3.8 (the current version is 3.7-IV4)
org.apache.kafka.common.errors.InvalidUpdateVersionException: Invalid update version 20 for feature metadata.version. Controller 5 only supports versions 1-19```

How to I upgrade the controller? it seems that operator failed to do so , but why.... I have no metadataVersion defined in the kafkas resource so operator should set it with success like it did for the other cluster....

scholzj · 2024-10-02T19:50:16Z

scholzj
Oct 2, 2024
Maintainer

It looks like you scaled dow some nodes before the upgrade. Unfortunately, Kafka does not support getting the registered nodes and unregister them right now. So you will have to unregister the node manually: https://kafka.apache.org/38/javadoc/org/apache/kafka/clients/admin/Admin.html#unregisterBroker(int)

12 replies

Tchirana Oct 2, 2024
Author

same:

INFO KafkaReconciler:963 - Reconciliation #328(watch) Kafka(: Kafka nodes [5] were removed from the Kafka cluster and will be unregistered ERROR KafkaAdminClient:4527 - [AdminClient clientId=adminclient-455] Unregister broker request for broker ID 5 failed: The given broker ID was not registered. WARN KafkaNodeUnregistration:93 - Reconciliation #328(watch) Kafka: Node 5 is not registered and cannot be unregistered from the Kafka cluster org.apache.kafka.common.errors.BrokerIdNotRegisteredException: The given broker ID was not registered. INFO KafkaReconciler:969 - Reconciliation #328(watch) Kafka: Kafka nodes [5] were successfully unregistered from the Kafka cluster INFO KRaftMetadataManager:165 - Reconciliation #328(watch) Kafka: Updating metadata version from 3.7-IV4 to 3.8-IV0

so 5 is not the id of that controller, it must be a ghost somewhere left from a unfortunate event in the past.

Tchirana Oct 3, 2024
Author

@scholzj I think I can wait until a feature is added to the tools to be able to do this without too much hassle. I am no developer so accessing that api programmatically using java is tricky for me. I see this in the cluster by using the :
bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --all --describe

All configs for broker 0 are: ........... inter.broker.protocol.version=3.8-IV0 sensitive=false synonyms={DEFAULT_CONFIG:inter.broker.protocol.version=3.8-IV0} .......... All configs for broker 1 are: ........... inter.broker.protocol.version=3.8-IV0 sensitive=false synonyms={DEFAULT_CONFIG:inter.broker.protocol.version=3.8-IV0} ........... All configs for broker 2 are: ...... inter.broker.protocol.version=3.8-IV0 sensitive=false synonyms={DEFAULT_CONFIG:inter.broker.protocol.version=3.8-IV0} ......
I really cannot find a way with the cli tools to get the "ghost" controller 5 to show up.
The only inconvenient now, as I see it, are those messages in the operator pod complaining it cannot migrate the metadata of controller 5 from 3.7..... to 3.8...

Tchirana Oct 3, 2024
Author

I may have found something looking in the metadata with:
bin/kafka-metadata-shell.sh --snapshot /var/lib/kafka/data-0/kafka-log1/__cluster_metadata-0/00000000000027285355-0000000873.checkpoint
[ Kafka Metadata Shell ]

tree image/cluster/
brokers:
0:
BrokerRegistration(id=0, epoch=27159043, incarnationId=xxx, listeners=[Endpoint(listenerName='PLAIN-9092', securityProtocol=PLAINTEXT, host='xxx', port=9092), Endpoint(listenerName='REPLICATION-9091', securityProtocol=SSL, host='xxx', port=9091), Endpoint(listenerName='TLS-9093', securityProtocol=SSL, host='xxx', port=9093)], supportedFeatures={metadata.version: 1-20}, rack=Optional.empty, fenced=false, inControlledShutdown=false, isMigratingZkBroker=false, directories=[xxx])
1:
BrokerRegistration(id=1, epoch=27163116, incarnationId=n-UlUXmaSCiXYMlbRn1Bmg, listeners=[Endpoint(listenerName='PLAIN-9092', securityProtocol=PLAINTEXT, host='xxx', port=9092), Endpoint(listenerName='REPLICATION-9091', securityProtocol=SSL, host='xxxc', port=9091), Endpoint(listenerName='TLS-9093', securityProtocol=SSL, host='xxx', port=9093)], supportedFeatures={metadata.version: 1-20}, rack=Optional.empty, fenced=false, inControlledShutdown=false, isMigratingZkBroker=false, directories=[VgeH_WzLojA4iKwXbT0dXA])
2:
BrokerRegistration(id=2, epoch=27167126, incarnationId=8EeQrpFhSWGx8U_n45thMg, listeners=[Endpoint(listenerName='PLAIN-9092', securityProtocol=PLAINTEXT, host='xcxx', port=9092), Endpoint(listenerName='REPLICATION-9091', securityProtocol=SSL, host='xxx', port=9091), Endpoint(listenerName='TLS-9093', securityProtocol=SSL, host='xxx', port=9093)], supportedFeatures={metadata.version: 1-20}, rack=Optional.empty, fenced=false, inControlledShutdown=false, isMigratingZkBroker=false, directories=[Rj9wvYRRXZn_UldEJ_E44Q])
controllers:
0:
ControllerRegistration(id=0, incarnationId=EspvilnsRF-v0C0HOssF4Q, zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLPLANE-9090', securityProtocol=SSL, host='0.0.0.0', port=9090)], supportedFeatures={metadata.version: 1-20})
1:
ControllerRegistration(id=1, incarnationId=PqE0HhSVQsC1XZaebE267A, zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLPLANE-9090', securityProtocol=SSL, host='0.0.0.0', port=9090)], supportedFeatures={metadata.version: 1-20})
2:
ControllerRegistration(id=2, incarnationId=NP6XrGvAS2KkC2_z08uGsA, zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLPLANE-9090', securityProtocol=SSL, host='0.0.0.0', port=9090)], supportedFeatures={metadata.version: 1-20})
3:
ControllerRegistration(id=3, incarnationId=3MSyohZcQreoJhVxl0zI-g, zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLPLANE-9090', securityProtocol=SSL, host='0.0.0.0', port=9090)], supportedFeatures={metadata.version: 1-19})
4:
ControllerRegistration(id=4, incarnationId=MMaIM82cTx2pOlxbUiBl2g, zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLPLANE-9090', securityProtocol=SSL, host='0.0.0.0', port=9090)], supportedFeatures={metadata.version: 1-19})
5:
ControllerRegistration(id=5, incarnationId=n9WaMxZ1T3KaRXdXscUSBg, zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLPLANE-9090', securityProtocol=SSL, host='0.0.0.0', port=9090)], supportedFeatures={metadata.version: 1-19})

scholzj Oct 3, 2024
Maintainer

Please keep in mind that the KIP will not be implemented at least until Kafka 4.0.0 as the KIP is not approved right now. That said, I do not know if any of the CLI tools can do it and can understand the reluctance to write code for it.

Tchirana Oct 3, 2024
Author

At least I found the culprits :)

Tchirana · 2024-10-03T16:20:14Z

Tchirana
Oct 3, 2024
Author

@scholzj Are you sure unregisterBroker will work?
because I look at this :

tree image/cluster/
brokers:
0:
BrokerRegistration(id=0, epoch=22029753, incarnationId=4qaSbje1Tbm_MmXVUicNsA, listeners=[Endpoint(listenerName='PLAIN-9092', securityProtocol=PLAINTEXT, host='xxx', port=9092), Endpoint(listenerName='REPLICATION-9091', securityProtocol=SSL, host='xxx', port=9091), Endpoint(listenerName='TLS-9093', securityProtocol=SSL, host='xxx', port=9093)], supportedFeatures={metadata.version: 1-19}, rack=Optional.empty, fenced=false, inControlledShutdown=false, isMigratingZkBroker=false, directories=[f2KpcX5jysMHLB8hpQPVOw])
1:
BrokerRegistration(id=1, epoch=22026028, incarnationId=av4to9BaRK6HEPfNdsEdgw, listeners=[Endpoint(listenerName='PLAIN-9092', securityProtocol=PLAINTEXT, host='xxx', port=9092), Endpoint(listenerName='REPLICATION-9091', securityProtocol=SSL, host='sxxx, port=9091), Endpoint(listenerName='TLS-9093', securityProtocol=SSL, host='xxxrokers.kafka.svc', port=9093)], supportedFeatures={metadata.version: 1-19}, rack=Optional.empty, fenced=false, inControlledShutdown=false, isMigratingZkBroker=false, directories=[9mwZbiWyVyKbo3HMZxkR9w])
2:
BrokerRegistration(id=2, epoch=22033493, incarnationId=O5mlGHrfQI2YPGXfj5SHeg, listeners=[Endpoint(listenerName='PLAIN-9092', securityProtocol=PLAINTEXT, host='xxxkafka-brokers.kafka.svc', port=9092), Endpoint(listenerName='REPLICATION-9091', securityProtocol=SSL, host='xxxkafka-brokers.kafka.svc', port=9091), Endpoint(listenerName='TLS-9093', securityProtocol=SSL, host='xxxkafka-brokers.kafka.svc', port=9093)], supportedFeatures={metadata.version: 1-19}, rack=Optional.empty, fenced=false, inControlledShutdown=false, isMigratingZkBroker=false, directories=[zpRXW2IVE8SOuTvRkvlZfw])
controllers:
0:
ControllerRegistration(id=0, incarnationId=_sTw-4DTSZaYrQYv34w66A, zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLPLANE-9090', securityProtocol=SSL, host='0.0.0.0', port=9090)], supportedFeatures={metadata.version: 1-19})
1:
ControllerRegistration(id=1, incarnationId=eZnGiSleRz-CvWqaGgqvBw, zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLPLANE-9090', securityProtocol=SSL, host='0.0.0.0', port=9090)], supportedFeatures={metadata.version: 1-19})
2:
ControllerRegistration(id=2, incarnationId=wTzHZWDoS-OsAjQGSsGlMw, zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLPLANE-9090', securityProtocol=SSL, host='0.0.0.0', port=9090)], supportedFeatures={metadata.version: 1-19})
3:
ControllerRegistration(id=3, incarnationId=P4qp6sZ-R5Gc27QcAJXPqg, zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLPLANE-9090', securityProtocol=SSL, host='0.0.0.0', port=9090)], supportedFeatures={metadata.version: 1-19})
4:
ControllerRegistration(id=4, incarnationId=QjW07J89RrusdWK78LxedQ, zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLPLANE-9090', securityProtocol=SSL, host='0.0.0.0', port=9090)], supportedFeatures={metadata.version: 1-19})
5:
ControllerRegistration(id=5, incarnationId=0SGNpoJ4QImPi7I3WzHx2Q, zkMigrationReady=false, listeners=[Endpoint(listenerName='CONTROLPLANE-9090', securityProtocol=SSL, host='0.0.0.0', port=9090)], supportedFeatures={metadata.version: 1-19})

Is there a method to alter the metadata in other ways? or a unregisterController ?

2 replies

scholzj Oct 3, 2024
Maintainer

I have no idea what it is you are listing here.

Tchirana Oct 3, 2024
Author

I am listing a
bin/kafka-metadata-shell.sh --snapshot /var/lib/kafka/data-0/kafka-log0/__cluster_metadata-0/00000000000027292391-0000000873.checkpoint
and after doing a
tree image/cluster/
inside the shell

Tchirana · 2024-10-03T19:35:13Z

Tchirana
Oct 3, 2024
Author

@scholzj I managed to do a trick. Because I am using kraft , i annotated the kafka nodepool with next-node-ids 3,4,5 and scaled the pool to 6 and it created 3 more brokers and 3 more controllers. This last creation , the controllers, made the "phantom" controllers reappear and also the errors from the cluster operator dissapeared. Now to remove those and come back to 3 nodes (broker+controller) I just have to scale down the nodepool? Or first should I use cruise control to remove the broker? Is there a special procedure to unregister a controller?

7 replies

scholzj Oct 3, 2024
Maintainer

Well, if you would have asked first I would have told you to use broker-only nodes because Kafka does not support scaling controllers right now. But as you already have it, you should try to scale it down.

Tchirana Oct 3, 2024
Author

hmmmm:
java.net.UnknownHostException: kafka-3.xxx-kafka-brokers.kafka.svc: Name or service not known
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method) ~[?:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at java.net.InetAddress$PlatformNameService.lookupAllHostAddr(InetAddress.java:934) ~[?:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1543) ~[?:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at java.net.InetAddress$NameServiceAddresses.get(InetAddress.java:852) ~[?:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at java.net.InetAddress.getAllByName0(InetAddress.java:1533) ~[?:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at java.net.InetAddress.getAllByName(InetAddress.java:1385) ~[?:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at java.net.InetAddress.getAllByName(InetAddress.java:1306) ~[?:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at org.apache.kafka.clients.DefaultHostResolver.resolve(DefaultHostResolver.java:27) ~[org.apache.kafka.kafka-clients-3.8.0.jar:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at org.apache.kafka.clients.ClientUtils.resolve(ClientUtils.java:124) ~[org.apache.kafka.kafka-clients-3.8.0.jar:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.resolveAddresses(ClusterConnectionStates.java:536) ~[org.apache.kafka.kafka-clients-3.8.0.jar:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.currentAddress(ClusterConnectionStates.java:511) ~[org.apache.kafka.kafka-clients-3.8.0.jar:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at org.apache.kafka.clients.ClusterConnectionStates$NodeConnectionState.access$200(ClusterConnectionStates.java:466) ~[org.apache.kafka.kafka-clients-3.8.0.jar:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at org.apache.kafka.clients.ClusterConnectionStates.currentAddress(ClusterConnectionStates.java:173) ~[org.apache.kafka.kafka-clients-3.8.0.jar:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1070) ~[org.apache.kafka.kafka-clients-3.8.0.jar:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:320) ~[org.apache.kafka.kafka-clients-3.8.0.jar:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1252) ~[org.apache.kafka.kafka-clients-3.8.0.jar:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1512) ~[org.apache.kafka.kafka-clients-3.8.0.jar:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1455) ~[org.apache.kafka.kafka-clients-3.8.0.jar:?]
strimzi-cluster-operator-67c44f9978-kw49q strimzi-cluster-operator at java.lang.Thread.run(Thread.java:840) ~[?:?]

scholzj Oct 3, 2024
Maintainer

Not sure why it is trying connect to it if you scaled it down.

Tchirana Oct 3, 2024
Author

ended up scaling back up not to get again phantom controllers, did a forced leader election, nodes were recreated corectly again by the cluster operator.
ended up with 1+1 on each node.
it is rather uncomfortable that controllers cannot be removed....

Tchirana Oct 3, 2024
Author

cluster is now healthy and all metadata is at 3.8-IV0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strimzi

kafka in kraft mode update from 0.40-3.7.0 to 0.43-3.8.0 fails to update metadata version #10672

{{title}}

Replies: 3 comments 21 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Strimzi

kafka in kraft mode update from 0.40-3.7.0 to 0.43-3.8.0 fails to update metadata version #10672

Tchirana Oct 2, 2024

Replies: 3 comments · 21 replies

scholzj Oct 2, 2024 Maintainer

Tchirana Oct 2, 2024 Author

Tchirana Oct 3, 2024 Author

Tchirana Oct 3, 2024 Author

scholzj Oct 3, 2024 Maintainer

Tchirana Oct 3, 2024 Author

Tchirana Oct 3, 2024 Author

scholzj Oct 3, 2024 Maintainer

Tchirana Oct 3, 2024 Author

Tchirana Oct 3, 2024 Author

scholzj Oct 3, 2024 Maintainer

Tchirana Oct 3, 2024 Author

scholzj Oct 3, 2024 Maintainer

Tchirana Oct 3, 2024 Author

Tchirana Oct 3, 2024 Author

Tchirana
Oct 2, 2024

Replies: 3 comments 21 replies

scholzj
Oct 2, 2024
Maintainer

Tchirana Oct 2, 2024
Author

Tchirana Oct 3, 2024
Author

Tchirana Oct 3, 2024
Author

scholzj Oct 3, 2024
Maintainer

Tchirana Oct 3, 2024
Author

Tchirana
Oct 3, 2024
Author

scholzj Oct 3, 2024
Maintainer

Tchirana Oct 3, 2024
Author

Tchirana
Oct 3, 2024
Author

scholzj Oct 3, 2024
Maintainer

Tchirana Oct 3, 2024
Author

scholzj Oct 3, 2024
Maintainer

Tchirana Oct 3, 2024
Author

Tchirana Oct 3, 2024
Author