javax.net.ssl.SSLException: Tag mismatch! error trying to connect via kafka client

[itaydj]

Hi Guys,
I am testing Strimzi for my company and I stumbled upon a minor issue.
I started a tls enabled cluster but when I try to connect to it I get the following error:
javax.net.ssl.SSLException: Tag mismatch!

I am using the latest strimzi operator version (0.24.0)
Steps to reproduce:

1. execute the following yaml to create a new kafka cluster:

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: my-cluster
spec:
kafka:
version: 2.8.0
replicas: 3
listeners:
- name: tls
type: nodeport
port: 9094
tls: true
authentication:
type: tls
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
log.message.format.version: "2.8"
inter.broker.protocol.version: "2.8"
storage:
type: persistent-claim
size: 10Gi
deleteClaim: false
zookeeper:
replicas: 3
storage:
type: persistent-claim
size: 10Gi
deleteClaim: false
entityOperator:
topicOperator: {}
userOperator: {}

2. extract the ca.crt using the following command:
   kubectl get secrets my-cluster-cluster-ca-cert -o jsonpath='{.data.ca\.crt}' | base64 -d > ca.crt

3.import the ca.crt to a jks:
keytool -import -alias kafka-ca -file ca.crt -storetype JKS -keystore kafka.jks

4. create a properties file containg all connection info (lets call it mycluster.properties):
   security.protocol=SSL ssl.truststore.password=123456 ssl.truststore.location=/home/itaym/projects/strimzi/kafka.jks ssl.endpoint.identification.algorithm=

5. connect to your cluster:
   ./kafka-topics.sh --list --bootstrap-server <worker_ip>:<nodeport_port> --command-config mycluster.properties

expected result:
I should get a list of all my topics

actual result:
I get the following error

2021-08-01 15:49:26,206] ERROR [AdminClient clientId=adminclient-1] Connection to node -1 (/172.25.10.52:30119) failed authentication due to: Failed to process post-handshake messages (org.apache.kafka.clients.NetworkClient) [2021-08-01 15:49:26,208] WARN [AdminClient clientId=adminclient-1] Metadata update failed due to authentication error (org.apache.kafka.clients.admin.internals.AdminMetadataManager) org.apache.kafka.common.errors.SslAuthenticationException: Failed to process post-handshake messages Caused by: javax.net.ssl.SSLException: Tag mismatch! at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292) at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:122) at java.base/sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:681) at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:636) at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:454) at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:433) at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) at org.apache.kafka.common.network.SslTransportLayer.read(SslTransportLayer.java:567) at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:95) at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:447) at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:397) at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) at org.apache.kafka.common.network.Selector.poll(Selector.java:481) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1329) at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1260) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: javax.crypto.AEADBadTagException: Tag mismatch! at java.base/com.sun.crypto.provider.GaloisCounterMode.decryptFinal(GaloisCounterMode.java:623) at java.base/com.sun.crypto.provider.CipherCore.finalNoPadding(CipherCore.java:1116) at java.base/com.sun.crypto.provider.CipherCore.fillOutputBuffer(CipherCore.java:1053) at java.base/com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:941) at java.base/com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:491) at java.base/javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:779) at java.base/javax.crypto.CipherSpi.engineDoFinal(CipherSpi.java:730) at java.base/javax.crypto.Cipher.doFinal(Cipher.java:2497) at java.base/sun.security.ssl.SSLCipher$T13GcmReadCipherGenerator$GcmReadCipher.decrypt(SSLCipher.java:1903) at java.base/sun.security.ssl.SSLEngineInputRecord.decodeInputRecord(SSLEngineInputRecord.java:240) at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:197) at java.base/sun.security.ssl.SSLEngineInputRecord.decode(SSLEngineInputRecord.java:160) at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:110) ... 16 more

Any guidance is highly appreciated as I am struggling with this one for a few days now
Thanks

[scholzj]

If you format the examples properly, it might be much more readable. But AFAIK you have enabled TLS client authentication, but you do not configure any keystore for it in the client.

[itaydj]

Hi Jakub,
as always, thanks alot for a super fast response.
sorry for the yaml formatting, github messed it up and I didnt check before sending it.
ill check your input and update.

[itaydj]

Indeed - it was a keystore issue.
thanks alot Jakub.

[austindev4]

Hi
I am facing the same error message, so it is great to see there is someone who has figured this out already.
Apologies but can you please elaborate on the missing step, how did you configure keystore in the client?

I am following https://developers.redhat.com/blog/2019/06/12/accessing-apache-kafka-in-strimzi-part-5-ingress#kubernetes_ingress

My test command is:-
/bin/kafka-console-producer --broker-list bootstrap.com:443 --producer-property security.protocol=SSL --producer-property ssl.truststore.password=password --producer-property ssl.truststore.location=/etc/security/keytabs/truststore.jks --topic test-topic

[austindev4]

Thanks @itaydj
I am placing the ca.crt in truststore and keystore. (First line after "Detailed steps to generate client truststore and keystore" above)

I now tried with kafka client 2.0.0 (confluentinc/cp-kafka:5.0.4) and got message:-
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2

(Previously was trying with kafkaclient 3.0.0 - confluentinc/cp-kafka:7.0.0 - when I got the Tag mismatch error)

Full output:-
root@producer-2:/# kafka-console-producer --broker-list bootstrap.com:443
--producer-property security.protocol=SSL
--producer-property ssl.truststore.password=password
--producer-property ssl.truststore.location=kafka.client.truststore.jks
--producer-property ssl.keystore.location=kafka.client.keystore.jks
--producer-property ssl.keystore.password=password
--producer-property ssl.key.password=password
--topic test-topic

[2021-11-22 08:58:48,171] ERROR [Producer clientId=console-producer] Connection to node -1 failed authentication due to: SSL handshake failed (org.apache.kafka.clients.Netw
orkClient)
[2021-11-22 08:58:48,171] ERROR Error when sending message to topic test-topic with key: null, value: 0 bytes with error: (org.apache.kafka.clients.producer.internals.Error
LoggingCallback)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1538)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:474)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:337)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:264)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:134)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:500)
at org.apache.kafka.common.network.Selector.poll(Selector.java:438)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:524)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:244)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:168)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLProtocolException: Handshake message sequence violation, 2
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:196)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:992)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:989)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:399)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:479)
... 9 more

[austindev4]

@itaydj Whats in your keystore? If you dont mind can you please share the steps you performed for generating keystore?

[scholzj]

@austindev4 I don't really get this part:

Generate a Client Certificate and sign it using the CA.
$ keytool -keystore kafka.client.keystore.jks -alias test_client -certreq -file client-cert-file
$ openssl x509 -req -CA /etc/security/keytabs/ca.crt -CAkey /etc/security/keytabs/ca.key -in client-cert-file -out client-cert-signed -days 365 -CAcreateserial

Import CA Certificate into the Client keystore.
$ keytool -keystore kafka.client.keystore.jks -alias CARoot -import -file /etc/security/keytabs/ca.crt
$ keytool -keystore kafka.client.keystore.jks -alias test_client -import -file client-cert-signed

You cannot authenticate with a random certificate you just generated. The keystore needs to contain a valid user certificate. You can use the User Operator and the KafkaUser resource to have it generated by Strimzi. Or you can tyke your custom CA and provide it as a custom Clients CA. But if you did what you just described, you just generated a random certificate and expect to be authenticated. That beats the purpose of the authentication.

[austindev4]

I am signing the "client-cert-file" here
$ openssl x509 -req -CA /etc/security/keytabs/ca.crt -CAkey /etc/security/keytabs/ca.key -in client-cert-file -out client-cert-signed -days 365 -CAcreateserial

Ok will try user certificate from User Operator

[austindev4]

Using user certificate from user operator for simplicity, PKCS12 format keystore and truststore do work with kafka3.0 producer client.

JKS format does not work with same kafka3.0 producer client. Error below:-
[2021-11-24 01:35:46,437] ERROR Error when sending message to topic test-topic with key: null, value: 0 bytes with error: (org.apache.k
afka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target
at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:349)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:292)
at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:287)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1357)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1232)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1175)
at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:551)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderExcep
tion: unable to find valid certification path to requested target
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
at java.base/sun.security.validator.Validator.validate(Validator.java:264)
at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335)
... 19 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
... 25 more

Secondly, did not work with Nifi consumeKafka client due to error similar to seglo/kafka-lag-exporter#270.

That led me to isolating the problem to java -version.
Commands:-
##keystore.jks
keytool -importkeystore -srckeystore user.p12
-destkeystore kafkadevjks.client.keystore.jks
-srcstoretype PKCS12 -deststoretype jks
-srcstorepass ${user.password}
-deststorepass password
-srcalias test-client -destalias test-client
-srckeypass ${user.password} -destkeypass password

#truststore.jks
keytool -keystore kafkadevjks.client.truststore.jks
-alias CARoot -deststoretype jks -import -file ca.crt
-deststorepass password -noprompt

Concluding:
If your knowledge of certificates and crypto are limited, and you are having certificate issues in your application, best build the keystore and truststore with keytool located on application host (assuming pre-existing jdk).
Again if having trouble with pkcs12/jks formats build on application host and test with both.

Hope that helps someone!
Thanks @itaydj and @scholzj for your assistance.