Sink connector is stucked without any log messages if Zeebe is unavailable

[Dabz]

Hi team!

While testing locally, it seems that the connector gets stuck if we provide invalid value to connect to Zeebe. If I deploy a connector locally, without having Zeebe running, I can observe that the thread is blocked indefinitely on https://github.com/zeebe-io/kafka-connect-zeebe/blob/master/src/main/java/io/zeebe/kafka/connect/sink/ZeebeSinkTask.java#L58.

My connector configuration is:

{
  "name": "zeebe",
  "config": {
    "connector.class": "io.zeebe.kafka.connect.ZeebeSinkConnector",
    "tasks.max": "1",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.storage.StringConverter",
    "topics": "kafka-to-zeebe",
    "zeebe.client.broker.contactPoint": "127.0.0.1:26500",
    "zeebe.client.security.plaintext": true,
    "message.path.messageName": "$.eventType",
    "message.path.correlationKey": "$.orderId",
    "message.path.variables": "$.['amount', 'orderId']"
  }
}

If I enable DEBUG log, I can only see:

ESC[36mconnect            |ESC[0m [2020-05-05 15:29:36,527] DEBUG Evaluating path: $['eventType'] (com.jayway.jsonpath.internal.path.CompiledPath)
ESC[36mconnect            |ESC[0m [2020-05-05 15:29:36,527] DEBUG Evaluating path: $['timeToLive'] (com.jayway.jsonpath.internal.path.CompiledPath)
ESC[36mconnect            |ESC[0m [2020-05-05 15:29:36,527] TRACE No timeToLive found, ignoring (io.zeebe.kafka.connect.sink.message.JsonRecordParser)
ESC[36mconnect            |ESC[0m [2020-05-05 15:29:36,527] DEBUG Evaluating path: $['amount','orderId'] (com.jayway.jsonpath.internal.path.CompiledPath)
ESC[36mconnect            |ESC[0m [2020-05-05 15:29:36,528] DEBUG Publishing message io.zeebe.kafka.connect.sink.message.Message@1573292e (io.zeebe.kafka.connect.sink.ZeebeSinkTask)

While looking at the thread, I can see that the thread of the connector is indeed stucked:

I understand that the root cause is my invalid configuration, but the fact that there is no log at all, even in DEBUG or TRACE, makes it really hard to debug. Having to open VisualVM to understand that there is a typo in the configuration is fun, but time-consuming ;)

[fr-ashikaumagiliya]

I think i am having a similar issue. Then checked Kafka Connect logs, I see following error

2020-10-15 09:03:18,938] INFO [Consumer clientId=connector-consumer-Zeebe_Sink_Connector-0, groupId=connect-Zeebe_Sink_Connector] Subscribed to topic(s): zeebe (org.apache.kafka.clients.consumer.KafkaConsumer)
[2020-10-15 09:03:18,939] INFO ZeebeSinkConnectorConfig values:
message.path.correlationKey = $.correlationKey
message.path.messageName = $.messageName
message.path.timeToLive = timeToLive
message.path.variables = variables
zeebe.client.broker.contactPoint = host.docker.internal:26500
zeebe.client.cloud.clientId = null
zeebe.client.cloud.clientSecret = null
zeebe.client.cloud.clusterId = null
zeebe.client.requestTimeout = 10000
zeebe.client.security.plaintext = true
(io.zeebe.kafka.connect.sink.ZeebeSinkConnectorConfig)
[2020-10-15 09:03:18,940] ERROR WorkerSinkTask{id=Zeebe_Sink_Connector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask)
java.lang.NoClassDefFoundError: Could not initialize class io.grpc.netty.NettyChannelBuilder
at io.zeebe.client.impl.ZeebeClientImpl.buildChannel(ZeebeClientImpl.java:122)
at io.zeebe.client.impl.ZeebeClientImpl.(ZeebeClientImpl.java:78)
at io.zeebe.client.impl.ZeebeClientBuilderImpl.build(ZeebeClientBuilderImpl.java:267)
at io.zeebe.kafka.connect.util.ZeebeClientHelper.buildClient(ZeebeClientHelper.java:62)
at io.zeebe.kafka.connect.sink.ZeebeSinkTask.start(ZeebeSinkTask.java:48)
at org.apache.kafka.connect.runtime.WorkerSinkTask.initializeAndStart(WorkerSinkTask.java:302)
at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:193)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:185)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:235)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
[2020-10-15 09:03:18,943] ERROR WorkerSinkTask{id=Zeebe_Sink_Connector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask)