NullPointerException when tasks.max > 1 and using s3a

[levin81]

Loaded kafka-connect distributed on 2 machines (using confluent kafka connect docker image). Configured it to work with streamx, created a job with tasks.max=1 while using s3a (configured in the hdfs-site.xml) - everything works fine.

Whenever I raise the number of tasks to be anything other than 1 I get the following error:

java.lang.NullPointerException
	at io.confluent.connect.hdfs.DataWriter.close(DataWriter.java:299)
 	at io.confluent.connect.hdfs.HdfsSinkTask.close(HdfsSinkTask.java:110)
 	at org.apache.kafka.connect.runtime.WorkerSinkTask.commitOffsets(WorkerSinkTask.java:302)
 	at org.apache.kafka.connect.runtime.WorkerSinkTask.closePartitions(WorkerSinkTask.java:435)
 	at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:147)
 	at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:140)
 	at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:175)
 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 	at java.lang.Thread.run(Thread.java:745)

I tried messing around with all the configurations, including kafka timeouts, heartbeat, etc. To no avail. After reading this issue: #30 - I tried using s3n instead and it works without exceptions now!

Thanks

[PraveenSeluka]

@levin81 - S3N is stable right now and am testing S3A now. Will have some updates soon for S3A.

[PraveenSeluka]

@levin81 Added a comment to #30 detailing the s3a issues there.