Use custom Kafka docker image

[ThePoipoi]

Hi

Long time no see.
I'm opening this thread for a simple question: how can we use custom Kafka docker image ?
I've read the documentation, saying that we can specify the image to use, but it's not working as expected: both zookeeper and kafka pods are running, but the entity-operator one doesn't work.
We have this requirement, cause we want to use our own Kafka image, but deploy it with Strimzi (to avoid more deployment dev'). To test this scenario, I have used your make build to generate docker images (everything went fine), and try to use one of the Kafka (2.8.0) image, renamed on my own.

My Kafka configuration:

Describe on the cluster-entity-operator that failing (CrashLoopBackOff):

Moreover, I have a weird behavior: sometime, the quay.io/strimzi/kafka image is downloaded by something, but not always. I remove it between each deployment, and it come back sometimes.

If I don't specify the image for zookeeper, topicOperator or userOperator, the Strimzi image is always downloaded during deployment. But when I do, it's random...

So, what I'm doing wrong ? Do you have any idea on why the cluster-entity-operator keep failing even if it's (in fact) your own Kafka docker image, just renamed with a custom name and specified by the "image" attribute ?
Thanks a lot :)

[scholzj]

The User and Topic operators are not using the Kafka images but the Operator images. So you are IMHO configuring it right, just to a bad value => you should use something like antoine/operator:0.25.0 for example.

Since you use node port listener, there will be also an init container being used => that might be one of the things pulling the Strimzi image. You should be able to configure it using the brokerRackInitImage option: https://strimzi.io/docs/operators/latest/full/using.html#property-kafka-brokerRackInitImage-reference ... again, this should be the operator and not the Kafka image.

BTW: Maybe you know it - but just in case you don't. You can configure the images like this in the Kafka custom resource. But you can also configure them centrally in the Cluster Operator deployment and its environment variables: https://github.com/strimzi/strimzi-kafka-operator/blob/main/install/cluster-operator/060-Deployment-strimzi-cluster-operator.yaml#L50-L91 ... this is normally easier unless you want every Kafka cluster to use different images.

[ThePoipoi]

Ok perfect, I'll try it and let you know.
Thanks again 👍

[ThePoipoi]

Hi Jakub.

Two different things today :

1. I don't succeed to use brokerRackInitImage. I don't know why, but every time I set it (to my custom image), Kafka pods refuse to start

Does it mean that I've failed the Kafka Image build ? It's weird, cause without this parameter, everything works fine.

2. finally, it's the entity-operator itself that always download and use the quay.io/strimzi/kafka image

So whatever I do, I'll always have 2 kafka images.
Am I doing something wrong ?

[scholzj]

Re 1) As I said before - the brokerRackInitImage is the operator image and not the Kafka image. So you are basically setting it to a wrong image and that is why it doesn't start.

Re 2) That is probably the TLS sidecar image? https://strimzi.io/docs/operators/latest/full/using.html#type-TlsSidecar-reference

[ThePoipoi]

I'm really sorry, I really need to clean my glasses...so, ok for point 1).
But as explained, we want to deliver a Kafka docker image without delivering the Operator's one, cause it can be already available and used on the customer environment. So we will never promote an image like "antoine/operator:X.X.X"
I guess there is no need to use brokerRackInitImage in this case, cause for me it's only the entity-operator that download and use the Strimzi image (I've checked it multiple times: the image is only downloaded when the entity-operator start, not before, nor by kafka or zookeeper images).

And obviously, thanks for your answer :)

[scholzj]

But as explained, we want to deliver a Kafka docker image without delivering the Operator's one, cause it can be already available and used on the customer environment. So we will never promote an image like "antoine/operator:X.X.X"

Ah, ok. Well, in that case you should not set the brokerRackInitImage and let it use the original operator image from Strimzi. There is no way to use the Kafka image for it. But the TLS sidecars use the Kafka image -> the one in Entity Operator I pointed out and if you use Cruise Control, there is another TLS sidecar.