-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adjust docker-compose single example #710
base: master
Are you sure you want to change the base?
Conversation
Using KAFKA_ADVERTISED_HOST_NAME is now deprecated https://kafka.apache.org/0110/documentation.html#brokerconfigs
@@ -9,7 +9,8 @@ services: | |||
ports: | |||
- "9092:9092" | |||
environment: | |||
KAFKA_ADVERTISED_HOST_NAME: 192.168.99.100 | |||
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 | |||
KAFKA_LISTENERS: PLAINTEXT://:9092 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By default, it'll only bind internally to the compose network
KAFKA_LISTENERS: PLAINTEXT://:9092 | |
KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am no Kafka expert, but this change should not be necessary as exposure to the outside world via localhost:9092
is assured by KAFKA_ADVERTISED_LISTENERS
on the above line, see docs.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This variable sets the bind address of the Java server. The advertised listeners are what's stored in Zookeeper and what are returned to clients for future connections. They are very different settings
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also worth mentioning that setting the advertised listener to localhost will not allow you to add other Kafka client services to the compose network, as they will then try to connect to themselves rather than the broker container. Perhaps you should add two listeners
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Have to admit I don't understand what scenario the proposed change is trying to solve...
- you have kafka available outside of compose to host machine thanks to ports mapping
9092:9092
andKAFKA_ADVERTISED_LISTENERS
atlocalhost:9092
- it is available within compose as well thanks to
KAFKA_LISTENERS
atkafka:9092
.
All this PR is trying to achieve is to replace deprecated KAFKA_ADVERTISED_HOST_NAME
, no further improvements in mind, feel free to open your own PR if you feel a need to have e.g. multiple listeners in this example, but there are no "other kafka client services in the compose network" to worry about. Trying to keep the example minimalistic.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it is available within compose as well thanks to KAFKA_LISTENERS at kafka:9092.
The bootstrap is. The actual client protocol is not.
Try this
$ docker-compose up -d
$ docker network ls # get the network
$ docker run --rm -ti --entrypoint=bash --network=<name> wurstmeister/kafka
root@41cab124273d:/# nc -vz kafka 9092
Connection to kafka (172.19.0.2) 9092 port [tcp/*] succeeded!
root@41cab124273d:/# hostname -f
41cab124273d
root@41cab124273d:/# /opt/kafka/bin/kafka-topics.sh --list --bootstrap-server kafka:9092
[2022-05-13 17:04:58,904] WARN [AdminClient clientId=adminclient-1] Connection to node 1001 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
Like I said, it'll try to connect to itself. That would be fine from outside the container, but not in the Compose bridge network.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The bootstrap is. The actual client protocol is not.
but the proposed change (KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092
) is not solving it, is it? It is likely my limited Kafka knowledge, but what s the addition of 0.0.0.0 good for here?
The steps you provided in you last post demonstrate the issue with localhost
, not why the 0.0.0.0
should be added, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You're correct that I'm proposing a different solution, I was simply referring to your previous comment that said the advertised listeners would work.
You've set this listeners value as the default, and so it'll bind to the container's canonical hostname. Since Compose allows containers over the bridge network to address hosts using service names, then it'll continue to work, but also, as linked setting to 0.0.0.0
"binds to all interfaces", which is generally more appropriate for allowing external connections.
Using KAFKA_ADVERTISED_HOST_NAME is now deprecated: https://kafka.apache.org/0110/documentation.html#brokerconfigs