Failing kafka when inference?

[easyfan327]

When try to inference a trained model, there is no response after post the queries.
logs from two predictor and inference workers as follows for reference:

2019-12-05 04:05:01,568 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:05:01,568 kafka.conn INFO Probing node bootstrap-0 broker version
2019-12-05 04:05:01,569 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:05:01,617 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:05:01,617 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:05:01,617 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:05:01,671 kafka.conn INFO Broker version identifed as 1.0.0
2019-12-05 04:05:01,671 kafka.conn INFO Set configuration api_version=(1, 0, 0) to skip auto check_version requests on startup
2019-12-05 04:05:01,671 kafka.consumer.subscription_state INFO Updating subscribed topics to: ('workers_b9287bc81988_bd250bae-a7d2-4362-ad91-76fa11981883_prediction',)
2019-12-05 04:05:01,672 kafka.cluster INFO Group coordinator for PREDICTIONS is BrokerMetadata(nodeId=1001, host='rafiki_kafka', port=9092, rack=None)
2019-12-05 04:05:01,672 kafka.coordinator INFO Discovered coordinator 1001 for group PREDICTIONS
2019-12-05 04:05:01,672 kafka.coordinator INFO Starting new heartbeat thread
2019-12-05 04:05:01,672 kafka.coordinator.consumer INFO Revoking previously assigned partitions set() for group PREDICTIONS
2019-12-05 04:05:01,673 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:05:01,773 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:05:01,773 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:05:01,874 kafka.coordinator INFO (Re-)joining group PREDICTIONS
2019-12-05 04:05:04,074 kafka.coordinator WARNING Heartbeat failed for group PREDICTIONS because it is rebalancing
2019-12-05 04:05:04,078 kafka.coordinator WARNING Heartbeat failed for group PREDICTIONS because it is rebalancing
2019-12-05 04:05:04,080 kafka.coordinator.consumer INFO Revoking previously assigned partitions {TopicPartition(topic='workers_b9287bc81988_0c83f3f4-22eb-4351-a639-83f74670baa0_prediction', partition=0)} for group PREDICTIONS
2019-12-05 04:05:04,082 kafka.coordinator INFO (Re-)joining group PREDICTIONS
2019-12-05 04:05:04,084 kafka.coordinator.consumer INFO Revoking previously assigned partitions {TopicPartition(topic='workers_b9287bc81988_37f92ee1-8015-4f1e-b04f-a139871edc6e_prediction', partition=0)} for group PREDICTIONS
2019-12-05 04:05:04,085 kafka.coordinator INFO (Re-)joining group PREDICTIONS
2019-12-05 04:05:04,091 kafka.coordinator INFO Elected group leader -- performing partition assignments using range
2019-12-05 04:05:04,097 kafka.coordinator.assignors.range WARNING No partition metadata for topic workers_b9287bc81988_bd250bae-a7d2-4362-ad91-76fa11981883_prediction
2019-12-05 04:05:04,101 kafka.coordinator INFO Successfully joined group PREDICTIONS with generation 20
2019-12-05 04:05:04,103 kafka.coordinator INFO Successfully joined group PREDICTIONS with generation 20
2019-12-05 04:05:04,104 kafka.coordinator INFO Successfully joined group PREDICTIONS with generation 20
2019-12-05 04:05:04,105 kafka.consumer.subscription_state INFO Updated partition assignment: [TopicPartition(topic='workers_b9287bc81988_0c83f3f4-22eb-4351-a639-83f74670baa0_prediction', partition=0)]
2019-12-05 04:05:04,106 kafka.consumer.subscription_state INFO Updated partition assignment: []
2019-12-05 04:05:04,106 kafka.consumer.subscription_state INFO Updated partition assignment: [TopicPartition(topic='workers_b9287bc81988_37f92ee1-8015-4f1e-b04f-a139871edc6e_prediction', partition=0)]
2019-12-05 04:05:04,107 kafka.coordinator.consumer INFO Setting newly assigned partitions {TopicPartition(topic='workers_b9287bc81988_0c83f3f4-22eb-4351-a639-83f74670baa0_prediction', partition=0)} for group PREDICTIONS
2019-12-05 04:05:04,107 kafka.coordinator.consumer INFO Setting newly assigned partitions set() for group PREDICTIONS
2019-12-05 04:05:04,108 kafka.coordinator.consumer INFO Setting newly assigned partitions {TopicPartition(topic='workers_b9287bc81988_37f92ee1-8015-4f1e-b04f-a139871edc6e_prediction', partition=0)} for group PREDICTIONS
2019-12-05 04:05:04,109 kafka.coordinator.consumer INFO Revoking previously assigned partitions {TopicPartition(topic='workers_b9287bc81988_37f92ee1-8015-4f1e-b04f-a139871edc6e_prediction', partition=0)} for group PREDICTIONS
2019-12-05 04:05:04,110 kafka.coordinator INFO (Re-)joining group PREDICTIONS
2019-12-05 04:05:07,122 kafka.coordinator WARNING Heartbeat failed for group PREDICTIONS because it is rebalancing
2019-12-05 04:05:07,122 kafka.coordinator.consumer INFO Revoking previously assigned partitions set() for group PREDICTIONS
2019-12-05 04:05:07,123 kafka.coordinator INFO (Re-)joining group PREDICTIONS
2019-12-05 04:05:07,180 kafka.coordinator WARNING Heartbeat failed for group PREDICTIONS because it is rebalancing
2019-12-05 04:05:07,184 kafka.coordinator.consumer INFO Revoking previously assigned partitions {TopicPartition(topic='workers_b9287bc81988_0c83f3f4-22eb-4351-a639-83f74670baa0_prediction', partition=0)} for group PREDICTIONS
2019-12-05 04:05:07,184 kafka.coordinator INFO (Re-)joining group PREDICTIONS
2019-12-05 04:05:07,189 kafka.coordinator INFO Elected group leader -- performing partition assignments using range
2019-12-05 04:05:07,197 kafka.coordinator INFO Successfully joined group PREDICTIONS with generation 21
2019-12-05 04:05:07,199 kafka.coordinator INFO Successfully joined group PREDICTIONS with generation 21
2019-12-05 04:05:07,199 kafka.consumer.subscription_state INFO Updated partition assignment: [TopicPartition(topic='workers_b9287bc81988_0c83f3f4-22eb-4351-a639-83f74670baa0_prediction', partition=0)]
2019-12-05 04:05:07,200 kafka.consumer.subscription_state INFO Updated partition assignment: [TopicPartition(topic='workers_b9287bc81988_37f92ee1-8015-4f1e-b04f-a139871edc6e_prediction', partition=0)]
2019-12-05 04:05:07,202 kafka.coordinator.consumer INFO Setting newly assigned partitions {TopicPartition(topic='workers_b9287bc81988_0c83f3f4-22eb-4351-a639-83f74670baa0_prediction', partition=0)} for group PREDICTIONS
2019-12-05 04:05:07,202 kafka.coordinator INFO Successfully joined group PREDICTIONS with generation 21
2019-12-05 04:05:07,203 kafka.coordinator.consumer INFO Setting newly assigned partitions {TopicPartition(topic='workers_b9287bc81988_37f92ee1-8015-4f1e-b04f-a139871edc6e_prediction', partition=0)} for group PREDICTIONS
2019-12-05 04:05:07,205 kafka.consumer.subscription_state INFO Updated partition assignment: [TopicPartition(topic='workers_b9287bc81988_bd250bae-a7d2-4362-ad91-76fa11981883_prediction', partition=0)]
2019-12-05 04:05:07,208 kafka.coordinator.consumer INFO Setting newly assigned partitions {TopicPartition(topic='workers_b9287bc81988_bd250bae-a7d2-4362-ad91-76fa11981883_prediction', partition=0)} for group PREDICTIONS

---

2019-12-05 04:01:03,404 kafka.conn INFO Broker version identifed as 1.0.0
2019-12-05 04:01:03,404 kafka.conn INFO Set configuration api_version=(1, 0, 0) to skip auto check_version requests on startup
2019-12-05 04:01:03,405 kafka.consumer.subscription_state INFO Updating subscribed topics to: ('workers_b9287bc81988_queries',)
2019-12-05 04:01:03,406 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:01:03,507 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:01:03,507 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:01:03,608 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:01:03,709 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:01:03,709 kafka.conn INFO Probing node bootstrap-0 broker version
2019-12-05 04:01:03,710 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:01:03,812 kafka.conn INFO Broker version identifed as 1.0.0
2019-12-05 04:01:03,812 kafka.conn INFO Set configuration api_version=(1, 0, 0) to skip auto check_version requests on startup
2019-12-05 04:01:03,812 kafka.consumer.subscription_state INFO Updating subscribed topics to: ('workers_b9287bc81988_queries',)
2019-12-05 04:01:03,813 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:01:03,914 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:01:03,914 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:01:04,015 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:01:04,116 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:01:04,117 kafka.conn INFO Probing node bootstrap-0 broker version
2019-12-05 04:01:04,117 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:01:04,219 kafka.conn INFO Broker version identifed as 1.0.0
2019-12-05 04:01:04,219 kafka.conn INFO Set configuration api_version=(1, 0, 0) to skip auto check_version requests on startup
2019-12-05 04:01:04,220 kafka.consumer.subscription_state INFO Updating subscribed topics to: ('workers_b9287bc81988_queries',)
2019-12-05 04:01:04,221 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:01:04,321 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:01:04,321 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:01:04,424 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:01:04,529 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:01:04,530 kafka.conn INFO Probing node bootstrap-0 broker version
2019-12-05 04:01:04,531 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:01:04,641 kafka.conn INFO Broker version identifed as 1.0.0
2019-12-05 04:01:04,641 kafka.conn INFO Set configuration api_version=(1, 0, 0) to skip auto check_version requests on startup
2019-12-05 04:01:04,644 kafka.consumer.subscription_state INFO Updating subscribed topics to: ('workers_b9287bc81988_queries',)
2019-12-05 04:01:04,649 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:01:04,751 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:01:04,752 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:01:04,855 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:01:04,960 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:01:04,961 kafka.conn INFO Probing node bootstrap-0 broker version
2019-12-05 04:01:04,962 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:01:05,077 kafka.conn INFO Broker version identifed as 1.0.0
2019-12-05 04:01:05,077 kafka.conn INFO Set configuration api_version=(1, 0, 0) to skip auto check_version requests on startup
2019-12-05 04:01:05,081 kafka.consumer.subscription_state INFO Updating subscribed topics to: ('workers_b9287bc81988_queries',)
2019-12-05 04:01:05,087 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:01:05,188 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:01:05,189 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:01:05,293 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:01:05,401 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:01:05,401 kafka.conn INFO Probing node bootstrap-0 broker version
2019-12-05 04:01:05,402 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:01:05,512 kafka.conn INFO Broker version identifed as 1.0.0
2019-12-05 04:01:05,513 kafka.conn INFO Set configuration api_version=(1, 0, 0) to skip auto check_version requests on startup
2019-12-05 04:01:05,515 kafka.consumer.subscription_state INFO Updating subscribed topics to: ('workers_b9287bc81988_queries',)
2019-12-05 04:01:05,520 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: connecting to rafiki_kafka:9092 [('10.0.0.233', 9092) IPv4]
2019-12-05 04:01:05,621 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Connection complete.
2019-12-05 04:01:05,622 kafka.conn INFO <BrokerConnection node_id=bootstrap-0 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.
2019-12-05 04:01:05,725 kafka.conn INFO <BrokerConnection node_id=1001 host=rafiki_kafka:9092 [IPv4 ('10.0.0.233', 9092)]>: Closing connection.