rabbitmq connection closed when the error CommandStatus.ESME_RSUBMITFAIL is encountered

[kelvin-bump]

All outgoing SMS messages are stopped when we observed the ESME response CommandStatus.ESME_RSUBMITFAIL is returned.
using the logs, we see that this is due to an exception in rsubmitfail

Unhandled error in Deferred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/jasmin/managers/dlr.py", line 108, in rejectMessage
yield self.amqpBroker.chan.basic_reject(delivery_tag=message.delivery_tag, requeue=requeue)
File "", line 3, in basic_reject

File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1905, in unwindGenerator
return _cancellableInlineCallbacks(gen)
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1815, in _cancellableInlineCallbacks
_inlineCallbacks(None, gen, status)
--- ---
File "/usr/local/lib/python3.11/site-packages/jasmin/managers/dlr.py", line 108, in rejectMessage
yield self.amqpBroker.chan.basic_reject(delivery_tag=message.delivery_tag, requeue=requeue)
File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1660, in _inlineCallbacks
result = current_context.run(gen.send, result)
File "/usr/local/lib/python3.11/site-packages/txamqp/protocol.py", line 72, in invoke
self._raise_closed(self.reason)
File "/usr/local/lib/python3.11/site-packages/txamqp/protocol.py", line 125, in _raise_closed
raise ConnectionClosed(reason)
txamqp.client.ConnectionClosed: Method(name=close, id=50) (541, 'INTERNAL_ERROR', 0, 0) content = None

[kelvin-bump]

found latest dlr.py version and applying it

[kelvin-bump]

didnt work - seems when amqp connection is closed due to error, the consumers are not reading from the queues even after Jasmin tries to reconnect. the error keeps repeating at least once a day and only solution is to restart jasmin.

2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> supervisor: {<0.607.0>,rabbit_channel_sup}
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> errorContext: shutdown
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> reason: reached_max_restart_intensity
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> offender: [{pid,<0.610.0>},
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> {id,channel},
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> {mfargs,
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> {rabbit_channel,start_link,
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> [1,<0.603.0>,<0.608.0>,<0.603.0>,
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> <<"192.168.80.7:46112 -> 192.168.80.3:5672">>,
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> rabbit_framing_amqp_0_9_1,
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> {user,<<"guest">>,
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> [administrator],
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> [{rabbit_auth_backend_internal,
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> #Fun<rabbit_auth_backend_internal.2.43373861>}]},
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> <<"/">>,[],<0.604.0>,<0.609.0>]}},
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> {restart_type,intrinsic},
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> {shutdown,70000},
2024-10-11 16:11:24.207005+00:00 [error] <0.607.0> {child_type,worker}]
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> Error on AMQP connection <0.603.0> (192.168.80.7:46112 -> 192.168.80.3:5672, vhost: '/', user: 'guest', state: running), channel 1:
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> {function_clause,
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> [{rabbit_binary_parser,parse_table,
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> [<<108,6,100,108,114,95,105,100,83,0,0,0,36,98,97,48,102,57,49,48,
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> 100,45,52,101,100,50,45,52,51,50,49,45,56,102,49,56,45,101,50,50,
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> 55,100,56,98,53,53,51,98,57,8,100,108,114,95,115,116,97,116,83,0,
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> 0,0,7,85,78,68,69,76,73,86>>],
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> [{file,"rabbit_binary_parser.erl"},{line,41}]},
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> {rabbit_binary_parser,parse_table,1,
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> [{file,"rabbit_binary_parser.erl"},{line,43}]},
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> {rabbit_framing_amqp_0_9_1,decode_properties,2,
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> [{file,"rabbit_framing_amqp_0_9_1.erl"},{line,924}]},
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> {rabbit_binary_parser,ensure_content_decoded,1,
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> [{file,"rabbit_binary_parser.erl"},{line,146}]},
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> {rabbit_channel,handle_method,3,
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> [{file,"rabbit_channel.erl"},{line,1318}]},
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> {rabbit_channel,handle_cast,2,[{file,"rabbit_channel.erl"},{line,647}]},
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1067}]},
2024-10-11 16:11:24.208226+00:00 [error] <0.603.0> {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,250}]}]}
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> Non-AMQP exit reason '{function_clause,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> [{rabbit_binary_parser,parse_table,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> [<<108,6,100,108,114,95,105,100,83,0,0,0,36,98,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> 97,48,102,57,49,48,100,45,52,101,100,50,45,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> 52,51,50,49,45,56,102,49,56,45,101,50,50,55,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> 100,56,98,53,53,51,98,57,8,100,108,114,95,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> 115,116,97,116,83,0,0,0,7,85,78,68,69,76,73,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> 86>>],
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> [{file,"rabbit_binary_parser.erl"},{line,41}]},
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> {rabbit_binary_parser,parse_table,1,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> [{file,"rabbit_binary_parser.erl"},{line,43}]},
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> {rabbit_framing_amqp_0_9_1,decode_properties,2,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> [{file,"rabbit_framing_amqp_0_9_1.erl"},
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> {line,924}]},
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> {rabbit_binary_parser,ensure_content_decoded,1,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> [{file,"rabbit_binary_parser.erl"},{line,146}]},
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> {rabbit_channel,handle_method,3,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> [{file,"rabbit_channel.erl"},{line,1318}]},
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> {rabbit_channel,handle_cast,2,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> [{file,"rabbit_channel.erl"},{line,647}]},
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> {gen_server2,handle_msg,2,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> [{file,"gen_server2.erl"},{line,1067}]},
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> {proc_lib,wake_up,3,
2024-10-11 16:11:24.208908+00:00 [warning] <0.603.0> [{file,"proc_lib.erl"},{line,250}]}]}'
2024-10-11 16:11:24.213576+00:00 [info] <0.603.0> closing AMQP connection <0.603.0> (192.168.80.7:46112 -> 192.168.80.3:5672, vhost: '/', user: 'guest')
2024-10-11 16:11:34.256638+00:00 [info] <0.2407.0> accepting AMQP connection <0.2407.0> (192.168.80.7:42818 -> 192.168.80.3:5672)
2024-10-11 16:11:34.259987+00:00 [info] <0.2407.0> connection <0.2407.0> (192.168.80.7:42818 -> 192.168.80.3:5672): user 'guest' authenticated and granted access to vhost '/'