value of time-key chopped off after every retry in AMQP 0.9.1 x-death header

[Alex-The-DevOp-Guy]

RabbitMQ-Info

Version: 3.13.4
Feature-Flags:

[[email protected] :~/]$ rabbitmqctl list_feature_flags
Listing feature flags ...
name	state
classic_mirrored_queue_version	enabled
classic_queue_type_delivery_support	enabled
detailed_queues_endpoint	enabled
direct_exchange_routing_v2	enabled
drop_unroutable_metric	enabled
empty_basic_get_metric	enabled
feature_flags_v2	enabled
implicit_default_bindings	enabled
khepri_db	disabled
listener_records_in_ets	enabled
maintenance_mode_status	enabled
message_containers	enabled
message_containers_deaths_v2	enabled
quorum_queue	enabled
quorum_queue_non_voters	enabled
restart_streams	enabled
stream_filtering	enabled
stream_queue	enabled
stream_sac_coordinator_unblock_group	enabled
stream_single_active_consumer	enabled
stream_update_config_command	enabled
tracking_records_in_ets	enabled
user_limits	enabled
virtual_host_metadata	enabled

Describe the bug

After every requeuing/retry of our Dead-Lettered Messages the time-value is chopped off, until it is 0.

Reproduction steps

1. Shovel a dead lettered message to an active queue.
2. When it comes back, the timestamp from the time-value is chopped off, resulting in an invalid timestamp

Expected behavior

The time value should behave like it is stated in the documentation: "When this message was dead lettered the first time from this queue for this reason." https://www.rabbitmq.com/docs/dlx

Additional context

__TypeId__:	our.type
x-death:	
count:	1
exchange:	our.exchange
queue:	our.queue
reason:	rejected
routing-keys:	
time:	1731593401

__TypeId__:	our.type
x-death:	
count:	2
exchange:	our.exchange
queue:	our.queue
reason:	rejected
routing-keys:	
time:	1731579

__TypeId__:	our.type
x-death:	
count:	3
exchange:	our.exchange
queue:	our.queue
reason:	rejected
routing-keys:	
time:	1731

__TypeId__:	our.type
x-death:	
count:	4
exchange:	our.exchange
queue:	our.queue
reason:	rejected
routing-keys:	
time:	1

__TypeId__:	our.type
x-death:	
count:	5
exchange:	our.exchange
queue:	our.queue
reason:	rejected
routing-keys:	
time:	0
__TypeId__:	our.type
x-death:	
count:	6
exchange:	our.exchange
queue:	our.queue
reason:	rejected
routing-keys:	
time:	0

As you can see, after every retry the time value is chopped off by 3 digits, until it is 0.
After the first retry, we still have a valid timestamp: "1731593401"

➜  ~ date -r 1731593401
Do 14 Nov 2024 15:10:01 CET

But after the second try the timestamp doesn't make any sense.

➜  ~ date -r 1731579
Mi 21 Jan 1970 01:59:39 CET

[ansd]

@Alex-The-DevOp-Guy please include the RabbitMQ version and list or enabled feature flags where you observe this issue.

[zicklam]

Checkout this pull request / issue, seems to be like this:

• 3.13.x only: fix timestamp when recovering x-death header #11608

[ansd]

This implies that @Alex-The-DevOp-Guy uses either 3.13.3 or 3.13.4, correct?

[Alex-The-DevOp-Guy]

That's correct, I am using 3.13.4

[ansd]

Okay, so the solution is to upgrade RabbitMQ to >= 3.13.5.

[Alex-The-DevOp-Guy]

Great, thanks for the support @ansd and @zicklam !

[michaelklishin]

And since 3.13.x is out of community support, the solution is to upgrade to 4.0.3.