Sorting on any message column in the /queues page errors when there is at least one quorum queue in the list

[kjnilsson]

Describe the bug

HTTP API: /api/queues slow query mode requested - extended sort on ["messages_unacknowledged"]
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>   crasher:
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>     initial call: cowboy_stream_h:request_process/3
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>     pid: <0.29338.0>
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>     registered_name: []
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>     exception error: no case clause matching
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                      {[{messages_unacknowledged,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages_unacknowledged,0}],
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                       [{consumers,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {publishers,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {consumer_capacity,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {consumer_utilisation,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages_ready,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {message_bytes_ready,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {message_bytes_unacknowledged,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {message_bytes,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {message_bytes_persistent,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages_persistent,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages_dlx,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {message_bytes_dlx,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {node,rabbit@GQN79FQYLF},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {delivery_limit,20},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages_ready_returned,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages_ready_normal,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages_ready_high,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {effective_policy_definition,#{}},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {memory,88724},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {online,[rabbit@GQN79FQYLF]},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {members,[rabbit@GQN79FQYLF]},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {open_files,[{rabbit@GQN79FQYLF,0}]},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages_ram,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {message_bytes_ram,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {name,<<"q1">>},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {vhost,<<"/">>},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {durable,true},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {auto_delete,false},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {exclusive,false},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {owner_pid,none},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {arguments,#{<<"x-queue-type">> => <<"quorum">>}},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {pid,{'%2F_q1',rabbit@GQN79FQYLF}},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {type,quorum},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {state,running},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {node,rabbit@GQN79FQYLF},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {members,[rabbit@GQN79FQYLF]},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {leader,rabbit@GQN79FQYLF},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {online,[rabbit@GQN79FQYLF]},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {reductions,110115},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {reductions_details,[{rate,72.4}]},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages_ready,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages_ready_details,[{rate,0.0}]},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages_unacknowledged_details,[{rate,0.0}]},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages,0},
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                        {messages_details,[{rate,0.0}]}]}
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>       in function  rabbit_mgmt_util:pget_bin/3 (rabbit_mgmt_util.erl, line 612)
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>       in call from rabbit_mgmt_util:sort_key/2 (rabbit_mgmt_util.erl, line 596)
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>       in call from rabbit_mgmt_util:'-sort_list/4-lc$^0/1-1-'/2 (rabbit_mgmt_util.erl, line 469)
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>       in call from rabbit_mgmt_util:sort_list/4 (rabbit_mgmt_util.erl, line 469)
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>       in call from rabbit_mgmt_util:sort/2 (rabbit_mgmt_util.erl, line 413)
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>       in call from rabbit_mgmt_util:run_augmentation/2 (rabbit_mgmt_util.erl, line 406)
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>       in call from rabbit_mgmt_util:augment_resources0/6 (rabbit_mgmt_util.erl, line 395)
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>       in call from rabbit_mgmt_util:with_valid_pagination/3 (rabbit_mgmt_util.erl, line 314)
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>     ancestors: [<0.29329.0>,<0.732.0>,<0.723.0>,<0.722.0>,<0.720.0>,
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>                   rabbit_web_dispatch_sup,<0.673.0>]
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>     message_queue_len: 0
2024-09-25 14:10:44.754089+01:00 [error] <0.29338.0>     messages: []

Reproduction steps

add QQ, try to sort on the messages / Ready column.

Expected behavior

...

Additional context

...

[chris-hln]

We are experiencing the same issue

[dcorbacho]

Similar to #11886

[chris-hln]

Similar to #11886

I don't believe this is related because it only happened after we upgraded to 4.0.2 (from 3.13.7)

[dcorbacho]

Similar to #11886

I don't believe this is related because it only happened after we upgraded to 4.0.2 (from 3.13.7)

It doesn't say related, it says similar. In both cases, we generate duplicated stats keys which cause the same crash. The origin of the keys is different, as #11886 uses CMQ and this bug is in the quorum queue code.