Fix: slow failover #146
Fix: slow failover #146
Conversation
678098
force-pushed
the
231107_slow_failover
branch
from
41c473c to
181c61d
Compare
| @@ -286,7 +286,7 @@ void RelayQueueEngine::onHandleConfiguredDispatched( | |||
| << downStreamParameters << ", but assuming success."; | |||
| } | |||
|
|
|||
| BALL_LOG_INFO_BLOCK | |||
| BALL_LOG_TRACE_BLOCK | |||
| { | |||
| mqbcmd::RoundRobinRouter outrr(d_allocator_p); | |||
| context->d_routing_sp->loadInternals(&outrr); | |||
There was a problem hiding this comment.
This loadInternals might be called for each consumer rapidly and might degrade overall performance when we have thousands of consumers and still continue to add new.
There was a problem hiding this comment.
Could we make this a debug level instead?
There was a problem hiding this comment.
How about BALL_LOGTHROTTLE_INFO_BLOCK?
There was a problem hiding this comment.
@hallfox changed trace->debug
There was a problem hiding this comment.
@hallfox will use throttle info block instead, so we have something in logs without changing default severity
| static const int k_MAX_INSTANT_MESSAGES = 10; | ||
| static const bsls::Types::Int64 k_NS_PER_MESSAGE = | ||
| bdlt::TimeUnitRatio::k_NANOSECONDS_PER_SECOND; | ||
| BALL_LOGTHROTTLE_INFO(k_MAX_INSTANT_MESSAGES, k_NS_PER_MESSAGE) |
There was a problem hiding this comment.
Decided to use in-place directive from ball and remove member throttle d_throttledSubscriptionInfo to simplify class.
This throttle is thread-safe: https://bloomberg.github.io/bde-resources/doxygen/bde_api_prod/group__ball__logthrottle.html#3.2
There was a problem hiding this comment.
Side effect: 2 places with throttle previously had a common throttle, and now they are independent. But in my opinion this doesn't matter in this case.
| @@ -286,7 +286,7 @@ void RelayQueueEngine::onHandleConfiguredDispatched( | |||
| << downStreamParameters << ", but assuming success."; | |||
| } | |||
|
|
|||
| BALL_LOG_INFO_BLOCK | |||
| BALL_LOG_TRACE_BLOCK | |||
| { | |||
| mqbcmd::RoundRobinRouter outrr(d_allocator_p); | |||
| context->d_routing_sp->loadInternals(&outrr); | |||
There was a problem hiding this comment.
How about BALL_LOGTHROTTLE_INFO_BLOCK?
| << ", downstreamSubQueueId = " << downstreamSubId | ||
| << ", upstreamId = " << upstreamId << "]"; | ||
| } | ||
| static const int k_MAX_INSTANT_MESSAGES = 10; |
There was a problem hiding this comment.
Maybe, move k_MAX_INSTANT_MESSAGES and k_NS_PER_MESSAGE out in a single place? They are the same for all occurrences most likely.
| // Maximum messages logged with throttling in a short period of time. | ||
|
|
||
| const bsls::Types::Int64 k_NS_PER_MESSAGE = | ||
| bdlt::TimeUnitRatio::k_NANOSECONDS_PER_MINUTE / k_MAX_INSTANT_MESSAGES; |
There was a problem hiding this comment.
Allow to log at most 10 records in a minute. Should be enough for us.
Previously each cycle of adding a new consumer consumed ~40 ms in total:
04NOV2023_14:09:16.926 (140284186806016) INFO mqbblp_relayqueueengine.cpp:824 For queue '...', rebuilt highest priority consumers: [count: 1598]
04NOV2023_14:09:16.885 (140284186806016) INFO mqbblp_relayqueueengine.cpp:824 For queue '...', rebuilt highest priority consumers: [count: 1597]
04NOV2023_14:09:16.842 (140284186806016) INFO mqbblp_relayqueueengine.cpp:824 For queue '...', rebuilt highest priority consumers: [count: 1596]
04NOV2023_14:09:16.802 (140284186806016) INFO mqbblp_relayqueueengine.cpp:824 For queue '...', rebuilt highest priority consumers: [count: 1595]
04NOV2023_14:09:16.761 (140284186806016) INFO mqbblp_relayqueueengine.cpp:824 For queue '...', rebuilt highest priority consumers: [count: 1594]
Signed-off-by: Evgeny Malygin <[email protected]>
Signed-off-by: Evgeny Malygin <[email protected]>
Signed-off-by: Evgeny Malygin <[email protected]>
Signed-off-by: Evgeny Malygin <[email protected]>
678098
force-pushed
the
231107_slow_failover
branch
from
3170366 to
1f4768f
Compare
No description provided.