JetStream stops publishing

[CodeTherapist]

I've developed a compact wrapper library that enhances the functionality of your NATS library. This addition is designed to streamline certain elements within our project's scope, such as the serialization and deserialization, as well as topic subscriptions. All these operations are performed via JetStream.

However, I've encountered an issue where, after a specific duration, the following log messages appear (Publishing):

NATS.Client.Core.NatsSubBase: Debug: End subscription Timeout
NATS.Client.JetStream.NatsJSContext: Debug: No response received, retrying 1/2
...
NATS.Client.Core.Internal.InboxSubBuilder: Warning: Unregistered message inbox received for _INBOX.HjgSHEdizxoWYykDc59WSr.qnJJ66ace9v8EPovchQA2v
...
NATS.Client.Core.NatsSubBase: Debug: End subscription Timeout
NATS.Client.JetStream.NatsJSContext: Debug: No response received, retrying 2/2

Subsequent to this, an exception is raised:

This entire sequence of events recurs once the exception is dismissed.
From my perspective, it appears that the private Inbox subscription has disappeared?
I've also examined the NATS Server logs, but haven't found anything unusual (no error).

The entire application operates as a single process, with publishing and receiving activities handled within a hosted BackgroundService. Moreover, the application maintains a single instance of both NatsConnection and NatsJSContext (registred as Singleton).

Adjusting the RetryAttempts and RetryWaitBetweenAttempts parameters within NatsJSPubOpts for publishing only served to delay the occurrence of the issue.

new NatsJSPubOpts()
{
    RetryAttempts = 10,
    RetryWaitBetweenAttempts = TimeSpan.FromSeconds(10)
}

Environment Info:
nats-server: 2.10.11 (Single Instance (no cluster), running as Docker Container)
NATS.NET: 2.2.3

Is it necessary to utilize two distinct NatsConnection objects due to the shared underlying TCP Socket for both publishing and receiving operations?

Any insights for debugging would be greatly appreciated.

[mtmk]

you can increase RequestTimeout in the connection options. I think by default it's 2 seconds.

[CodeTherapist]

I appreciate your swift response. Regrettably, the solution provided did not rectify the issue; the system's behavior remains unchanged even after adjusting the RequestTimeout to 10 seconds.

NATS.Client.Core.NatsSubBase: Debug: End subscription Timeout
NATS.Client.JetStream.NatsJSContext: Debug: No response received, retrying 9/10
NATS.Client.JetStream.Internal.NatsJSConsume: Debug: Check pending messages 500, 1000
NATS.Client.Core.Internal.InboxSubBuilder: Warning: Unregistered message inbox received for _INBOX.m2EzbJqgEMovGvbmkzyiir.L4Prom59p61b3L8Xj0TRAo
NATS.Client.JetStream.Internal.NatsJSConsume: Debug: Sending pull request for pull-loop(chk-msgs) 500, 0
NATS.Client.Core.Internal.InboxSubBuilder: Warning: Unregistered message inbox received for _INBOX.m2EzbJqgEMovGvbmkzyiir.L4Prom59p61b3L8Xj0TRF1
NATS.Client.JetStream.Internal.NatsJSConsume: Debug: Pull request issued for pull-loop(chk-msgs) 500, 0

[CodeTherapist]

When I configure the system to TimeSpan.FromMinutes(10), the messages disappear. Thanks for your support.

However, I'm encountering a challenge where the publishing thread doesn't receive adequate processing time due to the high volume of incoming messages. Is there a mechanism to manage this load balance, or is it something I need to implement manually?

[mtmk]

js.PublishAsync() sends the message to the server and awaits for an answer within the same call. we observed this to have issues like you described under heavy loads. Have a look at PR #532 for a different approach for publishing. maybe that helps.

[mtmk]

fyi #532 was released about a month ago with v2.3.0 if you want to have a go at JetStream batch publish:

var futures = new NatsJSPublishConcurrentFuture[Batch];

for (var i = 0; i < Batch; i++)
{
    futures[i] = await _js.PublishConcurrentAsync("bench_test2", i);
}

for (var i = 0; i < Batch; i++)
{
    await using var future = futures[i];
    var ack = await future.GetResponseAsync();
    ack.EnsureSuccess();
}

let us know how it goes if you get a chance to try it out.

[mtmk]

Is it necessary to utilize two distinct NatsConnection objects due to the shared underlying TCP Socket for both publishing and receiving operations?

reading your message again, yes that will help. I have seen reports from others with similar errors and we might have a head of line blocking issue when publishers and consumers are on the same connection.