How to detect a closed JetStream (due to server restart)?

[barafael]

We are using a WorkQueue consumer with StorageType::Memory and AckPolicy::Explicit. When the server restarts, it forgets the streams which existed before the shutdown. In a consumer, how can we observe that the server has shut down, so we can eventually recreate the stream when the server comes up again?

• We tried relying on the Stream interface, which states that the stream yields None when exhausted, but we do not observe this.
• We can observe the NATS connection state
  • we could lose messages currently queued in the consumer
  • we cannot detect that we have processed all of them, because of the earlier point

We DO observe an error on the stream, around the time of the reconnection:

2024-07-30T15:35:11.045783Z  WARN thing::module::consumer: pull consumer failed. error=pull request failed: request failed: missing field `stream_name` at line 1 column 129

[Jarema]

That looks like a bug. The consumer should recover on its own.
Will take a look.

[Jarema]

@barafael can you please share some snippet how you create and use the consumer?

[barafael]

https://github.com/barafael/njet-stream has a complete minimal example.

In three shells:

Step 1: nats-server -js --user nats --pass nats --addr 127.0.0.1 -p 23561
Step 2: cargo run --bin producer
Step 3: cargo run --bin consumer
Observe messages being delivered.
Step 4: Stop nats server
Observe no more messages
Step 5: Restart nats server
Observe still no messages.

This could of course be the intended behaviour, I don't know. One can work around it by having a dedicated tokio task to recreate the NATS consumer when errors come through the stream and forwarding all messages through a common hole. The normal behaviour of a stream is that it yields a None value on close, but in this case here we can't even rely on that.

[barafael]

I am currently using a dedicated tokio task which reacts to errors on the jetstream by recreating the consumer. Is it possible I am losing messages when the consumer is being reconstructed?

[Jarema]

@barafael If the stream is gone, being in memory stream, after for example server restart, you should not loose messages, as workqueue stream will remove them only after consumer receives ack.

However, I wonder why you are using memory based consumer if this is a persistent consumer?