raft: snapshot can not be installed while catchups running

[1995parham]

We are getting this error:

raft: snapshot can not be installed while catchups running

In the time that we have this error, we don't receive any messages.

[derekcollison]

I believe that is a warning, but it should clear itself as the catchup resolves. It just means that snapshots, which are being run periodically but the upper layers can not actually happen if the raft layer is trying to catchup.

When did you see the warning, at startup or after the server had been running?
Did the warning repeat or just show once?

[1995parham]

The server is running and the time of incidents we have many of this warning. Can it be related to our cpu or disk capacity?

[derekcollison]

So the server has been running for some time yes?
Was there an event that triggered it?

What do the following report?

nats server report jetstream <- from sys account
nats account info from the user account.

[1995parham]

╭──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                      JetStream Summary                                                       │
├───────────────────────────┬──────────────┬─────────┬───────────┬────────────┬─────────┬────────┬─────────┬─────────┬─────────┤
│ Server                    │ Cluster      │ Streams │ Consumers │ Messages   │ Bytes   │ Memory │ File    │ API Req │ API Err │
├───────────────────────────┼──────────────┼─────────┼───────────┼────────────┼─────────┼────────┼─────────┼─────────┼─────────┤
│ teh1js-nats-production-1  │ teh1-okd4-js │ 2       │ 39        │ 10,168,925 │ 2.8 GiB │ 0 B    │ 2.8 GiB │ 683     │ 1       │
│ teh1js-nats-production-2* │ teh1-okd4-js │ 5       │ 39        │ 23,731,643 │ 5.3 GiB │ 0 B    │ 5.3 GiB │ 274     │ 0       │
│ teh1js-nats-production-0  │ teh1-okd4-js │ 1       │ 33        │ 10,143,706 │ 2.8 GiB │ 0 B    │ 2.8 GiB │ 845     │ 48      │
├───────────────────────────┼──────────────┼─────────┼───────────┼────────────┼─────────┼────────┼─────────┼─────────┼─────────┤
│                           │              │ 8       │ 111       │ 44,044,274 │ 11 GiB  │ 0 B    │ 11 GiB  │ 1,802   │ 49      │
╰───────────────────────────┴──────────────┴─────────┴───────────┴────────────┴─────────┴────────┴─────────┴─────────┴─────────╯

╭─────────────────────────────────────────────────────────────────────╮
│                     RAFT Meta Group Information                     │
├──────────────────────────┬────────┬─────────┬────────┬────────┬─────┤
│ Name                     │ Leader │ Current │ Online │ Active │ Lag │
├──────────────────────────┼────────┼─────────┼────────┼────────┼─────┤
│ teh1js-nats-production-0 │        │ true    │ true   │ 0.14s  │ 0   │
│ teh1js-nats-production-1 │        │ true    │ true   │ 0.14s  │ 0   │
│ teh1js-nats-production-2 │ yes    │ true    │ true   │ 0.00s  │ 0   │
╰──────────────────────────┴────────┴─────────┴────────┴────────┴─────╯

[derekcollison]

What server version?

[1995parham]

We are on 2.8.0

[derekcollison]

If you could please update the server to the current version which is 2.8.4.

[1995parham]

Sure, I will update it and let you know if it happens again.

[derekcollison]

Thanks.