What do "Error replicating transaction" errors mean on replicas?

[tutiplain]

Hi all,

I am running ImmuDB in a network and using two different machines as replicas (the two machines are part of the same network).

I am starting the master database with the following CLI command:

/immudb --web-server-port=8081 --dir /data/immu_data

The replicas are being started with this command:

/immudb --web-server-port=8081 --replication-enabled --replication-follower-username=immudb --replication-follower-password=immudb --port=3323 --pgsql-server=false --replication-master- --dir /data/immu_data

This works fine. What I'd like to do is have one of my replicas start up as the new master database, and then have a new replica come up on yet another machine on the network. I already have a project that handles this and succesfully boots the old replica as the new master and creates a new replica elsewhere. The folder where the data is stored is never deleted, so there is a chance that a previous master will become a master again, but have an outdated data folder. And there is a chance that a replica might be created on a machine that already had data in it. When this happens, the replica process starting displaying this message: "Failed to replicate transaction XXX. Reason: illegal arguments". If a replica showing this message becomes the leader again, its data will be incomplete. I looked at the immudb code and, apparently the "illegal arguments" message means there is something invalid in the command line arguments. However, the command line I am using to start the replica works for the initial replica (and I have validated that the new master's address is correct). My question is: why do these "Failed to replicate tx" messages mean and how can I keep fix them? If the previous folder is deleted, the replica runs fine, but I figure the replica creation will take longer as the data size increases over time, so I'd like for the replica to use the existing folder and just... catch up to the most recent transaction. Maybe I'm misunderstanding how the whole replication thing works. Any help or info on this would be much appreciated.

[jeroiraz]

Hi all,

I am running ImmuDB in a network and using two different machines as replicas (the two machines are part of the same network).

I am starting the master database with the following CLI command:

/immudb --web-server-port=8081 --dir /data/immu_data

The replicas are being started with this command:

/immudb --web-server-port=8081 --replication-enabled --replication-follower-username=immudb --replication-follower-password=immudb --port=3323 --pgsql-server=false --replication-master- --dir /data/immu_data

This works fine. What I'd like to do is have one of my replicas start up as the new master database, and then have a new replica come up on yet another machine on the network. I already have a project that handles this and succesfully boots the old replica as the new master and creates a new replica elsewhere. The folder where the data is stored is never deleted, so there is a chance that a previous master will become a master again, but have an outdated data folder. And there is a chance that a replica might be created on a machine that already had data in it. When this happens, the replica process starting displaying this message: "Failed to replicate transaction XXX. Reason: illegal arguments". If a replica showing this message becomes the leader again, its data will be incomplete. I looked at the immudb code and, apparently the "illegal arguments" message means there is something invalid in the command line arguments. However, the command line I am using to start the replica works for the initial replica (and I have validated that the new master's address is correct). My question is: why do these "Failed to replicate tx" messages mean and how can I keep fix them? If the previous folder is deleted, the replica runs fine, but I figure the replica creation will take longer as the data size increases over time, so I'd like for the replica to use the existing folder and just... catch up to the most recent transaction. Maybe I'm misunderstanding how the whole replication thing works. Any help or info on this would be much appreciated.

Hi @tutiplain, I think you have a good understanding on how replication works. Although, there are some details which my lead to the situation you are describing "Failed to replicate transaction XXX..." when the replica started with an existent data folder.

When immudb starts, it creates two databases systemdb and defaultdb, some data is also inserted e.g. admin user. This means, that if you are starting another instance in replica mode, it won't create those databases as usual but will fetch initialized data from the master.
The error you are describing could happen when the data folder were you are running the replica was not created by fetching data from the master, e.g. data folder was initially created running as master.

Switching from master and replica back and forth should work fine as long the history of transaction is kept consistent. One instance ahead of others, but txs shouldn't diverge, this situation could happen when instances set as master commit transactions and then replication is attempted.

[tutiplain]

Thanks for the quick reply. What you said makes perfect sense and lines up with what I've been seeing. But how can I go about keeping the tx history consistent if just being a replica causes a different history (because the replica does not have the transactions to create the system and defaultdbs) to exist? Maybe running a replica inside the same machine as the current master and having the two remote replicas replicate from that one instead of directly from the master? Is there any alternative approach you can recommend?

[jeroiraz]

Thanks for the quick reply. What you said makes perfect sense and lines up with what I've been seeing. But how can I go about keeping the tx history consistent if just being a replica causes a different history (because the replica does not have the transactions to create the system and defaultdbs) to exist? Maybe running a replica inside the same machine as the current master and having the two remote replicas replicate from that one instead of directly from the master? Is there any alternative approach you can recommend?

to ensure history is kept consistent in replicas, you should always run immudb in replica mode e.g. ./immudb --replication-enabled=true, to ensure systemdb and defauldb won't be initialized in the replica but fetched from master. This will cover the data of existent data folder already in the replica; and at most once immudb server should be ran in master mode.

Note: this is intended to replicate systemdb and defaultdb, to enable replication of user created databases, immudb is ran in master mode, but each database settings will configure the database to be in master or replica mode.

[tutiplain]

I already run my replicas with the --replication-enabled flag. Are you saying the master need this flag, too?

[jeroiraz]

I already run my replicas with the --replication-enabled flag. Are you saying the master need this flag, too?

not in master, but to ensure to include it always in the replicas, because otherwise it will initialize its own data folder and then replication will fail with the error you are getting.

do you have a sequence of steps starting with fresh replicas that result in the replica not being to follow master anymore?

[tutiplain]

Not really, just what I mentioned above. When my master process dies, I stop one of the replicas and just start it again as the master, using the data it has replicated at the moment. I then spawn a new replica elsewhere.