Skip to content

Commit

Permalink
fix: Update naming conventions related to replication
Browse files Browse the repository at this point in the history 
Signed-off-by: Bartłomiej Święcki <[email protected]>
  • Loading branch information
Bartłomiej Święcki committed Nov 8, 2022
1 parent 49c8ec8 commit 7c36835
Show file tree
Hide file tree
Showing 4 changed files with 86 additions and 70 deletions.
12 changes: 6 additions & 6 deletions src/master/management/database.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -323,12 +323,12 @@ Replication settings:

| Name | Type | Description |
|------|------|-------------|
| replica | Bool | if true, the database is a replica of another one |
| masterDatabase | String | name of the database to replicate |
| masterAddress | String | hostname of the master immudb instance |
| masterPort | Uint32 | tcp port of the master immudb instance |
| followerUsername | String | username used to connect to the master immudb instance |
| followerPassword | String | password used to connect to the master immudb instance |
| replica | Bool | if true, the database is a replica of another one |
| primaryDatabase | String | name of the database to replicate |
| primaryHost | String | hostname of the primary immudb instance |
| primaryPort | Uint32 | tcp port of the primary immudb instance |
| primaryUsername | String | username used to connect to the primary immudb instance |
| primaryPassword | String | password used to connect to the primary immudb instance |

Additional index options:

Expand Down
43 changes: 29 additions & 14 deletions src/master/production/replication.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@

### Replication strategy

immudb includes support for replication by means of a follower approach. A database can be created or configured either to be a master or a replica of another database.
immudb includes support for replication by means of a follower approach. A database can be created or configured either to be a primary or a replica of another database.

<div class="wrapped-picture">

![replication using grpc clients](/immudb/replication-servers.jpg)

</div>

During replication, master databases have a passive role. The grpc endpoint `ExportTx` is used by replicas to fetch unseen committed transactions from the master.
During replication, primary databases have a passive role. The grpc endpoint `ExportTx` is used by replicas to fetch unseen committed transactions from the primary.

Replicas are read only and any direct write operation will be rejected. Using replicas allow to distribute query loads.

Expand All @@ -28,7 +28,7 @@ Replicas are read only and any direct write operation will be rejected. Using re
<WrappedSection>
### Replication and users

As shown in the diagram above, the replicator fetches committed transaction from the master via grpc calls. Internally, the replicator instantiates an immudb client (using the official golang SDK) and fetches unseen committed transactions from the master. In order to do so, the replicator requires valid user credentials with admin permissions, otherwise the master will reject any request.
As shown in the diagram above, the replicator fetches committed transaction from the primary via grpc calls. Internally, the replicator instantiates an immudb client (using the official golang SDK) and fetches unseen committed transactions from the primary. In order to do so, the replicator requires valid user credentials with admin permissions, otherwise the primary will reject any request.

</WrappedSection>

Expand All @@ -42,59 +42,74 @@ Creating a replica of an existent database using immuadmin is super easy:
$ ./immuadmin login immudb
Password:
logged in
$ ./immuadmin database create --replication-is-replica --replication-follower-username=immudb --replication-follower-password=immudb --replication-master-database=defaultdb replicadb
$ ./immuadmin database create \
--replication-is-replica \
--replication-primary-username=immudb \
--replication-primary-password=immudb \
--replication-primary-database=defaultdb \
replicadb
database 'replicadb' {replica: true} successfully created
```

::: tip
Display all database creation flags with

Display all database creation flags with:

```bash
$ ./immuadmin help database create
```

:::

### Creating a second immudb instance to replicate systemdb and defaultdb behaves similarly

Start immudb with enabled replication:

```bash
$ ./immudb --replication-is-replica --replication-follower-password=immudb --replication-follower-username=immudb --replication-master-address=127.0.0.1
$ ./immudb \
--replication-is-replica \
--replication-primary-username=immudb \
--replication-primary-password=immudb \
--replication-primary-host=127.0.0.1
```

::: tip
Display all replication flags

Display all replication flags:

```bash
$ ./immudb --help
```

:::

</WrappedSection>

<WrappedSection>

### Multiple replicas

It's possible to create multiple replicas of a database. Each replica works independently of the others.

<div class="wrapped-picture">

![multiple replicas of the same master database](/immudb/replication-multiple.jpg)
![multiple replicas of the same primary database](/immudb/replication-multiple.jpg)

</div>

Given the master database acts in passive mode, there are no special steps needed in order to create more replicas. Thus, by repeating the same steps to create the first replica it's possible to configure new ones.
Given the primary database acts in passive mode, there are no special steps needed in order to create more replicas. Thus, by repeating the same steps to create the first replica it's possible to configure new ones.

</WrappedSection>

<WrappedSection>

### Replica of a replica

In case many replicas are needed or the master database is under heavy load, it's possible to delegate the creation of replicas to an existent replica. This way, the master database is not affected by the total number of replicas being created.
In case many replicas are needed or the primary database is under heavy load, it's possible to delegate the creation of replicas to an existent replica. This way, the primary database is not affected by the total number of replicas being created.

<div class="wrapped-picture">

![a replica indirectly following the master](/immudb/replication-chain.jpg)
![a replica indirectly following the primary](/immudb/replication-chain.jpg)

</div>

Expand All @@ -104,19 +119,19 @@ In case many replicas are needed or the master database is under heavy load, it'

### External replicator

By creating a database as a replica but with disabled replication, no replicator is created for the database and an external process could be used to replicate committed transactions from the master. The grpc endpoint `ReplicateTx` may be used to externally replicate a transaction.
By creating a database as a replica but with disabled replication, no replicator is created for the database and an external process could be used to replicate committed transactions from the primary. The grpc endpoint `ReplicateTx` may be used to externally replicate a transaction.

</WrappedSection>

<WrappedSection>

### Heterogeneous settings

Replication is configured per database. Thus, the same immudb server may hold several master and replica databases at the same time.
Replication is configured per database. Thus, the same immudb server may hold several primary and replica databases at the same time.

<div class="wrapped-picture">

<img src="/immudb/replication-server.jpg" width="300" alt="a single immudb server can hold multiple master and replica databases"/>
<img src="/immudb/replication-server.jpg" width="300" alt="a single immudb server can hold multiple primary and replica databases"/>

</div>

Expand Down
81 changes: 41 additions & 40 deletions src/master/production/sync-replication.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,17 +52,18 @@ Synchronous replication is enabled per database. The following flags in the `imm

```bash
Flags:
--replication-allow-tx-discarding allow precommitted transactions to be discarded if the follower diverges from the master
--replication-allow-tx-discarding allow precommitted transactions to be discarded if the replica diverges from the primary
--replication-commit-concurrency uint32 number of concurrent replications (default 10)
--replication-follower-password string set password used for replication
--replication-follower-username string set username used for replication
--replication-is-replica set database as a replica
--replication-master-address string set master address
--replication-master-database string set master database to be replicated
--replication-master-port uint32 set master port
--replication-prefetch-tx-buffer-size uint32 maximum number of prefeched transactions (default 100)
--replication-primary-database string set primary database to be replicated
--replication-primary-host string set primary database host
--replication-primary-password string set password used for replication to connect to the primary database
--replication-primary-port uint32 set primary database port
--replication-primary-username string set username used for replication to connect to the primary database
--replication-sync-acks uint32 set a minimum number of replica acknowledgements required before transactions can be committed
--replication-sync-enabled enable synchronous replication

```
</WrappedSection>
Expand All @@ -84,13 +85,13 @@ Make sure you already have [immudb installed](../running/download.md).
1. Run the primary server:
```bash
$ immudb --dir data_master
$ immudb --dir data_primary
```
2. In a new terminal, start replica server:
```bash
$ immudb --dir data_follower \
$ immudb --dir data_replica \
--port=3324 \
--pgsql-server=false \
--metrics-server=false
Expand All @@ -104,9 +105,9 @@ Make sure you already have [immudb installed](../running/download.md).
$ immuadmin login immudb
```
Create a database `db` that requires 1 confirmation from the synchronous followers to do the commit.
Create a database `db` that requires 1 confirmation from the synchronous replicas to do the commit.
> Note that the number of confirmations needed (`--replication-sync-acks` option) should be set to `ceil(number of followers/2)`
> Note that the number of confirmations needed (`--replication-sync-acks` option) should be set to `ceil(number of replicas/2)`
to achieve majority-based quorum.
```shell
Expand All @@ -130,17 +131,17 @@ Make sure you already have [immudb installed](../running/download.md).
```shell
$ immuadmin database create replicadb -p 3324 \
--replication-is-replica \
--replication-master-address 127.0.0.1 \
--replication-master-database primarydb \
--replication-follower-username immudb \
--replication-follower-password immudb \
--replication-master-port 3322 \
--replication-primary-host 127.0.0.1 \
--replication-primary-port 3322 \
--replication-primary-database primarydb \
--replication-primary-username immudb \
--replication-primary-password immudb \
--replication-sync-enabled \
--replication-prefetch-tx-buffer-size 1000 \
--replication-commit-concurrency 100
```
At this point, the `replicadb` has been created on the replica server to sync with the `primarydb` on master server.
At this point, the `replicadb` has been created on the replica server to sync with the `primarydb` on primary server.
#### Step 2. Send a request
Expand Down Expand Up @@ -208,7 +209,7 @@ Make sure you already have [immudb installed](../running/download.md).
$ immuclient safeset foo bar
```
The client will block. This is because the primarydb requires 1 sync follower, and since the replica server is down, there is no ack from the replica server, hence synchronous transaction is blocked.
The client will block. This is because the primarydb requires 1 sync replica, and since the replica server is down, there is no ack from the replica server, hence synchronous transaction is blocked.
</WrappedSection>
Expand All @@ -226,11 +227,11 @@ the initial cluster setup, e.g.:
```shell
$ immuadmin database create replicadb -p 3324 \
--replication-is-replica \
--replication-master-address 127.0.0.1 \
--replication-master-database primarydb \
--replication-follower-username immudb \
--replication-follower-password immudb \
--replication-master-port 3322 \
--replication-primary-host 127.0.0.1 \
--replication-primary-port 3322 \
--replication-primary-database primarydb \
--replication-primary-username immudb \
--replication-primary-password immudb \
--replication-sync-enabled \
--replication-prefetch-tx-buffer-size 1000 \
--replication-commit-concurrency 100
Expand All @@ -255,11 +256,11 @@ This should be done while the database is unloaded:
```shell
$ immuadmin database create replicadb -p 3324 \
--replication-is-replica \
--replication-master-address 127.0.0.1 \
--replication-master-database primarydb \
--replication-follower-username immudb \
--replication-follower-password immudb \
--replication-master-port 3322 \
--replication-primary-host 127.0.0.1 \
--replication-primary-port 3322 \
--replication-primary-database primarydb \
--replication-primary-username immudb \
--replication-primary-password immudb \
--replication-sync-enabled \
--replication-prefetch-tx-buffer-size 1000 \
--replication-commit-concurrency 100
Expand Down Expand Up @@ -307,8 +308,8 @@ database 'replicadb' successfully unloaded
Current immudb cluster setup requires the primary node to be always predefined.
This means that in case of a primary node loss,
it is necessary to manually promote a replica to become the primary node.
Generally, electing the new master depends on the number of available instances,
their precommit state, and the replication-sync-acks setting on the master.
Generally, electing the new primary depends on the number of available instances,
their precommit state, and the replication-sync-acks setting on the primary.
#### Step 1. Inspect states of all replicas in the cluster and choose the new primary node
Expand All @@ -328,11 +329,11 @@ precommittedTxID: 734931
precommittedHash: 5e2f2feec159bc19c952a7a93832338a178936c5b258d0c906b7c145faf3a4b5
```
It's important to carefully choose the new master node in order to avoid losing committed transactions.
It is generally a good idea to promote some instance as a master that has already precommitted the largest
It's important to carefully choose the new primary node in order to avoid losing committed transactions.
It is generally a good idea to promote some instance as a primary that has already precommitted the largest
transaction contained in at least `replication-sync-acks` instances.
In the following scenario, we consider a three-node cluster with an unreachable master:
In the following scenario, we consider a three-node cluster with an unreachable primary:
```shell
# state in replica1
Expand All @@ -344,9 +345,9 @@ precommittedTxID: 734920
precommittedHash: 2a4f41c3d5b03ff014ca30b53d23ee3a098936c3b2a8a0d6e9b3b540cac166a1
```
In the event that the primary node becomes unavailable, a replica with a higher precommittedTxID should be chosen as the master.
If `replication-sync-acks` is 2, both replicas must acknowledge precommit before the master can commit.
In the scenario above, this would mean 734920 was the most recent committed transaction. Therefore, replica2 could also be selected as the new master.
In the event that the primary node becomes unavailable, a replica with a higher precommittedTxID should be chosen as the primary.
If `replication-sync-acks` is 2, both replicas must acknowledge precommit before the primary can commit.
In the scenario above, this would mean 734920 was the most recent committed transaction. Therefore, replica2 could also be selected as the new primary.
#### Step 2. Switch the selected replica to become new primary
Expand All @@ -358,15 +359,15 @@ $ immuadmin database update replicadb -p 3324 \
database 'replicadb' {replica: false} successfully updated
```
> Note that the number of required sync followers may be temporarily lowered due to the loss of the primary node.
> Note that the number of required sync replicas may be temporarily lowered due to the loss of the primary node.
#### Step 3. Switch other replicas to follow new primary
```shell
$ immuadmin database update replicadb -p 3325 \
--replication-master-address 127.0.0.1 \
--replication-master-database replicadb \
--replication-master-port 3324
--replication-primary-host 127.0.0.1 \
--replication-primary-port 3324 \
--replication-primary-database replicadb
```
#### Step 4. Truncate precommitted transactions on other replicas if needed
Expand All @@ -375,7 +376,7 @@ It may happen that the new replica will reject synchronizing with the new primar
In such case immudb will report an error in logs:
```text
immudb 2022/10/11 15:57:42 ERROR: follower precommit state at 'replicadb' diverged from master's
immudb 2022/10/11 15:57:42 ERROR: replica precommit state at 'replicadb' diverged from primary's
```
To fix this issue the replica may need to discard precommited transactions.
Expand Down
Loading

0 comments on commit 7c36835

Please sign in to comment.