Skip to content

Changing roles between nodes after failover

Jump to bottom
ᴀɴᴛᴏɴ ɴᴏᴠᴏᴊɪʟᴏᴠ edited this page Apr 19, 2024 · 1 revision

By default, RDS cluster works in standby mode. It means that Redis instances on master node will handle all read and write commands, and Redis instances on minion node will only synchronize with instances on master node.


RDS cluster in standby mode

Standby failover mode based on the use of virtual IP (for example, using keepalived, which is natively supported by RDS).

Important

We do not recommend using DNS for standby failover. The resolved IP of the node can be cached at the system/service level and your service will continue to send requests to the down node.

When the master node reboots or goes down, the virtual IP is transferred from the master to the minion node, and Redis instances on the minion node start serving requests. There is no automatic role change after failover, so you must change them manually.

To change roles between nodes, follow the next steps.

Caution

Note that all actions must be performed in the specified order. Performing actions in a different order or skipping them may result in data loss.

Actions on new master (ex-minion)

  1. Open configuration file /etc/rds.knf and change node role to the master in replication:role property;
  2. Change property replication:master-ip to the IP of machine;
  3. Stop syncing daemon using sudo service rds-sync stop command;
  4. Run sudo rds replication-role-set master command and follow the instructions;
  5. Start syncing daemon using sudo service rds-sync start command;
  6. Check log file /var/log/rds/rds-sync.log for errors or warnings.

Actions on new minion (ex-master)

  1. If syncing daemon is works stop it using sudo service rds-sync stop command;
  2. Open configuration file /etc/rds.knf and change node role to the minion in replication:role property;
  3. Change property replication:master-ip to the IP of master node;
  4. Run sudo rds replication-role-set minion command and follow the instructions;

Tip

If you are not sure about data consistency, or if you have a lot of data, you can delete all existing data with the command rm -rf /opt/rds/conf/* /opt/rds/meta/* /opt/rds/data/* /opt/rds/log/*.

  1. Start syncing daemon using sudo service rds-sync start command;
  2. Check log file /var/log/rds/rds-sync.log for errors or warnings.
Clone this wiki locally