CHT has high availability through failover

[garethbowen]

Key Results

• Proof of concept failover in AWS test instance complete and communicated
• CHT instance is still available during a datacentre outage

Initiatives

For ultimate availability we need to support automatic failover so if one server becomes unavailable another one can take over immediately. Once the prerequisite is complete the database layer will already have automatic failover if replicas are stored in separate locations. However we still need to support failover for most of the API service to get traffic to the db.

• Prevent data loss through re-sync when restoring data from backup cht-core#8093
• Refactor downwards continuous replication following the Nairobi protocol cht-core#8296
• enable couchdb replicas for multi-replica couch
• design failover architecture (stateless services, single sentinel, etc)
• implement failover architecture

Aspects to consider

• This should be a hot swap so all instances are serving traffic, and when one goes down the traffic automatically goes to the other
• Sentinel may need to be run on only one instance
• Check for state stored in API, eg: caches
• Work out how to do singleton tasks (eg: generate forms) only once to avoid conflict
• Work out orchestration
• Work out load balancing