Server to Server synchronization

1. Why?

RapidFTR is mostly used in emergency situations where the internet connection is very intermittent. Most of the times the captured child data might be lying in one of the emergency camps. Its very important to share this data -

• with another camps in the same emergency(which helps field workers who are moving across the camps to quickly identify the children and reunite them with their parents) 
• with a central server so that the quick and desired actions could be taken on the child data. 

This also helps in sharing the multiple copies of the data across all the authentic sources, which will recover the system quickly if a disaster/damage/theft of data happens with in one of the camps. Above are some of the reasons why we chose Couchdb to be our data store. Couchdb is reliable in the intermittent data connections scenario and excels at synchronization(replication) across multiple cocuhdb servers.

2. How it Works?

Couchdb server to server replication(called as synchronisation from now onwards) is always pushing and puling the data between two servers(by default bidirectional/master-> master and it provides us the apis to convert it into master -> slave mode). Plain vanila couchdb by default don't provide any authentication and authorization for synchronization across the servers. But it provides excellent api's to leverage the security and leaves it to the application to use apis to implement any kind of security we want.

The following pictures along with some relevant description would help you understand how it is implemented in RapidFTR.

Overview of how synchronisation works

For secured synchronisation of data across the servers we need to enable some kind of authentication for the couchdb servers. Couchdb by default have the concept of Server admins, database admins and database readers. We will be leveraging this to create Server admins from RapidFTR application. Once we create the Server admins for couchdb through RapidFTR application, we can share this credentials, along with the application url for other server whom we authorize to synchronize with(continue reading for more details).

Consider camp1 was first established in the emergency and field workers started capturing the data into the camp1 server. Due to the heavy influx of refugees lead agency decided to have one more camp(camp2) in a near distance. In order to avoid the duplicate entries and the other reasons we discussed above, camp2 should have same data before it(camp2) could be used for data capturing.

In order to synchronize the data across the camps, camp1 should share its synchronization credentials with camp2.

• camp1(say the application is running at http://10.10.1.2:3000) can create the credentials using the RapidFTR application by navigating to Manage System Users under System Settings.  
• Once the credentials are created (say the username and password are C1U1 and C1P1 respectively), camp1 system admin should share(verbally/offline) this credentials with camp2 system admin. 
• camp2 system admin will navigate to devices tab and configures a new server to synchronise with, by providing the usernmae(C1U1), password(C1P1) along with the application url(http://10.10.1.2:3000) for camp1 
• Once the camp2 system admin saves the new server configuration, replication starts and the status of the replication would be shown to the user. At the end of the replication we can see that both the camps(camp1 and camp2) would be sharing the same children data. Yes, you've read it right, currently we are sharing only the children data across the servers. We would be sharing other data going forward. 

End to End flow:


Synchronisation will also work in multi server mode:


3. Securing the Synchronisation process and more technical details

Below are some of the technical stuff which goes in background during the synchronisation process. There are also some security bits to follow for the production deployments.

• When ever System Users are created(if you are following above example, this should be done in camp1) using the RapidFTR application, these details would be directly captured under _users database of couchdb. _users is the general purpose database which couchdb uses to authenticate the users during replication process.
• Once the server is configured(this would be done in camp2 after receiving the username, password and url) and saved, the first request go to the application which is residing in the "url" specified during the server configuration, gets the required databases to replicate. Once we know the list of required databases, we push an entry into "_replicator" database of couchdb for each database to replicate. _replicator database does most of the heavy lifting around synchronisation and document conflict management. Couchdb writes back some important stats(_replication_stats, _replication_state, _replication_state_time) into the _replication database for each replication entry. We consume this statuses to display them back to the user.
• Couchdb never copies the revisions of the data during the synchronisation.
• More details about replication could be found at the couchdb wiki
• To know more about couchdb's _replicator database have a look at this gist https://gist.github.com/832610

Security bits for deploying RapidFTR during server to server synchronisation

• Couchdb supports ssl by default, its a bit of configuration that should be done from couchdb to enable ssl(either through futon or from the default/local.ini). If you are interested to try the couchdb ssl for replication have a look at this link
• For server to server synchronization to work, we need to expose the couchdb to the world(change the bind address from 127.0.0.1 to 0.0.0.0. To avoid unnecessary issues around authentication, its highly recommended to enable require_valid_user = true in local.ini. This will challenge the users with basic authentication if any one wants to directly access couchdb.
• Couchdb1.2 has loads of improvements around the optimisation of replication and with security around how the user passwords to be shown to the users through futon admin. Its again highly recommended to use couchdb1.2 for production deployments.
• To know more about the user types and security visit Matt woodward's blog
• Here are the list of threads which discussed about authentication and authorization of couchdb in detail.