Chapter 9.1 --- Chubby Lock Service.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chubby lock service\n",
    "\n",
    "* A **distributed lock service** is a service which provides distributed nodes a common lock implementation.\n",
    "* These locks are used for access to shared resources in a replication environment.\n",
    "* Chubby is Google's implementation of a distributed lock service, and it was very influential in further designs.\n",
    "* A distributed lock service can be used as a simple name server, and Chubby is often deployed for this use-case as well at Google (the alternative is using something like DNS).\n",
    "* In particular, it's often used to achieve distributed master elections (as in e.g. choosing a GFS master).\n",
    "\n",
    "\n",
    "* Chubby is a working Paxos wrapper, distributed using the lock service abstraction because that is what is most familiar to sequential-oriented developers.\n",
    "* Its use cases are all course-grained lock use cases. Thus the design prioritizes consesus over throughput.\n",
    "\n",
    "\n",
    "* A Chubby cluster has some number of machines (five is typical), of which one is the master and the rest are follower nodes.\n",
    "* Followers only copy the data. The master handles all of the reads and writes.\n",
    "* A master lease is given to the master, with a certain timeout. Nodes that vote on the master promise not to vote again until the cooldown is over, so an elected master is assured that it is the true master during that interval, and can act without consulting followers.\n",
    "* NB: this timeout pattern is what caught MongoDB out in the field. An epoch pattern, like in MongoDB v1, is probably better.\n",
    "* The resources being locked are files. The file system used is UNIX-like, but simplified.\n",
    "* Chubby locks have two states: exclusive, for writing, and shared, for reading.\n",
    "\n",
    "\n",
    "* An action on a distributed file is dependent on prior observed file state. If the state changes in the middle of a read-modify-write loop, for example, the data written may no longer be valid.\n",
    "* Actions against chubby locked resources must provide isolation gauarantees (this is a distributed service!) to be useful, otherwise corruption can occur due to unsafe writes.\n",
    "* Recall that providing full isolation requires strong sequential ordering.\n",
    "* The Chubby solution is to have clients ask, separately, for sequence numbers. When the client finally performs the action, it provides Chubby with this number. Chubby checks the sequence string to determine whether or not the action is still valid (presumptions are still satisfied).\n",
    "\n",
    "\n",
    "* Chubby also implements a cache layer on top.\n",
    "\n",
    "\n",
    "* In the case of a failover, there is a grace period after a lock lease ends during which the lock holder continues to hold the lock, and blocks the application layer, but doesn't perform any actions (so as to avoid inconsistent data).\n",
    "* If a master election succeeds before the grace period ends, the lock client and the master perform a three-roundtrip lease reaquisition dance.\n",
    "* If the master election does not succeed before the grace period ends, the lock client gives up and reports an error to the application layer.\n",
    "* Essentially configurable availability timeout behavior.\n",
    "\n",
    "\n",
    "* After a failover the master has to replicate the state of the old master in a state-preserving way. This means:\n",
    "  * Lock-giving and taking-away operations are synchronously replicated to disc (via consensus algorithms, specifically Paxos).\n",
    "  * Lock lease extensions are not synchronously replicated to disc. Instead, when a failover occurs, the new master refreshes the lease to the maximum lease the old master could possibly have given.\n",
    "    * Tradeoff between locks being held too long in case of a failover and lease extension speed.\n",
    "    * Synchronous replication is slow, asynchronous replication doesn't provide the necessary strong consistency guarantee.\n",
    "    * Lock hold length is un-important in a course-grained locking system. \n",
    "    * Lease extensions happen often, by contrast.\n",
    "    * Speed over failover holds is worth it.\n",
    "    \n",
    "    \n",
    "* How is Chubby used as a name service, and why is that advantageous?\n",
    "* The primary name service competitor is DNS.\n",
    "* DNS records have a time-to-live (TTL), which manages how long they live in cache.\n",
    "* It's impossible to flush the DNS cache globally (?), you have to just wait until the TTLs expire.\n",
    "* To provide rapid name changes (so that services aren't blocked by 404s to endpoints that have moved), you have to set a small TTL. But a small TTL results in more Keep-Alive traffic, which can overwhelm the DNS server.\n",
    "* A Chubby-based name service recipe allows an application to finely control endpoint changes.\n",
    "* To populate a service, take a lock on a file describing the endpoint.\n",
    "* To move a service, take a write lock and change the endpoint description.\n",
    "* Instant change propagation!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}