Add support for Spark 2.1.0's native dispatcher failover

[christek91]

According to the Spark documentation (http://spark.apache.org/docs/latest/running-on-mesos.html#cluster-mode), Spark 2.1.0 can write recovery state data to Zookeeper, which can then be read by new spark dispatchers during a dispatcher failover, or when restarting the service in Mesosphere.

Currently, if you restart a Spark dispatcher service in Mesosphere the previous dispatcher's job history is lost. It would be nice to persist this data so that it works similarly to the History Server (which persists its data in HDFS or elsewhere) after a restart.

Since the Spark package already does some zookeeper configuration (

spark-build/docker/runit/service/spark/run

Line 10 in a199353

export SPARK_DAEMON_JAVA_OPTS="$SPARK_DAEMON_JAVA_OPTS -Dspark.deploy.zookeeper.dir=/spark_mesos_dispatcher_${DCOS_SERVICE_NAME}"

)
this may only be a matter of exposing the spark.deploy.recoveryMode conf value through an env variable in the image and a config.json field.