A version of Kubernetes using Apache Hadoop YARN as the scheduler. Integrating Kubernetes with YARN lets users run Docker containers packaged as pods (using Kubernetes) and YARN applications (using YARN), while ensuring common resource management across these (PaaS and data) workloads.
This integration is under development. Please expect bugs and significant changes as we work towards making things more stable and adding additional features.
Kubernetes and Kubernetes-YARN are written in Go. Currently, only a vagrant-based setup is supported. That said, bringing up a vagrant-based Kubernetes-YARN environment is fairly easy.
Please ensure you have boot2docker, Go (at least 1.3), Vagrant (at least 1.6), VirtualBox (at least 4.3.x) and git installed. Run boot2docker to bring up a VM with a running docker daemon (this is used for building release binaries for Kubernetes).
$ $(boot2docker shellinit) #sets up docker env vars
$ echo $GOPATH
/home/user/goproj
$ mkdir -p $GOPATH/src/github.com/hortonworks/
$ cd $GOPATH/src/github.com/hortonworks/
$ git clone [email protected]:hortonworks/kubernetes-yarn.git
$ cd kubernetes-yarn
$ build/release.sh #builds kubernetes release binaries
$ hack/build-go.sh #builds kubernetes client binaries
$ cluster/kube-up.sh #brings up kubernetes cluster
Following these steps will bring up a multi-VM cluster (1 master and 3 minions, by default) running Kubernetes and YARN. Please note that, depending on your local hardware and available bandwidth, bringing the cluster up could take a while to complete.
By default, the kubernetes master is assigned the IP 10.245.1.2. The YARN resource manager runs on the name host. Once the vagrant cluster is running, the YARN dashboard accessible at http://10.245.1.2:8088/
The HDFS dashboard is accessible at http://10.245.1.2:50070/
For instructions on creating pods, running containers and other interactions with the cluster, please see Kubernetes' vagrant instructions here
In order to run a test map-reduce job, log into the cluster (ensure that you are in the kubernetes-yarn directory) and run the included test script.
$ vagrant ssh master
[vagrant@kubernetes-master ~]$ cd hadoop/install/hadoop-2.6.0-SNAPSHOT/
[vagrant@kubernetes-master hadoop-2.6.0-SNAPSHOT]$ sudo su #you need to be root to run this test script
[root@kubernetes-master hadoop-2.6.0-SNAPSHOT]# ./test-pi-yarn.sh