Exploring simple Vagrant setup for learning about Ceph Luminous
Some Ceph testing scripts that can be used with http://docs.ceph.com/docs/master/start/ to get familiar with components of Ceph.
The instructions in http://docs.ceph.com/docs/master/start/ relating to CentOS7 do not work – don’t try and follow them. In particular there appears to have been a discoonnect for the last two years between the RHEL development team and the CentOS7 downstream team.
https://bugzilla.redhat.com/show_bug.cgi?id=1292577 has some details on the disconnect. The entry describes a broken situation that seemingly has been unfixed for 2+ years. The docs at doc.ceph.com fail to mention this, which almost certainly has wasted a lot of time for people over the last two years. Although Redhat is the lead sponsor for Ceph and for CentOS7 there is apparently little overlap. Per the bugzilla thread:
The CentOS Storage SIG Ceph repos do not currently match up with the downstream RH Ceph Storage product. The two are maintained by almost entirely separate groups of people. They are not connected in the way that CentOS itself and RHEL Base are connected, for example, or RDO and RHEL OSP, etc.
Eventually we need to get more alignment there, but that's not the reality today. (If you have a support contract with RH it would be good to pass this feedback up through your RH support representatives.)
Another possible approach is ceph-ansible - https://github.com/ceph/ceph-ansible. This is an important go to location for production Ceph deployments. For learning, the ceph-ansible system is quite complicated. There are many YAML configuration files, only a few of which need site settings. The documentation at http://docs.ceph.com/ceph-ansible/master/ is relatively sparse.
Applying the ceph-ansible instructions and targetting CentOS7 as the backend distribution failed in similar ways to direct efforts with hand configuration. ceph-ansible does reportedly work for others with the Redhat Ceph Storage v1 (RHCS v1). RHCS is now at v3.0, so it is unclear whether working with v1 is good, bad or neutral.
Debian Stretch does seem to work as a backend based on Luminous following the instructions in http://docs.ceph.com/docs/master/start/. One instruction has a small missing qualifier. The command
ceph-deploy install {node}
needs the explicit flag --release luminous to work
with Luminous. It should read
ceph-deploy install {node} --release luminous
A vagrant setup, command notes and helper scripts for testing this way have been archived in this repo.
The current Debian NTP setup is generating some errors that should be fixed before using this setup for too much.
Vagrantfile - a vagrant setup that creates (1) a provisioner machine; (2) three Ceph nodes
that host both OSD and MON, MGR, etc... services; (4) a RBD client node that mounts an RBD pool as
a regular ext4 file system and exports the file system over regular NFS.
commands - a list of commands used by hand to get things running. Most of these could be in a
set of provision scripts.
createhosts.sh - a provisioning shell script that installs a /etc/hosts file on each of the vagrant VMs.
getml.sh - a provisioning shell script that gets a list of the vagrant VMs that have been instantiated.
preflight-all.scr - commands that do some distribtuion specific provisioning steps for all the VMs.
preflight-all.sh - a provisioing shell script that executes the steps in preflight-all.scr.
preflight.scr - commands that do some distribtuion specific provisioning steps for the admin VM.
preflight.sh - a provisioing shell script that executes the steps in preflight.scr.
setmasterkey.sh - a provisioing shell script that puts a common key for the admin node to use on all nodes.