lvmsync

Have you ever wanted to do a partial sync on a block device, possibly over a network, but were stymied by the fact that rsync just didn't work?

Well, fret no longer. As long as you use LVM for your block devices, you too can have efficient delta-transfer of changed blocks.

What is it good for?

Mostly, transferring entire block devices from one machine to another, with minimal downtime. Until now, you had to shutdown your service/VM/whatever, do a big cross-network dd (using netcat or something), and wait while all that transferred.

lvmsync allows you to use the following workflow to transfer a block device "mostly live" to another machine:

1. Take a snapshot of an existing LV.
2. Transfer the entire snapshot over the network, while whatever uses the block device itself keeps running.
3. When the initial transfer is finished, you shutdown/unmount/whatever the initial block device.
4. Run lvmsync on the snapshot to transfer the changed blocks

• The only thing transferred over the network is the blocks that have changed (which, hopefully, will be minimal)

1. If you're paranoid, you can md5sum the content of the source and destination block devices, to make sure everything's OK (although this will destroy any performance benefit you got by running lvmsync in the first lace)
2. Bring the service/VM/whatever back up in it's new home in a much shorter (as in, "orders of magnitude") time than was previously possible.

lvmsync also has a basic "snapshot-and-rollback" feature, where it can save a copy of the data in the LV that you're overwriting to a file for later application if you need to rollback. See "Snapback support" under "How do I use it?" for more details.

How does it work?

By the magic of LVM snapshots. lvmsync is able to read the metadata that device-mapper uses to keep track of what parts of the block device have changed, and use that information to only send those modified blocks over the network.

If you're really interested in the gory details, there's a brief "Theory of Operation" section at the bottom of this README, or else you can just head straight for the source code.

Installation

To run lvmsync, you'll need to have a working installation of Ruby 1.8 (or later) on both the machine you're transferring from, and the machine you're transferring to. On the source, you'll need vgcfgbackup (which is part of the core LVM2 toolset), and if you want to deal with thin snapshots, you'll also need thin_dump (which is part of the "thin provisioning tools" which are highly recommended for anyone working with thin-provisioned LVs). For transferring dumps between machines, you'll need SSH installed and working between the two machines.

Installing lvmsync itself is easiest using Rubygems: gem install lvmsync. This will install all the dependencies and (presumably) put the lvmsync command itself in root's PATH. If for some reason you want to install it all by hand, you'll need to copy the contents of lib/ into a directory in your Ruby library path, copy bin/lvmsync to somewhere on your PATH, and install the treetop and git-version-bump gems.

How do I use it?

For an overview of all available options, run lvmsync -h.

Efficient block device transfer

At present, the only part of the block device syncing process that is automated is the actual transfer of the snapshot changes -- the rest (making the snapshot, doing the initial transfer, and stopping all writes to the LV) you'll have to do yourself. Those other steps aren't difficult, though, and are trivial to script to suit your local environment (see the example, below).

Once you've got the snapshot installed, done the initial sync, and stopped I/O, you just call lvmsync like this:

lvmsync <snapshot LV device> <destserver>:<destblock>

This requires that lvmsync is installed on <destserver>, and that you have the ability to SSH into <destserver> as root. All data transfer takes place over SSH, because we don't trust any network, and it simplifies so many things (such as link-level compression, if you want it). If CPU is an issue, you shouldn't be running LVM on your phone to begin with.

The reason why lvmsync needs you to specify the snapshot you want to sync, and not the base LV, is that you might have more than one snapshot of a given LV, and while we can determine the base LV given a snapshot, you can't work out which snapshot to sync given a base LV. Remember to always specify the full device path, not just the LV name.

Example

Let's say you've got an LV, named vmsrv1/somevm, and you'd like to synchronise it to a new VM server, named vmsrv2. Assuming that lvmsync is installed on vmsrv2 and vmsrv2 has an LV named vmsrv2/somevm large enough to take the data, the following will do the trick rather nicely (all commands should be run on vmsrv1):

# Take a snapshot before we do anything, so LVM will record all changes
# made while we're doing the initial sync
lvcreate --snapshot -L10G -n somevm-lvmsync vmsrv1/somevm

# Pre-sync all data across -- this will take some time, but while it's
# happening the VM is still serving traffic.  pv is a great tool for
# showing you how fast your data's moving, but you can leave it out of
# the pipeline if you don't have it installed.
dd if=/dev/vmsrv1/somevm-lvmsync bs=1M | pv -ptrb | ssh root@vmsrv2 dd of=/dev/vmsrv2/somevm bs=1M

# Shutdown the VM -- the command you use will probably vary
virsh shutdown somevm

# Once it's shutdown and the block device isn't going to be written to
# any more, then you can run lvmsync
lvmsync /dev/vmsrv1/somevm-lvmsync vmsrv2:/dev/vmsrv2/somevm

# You can now start up the VM on vmsrv2, after a fairly small period of
# downtime.  Once you're done, you can remove the snapshot and,
# presumably, the LV itself, from `vmsrv1`

Snapback support

In addition to being able to efficiently transfer the changes to an LV across a network, lvmsync now supports a simple form of point-in-time recovery, which I've called 'snapback'.

The way this works is startlingly simple: as lvmsync writes the changed blocks out to the destination block device, it reads the data that is being overwritten, and stores it to a file (specified with the --snapback option). The format of this file is the same as the wire protocol that lvmsync uses to transfer changed blocks over the network. This means that, in the event that you need to rollback a block device to an earlier state, you can do so by simply applying the saved snapback files created previously, until you get to the desired state.

Example

To setup a snapback process, you need to have a local LV, with a snapshot, whose contents have been sent to a remote server, perhaps something like this:

lvcreate --snapshot -L10G -n somevm-snapback vmsrv1/somevm
dd if=/dev/vmsrv1/somevm-snapback bs=1M | pv -ptrb | \
     ssh root@vmsrv2 dd of=/dev/vmsrv2/somevm

Now, you can run something like the following periodically (say, out of cron each hour):

lvcreate --snapshot -L10G -n somevm-snapback-new vmsrv1/somevm
lvmsync /dev/vmsrv1/somevm-snapback vmsrv2:/dev/vmsrv2/somevm --snapback \
     /var/snapbacks/somevm.$(date +%Y%m%d-%H%M)
lvremove -f vmsrv1/somevm-snapback
lvrename vmsrv1/somevm-snapback-new somevm-snapback

This will produce files in /var/snapbacks named somevm.<date-time>. You need to create the somevm-snapback-new snapshot before you start lvmsync, so that you can guarantee no changes will go unnoticed.

There are some fairly large caveats to this method -- the LV will still be collecting writes while you're transferring the snapshots, so you won't get a consistent snapshot (in the event you have to rollback, it's almost certain you'll need to fsck). You'll almost certainly want to incorporate some sort of I/O freezing into the process, but the exact execution of that is system-specific, and left as an exercise for the reader.

Restoring data from a snapback setup is straightforward -- just take each snapback in reverse order and run it through lvmsync --apply on the destination machine (vmsrv2 in our example). Say at 1145 vmsrv1 crashed, and it was determined that you needed to rollback to the state of the system at 8am. You could do this:

lvmsync --apply /var/snapbacks/somevm.20120119-1100 /dev/vmsrv2/somevm
lvmsync --apply /var/snapbacks/somevm.20120119-1000 /dev/vmsrv2/somevm
lvmsync --apply /var/snapbacks/somevm.20120119-0900 /dev/vmsrv2/somevm

And you're done -- /dev/vmsrv2/somevm is now at the state it was at at 8am. A whole pile of fsck will no doubt be required, but hopefully you'll still be able to salvage something.

If you're wondering why I only restored the 0900 snapback, and not the 0800 one, it's because the snapback made at 0900 copied the changes that were sent at 0800 (and about to be overwritten at 0900) and wrote them to the 0900 snapback file. Confused much? Good.

Transferring snapshots on the same machine

If you need to transfer an LV between different VGs on the same machine, then running everything through SSH is just an unnecessary overhead. If you instead just run lvmsync without the <destserver>: in the destination specification, everything runs locally, like this:

lvmsync /dev/vg0/srclv-snapshot /dev/vg1/destlv

All other parts of the process (creating the snapshot, doing the initial data move with dd, and so on) are unchanged.

As an aside, if you're trying to move LVs between PVs in the same VG, then you don't need lvmsync, you need pvmove.

Taking a space- and IO-efficient snapshot of an LV

But wait, there's more! lvmsync also has the ability to dump out the snapshot data to disk, rather than immediately applying it to another block device.

To do this, use the --stdout option when you're running lvmsync, and instead of writing the changes to another block device, it'll instead dump the "change stream" to stdout (so redirect somewhere useful). This allows you to dump the changes to a file, or do some sort of fancy footwork to transfer the data to another lvmsync process to apply the changes to a block device.

For example, if you just wanted to take a copy of the contents of a snapshot, you could do something like this:

lvmsync --stdout /dev/somevg/somelv-snapshot >~/somechanges

At a later date, if you wanted to apply those writes to a block device, you'd do it like this:

lvmsync --apply ~/somechanges /dev/somevg/someotherlv

You can also do things like do an lvmsync from the destination -- this is useful if (for example) you can SSH from the destination to the source machine, but not the other way around (fkkn firewalls, how do they work?). You could do this by running something like the following on the destination machine:

ssh srcmachine lvmsync --stdout /dev/srcvg/srclv-snap | lvmsync --apply - /dev/destvg/destlv

Theory of Operation

This section is for those people who can't sleep well at night without knowing the magic behind the curtain (and to remind myself occasionally how this stuff works). It is completely unnecessary to read this section in order to work lvmsync.

First, a little bit of background about how snapshot LVs work, before I describe how lvmsync makes use of them.

An LVM snapshot "device" is actually not a block device in the usual sense. It isn't just a big area of disk space where you write things. Instead, it is a "meta" device, which points to both an "origin" LV, which is the LV from which the snapshot was made, and a "metadata" LV, which is where the magic happens.

The "metadata" LV is a list of "chunks" of the origin LV which have been modified, along with the original contents of those chunks. In a way, you can think of it as a sort of "binary diff", which says "these are the ways in which this snapshot LV differs from the origin LV". When a write happens to the origin LV, this "diff" is potentially modified to maintain the original "view" from the time the snapshot was taken.

(Sidenote: this is why you can write to snapshots -- if you write to a snapshot, the "diff" is written to some more, to say "here are some more differences between the origin and the snapshot").

From here, it shouldn't be hard to work out how LVM uses the combination of the origin and metadata LVs to give you a consistent snapshot view -- when you ask to read a chunk, LVM looks in the metadata LV to see if it has the chunk in there, and if not it can be sure that the chunk hasn't changed, so it just reads it from the origin LV. Miiiiighty clever!

In lvmsync, we only make use of a tiny fraction of the data stored in the metadata LV for the snapshot. We don't care what the original contents were (they're what we're trying to get away from). What we want is the list of which chunks have been modified, because that's what we use to work out which blocks on the original LV we need to copy across. lvmsync never actually reads any disk data from the snapshot block device itself -- all it reads is the list of changed blocks, then it reads the changed data from the original LV (which is where the modified blocks are stored).

By specifying a snapshot to lvmsync, you're telling it "this is the list of changes I want you to copy" -- it already knows which original LV it needs to copy from (the snapshot metadata has that info available).

See Also

Whilst I think lvmsync is awesome (and I hope you will too), here are some other tools that might be of use to you if lvmsync doesn't float your mustard:

• blocksync.py -- Implements the "hash the chunks and send the ones that don't match" strategy of block device syncing. It needs to read the entire block device at each end to work out what to send, so it's not as efficient, but on the other hand it doesn't require LVM.

• bdsync -- Another "hash the chunks" implementation, with the same limitations and advantages as blocksync.py.

• ddsnap -- Part of the "Zumastor" project, appears to provide some sort of network-aware block device snapshotting (I'm not sure, the Zumastor homepage includes the word "Enterprise", so I fell asleep before finishing reading). Seems to require kernel patches, so there's a non-trivial barrier to entry, but probably not such a big deal if you're after network-aware snapshots as part of your core infrastructure.