Backup strategy

[jeromegn]

If one wanted to do a backup of the database, what's the best practice here? Is there a way to do an "online" (not shutting down the process using the database) backup?

Since there are many directories and files, I assume the safest way is to persist from the process, exit and then create a tarball of the whole directory.

With SQLite, for example, it's possible to run a VACUUM INTO that will create a space-efficient snapshot of the database. SQLite can support multiple processes though, so it's a different ball game.

[marvin-j97]

There are some different ways to go about it. The easiest way would be to start a snapshot scan over every partition in the entire database and write them out to a backup directory. That will also get rid of any tombstones (and maybe old versions, could be configured), so the created snapshot will be as compact as can be. But it won't be the same database physically, and scanning through a database is slower than a simple file copy. It can kind of run passively in the background as it just relies on the snapshot feature, so compaction can still progress, because compaction keeps old versions around as needed. For a small-ish database, this option could be enough I think, and would probably be a good feature addition.

RocksDB has this feature called Checkpoint that hard links segments if possible. That makes the checkpoint really fast, but only if the backup is on the same file system. Otherwise, for standard backups, the files are simply copied. When copying files like that the keyspace would need to lock compaction as not to lose files temporarily.

[jeromegn]

When copying files like that the keyspace would need to lock compaction as not to lose files temporarily.

Is there a way to do this currently? I assume there are operations that can have that effect, but not sure which ones 😅

[marvin-j97]

All 3 approaches I mentioned were more like implementation details, not implemented features.

Do you have a rough estimate of the database size you're trying to keep backed up?

[jeromegn]

Not really, I'm thinking through future use cases where backup + restore is an important thing.

It's safe to imagine a dataset between 10GB and 100GB for what I'm thinking about.

[marvin-j97]

Personally I would try and go for a hardware solution like EBS snapshots, but I understand that may not always be possible. I think I need to push this back until I have figured out key-value separation because that will influence how to do online backups...

Writing to a single flat file using a snapshot scan would always work though, I will take a look at that and benchmark it... that would require loading the single file back into a database though (and rewriting it into segments), so it's not plug and play compared to the other solution...

And for an offline backup, using cp -R should just always work either way (and the segments could even be hard linked because they are immutable).

[marvin-j97]

I'm gonna close this in favour of #52. Realistically I think there will be two ways of backups: offline (trivial cloning) and checkpointing.