Build command line tool that can create a pre-built leveldb database to enable fast startup

[kriswehner]

In order to support clients with long offline durations, we need to maintain a very large local leveldb (for us, 7-10 days of WAL), but we cannot afford to take the time hit of loading that WAL at startup time.

What we need to do is:

• Build a tool that connects to PG, starts logical replication at 0/0 LSN and downloads all available WAL, then exits.
• Update the changeserver to find the max LSN in its local leveldb at startup time and then use that to resume replication

[gbrail]

What if we added the ability for a changeserver to dump out its own database so that another changeserver can import it? That might be easier to control than relying on PG -- one changeserver can bootstrap another. (Might be tricky to implement in some deployment environments, however.)

[kriswehner]

The reason I phrased it the way I did is just the operational aspect: I'd like to "cook" a leveldb database and have it warm but on something like s3, rather than doing an on-demand snapshot and then loading it from a peer changeserver.

That said, I was also imagining this as a command line wrapper around the changeserver internals to avoid re-inventing the wheel.

All that said, I don't per se care that much about the implementation as long as there's a clean set of operational steps to get a warm db.

[kriswehner]

@gbrail I've done an initial port to rocksdb using gorocksdb to enable this. I'd love you to give it a once over and get your opinion. The storage & changeserver tests are all green, but I'm not sure your degree of confidence in them.

master...seatme:feature/rocksdb

[gbrail]

Whoa -- big change, but probably a good one.

I've worked with RocksDB before and found that it's reliable and fast. The
only problem I had in the past was that it's a pain to depend on since it's
not available on as many platforms as LevelDB. The cockroachDB thing makes
that a lot easier though.

I'm curious, though -- the current "storage" interface just calls LevelDB
using the C API. The RocksDB C API is just the LevelDB C API with the
functions renamed, so it should be a very easy port. Did you consider just
doing that?

(Too many years of Node.js programming has made me a stickler for
understanding exactly what code we depend on and I try not to make a giant
dependency tree when I can avoid ...)

On Sun, Nov 13, 2016 at 11:38 PM, Kris Wehner [email protected]
wrote:

@gbrail https://github.com/gbrail I've done an initial port to rocksdb
using gorocksdb to enable this. I'd love you to give it a once over and get
your opinion. The storage & changeserver tests are all green, but I'm not
sure your degree of confidence in them.

master...seatme:feature/rocksdb
master...seatme:feature/rocksdb

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#6 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAf0a3nebdPUSTW-DzBIw0vWrPe-_EURks5q-A_8gaJpZM4Kpz9V
.

Greg Brail | apigee https://apigee.com/ | twitter @gbrail
http://twitter.com/gbrail @apigee https://twitter.com/apigee

[kriswehner]

My instinct is very much to let someone else smart who's focused on it maintain the C code. Since there's a strong, community driven database interface that we don't have to maintain, it lets us focus on the app code. In particular, my engineers don't tend to be C developers, and with gorocksdb, everything (including the comparator functions) is in golang, so it's much more accessible.

With all due respect to the node.js community, I don't think that instinct applies here :). Obviously we need to keep an eye on library quality, etc, but I'm very much a firm believer in using the community implementations.

[gbrail]

Go-rocksdb looks very nice. Let me take a look at your changes. I want to
be a little careful since we have the LevelDB stuff already working pretty
well. I also have a busy week. But the changes look like they're for the
best so I will work on it.

BTW RocksDB has column families -- I may want to extend this work to use a
separate column family for the "metadata." That also simplifies the
comparator which is nice.

On Mon, Nov 14, 2016 at 10:00 AM, Kris Wehner [email protected]
wrote:

My instinct is very much to let someone else smart who's focused on it
maintain the C code. Since there's a strong, community driven database
interface that we don't have to maintain, it lets us focus on the app code.
In particular, my engineers don't tend to be C developers, and with
gorocksdb, everything (including the comparator functions) is in golang, so
it's much more accessible.

With all due respect to the node.js community, I don't think that instinct
applies here :). Obviously we need to keep an eye on library quality, etc,
but I'm very much a firm believer in using the community implementations.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#6 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAf0a8gmvdn2URQ6PmO3LEffyD0Qd0Bcks5q-KGugaJpZM4Kpz9V
.

Greg Brail | apigee https://apigee.com/ | twitter @gbrail
http://twitter.com/gbrail @apigee https://twitter.com/apigee