Skip to content
This repository has been archived by the owner on Aug 3, 2022. It is now read-only.

Change snapshot server to return a sqlite database #35

Open
gbrail opened this issue Dec 9, 2016 · 0 comments
Open

Change snapshot server to return a sqlite database #35

gbrail opened this issue Dec 9, 2016 · 0 comments

Comments

@gbrail
Copy link
Contributor

gbrail commented Dec 9, 2016

In the case that the snapshot server produces a large amount of output, it won't do to return a big JSON blob. We added support to return a stream of protobufs, but this doesn't work either unless the consumer is written very carefully to process that without additional streaming.

A more scalable approach might be this:

  1. Have the snapshot server produce a sqlite database containing the requested data, with a standard set of mappings from Postgres to SQLite types.
  2. Have the snapshot server simply stream that result from disk.
  3. Clients can use the sqlite database directly if they wish, modify it, or read it and replace it with something else.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant