Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Schema Migration API endpoint #128

Open
dmitrizagidulin opened this issue Dec 23, 2015 · 0 comments
Open

Implement Schema Migration API endpoint #128

dmitrizagidulin opened this issue Dec 23, 2015 · 0 comments
Milestone

Comments

@dmitrizagidulin
Copy link
Contributor

One of the major user complaints with Riak Search (and Solr in general) is the inability to re-index existing documents when changing a Search schema.
The only way to perform schema migrations currently (to re-index all existing documents) is to manually write scripts to "touch" every object in an index (that is, read an object and write it back immediately) -- scripts that are usually powered by a streaming list keys or "select all documents" query.

Given that Explorer keeps cached lists of keys (that can be refreshed via either streaming list keys call or, better yet, something like issue #129 - key list via search query or issue #127 - throttled background key listing), we would make users' lives dramatically easier if we provided a push-button ability to do schema migration:

  1. Generate cached key lists for all the buckets in a given bucket type that has a search index, if not yet present.
  2. Set up a long-running Explorer job (throttled, connection-pooled and multi-threaded) to iterate over those key lists and "touch" the objects (read-then-write).
  3. Since the key lists are generated beforehand, provide polling capability to determine job progress on the client side.

Possibly only implement this in Dev mode. (Needs discussion, though).

@dmitrizagidulin dmitrizagidulin added this to the Query Tools milestone Dec 23, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant