Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RFC: Move VDiff to vttablet #9216

Closed
ajm188 opened this issue Nov 12, 2021 · 1 comment
Closed

RFC: Move VDiff to vttablet #9216

ajm188 opened this issue Nov 12, 2021 · 1 comment

Comments

@ajm188
Copy link
Contributor

ajm188 commented Nov 12, 2021

Feature Description

Currently vdiff runs at the vtctld layer, which is responsible for contacting tablets involved in a workflow, streaming the rows into memory, doing the diff, and output the results. This works but it does not scale well for running many vdiffs, or large vdiffs at once on vtctlds (which often are much less beefy than vttablets, hardware-wise as well).

If we were able to push down the work to just the tablets involved in the workflow, and leave the vtctld to only coordinate and collect results from the tablets, we could achieve much better vdiff throughput.

Design

This issue is a placeholder to start the discussion around how best to achieve this. We should eventually write up a design before doing significant work here.

@mattlord
Copy link
Contributor

Closing this as fixed in: #10382

Planned follow-up work can be seen here: #10494

Please feel free to re-open this if I missed or misunderstood something.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants