VDiff: Support a max diff time for tables

[mattlord]

Description

This PR adds a VDiff create flag called --max-diff-duration which allows a user to limit the maximum runtime for a single table diff run (the table diff can span N runs). This allows a user to further limit the impact of a VDiff on their production systems. Please see the issue for more information on the use case.

Click here to see a simple manual test:

./101_initial_cluster.sh

mysql < ../common/insert_commerce_data.sql

./201_customer_tablets.sh

./202_move_tables.sh

for i in {1..21}; do
  mysql commerce -e "insert into customer (email) values ('${RANDOM}_person_${RANDOM}@planetscale.com'); insert into customer (email) select email from customer"
done

sleep 180

vtctldclient vdiff --target-keyspace customer --workflow commerce2customer create --max-diff-duration 10s --update-table-stats

while true; do
  results=$(vtctldclient vdiff --target-keyspace customer --workflow commerce2customer show last)
  echo "${results}"
  if echo "${results}" | grep -q "State:        completed"; then
    echo "#################### DONE ####################"
    vtctldclient vdiff --target-keyspace customer --workflow commerce2customer show last --verbose
    break
  fi
done

command mysql -u root --socket=${VTDATAROOT}/vt_0000000201/mysql.sock vt_customer -e "select count(*) as corder_rows from corder; select count(*) as customer_rows from customer;"

let primaryuid=$(vtctldclient GetTablets --keyspace=customer --tablet-type=primary | awk '{print $1}' | cut -d "-" -f2)+0
echo

curl -s localhost:15${primaryuid}/debug/vars | grep -i VDiffRestarted

Final results being:

VDiff Summary for customer.commerce2customer (dbc99b78-dfac-4b80-a789-d01fac7f152d)
State:        completed
RowsCompared: 14680067
HasMismatch:  false
StartedAt:    2023-12-15 00:37:39
CompletedAt:  2023-12-15 00:39:03

Table corder:
	State:            completed
	ProcessedRows:    5
	MatchingRows:     5

Table customer:
	State:            completed
	ProcessedRows:    14680062
	MatchingRows:     14680062

Use "--format=json" for more detailed output.

+-------------+
| corder_rows |
+-------------+
|           5 |
+-------------+
+---------------+
| customer_rows |
+---------------+
|      14680062 |
+---------------+

"VDiffRestartedTableDiffsCount": {"customer": 2},

Related Issue(s)

• Fixes: Feature Request: VDiff to support periodically refreshing its DB snapshots to limit impact #14847

Checklist

• "Backport to:" labels have been added if this change should be back-ported
• Tests were added
• Did the new or modified tests pass consistently locally and on the CI
• Documentation: vtctldclient docs will automatically be added

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[ajm188]

lgtm! just a few questions that shouldn't affect mergability

[ajm188]

333 seems like an oddly-specific duration to me ... is there a reason you went with this?

[mattlord]

Only in that I needed to pick something specific and this is the random number I typed. 🙂

[ajm188]

nice

[ajm188]

this is more a general question, since the other duration-type in this message is also int64: is there a reason we do the "int as seconds" thing here rather than using vttime.Duration like we do in vtctldata and others?

(also extremely nit-picky and GH won't let me comment on the line, but i'm spotting a double-space:

- int64  max_rows = 3;
+ int64 max_rows = 3;

[mattlord]

I thought about that too, but vttime is a struct with a seconds and nanoseconds value and in this case I only cared about the seconds so just used that part of it. I'm fine changing it back though too.

I'll fix the spacing. Thanks!

[ajm188]

i'm good with the consistency! (i.e. don't change it)