Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Schema Engine's TablesWithSize80 query #17066

Merged
merged 13 commits into from
Oct 28, 2024
Merged

Improve Schema Engine's TablesWithSize80 query #17066

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
51741ed
Improve Schema engine's TablesWithSize80 query
mattlord Oct 25, 2024
5da3b82
Use a longer timeout when waiting for vrep pos prior to locks
mattlord Oct 25, 2024
4558c4e
Correct query so that we're not matching table name prefixes
mattlord Oct 25, 2024
5d71b8e
Make query exponentially faster
mattlord Oct 25, 2024
94d4c96
Minor change from self review
mattlord Oct 25, 2024
bed84fa
Be less aggressive in timeout extension
mattlord Oct 25, 2024
1b3511b
Use full partition name indicator (second p before the num)
mattlord Oct 25, 2024
d68b561
Merge branch 'main' into 1000s_tables_perf
shlomi-noach Oct 27, 2024
ea56d4f
Use safer query that's only ~ 50% slower.
mattlord Oct 27, 2024
01b08a1
Remove no longer relevant comments
mattlord Oct 27, 2024
0467caa
support NULL for TABLE_OPTIONS
shlomi-noach Oct 27, 2024
3fffcb7
Merge branch 'main' into 1000s_tables_perf
shlomi-noach Oct 28, 2024
8df9040
using subselect. Adding comments
shlomi-noach Oct 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go/mysql/flavor_mysql.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -430,7 +430,7 @@ const TablesWithSize80 = `SELECT t.table_name,
SUM(i.allocated_size)
FROM information_schema.tables t
LEFT JOIN information_schema.innodb_tablespaces i
ON i.name LIKE CONCAT(t.table_schema, '/', t.table_name, IF(t.create_options <=> 'partitioned', '#p#%', '')) COLLATE utf8mb3_general_ci
ON SUBSTRING_INDEX(i.name, '#p#p', 1) = CONCAT(t.table_schema, '/', t.table_name) COLLATE utf8mb3_general_ci
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#p#p is only true if partitions are named as p0, p1, etc. but not if they're named foo1, foo2, f3. I'm checking if all that's needed is to change into #p#, otherwise momentarily marking as "request changes" so the PR isn't merged yet.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I actually thought the tablespace names were always 0,1,2... but you're right:

mysql> CREATE TABLE trb3 (id INT, name VARCHAR(50), purchased DATE)     PARTITION BY RANGE( YEAR(purchased) ) (         PARTITION my1990 VALUES LESS THAN (1990),         PARTITION my1995 VALUES LESS THAN (1995),         PARTITION p2 VALUES LESS THAN (2000),         PARTITION p3 VALUES LESS THAN (2005)     );
Query OK, 0 rows affected (0.06 sec)

mysql> select name from information_schema.innodb_tablespaces where name like "vt_commerce/trb3%";
+---------------------------+
| name                      |
+---------------------------+
| vt_commerce/trb3#p#my1990 |
| vt_commerce/trb3#p#my1995 |
| vt_commerce/trb3#p#p2     |
| vt_commerce/trb3#p#p3     |
+---------------------------+
4 rows in set (0.03 sec)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes partition detection (based on #p#):

Suggested change
ON SUBSTRING_INDEX(i.name, '#p#p', 1) = CONCAT(t.table_schema, '/', t.table_name) COLLATE utf8mb3_general_ci
ON SUBSTRING_INDEX(i.name, '#p#', 1) = CONCAT(t.table_schema, '/', t.table_name) COLLATE utf8mb3_general_ci

WHERE
t.table_schema = database()
GROUP BY
Expand Down
12 changes: 8 additions & 4 deletions go/vt/vttablet/onlineddl/executor.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -891,8 +891,8 @@ func (e *Executor) cutOverVReplMigration(ctx context.Context, s *VReplStream, sh

migrationCutOverThreshold := getMigrationCutOverThreshold(onlineDDL)

waitForPos := func(s *VReplStream, pos replication.Position) error {
ctx, cancel := context.WithTimeout(ctx, migrationCutOverThreshold)
waitForPos := func(s *VReplStream, pos replication.Position, timeout time.Duration) error {
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
// Wait for target to reach the up-to-date pos
if err := tmClient.VReplicationWaitForPos(ctx, tablet.Tablet, s.id, replication.EncodePosition(pos)); err != nil {
Expand Down Expand Up @@ -950,7 +950,11 @@ func (e *Executor) cutOverVReplMigration(ctx context.Context, s *VReplStream, sh
return err
}
e.updateMigrationStage(ctx, onlineDDL.UUID, "waiting for post-sentry pos: %v", replication.EncodePosition(postSentryPos))
if err := waitForPos(s, postSentryPos); err != nil {
// We have not yet locked anything, stopped anything, or done anything that otherwise
// impacts query serving so we wait for a multiple of the cutover threshold here, with
// that variable primarily serving to limit the max time we later spend waiting for
// a position again AFTER we've taken the locks and table access is blocked.
if err := waitForPos(s, postSentryPos, migrationCutOverThreshold*3); err != nil {
return err
}
e.updateMigrationStage(ctx, onlineDDL.UUID, "post-sentry pos reached")
Expand Down Expand Up @@ -1148,7 +1152,7 @@ func (e *Executor) cutOverVReplMigration(ctx context.Context, s *VReplStream, sh
}

e.updateMigrationStage(ctx, onlineDDL.UUID, "waiting for post-lock pos: %v", replication.EncodePosition(postWritesPos))
if err := waitForPos(s, postWritesPos); err != nil {
if err := waitForPos(s, postWritesPos, migrationCutOverThreshold); err != nil {
e.updateMigrationStage(ctx, onlineDDL.UUID, "timeout while waiting for post-lock pos: %v", err)
return err
}
Expand Down
Loading